RESULTS OF THE CATS USER SURVEY
The follow-up work we conducted in response to the AAG/A's second request consisted of a survey of the 11 CATS participant agencies' asset forfeiture program officials to determine their satisfaction with CATS. In October 1995, we conducted interviews with each of the 11 program managers during which time we distributed the user survey. All 11 surveys were completed and returned. We also conducted interviews with the asset forfeiture managers from the IRS, a former participant agency, and the USCS, which chose not to participate in the CATS project. We focused our survey on asset forfeiture program managers rather than on those who actually enter and retrieve data from the system (end users) because we were interested in decision makers' perceptions and concerns. Furthermore, the system training contractor surveyed end users in FY 1994 and was planning to conduct another similar survey in FY 1996. It is our understanding, based on our participation at a CATS user conference, that end users were generally pleased with the system's capabilities.
Our CATS user survey obtained general participant agency information and covered six specific issues identified as areas of concern by agencies' representatives whom we interviewed during our initial review. The six specific issues were: (1) implementation and installation schedules; (2) implementation and installation coordination; (3) data conversion efforts; (4) participant agency legacy systems; (5) training; and (6) security. See Appendix III for a copy of our user survey instrument. Complete participant agency responses are found in Appendices IV through X and are summarized below. The inspection team did not independently validate the information provided in response to the user survey. The views expressed by the participant agencies' asset forfeiture program managers are their own and are subject to their interpretations, perceptions and experiences with the CATS program. We met with members of the AFMS on November 8, 1995, June 26, 1996 and July 11, 1996 to share the results of our user survey and to update the information wherever possible.
General Participant Agency Information
To obtain general agency background information, representatives responded to questions pertaining to the date when their agencies became participants of CATS, the planned date for full implementation for their agencies and what level of participation their agencies would have when the system was fully implemented.
Agency participation in the CATS project began in 1989. Participant agencies' representatives stated that once the system is fully operational, their level of participation is estimated to range from 27 users and two terminals in the EOAF-T to 500 users and 577 terminals in 178 offices in the USMS.
CATS Implementation and Installation Schedule and Coordination
During our status review, several agencies' representatives cited dissatisfaction with CATS installation and implementation scheduling and coordination. In the user survey, we asked if participant agencies were satisfied with the scheduling and coordination of CATS installation and implementation.
At the time of our survey, only the Criminal Division and the EOAF-T representatives stated that CATS was fully implemented in their agencies. These participant agencies had a combined total number of 37 users. The other respondents were not sure when full implementation would occur in their respective agencies. For instance, the USMS representative stated that, "No planned full implementation date exists to [the USMS'] knowledge, but the target date is by the end of calendar 1996." Additionally, the USPS representative stated that the date of full CATS implementation in his agency ". . .cannot be determined at this point due to delays in the installation of CATS terminals across the country."
All participant agencies' representatives stated that they received a general schedule for CATS implementation. However, several stated that the implementation schedule they received was not up to date; only identified the geographical sequence of installations, not specific installation dates; or the published schedule and subsequent site installation and implementation did not agree.
These responses indicate that there has been confusion among participant agencies regarding the implementation schedule for CATS and several agencies do not know when the system will be fully implemented. We believe that the apparent lack of a current implementation schedule with set, meaningful milestones makes it difficult for the AFMS and participant agencies to adequately coordinate and schedule CATS installation and implementation efforts.
In addition, the majority of agencies' representatives stated that CATS was not installed in a timely manner. They cited a number of problems for the delays, including obtaining RWAs from the GSA and the DEA's temporary withdrawal from CATS. The respondents attributed the majority of delays to the lack of coordination between the participant agencies, CATS contractors and the AFMS.
For example, the ATF representative stated that "Sometimes there does not seem to be any coordination between [the CATS contractors] and the [participant] agencies." The USSS representative stated that ". . .installations have not been coordinated satisfactorily." The USSS, in conducting background checks on CATS contractor personnel, found that some had criminal histories. The USSS, therefore, did not permit these contractor personnel access to its facilities. The CATS Project Manager stated that the AFMS made every effort to effectively coordinate installation. Some participant agencies' representatives stated that the AFMS continues to work with them to resolve these types of problems as they arise. The CATS Project Manager told us that an implementation schedule with set, determinable milestones was developed at the start of the CATS project. However, due to numerous problems experienced throughout the course of the project, including the need to develop a national telecommunications network, milestones were often not reached on schedule. As a result of this situation, she stated that it is no longer feasible to continue to use and adhere to a set implementation schedule.
Our user survey shows that the documentation provided to the participant agencies regarding CATS installation and implementation dates is confusing and a single schedule with firm completion dates and measurable milestones was not provided to participant agencies. Although several respondents stated that they have received revised installation and implementation information, it is our opinion that agencies are unclear as to the timing of site installations and full CATS implementation for their agencies. The AFMS provided us with a number of installation and implementation schedules, including one listing the agencies and geographical areas identified for installation. However, these schedules did not provide a specific timetable of events. We were told by the CATS Project Manager that for a number of reasons, including expanded agency requirements, redesign of the system and no Department-wide telecommunications system, the system installation and implementation schedule is constantly changing.
We recognize the need for some flexibility in any installation and implementation schedule. Nevertheless, in our opinion, without an installation and implementation schedule with measurable milestones, it is difficult for participant agencies to maintain a high level of commitment to the project. In addition, we believe that the lack of coordination among the agencies, CATS contractors and the AFMS is a continual source of dissatisfaction for participant agencies and the cause of system installation and implementation delays. If the coordination problems among the agencies, CATS contractors and the AFMS are not addressed and resolved, user dissatisfaction with the system may increase and agencies may reconsider the level of their participation in the system.
The CATS Project Manager told us that she believes it is unlikely that participant agencies will reconsider their participation in the CATS project. It is her opinion that the participant agencies should dedicate more resources to the installation phase of the project. She believes dedicating more resources to this phase of the project could facilitate more timely installation of CATS terminals.
CATS Data Conversion Efforts
During our status review, several participant agencies' representatives stated that they were experiencing difficulty converting asset data into CATS. During our user survey, we asked participant agencies' representatives to respond to questions identifying the percentage of total assets converted into CATS and explain any problems with the data conversion efforts. Participant agencies are currently in the process of converting asset data into the CATS data base. The FBI and ATF representatives stated that their agencies had entered 100 percent of their assets into the system as of October 1994, while the remaining agencies estimated that they have converted between 14 and 44 percent of their data.
Several participant agencies' representatives stated that they were experiencing problems that affected their ability to complete the conversion process. These problems included reliance on other agencies to enter information into CATS and the inability of the system to directly transfer agency legacy system information into the data base. Specifically, the USMS and the USPS stated that they are dependent upon other agencies accurately entering information into the system in a timely manner. The USMS stated that it ". . .has had much difficulty getting [the] DEA to enter or correct assets in CATS for conversion purposes" and this has affected the USMS' ability to complete the conversion process. Furthermore, the USSS stated that when the initial data conversion was completed, only four percent of the data in its legacy system was transferred into CATS. As a result, the USSS must reenter the assets directly into CATS.
In our opinion, problems experienced by agencies in their data conversion efforts have already significantly delayed, and are likely to delay further, the full implementation of CATS. The slow progress of data conversion into the system and the inability of the system to directly transfer certain agency data from the legacy systems into CATS is a cause for concern. The CATS Project Manager stated that data conversion from the participant agencies' legacy systems to CATS is an extremely time consuming and complex project. She stated that the conversion process involves incorporating a number of individual agency asset records from a number of legacy systems into one consolidated asset record in CATS. In addition, records entered into CATS must be validated. If data cannot be converted into CATS accurately and in a timely manner, we believe records in the system will be incomplete, use of the system will be significantly hindered, dissatisfaction with the system will increase and participant agencies will be forced to maintain legacy systems indefinitely.
The CATS Project Manager told us that the Assets Forfeiture Fund provides the funding for the DOJ participant agencies' legacy systems. She stated that if DOJ participant agencies continue to maintain their legacy systems once CATS is fully implemented and operational, the AFMS will no longer provide funding to the agencies to maintain these systems.
CATS Participant Agency Legacy Systems
The purpose of the CATS project is to provide a consolidated asset forfeiture system for use by all asset forfeiture agencies, thus eliminating the need for competing parallel systems. During our status review, several participant agencies' representatives stated that their agencies had an asset forfeiture tracking system prior to the development of CATS and that they would continue to run their systems parallel with CATS during and after full implementation of the consolidated system. We asked participant agencies' representatives to respond to questions pertaining to their legacy systems.
Ten representatives stated that their agencies had an asset forfeiture tracking system prior to the development of CATS and nine of these representatives stated that their agencies plan to continue operation of these legacy systems in parallel with CATS until the system is fully implemented. The DEA representative added further that the DEA will continue ". . .operation of its legacy reporting system in parallel with CATS until the system is. . .validated by [the] DEA."
The Criminal Division representative stated that the Criminal Division will continue to maintain its legacy system because assets logged onto their legacy system must be closed out on the same system. However, once all assets are closed out on the legacy system, the Criminal Division can use CATS exclusively. The FBI representative stated that it will continue to run its parallel system because the FBI legacy system has proven reliable and accurate, and the legacy system can hold sensitive information that cannot be entered into CATS. The USPP representative stated that although the USPP has access to CATS, it does not input or edit information in the system; i.e., the USPP has read-only authority on the system. As a result, the USPP must maintain its own legacy system to track USPP seizures.
Once CATS is fully operational, the USMS, Executive Office for United States Attorneys (EOUSA), INS and ATF representatives stated that their agencies will no longer continue to run legacy systems. The DEA, USPS and USSS representatives stated that their decision to discontinue use of their legacy systems will depend upon several factors including successful operation of CATS, timely software enhancements and the ability of CATS to meet the agencies' reporting needs.
We believe that maintaining legacy systems results in a duplication of data input for participant agencies and is not a cost-effective use of the Assets Forfeiture Fund. We also believe that if the CATS data base cannot meet participant agencies' requirements, managers will continue to maintain legacy systems.
During our status review, several participant agencies' representatives stated that the AFMS used a "train the trainer" approach to CATS training. Using this approach, agencies' representatives took part in CATS training courses conducted by the system training contractor. These courses were designed to provide participants with the knowledge, skills and abilities necessary to integrate CATS into their agencies. Upon completion of the training, participant agencies' representatives returned to their agencies as trainers to educate other CATS users. Agencies' representatives we interviewed during our implementation study believed that the "train the trainer" approach worked well; however, the representatives expressed concern with the timing of CATS workstation installation and the subsequent training sessions. As a result, we asked participant agencies to respond to questions pertaining to the timing of and their satisfaction with training.
The majority of agencies' representatives stated that training occurred between a few weeks and three to four months prior to workstation installation. Several representatives stated that as a result, their agencies' staff required retraining by the time workstations were installed and operational. In one case, the DEA representative stated that training was provided without any site preparation work having been completed in three divisions. Subsequently, DEA was ". . . faced with trained users that returned to their offices without CATS terminals available. Due to this situation, the DEA temporarily withdrew from CATS."
The Director, AFMS told us ". . .that the DEA took a timeout from the CATS project" and that the agency did not really withdraw from the project. He stated that the DEA underwent a number of organizational and personnel changes within the asset forfeiture program. In addition, he stated that the temporary withdrawal from the project was the result of internal communication problems in the DEA and the DEA not completing the required RWA process for some of its CATS sites. The Director, AFMS stated that the DEA decision to take a timeout from CATS was a ". . .unilateral action which had a trickle-down effect on all other agencies in that area." The CATS Project Manager stated that the DEA's temporary withdrawal had a significant effect on the overall CATS installation schedule because the DEA was a hub site. The Director, AFMS added that the AFMS has worked closely with the DEA to resolve this issue.
Nevertheless, the time-lapse between CATS workstation installation and CATS training has resulted in users being unable to work effectively and efficiently and requires their retraining. Therefore, training should coincide as closely as possible with workstation installation. The CATS Project Manager acknowledged the importance of closely coordinating CATS workstation installation and training and stated that the AFMS planned for training sessions to take place immediately following workstation installation. However, in some areas where workstation installation was delayed and where training facilities were scheduled in advance, training occurred despite the workstation installation delay. In some instances, this occurred to avoid cancellation fees for training locations not used.
Participant agencies' representatives told us during our status review that the final security classification of CATS was an area of concern because some agencies may be entering sensitive data into the system. We asked agencies' representatives to respond to questions pertaining to system security.
When asked to describe the information handled by CATS for each agency, the representatives stated that CATS handles information for their agencies ranging from requests for equitable sharing of assets by state and local law enforcement agencies to information pertaining to seizure, custody and disposal of assets. All representatives stated that their agencies were not entering sensitive data into CATS.
When asked to identify the protective measures necessary for asset information, representatives provided varied responses. Although it appeared that the use of a password and USERID was sufficient for some agencies, the DEA representative stated that his agency continues to have serious concerns regarding the accessibility of CATS to unauthorized users. Nevertheless, he stated that the DEA is generally satisfied with the data protection available on CATS. The ATF and USPS representatives stated that their agencies are not satisfied with the data protection available on CATS. The ATF representative stated that he did not feel participant agencies should have access to all CATS data, particularly ATF and IRS tax data.
Based on these responses, the agencies' representatives believe that the security classification of CATS is adequate, although not ideal. In the future, if participant agencies determine that it is necessary to enter sensitive data into the system, the security classification of the system as a whole will need to be reevaluated.
Based on information obtained during our status review and our user survey, we believe the AAG/A has reached a critical decision point regarding the future of the CATS program. The significant delays and continual revisions of the CATS installation and implementation schedule and the ever-increasing costs of the system are threatening the Department's ability to meet the Attorney General's directive to implement a consolidated asset tracking system.
We believe that the continual delays in system implementation, increasing system costs, the impact of the IRS withdrawal from CATS and the USCS' decision not to participate in the system have been detrimental to the success of CATS. The CATS feasibility study was completed in late 1990 and the initial projected system completion date was December 1992 at a cost of $24 million. However, full system implementation has not yet occurred and is now estimated for the Spring of FY 1997. The estimated costs have risen to $106.2 million. In the JMD Monthly Administrative Mangers' Report, dated June 1996, the AFMS estimated that 50 percent of the total number of planned CATS sites are installed and operational and 59 percent of planned workstations are installed and operational. During our review, we were unable to determine a complete and accurate status of CATS because implementation dates and costs continue to fluctuate. We question if CATS can be fully implemented and operational by the Spring of FY 1997. If the planned date for full implementation and operation continues to change, we believe participant agencies almost surely will reevaluate their level of participation in CATS and either withdraw from the system completely or, at a minimum, curtail their use of the system.
In our view, the following areas summarize the most critical issues affecting the future success of CATS:
· Two agencies have decided not to participate in CATS because, in part, the system does not meet their financial reporting requirements. The AFMS told us that the data is available in CATS, but it cannot be produced in the reports some agencies require. Until the issues pertaining to financial reporting requirements are resolved, we are concerned that CATS may not be able to meet the participant agencies' financial reporting needs.
· Despite assurances that the system security classification issue has been resolved, the documentation provided to us was not conclusive, in our opinion. The security classification status is not clear and at least one DOJ agency believes that further work is needed in this area.
· At the time of our review, data conversion was languishing and, in our judgment, the comprehensiveness of the CATS data base was in jeopardy. If all asset data is not converted into CATS in a complete and timely manner, agencies will continue to rely on their legacy systems. Continued use of the legacy systems contradicts the overall concept of a consolidated asset tracking system.
· Our interviews with participant agencies revealed that installation problems have overshadowed installation accomplishments. Many site installation problems resulted from poor coordination of contractors and insufficient site preparation. Given the history of installation delays, we believe that exhaustive efforts must be made to ensure that planning and execution of site installations are carefully coordinated among all parties. Although the AFMS stated that they are aware of the installation problems, and have recently taken some remedial actions, we are not certain sufficient steps have been taken to eliminate the problems of the past.
· The single largest factor affecting user satisfaction has been the continuous delays in the implementation of CATS. The projected system implementation date has slipped frequently from 1992 to 1997. We do not believe that the most recent estimated implementation date of the Spring of 1997 is credible given that about one-half of the approximately 754 sites must still be implemented, staff need to be trained and data needs to be converted. We believe that the date for full system implementation, as reported to the participant agencies and Department management, should be based on a candid and realistic assessment of the many complex tasks remaining and a reasonable implementation schedule. Moreover, we believe that AFMS cannot achieve this goal without the full cooperation of the participant agencies and their dedication of sufficient resources.
In our view, the success of CATS is contingent upon effective cooperation and coordination among the participant agencies. In September 1993, the DOJ and Treasury signed a memorandum of understanding agreeing to jointly support CATS. Since that time, the IRS has withdrawn from the project. The USCS, a large agency involved in the asset forfeiture program, has decided that it will not participate in CATS. Representatives of these agencies stated that their agencies will not participate in CATS because, among other reasons, the USCS is developing SEACATS, and CATS is not providing the type of reports required to comply with the Chief Financial Officers Act of 1990. In the opinion of IRS, CATS installation is significantly behind schedule and the system will not be available nationwide in the foreseeable future.
Given the breakdown of consolidation efforts between the Treasury and the DOJ, and the development of another asset tracking system by the USCS, we suggest the Department continue to meet with Treasury asset forfeiture representatives to reconcile the differences currently preventing final implementation of the consolidated multi-agency asset tracking system envisioned by Congress.