|
Pursuant to the OIG’s standard audit process, the OIG provided a draft of this audit report to the FBI on December 20, 2004, for its review and comment. The FBI’s January 26, 2005, response is included as Appendix 7 of this final report. The FBI concurred with the nine recommendations in the audit report and also provided comments regarding three general issues in the report. Our analysis of the FBI’s response to these three issues and to the nine recommendations is provided below.
FBI’s General Comments
- In its response, the FBI disagreed with the OIG’s assertion that "the continuing lack of an effective case management system hinders the FBI’s capability to perform its critical national security mission." According to the FBI, such statements "overlook the substantial IT improvements that directly support our counterterrorism mission."
In our report, we note that there are national security implications when the FBI must continue to rely on an archaic case management system, ACS, which hampers analysts and agents from adequately searching and sharing investigative information. We believe the ability to timely receive, link, analyze, and share investigative leads and case information is essential to help prevent future terrorist attacks. As documented in prior OIG reports, the ACS is an inefficient and burdensome tool that does not contain all of the investigative lead and case information available within the FBI. We found that many FBI employees do not fully use ACS because of its deficiencies. Further, as stated in the final report of the National Commission on Terrorist Attacks Upon the United States (the 9/11 Commission): "Finally, the FBI’s information systems were woefully inadequate. The FBI lacked the ability to know what it knew: there was no effective mechanism for capturing or sharing its institutional knowledge."
In responding to the audit, the FBI points to improved capabilities resulting from the Trilogy infrastructure as well as the availability of various other FBI systems, databases, and analytical tools to facilitate intelligence analysis and sharing. We do not dispute the FBI’s progress in these areas, although we did not examine these other systems because they were not the subject of our audit. We did
analyze the FBI’s progress in implementing the VCF case management application, which was intended to replace ACS and serve as the FBI’s sole case management system. The VCF was intended to contain all investigative lead and case file information in a paperless system. Due to the FBI’s failure to develop a workable and complete VCF after spending more than 3 years and $170 million, the FBI continues to lack a modern case management system containing complete and accessible investigative lead and case information.
In responding to several prior OIG reports that identified substantial deficiencies in the ACS system, the FBI stated that the deficiencies found within the ACS system would be corrected through implementation of the VCF.22 However, the VCF still remains under development without a clear timetable of when it will be implemented and at what cost. The FBI’s response recognizes the urgent need to replace ACS, stating that doing so remains the FBI’s top IT priority.
As pointed out in the prior OIG reports and as noted in this report, ACS is based on obsolete technology and has been extraordinarily difficult to use. Furthermore, because agents did not always use the ACS, the system does not contain complete investigative case data and records. Also, due to its obsolete technology, the ACS is not readily or timely used to acquire and link information across the FBI. Although the FBI recently has reduced the number of steps needed to use the ACS from 12 to 3, the system remains archaic. According to a 9/11 Commission Staff Statement on information sharing within the FBI:
Although there are many explanations for the failure to share information internally, one of the most common in the FBI’s outdated information technology, the Automated Case Support system in particular. It employs 1980s-era technology that is by all accounts user-unfriendly. More troubling, the system cannot be used to store or transmit top secret or sensitive compartmented information. For a variety of reasons, significant information that gets collected by the FBI never gets uploaded into the Automated Case Support system, or it gets uploaded long after it is learned.
In its response, however, the FBI cites other IT initiatives to support its assertion that national security remains uncompromised by the delay in deploying the VCF. For example, the FBI’s response states that the newly created Investigative Data Warehouse allows FBI employees greater access to a variety of counterterrorism information. Although we did not include the Investigative Data Warehouse in our audit and have not evaluated its capabilities, we believe that the VCF case management system would have many features that a data warehouse does not. The VCF was intended to be the backbone of the FBI’s information systems, replacing the FBI’s paper case files with electronic files. Case data in the VCF could be approved electronically, and the electronic files would be available throughout the FBI immediately as entered, and various lead and case information could be associated for analysis. The VCF not only would allow dissemination of this information immediately across the FBI, but would “push” information to agents and analysts who have a need to receive it timely. The Investigative Data Warehouse requires that information be affirmatively uploaded into the warehouse so that employees can then conduct searches. Updates must be made to the information in the warehouse, and the information in the warehouse is not available immediately or universally, unlike information in the VCF.
In sum, the Investigative Data Warehouse, while perhaps a useful tool, does not manage case workflow and does not substitute for an effective case management system. Consequently, the FBI continues to lack critical tools necessary to maximize the performance of both its criminal investigative and national security missions.
- In its response, the FBI clarified the relationship between the VCF and the proposed Federal Investigative Case Management System (FICMS). During our audit, the FBI offered evolving explanations of the status of the VCF effort and its relationship with FICMS. In June 2004, the FBI had announced a two track corrective plan for the VCF, which included the development of what the FBI called the Initial Operational Capability (IOC) and the Full Operational Capability (FOC) for the VCF. The IOC represented a pilot test consisting of an electronic approval process. Employees were to create electronic documents on standard FBI forms, and these forms were to be sent for approval electronically from within the VCF IOC system. The goal of the pilot test was to determine the efficiency and effectiveness of the electronic approval process. Once the initial testing and evaluation were completed, a determination was to be made as to how the FBI would proceed with future system deployments.
The FOC was intended to resolve outstanding issues, provide advice, and reevaluate requirements for the VCF. Additionally within the FOC, the processes for document creation, approvals, leads management, and evidence management were to be developed to determine the best phased approach to obtain the necessary components for the FOC.
However, FBI managers told the OIG audit team in November 2004 that the FICMS would replace the VCF and that requirements for the FICMS were to include user activities and processes for creating and approving documents and managing investigative leads, evidence, and cases. These requirements were expected to come from the VCF efforts already undertaken, and the VCF was described as the leading driver for the FICMS. Additionally, the creation of a Request for Proposal (RFP) from vendors for the FICMS was to include a full case management capability for the FBI. At the time, the OIG was left with the impression that, as the name implies, the FICMS was to be an actual system that could adopt aspects of the stalled VCF effort.
In commenting on a draft of this report, the FBI suggested that the FICMS represented a "framework" that would lead to a case management system. Whether the FBI intended the resulting case management system to be part of the FICMS effort or a separate follow-on effort remained unclear from the FBI’s earlier comments.
However, in its formal response to this report, which is contained in Appendix 7, the FBI describes the VCF and the FICMS as two different projects proceeding in parallel. The FBI states that it is continuing to pursue the VCF through development of an implementation plan that will result in the deployment of a fully functional case and records management system. According to the FBI, it hired Aerospace Corporation to evaluate currently available software products to determine if they meet the FBI’s requirements. The FBI also asked Aerospace to evaluate the adequacy of the VCF as delivered by SAIC to determine what might be salvaged from that effort.
In its response, the FBI also states that the FICMS project will develop a "framework that will govern development of DOJ and Department of Homeland Security (DHS) investigative case management systems to ensure the high level of inter-agency compatibility needed to facilitate information sharing." The response states that the FICMS represents a broad blueprint for such systems and does not replace the VCF. As FICMS is being developed, the FBI plans to continue working on a case management system, whether it continues to be called the VCF or is given a new name.
As a result of this information, we have clarified the report to reflect the FBI’s current description of the relationship of FICMS and the VCF.
- In its response, the FBI agrees that its oversight of the Trilogy contracts should have been stronger, but it notes that the FBI had recognized its limitations and outsourced elements of the project to other entities, in accordance with guidance from the Department. The response discusses the role of FEDSIM in acting as the contracting office for key Trilogy contracts and of others in assisting the FBI with managing the project. The FBI’s response also describes steps the FBI has taken to improve its IT management processes as part of what it called a "top-to-bottom reorganization" of FBI IT resources.
We agree that the responsibility for the contracting and management of the Trilogy project was shared among several parties, including the FBI and FEDSIM. However, the FBI selected FEDSIM and its Millenia contracting mechanism because the process could be expedited without the need to first fully develop the design requirements for the Trilogy project. Consequently, the FBI assumed certain risks in taking this path in attempting to expedite Trilogy’s implementation. That approach created problems because the management processes requiring the development of schedule, cost, technical, and performance targets for Trilogy were not in place. Moreover, the FBI, as the consumer, had the primary responsibility for managing the project and overseeing those retained to assist in that management. However, we agree that having the FBI act as the project integrator was unrealistic, given the FBI’s lack of expertise and experience to perform such a key function of ensuring that all three components of the project would mesh.
Additionally, while the FBI stated in its response that while it recognized its management limitations at the start of the project, it did not take the necessary steps to correct identified problems. Even internal FBI reports on Trilogy issued by the FBI’s Inspection and Criminal Justice Information Service Divisions repeatedly pointed to the lack of both requirements and project oversight, including improper scheduling and cost estimates.
The OIG agrees that the FBI has taken a number of positive steps to improve its IT management recently. We have noted key improvements, including those cited by the FBI in its comments, beginning on page 33 of this report. However, these steps do not mitigate the significant deficiencies in the FBI’s management of the Trilogy project, and the problems created by the delay in deploying the VCF and the associated cost increases. Moreover, while the FBI states that the VCF’s Initial Operating Capability (IOC) was executed using the FBI’s new approach to IT management "on schedule and within budget," the IOC is only one step towards a fully functioning case management system. The FBI still has significant work before it can develop and deploy a modern case management system.
Finally, in response to the FBI’s "Conclusion" paragraph in its formal response, we agree that the FBI has taken steps to modernize its IT, particularly its infrastructure. But the FBI has not succeeded in its goal to replace the antiquated ACS system with a fully functional and effective case management system, despite more than 3 years in development and $170 million. This critical IT project affects the FBI’s ability to perform its mission as effectively and efficiently as it should, and implementing an effective case management system should remain the FBI’s top IT priority.
Response to Recommendations:
- Resolved. In response to this recommendation, the FBI stated that it agrees that it must replace the ACS as quickly and as cost-effectively as feasible. The FBI reports that it has contracted to examine the VCF software, as well as available off-the-shelf software, to determine whether these products could be adapted to meet the FBI’s needs. The FBI also points out in its response that technological innovation may have overtaken the FBI’s original vision for the VCF, and that new products may exist to meet the FBI’s need that did not exist when Trilogy began. We agree that the FBI must consider new and available technologies. In light of the FBI’s agreement that it must replace the obsolete ACS system, this recommendation is resolved. The recommendation can be closed when the FBI demonstrates that a fully functional case management application replacing the ACS has been developed and deployed, and is being utilized throughout the FBI.
- Resolved. This recommendation is resolved based on the FBI’s agreement to devote all necessary resources to support a new case management system. This recommendation can be closed when the FBI provides evidence that it has reprogrammed the resources necessary to meet its need for a functional case management system.
- Resolved. This recommendation is resolved based on the FBI’s agreement to take steps to establish critical design requirements for the case management system before initiating a new contract and ensuring that the contractor fully understands the requirements and has the capability to meet them. This recommendation can be closed when the FBI provides documentation demonstrating that the critical design requirements for the case management system have been established before initiating a new contract, and that the future contractor fully understands the requirements and has the capability to meet them.
- Resolved. The FBI agrees with this recommendation, stating that the VCF project “suffered in part from runaway scope.” The FBI response also states that it has learned from the VCF experience and that it has “adopted a process that will avoid a recurrence.” This recommendation is resolved based on the FBI’s agreement to incorporate the development efforts for the VCF into the requirements for any successor case management system. This recommendation can be closed when the FBI provides documentation demonstrating that the development of requirements for a future case management system incorporates the VCF development efforts for any successor case management system.
- Resolved. The OIG acknowledges that the FBI has issued a Life Cycle Management Directive (LCMD), which resulted in the closing of a broader FBI-wide recommendation in December 2004 from a prior OIG report. However, the recommendation in this audit report addresses specifically the FBI Inspection Division’s review of the Trilogy project’s finances, rather than the broader new LCMD process. This recommendation can be closed when the FBI provides documentation that the financial systems used for tracking Trilogy project costs have been validated and improved to ensure complete and accurate data on project costs.
- Resolved. This recommendation is resolved based on the FBI’s agreement with the recommendation. The FBI’s issuance of the LCMD resulted in closing a general FBI-wide recommendation in December 2004 from a prior OIG report. However, the current recommendation can be closed when the FBI provides documentation demonstrating that upcoming Trilogy-related contracts have applied the LCMD and include defined requirements, progress milestones, and penalties for deviations from cost, schedule, performance, and technical baselines.
- Resolved. This recommendation is resolved based on the FBI’s agreement with the recommendation. Again, the OIG acknowledges that the FBI has issued a LCMD. However, this recommendation deals specifically with both existing and future Trilogy-related contracts. We also note that a recommendation from the OIG’s report on the FBI’s IT Investment Management regarding the need for establishing baselines has not been closed because the FBI has not provided evidence that it has established final cost and schedule baselines for the Trilogy project. Further, the Full Operational Capability phase of the VCF project is ongoing, and additional contracts will be required to complete Trilogy with a functional case management system. This recommendation can be closed when the FBI provides documentation demonstrating that it has established management controls and accountability to ensure that baselines are met for current and upcoming Trilogy-related contracts.
- Resolved. This recommendation is resolved based on the FBI’s agreement to apply ITIM processes to future IT projects, including additional Trilogy-related projects. Because key ITIM-related recommendations have not yet been closed from a prior report and the VCF portion of Trilogy remains in flux, we have included this recommendation to ensure the FBI provides appropriate follow-up as it develops a framework for a case management system and to completes and deploy such a system. This recommendation can be closed when the FBI provides documentation that it has applied ITIM processes to both existing and upcoming Trilogy-related projects.
- Resolved. This recommendation is resolved based on the FBI’s agreement with the recommendation and its stated progress toward completing an Enterprise Architecture. The OIG previously closed a recommendation from a prior OIG report that called for the FBI to continue its effort to establish an Enterprise Architecture. Due to the clear need for the FBI to have a blueprint to effectively manage its overall IT, including Trilogy, the current recommendation is for the FBI to monitor, or track, the development of its Enterprise Architecture to ensure that it meets its September 2005 deadline. This recommendation can be closed when we receive documentation of an effective monitoring system for ensuring that the FBI’s Enterprise Architecture is completed and implemented on schedule.
- Problems with the ACS were discussed in the following OIG reports: The Handling of FBI Intelligence Information Related to the Justice Department's Campaign Finance Investigation (July, 1999); An Investigation of the Belated Production of Documents in the Oklahoma City Bombing Case (March 2002); and The FBI's Management of Information Technology Investments (December 2002).
|