The Federal Bureau of Investigation's Implementation of the
Laboratory Information Management System
Audit Report 06-33
June 2006
Office of the Inspector General
|
Inadequate Management of the Laboratory Information Management System Project
JusticeTrax proposed installing its LIMS software within 90 days of the September 2003 contract award. However, a series of delays began soon after the contract was awarded. One of the reasons for the delays was that JusticeTrax’s president and chief shareholder was a foreign national, which created security concerns requiring an evaluation. Also, the firm lacked IT personnel in Quantico, Virginia with security clearances to work on the project. Moreover, extensive customization of JusticeTrax’s off-the-shelf system was needed to meet the FBI’s requirements, but the LIMS software used an outdated programming language that made customization difficult and slow. In January 2004, 4 months after the LIMS contract was awarded, the FBI’s contracting officer, who is responsible for the overall implementation of the contract, and the contracting officer’s technical representative (COTR), who directly monitors the contract, were both replaced due to personnel changes in the FBI’s Laboratory Division. Both of the individuals replaced were involved in the initial development of the information management project, including the system requirements. Shortly afterward, a series of problems arose in the implementation of the LIMS project. In March 2004, the president of JusticeTrax informed the new COTR that he was a foreign national. While the former COTR was aware of the president’s status prior to awarding the contract, he did not view the lack of U.S. citizenship as a problem because he believed the president was not going to be involved in the coding of the system. Additionally, the contract did not specify work to be performed at the classified level, even though the LIMS database was to include classified and other sensitive information such as grand jury data. The newly appointed COTR stated that she believed a risk existed with the project because the LIMS would include sensitive information and the JusticeTrax president might be directly involved in the LIMS development. Additionally, the RFP included a Department of Justice mandated provision prohibiting non-U.S. citizens from having access to or being involved in the development of any Department IT system. After evaluating the security risk, the Laboratory Division, the Security Division, the Financial Division, and the Office of General Counsel agreed that the JusticeTrax president being a foreign national was a low risk; therefore the FBI decided to continue the contract. In our view, it was predictable that because JusticeTrax is a small organization of about 20 employees, the president would need to be involved in managing the project. The FBI’s security concerns led the JusticeTrax president to sign an agreement in April 2004 not to be involved in the development, operation, management, or maintenance of LIMS. The COTR followed up on her concerns, believing that the sensitivity of the LIMS and the data it would hold required additional assurances. As a result, the FBI performed a Community Acquisition Risk Center (CARC) threat analysis. In August 2004, the FBI’s Counterintelligence Division issued a CARC Company Threat Analysis memorandum stating JusticeTrax was eligible to perform the contract. Finally, in September 2004, 1 year after the contract was signed, the JusticeTrax president became a U.S. citizen, and the recusal agreement was rescinded. The foreign ownership issue should have been addressed by the FBI during the pre-acquisition phase of the project. Because of the secure nature of the LIMS system, the FBI should have taken steps to ensure that all of the potential contractors were familiar with the security requirements of the system and of the Department of Justice’s mandate prohibiting non-U.S. citizens from being involved in the development of a Department system. As a result of not taking measures to ensure that the potential contractors for the project met these requirements, the COTR had to take actions that delayed the project’s implementation after the contract had been awarded. Another obstacle to the implementation of the LIMS was a lack of personnel with security clearances at JusticeTrax to work on the project in Quantico, Virginia. JusticeTrax did not provide the FBI with security clearance information on its personnel until almost 2 months after the contract award, and the security clearance process took an additional 3 to 8 months. This meant that JusticeTrax could not begin implementing LIMS until early 2004, after the basic product was to have been deployed in accordance with JusticeTrax’s schedule. A third problem required the basic LIMS product to have extensive customization to meet the FBI’s requirements, resulting in further delays. According to an FBI official in May 2005, the COTS product was 95-percent customized. In essence, the FBI’s LIMS would no longer be a COTS product but an FBI-unique system. This process was slow because the LIMS software relies on a dated code format, Visual FoxPro, requiring more intensive coding than more modern formats.14 Visual FoxPro is considered an outdated form of code, but it is still compatible with today’s technology. While the FBI’s requests for a customized system caused delays, the old code used in the LIMS software exacerbated these delays. FBI Attempts to Correct Project Delays The FBI became aware of the delays and deficiencies with LIMS early in the project. While the LIMS software was functional, it had security vulnerabilities and did not yet meet the FBI’s requirement for a web-browser interface. Although the basic LIMS was to be implemented in 90 days (December 2003), the delays in the project resulted in two no-cost extensions, with the base year slipping 15 months. In 2004, it became increasingly apparent to the FBI that full implementation of LIMS appeared unlikely, even though JusticeTrax had already trained laboratory personnel in operating the system. On December 6, 2004, the FBI issued a Show Cause Notice to JusticeTrax stating that JusticeTrax failed to meet the deadline for implementation.15 The notice also provided JusticeTrax with a list of failed tasks including: (1) ensuring system security, (2) migrating legacy ECS data to LIMS, and (3) passing acceptance testing of the system. The Show Cause Notice stated that although the LIMS was delivered, the system had to pass security testing as well as a acceptance testing. On December 9, 2004, JusticeTrax responded that the delays the FBI detailed in the Show Cause Notice were requirements not immediately apparent in the contract. JusticeTrax also stated that neither it nor FBI staff had any detailed information regarding the process and what was to be tested. We also noted that the FBI did not provide JusticeTrax with specifics of how to meet the certification and accreditation (C&A) requirements. On February 11, 2005, the FBI issued a letter to JusticeTrax stating the initial security review of LIMS during the security testing process identified risks that had to be corrected before further testing could proceed. The FBI awarded the LIMS contract 14 months prior to the implementation of its LCMD, a critical initiative that provided the FBI with sound and structured IT investment management processes to help ensure successful IT projects. Once the LCMD was implemented, the FBI required all ongoing IT projects to follow the LCMD processes for the projects’ current stages of development. The FBI’s Chief Information Officer (CIO) stated the FBI’s IT investment review boards began reviewing ongoing projects that predated the LCMD. The review boards examined high-dollar, high-risk projects first, concentrating on the top 30 to 40 projects. LIMS was not reviewed for about 6 months because the project did not meet the criteria for priority review. On May 20, 2005, the FBI’s Information Management Project Review Board (IMPRB), one of the review boards established in the LCMD, reviewed the LIMS project. During the review, laboratory officials described the history of LIMS, including the laboratory’s need for an information management system and the delays experienced in trying to implement the LIMS project. At the time of the review, JusticeTrax had already trained the FBI’s would-be LIMS users. Although LIMS was functional, it had not yet been brought online because it did not meet all of the FBI’s security requirements. The review board also learned that although JusticeTrax’s basic LIMS was a COTS system, the software had undergone extensive modification so that about 95 percent of the FBI’s version of LIMS was based on custom code. A member of the IMPRB doubted the project would pass the FBI’s security certification and accreditation testing. The FBI’s Security Division provides C&A, authorizing the deployment and operation of a system, only if it deems a system secure based on its testing and evaluation. FBI officials agreed that if LIMS could not pass C&A, then the project should be cancelled. The IMPRB expressed additional concerns about project risks, including the fact that the Visual FoxPro code used for JusticeTrax’s LIMS is old technology and whether the small firm could adequately support the system into the future. The IMPRB recommended that a Red Team be assembled to review the LIMS project and consider alternative approaches.16 The FBI formed a LIMS Red Team in July 2005 with representatives of the Laboratory Division, the Office of General Counsel, the Office of the CIO, the Finance Division, and the ITOD. The team held meetings from July through October 2005 and presented its findings, conclusions, and recommendations to the FBI’s CIO in October. From the beginning of its review, the Red Team identified serious technical deficiencies with LIMS, which included:
The Red Team recommended terminating the JusticeTrax LIMS contract because the system could not pass C&A. The team also suggested that BizFlow, a product the FBI is licensed to use, might be a suitable alternative.17 According to the Red Team, BizFlow has the capability to integrate workflows with information management, create and replicate forms, provide formatted and customizable reports, and handle bar-coding equipment. Certification and Accreditation As the IT review board predicted, C&A testing led to the termination of the LIMS contract. As part of the LCMD, C&A is the FBI’s management control for ensuring the adequacy of computer systems’ security. The C&A testing and evaluation process is designed to ensure the FBI’s systems are designed securely and remain secure throughout their life cycle. If the Security Division’s testing and evaluation determine that a new system is secure, the Security Division provides accreditation and approves the system to enter into operations within the FBI’s IT architecture. The LIMS RFP required security to be part of the system. However, due to several high-profile espionage-related security breaches within the FBI, the FBI strengthened C&A requirements after the September 2003 award of the LIMS contract. The specifics were not available to JusticeTrax until the FBI provided the results of the FBI’s Security Division’s Certification Test Report to JusticeTrax in August 2005. The report stated that LIMS failed testing in four key areas: (1) password storage, (2) auditing capability, (3) control of grand jury evidence, and (4) shared directory (information sharing outside the laboratory). In September 2005, the Security Division began testing for a second Certification Test Report after JusticeTrax provided patches to the LIMS software based on the first report. The FBI performed tests to ensure that the system was at an approved baseline security configuration and that the system presented little or no risk to FBI systems or data. However, the Security Division identified 14 vulnerabilities according to the ease of exploiting the system. The 14 findings ranged from “requires expert-level knowledge to exploit the vulnerability to gain access to the system” to “does not require tools or expert-knowledge to exploit and gain access to the system.” The significance level, meaning impact if exploited, for all 14 vulnerabilities was rated high.18 By October 2005, it became clear to the FBI that LIMS would not meet the FBI’s security and other requirements. The FBI gave JusticeTrax an opportunity to correct the system’s deficiencies, but those efforts were unsuccessful. Eventually, after 28 months of effort, the FBI terminated the LIMS contract. On October 4, 2005, the FBI issued a Cure Notice to Justice Trax stating that the LIMS software application was not able to successfully pass the FBI’s Security C&A Testing.19 In the Cure Notice, the FBI identified two outstanding concerns: (1) system security, and (2) the lack of a fully functional web-browser interface. JusticeTrax attempted to correct the security flaws, but the FBI’s Security Division did not accept the corrections. JusticeTrax planned to provide the web browser at a later date. Based on the Certification Test Report and its finding that LIMS posed a very high security risk, the Security Division recommended on October 17, 2005, that LIMS not be accredited. The C&A process found that the system’s vulnerabilities could not be mitigated due to the inherent design of the software. Therefore, the certifier recommended against granting an approval to operate the system.20 At the end of October 2005, the FBI issued a Stop-work Order to JusticeTrax. According to the Federal Acquisition Regulation, situations may occur during contract performance that cause the government to order a suspension of work, or a work stoppage. A Stop-work Order may be issued in any negotiated fixed-price or cost-reimbursement supply, research and development, or service contract due to advancement in the state-of-the-art, production or engineering breakthroughs, or realignment of programs. In January 2006, the FBI issued a contract termination letter to JusticeTrax. In March 2006, the FBI and JusticeTrax agreed to terminate the contract. The FBI agreed to pay JusticeTrax an additional $523,932, and the contractor waived any claims arising from the contract. CIO’s Observations The FBI’s CIO noted to the OIG that the LIMS contract was awarded before the FBI’s IT investment management controls were implemented through the LCMD. He stated that in his opinion, the LIMS project demonstrates the success of the FBI’s LCMD because the FBI terminated the project after the IMPRB review and the C&A process showed that the LIMS system’s serious deficiencies could not be corrected. The CIO noted that the LCMD process now requires project managers to come before review boards so that the FBI’s divisions no longer manage IT projects in isolation. The CIO stated that the controls provided by the LCMD help to detect problems earlier in a project’s life cycle. JusticeTrax’s Observations JusticeTrax officials stated that in their opinion, the failure of the LIMS project was due to the FBI’s lack of communication, information sharing, and resources. They also stated that the FBI did not provide a “champion,” that is, an FBI official who would work to ensure the success of the project. Finally, JusticeTrax officials said that the FBI insisted on requirements, especially regarding system security, that were not specified in the contract. Although the contract included a provision for security, JusticeTrax officials stated that details for the C&A requirements were never provided. After reviewing the requirements in the contract, we agree that the security requirements were too general to provide enough detail on how to meet the requirements. Laboratory Division’s New Review Process In addition to the FBI’s LCMD, the Laboratory Division had established in October 2005 a division-wide Major Acquisition Review Committee (MARC) to strengthen the oversight of the Laboratory Division’s acquisitions, including IT investments. The MARC will assist Laboratory managers to ensure that Laboratory projects adhere to all Department of Justice and FBI requirements for sound project and financial management. The MARC mirrors the LCMD, but covers all projects rather than only the IT projects covered by the LCMD. The purpose of the MARC is to:
The base year of the LIMS contract was September 2003 to September 2004, with a $1.6 million budget. The base year could be extended by four 1-year contract options, bringing the total contract budget to $4.3 million. Prior to the Red Team’s decision to recommend termination, the FBI paid JusticeTrax a total of $856,219 in personnel, training, and equipment costs. This included $205,136 in hardware that the Laboratory Division purchased from JusticeTrax that can be used by the FBI laboratory separate from LIMS.21 During our audit, we reviewed and verified that all expenses were supported by invoices. When the FBI terminated the LIMS contract, the FBI and JusticeTrax agreed to a settlement of $523,932. Therefore, the FBI spent a total of $1,380,151 on the LIMS contract as shown in the table below. FBI Payments to JusticeTrax
The FBI wasted $1,175,015 on the LIMS project: $1,380,151 paid to JusticeTrax less the reusable equipment totaling $205,136.22 The FBI Laboratory Division’s need for an information management system remains. To fulfill the need, the FBI is considering other COTS systems. For example, the Red Team that evaluated JusticeTrax’s LIMS recommended Bizflow software, which is used for workflow and information management. The FBI purchased Bizflow to use within the FBI in general, but the software has not yet gone through C&A testing or other LCMD processes. Alternative solutions might also be found in other Department of Justice components’ or other federal agencies’ laboratory information systems. For example, the FBI has obtained information from the Drug Enforcement Administration on its ongoing project to acquire a system for managing evidence. The Bureau of Alcohol, Tobacco, Firearms and Explosives is also expected to deploy a new laboratory information system in the spring of 2006 that has been under development for over 5 years. We concluded that the FBI’s inability to implement the LIMS system and its loss of nearly $1.2 million in the attempt was a shared responsibility between the FBI and JusticeTrax. The project began before the FBI had established its ITIM processes. When those processes were implemented, they helped identify problems with the project that ultimately led to terminating the contract before losing additional money. Still, the FBI did not do its homework before awarding the contract, including adequately identifying and assessing the risks in selecting JusticeTrax, and in vastly modifying the company’s COTS LIMS product. The FBI had a responsibility to not only ensure that JusticeTrax understood the system requirements, but that JusticeTrax also had the technical capacity to fulfill the requirements. In addition, the FBI did not adequately document for JusticeTrax the security requirements for certification and accreditation of the LIMS software. To the extent security requirements evolved, those changes should have been made clear through contract modifications, if necessary. The FBI also should have identified the citizenship problem of the JusticeTrax president, foreseen the security clearance requirements for JusticeTrax personnel, and assessed the problems and delays inherent in requiring major modifications to tailor a COTS system — especially one based on an outdated code. A firmly managed schedule, and cost, technical, and performance benchmarks, would have raised danger signs early in the project and perhaps led to resolution much more rapidly. Among the FBI’s weaknesses were: (1) the lack of established IT management processes to ensure a sound project and identify problems early, and (2) not designating a project manager to oversee the project. Also, two key contracting personnel, both of whom were involved in the development of the LIMS requirements, left the project only 4 months after the contract was awarded. This lack of continuity and institutional knowledge likely contributed to the poor outcome of the LIMS project. Because JusticeTrax did not provide personnel with security clearances to work on the system, and its president was not a U.S. citizen, JusticeTrax contributed to the early delays in starting the project. It was incumbent upon JusticeTrax to meet all FBI requirements for the system, including mandatory security protections. However, JusticeTrax has a legitimate point that some details of the requirements were unknown at the start of the project. JusticeTrax’s use of outdated code made modifications difficult and time-consuming, and JusticeTrax did not properly assess its ability to perform the work required to adapt its system to operate in the FBI environment. Also, while JusticeTrax intended to make its system web-based, the delays in the project prevented that before the contract was terminated. Because JusticeTrax was unable to mitigate unacceptable security vulnerabilities, the FBI had no choice but to terminate the LIMS contract. As a result, the FBI’s Laboratory Division continues to lack a modern system to track evidence through the laboratory and otherwise manage its laboratory operations because it is difficult to determine the location and status of evidence at any given point in time or to determine how long the process is taking. We believe the FBI should consider adopting a COTS workflow system for its laboratory information system or an acceptably secure information management system used by another federal law enforcement entity. We agree with FBI officials who stated that the FBI’s LCMD should prevent problems such as those encountered with LIMS if the processes are applied as intended with detailed requirements for the contracting process, management oversight boards, and other controls to ensure troubled projects are identified sooner and can be remedied. We recommend that the FBI:
Footnotes
|
| « Previous | Table of Contents | Next » |