The Federal Bureau of Investigation's Implementation of the
Laboratory Information Management System

Audit Report 06-33
June 2006
Office of the Inspector General


Executive Summary


The Federal Bureau of Investigation’s (FBI) laboratory is one of the largest and most comprehensive forensic laboratories in the world. The laboratory, which conducts over one million examinations of physical evidence annually, supports FBI investigations and provides forensic and technical services to other federal, state, and local law enforcement agencies. The FBI manages the flow of evidence through the laboratory in a largely paper-based process, with a limited “in-and-out” database that shows when an item enters the laboratory for testing, when analyses are performed, and when the item leaves the laboratory. However, the FBI cannot readily determine where the evidence is during the examination process and what work remains to be completed. The FBI also does not have the capability to generate statistical reports to help manage laboratory operations, such as how long it takes to examine evidence or where delays might occur.

To provide a modern information system that would allow the FBI to better track and manage evidence as it passes through the laboratory, the FBI’s Laboratory Division awarded a $1.6 million contract, with 4 additional option years for a total of $4.3 million, to JusticeTrax, Inc. in September 2003. The contract was to provide the FBI with JusticeTrax’s commercial off-the-shelf (COTS) Laboratory Information Management System (LIMS).1 The JusticeTrax LIMS was intended to allow the tracing and tracking of evidence using bar-code technology and provide a variety of reporting capabilities.

However, after many delays and extensive customization of the COTS LIMS, the system was unable to meet the FBI’s security requirements. In January 2006, the FBI notified JusticeTrax that the FBI had terminated the LIMS contract. In March 2006, the FBI and JusticeTrax agreed to a settlement that terminated the LIMS contract, resulting in an overall loss to the FBI of $1,175,015.

The OIG performed this audit to determine the status of the LIMS project, assess the Information Technology Investment Management (ITIM) processes and other management controls over the project, and determine the overall project costs. We found that the LIMS project was poorly managed. In addition, JusticeTrax was unable to meet the FBI’s more rigorous requirements implemented as a result of information technology (IT) system security breaches. With LIMS not able to obtain security certification and accreditation, coupled with other disadvantages such as the delayed implementation of a web-browser interface, the FBI terminated the contract. Although the FBI has now improved ITIM processes through its Life Cycle Management Directive (LCMD) and has established other improved controls, the failure of the system results in the FBI laboratory continuing to operate without an effective information system to adequately trace the flow of evidence through the laboratory.

Background

To track evidence arriving and leaving the laboratory, the FBI continues to use the Evidence Control System (ECS) that was created in 1978 and converted into a database in 1998. The FBI uses the ECS to record when an item of evidence is received by the laboratory for analysis, when analyses are performed, and when the item is released by the laboratory back to its originator. In comparison to the ECS’s limited database, a modern laboratory information system can provide a much greater level of functionality, including: the ability to trace evidence throughout the analysis process; Internet capabilities that allow external agencies to review and request information about evidence they have submitted; extensive reporting, workload analysis, and responses to ad-hoc querying; and data searching regarding the disposition of evidence.

FBI’s LIMS Project

In 1998, the FBI’s Laboratory Division hired a contractor to develop requirements for a more functional information system. However, the implementation of such a system was not fully funded until the Laboratory Division reprogrammed money from its own projects to fund the development in 2002. By this time, the system requirements needed to be upgraded. In February 2003, the FBI issued a Request for Proposal (RFP) for a laboratory information management system.

The FBI received six responses to the RFP. Cost and technical committees comprised of personnel from the FBI’s Finance and Laboratory Divisions evaluated the proposals. In September 2003, the FBI awarded JusticeTrax, Inc., of Mesa, Arizona, a $4.3 million firm-fixed-price contract to provide its LIMS product to the FBI.2The FBI selected JusticeTrax because it submitted the lowest cost bid and had an exceptional technical evaluation. According to JusticeTrax’s proposed project plan, LIMS installation, training, and roll-out would be completed in December 2003, 90 days from the contract award.

Schedule Delays

Although JusticeTrax planned to install the LIMS software within 90 days of the September 2003 contract award, a number of problems arose: (1) JusticeTrax’s president was a foreign national and thus not eligible to be involved in the development of the software for the FBI; (2) all JusticeTrax personnel lacked security clearances; and (3) although extensive software customization was required to meet FBI requirements, the LIMS used an outdated programming language that made modifying the software difficult and time-consuming.

The RFP for the information system stated that non-U.S. citizens may not have access to or be involved in the development of any Department of Justice IT system. By signing the contract or commitment document, the contractor agreed to this condition, even though the JusticeTrax president was not a U.S. citizen. However, after a security assessment, the FBI determined the risk was low and decided to continue with JusticeTrax. In April 2004, the JusticeTrax president signed a non-disclosure agreement to not access or assist in the development, operation, management, or maintenance of the FBI’s LIMS. In September 2004, 1 year after the contract was signed, the JusticeTrax president became a U.S. citizen and the non-disclosure agreement was rescinded.

Another obstacle to the timely implementation of the LIMS system was the lack of security clearances for JusticeTrax employees. The background investigations to obtain security clearances took from 3 to 8 months.

The third problem was the FBI’s numerous customization requests to tailor LIMS to the FBI’s specific needs. The customization was a slow process because the JusticeTrax LIMS relies on an aging code format, Visual FoxPro.3 While Visual FoxPro is outdated, it is still compatible with today’s technology. However, according to FBI personnel, Visual FoxPro is difficult and slow to customize compared to newer programming languages. While the extent of customization was the main obstacle, having to use the old code increased the delays.

FBI’s Project Controls

The FBI had no management control structure in place for LIMS such as establishing firm cost, schedule, technical, and performance benchmarks. The FBI also did not have a specific IT project manager for the LIMS project. Instead, the FBI relied on two contracting personnel to oversee the project as part of their contract-related duties. However, about 4 months after the FBI awarded the LIMS contract, there was turnover in these two key positions.

The FBI awarded the LIMS contract prior to the development and implementation of the FBI’s Life Cycle Management Directive. However, upon the LCMD’s implementation in November 2004, the FBI required all IT projects to follow the LCMD and meet the requirements for the stage of development the project had achieved. In May 2005, over a year after the LIMS was to be implemented, the FBI’s Information Management Project Review Board (IMPRB), one of the FBI’s IT investment boards, reviewed the LIMS project. During this review, Laboratory officials explained that although there were delays in implementing LIMS, the system could function and JusticeTrax had completed training the system’s users. However, LIMS had not yet achieved all of the FBI’s requirements, such as being a web-based system, and it was unlikely that the project would pass the FBI’s certification and accreditation (C&A) testing to ensure the security of the system. FBI officials agreed that if the project could not pass C&A, then the project should be cancelled. An IMPRB member recommended that a Red Team be assembled to review the procurement and consider alternatives.4

The Red Team included members from the FBI’s Laboratory Division, Office of General Counsel, Office of the Chief Information Officer (CIO), Finance Division, and ITOD. The Red Team review began in July 2005, and the team presented its findings, conclusions, and recommendations to the FBI’s CIO in October 2005. The Red Team recommended terminating the JusticeTrax contract because the LIMS system could not pass C&A, and additional work would not rectify the security weaknesses. In addition to the lack of a web-browser interface, identified deficiencies included several security vulnerabilities related to the lack of auditable records, insecure transmission between client and server, and a technical architecture that did not meet chain-of-custody requirements. In lieu of LIMS, the Red Team suggested the FBI use a standard COTS workflow software package already licensed to the FBI.

The FBI’s CIO stated the LIMS contract was awarded before the FBI’s IT investment management controls were implemented, and that LIMS is an example of the success of the FBI’s new ITIM processes because the problems with the project were quickly identified for resolution based on the IMPRB review.

Certification and Accreditation

The C&A program is the FBI’s management control for ensuring the adequacy of computer system security. The FBI’s Security Division tests the security of all new IT systems and approves the C&A if it deems a system secure. The testing ensures that the FBI’s IT systems have an approved baseline security configuration and that the systems present little or no risk to FBI systems or data. The FBI required the C&A process to be completed and approval to operate the system be obtained from the Security Division before the LIMS system could be made operational. Although the RFP included the requirement for security to be part of the system, specific guidance on the LCMD C&A requirements had yet to be established at the time the contract was awarded and was not provided to JusticeTrax until August 2005 when the FBI provided the results of the FBI Security Division’s LIMS Certification Test Report to JusticeTrax. The C&A testing delayed and then prevented the implementation of LIMS, and it ultimately led to the termination of the contract.

In September 2005, the Security Division began system testing, which resulted in a Certification Test Report identifying 14 security vulnerabilities in the LIMS system. In October 2005, the Security Division recommended against accrediting the system based on these high-risk vulnerabilities, which could not be mitigated due to the inherent design of the system. One weakness cited by the Security Division was the inability of LIMS to meet the confidentiality and integrity requirements for protecting evidentiary or grand jury data. The certifier recommended against granting an approval to operate. Because of these critical security flaws, the FBI determined that LIMS could not be used.

Contract Termination

The FBI became aware of delays and deficiencies with developing the LIMS system early in the contract period. While the LIMS software is functional, it has major deficiencies for FBI use, including the lack of a web-browser interface and numerous security vulnerabilities. Although the FBI and JusticeTrax signed the contract in September 2003, with the project to be implemented in 90 days, delays resulted in no-cost extensions through December 2005.

In December 2004, the FBI issued a Show Cause Notice to JusticeTrax stating that it failed to meet the deadline for the initial implementation of the system.5 JusticeTrax responded that the delays resulted from requirements not immediately apparent in the contract and that it did not have detailed information regarding the C&A process and what would be tested. Early in 2005, the FBI issued a letter to JusticeTrax stating the results of the initial security review of the LIMS system during the C&A testing process and identifying security risks that had to be corrected before further certification testing could proceed.

In October 2005, the FBI issued a Cure Notice to JusticeTrax stating that the LIMS system was not able to successfully pass the FBI’s Security C&A Testing.6 In the Cure Notice the FBI identified two outstanding concerns, the lack of auditable records (known as administrative shares) and the lack of a fully functional web-browser interface. JusticeTrax tried to resolve the security concerns, including the lack of auditable records, but the FBI’s Security Division found that the actions taken did not adequately resolve the concerns. JusticeTrax intended to work on the web-browser interface at a later date. However, in its response to the RFP, JusticeTrax had committed to providing the web-browser interface by early 2004.

At the end of October 2005, the FBI issued a Stop-work Order to JusticeTrax, and in January 2006 issued a contract termination letter.7In March 2006, the FBI and JusticeTrax agreed to terminate the contract for the convenience of the government. The FBI agreed to pay JusticeTrax an additional $523,932, and the contractor waived any claims arising from the contract.

In addition to considering other COTS workflow management systems to meet its information management needs, we recommend that the FBI consider systems being developed by other Department of Justice components. For example, we found that the Drug Enforcement Administration (DEA) and the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) are both working on laboratory information systems.

Costs

The base-year budget beginning September 2003 for the JusticeTrax contract was $1.6 million, with a total contract budget of $4.3 million including four additional 1-year contract options. Prior to the Red Team’s decision to recommend terminating the LIMS contract, the FBI paid JusticeTrax a total of $856,219. We reviewed and verified that all expenses were supported by invoices. Consistent with the contract, the FBI Laboratory Division purchased hardware from JusticeTrax, including bar-coding equipment, totaling $205,136. The equipment purchased can be used within the laboratory separate from the LIMS system.

In January 2006, the FBI ended the LIMS project, and in March 2006 the FBI and JusticeTrax agreed to terminate the contract for the convenience of the government. The FBI agreed to pay a settlement of $523,932 to the company in addition to the money already spent on developing the system and obtaining hardware. Therefore, the FBI spent a total of $1,380,151 on the project. With only the hardware usable, the FBI lost $1,175,015 on the unsuccessful LIMS project.

JusticeTrax’s Observations

During our fieldwork, we met with JusticeTrax officials to discuss their perspective on the LIMS contract. In the opinion of the officials, the failure of the LIMS project was due to the FBI’s lack of communication, information sharing, and resources. Also, JusticeTrax said the FBI should have provided a champion, or advocate, to ensure the success of the project. Finally, JusticeTrax stated that the FBI held JusticeTrax to requirements that were not in the contract. JusticeTrax acknowledged the contract included a provision for security but said it had no details about the C&A requirements. We agree with JusticeTrax that the FBI did not include specific details in the contract on how to meet the C&A requirements.

Conclusion

The failure to implement the LIMS system and the resulting loss of nearly $1.2 million in the attempt should be attributed to both the FBI and JusticeTrax. The project began before the FBI had established its ITIM processes, and those subsequent processes helped identify problems with the project that ultimately led to terminating the contract before losing additional money. The FBI did not do its homework before awarding the contract, including adequately identifying and assessing the risks in selecting JusticeTrax when the company’s COTS LIMS product had to be vastly modified. The FBI had a responsibility to not only ensure that JusticeTrax understood the system requirements, but also that JusticeTrax had the technical capacity to fulfill the requirements. The FBI did not adequately document for JusticeTrax the security requirements for certification and accreditation of the LIMS software and, to the extent security requirements evolved, did not clarify those changes through contract modifications.

The FBI should have assessed the problems and delays inherent in requiring major modifications to tailor a COTS system, especially one based on an outdated code. Firmly managed schedule, cost, technical, and performance benchmarks would have raised warning signs earlier in the project and perhaps led to resolution much more rapidly. Among the FBI’s weaknesses was the lack of established IT management processes when the project began and the failure to designate a LIMS project manager to oversee the implementation of the project. Also, two key contracting positions experienced turnover within months after the contract award.

Because JusticeTrax did not provide cleared personnel to work on the system and its president was not a U.S. citizen, JusticeTrax contributed to the early delays in getting the project started. It was incumbent upon JusticeTrax to meet all FBI requirements for the system, including mandatory security protections and a web-browser capability. However, JusticeTrax is correct in that some requirements were unknown at the start of the project. JusticeTrax’s use of outdated code also made modifications difficult and time-consuming. JusticeTrax did not properly assess its ability to perform the work required to adapt its system to operate in the FBI environment. In addition, while JusticeTrax intended to make its system web-based, the delays in the project prevented that before the contract was terminated.

Because JusticeTrax was unable to address unacceptable security vulnerabilities, the FBI terminated the LIMS contract. The FBI’s Laboratory Division continues to lack a modern system to track evidence through the laboratory and otherwise manage its laboratory operations. It remains difficult to determine the location and status of evidence at any given point in time or to determine how long the process is taking. We believe the FBI should consider adopting a COTS workflow system for its laboratory information system or an acceptably secure system used by another federal law enforcement entity, such as the Drug Enforcement Administration or Bureau of Alcohol, Tobacco, Firearms and Explosives, if it meets the FBI’s needs.

We agree with FBI officials who stated that the FBI’s LCMD should prevent problems such as those encountered with LIMS if the processes are applied as intended with detailed requirements for the contracting process, management oversight boards, and other controls to ensure troubled projects are identified sooner and remedied.

OIG Recommendations

We make three recommendations for the FBI to help ensure the FBI’s laboratory meets its need for an information management system. The recommendations are summarized below.

  • Consider whether a COTS workflow system or laboratory information management system currently in use or under development within the federal government will meet the needs of the FBI laboratory.
  • Ensure that any future laboratory information management system follows the FBI’s LCMD and is overseen by an experienced IT project manager.
  • Establish cost controls to ensure that training or other expenses are not incurred prematurely in the development of a successor to the LIMS project.



Footnotes
  1. The JusticeTrax product is known as LIMS-plus, but we refer to the system as LIMS throughout this report.

  2. The contract included a base year award of $1.6 million and four additional 1-year option contracts. The base year was September 2003 to September 2004. The contract also included cost-reimbursable delivery orders to convert the legacy ECS data to the new LIMS-plus system.

  3. Visual FoxPro, first developed by Fox Software in 1984, is a programming language used to develop database applications.

  4. Red Teams review and advise on FBI IT projects that miss cost, schedule, or performance thresholds.

  5. A contracting agency sends a Show Cause Notice to the contractor when problems occur. The notice includes a description of the problems and a timeframe for resolving the problems.

  6. A cure notice specifies to the contractor the problems requiring correction and establishes a timeframe for doing so.

  7. According to the Federal Acquisition Regulation, situations may occur during contract performance that cause the government to order a suspension of work, or a work stoppage. A Stop-work Order may be issued in any negotiated fixed-price or cost-reimbursement supply, research and development, or service contract due to advancement in the state-of-the-art, production or engineering breakthroughs, or realignment of programs.



« Previous Table of Contents Next »