On March 16, 2006, the Federal Bureau of Investigation (FBI) announced that it had awarded a contract to Lockheed Martin Services, Incorporated (Lockheed Martin) to develop the Sentinel information and investigative case management system in 4 phases. The cost of the four phases of the Lockheed contract was $305 million, and the FBI estimated that it would cost an additional $120 million to provide various contractor support and staff the FBI’s Sentinel Program Office, with the total estimated cost of Sentinel at $425 million. The initial schedule for the Lockheed Martin contract calls for all phases to be completed in December 2009.

The Sentinel project, which uses commercial off-the-shelf components, is intended to provide the FBI with an electronic information management system, automated workflow processes, search capabilities, and information sharing with other law enforcement agencies and the intelligence community. The FBI Director has stated, “Sentinel will strengthen the FBI’s capabilities by replacing its primarily paper-based reporting system with an electronic system designed for information sharing. Sentinel will support our current priorities, including our number one priority: preventing terrorist attacks.”¹

Sentinel follows the FBI’s unsuccessful 3-year, $170 million effort to develop a modern investigative case management system called the Virtual Case File (VCF) as part of the FBI’s Trilogy information technology (IT) modernization project. The VCF, and now Sentinel, was intended to provide the FBI with a modern system so that the existing obsolete Automated Case Support (ACS) system could be retired. As detailed in the Office of the Inspector General’s (OIG) February 2005 audit report on the FBI’s Trilogy project, the VCF project failed for a variety of reasons, including poorly defined design requirements, lack of mature Information Technology Investment Management (ITIM) processes, and poor management continuity and oversight.²

The Sentinel contract, awarded to Lockheed Martin through a Government-Wide Acquisition Contract (GWAC), is a cost-plus-award-fee contract that uses task orders to complete work for each phase of the project.³ While this type of contract proved problematic under Trilogy, we have found that the FBI has made considerable progress in establishing controls and processes required to adequately manage a major IT development project such as Sentinel and to bring it to a successful conclusion – if the processes are followed and controls are implemented as intended.

The OIG performed this audit of the Sentinel project at the request of the FBI Director and congressional appropriations and oversight committees. This audit is the second in a series of audits that the OIG intends to conduct, as Sentinel progresses, to evaluate the progress and implementation of Sentinel. The first audit, issued in March 2006, assessed the FBI’s pre-acquisition planning for and controls over Sentinel.

The objective of this second audit was to determine: (1) the progress the FBI has made in resolving the concerns identified in our first report on the planning for Sentinel, and (2) if the contract with Lockheed Martin and the FBI’s ITIM processes and project management are likely to contribute to the successful implementation of Sentinel. Our future audits will examine the progress of Sentinel over its four phases and assess whether cost, schedule, performance, and technical benchmarks are being met.

Background of Sentinel

A major objective of the FBI’s IT modernization project is to replace the FBI’s antiquated ACS system. During a variety of OIG reviews over the past several years, we reported that ACS uses outmoded technology, is cumbersome to operate, and does not provide necessary workflow and information-sharing functions.

The FBI expects that Sentinel will provide it with a web-enabled case management system that includes records management, workflow management, and evidence management; and records search and reporting capabilities, all of which will replace its current paper-based case management system. The FBI intends to implement Sentinel in four phases over 45 months, with each phase providing distinct capabilities until the overall project is completed in December 2009. The FBI expects to complete each of the phases in 12 to 16 months, with the phases overlapping by 1 to 2 months. For example, Phase 2 will begin about 2 months before Phase 1 is completed.

According to the FBI, the four phases will provide the following capabilities.

• Phase 1 will provide the web-based Sentinel portal. Initially, the portal will allow access to ACS data and eventually to data in the new case management system. It will also include a case management “workbox” that will summarize a user’s workload (the case files an agent or analyst is working on), and provide automatic indexing in case files according to person, place, or thing.

• Phase 2 will begin the transition to a paperless case records system by providing electronic case document management and a records repository. A workflow tool will support the movement of electronic case files through the review and approval process, while a security framework will provide access controls and electronic signatures.

• Phase 3 will provide a new Universal Index (UNI), which is a database of people, places, or things that relate to a case. Expanding the number of attributes in the system will enable more precise searching and will enhance agents’ ability to “connect the dots” among cases.

• Phase 4 will implement Sentinel’s new case management and reporting capabilities, including the management of tasks and evidence. During this phase, Sentinel will be connected to ACS, data on closed cases will be migrated from ACS to Sentinel, and the process to retire ACS will begin.

We reviewed the progress the FBI has made since our March 2006 report, the requirements of the Sentinel contract, the FBI’s application of its ITIM processes through its Life Cycle Management Directive (LCMD), and the controls the FBI has established over the Sentinel project to help avoid the problems the FBI encountered with the Trilogy project.

We found that the FBI has resolved most of the concerns we identified in our first Sentinel audit, although some aspects of those concerns as well as some new concerns identified in our current audit bear continued monitoring. Specifically, the FBI has made progress in: (1) establishing cost tracking and control processes, (2) implementing an Earned Value Management (EVM) system to help measure progress toward project baselines, (3) developing plans for the Independent Verification and Validation (IV&V) of the system software to ensure it will operate as intended, (4) developing information sharing capabilities, and (5) hiring more Program Management Office (PMO) staff.

Among the areas warranting continued monitoring by the FBI, the OIG, and other oversight entities are the: (1) funding of the Sentinel project and the effect on the FBI’s operations or other FBI projects of any reprogramming of funds that might be required (2) accuracy of the estimated cost of the project, (3) availability of contingency plans for identified project risks, and (4) completion of Sentinel PMO staffing.

In sum, the project is still in its early stages and has not yet reached the most difficult phases. However, we believe that the processes the FBI has established to manage and control the Sentinel project – if implemented and carefully followed as Sentinel develops – can provide reasonable assurance that Sentinel can be successful and that any deviations from cost, schedule, technical, or performance baselines can be identified.

Sentinel Contract

The FBI awarded to Lockheed Martin a cost-plus-award-fee contract through a National Institutes of Health government-wide acquisition contract (GWAC).⁴ Actual work under the contract will occur project phase by project phase through task orders.⁵ The cost of the task order for Phase 1 of Sentinel is $57 million. According to the contract, the FBI may exercise options for $248 million to cover three additional phases of the project plus operations and maintenance. Therefore, the total contract with Lockheed Martin could total $305 million. According to the contract, Lockheed Martin can also be rewarded for meeting established goals in four areas: project management, cost management, schedule, and technical performance. The award fee cannot exceed [redacted] percent of the $232.4 million total development costs for Sentinel, or approximately $26 million, and will be allocated across the four areas based on risk. This type of contract and award fee structure is common for large government IT projects.

In our 2005 report on the FBI’s Trilogy project, we described our concerns with the cost-plus-award-fee contract as it was implemented by the FBI in that project. The cost-plus-award-fee contract used for Trilogy did not: (1) require specific completion milestones, (2) include critical decision review points, and (3) provide for penalties if the milestones were not met. With regard to the Sentinel contract, the FBI is establishing clear milestones and requiring critical decision review points. If the contractor does not meet its milestones, it will be penalized by loss of the award fee.

Progress in Addressing the OIG’s Past Concerns

The FBI has made good progress in addressing the concerns we identified in our March 2006 audit report. As we describe in the following sections, our audit found that although some concerns remain, the FBI has: (1) hired or selected staff to meet current vacancies for the Sentinel PMO and has had management stability, (2) required that Sentinel meet a new joint Department of Justice and Department of Homeland Security information sharing standard, which will allow Sentinel to communicate with other systems built to the standard, (3) established an EVM system to monitor the Sentinel project’s costs and schedule, (4) established layers of review, approval, and reporting for Sentinel spending, and (5) completed plans for the IV&V of Sentinel’s software to ensure it will perform as intended.

Staffing

The FBI has made progress in staffing the Sentinel PMO since our first report. Of a total planned staff of 73, as of October 2006, 65 positions had been filled compared to 51 in March 2006.⁶ The FBI said it has intentionally delayed filling six of the vacant positions until the second phase of Sentinel. Two other positions remain vacant, an intelligence analyst and a planner. The Chief of the Business Management Unit said the PMO has taken steps to expedite hiring, including interviewing applicants who had applied to an FBI-wide job announcement for computer scientists.

Information Sharing

In our March 2006 report, we expressed concerns that the FBI was focused on sharing information within the FBI but had not paid sufficient attention to Sentinel’s ability to share information with other law enforcement and intelligence agencies’ systems. Since that report, the FBI has focused more attention on external information sharing needs and has been coordinating with the Departments of Justice and Homeland Security and other federal entities, including the Drug Enforcement Administration, Immigration and Customs Enforcement, and the Office of the Director of National Intelligence. Sentinel will be built to meet the standards of the new National Information Exchange Model, a joint Department of Justice and Department of Homeland Security standard, which is also supported by the Director of National Intelligence. When finalized, the standard will become the government-wide standard for any new law enforcement and intelligence systems being developed.⁷ However, the standard is still evolving, and Sentinel’s design may have to be modified as the standard evolves.

Earned Value Management

EVM is a tool that measures the performance of a project by comparing the variance between established cost, schedule, and performance baselines with what is actually taking place. These variances are measured periodically to give project managers a perspective on the status of a project and an early warning if a project is heading for trouble. EVM reporting is an important risk-management tool for a major IT development project.

The FBI and Lockheed Martin have implemented EVM systems in accord with Office of Management and Budget (OMB) requirements to track and validate Sentinel project costs throughout the life of the project. In addition to data provided by Lockheed Martin, the FBI’s EVM system relies on cost data provided through invoices from support services contractors and the FBI’s Budget Execution and Analysis Reporting System, which extracts purchase order information from the FBI’s Financial Management System and generates reports on funds requested, amounts approved and spent, and obligations that have not yet entered the FBI’s overall Financial Management System. The FBI is required to report to the OMB any net cost or schedule variations by the FBI and the contractor that meet a reporting threshold.

The FBI is using the EVM system to help manage project risks by providing an early warning of unexpected costs and problems that could delay Sentinel’s completion. We are monitoring the FBI’s EVM reporting to identify any unexplained growth in overall project costs or any schedule delays. Three early EVM reports indicated some variances, but the variances were due to estimating errors by the contractor, which have been corrected.

Cost Tracking and Controls

The OIG’s prior reviews of the Trilogy project found that the FBI lacked an effective, reliable system to track and validate the Trilogy project’s costs. In our current audit work, we found that in addition to EVM reporting, the FBI has established controls to help ensure that Sentinel expenditures are authorized in advance and that items are verified when delivered and validated when invoiced. For example, the FBI has developed a system of overlapping responsibilities for the oversight of Sentinel’s costs that include: accounting, auditing, and budget monitoring by the FBI’s Finance Division; detailed tracking of Sentinel’s costs by the Office of the Chief Information Officer’s IT Financial Management Unit; and tracking and controlling program and development costs and developing policies and procedures for processing invoices, requisitioning and procuring equipment, reviewing contractor time charges, and resolving discrepancies by the Sentinel PMO Business Management Unit. We believe that the tracking systems and controls the FBI has implemented will allow the FBI to be better monitor and control project costs for Sentinel than was the case under Trilogy.

Documentation Required by ITIM Processes

Although the FBI had established sound IT investment management processes through its Life Cycle Management Directive (LCMD), we noted in our last report on Sentinel that two key plans had not yet been developed because the project design had not been completed: IV&V and the system security plan. The IV&V process provides an independent control to monitor the testing of the system software and ensure it functions as intended. The FBI’s Chief Information Officer (CIO) recently told us that the FBI awarded its IV&V contracts to eight vendors and that it had awarded a task order to Booz Allen Hamilton to monitor Lockheed Martin’s testing of the system software during the development of the Sentinel system.⁸

A system security plan is also critical to help ensure that Sentinel will meet the FBI’s security standards and can be certified and accredited for use within the FBI’s operating environment. The CIO recently told us that the security plan has been drafted and is in the approval process.

In accordance with the FBI’s LCMD, the final design for the first phase of the Sentinel project will occur in October 2006. However, because Lockheed Martin will be using off-the-shelf components to develop Sentinel, the complication and risk of the project design should be lessened, although configuring all of the components into one seamless system will remain a greater challenge. The FBI stated that it will conduct future planning, including requirements verification, prior to the initiation of subsequent phases in order to solidify the design and deliverables for each phase

Current Concerns

The FBI has made strides in resolving most of the concerns discussed in our March 2006 audit report, although some aspects of those concerns remain. Also, in our current audit work we have identified additional concerns that warrant continued monitoring by both the FBI and the OIG. One concern carries over from our previous report – the possibility of a reprogramming of the FBI’s non-IT funds to cover fiscal year 2007 Sentinel expenses, which may have an adverse affect on the FBI’s mission capabilities. In addition, we were unable to validate the FBI’s cost estimate for Sentinel, and we found that the FBI lacks contingency plans for all of the highly rated project risks it has identified.

Project Funding and Reprogramming

We found that the FBI faces uncertainty over the source of the approximately $150 million the FBI says it needs in fiscal year (FY) 2007 to continue the Sentinel project. The President’s FY 2007 budget request includes $100 million for Sentinel, and the FBI would need an additional $56.7 million to bridge the gap between the requested funds and its FY 2007 requirements for Sentinel. The FBI expects to have about $50 million remaining from the first phase of Sentinel and prior year unexpended balances from other sources. Moreover, the FBI’s CIO recently told us that an FY 2007 appropriation of less than $100 million would be cause for concern and could result in an unanticipated level of reprogramming of FBI resources to fund the Sentinel project. In our judgment, any reprogramming significantly above $50 million will require the FBI to carefully consider which programs and activities will be affected and how to monitor the overall impact on the FBI’s mission.

As we reported in our first Sentinel audit, various FBI managers told us that a second reprogramming of FBI funds similar in size to the $97 million reprogramming that occurred in November 2005 could erode the FBI’s mission capability in counterterrorism, cybercrime, and other important operational areas. Therefore, until the funding issues are addressed, we remain concerned about the impact that reprogramming significant amounts of non-IT funds to support Sentinel would have on other critical FBI priorities.

With respect to total project costs, the FBI CIO told us that he stands by the FBI’s estimate that the full cost of Sentinel will be $425 million, with $305 million to cover work by Lockheed Martin on a variety of task orders and an additional $120 million to cover costs such as staffing the FBI’s Program Management Office, contractor support, and management or risk reserve for contingencies. Training costs are included in the Lockheed Martin portion of the estimate, which was a concern we noted in our last Sentinel report when the FBI had not yet developed a complete cost estimate for its training plans.

Cost Estimates

We reviewed the processes used to derive the $425 million cost estimate for the Sentinel project, noted some inconsistencies in the process and the results, and concluded that the estimate is a rough approximation of Sentinel’s overall costs. The estimate is, in our view, tentative given the variances in the supporting cost estimates and the inherent complexity of estimating costs for a major IT system before the design is finalized.

In examining the underlying estimates for the overall project cost, we are unclear as to whether the initial cost estimate accurately included the project’s operations and maintenance costs through FY 2011. We found that some portions of the estimate provide costs for 2 years, while other portions include costs for 3 years. Another estimate showed significant disparities in Lockheed Martin’s labor costs. Variations in the estimates of Sentinel’s projected costs demonstrate the difficulty of estimating the cost of such a complex information technology project at its outset.

Because of these estimation difficulties, and because the project is in its early stages, we could not validate the FBI’s overall estimate of $425 million for Sentinel, and we believe that the ultimate cost could be lower or higher. We noted that the overall management reserve for the project – a budgeted amount to cover any unanticipated expenses – currently amounts to about 15 percent of Sentinel’s development costs. The Sentinel Program Manager told us that based on his experience, an 11 percent reserve would be adequate (the difference amounts to about $8.6 million). The FBI expects to adjust the amount of the reserve so that over the length of the project the reserve will equal 11 percent of the development cost. As the FBI finalizes Sentinel’s design and gains experience with actual project costs, the FBI should regularly update its estimate of the overall project costs to keep Congress and the Department informed. In addition, we intend to continue to monitor the cost of the project as it progresses.

In addition to the Sentinel project cost estimates, we identified costs that could be considered as associated with Sentinel but are separate projects and therefore not included as part of Sentinel’s projected $425 million cost. For example, the implementation of Sentinel will require changes to the FBI’s National Name Check system. In response to a request from a federal, state, or local agency, the National Name Check Program queries FBI records to determine whether the person named in the request has been the subject of an FBI investigation or mentioned in an FBI investigation. The data system used by the program relies very heavily on the ACS system, which Sentinel is intended to replace. The estimated cost of updating the existing name check system to work with Sentinel is over [redacted]. In addition, the FBI has ongoing agency-wide security efforts that will benefit Sentinel. If these separate projects were included as Sentinel costs, the $425 million cost estimate could be at least $25 million higher.

The FBI’s position is that these separate projects are enterprise-wide endeavors that will benefit the FBI’s overall IT structure, including Sentinel but also many other FBI systems. The CIO and the Sentinel Program Manager contend that these other projects were initiated on their own merits, would be undertaken regardless of Sentinel, and their costs ought not be considered as Sentinel costs. While we agree that these Sentinel-related projects may not be direct Sentinel costs, in our view the scope of the Sentinel project would be larger if it was not supported by these other investments.

Risk Management

The purpose of risk management is to assist the project management team in identifying, assessing, categorizing, monitoring, controlling, and mitigating risks before they negatively affect a program. A risk management plan identifies procedures used to manage risk throughout the life of the program. Risks are categorized by severity and identified as either open or resolved. Open risks are tracked until resolved.

The FBI has created a list of 20 risks associated with the Sentinel project that it is monitoring. While the FBI’s establishment of a risk management program is a positive step, contingency plans, and the triggers for activating such plans, currently exist for only three risks – including only one of the top five risks. The Program Manager told us that in some cases it is difficult to develop a contingency plan before the FBI’s preventive actions mitigate the likelihood or severity of the risk or before the risk. He explained that the focus is on preventing problems that would rise to the level of requiring mitigation, and that if a problem occurs, a corrective action will be developed. He also told us that many risks are temporary and as a project phase progresses, the risk may become moot and is closed. However, we believe the FBI should have a plan for risks that have the potential to result in a significant cost, schedule, or performance deviation from the project baselines.

With respect to currently identified project risks, we view the FBI’s ability to successfully migrate data from the antiquated ACS system to Sentinel as a potentially significant challenge. If the migration were to fail or be seriously delayed, the FBI would need to try maintaining its legacy ACS system with all of its flaws. An inability to migrate the ACS data would also result in a Sentinel system that builds its data from the present day forward, without the benefit of years of investigative data compiled in the old system. Further, should ACS cease to be maintainable, that data could effectively be lost. The Sentinel Program Manager told us that the task of “cleaning” and reconciling the ACS data for migration into Sentinel is not technically difficult and the FBI plans to use an available software tool for that purpose. However, he pointed out that it will take a significant amount of work to accomplish. He also said that as a preventative measure intended to eliminate any delays in the overall project due to data cleansing, the FBI plans to cleanse data in the phase preceding the phase in which the data will be transferred to Sentinel.

Another potential risk is the extent to which Sentinel will actually use commercial-off-the-shelf software modules as intended. A high degree of customization of the software could result in increased costs and schedule delays. The Program Manager told us that the components for Sentinel are all off-the-shelf and little or no customization is anticipated. However, the key task will be configuring Sentinel’s various applications – such as the workflow, document management, searching and reporting, and electronic signatures – to all work together. The Program Manager noted that Lockheed Martin has successfully configured similar systems in other major projects, using some of the same software modules, including one at the Social Security Administration.

IT Investment Management Processes

In November 2004, the FBI established its IT investment management processes through its LCMD, which it has since refined and is applying to the Sentinel project. The LCMD governs all aspects of an IT project, including planning, acquisition, development, testing, and operations and maintenance. The FBI’s LCMD contains four overlapping components: life cycle phases, control gates, project level reviews, and key support processes.

The LCMD has established nine phases that occur during the development, implementation, and retirement of IT projects: (1) concept exploration, (2) requirements development, (3) acquisition planning, (4) source selection, (5) design, (6) development and testing, (7) implementation and integration, (8) operations and maintenance, and (9) retirement.⁹ As of August 2006, the Sentinel project had passed through the first four life cycle phases and is currently in the fifth phase – Design.

During the life cycle phases, specific requirements must be met for the project to obtain the necessary FBI management approvals to proceed to the next life cycle phase. The approvals occur through control gates, where FBI management boards meet to discuss and approve or disapprove a project’s progression to future phases of development or implementation. The control gate reviews provide management control and direction, decision-making, coordination, confirmation of successful performance of activities, and determination of a system’s readiness to proceed to the next life cycle phase. Decisions made at each control gate review dictate the next step for the IT program or project and may include: allowing an IT program or project to proceed to the next segment or phase, directing rework before proceeding to the next segment or phase, or terminating the IT program or project.

The Sentinel project has received management approval for the first two of the LCMD control gates: the system concept on July 15, 2005, and the acquisition plan on July 29, 2005. As of September 2006, the Sentinel program had not requested or received approval for the third control gate. According to the Sentinel Program Manager, Phase 1 of the Sentinel project is scheduled to pass through Control Gate 3, the Final Design Review, in late October 2006. Depending upon the development model employed, programs or projects may pass through the control gates more than once. Because Sentinel is being developed in phases, and the contractor must provide a system design for each phase, the project will pass through Control Gate 3 four times.

Conclusions

By establishing stronger IT investment management processes and an array of monitoring and control mechanisms, the FBI has positioned itself to better manage the Sentinel project and avoid the problems that occurred in the Trilogy and VCF projects. However, FBI officials agree this does not mean that the development of Sentinel is risk-free. While the FBI has corrected or alleviated most of the concerns we raised in our March 2006 audit report on Sentinel, several areas warrant attention to avoid potentially serious problems as the project progresses:

• the ability to fully fund the project and, if required, reprogram funds without adversely affecting other FBI mission-critical operations,

• monitoring and adjusting as necessary the estimates of total project costs,

• developing contingency plans for high-risk areas that could affect project costs, schedule, or performance, and

• completely staffing the PMO.

In future audits, we will continue to monitor Sentinel’s progress and whether the project is meeting the cost, schedule, technical, and performance baselines.

OIG Recommendations

In this second Sentinel audit, we make five recommendations to the FBI to help ensure the success of the Sentinel case management system and manage project costs. The recommendations are:

• Ensure the management reserve is based on an assessment of project risks for each phase and for the project overall.

• Periodically update the estimate of total project costs as actual cost data is available.

• Complete contingency plans as required by the Sentinel Risk Management Plan.

• Ensure that the independent verification and validation process is conducted through project completion.

• Complete hiring as soon as possible for the vacant PMO positions needed during the current project phase.

1. FBI Press Release entitled FBI Announces Award of Sentinel Contract

2. The Department of Justice, Office of the Inspector General, The Federal Bureau of Investigation’s Management of the Trilogy Information Technology Modernization Project, Audit Report Number 05-7, February 2005.

3. An award fee is a financial incentive provided to a contractor based on the contractor’s performance.

4. The development contract under the GWAC is cost-plus-award-fee. However, all materials are cost-plus-fixed-fee and travel is cost reimbursable only.

5. A task order specifies the services required and the negotiated terms at which they will be provided, subject to the terms of the contract.

6. The number of filled positions includes three candidates who had accepted positions and were in the process of being hired.

7. The Sentinel statement of work, which was developed prior to the release of the draft National Information Exchange Model, requires Sentinel to be built to the Global Justice XML Model. However, the Sentinel Program Manager said that Sentinel’s design will ultimately conform to the new National Information Exchange Model standards.

8. At the time our audit, the specific IV&V activities for Sentinel had not been determined. However, IV&V may include oversight of program management processes and assessments related to the development contractor’s performance.

9. The life cycle phases are not to be confused with the Sentinel project’s four development phases.