In March 2006, the Federal Bureau of Investigation (FBI) announced that it had awarded a contract to Lockheed Martin Services, Incorporated (Lockheed Martin) to develop the Sentinel information and case management system. The cost of the four phases of the Lockheed contract was $305 million, and the FBI estimated that it would cost an additional $120 million to staff and administer the FBI’s Sentinel Program Management Office (PMO), with the total estimated cost of Sentinel at $425 million. The initial schedule for the Lockheed Martin contract called for all phases to be completed in December 2009.

On June 19, 2007, the FBI announced that it had fully deployed Phase 1 of Sentinel to provide FBI employees with user-friendly, web-based access to information currently in the FBI’s antiquated Automated Case Support (ACS) system and improved search capabilities.¹ Phase 1 of Sentinel features a personal workbox, which summarizes a user’s cases and leads, and a squad workbox, which allows supervisors to better manage resources and make assignments.²

The Sentinel project integrates commercial off-the-shelf (COTS) components and eventually is intended to provide the FBI with an electronic information management system, automated workflow processes, search capabilities, and information sharing with other law enforcement agencies and the intelligence community. The FBI Director has stated that, “Sentinel will strengthen the FBI’s capabilities by replacing its primarily paper-based reporting system with an electronic system designed for information sharing. Sentinel will support our current priorities, including our number one priority: preventing terrorist attacks.”³

Audit Approach

The Office of the Inspector General (OIG) is performing audits of the Sentinel project at the request of the FBI Director and congressional appropriations and oversight committees. This audit is the third in a series of audits on Sentinel that the OIG intends to conduct to evaluate Sentinel’s progress and implementation. The objectives of this third audit were to evaluate: (1) the status of the project, including the FBI’s monitoring of the contractor’s performance during Phase 1, (2) the planning for and progress of Phase 2, and (3) the resolution of concerns identified in our two previous Sentinel audits.⁴ Future OIG audits will continue to examine the progress of Sentinel over its remaining phases and assess whether Sentinel’s cost, schedule, performance, and technical benchmarks are being met.⁵

OIG Audit Results in Brief

Phase 1 of Sentinel, which was completed on June 19, 2007, delivered two key project components: a web-based portal to ACS and workboxes that summarize case information. The user friendliness of the portal and workboxes should enhance access to information and case management within the FBI. The FBI deferred one deliverable initially planned for Phase 1 because it would be more technically feasible to accomplish it in Phase 2, and the FBI did not clearly articulate which components of another deliverable would be accomplished in Phase 1 and which components would be accomplished in later project phases. While we cannot yet assess the full impact of completing an original Phase 1 deliverable in a subsequent project phase, some future cost and schedule pressures may result. In addition, we question why cost adjustments did not occur in Phase 1 due to reduced requirements.

In addition, Phase 1 was completed in about 14 months instead of the planned 12 months. Our audit found the following four primary causes for this short delay: (1) an unrealistic schedule, (2) delays by Lockheed Martin in fully staffing the project with appropriately experienced personnel, (3) challenges in integrating the various COTS software components to work as a system, and (4) problems in assessing the project’s progress against the approved schedule.

Our audit found that one of the four deliverables initially planned for completion in Phase 1 was deferred to Phase 2: cleansing the data in the electronic case file module of ACS so that the data is in a uniform format for eventual transfer (migration) to Sentinel. As the Sentinel project progressed, the FBI determined that the data cleansing planned for Phase 1 posed significant risks to the integrity of the data and should be moved to Phase 2. In addition, the FBI did not adequately define one of the four Phase 1 deliverables, the foundational components of a service-oriented architecture.⁶ Because the FBI’s expectations for implementing a service-oriented architecture in Phase 1 were vague, we could not assess whether Phase 1 achieved its objectives in this area. FBI officials said that Phase 1 delivered an enterprise service bus, which they said was the only foundational component of a service-oriented architecture that was appropriate for this initial phase of the project.⁷

Our audit also found that the costs for the Sentinel project have increased a small amount from the initial estimates for Phase 1. As a result of a series of contract modifications, some of which pre-purchased software for Phase 2, the budget for the Phase 1, including award fees, increased from $57.2 to $59.7 million. However, the overall contract value of $305 million did not change. Lockheed Martin also estimates that its costs exceeded the revised contract amount by approximately $4.4 million due to requirements the FBI added but did not include in contract modifications. However, both parties agreed that Lockheed Martin would be paid the $59.7 million amount in the revised budget, which includes the $2 million budgeted for award fees.⁸ Over the course of Phase 1, the FBI deferred a total of 57 mostly low-level requirements from Phase 1 to later phases because they were outside of the scope of Phase 1, did not add value to Phase 1, or required the modification of ACS. Despite the somewhat decreased functionality Lockheed Martin was required to deliver in Phase 1, none of these deferrals resulted in a decrease in the cost of Phase 1.

At the time of our audit, the FBI’s activities for Phase 2 of the Sentinel project were limited to planning for that phase. However, we believe the FBI gained valuable experience during Phase 1, and the lessons learned can improve the implementation of Sentinel’s remaining phases. Based primarily on the FBI’s experience in dealing with the legacy ACS system during Phase 1 of the project, the FBI has begun to reexamine whether dividing development and implementation of Sentinel into four phases was still the most effective way to manage the work on the project. When our audit concluded in May 2007, the FBI had not yet decided on the number of remaining phases or the content of them.

Since the project began, the FBI has implemented several management controls and processes designed to help it adequately manage the development of Sentinel and bring it to a successful conclusion. We reviewed four of these controls and processes in-depth: (1) earned value management, (2) independent verification and validation, (3) risk management, and (4) bill of materials. We found that the FBI has made significant progress in each of the four, but that additional progress needs to be made in the implementation of earned value management, risk management, and the bill of materials. In our opinion, if implemented correctly these processes and controls can provide reasonable assurance of project success.

However, while the FBI has implemented earned value management to monitor Sentinel, the quality of Lockheed Martin’s cost data concerns us and the FBI. For example, when Lockheed Martin notified the FBI that its costs exceeded the revised budget by $4.4 million, Lockheed Martin’s earned value management data continued to show that Lockheed Martin was within budget on the project.

The FBI also has created a list of 16 risks it is monitoring that are associated with the Sentinel project. While the FBI’s establishment of a risk management program is a positive step, we have several concerns with the program’s implementation, including irregular review of the risks, a lack of contingency plans, and incomplete plans to mitigate identified risks. We are also concerned that the personnel assigned to manage these risks may not have sufficient time or expertise to adequately develop and implement a strategy to reduce the risks Sentinel faces.

Our audit also determined that the FBI has made good progress in addressing most of the concerns we identified in our two previous audits of the Sentinel project. Five of the 12 recommendations made in our prior reports have been closed, and the FBI is in the process of taking action to close the remaining recommendations. For example, in addressing one of our key recommendations, the FBI developed a plan and hired a contractor to perform independent verification and validation of the project’s development. To close the remaining recommendations, the FBI must complete system security and training plans, fully staff the PMO, determine the appropriate amount of management reserve for the phases of Sentinel, and develop adequate contingency plans for Sentinel. We will continue to monitor the FBI’s progress in implementing the remaining open recommendations.

In sum, the first phase of the Sentinel project is complete, although with some difficulty and without providing all of the deliverables originally intended for this phase of the project. Moreover, the most difficult portions of the project lay ahead. As Sentinel progresses, the FBI must ensure the deliverables for each phase are clearly documented and communicated to FBI management and oversight entities. We believe that the lessons learned during Phase 1, combined with the processes the FBI has established to manage and control the Sentinel project, can help provide reasonable assurance of Sentinel’s ultimate success. However, rigorous implementation of processes and lessons learned is necessary to minimize any significant deviations from cost, schedule, technical, or performance baselines.

Background

The Sentinel project follows the FBI’s unsuccessful 3-year, $170 million effort to develop a modern investigative case management system called the Virtual Case File as part of the FBI’s Trilogy information technology (IT) modernization project. The Virtual Case File originally was intended to provide the FBI with a modern system so that the existing obsolete ACS system could be retired. During multiple OIG reviews over the past several years, we reported that ACS uses outmoded technology, is cumbersome to operate, and does not provide necessary workflow and information-sharing functions.

The Sentinel contract, awarded in March 2006 to Lockheed Martin through a government-wide acquisition contract, is a cost-plus-award-fee contract that uses task orders to complete work for each phase of the project.⁹ The cost of the original task order for Phase 1 of Sentinel was $57 million. According to the contract, the FBI may exercise options for $248 million to cover three additional phases of the project and future operations and maintenance costs. Under the terms of the contract, Lockheed Martin can also be rewarded for meeting established goals in four areas: project management, cost management, schedule, and technical performance. This type of contract and award fee structure is common for large government IT projects.

While this type of contract proved problematic under Trilogy, our two prior Sentinel audits found that the FBI has made considerable progress in establishing controls and processes required to adequately manage a major IT development project such as Sentinel and to bring it to a successful conclusion – if the processes are followed and controls are implemented as intended. As we reported in each of our two previous Sentinel audits, we believe t he FBI is establishing clear milestones and requiring critical decision review points in managing this contract. For instance, if the contractor does not meet its milestones, it is penalized by loss of the award fee.

The FBI’s initial plan called for implementing Sentinel’s 4 phases over 45 months, with each phase providing distinct capabilities until the project is fully functional in December 2009. Originally, the FBI expected to complete each of the phases in 12 to 16 months. As discussed later in this report, however, the FBI is now considering a modification of the four-phase approach based on its experience with the first phase.

According to the FBI, the four phases will provide the following capabilities:

• Phase 1 introduces the Sentinel portal to provide access to data from the existing ACS system and eventually, through incremental changes in subsequent project phases, will support access to the newly created investigative case management system. Phase 1 also provides a case management personal workbox that presents a summary of all cases in which the user is involved, rather than requiring the user to perform a series of queries to find the cases as is necessary in the ACS system. In addition, a squad workbox will facilitate management of cases. The Findings and Recommendations section of this report contains a more comprehensive discussion of the Phase 1 deliverables.

• Phase 2 will begin the transition to a paperless case records system by providing electronic case document management and a records repository. A workflow tool will support the movement of electronic case files through the review and approval process, while a security framework will provide access controls and electronic signatures.

• Phase 3 will provide a new Universal Index, which is a database of people, places, or things that relate to a case. Expanding the number of attributes in the system will enable more precise searching and will enhance FBI employees’ ability to “connect the dots” among various pieces of information and cases.

• Phase 4 will implement Sentinel’s new case management and reporting capabilities, including the management of tasks and evidence. During this phase, Sentinel will be connected to ACS, data on closed cases will be migrated from ACS to Sentinel, and the process to retire ACS will begin.

Phase 1 Schedule, Cost and Performance

When Lockheed Martin delivered Phase 1 of Sentinel on June 19, 2007, 2 months behind the proposed schedule, the revised contract amount had increased from $57.2 million to $59.7 million due to an overall increase in the scope of work, including pre-purchasing software for Phase 2. However, Lockheed Martin’s costs exceeded the revised contract amount – including $2 million budgeted for award fees – by approximately $4.4 million. Lockheed Martin and the FBI agreed that Lockheed Martin would only be paid $59.7 million, the amount of the revised budget, rather than being paid the entire $4.4 million overage. FBI officials stated that the net project cost remained the same due to offsetting adjustments to the Phase 1 and Phase 2 budgets, and there was no change in the overall contract value.

At the conclusion of Phase 1, Lockheed Martin delivered two key deliverables: a web-based portal to ACS and case management workboxes. The FBI deferred to Phase 2 another deliverable, the cleansing of data in ACS’s electronic case file module for migration into Sentinel. As a result of deferring this deliverable to Phase 2, Sentinel’s total costs may be higher than currently projected. The FBI’s expectations for implementing a service-oriented architecture in Phase 1 were vague, so we could not fully assess whether Phase 1 achieved its objectives in this area. However, the FBI’s explanation that the enterprise service bus was the only appropriate component of a service-oriented architecture for Phase 1 appears reasonable, and that component was delivered.

Schedule Delay

Our audit found the following four primary causes for the 2-month delay in the delivery of Phase 1: (1) an unrealistic schedule, (2) delays by Lockheed Martin in fully staffing the project with appropriately experienced personnel, (3) challenges in integrating the various COTS software components to work as a system, and (4) problems in assessing the project’s progress against the approved schedule.¹⁰

Unrealistic Schedule

According to FBI officials, Lockheed Martin based its Phase 1 project schedule on the FBI’s proposed notional, or hypothetical, schedule created prior to formally soliciting proposals for development of Sentinel. That schedule divided the project into four phases, identified deliverables for each phase, and provided an estimated timeline for completion of each phase. While information the FBI provided potential vendors advised that they were free to propose a different number of phases or change the deliverables of each phase, vendors still had to meet the FBI’s target completion date of 2009. In addition to broad outlines of the project’s overall schedule, the FBI also dictated certain project milestones in the Sentinel Statement of Work. The Sentinel Program Manager told us that, in retrospect, the timeframes outlined in the Statement of Work were overly aggressive because they did not allow Lockheed Martin adequate time to staff the project.

Delays in Staffing

Almost immediately following the contract award, Lockheed Martin fell behind in its projected staffing levels. The FBI attributed this to the difficulty in hiring qualified personnel with top secret clearances and personnel costs 25 to 40 percent higher than Lockheed Martin projections. A 6-month suspension in processing security clearances for government contractors shortly after the Sentinel contract was awarded also depleted the supply of cleared contractor personnel and increased the cost of hiring those who were available.¹¹

In addition, Lockheed Martin and the FBI also underestimated the level of expertise in integrating COTS software that personnel would need for the Sentinel project. In a January 2007 briefing to the FBI’s Associate Deputy Director, the Sentinel Program Manager said that both the FBI and Lockheed Martin based their original personnel cost estimates on the assumption that most of the work could be completed by recent college graduates, an approach Lockheed Martin had successfully used on a large scale information technology project at the Social Security Administration. However, several PMO and FBI Chief Information Office personnel said that throughout Phase 1 of Sentinel, the level of expertise required of the Lockheed Martin staff to deal with Sentinel’s COTS software was not sufficient for the project, although they said that Lockheed Martin eventually added the required expertise. FBI officials said the quality of the Lockheed Martin staff had improved during the first phase, but that additional improvements need to be made if the subsequent phases of the project are going to be successful. Other FBI officials said, however, that Lockheed Martin should have considered contracting with the software manufacturers who developed the most challenging pieces of software to help with implementation.

Challenges of Integrating the Software

Several PMO officials, including the Sentinel Program Manager and Lockheed Martin’s Deputy Project Manager, stated that integrating the various commercial off-the-shelf software modules that comprise Phase 1 of Sentinel into a system that functions as intended was a major challenge. For example, analyzing why a particular software problem occurred within such an integrated system was difficult due to the number of variables in complex systems such as Sentinel. The COTS software used in Sentinel is so complex that the Lockheed Martin Project Manger said that it is virtually impossible to complete a COTS-based system without hands-on experience with its component software packages. Another factor that compounded the general challenge of COTS integration was that Sentinel is based on cutting-edge software, some of which had bugs. In at least one case, the software manufacturer was not aware of a problem until notified by the FBI and Lockheed Martin. Because this was a new bug, the manufacturer had to research its cause and develop a solution before Lockheed Martin could implement the software patch.

Problems in Assessing Progress

PMO personnel said that the methodology used by Lockheed Martin to construct the Sentinel project’s schedule made it difficult to assess the project’s progress. Specifically, they cited the following concerns about the schedule:

• Overuse of “hard constraints.” Hard constraints are specific dates entered into a schedule that require a task to begin or end on that date, regardless of any other activity within the schedule. Hard constraints cloud an assessment of the impact of schedule slippages because the scheduling software will assume that the task met the constraint, regardless of whether or not it did. Lockheed Martin’s project schedule contained many hard constraints, which made assessing progress difficult.

• Logic problems. PMO officials said Lockheed Martin’s schedule did not always accurately reflect the interdependence between tasks, often linking some that were not interdependent and not linking others that were interdependent.

• High percentage of “level-of-effort” tasks. Some of the tasks in the development of an IT system are referred to as “level of effort,” meaning that progress toward completion of a task is measured by the passage of time rather than progress toward completing the task. Tasks that do not have a defined deliverable, such as project management, are often measured using level of effort. However, because level-of-effort tasks are not tied to discrete deliverables, it is difficult to determine how much their completion contributes to the overall progress of a project. As a result, it is prudent to have a schedule with as few level-of-effort tasks as possible. Lockheed Martin’s project schedule contained a significant number of level-of-effort tasks.

Cost and Deliverables

The contract awarded to Lockheed Martin to develop Sentinel represents about 72 percent of the total cost of the entire Sentinel project, so Lockheed Martin’s ability to deliver its portion of Sentinel within budget is critical to the cost performance of the overall Sentinel project. As the result of a series of contract modifications, the value of Lockheed Martin’s task order for Phase 1 increased from $57.2 million at the time of the integrated baseline review (IBR) in May 2006 to $59.7 million in March 2007. However, in June 2007 Lockheed Martin advised the FBI that it had incurred costs totaling $64.1 million in the performance of Phase 1. Lockheed Martin attributed the cost overruns to unanticipated work in interfacing with existing FBI computer systems and modifications to the FBI’s testing approach.

However we found that three factors obscured a precise accounting of Lockheed Martin’s cost performance. First, even though the FBI transferred some Phase 1 requirements to later phases of the project, it received minimal cost reductions on Phase 1 from Lockheed Martin for deferring completion of these requirements. Second, the FBI did not adequately define the foundations of a service-oriented architecture expected to be delivered in Phase 1 and did not tie all of the deliverables to the requirements agreed upon for Phase 1, making it difficult to evaluate what the Phase 1 budget was supposed to pay for. Third, the FBI transferred $2.5 million in materials and services from Lockheed Martin’s budget to the PMO’s budget and increased the amount of equipment the FBI furnished for the project. As a result, the amount paid to Lockheed Martin understates the cost of the work Lockheed Martin was originally tasked with.

Requirements Deferred

Over the course of Phase 1, the FBI deferred a total of 57 mostly low-level requirements from Phase 1 to later phases.¹² Despite decreasing the amount of functionality Lockheed Martin was required to deliver in Phase 1, none of these deferrals resulted in a decrease in the cost of Phase 1. According to the FBI, it deferred most of the 57 requirements because it decided the requirement was outside of the scope of Phase 1, did not add value to Phase 1, would require the modification of ACS, or would duplicate a capability included in a future phase of Sentinel. FBI officials said they did not believe it was prudent to invest in upgrading ACS because Sentinel is intended to replace it.

We recognize that phased projects using COTS components often transfer requirements from one phase to another and, in general, we do not disagree with the FBI’s transfer of these 57 requirements. However, as noted previously, we are concerned that the FBI did not require that Lockheed Martin determine the financial impact of not doing this work in Phase 1 and adjust the cost of Phase 1 accordingly.

Phase 1 of Sentinel has delivered a web-based user interface to ACS data, giving a much more modern look and feel to ACS data and allowing users to navigate through the database using a mouse. For example, users can view and download ACS documents. However, this version of the web-based portal does not allow users to perform all of the functions included in ACS, meaning that FBI personnel may also need to continue using the old system as well. Because many of the functions now performed by ACS will not be incorporated into Sentinel until Phase 2, the Phase 1 web portal to ACS will be used only until the completion of Phase 2.

As a result, the FBI decided that duplicating all of ACS was not cost effective and chose instead to include only the most frequently used functions in the Phase 1 portal. FBI officials said they recognize in retrospect that they overlooked some critical functions in the Phase 1 portal, such as the ability to upload documents into ACS, and that Phase 1 should have incorporated those functions.

Deliverables Ill-Defined

Throughout the Sentinel project, FBI documents, including slides from weekly briefings of the FBI Director, have shown four major anticipated deliverables for Phase 1: (1) a web-based portal to ACS, (2) a case workbox, (3) the foundational components of a service-orientated architecture, and (4) data cleansing of the electronic case file portion of ACS. As implemented, Phase 1 delivered the most important deliverables, the ACS portal and case workbox. Because the foundational components of a service-oriented architecture were ill-defined, we could not evaluate the extent to which this deliverable was achieved. However, FBI officials stated that the only component applicable to Phase 1 was the enterprise service bus, which was delivered. They said that the fourth planned deliverable, the data cleansing of the electronic case file portion of ACS, was deferred because it was more technically feasible to do so.¹³ As Sentinel progressed through the life cycle management process, the FBI’s internal technical reports have noted this divergence from the original set of deliverables.

Neither the foundational components of a service-oriented architecture nor the data cleansing of electronic case file data were specified in the requirements for Phase 1, so the deferral of these goals did not require the deferral of requirements. However, achieving both of these goals may potentially require additional financial and personnel resources. And as mentioned previously, deferral of these goals did not result in a corresponding decrease in the Phase 1 contract amount.

The FBI’s Incremental Development Plan, which was provided to all potential Sentinel bidders as a framework from which to describe the intent of the Sentinel program, refers to a service-oriented architecture framework and foundational services but does not define these terms. The FBI said that as a result, it had no expectation that Lockheed Martin would specifically address the commonly recognized basic components of a service-oriented architecture in Phase 1.

The Incremental Development Plan does not include any data cleansing or data migration capabilities for Phase 1. Rather, the plan states “There are no specific requirements for migration of case data in Phase 1.” However, Lockheed Martin’s proposal included data cleansing of electronic case file data as part of Phase 1 in preparation for the data’s transfer, or migration, to the Sentinel database in Phase 2. The FBI subsequently agreed to Lockheed Martin’s data cleansing approach and the proposed scope of the data cleansing efforts was built into the project’s integrated master schedule. However, as stated above, after further consideration the FBI deferred data cleansing until Phase 2 because it had technical concerns with cleansing data in advance of migrating it.

While deferring the data cleansing to Phase 2 did not affect the functionality of Phase 1, it pushed time-consuming activities into Phase 2 and the FBI did not adjust the Phase 2 end date. In addition, similar to the deferral of requirements, the deferral of the data cleansing did not result in a decrease in the amount of the Phase 1 contract.

Costs Transferred

Through a series of six contract modifications during Phase 1, the FBI increased the total contract value of Phase 1 by $2.5 million, from $57.2 million to $59.7 million. As expected in a project of Sentinel’s size and complexity, some of the modifications increased the scope of Phase 1, while others decreased it. However, the decreases either transferred the cost for the tasks to the PMO budget or to the amount budgeted for Lockheed Martin’s award fee. For example, in March 2007 the FBI issued a modification which deleted $2.1 million for tape silos from the Phase 1 contract.¹⁴ Although the tape silos were still necessary for Phase 1, the FBI purchased silos with more storage capacity using funds from the PMO’s budget and used the funds in the Lockheed Martin contract originally allocated to tape silos to offset the cost of additions to the scope of Phase 1. FBI officials stated that the various cost adjustments did not affect the overall contract value of $305 million.

Phase 2 Planning and Project Management

The FBI has implemented several management controls and processes in addition to its life cycle management directive that are designed to help it adequately manage the development of Sentinel and bring it to a successful conclusion. In this audit, we reviewed four of these controls and processes in depth: earned value management (EVM), risk management, independent verification and validation, and bill of materials. We concluded that the FBI has made significant progress in each of the four areas, but that substantial additional progress is needed in risk management and the bill of materials.

In addition to these four areas, the FBI recognizes that the lessons learned during Phase 1 will aid the FBI in its planning of Phase 2. Although Phase 1 is complete, the most difficult portions of Sentinel development and implementation lay ahead. To reduce the risk to Phase 2 and subsequent phases of Sentinel, that the FBI must implement corrective actions resulting from the problems encountered during Phase 1.

It is also important to note that the FBI has taken action to alleviate or resolve most of the concerns identified in our first two audits of the Sentinel project relating to project management. We believe that the FBI’s efforts to improve its project management capabilities can help provide reasonable assurance that the Sentinel project can be successfully completed, if the processes are implemented as intended.

Earned Value Management

As required by the Office of Management and Budget (OMB), and with Department of Justice (Department) guidance, the FBI has established an Earned Value Management (EVM) system for Sentinel. EVM helps manage project risks by achieving reliable cost estimates, evaluating progress, and allowing the analysis of project cost and schedule performance trends. EVM compares the current status of a project, in terms of both cost and schedule, to the established cost and schedule baselines. Deviations between the baselines and the current status should demonstrate the project’s progress and the overall level of performance, thereby enabling a level of accountability to be imposed on the project. When properly implemented and utilized, EVM allows project management to pinpoint potential problems and address them before they escalate.

The Sentinel contract requires Lockheed Martin to fully implement EVM in accordance with the Sentinel EVM plan, including having an EVM system that complies with American National Standards Institute (ANSI) /Electronic Industries Alliance (EIA) Standard 748-A.¹⁵ This allows the FBI to gather EVM data on the development portion of the project through monthly electronic data transfers from Lockheed Martin.

Our review of EVM reporting from September 2006 to March 2007 showed that the FBI has continued to implement EVM and use that data to help manage Phase 1 of Sentinel. However, several issues decreased the effectiveness of EVM as a tool to manage the Sentinel development contract. The most significant issue was the reliability of the EVM data Lockheed Martin provided the FBI. In June 2007, the FBI rejected Lockheed Martin’s April 2007 EVM data after Lockheed Martin notified the FBI that it estimated that it had incurred approximately $64.1 million in costs during Phase 1 of Sentinel. Because the EVM baseline for Phase 1 was $59.7 million, Lockheed Martin’s estimate showed that its EVM system was not collecting accurate data on Sentinel costs as Lockheed Martin was accruing the costs – one of the primary purposes of an EVM system.

Further, while the FBI’s implementation of EVM comports with the Department’s guidance, it does not provide all the data that OMB believes necessary for oversight purposes. As a result of OMB concerns that the FBI reprogrammed or rebaselined Phase 1 of Sentinel without required OMB approval, we reviewed all changes to the time-phased budget used to measure Sentinel’s progress.¹⁶ We concluded that the FBI had not rebaselined the project, but that frequent replanning diminished the quality and usefulness of the EVM data for higher-level oversight.¹⁷

Independent Verification and Validation

In September 2006, the FBI obtained the services of Booz Allen Hamilton (Booz Allen) to perform the independent verification and validation function for the Sentinel project. Since then, Booz Allen has participated in FBI-only project meetings and joint FBI-Lockheed Martin project reviews. In addition, Booz Allen has provided written comments and recommendations on many project documents, and produced 15 project-status briefings and monthly reports. Booz Allen also produced monthly reports and biweekly briefings that were sent directly to the FBI’s Chief Information Officer.

These reports and briefings highlight recent activities, upcoming events, and the independent verification and validation team’s view of the overall status of the project, including a discussion of risks that could affect the project’s cost, schedule, or performance. The independent verification and validation products also included recommendations and best practices observed by Booz Allen. As of May 2007, Booz Allen had made over 70 recommendations based on risks and other areas it identified. Booz Allen also reported several project management and oversight weaknesses that increased the risks associated with Sentinel, including concerns about:

• the ability of Lockheed Martin’s developers to build Phase 1 to meet the FBI’s needs before Lockheed Martin had completed the design; and

• the quality of most of the test procedures submitted by Lockheed Martin for the test readiness review.

Risk Management

The purpose of risk management is to assist the project management team in identifying, assessing, categorizing, monitoring, controlling, and mitigating risks before they negatively affect a program. A risk management plan should identify procedures used to manage risk throughout the life of the program. Risks are categorized by severity and identified as either open or resolved. Open risks are tracked in a risk register maintained by the risk manager until resolved.

The FBI has created a list of 16 risks it is monitoring that are associated with the Sentinel project. While the FBI’s establishment of a risk management program is a positive step, we have several concerns with the program’s implementation, including irregular review of the risks, a lack of contingency plans, and incomplete plans to mitigate identified risks.

As required, the FBI has developed plans to mitigate the highest ranked risks. However, the mitigation plans for these top-ranked risks are incomplete because they do not include a method to measure whether the steps in the mitigation plan are effective. In addition, the FBI has developed contingency plans for only 50 percent of the risks that are required to have such plans. The Sentinel risk manager said that the mitigation plans do not include a method to measure their effectiveness because it is very difficult to develop accurate measures. However, we believe that risk management is critical to the success of Sentinel.

When a new risk is opened, the Sentinel Risk Review Board assigns an owner to that risk to develop a mitigation and contingency plan and to ensure that the mitigation plan is implemented. We support the idea of having one person taking a lead role – having “risk ownership” in managing each risk. However, this process does not appear to be functioning as intended. During interviews with risk owners we found that some could not explain the nature of the risks they had been assigned, while others said they thought they did not have the authority or capability to implement a risk’s mitigation strategy.

With respect to currently identified project risks, we view Sentinel’s ability to interface with existing FBI systems from which it will extract data as a potentially significant challenge. In Phase 1, Lockheed Martin had unanticipated problems connecting Sentinel with ACS because there was no detailed documentation describing how ACS works. Consequently, Lockheed Martin had to create the documentation itself. The Sentinel PMO did not anticipate this task because the managers of ACS told the PMO that all of the necessary documentation existed. Because this unexpected task strained both the Phase 1 budget and schedule, the PMO is now tracking documentation for the other systems with which Sentinel must interface as a risk.

Bill of Materials

A bill of materials is a complete listing of all parts, assemblies, equipment, and software that make up an IT system, as well as the information required to construct new units of the system or order spare parts for it. Contractually, Lockheed Martin is required to purchase the list of items on the bill of materials it submitted with its proposal. As is to be expected in a project as complex as Sentinel, Lockheed Martin has needed to revise the bill of materials for several reasons including revisions to the project’s design.

An accurate bill of materials is critical to ensuring the FBI approves all changes to Sentinel’s design and that an accurate list of Sentinel’s components is available to the FBI to use when reviewing Lockheed Martin’s invoices. Because of the importance of an accurate bill of materials, the PMO established a Bill of Materials Deviation Policy, which establishes the criteria for what constitutes a change to the bill of materials and the criteria to assess whether Lockheed Martin needs the FBI’s approval prior to making a change to the bill of materials.

According to the Sentinel Contracting Officer’s Technical Representative, Lockheed Martin also established its own internal procedures for making changes to the bill of materials. However, Lockheed Martin employees viewed internal approval of changes as final approval to change the bill of materials and therefore did not submit all changes to the FBI as required. The PMO is aware of this and other shortcomings and has established a joint Lockheed Martin-FBI team to revise Sentinel’s bill of materials policies and procedures to address these issues.

In addition to the issues with which the FBI was aware, we identified a significant flaw in Sentinel’s Bill of Materials Deviation policy. While the policy states that a deviation is any addition or deletion to the bill of materials, the policy does not require FBI approval for additions or deletions. Instead, the policy only requires approval for cost increases and changes to purchase dates for items already on the bill of materials. FBI officials agreed that the deviation policy needs to address this issue.

Lessons Learned

Although Phase 1 was slightly delayed and over budget, the FBI appears to have learned important lessons that may allow it to reduce the risk to subsequent phases. We examined what the FBI had learned about integrating COTS software, interfacing with ACS, measuring progress, clarifying with Lockheed Martin the details of what must be done to meet a given requirement, scheduling reviews of Sentinel’s design, and ensuring Sentinel’s schedule accurately measures the project’s progress. We believe that if the FBI implements the planned corrective actions resulting from these lessons, we believe the risk to subsequent phases of Sentinel should be reduced.

Actions Taken on Prior OIG Recommendations

During our audit, we examined the FBI’s actions to address recommendations we made in our audit reports on Sentinel and found that the FBI was, in general, taking action to resolve our concerns. Based on the FBI’s actions, we closed 5 of the 12 recommendations. We also noted that the FBI agreed with the remaining recommendations and was in the process of taking corrective action. Our recommendations dealt generally with the FBI’s need to complete required planning and general management oversight policies and procedures for Sentinel in order to help ensure its success.

In our March 2006 report, we made seven recommendations, of which four have been closed. The FBI continues to address the remaining three recommendations that involve completing a system security plan, filling vacant positions within the Sentinel PMO, and completing comprehensive training plans for the project. We found during our audit that the FBI had completed its independent verification and validation plan, which partially closes one of the recommendations, but the system security plan still needs to be completed for full closure of the recommendation.

In our December 2006 report, we made five recommendations, one of which has been closed. The four remaining recommendations were that the FBI should: (1) develop contingency plans as required by the Sentinel Risk Management Plan, (2) provide experienced contractors to conduct an independent verification and validation process throughout the Sentinel project, (3) determine the appropriate amount of management reserve for each phase of the project, and (4) fill the vacant Sentinel PMO positions needed to complete Phase 1 of the project.

As discussed earlier, we found that the FBI has improved in developing contingency plans as required by the Sentinel Risk Management Plan. However, as noted, we continue to have concerns about the FBI’s management of risks in the Sentinel project. We also found that the FBI has begun utilizing a contractor to perform independent verification and validation and the contractor offered numerous recommendations during Sentinel’s Phase 1. Because the independent verification and validation is being performed, this recommendation will be closed through our normal audit follow-up process.

For the two remaining open recommendations from our December 2006 report, we found that the FBI had not determined management reserve amounts for the remaining phases of Sentinel and did not fully staff the Sentinel PMO. Because Sentinel consists of four phases, each phase will have a separate task order and its own funding. At the end of our fieldwork for the current review, the PMO was still in the planning stage for Phase 2, which includes a risk analysis and cost implications for those risks. As a result, the management reserve had not yet been determined.

Regarding the staffing of the Sentinel PMO, we found that the FBI continued to make progress in its hiring of personnel. We found that 70 of 76 positions had been filled, and 3 individuals were pending for the remaining positions. The FBI also was planning for changes to be made in the PMO as the project evolved from planning and development to operations and maintenance.

We will continue to monitor the progress made by the FBI in implementing the remaining recommendations identified in our prior audits to ensure that the required planning and general management oversight policies and staff continue to be utilized effectively.

Conclusion

The FBI deployed the two most critical deliverables planned for the first phase of the Sentinel project – a web-based portal to ACS and personal and squad workboxes that summarizes users’ cases and leads. These deliverables were completed slightly behind schedule and over budget. However, the most difficult portions of Sentinel to implement lay ahead and several tasks originally planned for Phase 1 have been deferred to later phases.

We will monitor the quality of the Phase 1 product in our next Sentinel audit as FBI employees gain experience in using the portal and workboxes.

We believe that the lessons learned during Phase 1, combined with the processes the FBI has established to manage and control the Sentinel project, can help provide reasonable assurance of Sentinel’s ultimate success. However, rigorous implementation of processes and lessons learned is necessary to minimize any significant deviations from cost, schedule, technical, or performance baselines.

OIG Recommendations

In this third Sentinel audit, we make nine recommendations to the FBI to help ensure the success of the Sentinel case management system and to better manage project costs. Among the recommendations are that the FBI limit the scope and duration of future project phases to make them more manageable; adjust the amount of task orders to reflect changes in project requirements; include both initial and revised performance baselines in EVM reports; improve the requirements for Lockheed Martin’s cost reporting; improve risk management and the tracking of project deficiencies; and improve the bill of materials process.

1. ACS is the FBI’s current case management system. Deployed in 1995, ACS is a mainframe computer system.

2. A lead is a request from any FBI field office or headquarters for assistance in the investigation of a case.

3. FBI Press Release entitled FBI Announces Award of Sentinel Contract,
   March 16, 2006.

4. See Department of Justice Office of the Inspector General, The Federal Bureau of Investigation’s Pre-Acquisition Planning For and Controls Over the Sentinel Case Management System, Audit Report Number 06-14, March 2006; and Department of Justice, Office of the Inspector General, Sentinel Audit II: Status of the Federal Bureau of Investigation’s Case Management System, Audit Report Number 07-03, December 2006.

5. Although we originally intended to cover the early stages of Phase 2 of Sentinel in this report, Phase 2 had not yet begun when our audit fieldwork was completed in May 2007. However, we evaluated the impact the FBI’s experience with Phase 1 had on how the FBI plans to approach Phase 2. We will evaluate progress under Phase 2 of the project in our next audit.

6. A service-oriented architecture is a software design approach in which software components, called services, can be re-used by multiple software applications.

7. An enterprise service bus is software “middleware” that connects software components and allows the components to communicate with each other.

8. Lockheed Martin did not receive an award fee. Instead, the FBI allowed Lockheed Martin to transfer the $2 million budgeted for the award fee to the cost portion of the budget to cover the cost overruns.

9. An award fee is a financial incentive provided to a contractor based on the contractor’s performance. A task order specifies the services required and the negotiated terms at which they will be provided, subject to the terms of the contract.

10. Although we view the schedule as unrealistic, FBI officials in commenting on a draft of this report stated that they would describe the schedule as aggressive, rather than unrealistic, because had Lockheed Martin been able to provide adequate staffing from the beginning, the 12-month schedule might have been met.

11. According to the FBI, shortly after the Sentinel contract was awarded the Defense Security Service, the organization responsible for performing background investigations and granting clearances, suspended its processing of clearances for all government contractors for 6 months due to significant backlogs.

12. For example, the requirement that Sentinel be able to perform unstructured searches against items collected during investigations was deferred from Phase 1 to Phase 2.

13. See page 34 for a discussion of the common components of a service-oriented architecture.

14. A tape silo is computer hardware that uses tapes to store large amounts of computer data.

15. ANSI/EIA Standard 748-A is the criteria selected by the OMB for EVM systems. The standard includes 32 specific criteria in 5 process areas necessary for a sufficient EVM system: (1) organization; (2) planning, scheduling and budgeting; (3) accounting; (4) analysis and management reports; and (5) revisions and data maintenance.

16. Reprogramming, or rebaselining, revises the project baselines and eliminates all cost and schedule variances. Rebaselining usually occurs when a project’s progress deviates significantly from the original plan and the remaining time and funds are not sufficient to complete the project.

17. Replanning revises the time-phased budget for completing the work remaining in a project without any changes to the total scope of work, baselined cost, or scheduled completion of the project.