The Federal Bureau of Investigation's Pre-Acquisition Planning for
and Controls Over the Sentinel Case Management System
Audit Report 06-14
March 2006
Office of the Inspector General
|
PLANNING THE DEVELOPMENT OF SENTINEL
Improved Management Processes and Controls In the early stages of the Trilogy project, the OIG and GAO recommended that the FBI establish an ITIM process to guide the development of its IT investments. In response, the FBI instituted a Life Cycle Management Directive (LCMD) in 2004 while Trilogy was well underway. The LCMD established policies and guidance applicable to all FBI IT programs and projects, including Sentinel. We believe the structure and controls imposed by the LCMD can help prevent many of the problems encountered with the failed development of the VCF. The LCMD covers the entire IT system life cycle, including planning, acquisition, development, testing, and operations and maintenance. As a result, the LCMD provides the framework for standardized, repeatable, and sustainable processes and best practices in developing IT systems. Application of the IT systems life cycle within the LCMD can also enhance guidance for IT programs and projects, leverage technology, build institutional knowledge, and ensure that development is based on industry and government best practices. The LCMD is comprised of four integrated components: life cycle phases, control gates, project level reviews, and key support processes. A diagram showing how these components relate to each other is found in Appendix 4. According to the FBI CIO, since the inception of the LCMD all FBI IT programs and projects have been reviewed and managed according to the processes described in the LCMD. New IT programs and projects have been managed under the LCMD from inception and will continue to be managed through retirement or replacement. Existing IT programs and projects were reviewed and placed within the relevant life cycle phase according to their maturity and other factors. System Life Cycle Phases The LCMD has established nine phases that occur during the development, implementation, and retirement of IT projects. During these phases, specific requirements must be met for the project to obtain the necessary FBI management approvals to proceed to the next phase. The approvals occur through seven control gates, where management boards meet to discuss and approve or disapprove a project's progression to future phases of development, implementation, or retirement. As of December 6, 2005, the Sentinel project had passed through the first three of the nine phases and is currently in the fourth phase - Source Selection. The following table shows the nine phases of development, implementation, and retirement. FBI LCMD DEVELOPMENT PHASES
Control Gate Reviews The seven control gate reviews provide management control and direction, decision-making, coordination, confirmation of successful performance of activities, and determination of a system's readiness to proceed to the next life cycle phase. Decisions made at each control gate review dictate the next step for the IT program or project and may include: allowing an IT program or project to proceed to the next segment or phase, directing rework before proceeding to the next segment or phase, or terminating the IT program or project. The FBI's Investment Management Project Review Board (IMPRB) - comprised of 12 representatives from each FBI division at the Assistant Director level and 4 representatives from the Office of the Chief Information Office, including the CIO - is responsible for approving an IT project's passing through each control gate. The Sentinel project has been approved through the first two of the LCMD control gates: the system concept on July 15, 2005, and the acquisition plan on July 29, 2005. The following table shows the seven control gate reviews that govern the approval of an IT project and the related LCMD phases. FBI LCMD CONTROL GATE REVIEWS
At each control gate, executive-level reviews determine system readiness to proceed to the next phase of the IT systems life cycle. Evidence of readiness is presented and discussed at each control gate review in the form of deliverables, checklists, and documented decisions. Regardless of the development model used for a particular program or project, all control gate reviews should be performed unless an agreement is made to skip or combine them. Depending upon the development model employed, programs or projects may pass through the control gates more than once. Because Sentinel is being developed in phases, and the contractor must provide a system design for each phase, the project will pass through Control Gate 3 four times. The control gate reviews also provide executive-level controls to ensure that IT projects are adequately supported and reviewed before a project receives additional funding. Five executive-level review boards serve as the decision authority for the control gate reviews.
The Gate 2 approval for Sentinel on July 29, 2005, signified that the IMPRB accepted the overall project approach and cost estimate for acquiring the Sentinel system. Our review of the approval documents showed that the FBI generally complied with the requirements of the LCMD in performing the control gate reviews for Sentinel. However, two documents required by the LCMD had not been completed at the time the control gate review was conducted because: (1) the system security plan could not be developed since the vendor needs to provide the project design details and, as of the date of the control gate review, the vendor had not been selected, and (2) the IV&V plan has to be carried out by a separate contractor to provide for an independent control to assess the implementation of the system according to technical and performance baselines. As of February 2006, the FBI had not yet awarded the IV&V contract. The system security plan will provide the detail necessary for the completion of certification and accreditation of the applications being created for Sentinel. The IV&V plan is, in our opinion, crucial to ensuring the success of the Sentinel project. We will continue to monitor these two items in our subsequent audit work, including whether the IV&V is being implemented by an independent contractor. At the Gate 2 review, the IMPRB approved Sentinel prior to the approval of the acquisition plan. The OMB requires non-phased IT projects to demonstrate funding for the entire project prior to the signing of a contract. The FBI's LCMD incorporates this process for most of its IT projects. However, because Sentinel is a multi-phased project, the FBI has modified this part of the LCMD. According to the FBI, for Sentinel the FBI will identify funds for each phase of the project prior to work being initiated for that phase rather than identifying the funds for all four phases from the outset. The FBI will perform separate acquisition plan reviews for each phase prior to its initiation, and each phase must receive Control Gate 2 approval before proceeding. We agree with this modification to the LCMD for Sentinel because it provides greater oversight of the project and requires a distinct commitment of funds prior to the initiation of each phase. Had such control gates and management reviews been in place during the Trilogy project, many of the problems with that project could have been avoided or identified earlier for corrective action. Project-Level Reviews Project-level reviews help determine a project's readiness to proceed to the next phase of the project life cycle. Each project-level review provides information to the executive-level control gates as data is developed and milestones are completed. At the conclusion of our field work for this audit in December 2005, the FBI had conducted two project-level reviews for Sentinel:
Key Support Processes The LCMD also contains 23 key support processes that provide additional support to the development of projects within the FBI. While the key support processes are not developed for projects specifically, these processes cover organization-wide management functions, and as a result the key support processes affect how individual projects are managed. For example, one key support process is the FBI's Strategic Plan. For Sentinel, the Strategic Plan defines the organizational need that Sentinel will address once it is implemented. However the FBI's Strategic Plan was not created specifically for Sentinel. Key process areas are performed independently of the life cycle phases and the deliverables associated with each key process are integrated into the control gate and project-level reviews where applicable. Appendix 5 lists the 23 key process areas. Based on our review of planning documents and interviews with key FBI personnel including the CIO, we believe that the FBI is applying more rigorous management controls and ITIM processes in planning for Sentinel. Moreover, during the 3 years of Trilogy's development, the FBI had five different CIOs or acting CIOs. Since the start of Sentinel's development, the FBI has had stability in the CIO position. In addition, as a result of a July 2004 reorganization, the CIO's office has much greater authority over all FBI IT management and resources than it did in the pre-Sentinel era. Sentinel Program Management Office The PMO plays a critical role in assuring that the FBI implements a case management system that meets its needs. The PMO's contract and program execution responsibilities include: (1) cost, schedule, and performance oversight; (2) LCMD project reviews; (3) award fee evaluations; (4) primary contractor's documentation review and acceptance; (5) requirements and risk management; and (6) budget and financial management. In light of these responsibilities, having a qualified, dedicated PMO staff focused on program execution is critical to the success of the Sentinel project. Since the PMO's creation soon after the inception of the Sentinel project, the FBI has made progress in staffing the office. As of January 30, 2006, the PMO consisted of 51 of the 76 IT personnel identified in the FBI's Sentinel Staffing Plan (67 percent) as required to properly oversee the project. According to the FBI, the objective in staffing the PMO is to form an integrated team of subject matter experts from government, federally funded research and development centers, and system engineers and technical assistance contractors to maximize program expertise.12 The Sentinel program manager told the OIG that because of the pre-award spending caps placed on the program, it was premature to staff the entire PMO during the pre-award effort. As a result, he said the FBI is hiring essential program management oversight personnel to ensure that the PMO is prepared to handle contract award activities. In addition, another FBI official told us that delays in hiring PMO staff have resulted from the FBI's lengthy background investigation and clearance process. However, due to the aggressive scheduling of Sentinel, it is critical for the FBI to fully staff the PMO office as soon as possible. In our opinion, the significant turnover of project management during the Trilogy project - 15 different key IT managers over the course of its life, including 10 individuals serving as project managers for various aspects of Trilogy - was a major reason for Trilogy's problems. We believe that fully staffing the Sentinel PMO before the project begins is key to establishing the stable management staff required to properly oversee the project. The Sentinel program manager, on loan to the FBI from the Central Intelligence Agency since November 2005, is experienced with large IT systems acquisitions and should provide strong leadership. However, he is detailed to the FBI for 2 years, with an option to extend for another year. As a result, he is expected to return to his home agency before Sentinel is completed. When questioned about the program manager's planned tenure, the FBI CIO said that a potential replacement will be assigned to work directly with the program manager in the event of the program manger's departure. In addition, the FBI said that it continues to build management depth in the Sentinel PMO to ensure that each position has a trained backup to ensure continuity. In light of the likelihood of the program manager's return to the CIA before Sentinel is completed, we believe that the FBI needs to ensure a seamless transition to a qualified successor. Moreover, as discussed in our February 2005 report on Trilogy, given the turnover of key personnel during that effort and the resulting lack of continuity and oversight, it is important for the FBI to maximize leadership stability throughout the project, not only with respect to the program manager but also other key PMO positions. The following table summarizes the PMO's staffing level as of January 31, 2006. SENTINEL PMO STAFFING REQUIREMENTS
For a more complete description of PMO staff and their duties, see Appendix 6. Although we are concerned about the incomplete staffing of the PMO given its vital role in helping ensure the success of the Sentinel project - particularly since project management was one of the major reasons for the VCF failure - the FBI has filled some of the more critical PMO positions, such as program leadership, system engineers, contracting officer, and business manager. The OIG will continue to monitor the staffing of the PMO and the stability of the program's leadership in future audit reports to ensure that Sentinel has the needed staff in place to help ensure its success. Sentinel Oversight In addition to its ITIM processes represented by the LCMD, the FBI has identified four external oversight or advisory entities in addition to the OIG and congressional committees that will provide feedback on Sentinel's development: (1) the FBI's Science and Technology Board, (2) RAND, (3) the Markle Foundation, and (4) a retired corporate chief technology officer to advise the FBI on areas of information sharing and privacy, IT strategic planning and investments, and management of large IT acquisitions.13 The FBI also holds monthly meetings with representatives of the OMB and the Department - and weekly meetings with the FBI Director - to track Sentinel's progress. We found that progress briefings during the VCF-development process proved ineffective. Therefore, we believe that vigorous reporting and analysis of Sentinel is needed to maintain transparency over the project's progress and identify any problems encountered as Sentinel unfolds. Our future audits of Sentinel will examine the extent and effectiveness of such project oversight. In its February 2005 audit report on the Trilogy project, the OIG cited the lack of an Enterprise Architecture as one of the reasons for the failure of the VCF effort. Since then, the FBI has made progress in establishing an Enterprise Architecture to more effectively and efficiently manage its current and future IT infrastructure. In March 2005, the FBI completed an Enterprise Architecture baseline report on the status of its "as is" Enterprise Architecture activities. The purpose of the report was to provide a high-level snapshot of current FBI business processes and supporting IT structures and systems. In May 2005, the FBI issued a similar report on its "to be" architecture activities and an interim architecture report showing how Sentinel will help the FBI in attaining the future IT environment outlined in the "to be" architecture report. The FBI stated that while its Enterprise Architecture continues to mature, it now provides a roadmap to help the FBI more effectively develop systems that directly support its mission. Currently, the FBI is in the approval process for its Enterprise Architecture development methodology documentation, which will help ensure that each FBI component follows the same set of guidelines when developing IT systems. If the FBI continues to use the new Enterprise Architecture documentation to drive its IT investments, it minimizes the risk of investing in IT that is duplicative, poorly integrated, costly, or not supportive of the FBI's mission. The FBI still needs to develop a transition plan, a step-by-step process to move from the current architecture to the target architecture. In addition to establishing a fully mature Enterprise Architecture, the FBI must also begin to use the Enterprise Architecture to drive its IT investments. In our opinion, the FBI's lack of a fully mature Enterprise Architecture, which few federal agencies have achieved, should not prevent the Sentinel project from going forward. The FBI has instituted a risk management process to identify and mitigate the risks associated with the Sentinel project. The Sentinel IT risk process is managed by the Sentinel program manager and a Risk Review Board. While Risk Review Board meetings have been held biweekly during the pre-acquisition phase, the FBI plans to hold weekly meetings once the Sentinel contract is awarded. The most significant risks identified by the board are examined at monthly Program Management Review sessions and other Sentinel oversight meetings in accordance with the LCMD. The purpose of risk management is to assist the program management team in identifying, assessing, categorizing, monitoring, controlling, and mitigating risks before they negatively affect a program. A risk management plan identifies the procedures used to manage risk throughout the life of the program. In addition to documenting the risk approach, the plan focuses on how the risk process is to be implemented; the roles and responsibilities of the program manager, program team, and development contractors for managing risk; how risks are to be tracked throughout the program life cycle; and how mitigation and contingency plans are implemented. Program risks include risks that are identified and managed by the development contractor as well as risks that can only be identified and managed by the FBI. This requires that risk management be performed by the vendor and subcontractors to identify risks from the contractor perspective, and by the FBI program management team to identify risks from the FBI's perspective. According to Sentinel Risk Management Plan, Sentinel risks are to be identified, assessed, and tracked throughout the life of the program. The PMO is responsible for reviewing new or "proposed" risks to determine if the items should be accepted as an "open" risk. Open, or unresolved, risks are supposed to be analyzed, updated, and assigned impact and severity ratings by each voting board member. The program manager ranks the risks so that the highest priority risks get immediate attention. The PMO has the responsibility to track and periodically review risks that are closed or resolved to prevent recurrence and to document the effectiveness and any unintended consequences of the mitigation strategy employed. In the initial Concept Exploration Phase of the life cycle, the PMO developed a mission-needs statement that identified the following five potential areas of risk in the Sentinel project.
In addition, the acquisition plan created in the planning phase of the life cycle identified the following risks for the Sentinel project:
The plan also considered consequences for each risk area and offered mitigation plans. We agree with the risks the FBI has identified. However, the FBI's mitigation plans, along with its LCMD processes and other controls, if followed, will reduce the potential effects of each risk. A detailed listing of each risk and the FBI's mitigation strategy is outlined in Appendix 7. Leveraging the VCF for Sentinel In his February 2005 congressional testimony, the FBI Director cited a loss of $104.5 million out of the $170 million spent on the 3-year VCF development effort. However, during the current audit we were unable to determine how much of the VCF investment the FBI was able to transfer to the Sentinel project.15 The FBI did not maintain records identifying or estimating the cost of any VCF products that can be incorporated into Sentinel. According to independent evaluations of the VCF product by Aerospace Corporation, the code used for developing the VCF was inadequate and therefore should not be useful for Sentinel. Further, the FBI intends to maximize the use of off-the-shelf products for Sentinel. Although the FBI likely applied lessons learned from the VCF effort, including a better understanding of what features it wanted in a case management system, we were unable to quantify what, if anything, was transferable from the VCF to Sentinel. One FBI system engineer said he thought that as much as 40 percent of the VCF specifications would apply to Sentinel, but he was uncertain and had no documentation to support his estimate. Another FBI official explained that a limited amount of hardware left over from the VCF effort was used by the FBI for purposes other than Sentinel. The only clear-cut transfer from the VCF was $3,542,000 in fiscal year (FY) 2004-2005 funding that has been redirected to Sentinel. Because this first Sentinel audit focused on the FBI's pre-acquisition planning, and given the procurement sensitive nature of the information, the FBI did not disclose to the OIG the estimated cost of the planned four-phase Sentinel project. However, in response to a Senate Appropriations Committee inquiry in October 2005, the FBI estimated that it would cost between $400 and $500 million to develop Sentinel. According to the Sentinel program manager, the precise cost estimate will not be known until the FBI awards the contract, which has been postponed to early 2006.16 Our next audit will examine in detail the winning bidder's cost estimates. According to the FBI's Deputy Assistant Director of Finance, during the summer of 2005 the FBI met with representatives from the Department of Justice and the OMB to discuss options to fund the project. In the end, the FBI decided to seek funding for Sentinel using both reprogrammed and appropriated funds: the first two phases would be funded using FBI funds reprogrammed from other projects and operations and the third and fourth phases would be funded using appropriated funds. Reprogramming Request According to an FBI official, the OMB required the FBI to identify the funding for each phase of Sentinel before work on that phase could begin. As a result, on September 27, 2005, the FBI submitted a $97 million reprogramming request to Congress for the first phase of Sentinel. Congress approved the request on November 15, 2005. The FBI's reprogramming request did not offer sufficient detail for us to render a detailed opinion on the specific amount of the request. Yet, because of the FBI's extreme need for a new case management system, this initial reprogramming request appears reasonable, and in our judgment, the Sentinel program should move forward. The FBI currently is developing a second reprogramming request to fund the second phase of Sentinel at an amount which we believe will be similar to the first request - approximately $100 million. The size of the appropriations the FBI expects to seek from Congress to complete the third and fourth phases of the Sentinel program are unknown to us, as are the funds that will be needed to operate and maintain the program on an ongoing basis. The FBI has agreed to provide a more precise cost estimate for the remainder of the project after the Sentinel contract is awarded. With regard to training, the FBI's initial $97 million reprogramming request includes $1.2 million in training costs in the first phase of the Sentinel program. However, the FBI has not yet developed a comprehensive training plan for Sentinel or an estimate for its full training costs. In our judgment, training costs over the life of the project will be substantial. The reprogramming request also cites approximately $10 million as management reserve. In our judgment, maintaining a management reserve is a prudent practice given the uncertainties of developing a new IT system. However, when attempting to calculate the amount of the management reserve required for a major IT project, an organization should consider the degree of risk associated with the project and use Earned Value Management (EVM) tools to quantify the effect on the project should the potential risk materialize. We do not have enough information at this time to evaluate the adequacy of the FBI's proposed reserve for the first phase of Sentinel or what amount of reserve might be required over the life of the entire program. As the project progresses, the FBI must continue to monitor and reassess the level of the reserve fund. According to the FBI, more than $14 million of the initial reprogramming will come from the Counterterrorism Division budget, $13 million from intelligence-related activities, and $2 million from the Cyber Division. We interviewed officials at FBI headquarters to assess the effect of the $97 million reprogramming on FBI operations. Generally, these officials said their divisions and offices can withstand the diversion of funds to Sentinel for the first reprogramming. However, we are concerned that diverting substantial funds from such mission-critical areas could begin eroding the FBI's operational effectiveness, only to be compounded by an anticipated second reprogramming. Although most FBI divisions and offices seemed confident about their ability to absorb the initial reprogramming of funds to Sentinel, they stated that a second reprogramming of the same magnitude would damage their ability to fulfill their mission. According to FBI CIO, the FBI intends to send another reprogramming request to Congress to fund the second phase of the Sentinel program in FY 2006. The OIG plans to assess the operational impact of these reprogrammings in subsequent Sentinel audits to assess whether the FBI's critical missions are adversely affected while the FBI also seeks to provide its employees with a case management system that will help them do their jobs more effectively and efficiently. Cost Tracking and Control In the Trilogy project, the FBI lacked an effective, reliable system to track and validate the contractors' costs.17 We highlighted this concern in our February 2005 report on Trilogy and the VCF. Further, in February 2006 draft report, the GAO stated its preliminary finding that the FBI's poor cost controls resulted in the payment of about $10 million in questionable contractor costs. Although the FBI stated that it is evaluating a tool to track Sentinel project costs, we view the potential weaknesses in cost control as a project risk. One approach to achieving reliable program cost estimates, evaluating current progress, and analyzing schedule and cost performance trends is to employ the discipline of EVM. EVM enables project teams to report progress to program managers to evaluate performance against initial baselines. In essence, EVM is a method of imposing accountability on a project and exposing potential problems while there is still time to fix them. In a memorandum dated August 4, 2005, the OMB required federal CIOs to manage and measure all major IT projects to within 10 percent of baseline goals by using an EVM system. The OMB required each agency to develop agency policies for full implementation of EVM on IT projects by December 31, 2005. In August 2005, the FBI developed a Sentinel Program EVM Capability Implementation Plan which, in our judgment, satisfied the OMB requirement for the project. According to the plan, the Sentinel PMO will use the plan to measure its earned value performance, and the performance of the vendor, and report the result to oversight entities. The Statement of Work requires that Sentinel's vendor and its contractors implement EVM in accordance with the plan. According to the FBI, it has evaluated several tools to track and manage EVM results. The evaluation consisted of examining technical and functional capabilities of the tools, learning about the requirements for the associated system environment, reviewing implementation methodologies and training materials, evaluating tool acquisition and installation costs, and viewing demonstration sessions of potential tools. As a result of this review, the FBI intends to use the following tools to track and manage Sentinel in the short term.
In the long term, the FBI expects that its EVM performance metrics will be developed, maintained, and reported using Métier's WorkLenz software suite. The FBI is acquiring the software but will need to complete security certification and accreditation for the software to be certified for use on FBI systems. According to the FBI, full implementation and execution of the EVM capabilities for the Sentinel project are scheduled to be completed after the Integrated Baseline Review occurs approximately 2 months after the award of the Sentinel contract. Based on our initial review, the FBI's EVM strategy appears adequate. We will monitor the FBI's implementation of EVM in future audits. Capability Maturity Model Integration The FBI's Statement of Work for the Sentinel project requires that bidders obtain an independent appraisal certifying that their systems development, software engineering, and integration processes are at a Level 3 or higher on the Carnegie-Mellon University's Capability Maturity Model Integration (CMMI) 5-level maturity scale. This requirement includes all vendors and any subcontractor that will contribute a minimum of 10 percent of the total Sentinel effort in developing or integrating software. Sentinel's Statement of Work also gives the FBI the right to interview the lead appraiser who conducted the assessment and to conduct independent assessments during the development of the project to verify compliance with the appraised processes. We believe that by requiring the vendor to perform at a CMMI Level 3, the FBI reduces the risk of selecting a vendor that is not capable of completing the Sentinel project and integrating all four project phases. Additionally, because the vendor will be independently reviewed by a CMMI appraiser, the FBI has assurance that the processes the vendor will use to develop Sentinel are rated favorably in relation to best industry practices. In our upcoming audit work, we plan to verify that the appraisal was conducted, review its results, validate the appraiser's independence, and review the results of the appraisal. In selecting the appropriate contract type for the development of Sentinel, the FBI originally identified 16 Government-wide Acquisition Contracts (GWAC) that were suitable for a project as extensive as Sentinel. The FBI eliminated 11 of the 16 GWACs as inappropriate vehicles for Sentinel because the contract vehicle's task scope was inadequate, task-order cost reimbursement was not allowed, or the contractors available through the GWAC lacked the expertise needed for the project. The FBI further analyzed the other five GWACs to determine which were the most suitable for the project. The analysis included a 29-item questionnaire with 6 discriminator areas.18 The discriminator areas are listed below.
Based on the information obtained from the questionnaires, the FBI eliminated two of the remaining five GWACs for two reasons:(1) the GWAC did not allow direct order, and (2) the GWAC may not support the acquisition strategy of having all task orders awarded by January 2006 and be of no more than five years in duration.19 From the other three GWACs, the FBI chose the National Institute of Health's (NIH) Chief Information Officer-Solutions Partners 2 Innovations (CIO-SP2i) contract vehicle because it gave the FBI the greatest flexibility and included 37 potential bidders. The Federal Acquisition Regulations (FAR) § 15.201 encourages agencies to promote early exchanges of information prior to the release of the Request for Proposals (RFP). The purpose of exchanging information is to improve the understanding of government requirements and industry capabilities, thereby allowing potential bidders to judge whether or how they can satisfy the government's requirements. An early exchange of information can identify and resolve concerns regarding: the acquisition strategy, including the proposed contract type; terms and conditions; acquisition planning schedules; requirements; statements of work; data requirements; and any other industry concerns or questions. The FAR also identifies techniques to promote early exchanges of information, including industry or small business conferences, public hearings, market research, and one-on-one meetings with potential bidders. On June 27, 2005, the FBI held an Industry Day to exchange information with potential bidders. All NIH CIO-SP2i contractors were invited to participate. According to the FBI, the potential contract bidders attending the session submitted both contractual and technical questions. However, the FBI would not provide these questions for our review because they were deemed procurement sensitive. On August 5, 2005, the FBI issued an RFP with responses due by September 19 and a contract award date of November 15. According to FBI officials, the due date for the proposals was extended one week to September 26, 2005, because vendors needed more time to complete the technical, management, and cost sections of the proposal. Subsequently, the contract award date was rescheduled for December 31, 2005, and later postponed again to an unspecified date in 2006. The FBI said that the source selection evaluation team, during its initial review of the proposals, identified the need for additional data from the bidders. As a result, the FBI said it will not establish a new contract award date until the source selection evaluation team receives and reviews the additional data. According to the FAR § 15.203, RFPs for competitive acquisitions should state the government's requirements, anticipated terms and conditions that apply to the contract, information required in the bidder's proposal, and factors that will be used to evaluate the proposal. To meet this requirement, the Sentinel RFP contained the following documents.
Based on the above, in our judgment the FBI issued the Sentinel RFP in accordance with the FAR requirements. While delays have occurred in awarding a contract for Sentinel, we believe it better for the FBI to take a reasonable amount of time at the outset of the project to ensure that the bidders fully understand the FBI's needs, system specifications, and expectations. According to Sentinel program manager, The FBI is evaluating the proposals based on the following criteria.
The FBI solicited assistance from federally funded research and development centers and other organizations for administrative, technical, and cost analysis support during source selection. These companies were also used as advisors in the evaluation of the proposals. However, the FBI retained the responsibility for selecting the contractor. At the end of source selection, the FBI intends to award a cost-plus-award-fee task order contract to develop the Sentinel system. A cost-plus-award-fee contract provides an estimated cost plus a fee consisting of a base amount fixed at inception of the contract and an award amount. The award amount is a pool of dollars available to the vendor to earn based on performance. The government makes the award fee determination based on periodic evaluations of vendor performance. One important aspect of a cost-plus-award-fee contract is that the award fee amount must be sufficient to motivate the vendor's performance. According to the Sentinel Award Fee Plan, the FBI anticipates capping the overall contract award amount for the development of Sentinel at 12 percent of development costs. This type of contract is common for large government IT projects. In our 2005 report on Trilogy, we stated our concerns with the cost-plus-award-fee contract as it was implemented by the FBI in that project. The cost-plus-award-fee contract used for Trilogy did not: (1) require specific completion milestones, (2) include critical decision review points, and (3) provide for penalties if the milestones were not met. However, the FBI's improved management processes and controls should minimize the risk of such problems recurring for Sentinel since the FBI intends to establish clear milestones, penalties for not meeting milestones, and critical decision review points. Executive Order 13356 requires that federal agencies design information systems with priority given to the interchange of terrorism information among agencies. Although the FBI has planned extensively for information to be shared among its divisions and offices, we found that it has expended little effort in assessing information sharing needs with other federal agencies. In particular, we have no assurance that the FBI has identified all external systems with which Sentinel must connect. While the Sentinel PMO told us that all external interfaces have been identified, we found that the external information sharing requirements for Sentinel have not yet been fully established but are scheduled to be completed by April 2006. Because these requirements have yet to be established, we anticipate a modification to the contract. In our opinion, such modifications represent a potential risk of requirements creep. The FBI is developing Sentinel using architectural models not widely used in the Department of Justice, which may require retrofitting or modifying other Department information systems as well as those of other agencies to effectively share information with Sentinel. The cost, extent, and timing of those modifications are not known. In our judgment, the FBI needs to focus more attention on the sharing of information between Sentinel and other agencies' data systems in these early stages of Sentinel's development. As discussed below, if Sentinel is developed without defining adequate external information sharing requirements, the system may not meet the information sharing mandate of Executive Order 13356, and costs may escalate due to the addition of these requirements later. Information Sharing Requirements During our audit, we interviewed several FBI and Department officials to better understand the process used to identify Sentinel's information sharing requirements. We found that the process the FBI used to identify the internal information sharing requirements was extensive, while the process to identify external information sharing requirements and compatibility appeared non-existent. According to the FBI, during the development of Sentinel's requirements system engineers held working sessions with future Sentinel users in the FBI to gain an understanding of what the system needed to do. The results of these sessions were compiled into a working draft of the Sentinel system requirements, which was then circulated to internal users for comment. According to FBI officials, approximately 1,200 comments were received, and many were integrated into the final systems requirements document. As a result of this interaction with internal users, the Sentinel requirements detailed how the system should interact with internal systems. For example, the system requirements show how data would be entered into and extracted from Sentinel as well as how Sentinel will generate reports currently produced by other FBI systems. In response to our concerns about information sharing, the FBI CIO stated that the FBI is working with the OMB, DHS, and the Directorate of National Intelligence (DNI) to ensure external interface requirements are adequately considered. However, the FBI CIO noted that while the OMB is taking steps to encourage external agencies' involvement, the level of involvement of these agencies cannot be controlled by the FBI. With respect to external IT system connections with Sentinel, the FBI said that in July 2005 it invited the Department of Homeland Security (DHS), the Drug Enforcement Administration (DEA), and the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) to participate in its development of Sentinel's requirements and has since begun discussions with the OMB and DNI on the need for system connections. We interviewed representatives from the DHS, DEA, and ATF to determine the extent of each agency's involvement in the development of Sentinel's requirements. The DHS representative stated that the DHS was given the opportunity to review the requirements document after the document was finalized by the FBI. The DHS has committed to providing the FBI with subject matter experts for 3 years in the areas of Enterprise Architecture, system engineering, security, privacy, and data to the project. At the time of our audit, the DHS was in the process of identifying the personnel to detail to the FBI. A DEA official stated that the FBI initially wanted the DEA to participate in an advisory capacity on the Sentinel steering committee and to have someone assigned full-time to Sentinel. While the DEA was not able to provide a full-time staff member, two officials participated on the steering committee. In addition, a DEA official reviewed the requirements for Sentinel to ensure that Sentinel addressed DEA information sharing needs. Although the DEA plans to deploy its own new case management system to its field offices in early 2006, the DEA said it intends to stay abreast of any developments with Sentinel. The DEA anticipates that staying informed about Sentinel will enable it to make changes to its case management system as the Sentinel project develops, thereby reducing the need of major retrofitting after Sentinel is completed. However, before Sentinel can connect with the DEA's case management system, a gateway from the classified operating environment of Sentinel to the sensitive but unclassified environment of the DEA's case management system must be established. Overall, DEA managers said they believe that Sentinel will meet the agency's information sharing needs as long as the FBI executes the project as planned. ATF officials told us that in late September 2005, an ATF official met with the Sentinel program manager to introduce himself as a point of contact for the ATF and provide information about the ATF's research into off-the-shelf products to enhance case management inquiry capability and facilitate information sharing. ATF officials said that they had not reviewed any of the requirements for Sentinel, and have had no other involvement with Sentinel. According to the ATF, it is too early in the Sentinel project for it to determine whether any retrofitting of ATF programs will be required once Sentinel is completed to enable information sharing to occur between the two agencies. During our audit work, we reviewed briefing documents, prepared by the FBI Office of IT Program Management for the FBI Deputy Director, in which the FBI indicated that the external interfaces for information sharing with the intelligence and law enforcement communities were not well-defined. When questioned about its uncertainty regarding Sentinel's compatibility with other agencies' systems, the FBI said that it has identified all known external interfaces that would fall under the FBI's information-sharing requirements. In addition, the FBI said that previously agreed-upon standards for information sharing across the law enforcement, intelligence, and defense communities will be followed in the development of Sentinel. However, we have not seen evidence of a comprehensive list of these information-sharing requirements. In fact, an FBI division head told us that the FBI's list of external information-sharing requirements should be completed by April 2006. As noted previously, if Sentinel is developed without adequately defining such external information sharing requirements, the system may not meet the information sharing mandate of Executive Order 13356 and the cost of the project may escalate because of the inclusion of these requirements at a later date. Target Architecture Sentinel will be developed using the Global Justice Extensible Markup Language (XML) Data Reference Model (GJXDM) and its extension, the National Information Exchange Model (NIEM). (See Appendix 9 for a discussion of these models.) The GJXDM and NIEM can make information exchange substantially more efficient by defining how information should be documented. In addition, the intelligence agencies connected to Sentinel will use the Terrorist Watchlist Person Data Exchange Standard.20 The FBI expects its new investigative case management architecture to capture and define processes for performing investigations and for collecting, controlling, analyzing, and sharing law enforcement data. Consequently, the target architecture for Sentinel that is expected to enable greater information sharing and improved management reporting is a key deliverable of the Sentinel case management system. According to a Department of Justice system architect, the GJXDM is not yet in use in most of the systems in the Department. However, he said the Department is moving forward on a number of initiatives to ensure its broader implementation. We believe the FBI and the Department need to focus more attention on this connectivity issue, because external entities' systems have not been developed with the same architectural model. Therefore, retrofitting or modifying the external agencies' systems may be necessary, and the cost, extent, and timing of such retrofitting is unknown at this time. According to FBI officials, external collaboration, including information sharing with the intelligence community and law enforcement partners, is envisioned with secure connections to a data mart.21 The following figure depicts the FBI's target architecture for such external information sharing.
The terrorist attacks of September 11, 2001, underscore the need for agencies involved in combating terrorism to be able to communicate with one another effectively. An intelligence agency may have only partial information on a suspected terrorist, but when coupled with information that other agencies possess, a threat may become more clear. In our judgment, there is no assurance that the requirements for Sentinel have been sufficiently defined to allow such interagency information sharing without potentially costly and time-consuming modification of agencies' existing systems to achieve compatibility with Sentinel. While Sentinel is first and foremost a system that must address the FBI's needs, in our judgment it may not serve the FBI's goal to prevent future terrorist attacks if this new system is isolated from information that exists within other agencies' information systems. Federal Investigative Case Management System In addition to developing its own case management system, the FBI is also the lead agency for the interagency Federal Investigative Case Management System (FICMS) initiative, as stated in a memorandum of understanding (MOU) signed by the FBI, DOJ, and DHS CIOs in June 2005. As lead agency, the FBI is expected to develop an architectural framework that will establish case management data and technology standards that enable electronic information sharing among government agencies. In April 2005, the FBI developed a draft FICMS framework which, according to the FBI CIO, was submitted to the Department for consideration. He added that the Department is refining the draft framework into a more mature framework. The June 2005 MOU also states that Sentinel will be the first implementation of the FICMS framework. The FBI CIO stated that the FBI is using the draft framework to drive the development of Sentinel, and when Sentinel is completed it will provide the FICMS framework with various case management services that can be adopted by other agencies. According to the 2005 MOU, two mission needs drive the development of the Sentinel project as the initial implementation of the FICMS:
The DHS said it provided $500,000 in FY 2005 to the Department of Justice for FICMS and will contribute up to that amount in FY 2006. A DHS official said that the DHS would have to wait and see if the FBI establishes its business processes within Sentinel in such a way that allows the processes to be modified to meet the needs of other agencies or not. However, if the FBI develops Sentinel as intended - using a service-oriented architecture - the DHS anticipates using approximately 40 to 60 percent of the system. Other potential users of the FICMS framework outside the Department of Justice include the Departments of Energy and Treasury, and the DNI. Therefore, the FBI should more closely consult with other intelligence and law enforcement agencies as the FBI moves forward in developing Sentinel. In our judgment, the FBI has taken a variety of positive steps to address its past IT development mistakes and to plan for the development of Sentinel. Specifically, the FBI has made significant progress by developing ITIM processes, a more mature Enterprise Architecture, and other management improvements since the Trilogy project, including establishing a Sentinel Program Management Office. However, we have several concerns about the project that require action and continued monitoring by the FBI, the OIG, and other interested parties: (1) the incomplete staffing of the PMO, (2) the FBI's ability to reprogram funds to complete the second phase of the project without jeopardizing its mission-critical operations, (3) Sentinel's ability to share information with external intelligence and law enforcement agencies and provide a common framework for other agencies' case management systems, (4) the lack of an established EVM process, (5) the FBI's ability to track and control Sentinel's costs, and (6) the lack of complete documentation required by the FBI's ITIM processes. Unlike during its failed VCF effort, the FBI now has a maturing Enterprise Architecture and a sound ITIM process in its LCMD. We found that the FBI generally is managing the Sentinel project in accordance with the LCMD. By following the LCMD, the FBI appears to have implemented adequate management controls through a variety of review boards and other oversight structures. This includes the identification of project risks and the development of mitigation strategies for those risks. The addition of an effective EVM process will also enhance the FBI's control over the project cost and schedule. According to the FBI, full implementation of an EVM process for the Sentinel project is scheduled to occur approximately 2 months after the Sentinel contract is awarded. Based on our initial review, the FBI's EVM strategy appears adequate. We will monitor the FBI's implementation of EVM in future audits. The FBI continues to build a PMO specific to the Sentinel project, an entity critical to the project's successful management continuity and oversight. However, as of January 30, 2006, the Sentinel PMO was staffed with 51 of the 76 staff the FBI determined are needed to successfully manage Sentinel. Unless the FBI fully staffs the PMO during the first phase of the project, the FBI runs the risk of not being able to oversee adequately Sentinel's aggressive delivery schedule. We believe that it is imperative for the FBI to fully staff the PMO with qualified personnel as quickly as possible and to continue to follow the guidelines, requirements, and controls established in the LCMD. While we support in principle the FBI's initial $97 million reprogramming request for the Sentinel program, we have concerns about the effect of a second large reprogramming request on the FBI's mission-essential operations. It is not clear to us how the FBI can effectively carry out its wide-ranging and complex mission if funds of this magnitude need to be diverted from other FBI programs in a second reprogramming. Additionally, the FBI's ability to track Sentinel's costs needs to be firmly established by the time the contract is signed to ensure that all of the funding for the project is adequately accounted for. Although the FBI has tried to use its past work on VCF in the Sentinel effort, neither the FBI nor we could quantify how much hardware and development work from the VCF had been transferred to the Sentinel project. With regard to information sharing, we found that the development of Sentinel and the architecture for the interagency FICMS are being performed largely in parallel. Sentinel is being developed to be compliant with the GJXDM language and data reference and the Terrorist Watchlist Person Data Exchange Standard. There are risks associated with this tandem development approach, because Sentinel is essentially defining the standards for FICMS. Furthermore, the ultimate connectivity between Sentinel and external systems remains unclear, as most Department of Justice systems are not using the GJXDM model and may require significant modifications to facilitate information exchange. The cost and extent of those modifications are unknown at this time. In our judgment, Sentinel's requirements, including those for information sharing, must be firm before work begins on the project in order to avoid delays and cost increases and if Sentinel is to serve one of its intended purposes - to provide an investigative case management system that other federal law enforcement agencies can adapt for their own use and that will allow for information sharing among federal law enforcement and intelligence community agencies. Although the FBI appears to have thoroughly examined internal FBI information sharing requirements in developing Sentinel, it has not ensured compatibility with other agencies' systems. We have found that in addition to continuing to develop an EVM process and the capability to track costs, the FBI has yet to complete system security and verification and validation plans as established in the FBI's ITIM. These plans, which the FBI intends to complete after the Sentinel contract is awarded, are required to ensure that the system meets the FBI's security requirements and is implemented according to established control mechanisms. The OIG will continue to monitor and periodically issue audit reports throughout the Sentinel project in an effort to track the FBI's progress and identify any emerging concerns over the cost, schedule, technical, and performance aspects of the project. As a result of our review of the pre-acquisition phase of the Sentinel project, we make the following recommendations. We recommend that the FBI:
Footnotes
|
| « Previous | Table of Contents | Next » |