Return to the USDOJ/OIG Home Page

Federal Bureau of Investigation's Management of Information Technology Investments

Report No. 03-09
December 2002
Office of the Inspector General


APPENDIX I

OBJECTIVES, SCOPE, AND METHODOLOGY

Objectives

The primary objectives of the audit were to: (1) determine whether the FBI was effectively managing its IT investments; and (2) assess the FBI's IT strategic planning and performance measurement activities.65 In determining whether the FBI was effectively managing its IT investments, we also examined the FBI's efforts in developing enterprise architecture and project management functions. These two functions both complement and facilitate IT investment management. Additionally, we performed a case study of Trilogy, a significant IT project, to determine how the FBI's IT investment management practices affected the project's progress.

Scope and Methodology

The audit was performed in accordance with Government Auditing Standards, and included tests and procedures necessary to accomplish the audit objectives. We conducted work at: (1) FBI Headquarters in Washington, D.C. (2) FBI Laboratory facilities in Quantico, Virginia, and (3) FBI field offices in New York City, New York; Los Angeles, California; Chicago, Illinois; Miami, Florida; and Washington, D.C.

To perform our audit, we conducted approximately 85 interviews with 70 officials from the FBI (including field offices), DOJ, OMB, and GAO. The FBI officials interviewed were from the Director's office, Information Resources Division, Criminal Justice Information Services Division, Laboratory Division, Inspection Division, and Finance Division. Additionally, we reviewed over 200 documents related to IT management policies and procedures, project management guidance, strategic and program plans, IT project proposals and management plans, budget documentation, organizational structures, Congressional testimony, and prior GAO and OIG reports.

To determine whether the FBI is effectively managing its IT investments, we applied the GAO's ITIM framework and the associated assessment method. As part of the Framework's assessment method, the FBI conducted a self-assessment of its IT investment management activities using the Framework. The self-assessment included those processes that the FBI had in place as of the beginning of our audit. Additionally, the self-assessment covered those processes that the FBI was planning to implement based on its IT Investment Management Model and Transition Plan.66 In the self-assessment, the FBI indicated whether it executed each of the key practices in Stages Two through Five. The FBI asserted that it executed 27 of the 38 key practices from Stage Two, 3 of the 53 key practices from Stage Three, and none of the key practices from Stages Four and Five. Additionally, it stated in the self-assessment that the IT Investment Management Model and Transition Plan would be supplemented so that the FBI would eventually implement the critical processes necessary to achieve Stage Four maturity, as well as many of the key practices from Stage Five.

Because FBI officials stated in the IT Investment Management Model and Transition Plan that its initial goal was to advance to Stage Two, by default, the FBI indicated that it was in Stage One maturity. As a result, we validated the FBI's execution of the 38 key practices from Stage Two, and assessed the FBI's ability to improve its IT investment management practices through implementation of the ITIM process defined in the IT Investment Management Model and Transition Plan.

The Stage Two critical processes and key practices we examined focus primarily on the FBI's ability to effectively select and control its IT investments. To determine whether the FBI had implemented the critical processes and key practices in Stage Two, we evaluated policies, procedures, and guidance related to the FBI's IT investment management activities.

We compared the evidence collected from our document reviews and interviews to the key practices and critical processes defined in the Framework. Because the Framework is a hierarchical model, the rating of each critical process is dependent on the key practices below it. Therefore, we first rated the key practices. In accordance with the Framework's assessment method, we rated a key practice as "executed" when we determined that the FBI was executing the key aspects of the practice. A key practice was rated as "not executed" when we determined that there were significant weaknesses in the FBI's execution of the key practice and the FBI offered no adequate alternative, or when we found no evidence of a practice during the review.

Once the key practices were rated, we rated each of the Stage Two critical processes we reviewed. A critical process was rated "implemented" if all of the underlying key practices were rated as being executed. A critical process was rated as "not yet implemented, but substantial progress made" if over half, but not all, of its underlying key practices were rated as being executed. A critical process was rated as "not implemented" when there were significant weaknesses (i.e., fewer than 50 percent of the key practices had not been implemented) in the FBI's implementation of the underlying key practices and no adequate alternative was in place.

Beginning in March 2002, the FBI pilot tested the select phase of its new ITIM process. To measure the FBI's progress in improving the execution of Stage Two key practices during the course of our audit, we documented the key practices executed: (1) before the implementation start of the test pilot in March 2002, and (2) as of the end of our fieldwork in June 2002.

Our assessment of the FBI's ability to improve its IT investment management consisted of the following four areas:

  1. the Plan's coverage of Stage Two key practice activities that were not being executed during our fieldwork;
  2. the amount of participation from ITIM users in developing the ITIM process;
  3. the support from the project management function; and
  4. the support from the enterprise architecture function.

In addition, we performed a case-study of the Trilogy project to determine how the FBI's IT investment management practices have affected its progress. Trilogy was selected for a case-study because it is currently the FBI's most expensive IT project and its implementation is critical to the FBI's ability to achieve its mission. Trilogy is intended to provide the right hardware and software tools to the FBI's agents and analysts, enable the FBI's investigative personnel to easily and rapidly find, present, and manipulate required information, and transport and share information quickly and efficiently across the Bureau. We performed the case-study both at FBI Headquarters where we interviewed individuals responsible for the project, as well as at five FBI field offices (New York, District of Columbia, Los Angeles, Miami, and Chicago) where we interviewed individuals responsible for assisting in the deployment of the new system, as well as agents utilizing the system.

To assess the FBI's IT strategic planning and performance measurement activities, we reviewed strategic and performance planning documentation from the FBI, the DOJ's Strategic Plan for FYs 2001 to 2006, the DOJ's FY 2001 Performance Report, the DOJ's FY 2002 Revised Final Performance Plan for FY 2003, and the DOJ's IT Strategic Plan. To supplement our document review, we also interviewed officials responsible for creating FBI strategic and performance plans.


Footnotes

  1. During our audit fieldwork, we initiated work relating to a third objective: to determine if the FBI has implemented prior information technology related recommendations and improved its information technology. We will issue a separate report on this objective.
  2. Although the FBI's IT Investment Management Model and Transition Plan, issued in January 2002, states that its primary goal is to provide the conceptual framework for Stage Two maturity, it also outlines the steps the FBI must take to advance to Stage Four maturity in preparation to achieving Stage Five maturity.