Integrating budget and performance has been an important government initiative to ensure “that performance is routinely considered in funding and management decisions, and that programs achieve expected results and work toward continual improvement.”¹ Over the years, the integration of budget and performance has been addressed in the Government Performance and Results Act (GPRA) of 1993, the Reports Consolidation Act of 2000, and is also addressed in the implementation guidance in the Office of Management and Budget (OMB) Circulars A‑11 and A‑136.² In order to measure performance, federal agencies develop long‑term strategic plans to provide the President, Congress, and the public an overview of their goals and objectives and the problems and challenges they face.³

The Department of Justice's (DOJ) current Strategic Plan identifies three general strategic goals.⁴ However, due to the timing of our audit work and our audit scope, this report discusses the FYs 2003‑2008 Strategic Plan, which identified the following four strategic goals:

1. Prevent Terrorism and Promote the Nation's Security

2. Enforce Federal Laws and Represent the Rights and Interests of the American People

3. Assist State, Local, and Tribal Efforts to Prevent or Reduce Crime and Violence

4. Ensure the Fair and Efficient Operation of the Federal Justice System

According to OMB Circular A-11, Section 210, Preparing and Submitting a Strategic Plan, a strategic goal is a statement of aim or purpose that defines how an agency will carry out a major segment of its mission over a period of time. Strategic goals are used to group multiple long-term program outcome goals that identify what DOJ hopes to achieve several years into the future. Outcome goals, which represent the activities that are considered DOJ’s highest priorities, are measured annually by key indicators.

For example, within Strategic Goal II, one outcome goal is to “Shut Down a Cumulative Total of 1,850 Child Pornography Websites or Web Hosts.” The Federal Bureau of Investigation’s (FBI) key indicator measuring the annual progress towards achieving this outcome goal is the “Number of Child Pornography Websites and Web Hosts Shut Down.”

Figure 1 illustrates the relationship between strategic goals, outcome goals, and key indicators.

FIGURE 1: RELATIONSHIPS BETWEEN STRATEGIC GOALS, OUTCOME GOALS, AND KEY INDICATORS

Recent Guidance

In November 2007, the President issued Executive Order 13450, Improving Government Program Performance, which details the need for performance goals that are measurable and clearly defined. Specifically, in addition to several other requirements, the Order directs agency heads to establish clear annual and long-term goals defined by objectively measurable outcomes and plans for achieving them.

Additionally, the Order states that the head of each agency assists in making budget and appropriation recommendations to Congress that are justified based on objective performance information. Finally, the Order discusses the establishment of Performance Improvement Officers who will supervise the performance management activities.

In our judgment, Executive Order 13450 places additional emphasis on the importance of key indicators and the quality of data used to report on these indicators, since those indicators are the foundation for agencies to measure how well they accomplish their goals and assist in their budgeting processes.

Background

During our audit, Justice Management Division (JMD) officials stated that key indicators are the best representation of DOJ’s current activities, objectives, and direction for the future. Further, key indicators demonstrate progress toward achieving outcome-oriented goals. DOJ components and JMD work together to select key indicators that best represent a measurement of DOJ’s activities. Each key indicator must be approved by DOJ management and the OMB.

Key indicator performance information is published in the components’ annual budget requests within the Budget and Performance Summary. This information discusses DOJ achievements and plans for future accomplishments. Additionally, this information can be used for budget and management decisions. Therefore, accuracy is essential in key indicator reporting. JMD officials stated that to ensure confidence in DOJ reporting, the Department must provide the most accurate data available.

JMD officials also stated that when a key indicator target is not achieved, the DOJ component may meet with the Assistant Attorney General for Administration to discuss the program performance and the actions necessary to achieve that target in the future.

Key indicators are part of DOJ’s annual Budget and Performance Summary, which accompanies the President's Budget. In addition, at the close of each fiscal year, key indicator performance data is reported in the Department’s Performance and Accountability Report (PAR), which consolidates financial and performance information into a single report.

The PAR includes a discussion on the data collection and storage processes, data validation and verification processes, and any data limitations for each key indicator. The PAR also reports key indicator performance data for the reporting period and includes narrative sections describing whether the key indicator target was achieved and progress toward achieving the outcome goals.⁵ If key indicator performance data for the fiscal year is incomplete in that year’s PAR, revised data can be included in the next calendar year’s Budget and Performance Summary and the subsequent PAR.

Department of Justice Guidance

Annually, JMD Budget Staff assist in developing and revising the DOJ Financial Statement Requirements and Preparation Guide. Chapter 4 of the FY 2006 version of this guide provides specific information on the Management Discussion and Analysis (MD&A) that the components submit to JMD. That chapter contains the key indicator requirements that must be included in the MD&A and lists the key indicators that each component is responsible for reporting.

Reporting by the components begins in May of each year when they provide a draft MD&A to JMD. Revisions can be made to the MD&As until the final drafts are submitted to JMD in October. Then the actual performance data and a discussion of performance for the fiscal year are added. JMD extracts the information provided in the components’ final draft MD&As and certified final MD&As to prepare DOJ’s PAR. The PAR consolidates the components’ MD&As, so JMD does not generally adjust the information provided by the components in their MD&As.

The PAR must be submitted to the President, Congress, and the Office of Management and Budget (OMB) 45 days after the September 30 fiscal year end. The draft PAR should be provided to OMB 10 working days before the due date for review and clearance. The established timeline for DOJ’s reporting process is illustrated in Figure 2.

Figure 2: DOJ Key Indicator Reporting Process Timeline

[Image Not Available Electronically]

Source: DOJ Financial Statement Requirements and Preparation Guide, August 2006; OMB Circular A-136; OMB Circular A-11; and JMD management

Audit Approach

The objectives of our audit were to determine whether each key indicator met the following criteria: (1) data collection and storage processes were complete and accurate, (2) data validation and verification processes were complete and accurate, and (3) data limitations provided by the agency were complete and accurate. Our audit did not assess whether the key indicators were the best measures to assess progress on achieving DOJ’s goals.

Our audit scope included the FY 2006 PAR, which contained 28 DOJ key indicators. Additionally, the audit scope included the revisions made to the performance data reported in the FY 2006 PAR, which are disclosed in the 2008 Budget and Performance Summary. Our audit scope did not include seven key indicators that were either discontinued or long-term measures still in process of developing a baseline at the time of our audit.⁶

Our audit work was conducted at JMD and the 17 DOJ components responsible for reporting on 21 key indicators. These key indicators are listed in Table 1, categorized by the four strategic goals reported in the DOJ’s Strategic Plan for FYs 2003‑2008.

TABLE 1: DOJ KEY INDICATORS INCLUDED IN THE AUDIT

As part of our audit, we reviewed current laws, regulations, guidelines, and policies in order to understand the key indicator reporting requirements with which federal agencies must comply. Our fieldwork consisted of interviewing component personnel; reviewing relevant policies and procedures; reviewing processes related to data collection, storage, validation, and verification; and discussing applicable data limitations with component personnel. Additionally, we verified the performance data reported in the FY 2006 PAR and the 2008 Budget and Performance Summary.

Our fieldwork was conducted while DOJ components were preparing the MD&As for the FY 2007 PAR, which was issued in November 2007. Due to the timing of this audit, we communicated our preliminary audit results to DOJ components and JMD officials before the PAR was completed. This process was important to ensure that components were able to implement revisions to the FY 2007 MD&As, which were used to compile the FY 2007 PAR, in response to our preliminary findings. Delayed presentation to the components of the issues and deficiencies identified in our audit would have resulted in the Department’s reporting of inaccurate and incomplete information in the DOJ’s FY 2007 PAR. As a result, as part of this audit we analyzed the FY 2007 PAR, which was modified in response to our preliminary findings, and we are including the components’ revisions throughout this report.⁸

We determined that 12 key indicators out of the original 21 had no definitive problems. Therefore, we made no recommendations regarding those key indicators. We identified deficiencies and issues related to 9 of the 21 key indicators included in our audit. These deficiencies and issues related to: (1) the data collection processes, which could, and in some cases did, result in under- or over‑reporting; (2) incomplete and inaccurate data; (3) the scope of the data collected by some of the components was incomplete; (4) the scope of the data presented in the FY 2006 PAR for some key indicators was inaccurate; (5) detailed information in the FY 2006 PAR that did not directly illustrate what the title described; and (6) some key indicators did not fully disclose data limitations and other information that needed to be disclosed in the PAR.

Generally, we found that component management had implemented a variety of processes to review and validate the MD&As prior to providing them to JMD for compilation in the PAR. Yet, the deficiencies and issues identified in 9 of the 21 key indicators suggest that the review and validation processes were not sufficiently effective to ensure accurate and complete performance data and an accurate presentation of the performance information in the MD&As and the PAR. In our opinion, component management should enhance validation processes by verifying the accuracy of MD&A narratives covering the key indicators and the supporting information necessary to ensure the accuracy of the key indicator performance data. In addition, components should communicate to staff the need for accuracy of the key indicator information presented in the MD&A for PAR reporting.

Furthermore, while we recognize that JMD officials cannot currently verify or adjust information provided by the components, we believe that JMD should expand its role to emphasize improved accuracy of the key indicator performance data reported in the components’ MD&As, which are subsequently compiled for the PAR. To accomplish this, we believe that through formal policy JMD should require the necessary supporting information from the components to authenticate the accuracy of the key indicator performance data.

The deficiencies and issues identified in 9 of the 21 audited key indicators are discussed in detail in the Findings and Recommendations section of this report, organized by the 4 strategic goals reported in the DOJ’s Strategic Plan for FY 2003‑2008. Detailed information regarding our audit objectives, scope, and methodology appears in Appendix I.

1. U.S. Office of Management and Budget, The Federal Government is Results‑Oriented: A Report to Federal Employees, August 2004, which summarizes how the federal government is focusing on results, the role of the President’s Management Agenda, and the key to future success.

2. This guidance is discussed further in Appendix II.

3. All federal agencies are required by GPRA to develop a 5-year Strategic Plan.

4. The current three strategic goals from the DOJ Strategic Plan for FYs 2007‑2012 are: (1) Prevent Terrorism and Promote the Nation’s Security; (2) Prevent Crime, Enforce Federal Laws, and Represent the Rights and Interests of the American People; and (3) Ensure Fair and Efficient Administration of Justice.

5. For a sample key indicator narrative, see Appendix V.

6. See Appendix III for a comprehensive list of all 28 key indicators included in the FY 2006 PAR and Appendix IV for background information on these key indicators.

7. The following six litigating divisions are included in this indicator: Antitrust, Civil, Civil Rights, Criminal, Environment and Natural Resources, and Tax Divisions. Throughout this report we use the term “litigating divisions” to refer to these six divisions. The key indicator data for the Executive Office for U.S. Attorneys (EOUSA) and the litigating divisions is consolidated by JMD Budget Staff.

8. Our review of the FY 2007 PAR and any changes that were made as a result of our audit are discussed in detail in the Findings and Recommendations section of this report.