Identification and Review of the Department’s Major Information Technology Systems Inventory

Audit Report 07-37
June 2007
Office of the Inspector General


Findings and Recommendations

Inventory of Major DOJ IT Systems and Related Costs

We identified 38 IT systems operated by, or under development in, 7 DOJ components as the inventory of major DOJ systems. The FBI has the largest number of IT systems in the inventory with 21 – clustered in the OCIO, Science and Technology Branch (STB), and National Security Branch (NSB) and Security Division. The DEA has seven systems on the list and JMD has six.20 Because these three components make up nearly 90 percent of the total inventory, we focused on their IT cost reporting practices in this audit.

The following chart shows the distribution of all 38 major DOJ IT systems and projects.

Distribution of IT Inventory

FBI-STB:11, FBI-CIO:5, FBI-NSB:5, DEA:7, JMD:6, EOIR:1, OJP:1, ATF:1, BOP:1.

Source: OIG

After an initial survey that included interviewing Department and component IT and finance staff, we determined no component-level or DOJ-wide systems collect data on individual IT system costs. To obtain the best available data on the 38 major IT systems in the inventory, we asked the managers of each system to provide us with detailed cost and other information on the systems they manage. System managers reported $5.7 billion in system costs through FY 2005 and an additional $9.3 billion estimated through FY 2012, for a total of $15 billion.

The IT systems managers completed an OIG-developed questionnaire, distributed through the DOJ CIO, that requested costs and timeframes associated with each phase of the project’s system development life cycle, total costs incurred through FY 2005, funding amounts and funding sources necessary to complete the system, contractor information, a brief description of the methods used to determine reported costs, and other information required to respond to the congressional request.

Based on the responses to our questionnaire and additional interviews of component IT staff, we confirmed that IT system cost reporting across the Department is fragmented and inconsistent. The methodologies used to track the costs of IT systems vary widely among and within components, and there is a general lack of controls necessary to ensure accuracy and completeness.

The primary objective of the OIG questionnaire was to obtain comparable cost data on all 38 major IT systems. The following table shows the detailed cost and the estimated funding required for the inventory of 38 major DOJ IT systems.

IT System Costs Reported on OIG Questionnaire

System To Date Costs
through
9/30/2005
Funding
Requests –
FY 2012
Total Costs
through
FY 2012
FBI - Science and Technology Branch
1 Integrated Automated Fingerprint Identification System
$ 1,515,162,000
$ 812,008,000
$ 2,327,170,000
2 Next Generation Integrated Automated Fingerprint Identification System
$ 14,094,000
$ 438,726,000
$ 452,820,000
3 National Instant Criminal Background Check System
$ 393,684,000
$ 408,706,000
$ 802,390,000
4 National Crime Information Center
$ 315,404,000
$ 162,276,000
$ 477,680,000
5 Law Enforcement Online Re-engineering/Relocate
$ 114,536,000
$ 237,989,000
$ 352,525,000
6 Law Enforcement National Data Exchange Rev 9/7
$ 25,235,000
$ 195,242,000
$ 220,477,000
7 Combined DNA Index System
$ 4,780,000
$ 92,759,000
$ 97,539,000
8 Digital Collection
$ 200,150,000
$ 205,530,000
$ 405,680,000
9 Electronic Surveillance Data Management System
$ 25,454,195
$ 148,120,000
$ 173,574,195
10 Biometric Reciprocal Identification Gateway /CJIS Interoperability Initiative
$ -
$ 346,662,000
$ 346,662,000
11 Computer Assisted Response Team Storage Area Network
$ 16,328,000
$ 92,859,000
$ 109,187,000
FBI - OCIO Branch
12 Sentinel
$ 4,300,000
$ 433,667,000
$ 437,967,000
13 Data Centers
$ 450,000,000
$ 200,895,322
$ 650,895,322
14 Technical Refresh Program
$ 24,400,000
$ 380,400,000
$ 404,800,000
15 Investigative Data Warehouse
$ 84,436,523
$ 135,065,296
$ 219,501,819
16 Multi-Agency Information Sharing Initiative Regional Data Exchange
$ 7,957,843
$ 10,000,000
$ 17,957,843
FBI - National Security Branch and Security Division
17 Terrorist Screening Center
$ 22,030,000
$ 453,676,000
$ 475,706,000
18 Foreign Terrorist Tracking Task Force
$ 82,046,000
$ 142,900,000
$ 224,946,000
19 Security Management Information System
$ 3,633,190
$ 76,993,000
$ 80,626,190
20 Information Assurance Technology Infusion
$ 2,554,379
$ 5,300,000
$ 7,854,379
21 Sensitive Compartmented Information Operational Network
$ 38,082,620
$ 305,439,304
$ 343,521,924
DEA
22 Model 204 Corporate Systems
$ 334,556,000
$ 96,159,860
$ 430,715,860
23 E-Commerce-Controlled Substances Ordering System
$ 24,315,070
$ 79,310,032
$ 103,625,102
24 EPIC Information Systems
$ 63,821,000
$ 71,830,108
$ 135,651,108
25 Concorde
$ 19,784,000
$ 50,813,000
$ 70,597,000
26 Firebird
$ 639,533,346
$ 551,397,000
$ 1,190,930,346
27 Merlin
$ 90,904,000
$ 109,730,144
$ 200,634,144
28 Organized Crime & Drug Enforcement Task Force Fusion Center System
$ 3,524,487
$ 140,331,409
$ 143,855,896
JMD
29 Integrated Wireless Network
$ 752,302,000
$ 1,818,096,000
$ 2,570,398,000
30 Unified Financial Management System
$ 25,580,973
$ 403,062,668
$ 428,643,641
31 Litigation Case Management System
$ 3,500,000
$ 99,563,360
$ 103,063,360
32 Classified Information Technology Program
$ -
$ 38,118,003
$ 38,118,003
33 Justice Consolidated Office Network
$ 193,567,064
$ 270,078,542
$ 463,645,606
34 Public Key Infrastructure
$ 9,511,265
$ 157,900,000
$ 167,411,265
ATF
35 National Integrated Ballistics Information Network
$ 125,000,000
$ 122,000,000
$ 247,000,000
BOP
36 Inmate Telephone System-II
$ 361,693
$ 205,095
$ 566,788
EOIR
37 eWorld
$ 19,482,000
$ 26,352,000
$ 45,834,000
OJP
38 Grants Management System
$ 77,753,000
$ 23,702,000
$ 101,455,000
Totalsa $5,727,763,649 $9,343,862,143 $15,071,625,792
Source: OIG analysis of completed questionnaires

a Due to rounding, values do not sum.

After obtaining and analyzing the data provided by the individual system managers, we attempted to verify costs associated with one IT system from each of the three DOJ components (FBI, DEA, and JMD), which collectively represent over 97 percent of all the IT spending in our audit inventory.

In the following sections of this report, we briefly discuss our consideration of the Department’s IT Capital Planning and Investment Control environment, budget, and financial systems in order to complete our verification of the Department’s major IT inventory.

IT Capital Planning and Investment Control Environment

Since FY 2001, Congress has authorized more than $12 billion for DOJ IT equipment, software, and services – an average of more than $2 billion annually.21 DOJ spending authority represents approximately 4 percent of the $64 billion authorized on IT across the federal government in FY 2007 and 11 percent of DOJ’s annual budget.22 These funds are used to acquire new computers and other assets as well as to cover expenses associated with operating and maintaining legacy IT systems. Funding for new acquisitions and improvements to existing IT is referred to as Development, Modernization, and Enhancement (DME) costs, while operating expenses are known as “steady state” costs. In FY 2005, DOJ spending authority related to steady state activities amounting to nearly $1.5 billion – more than double the $702 million allocated to DME expenditures.

More than 40 DOJ components or organizational units use IT systems to assist DOJ’s 104,000 employees in the performance of their duties and to provide support for the state and local law enforcement community.

In addition to the DOJ CIO, each major component has a CIO.23 Each of these components has either its own Capital Planning and Investment Control (CPIC) policies and IT investment management (ITIM) processes, or follows DOJ’s policies and processes. In prior audit reports, the OIG reviewed the ITIM processes at three DOJ components – the FBI, DEA, and JMD – and found they are in the beginning or middle stages of their ITIM processes.24

Although the DOJ and component CIOs have important roles and input to decision-making on IT matters, they often do not control the budgets for the IT systems they are responsible for monitoring. DOJ officials could not estimate what percentage of DOJ spending is controlled by offices other than the OCIO, because each component budgets its IT spending differently.

Elements of an agency’s CPIC process include IT planning, establishing an IT architecture, and monitoring the status of IT systems. Although the DOJ CIO and component CIOs we reviewed have significant responsibilities for their organization’s IT systems, we found they also have varying degrees of control over those IT system budgets.

The Clinger-Cohen Act outlines the required content of an agency’s CPIC process:

Because we had difficulty obtaining independently verifiable cost data for the major IT systems in the inventory, we discussed with DOJ OCIO staff whether the DOJ and its components are complying with the Clinger-Cohen Act provision cited above.

OCIO officials told us that to meet the Clinger-Cohen Act requirements, they are now using Earned Value Management tools for IT investments under development. Because OMB has only recently required Earned Value Management for new IT investments, historical cost data for these systems would not have been available to use for this audit. Therefore, the Clinger-Cohen Act provisions would be met only for new systems under development.

The IT Investment Portfolio

A report known as the IT Investment Portfolio or Exhibit 53, required by the Clinger-Cohen Act and the Paperwork Reduction Act of 1995, is designed to help OMB and other federal agencies provide a full and accurate accounting of an agency’s IT investments. The Exhibit 53 is a product of the IT CPIC and ITIM processes. As the name suggests, the CPIC process should enable an agency to plan, acquire, and manage its IT acquisitions. During the CPIC process, the agency CIO is responsible for collecting and managing the data necessary for the preparation of the IT Investment Portfolio.

The IT Investment Portfolio is similar to the President’s Budget in that each item in the IT Exhibit 53, whether a system, project, or service, is reported with 3 years of associated spending. The 3 years used in the Exhibit 53 are the budget year, the current year, and the prior year. The spending reported for all years reflects the appropriated amounts or budgetary resources available for that year.

The costs of individual IT systems are usually included in the programs and activities of a component with other costs, including those of other IT systems. These other costs are called object classifications, or object classes.25 DOJ component CIOs do not generally compile the actual cost data they report in the IT Investment Portfolio because they rely on individual IT system managers to collect and report cost data that the CIOs review before it is submitted to OMB.

To inform Congress of IT spending, OMB provides Congress with an Exhibit 53 at the time it submits the President’s Budget. Both DOJ’s budget and Exhibit 53 include IT spending but use different formats to present the same appropriations – actual, enacted, and proposed. Both of these documents may appear to be integrated, but they are created for different purposes by separate offices and use different databases to gather information.

To collect and process most of the information needed to prepare DOJ’s budget, OMB uses the MAX budget system.26 DOJ enters its budget and financial data into the MAX system, which organizes the data using a series of schedules, including the object classification schedule.

The following table from Circular A-11 shows how various IT obligations should be assigned to object classes:

MAX Budget System Object Classes for IT Spending

Description of Obligation Object Class Number Object Class Title
IT services or rental of IT equipment 23.3 Communications, utilities, and miscellaneous charges
Operation and maintenance of IT systems by the private sector 25.7 Operation and maintenance of equipment
Operation and maintenance of IT systems by another Federal Government Account 25.3 Purchases of goods and services from Government Accounts
IT hardware and software 31.0 Equipment
IT supplies and material such as manuals, diskettes, and toner cartridge 26.0 Supplies and materials
IT consulting for management, studies, analyses, and evaluations, or engineering and technical services 25.1 Advisory and assistance services
Source: OMB MAX

In addition to the instruction on how to assign different types of IT obligations to object classes, OMB directs agencies to include employee wages in the personnel compensation and benefits object class even when these employees are working to develop an IT system.27

While the MAX budget system is used for the President’s Budget, OMB uses the Electronic Capital Planning and Investment Control (eCPIC) system, which is installed on agency intranets, to collect IT Investment Portfolio data. The eCPIC system is designed to manage and control IT spending and help prepare information requested by OMB. In addition to facilitating the preparation of the IT Investment Portfolio, the eCPIC system also contains the data used to prepare Exhibits 300 for individual IT systems.

The following table summarizes the differences between DOJ-related sections of the President’s Budget and the IT Investment Portfolio.

Comparison of the President’s Budget to the
IT Investment Portfolio

Cost Report President’s Budget IT Investment Portfolio
Scope All Spending IT Spending Only
DOJ amounts in
FY 2005
$28 billion $2 billion
OMB system to collect data MAX eCPIC
Related agency process Budget Execution Capital Planning and Investment Control
Related responsible agency officer Chief Financial Officer Chief Information Officer
Related agency staff preparing cost data Chief Financial Officer's Budget and Finance Staff Individual IT system owners
Related agency systems Core Financial Systems capturing all agency transactions - audited Various ad hoc methods and cost databases - no one system capturing all agency transactions - not audited
Source: OMB and DOJ documents

DOJ Core Financial Systems

Core financial systems are used for the funds management function of an agency. These systems should ensure that agency financial transactions are captured and processed in a uniform and consistent manner.

Data from DOJ components’ core financial systems is put in a format that defines all costs of individual IT systems. Without the ability to identify and extract data in a useable format, it is cumbersome and time consuming to identify the costs of these IT systems. Therefore, auditing the costs of the 38 major IT systems in DOJ’s inventory would, in essence, require a detailed audit of each system.

In this review, we focused on the core financial systems maintained by the FBI, DEA, and JMD. The FBI’s core financial system, Federal Management System (FMS), is a legacy system dating back to the 1980s. In general, FMS limits the FBI’s ability to process financial information effectively and efficiently, including timely financial statements.

At the start of our audit, we discussed with FBI officials familiar with FMS the possibility of extracting and summarizing cost data in FMS for the individual IT systems in the inventory. These officials told us that financial data in FMS is not coded in sufficient detail to allow this kind of independent identification and grouping for all IT systems.

We also met with DEA and JMD officials to determine whether their respective financial systems – the Federal Financial System (FFS) and Financial Management Information System (FMIS) – could be used to identify or verify costs of individual IT systems. Although the FFS and FMIS are newer systems and do not have the same degree of limitations as the FBI’s FMS, neither system can easily produce reports that identify all the costs associated with its components’ IT systems.

Despite the age and weaknesses of these financial systems, officials told us this financial data could have been more useful had accounts been created and transactions consistently coded to track IT systems as defined in the CPIC process. This did not occur because the primary cost tracking concern of the components is related to the overall budget and spending process rather than the CPIC process. Additionally, cost reporting for CPIC purposes is not specifically required by law for federal financial systems.28

Verification of IT System Cost Data

We selected three IT systems within the FBI, DEA, and JMD, – LEO, Concorde, and JCON, respectively – based on the results of the OIG-developed questionnaire. Within those three systems, we tested the accuracy and completeness of the actual cost data reported. Due to the lack of sufficient internal controls or any routine testing of the accuracy of the costs reported, we assessed the risk as “high” that all IT system costs may be understated. Accordingly, we planned our work and focused on determining whether the costs reported by the IT systems were complete. Testing for completeness without the ability to sample transactions from the entire financial system is considerably more difficult than simply confirming the reported costs exist. Although we used component financial and budget system data to the extent possible, there is no assurance that our work has identified all the costs associated with these three IT systems.

FBI Law Enforcement Online

Law Enforcement Online (LEO), which was formed through a cooperative agreement between the FBI and Louisiana State University (LSU), is the FBI’s Internet-based communication system and information service for law enforcement agencies nationwide. Thousands of police officers and other employees of local, state, and federal law enforcement agencies are able to access LEO 24 hours a day, 7 days a week.

Examples of the services available to users of LEO include:

In response to our questionnaire, the FBI Criminal Justice Information Services (CJIS) Financial Management Unit provided the cost information.29

LEO Costs
($ in millions)

FY Start Costs incurred through FY 05 Estimated Funding through FY 2012 Total Costs Through FY 2012
-- a $ 115 $ 238 $ 353
Source: OIG analysis of completed LEO questionnaire

a The FBI did not provide a LEO start date on its completed questionnaire.

Although the LEO response did not indicate a start date, FBI and LSU documents indicated the system was created in 1995 and transferred from a FBI unit to CJIS in 1997.

We met with CJIS Financial Management Unit officials to discuss the source of the amounts reported in our questionnaire and their methods for tracking actual costs. These officials told us the $115 million in costs through FY 2005 comprised personnel and non-personnel costs, but no indirect costs. In addition, non-personnel costs included costs associated with a cooperative agreement with LSU. The following table presents the breakdown of actual LEO costs through FY 2005.

LEO Reported Costs for FYs 1995-2005
($ in millions)

Non-personnel
     Cooperative Agreement with LSU $ 52  
     Other $ 33  
Total non-personnel $ 85
Total personnel $ 30
Total costs $ 115
Source: CJIS documents

LEO’s direct, non-personnel requisitions are tracked in the CJIS Budgetary and Evaluation and Reporting System (BEARS) database.30 However, the CJIS Financial Management Unit does not use BEARS to report LEO’s costs for CPIC purposes. Instead, officials use electronic spreadsheets primarily for budget monitoring. CJIS Financial Management Unit officials provided us with copies of these electronic spreadsheets and demonstrated how they sorted the data to calculate the direct, non-personnel amount included on our questionnaire. This amount, $85 million, comprises 74 percent of the total LEO costs reported through FY 2005.

Not all of the LEO-related expenditures reported were incurred by CJIS, since LEO was under different management until 1997. Since that time, other FBI divisions have participated with funding and managing LEO activities.

At the time CJIS Financial Management Unit officials provided us the LEO cost spreadsheets, they identified an error in their original response. As a result, the amount related to LSU was increased from $52 million to $74 million. However, this restatement did not change the overall reported costs of $115 million or the percent of direct, non-personnel costs described above.

To test the completeness of the reported LEO costs, we compared the values in the CJIS electronic spreadsheets to a number of sources. First, we tested the completeness of the LEO requisitions in the CJIS spreadsheets with the LEO requisitions in the BEARS database. Because BEARS was not in use until 2002, we did not expect the LEO requisitions in BEARS to contain all of the CJIS requisitions contained in the LEO spreadsheets. We did expect that all of the LEO requisitions coded in BEARS to be present in the CJIS electronic spreadsheets. Our review of the 86 LEO requisitions in BEARS showed, however, that 10 requisitions were not included in the CJIS spreadsheets. The invoiced amounts associated with these 10 requisitions were less than 1 percent of the total invoiced amount in BEARS of $41 million. The following table shows the requisitions related to LEO in BEARS.

LEO-Related Requisitions CJIS BEARS Database

Requisitions Number of Requisitions Invoiced Amounts Associated with Requisitions
Included in the CJIS spreadsheets 76 $ 40,635,330
Not included in the CJIS spreadsheets 10 $ 274,001
Total 86 $ 40,909,331
Source: CJIS Financial Management Unit

We asked the CJIS Financial Management Unit to explain why all 86 requisitions were not included in its spreadsheets, and were told that 9 of the 10 requisitions totaling $150,421 in invoiced amounts related to office furniture and office equipment supporting the LEO program, but were not part of the LEO IT infrastructure.

CJIS Financial Management Unit officials told us the last requisition related to LEO – $123,580 paid to an FBI contractor –should have been captured in spreadsheets and included in the costs reported for CPIC purposes. Officials also told us the expenditure related to this requisition was paid from CJIS funding not designated for the LEO project, which explains why it was incorrectly excluded.

We next obtained a listing from FMS of all payments made to LSU since October 1999.31 This listing contained 387 payments totaling $45,813,271. With the help of the CJIS Financial Management Unit staff, we attempted to reconcile the payments from FMS with the requisitions contained in the LEO spreadsheets. The reconciliation proved to be difficult because the CJIS spreadsheets track requisitions and FMS data is based on payments. Because one requisition may result in a number of payments, we did not independently research each of the hundreds of payments.

However, from a limited review of some of the FMS-listed payments, we identified five requisitions related to LEO totaling $850,000 that were not included in the CJIS cost spreadsheets. Although our review was limited, it clearly demonstrates that not all LEO-related costs have been captured by CJIS or reported in the OIG questionnaire.

We contacted LSU to collect any information it maintained on LEO to compare the costs provided by the FBI. The LEO Director at LSU provided this data and informed us of other activities related to LEO that the FBI has funded. LEO activities funded by the FBI include the InfraGuard, the Hostage Barricade System (HOBAS) and Bomb Data Center (BDC), the National Center for Missing and Exploited Children (NCMEC), the Katrina Fraud Task Force (KFTF), and the Law Enforcement Linguistic Access (LELA) system at LSU.32

We grouped the requisitions contained in the CJIS spreadsheets by their descriptions to compare them to the amounts provided by LSU as follows.

Comparison of LSU and CJIS Cost Data Related to
Contracted Activities FYs 1995 through 2005

Activity LSU Amount CJIS Amount Difference
Cooperative Agreement
$ 72,716,134
$ 62,349,855
$ 10,366,279
InfraGuard
$ 9,282,571
$ 7,850,850
$ 1,431,721
HOBAS and BDC
$ 1,710,086
$ 854,926
$ 855,160
NCMEC
$ 1,273,169
$ 1,195,916
$ 77,253
KFTF
$ 639,745
$ -
$ 639,745
LELA
$ 316,594
$ 163,572
$ 153,022
Total
$ 85,938,299
$72,415,119
$ 13,523,180
Source: OIG analysis

Similar to the difficulty we had comparing the amounts in FMS based on payments to the amounts in the CJIS spreadsheets based on requisitions, LSU’s costs are based on amounts invoiced to the FBI.

We asked the CJIS Financial Management Unit to explain the differences in the two sets of costs. Although officials pointed to the possibility of timing differences between the CJIS and LSU data, they acknowledged timing issues alone cannot explain these significant differences. Instead, CJIS Financial Management Unit officials told us the bulk of these differences appear to be attributable to transactions with LSU made by other FBI components, not by CJIS. Although CJIS manages LEO and all CJIS requisitions are included in the CJIS BEARS database, funding for LEO projects can come from outside CJIS. The spreadsheets maintained by CJIS included amounts for LEO related to non-CJIS FBI components, but the CJIS Financial Management Unit can only track non-CJIS requisitions and payments when informed of them. We did not attempt to reconcile the LSU and CJIS amounts because CJIS officials did not believe the differences could be reconciled.

To calculate the FTE costs for LEO, CJIS Financial Management Unit staff obtains data from the Bureau Personnel Management System.33 The Human Resources Division uses the Bureau Personnel Management System to capture and manage data on FBI personnel. The Bureau Personnel Management System tracks FBI employee information including salary information and the cost code to which they are assigned.34 CJIS officials told us that employees working with LEO are assigned one of three different codes. One of these codes includes employees assigned to LEO as well as another FBI IT system. After checking with administrative staff working for the LEO program, CJIS Financial Management Unit staff adjusted the list of employees to include only those working on LEO. The salaries of those employees determined to be working on LEO is then totaled.

For FY 2005, the FBI salaries related to LEO amounted to $2,323,795. To this amount the CJIS Financial Management Unit applied the fringe benefit rate of 32.8 percent prescribed by OMB, and the resulting total amount of LEO FTE costs for FY 2005 was reported as $3,086,000.

To test these FTE costs, we reviewed the Bureau Personnel Management System’s current listing of employees and their salaries for the three cost codes related to LEO. According to the Bureau Personnel Management System, there are 34 employees assigned to the three LEO-related cost codes with a combined salary of $2,502,314.

Although the current combined salary amount in the Bureau Personnel Management System for the three cost codes is 8 percent higher than the reported LEO amount in FY 2005, we do not believe this, by itself, indicates reported costs are understated. Instead, the relatively small difference may be attributable to the following causes:

In addition to our analysis using the Bureau Personnel Management System, we asked the LEO Director at LSU to provide the number of FBI employees working on LEO. She estimated that during FY 2005, approximately 30 FBI employees worked full time on LEO-related tasks.

Finally, as shown in the following table, we compared the costs reported for LEO in the CJIS Financial Management Unit spreadsheets with the outlays for LEO in OMB’s eCPIC database.

LEO PMO and eCPIC Cost Data
FYs 1995 - 2005

($ in millions)

Costs Amounts Per
PMO
Amounts Per
eCPIC
Difference
Non-personnela $ 85 $ 101 $ 15
FTE costs $ 30 $ 30 -
Totala $ 115 $ 130 $ 15
Source: CJIS and DOJ documents

a Due to rounding, not all values sum.

As shown in the table above, FTE costs were the same in both the CJIS spreadsheets and eCPIC database. However, the non-personnel amount in eCPIC was nearly $15 million more than the amount in the CJIS spreadsheets. We discussed this difference in reported non-personnel costs with the FBI CIO official responsible for entering the data in eCPIC.35 The official confirmed that the CIO’s records showing the same amounts reported by CJIS and the eCPIC non-personnel amount was incorrect. The official explained that the eCPIC value is incorrect because the FBI did not update prior year outlays for LEO and some other FBI IT systems during the last budget cycle. The FBI updated the prior year outlays for only those 16 systems it considers major systems. The official also explained that prior year costs for LEO and many other systems were not updated in eCPIC because of the CIO’s limited staff and resources.

In summary, the CJIS Financial Management Unit maintains records that provide reasonable assurance that the amounts reported in the OIG questionnaire can be verified. However, we determined these records do not capture all FBI costs related to LEO. From FMS payment data and LSU records, we determined the FBI investment in the LEO project could be as much as $13 million more than the $115 million reported.

DEA Concorde

Concorde is a DEA IT system designed to integrate DEA’s IT functions, improve business processes, and enable information sharing within the component. It is intended to allow Special Agents, Intelligence Analysts, and other investigative professionals to manage investigative case files digitally. The central feature of the Concorde system is the Investigative Management Program and Case Tracking System (IMPACT), a web-based case management system.

The DEA’s response to our request for Concorde cost information showed that the program began in 2000, with related costs through FY 2005 totaling $19.8 million. The DEA also reported government FTE costs amounting to $3.7 million, with five contractors individually paid at least $1.3 million over this same period. The following table presents a selection of the cost data DEA initially provided.

Concorde Reported Costs FYs 2000-2005
($ in millions)

Costs associated with 5 largest contractors $ 13.1
Personnel - DEA FTEs $ 3.7
Other contracted costs $ 3.0
Total Costs $ 19.8
Source: OIG analysis of completed Concorde questionnaire

We met with DEA officials to review their cost-tracking methods and supporting documentation, and analyzed the cost information provided. After we requested that DEA ensure the cost data extended back to the beginning of the Concorde project, DEA officials provided information noting that the program was conceived in 1997 with the name “UMBRELLA.” DEA officials told us the goals of UMBRELLA were the same as Concorde and only the name had changed.

DEA officials also provided us with copies of the PMO electronic spreadsheets they used to track Concorde’s contracts and related invoices since FY 2000. The following table presents Concorde’s contract-related expenditures between FYs 2000 through 2005.

Concorde Contract Invoices
FYs 2000 - 2005

Fiscal Year Amount
2000 $ 947,510
2001 $ 1,891,135
2002 $ 3,729,187
2003 $ 3,698,410
2004 $ 4,843,233
2005 $ 741,749
Totala $ 15,851,226
Source: Concorde invoice detail

a Due to rounding, values do not sum.

Although we previously determined that the DEA’s financial system, the FFS, is not organized in such a way to easily and reliably identify all the costs associated with any particular IT system, DEA finance staff told us a unique code to track funding for Concorde was created in FY 2005. Prior to 2005, Concorde did not have a defined allocation within the DEA’s budget.

DEA finance staff provided us with an FFS report that captured all activity associated with the Concorde fund code in FYs 2005 and 2006. To the extent possible, we tested the completeness of the Concorde PMO spreadsheets by using this FFS report. Although the report tracks the use of Concorde funding allotment within the DEA budget, it does not necessarily capture all costs related to Concorde. From the PMO spreadsheets, we identified costs related to Concorde that were funded from DEA sources other than the Concorde budget allotment. DEA officials told us these other sources funded Concorde as well a number of other projects.

The following table presents the amounts in FFS related to Concorde funding for operations and equipment during FYs 2005 and 2006.

Amounts Expended from Concorde Funding in DEA Budget
($ in millions)

Fiscal Year Operations Equipment Total
2005 $ 4.379 $ 0 $ 4.379
2006 $ 1.870 $ 0.068 $ 1.938
Total $ 6.249 $ 0.068 $ 6.317
Source: DEA

We analyzed the detail of the FFS amounts above and compared them with the PMO spreadsheets. From this analysis, we identified 19 task orders with invoiced amounts of $717,581 that were not included in the PMO spreadsheets. We determined that 13 task orders totaling $15,027 were for relatively small DEA travel and credit card expenditures. These amounts were not included in the reported costs because the PMO spreadsheets include only contracted activity. However, the remaining $702,555 in expenditures were related to Concorde contracts and should have been included in the PMO spreadsheets. DEA officials told us the incomplete project management cost data may have resulted from the accounting treatment for Concorde’s software, which requires such costs to be reported as an asset in DEA’s financial statement throughout the development phase rather than as an expense.

We considered testing the completeness of the $13.095 million associated with Concorde’s major contractors by obtaining payments to these contractors from FFS. However, DEA finance staff informed us that these particular contractors performed a large volume of services for the DEA, of which Concorde is only a relatively small portion. DEA staff said they could not readily identify those payments specific to Concorde.

We next evaluated the government FTE costs of $3.7 million reported by DEA in our questionnaire. This amount reflects all DEA personnel costs since FY 2000 and translates to approximately five FTEs per year. DEA officials provided us with a spreadsheet used to estimate Concorde FTE costs. This spreadsheet included estimates for FYs 2003 through 2020. Total estimates are calculated by identifying the number of FTEs in each of eight job categories and multiplying this number by the amounts for salaries and benefits related to each category. In addition to salary and benefits, the FTE spreadsheets include estimated amounts for other DEA employee expenses, such as training and travel. The following table presents the Concorde FTE cost data from FYs 2003 to 2005.

Concorde FTE Cost Data
FYs 2003 - 2005

Fiscal
Year
Number
of FTEs
Salary &
Benefits
Other FTE
Expenses
Total
2003 4.8 $ 93,218 $ 24,032 $ 562,800
2004 5.3 $ 93,218 $ 24,032 $ 619,080
2005 5.0 $ 97,040 $ 24,032 $ 610,202
Annual Average 5.0

$ 597,361
Source: DEA

The DEA did not have this type of information available for any years prior to FY 2003. However, we used the average annual FTE costs from the table above to estimate the FTE costs from FYs 2000 through 2005. The total FTE costs we estimated – $3,584,165 – is only 3 percent less than the FTE costs of $3,703,000 the DEA reported.

Although this analysis suggests that the FTE costs reported by the DEA for Concorde are supportable, these amounts are not based on actual results. DEA officials told us they have no procedures for tracking the time employees spend working on any particular project and the FTE costs are essentially estimates. In addition, the DEA does not have a time utilization tracking system that would identify the actual time spent on Concorde-related activities for all job categories.

Because the Concorde program began in 1997 and the earliest cost data we reviewed was for FY 2000, we asked DEA officials to provide any additional spreadsheets or cost data they could identify with this data. Officials provided us with past IT spending plans for FYs 1998 and 1999 showing Concorde total obligated costs of $770,000.36

Finally, we obtained from DOJ’s OCIO the reported amounts for Concorde contained in OMB’s eCPIC database. The following table compares these amounts with the amounts contained in the PMO spreadsheets already discussed.

Concorde PMO and eCPIC Cost Data
Excluding Government Personnel Costs
FYs 2000 – 2005

($ in thousands)

Fiscal
Year
Amounts per
PMO
Amounts per
eCPIC
Difference
2000 $ 948 - $ 948
2001 $ 1,891 - $ 1,891
2002 $ 3,729 $ 5,730 $ 2,001
2003 $ 3,698 $ 5,634 $ 1,936
2004 $ 4,843 $ 3,576 $ 1,267
2005 $ 742 $ 5,634 $ 4,892
Total $ 15,851 $ 20,574 $ 4,723
Source: DEA and DOJ officials

We asked DEA officials to explain the significant differences between these two amounts’ sources. Although the eCPIC data appears to represent prior years actual outlays, the DEA official responsible for entering IT system data into eCPIC told us this is not the case. He explained that budget estimate amounts from the start of the year are often used to report the outlays at the end of the year. The DEA does not routinely report actual amounts for prior year outlays because they view eCPIC as a planning tool, where the emphasis is placed on estimated future amounts, not costs previously incurred.

In summary, the Concorde costs we reviewed did not include all program costs since inception. We identified approximately $700,000 in software–related expenditures missing from project cost sheets, and another $770,000 of expenditures from the first 2 years of program that were not reported. In addition, Concorde’s actual reported FTE costs are estimated at the beginning of each year. In our view, the true costs of Concorde should include all costs incurred since 1997 when Concorde began.

JMD Justice Consolidated Office Network

JMD’s Justice Consolidated Office Network (JCON) is the common office automation platform that over 70,000 employees from 16 DOJ components use daily. JCON provides the IT tools and services that allow these employees to perform their computer-based work duties.37 Specifically, the JCON provides the basic IT computing framework for DOJ, which includes hardware such as networked workstations and printers, and applications such as e-mail and word processing. JCON also provides the infrastructure for components to access other IT systems such as case management databases and DOJ’s Financial Management Information System.

The JCON program is a partnership between the program management office (PMO) and the 16 DOJ components that rely on JCON. In this partnership, the PMO is responsible for funding and implementing all aspects of JCON planning and deployments, including acquiring hardware and software for the related components. The components themselves are responsible for and fund the operations and maintenance of these assets.

In response to our questionnaire, JCON PMO officials provided the cost information presented in the following table.

JCON Costs
($ in millions)

FY
Start
Costs incurred
through FY 2005
Estimated Funding
through FY 2009
Total Costs
FY 2001 - FY 2009
2002 $ 193.567 $ 270.079 $ 463.646
Source: OIG analysis of completed JCON questionnaire

The JCON response also included the following information:

We requested the amounts paid to BAE Systems, the single largest contractor that provides full life cycle support services for JCON, so we could verify these costs from a source independent of the PMO. Using the BAE’s tax identification number, we requested a listing of all payments made to the contractor from the JMD Finance staff. From a listing of payments made containing over 1,400 transactions totaling more than $149 million, we identified 504 payments related to JCON between FYs 2002 and 2005 totaling $49,537,777. The difference between the value we calculated and the value reported by JCON is $89,135, or less than 1 percent of the reported costs.

We met with JCON PMO staff to discuss the potential for using financial system and budget execution data to verify the remaining reported costs through FY 2005. We previously determined the various DOJ financial systems would not be able to identify the costs associated with all of the IT systems in the inventory. We discussed financial issues further with the JCON PMO and learned it would be possible in this case to create a report from the financial system that could identify planning and acquisition-related costs for JCON. The JCON PMO then prepared a JMD financial system (FMIS) report that matched the $193.567 million it reported on our questionnaire. JCON officials told us this amount included the government FTE costs incurred over the period.

From our discussions with the JCON PMO, we also learned that in FYs 2004 and 2005, JCON planning and acquisition costs were funded entirely from the Legal Activities Office Automation appropriation. Because JCON planning and acquisition in the CPIC process was essentially the same as the Legal Activities Office Automation appropriation in the budget process, we compared the costs from these two processes. First, we matched the FY 2005 Exhibit 53 amount for JCON of $39.967 million with the $40 million reported for the Legal Activities Office Automation appropriation. Both these amounts reflect the budget authority. We then compared the amount of outlays for planning and acquisition in the Exhibit 300 with the same amount reported for the Legal Activities Office Automation appropriation in FY 2005. The Exhibit 300 reported about $49 million in outlays for FY 2005, while the President’s Budget included outlays of $43 million for the same period – a difference of over $6 million.

We asked JCON officials about this discrepancy, and were told the $49 million included in the Exhibit 300 represented appropriations instead of outlays. We confirmed that the President’s Budget reported $49 million for obligations in FY 2005 and asked JCON officials why outlays were not provided as required by OMB Circular A-11. JCON officials explained that they began reporting obligations because it was difficult to retrieve data on outlays from the previous version of the FMIS. To be consistent and avoid reporting the same cost twice, the JCON PMO continued reporting obligations rather than outlays.

In addition to the comparison of FY 2005 CPIC and budget data, we compared the $246 million total amount reported in the JCON Exhibit 300 for planning and acquisition to the $193.6 million total amount reported in the OIG questionnaire. JCON officials told us total costs in the Exhibit 300 costs were $53 million more than total costs reported to the OIG because the Exhibit 300 included expenditures from FY 2000 and 2001 and were not considered part of JCON IIA.

We researched government and industry sources for more information on JCON and JCON contracts. This search identified a $500 million JCON contract awarded in 1996, more than 5 years before the first costs reported in our questionnaire. Although the JCON PMO response made clear that the reported costs related to JCON IIA, we consider the JCON initiative to have begun in 1996 when the decision was made to replace a collection of separate office automation systems with a consolidated DOJ platform.

The JCON PMO confirmed that the first attempt at replacing the existing office automation systems, known as JCON I, began in 1996 and was terminated in 1998 when it did not work. JCON II followed and evolved into JCON IIA by FY 2002. Because the earlier JCON efforts predate the current JCON Project Manager and other staff, they were not able to provide us with the complete costs of JCON prior to FY 2002.

Although JCON has not yet developed cost estimates for FYs 2010 through 2012, the JIST Budget Officer told us that funding in these years is expected to equal at least the FY 2009 level of $89.5 million.

In summary, by using financial system and budget data we were able to verify the costs the JCON Project Management Office reported to us. However, the Project Management Office said the costs only represent the current standard architecture. Although JMD views the various versions of JCON as separate systems, we believe the true cost of JCON should include all costs incurred since 1996 when the JCON project was initiated. Therefore, we conclude that JCON’s costs since 1996 should be at least $53 million more than the $193.6 million we verified. In addition, because complete cost data was not available for JCON prior to FY 2002, we were unable to determine what amounts, if any, were paid in connection with the 1996 $500 million contract.

CIOs’ Role in IT Spending and Budgeting

Although the DOJ CIO and component CIOs have significant responsibilities for their organization’s IT systems, we found they have varying degrees of control over IT system budgets.

CIOs Control of IT Spending

In testing the completeness of reported costs for selected IT systems, we found that the control of IT spending and budgets by component CIOs varies. The FBI CIO’s control over IT spending has increased over the last few years. In 2004, the FBI reported in its first IT Strategic Plan that its CPIC process and IT spending were not centrally managed. However, the budget for the CIO now includes nearly 50 percent of the FBI’s IT spending, and the CIO also has approval authority for IT spending not within the CIO’s budget. This change since 2004 indicates more centralized management of the FBI’s IT budget within the CIO’s office.

At the DEA, the CIO controls approximately 60 percent of the agency’s budgets relating to IT spending, and other DEA managers control the remainder. To help oversee DEA IT projects, the DEA CIO has created an Integrated Project Review process that allows the CIO to:

While the Integrated Project Review is used for IT systems utilizing CIO funding, the Major Investment Review is used for any major DEA IT system using DEA funds or personnel. The objectives of the Major Investment Review are similar to those for the Investment Project Review.

At JMD, the creation of the JIST appropriation has given the CIO budget control over five of the six JMD IT systems in our inventory. Based on our discussion with the JIST Budget Officer, the CIO should have reliable cost data in future years.

Costs of DOJ IT Systems Contained in OMB eCPIC Database

Although the CIOs at the FBI, DEA, and JMD control the budgets of many IT systems, employ tools that monitor current spending, and review future year spending requests, we found the CIOs are generally unable to identify or verify all prior costs related to individual IT systems.

As part of our testing during this audit, we examined the prior years’ cost data contained in the OMB eCPIC database. OMB uses eCPIC to collect cost and other data on individual IT systems. Of particular interest for this audit were the amounts contained in eCPIC for prior year outlays of the three IT systems we tested. In all three cases, we found the eCPIC prior year amounts were different from the amounts we tested in the OIG questionnaire, which indicates the eCPIC costs are inaccurate.

At the three components, we discussed with IT system project managers and CIO staff the differences between the eCPIC and questionnaire data. For each component, we determined a different cause behind these cost differences. In our view, it is critical for the adequate oversight of DOJ IT systems and projects that complete and accurate cost data be available on an individual system basis.

At the FBI, we determined that the CIO had stopped updating the prior year outlays for some IT systems, including LEO. For the DEA, we determined the CIO staff had not entered the actual outlay amount but instead used budgeted amounts. For JCON, we determined the difference in prior year questionnaire data related to the PMO’s use of obligations rather than outlay data. In addition, we found that total costs extended 2 years further back than the data we initially were provided.

We also learned that each of the CIO staffs do not have a formal verification process of the prior year amounts reported by the PMOs for inclusion to eCPIC. OMB’s eCPIC is also used by DOJ’s OCIO to prepare the Exhibit 53.

Consistent with the President’s Budget, the Exhibit 53 shows the appropriated amounts associated with individual DOJ IT systems.38 The Exhibit 53, in effect, extracts and provides detail on amounts for IT spending already included in an agency’s overall budget. According to OMB Circular A-11, an agency’s Exhibit 53 must be fully integrated with that agency’s overall budget submission. In addition, agencies must update each Exhibit 53 and the accompanying Exhibits 300 to reflect any changes due to final budget decisions.

We discussed with officials at the JMD OCIO the extent to which the Exhibit 53 amounts could be traced to the component’s overall budgets and financial systems. These officials told us the two documents are not completely integrated, but the OCIO is working with DOJ budget staff and the components on this issue.

Other Congressional Requests for IT System Costs

During our audit, the Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security of the Senate Committee on Homeland Security and Governmental Affairs held hearings on federal IT projects at risk. Following this hearing, the Subcommittee asked all major federal agencies to list all of their on-going IT projects and whether they had experienced cost overruns. In response to this request, DOJ provided data on its IT systems, including the three IT systems we tested – LEO, Concorde, and JCON. The following table compares the responses to our questionnaire and the responses to the Subcommittee’s request.

DOJ Responses to IT System Cost Inquiries
Made by Senate Subcommittee and the OIG
($ in millions)

IT System Senate Subcommittee’s Request
as of 10/2006
OIG Questionnaire
as of 9/2005
LEO N/A $ 114.5
Concorde $ 25.1 $ 19.8
JCON $ 65.3 $ 193.6
Source: Senate Subcommittee’s request and OIG questionnaire

Although we did not expect the amounts reported to the Senate Subcommittee and the OIG to exactly match due to the difference in reporting dates, the amounts for LEO and JCON did not appear to be consistent.

The FBI included a comment in the response to the Senate Subcommittee indicating that LEO is not an ongoing project, and FY 2006 development, modernization, and enhancement DME funds were used for tactical enhancements only. We discussed this issue further with LEO and other FBI officials. They told us their initial response was made in error, and said they subsequently amended the response to be consistent with the information provided to us.

Although the amounts reported by the DEA for Concorde appeared consistent, the DEA indicated to the Senate Subcommittee that Concorde began in 1998, while we were initially provided cost data only as far back as 2000. Subsequently, the DEA confirmed a 1997 start date and provided us with the cost data for the years omitted from their initial response to us.

We discussed the difference in the JCON amounts with JMD officials, who told us the amounts reported to the Senate Subcommittee were taken from Earned Value Management data on the current baseline and included JMD as well as the component’s share of JCON costs.

Conclusion

We concluded that the actual costs of DOJ IT systems that are provided to Congress, OMB, and senior management within DOJ, including the CIO, are unreliable. IT system cost reporting within DOJ is fragmented, uses inconsistent methodologies, and lacks control procedures necessary to ensure that cost data for IT systems is accurate and complete. Furthermore, DOJ does not have complete cost data for the three IT systems we tested and, based on our testing and other audit work, we lack confidence in the cost data reported for DOJ IT systems in general. We determined that the $327.9 million combined costs reported for these three systems was understated by at least $68 million. In our opinion, the lack of complete cost data that is verifiable for DOJ’s IT systems compromises the effectiveness of DOJ’s IT oversight entities, including Congress, OMB, the DIRB, and DOJ and component CIOs.

Although the primary purpose of DOJ’s CPIC processes is IT planning and less emphasis is placed on prior years’ costs, we believe it is important that the cost data used by decision-makers is reliable. Our audit casts doubt on the reliability of the data that DOJ components report to the CIO, the DIRB, OMB, Congress, and other oversight entities.

Recommendations

We recommend that the Assistant Attorney General for Administration:

  1. Ensure that component CIOs develop and implement cost effective means to report accurate, complete, and verifiable costs for individual IT systems.

  2. Ensure that DOJ’s CIO improves the integration of the Exhibit 53 and budget submissions in accordance with OMB Circular A-11 so that Exhibit 53 amounts can be traced to the components’ overall budgets and financial systems.

  3. Assess the feasibility of using the DOJ’s planned Unified Financial Management System for Capital Planning and Investment Control cost reporting.



Footnotes
  1. Although the Organized Crime Drug Enforcement (OCDETF) Fusion Center is located within the Office of the Deputy Attorney General, for purposes of this audit we included the Fusion Center IT system as part of the DEA. The DEA’s unobligated funds developed the Fusion Center.

  2. DOJ IT spending authority for FY 2006 is an enacted amount.

  3. The DOJ’s FY 2006 enacted budget is approximately $22 billion.

  4. The DOJ CIO is organizationally located in JMD and serves as CIO for both the DOJ and the JMD component.

  5. See Appendix III for information on prior reports examining the DOJ’s ITIM process.

  6. Object classification is spending based on an item or service purchased by the federal government. In this case, object classes break down total IT system costs into smaller costs such as personnel compensation, equipment, and advisory or assistance services.

  7. The MAX Budget Information System is used to support the federal budget process. OMB uses the MAX Budget Information System to collect, validate, analyze, model, and publish budget information.

  8. OMB Circular A-11 states that the object classes present obligations according to their initial purpose, not the end product or service. For example, for a federal employee who constructs a building, the obligations for the employee’s wages should be classified under personnel compensation and benefits rather than acquisition of assets.

  9. The Office of Federal Financial Management’s Core Financial System Requirements issued in January 2006 requires the following seven functions: financial system management, general ledger management, funds management, payment management, receivable management, cost management, and reporting.

  10. Established in 1992, CJIS serves as the focal point and central repository of criminal justice information services at the FBI.

  11. BEARS is a stand-alone system used to track requisitions more easily and in greater detail than is possible using the FBI’s core financial system, FMS.

  12. FMS data prior to FY 2000 has been archived and retrieving this data took more time than our analysis permitted.

  13. See Appendix V for more information on these activities.

  14. The CJIS Financial Management Unit staff uses Bureau Personnel Management System data and follows the method described above for LEO and other projects they consider small. Larger CJIS projects, such as the Integrated Automated Fingerprint Identification System, use an Activity Based Cost model. After the number of FTEs is identified, personnel cost rates provided by the FBI’s Finance Division are used to calculate total personnel costs.

  15. The cost codes used by the Bureau Personnel Management System are the same used by FMS.

  16. Although eCPIC is accessible from most federal agencies’ intranet systems, this is not the case at the FBI. Instead, the FBI’s CIO Investment Management Unit collects and enters the data for the individual IT systems into eCPIC.

  17. The FY 1998 IT spending plan includes FY 1997 obligations for Concorde. The total obligated costs of $770,000 reflects FYs 1997 through 1999.

  18. The amounts discussed below represent JCON Planning and Acquisition only. JCON maintenance is funded by the 16 participating DOJ components.

  19. DOJ’s Exhibit 53 summarizes office automation and infrastructure IT systems in the Consolidated Enterprise Infrastructure.



« Previous Table of Contents Next »