Combined DNA Index System Operational and Laboratory Vulnerabilities

Audit Report 06-32
May 2006
Office of the Inspector General

Appendix I
Objectives, Scope and Methodology


We conducted our audit in accordance with the Government Auditing Standards and included such tests as were considered necessary to accomplish the audit objectives. Our audit generally covered the period from October 2003 through November 2005, although in some instances it was necessary to consider documentation from outside that timeframe. The objectives of this audit were to:

  1. assess the adequacy of the FBI’s administration of CODIS, including its oversight of the national DNA database;

  2. analyze findings from DNA laboratory audits, both OIG‑conducted audits and external quality assurance audits, to determine if they reveal trends and vulnerabilities; and

  3. evaluate the FBI’s implementation of corrective actions in response to findings from the OIG’s September 2001 audit, The Combined DNA Index System.50

To accomplish the objectives of this audit we:

  • Developed and conducted a survey of 174 NDIS participating laboratories to obtain feedback from CODIS administrators on the FBI’s administration of CODIS.

  • Interviewed CODIS Unit management regarding staffing, position responsibilities, and the planned timeline for filling vacant CODIS Unit positions.

  • Interviewed FBI management regarding the mission, goals, objectives, and performance measurements for the CODIS Unit, and obtained copies of all supporting documentation for those strategic planning items.

  • Reviewed contract and operations documents to verify the operation, maintenance, and security of the CODIS System.

  • Reviewed FBI documentation and interviewed FBI management to verify that the proper changes have been made to the database as required by the Justice for All Act of 2004.51

  • Reviewed FBI documentation and interviewed FBI management regarding the current status and plans of each of the corrective action measures implemented as a result of the prior OIG audit of CODIS.

  • Reviewed 18 OIG CODIS laboratory audits and identified trends in the findings.

  • Reviewed a random sample of 41 external laboratory evaluation reports and supporting documentation for corrective action taken, if any, to determine if any trends or vulnerabilities could be detected from a collective review of quality assurance laboratory findings.52

  • Analyzed the tracking system maintained by the CODIS Unit for the processing of audits through the NDIS Audit Review Panel (Review Panel), to determine the efficiency of the process and the timeliness of Review Panel member submissions on their assessment of each audit.

The following sections provide additional detail for work that specific actions listed in the preceding list.

OIG CODIS Administrator Survey

Using information obtained during meetings with the FBI and CODIS administrators, including issues that were raised during open discussion at the SDIS administrator's meeting in May 2005, we developed a survey for completion by CODIS administrators at NDIS‑participating laboratories. The survey provided us with feedback on the FBI’s administration of CODIS and laboratory concerns about quality issues or problems in the CODIS community. We included open-ended and static-option questions. For those questions where we provided static options, we included space for miscellaneous comments. In addition, we assured the CODIS administrators that responses would be confidential and individual responses would not be singled out in a way that could identify the source of the information.

After the initial draft of the survey was created, we tested the survey on members of the CODIS community. We used information received from the FBI to select experienced individuals to test the survey, being careful not to select CODIS administrators to preserve the universe for our final survey. We contacted six members of the DNA community and all six responded. Additional revisions were made to the survey based upon the test respondents’ comments, and were reflected in the final version.

The final version of the survey was e-mailed to 174 CODIS administrators on June 7, 2005. A list of the CODIS participating laboratories to which we sent the survey appears in Appendix II. CODIS administrators were initially given until June 24, 2005, to complete the survey and return it via e-mail, fax, or U.S. mail. On June 21, 2005, we sent out a reminder e-mail and, on June 28, 2005, we sent out a third e-mail extending the deadline to July 7, 2005. We extended the deadline to give non-responding states a chance to reply.

As of July 7, 2005 (the extended deadline), we had received 139 surveys. However, there were 6 states from which we still had not received a response. In a final attempt to give these six states a chance to submit a survey we contacted them via e-mail on July 29, 2005.

After all the extensions (August 15, 2005 was the final cut-off date) we still had not heard from Idaho and Rhode Island. We noted that a member of the Connecticut laboratory had provided a response, but did so during the test phase of the survey, and since the survey changed after that response was received, we could not include it in our results.

The survey contained 46 questions which were broken into seven sections: (1) demographics, (2) FBI CODIS Unit responsiveness, (3) allowability of DNA profiles, (4) laboratory quality, (5) general CODIS operations, (6) NDIS Audit Review Panel, and (7) FBI guidance to the CODIS community.

Additional information about how we tallied the survey responses, and a summary of the actual responses received can be found in Appendix VII.

OIG CODIS Laboratory Audits

During our audit we analyzed a total of 18 OIG CODIS laboratory audits, of which 6 were issued in final for FY 2004 and 12 were issued in final for FY 2005.53 A list of the 18 OIG CODIS laboratory audits is contained in Appendix V.

We identified and analyzed trends from each audit specific to profile allowability as well as the number of findings for the five different QAS sections reviewed during the audits, as follows.

  • NDIS participation requirements,

  • quality assurance standards (QAS),

  • forensic profiles,

  • convicted offender profiles, and

  • other reportable matters.

External QAS Evaluations

We tested the FBI’s records for the Review Panel and the external QAS evaluations submitted to it. We judgmentally selected 10 participating states and compared the FBI’s electronic records against the written records used to create the electronic records. We found no material differences in the tracking system. As a result we did not expand our sample and relied on the information contained in the electronic records.

In addition to reviewing the tracking system for data reliability, we determined if delays in the process were significant, if the timeliness was improving, and whether the Review Panel members were meeting the 30‑day deadline set forth in the NDIS procedures for reviewing audits. In order to see if the timeliness had improved we analyzed the information contained in the FBI’s records for both 2003 and 2004.54

Working with the information provided to us from the FBI, we determined that there were 72 closed evaluations in 2004 and 11 closed evaluations in 2005. We tested 50 percent, or 41, of these evaluations.

We used random sample selection over the 2 years of interest since our goal was to conduct a trend analysis and to look at findings at more laboratories. Further, we stratified our sample to ensure we selected a percentage of SDIS and LDIS laboratories representative of the whole universe. We excluded laboratories the OIG had already audited, to avoid requesting information similar to what had been requested previously.

We notified the CODIS Unit Chief that we would be contacting the laboratories to request copies of external QAS evaluations and related correspondence. We provided him with copies of any written correspondence we issued in order to acquire the documentation we needed to complete our review.

Using contact information obtained from the FBI we contacted CODIS administrators for each of the laboratories and explained the following:

  • We were conducting an audit of the FBI's CODIS Unit, and in connection with that, we needed to obtain documentation to confirm the FBI's records for the Review Panel.

  • We had selected a sample of the evaluations conducted and cleared from 2004 through July 2005, and their laboratory’s evaluation was one of those selected.

  • Since the FBI returns all documentation from the submitting laboratories, we needed to obtain copies of documentation directly from them.

  • For the evaluation selected (specific dates were provided), we needed a copy of the completed evaluation document and any correspondence that had been sent to or received from the FBI related to that evaluation (not including complete corrective action documentation, such as revised policies or procedures).

The 41 laboratories in our sample represent 19 states and 1 federal agency. We analyzed the evaluations in our sample for trends and statistics. Specifically, we calculated:

  • the number of findings (based on QAS section numbers);

  • the average number of findings per laboratory, with and without adjustments for overturned findings;

  • the number and percentage of overturned findings; and

  • the number of laboratories with common findings, without common findings, and with no findings, divided into categories of SDIS and LDIS laboratories.

In our analysis, we relied upon the findings and conclusions of the QAS evaluators within the DNA community, and did not perform any assessment as to the scope of their work. In addition, we did not confirm whether those evaluators met the requirements for conducting external QAS evaluations, specifically the requirement that they successfully complete the FBI’s QAS auditor training.

We compared the documentation received from the laboratories to the information the FBI had provided in its Review Panel record spreadsheets, to verify accuracy of those records. We also tracked whether the Review Panel had to follow up with the laboratories and whether findings were challenged by the laboratories, to determine if those issues impeded the timeliness of the Review Panel process.


Footnotes

  1. Department of Justice, Office of the Inspector General. Audit Report No. 01-26, The Combined DNA Index System, September 2001.

  2. Pub. L. No. 108-405 (2004).

  3. The QAS require that laboratories undergo annual audits and, that at least every other year, the audit must be performed by an external agency that performs DNA identification analysis and is independent of the laboratory being reviewed. These annual audits are not required by the QAS to be performed in accordance with the Government Auditing Standards (GAS) and are not performed by the Office of the Inspector General. Therefore, we will refer to the annual audits as evaluations (either an internal laboratory evaluation or an external laboratory evaluation, as applicable) to avoid confusion with our audit, which was conducted in accordance with GAS.

  4. In our analysis, we included two audit reports for audits completed in FY 2005 that were not issued until early FY 2006.

  5. All of our analysis was done based on calendar days.


« Previous Table of Contents Next »