Combined DNA Index System Operational and Laboratory Vulnerabilities

Audit Report 06-32
May 2006
Office of the Inspector General

The following guidance was provided to the survey respondents at the beginning of the survey: As a rule, please select only one answer to each of the survey questions. Guidance on how to interpret the question is presented in italics. Note that throughout the survey, “NDIS requirements” is used to refer to all of the requirements with which an NDIS-participating laboratory has to comply to use and maintain their CODIS system, including the NDIS operating procedures and NDIS data acceptance standards. However, we do distinguish NDIS requirements from the QAS, even though compliance with the QAS is required for NDIS participation. Survey respondents were also instructed to provide their responses directly to the OIG, with no copy to any other organization, such as the FBI, the National Institute of Justice, or accrediting organizations. Below, we describe our strategy for tallying the survey responses, as well as give a list of the survey results, by question. Tallying of Survey Responses A few different systems were devised that would allow us to summarize the survey results as well as calculate averages and percentages for various questions. The system was based on the type of question and the calculation that would best portray the results of that question. We tallied the number of “yes” and “no” responses for questions 5, 23, 27, 33, 36, 37, 38, 39, 40, and 41. In addition, we assigned a number value to questions, 1, 3, 4, 11, 16, 32, 34, 42, 44b, and 46, but the numbers had no positive or negative significance. Questions 7, 8a, 9, 10, 13, 14, 17, 18, 21, 22, 26, 28, 29, 35, 43, and 44a were assigned a numerical value where the numbers had a positive or negative implication, moving from negative to positive as the numbers became larger. For questions 2, 8b, 12, 19, 20, 30, and 45 we created our own alpha key (i.e., we assigned alphabetic designators, similar to acronyms, that allowed us to tally the responses in the limited space of our spreadsheet). Tallying results for question 15 was more complex than the other questions because there was not typically one correct response for the scenarios given in the question. T o each scenario, respondents could select "Yes," "Yes, under the following conditions," or "No, for the following reason(s)." As a result we developed a complex grading matrix that could help us evaluate the different factors that a respondent could cite to justify their response. We distinguished between primary, secondary, and peripheral factors that would need to be considered in evaluating each scenario. Some scenarios were simpler and did not have that many factors, but the more complex scenarios had many factors. From that we developed a grading scale where responses were graded based upon the number of factors that they provided. The scale was 1 to 5, with 1 = poor, 2 = marginal, 3 = adequate, 4 = good, and 5 = exceptional. We averaged the grades, but also tracked the number of "1" responses, or “poor” responses received. For questions 24 and 25 we summarized the comments into phrases. Then we categorized the phrases and grouped them into similar categories. For questions 6 and 31 the actual percentages and dates given by the respondents were used. In addition, 26 of the questions allowed respondents to provide comments. Some comments were modified slightly by auditors to correct for grammar and sentence structure. The 26 questions were 8, 13, 11, 17 (in two separate places), 20 (in two separate places), 23, 27, 28, 29, 30, 32, 34, 35, 36, 37, 38, 39, 40, 41, 42, 44.b., 45 (in two separate places), and 46. The comments provided were analyzed and trends were identified for each question, as well as across all of the comments. Survey Results Described below are the results for each question of our survey, along with an explanation of the various options offered to the administrators with each question. Throughout the survey questions, italicized text was used to give instructions to the administrators on how to interpret our questions and proceed through the survey. Demographics What CODIS level is your laboratory? Ninety-five respondents were from LDIS laboratories and 49 were from SDIS laboratories. What is your current role in the laboratory? How long have you been the CODIS Administrator at your laboratory (include past experience if you are not currently the CODIS Administrator)? The average time the respondents were CODIS administrators was 3 to 5 years. What is the size of the DNA section in your laboratory (include technicians, examiners, managers, etc., but not clerical support or management that are not specific to the DNA section)? The average size of respondents’ DNA laboratory was 6 to 10 positions. This includes all staff specific to the DNA portion of their laboratory. Are you an FBI-trained quality assurance auditor? FBI CODIS Unit Responsiveness For your communications with the CODIS Unit during the last 2 years, please estimate what percentage of those communications were for the following purposes: _% confirmation on whether a profile is allowable for NDIS _% support to defend my decisions to investigators or attorneys _% annual certifications, filings, or paperwork _% assistance in managing CODIS user information _% QAS audit and corrective action submission or responses _% guidance on hit counting or match resolution _% clarification on NDIS Operating Procedures _% submission of other routine required paperwork _% Other [please specify]: _________________ Percentages were highest for the purposes listed below. QAS audits and corrective action, information technology matters (mentioned under “Other”), and annual certifications and paperwork. How would you rate the timeliness of responses you have received, in the last 2 years, from the CODIS Unit to questions or concerns you have had on the following topics: Purposes Often, no response is received Often, response delayed more than 2 weeks Generally, received within 2 weeks Generally, received in less than 1 week N/A Appropriateness of a profile for NDlS     X   69% Support to defend your decisions to investigators or attorneys     X   81% Compliance with / clarification of NDlS requirements     X   51% Compliance with / clarification of QAS     X   53% Assistance in managing CODlS user information       X 42% Completion of misc. required paperwork       X 44% While 139 responses were received to this question, a significant percentage marked “N/A” if the reason we offered did not apply to their usual contact with the FBI. The higher the percentage of “N/A” responses, the fewer the people who contacted the FBI on that topic. The percentage of “N/A” responses can be seen in the “N/A” column for each category. The chart also shows the average responses for each topic given. The rounded average responses ended up being in the latter two columns. Please complete the following questions if you selected “Often, no response received” or “Often, response delayed more than 2 weeks” to any of the options in the above question. How often did you feel that the untimely responses from the CODIS Unit has limited your ability: To what cause(s) do you attribute the untimely responses from the CODIS Unit? [Check all that apply] For those matters for which the CODIS Unit has been timely in responding (responses received in less than 2 weeks), to what degree did that response address your questions or concerns? Purposes Not at all Minimally: significant aspects were unaddressed Partially: minor aspects were unaddressed Fully N/A Appropriateness of a profile for NDlS     X   65% Support to defend your decisions to investigators or attorneys     X   70% Compliance with / clarification of NDlS requirements       X 47% Compliance with / clarification of QAS       X 51% Assistance in managing CODlS user information       X 35% Completion of misc. required paperwork       X 39% While 125 responses were received for this question, a significant percentage marked “N/A” if the purpose did not apply to their usual contact with the FBI. The higher the percentage of “N/A” the fewer the people who contacted the FBI on that topic. The percentage of “N/A” responses can be seen in the “N/A” column for each category and the rounded average responses ended up being in the latter two columns. If you have raised concerns about CODIS or NDIS operations to the CODIS Unit and those concerns remain unaddressed, to what degree do you believe those concerns have the potential to undermine the long-term success of CODIS (i.e., the ability of CODIS to accomplish its mission)? No potential, since the question or concern was a one-time issue limited to my laboratory Minimal potential, since the question or concern is probably an isolated issue for laboratories similar in size or level to mine Moderate potential, since the question or concern is probably a recurring issue for laboratories similar in size or level to mine Significant potential, since the question or concern is a recurring issue for many laboratories in the CODIS community Fifty responses were received for question 10 and the average response was between moderate and minimal potential. What suggestions do you have for how the CODIS Unit can improve its responsiveness to the CODIS community’s questions or concerns? We received 39 comments. Trends in the comments were that the CODIS Unit needs more staff, and the CODIS Unit should disseminate more info to the CODIS community via the CODIS Website or Criminal Justice Information System Wide Area Network (CJIS WAN). In addition, disseminating more information to the CODIS community via the CJIS WAN was also a comment trend identified when all 636 comments were analyzed. Specifically, our analysis showed 37 respondents made a total of 51 comments regarding posting information through the CJIS WAN. A related comment trend is that while the FBI’s accessibility and responsiveness has improved, more improvements are needed; our analysis showed 20 respondents made a total of 28 comments regarding the FBI’s inaccessibility and its untimely responses. Suggestions were made for the FBI to set standards on timeliness of responses and to have a mechanism for making sure all responses are addressed, similar to the type of standards or tracking that is done for the CODIS contractor's help desk with information technology questions. In situations where a response cannot be formulated in a timely fashion, suggestions were made for at least a response indicating something along the lines of "X person will respond by Y time with the information you requested." Allowability of DNA Profiles In your laboratory, who currently is ultimately responsible for ensuring that casework profiles are uploaded in accordance with NDIS requirements? [Having the final responsibility does not preclude the possibility that the person responsible may consult with another member of the laboratory to confirm their conclusion.] If your answer to question 12 indicates you are partially or fully responsible for designating which profiles are uploaded to NDIS, please complete questions 13-15. How difficult is it to determine what categories of profiles can be uploaded to NDIS? [In the options offered below “another source” is any source other than the documents supplied by the FBI to inform you on these topics, such as the CODIS Administrator’s Handbook, or the NDIS Operational Procedures. Examples of other sources you might consult are the NDIS Custodian or another CODIS Administrator.] Very difficult (routinely requires clarification from another source) Difficult (occasionally requires clarification from another source) Routine (rarely requires clarification from another source) Easy (does not require clarification from another source) We received 130 responses to question 13 and the average response was “Routine.” How confident are you personally that when you conclude that a profile is permitted in NDIS, you are correct? Completely confident. Consistently confident. On rare occasion I appreciate having additional confirmation from another source on my decision. Generally confident, but would occasionally appreciate having confirmation from another source. Somewhat confident, and often solicit confirmation from another source on my decision. Minimally confident, and routinely solicit confirmation from another source on my decision. We received 130 responses to question 14 and the average response was closest to the “consistently confident” option. Would you categorize the following samples as a “forensic unknown” suitable for NDIS? [Please base your analysis only on the information provided in the question, and assume the profiles have >=10 loci.] Each scenario offered the following options: Yes Yes, under the following conditions: No, for the following reason(s): We used a grading scale that evaluated the quality of response we received from respondents, with 1 = poor and 5 = exceptional. A profile developed from crime scene evidence that once analyzed is revealed to match the suspected perpetrator? We received 131 responses and the average grade was 3.4. A profile developed from crime scene evidence that does not match any reference sample provided (suspect, victim, or elimination)? We received 132 responses and the average grade was 3.6. A profile developed from an item submitted by a law enforcement agency for analysis (item source or crime committed is unclear) that does not match any reference sample provided? We received 132 responses and the average grade was 4.3. A profile developed from crime scene evidence that is confirmed to be a mixture of the victim and suspected perpetrator (reference samples are available)? We received 132 responses and the average answer was 4.1. A profile developed from crime scene evidence that is a mixture of two contributors that could include the victim, but reference samples are not available? We received 133 responses and the average answer was 3.5. A profile developed from crime scene evidence that is a mixture of three contributors, including the victim, and only the victim’s reference sample is available? We received 133 responses and the average grade was 3.7. What (as in a law or policy document) or who (as in a position) do you believe to be the final authority on what profiles your laboratory uploads? For question 16 most respondents gave multiple answers, even though we asked for a single “final authority.” Consequently, the percentages overlap and do not total 100 percent, which is why we did not put percent labels on these charts, to preclude misinterpretation. However, the dominant responses are clear. “N/A” responses are not reflected but were few. In the preceding chart it is clear that CODIS administrators see the LDIS administrator as the primary authority over what goes into LDIS. In the preceding chart it is clear that CODIS administrators see the SDIS administrator as the primary authority over what goes into SDIS, although roughly one-third of the responses included an emphasis on the state law or policy. In the preceding chart we see NDIS is the only level where respondents weighed the national law or policy almost as heavily as they did the national representative. From your experience in the CODIS community, do you believe that NDIS-participating laboratories have the same understanding of what profiles are suitable for inclusion in NDIS? Yes, all laboratories have the same understanding Yes, laboratories have the same understanding with only rare exceptions No, not all laboratories have the same understanding, but community understanding is improving If possible, please explain: No, and community confusion is increasing If possible, please explain: Unsure or not applicable based upon limited experience The average response to question 17 was “No, not all laboratories have the same understanding, but community understanding is improving.” Administrators who said that the CODIS community does not have the same understanding had the option of providing additional comments, and 70 respondents did. Most of the respondents focused their answers on their participation in discussions at the National CODIS Conferences (NCC). Some took the perspective that the discussions further confused people, while others felt that the discussions helped by clarifying troublesome scenarios. Do you believe there are NDIS-participating laboratories (including your own) that knowingly upload profiles that they believe to be “borderline” (i.e., probably unallowable) if they believe it will further an investigation? No Yes, but they are the rare exception Yes, and it could be occurring beyond a rare exception Unsure or not applicable based upon limited experience Twenty-three percent of respondents to question 18 said “Unsure or not applicable” and the majority of the remaining responses were “Yes, but they are the rare exception.” If a member of your DNA laboratory has a question regarding whether a profile is allowable for upload to NDIS, who or what would be their most likely source for clarification? CODIS Administrator’s Handbook or NDIS Operating Procedures CODIS Administrator in their laboratory CODIS Administrator in another laboratory NDIS Custodian An examiner in their laboratory Other: Respondents gave multiple answers to question 19, and therefore we were not able to calculate true percentages. Instead, we focused on which sources of guidance were the top three named. We received 143 responses, with three options selected the most by respondents as all or part of their answer: 111 respondents cited “CODIS Administrator in their laboratory”; 27 respondents cited “CODIS Administrator Handbook”; and 27 respondents cited “CODIS Administrator in another laboratory.” What do you believe the CODIS Unit can do to improve community understanding of what profiles are permitted at NDIS? [Check all that apply] Conduct specific training Increase discussion at the annual CODIS Conference Disseminate better information and guidance Please describe: __________ Other: __________ Since this question permitted multiple responses, the chart below is only intended to convey the magnitude of response. The three primary options we offered were selected pretty evenly, as ways that would help community understanding. Laboratory Quality Please rank your laboratory’s quality of operations as one of the following: 1 = Poor, since there are still fundamental quality controls we fail to consistently apply OR we have one or more staff members that are not fully committed to QAS compliance. 2 = Fair, since we routinely apply most quality controls in our operations, but still need occasional improvement. All staff are committed to QAS compliance, but occasionally are not properly informed about the standards. 3 = Good, since we consistently apply all appropriate quality controls in our operations. All staff are fully committed to QAS compliance and are proficient in what those standards are. 4 = Excellent, since we apply all appropriate quality controls, and actively pursue enhancing those controls. All staff are committed to QAS compliance, are proficient in what those standards are, and are committed to surpassing those standards whenever warranted to ensure excellence. We received 143 responses to question 21 and the average answer was 3.6. How would you rank your laboratory’s quality of operations in relation to other laboratories? 1 = Below average (the majority of laboratories surpass our laboratory) 2 = Average (our laboratory is comparable to the majority of laboratories) 3 = Above average (our laboratory surpasses the majority of laboratories) 4 = Outstanding (our laboratory is a leader in quality in the DNA community) 5 = Unsure or not applicable based upon limited experience We received 142 responses to question 22 and the average answer was 3.2. Do you know of any NDIS-participating laboratory (including your own) that is currently operating with what you would consider to be a material weakness in its quality of operations? For question 23 we received 140 responses and the respondents who said “yes” had the opportunity to provide additional comments and 10 did. Comments included statements regarding weaknesses related to the following areas: the inherent limitations of one-person DNA laboratories; uploading profiles that have not been fully reviewed, or on behalf of other laboratories where quality has not been confirmed; uninvolved off-site technical leaders; and first-hand knowledge of laboratories (public and private) that emphasize productivity at the expense of quality. General CODIS Operations What three issues pose the greatest challenge to the mission of CODIS in the next five years? We received 126 responses to question 24 and the top challenges are listed below. 19 percent – Expansion and Change, 18 percent – Resources, 12 percent – Profile Integrity or System Operations, and 11 percent – Data Management or System Administration. What three aspects of CODIS do you believe have been its most important successes? We received 122 responses to question 25 and the most important successes are listed below. 34 percent – Crime Solving and Prevention, 25 percent – System Benefits, and 12 percent – Community Assistance or Communication Connections. How would you rate the FBI CODIS contractor’s overall performance? The numeric rating scale used is below. 1 = Unacceptable 2 = Poor 3 = Fair 4 = Good 5 = Excellent The average answer was 4.5. Do you believe the CODIS software has addressed the needs of the CODIS community? How would you characterize the FBI’s current management of CODIS? The numeric rating scale used is below. 1 = Unacceptable 2 = Poor 3 = Fair 4 = Good 5 = Excellent The average answer was 4 and, of the 143 responses we received to this question, 24 gave additional comments. To what extent has the FBI’s management of CODIS improved over the last 2 years? The following numeric rating scale was used: 1 = No improvement observed     2 = Minimally 3 = Moderately 4 = Substantially 5 = Extensively N/A = Unsure or not applicable based upon limited experience Of the 143 responses received, 21 provided additional comments and 27 percent said “N/A.” The average answer was 3.1. Trends in this section of the survey are listed below. SAIC does an excellent job. The software has become more user-friendly. CODIS Unit communication and accessibility have improved. The NCC, the CODIS website, and NDIS procedure updates have helped but further improvements could be made. NDIS Audit Review Panel Do you believe the NDIS Audit Review Panel has improved community compliance with the QAS? [Check all that apply] Note that this chart does not include a small number of “other” designations that were received, accompanied by supplemental comments. The comments further emphasized that individual interpretations of standards still exist. For your most recently completed audit panel review, when was the audit and accompanying corrective action documentation submitted to the FBI? We received 137 responses: 18 percent selected the year “2003,” 39 percent selected the year “2004,” 9 percent selected the year “2005”; and 34 percent selected “Not applicable,” which was offered as an option to those respondents who did not know or were not involved in this process. These respondents were then asked to skip to question 36. In your most recently completed audit panel review, please estimate how long it took from the time that your audit and corrective action documentation were originally submitted until you received notification that the audit was closed? When we asked about the total time it took for the processing of their last completed external QAS audit, we observed that there was slightly less than one-third who said it took longer than 6 months, roughly one-third who said it took from 4 to 6 months and slightly more than one-third who said it took 0 to 3 months. This does not include the less than one percent of “N/A.” In this same completed audit panel review, were you asked to supply additional corrective action documentation after you made your original submission? This information sheds some light on potential causes of delay in closing out audits, since nearly one-third of the respondents indicated that the Audit Review Panel had followed-up to get more corrective action documentation after the original submission by their laboratories. For this same completed audit panel review, please estimate how much time elapsed after you supplied all additionally requested corrective action documentation until you received notification that the audit was closed? This question was conditional upon the response to the preceding question, therefore non-valid responses were disregarded. How would you characterize your perception of any improvements made in the last 2 years in the NDIS Audit Review Panel’s timeliness of review? The numeric rating system used is below. 1 = Timeliness does not seem to be improving. 2 = Timeliness is improving slowly. 3 = Timeliness is actively improving. 4 = Necessary improvements have already been made. We received 83 responses and the average was 2.7, closest to the “actively improving” designation. Respondents also had the option of selecting “other” and providing a comment. Eleven of the 81 respondents provided comments and the trend showed that respondents had no basis to form an opinion as to whether the NDIS Audit Review Panel improved the timeliness of their reviews. FBI Guidance to the CODIS Community Does the FBI provide sufficient guidance on complying with the QAS, to ensure CODIS participants understand and comply with those standards? Note this does not reflect the less than 1 percent of respondents who said “N/A.” The respondents who said “no” had the option of providing additional comments, and 31 did. The trend in the comments was interpretation of standards varies between auditors and the CODIS community as a whole. Does the FBI provide sufficient guidance on complying with the NDIS requirements, to ensure NDIS participants understand and comply with those requirements? Note this does not reflect the less than 1 percent of respondents who said “N/A.” The respondents who selected “no” had the option of providing comments, and 20 did. The trends identified in the comments were interpretation of standards varies between auditors and the CODIS community as a whole and the CODIS Unit does not respond to questions in a timely manor. Do you believe that the FBI’s audit document enables an external QAS auditor to identify all of a laboratory’s quality assurance weaknesses? We received a total of 140 responses to question 38. The respondents who selected “no” had the option of providing additional comments, and 33 did. The trends identified in the comments were interpretation of standards varies between auditors and the CODIS community as a whole and the standards and the audit document need to be updated. Do you believe the FBI has provided adequate training on the proper use of the QAS audit document to ensure that community QAS auditors are consistent and thorough in their assessment of compliance with the QAS? We received a total of 140 responses to question 39. The respondents who selected “no” had the option of providing additional comments, and 48 did. The trends identified in the comments were interpretation of standards varies between auditors and the CODIS community as a whole. To be effective, do you believe external QAS auditors should be qualified in the method or platform they are auditing? We received 143 responses to question 40. The respondents who said “no” were given the opportunity to provide comments, and 29 did. To be effective, do you believe external QAS auditors should be qualified in the specific application (casework or offender) they are auditing? We received 143 responses to question 41. The respondents who said “no” were given the opportunity to provide comments, and 26 did. The responses received about auditor qualifications to both question 40 and 41 were put into context with the comments that were provided. A few examples are below: Technology, platforms, and methods are similar enough where an auditor who is proficient or qualified in one will know enough to audit them all. A casework-qualified auditor can audit an offender laboratory but not vice versa. The auditors should be qualified or previously qualified, or at least one person from the audit team should be qualified. Do you have any suggestions for how the DNA community’s auditing structure (i.e., the way audits are conducted, processed, and reviewed) can be improved, to better aid national CODIS laboratory quality? The trends from comments we identified in question 42 are below: Interpretation of standards should be made more consistent. Actual QAS should be revised and clarified. Use the CODIS website to disseminate auditing information to CODIS the community. What level of consistency do you believe has existed in guidance to the CODIS community regarding compliance with NDIS requirements under the current CODIS Unit management? [Do not count as inconsistencies the changes resulting from expansions in the law to what is permitted in NDIS.] The numeric rating scale used for question 43 is below. 1 = Inconsistent. The messages conveyed at meetings or conferences do not match what is contained in written guidance or what is conveyed in individual responses. 2 = Somewhat consistent. The messages conveyed at meetings or conferences periodically match what is contained in written guidance or what is conveyed in individual responses. 3 = Consistent. The messages conveyed at meetings or conferences match what is contained in written guidance or what is conveyed in individual responses, with rare exception. 4 = Very consistent. The messages conveyed at meetings or conferences always match what is contained in written guidance and what is conveyed in individual responses. N/A = Unsure or not applicable based upon limited experience [skip to question 45] The average rating to this question was 2.8, which is closest to the “consistent” designation. In addition, 12 percent of the respondents said “N/A." If you designated a rating for question 43 as less than consistent: How often did you feel that the inconsistency of guidance has limited your ability to perform your CODIS administrator duties or to comply with the NDIS Requirements? To what cause(s) do you attribute the inconsistency of guidance? [Check all that apply] [Image Not Available Electronically] The categories in the preceding graphic were offered as responses and since multiple responses were permitted to this question we could not calculate true percentages. This graphic is intended only as a way to convey the magnitude of the responses given, by the 30 people who responded to this question. What do you believe would improve the consistency of understanding within the CODIS community regarding compliance with NDIS requirements? [Image Not Available Electronically] Multiple responses were permitted for question 45, and the options displayed above were the ones we offered as responses. In addition there were a total of 47 respondents who provided additional comments. The trends identified in these comments are listed below: The QAS and the related audit document need to be updated. Post frequently asked questions and answers on QAS and NDIS requirements. Do you have any other comments, suggestions, or concerns that you can offer regarding FBI administration of CODIS, CODIS operations in your laboratory or in the CODIS community as a whole, or factors that have the potential to adversely impact CODIS operations in the future? We received a total of 40 comments to question 46 and the trends we identified are listed below: There are inconsistencies in the way standards are interpreted throughout the CODIS community. The CODIS community needs more resources. Inconsistencies in interpretation in the standards throughout the CODIS community was also one of our comment trends; specifically we received 161 comments from 83 respondents on the subject. The second comment trend in question 46 regards resources that include personnel, better technology, and tools, such as expert systems. In addition, comments were made that indicated that the lack of resources force laboratories to make difficult decisions regarding resource allocation, and thus place pressure on quantity versus quality, which may not be best for the CODIS community as a whole.