|Return to the USDOJ/OIG Home Page
Return to the Table of Contents
The FBI DNA Laboratory: A Review of Protocol and Practice Vulnerabilities
Office of the Inspector General
Blake's misconduct occurred even while the DNAUI received satisfactory audit reports from both internal and external auditors. As described previously, Blake was able to conceal her actions for almost a 2-year period because of inadequate DNAUI internal controls.74 As a result, the OIG undertook an assessment of the Laboratory's DNA protocols and practices to determine whether oversight vulnerabilities exist within the DNAUI.75
I. ASSESSMENT FOUNDATION AND PROCESS
OIG staff designed the assessment to examine comprehensively the DNAUI's protocols and their application. Our objectives were to:
These objectives were accomplished in two phases. In the first phase, the OIG team reviewed the written DNAUI protocols for vulnerabilities. The second phase consisted of OIG fieldwork at the DNAUI laboratory.
To facilitate the assessment, particularly the review of the protocols, we recruited scientists from the national DNA community to assist the OIG. We selected the scientists by first soliciting recommendations from various contacts in the public and private DNA laboratory community, and then selecting three persons who could provide the team with varied experiences and a full understanding of the standards that govern forensic DNA laboratories. The scientists selected were crucial contributors to the OIG's work and brought to the assessment expertise in academic research and instruction; federal, state, and private DNA laboratory analysis and administration; and national DNA advisory committee participation.
The scientists we recruited were:
Each of these individuals has a distinguished record in the field of DNA analysis. Appendix 5 to this report contains brief biographies of the scientists.
To formalize the scientists' participation in the OIG's work, we conducted an orientation meeting in September 2002. At that time, OIG staff briefed the scientists on the circumstances surrounding the assessment, including Blake's misconduct.
During the orientation meeting, the FBI DNAUI Chief explained to the scientists the timeline of events and the actions taken by the FBI after their discovery of Blake's actions. The scientists and OIG staff also toured the DNAUI facilities - at that time housed at FBI Headquarters in Washington, D.C. - to gain an overview of the DNAUI's operations.
The OIG staff distributed to the scientists checklists and supplemental guidance designed to facilitate the scientists' review of the DNAUI protocols. The checklists and guidance contained key terms and definitions specific to our assessment, and supplemented the verbal explanations provided by OIG staff describing what the scientists were to consider when determining whether a protocol is vulnerable. OIG staff structured the checklists so that they corresponded to the table of contents for each of the protocols reviewed to ensure that each section of the documents was assessed. See Appendix 6 (examples of the checklists and guidance provided to the OIG scientists).
For purposes of this assessment, we defined "vulnerability" as a function of "impact" and "risk" as follows:
In addition to these definitions, categories of severity were defined so that numeric ratings could be generated. The scientists were given descriptions of the "impact" characteristics that a protocol or procedure would have if it fell into low, medium, or high impact categories.77 They also were given similar descriptions for the low, medium-low, medium, medium-high, and high risk78 categories.79 These categories are further defined in the guidance contained in Appendix 6. Along with the checklists and guidance, OIG staff and the scientists reviewed the DNAUI protocols described below, which had been supplied to the OIG by the FBI. According to the FBI, these documents were the most current version of each of the protocols governing DNAUI activities at that time.
The OIG staff established guidelines for implementation of the assessment, including procedures that ensured that the scientists would document their conclusions on the checklists provided by the OIG. In addition, the scientists were instructed to base their conclusions solely on their analysis of the written materials above. In other words, when gauging vulnerability within the DNAUI, the scientists were to consider the contents of the protocols only, not factoring in any understanding they might have of the FBI's DNA methods or their observations and conversations with DNAUI staff members and management during the tour of the DNAUI. OIG staff intended the guidelines to enable the scientists to take a fresh look at subjects with which they were obviously familiar, so that they could find weaknesses in the internal controls that others may have missed.
The scientists reviewed the protocols in two sequential phases. OIG staff worked with the scientists to divide the document sections into two groups for review, referred to simply as Phase 1 and Phase 2. Phase 1 covered pre-analysis protocols; Phase 2 focused on the remaining protocols, including those related to the actual analysis of DNA samples. The checklists used by the scientists reflect the division that was made. See Appendix 6.
The scientists were given a period of time to review the protocol documents for each phase and then met with OIG staff to discuss and record the vulnerabilities identified. The meetings generated a consensus on the impact and risk ratings that should be assigned to each protocol section. In addition, the OIG recorded the underlying concerns for the agreed-upon ratings assigned by the scientists to ensure that the fieldwork conducted later and the conclusions and recommendations ultimately reported were an accurate reflection of the specific underlying weaknesses.
At the conclusion of the Phase 2 meeting, OIG staff asked the scientists to assist them in devising fieldwork to identify the actual work practices of DNAUI staff members. In preparation for fieldwork design, OIG staff summarized the protocol sections in which the scientists had identified key vulnerabilities, and analyzed the results to determine if any of the sections pertained to similar subject matter. In addition, we analyzed the comments voiced by the scientists for recurring themes and categorized them into key concern areas. From this analysis, we designed fieldwork to verify actual laboratory practices for protocols deemed vulnerable, and to assess whether these practices served to mitigate the vulnerabilities identified.
OIG staff conducted this fieldwork from March 12 through 21, 2003. The fieldwork generally was comprised of a tour of the new DNAUI facility in Quantico, Virginia, and a series of interviews of staff members from within the DNAUI and the Laboratory Division. Where possible, interview responses and observations made during the tour were checked against supporting documentation for verification.
Fieldwork interviews served as our primary source of insight into the DNAUI's operations. We recognized that it would be important to collect information from a broad cross-section of personnel, since we intended to analyze their responses for consistency with the protocols, with others of the same operational position, and with respondents in different positions. Therefore we took the following steps to ensure variety in our sources of information.
Since the DNAUI staff function as teams, with each team generally consisting of a Serologist, a PCR Biologist, and an Examiner, we interviewed multiple staff members in each of these positions.
We also recognized that the amount of time that a person had held a position could affect his or her fluency in describing certain processes. Consequently, we interviewed the most senior and the most junior employees in each position, and judgmentally picked a third person in that same role. For the third Serologist and Biologist interviewees, we selected a staff member who was currently in training for a different team position and thus would have a level of familiarity with the duties performed in both roles.
This interviewing scheme was expanded to include a fourth interviewee from among the Examiners, so that our interviewees would include, in addition to the most senior and most junior Examiners, the Examiners who also supervised the key programs within the DNAUI: the Examiner-Supervisor of the Serology Program and the Examiner-Supervisor of the PCR (STR) Program.
Finally, we interviewed DNAUI management, including the Unit Chief, the Assistant Unit Chief, and the Quality Assurance Manager; and Laboratory Division management, including the Laboratory Director, the Deputy Director, and the Chief of the Scientific Analysis Section.
We also reviewed documentation and interviewed key personnel regarding: 1) the factors considered in the design of the new DNA facility; 2) the training curriculum and methods used within the DNAUI, along with various staff training records; and 3) the status of development of the Laboratory Information Management System (LIMS), a computerized tracking system for evidence, samples, and other information. However, we did not include in our fieldwork design an analysis of case file documentation for two reasons:
In addition, we relied upon the work performed by DNAUI management and staff members, as described in Chapter Four, Section III.A, to determine whether other DNAUI Biologists had failed to process the negative controls prior to the discovery of Blake's misconduct. That work determined that the controls were completed as required.
We analyzed the results of our fieldwork and compared them with the concerns voiced and vulnerabilities detected by the scientists to discern whether information gathered during fieldwork confirmed the extent and nature of the scientists' conclusions. We then conducted a follow-up meeting with the scientists to discuss the fieldwork results and to adjust, if necessary, their earlier conclusions that had been based strictly on the document review. The scientists made only a few minor updates to their earlier observations to reflect the information obtained during fieldwork. Generally, they did not change their conclusions regarding protocols previously identified as vulnerable.
II. DNA UNIT I PROTOCOLS AND PRACTICES IDENTIFIED AS VULNERABLE TO ABUSE
During our assessment within the DNAUI, we examined two types of vulnerabilities: protocol vulnerabilities and practice or operational vulnerabilities. Our textual analysis of the FBI protocols that govern the DNAUI revealed various weaknesses, which are described in detail immediately below. In addition, in the course of completing field work that examined how staff members implemented the protocols that we identified as problematic, we discovered numerous practice vulnerabilities. These are described along with our analyses of the various protocols. The specific vulnerabilities we examined within the two general categories (protocol and practice) are presented in order of significance based upon the scope of the vulnerability (generally, the number of document sections associated with it or the pervasiveness of the problem across DNAUI functions) and its severity (generally, the extent to which the vulnerability could undermine the DNAUI mission).
As explained in Chapter Five, Section I.C of this report, our analysis of protocol vulnerabilities is based on a review of 5 FBI manuals and 5 instructional documents that collectively contain 172 topical sections.85 See discussion supra at page 33. From this review we identified 31 sections as significantly vulnerable86 to inadvertent or willful noncompliance. It is important to note that our identification of a "vulnerability" should not be misconstrued as an invalidation of the science or techniques used by the DNAUI, or as an indication of the inadequacy of the entirety of DNAUI policies on a particular subject. Our use of the term "vulnerability" is limited to its definition as set forth in Chapter Five, Section I.C.
The sections we identified as significantly vulnerable to inadvertent or willful noncompliance are identified below, with the most vulnerable sections in bold italics:
Although we identified 31 document sections as vulnerable, the causes of the vulnerabilities were few in number. In general, one or more of four reasons accounted for each of the vulnerability designations: 1) the protocol lacks sufficient detail; 2) the protocol fails to inform the exercise of staff discretion; 3) the protocol fails to ensure the precision of manual notetaking; and 4) the protocol is outdated.
The following chart depicts the categories of vulnerabilities and the document sections to which each category applies.
[Image not available electronically]
Given that each type of vulnerability was observed in multiple document sections, for ease of comprehension and to avoid repetition, we describe below each of the protocol vulnerabilities according to its cause rather than by individual document section.
Our review of the DNAUI protocols revealed that 29 of 172 document sections lacked the detail necessary for a technically qualified DNA scientist to reproduce all aspects of the analysis procedures in use in the DNAUI without the potential for variation. In our view, a qualified DNA scientist should be able to locate the DNAUI's essential testing requirements in its protocols and not have to resort to peripheral sources. Further, the protocols should contain guidance sufficient to ensure that qualified scientists are consistent in their interpretation of the testing requirements.
While the sections we have identified as lacking essential detail are a relatively small percentage of the total protocols examined in our review, we consider this vulnerability to be the most significant of the ones we identified and the most important indicator of the DNAUI's susceptibility to inadvertent noncompliance. Protocols that lack essential detail can create a work environment that encourages use of disparate and unproven laboratory practices. When laboratory staff members must rely on ad hoc verbal cues from their peers to complete their duties, the risk increases that they will deviate from the practices that are necessary to generate valid and reliable testing results. In addition, protocols that lack essential detail can foster a perception among staff members that the protocols are not authoritative and can be disregarded, even though they should serve as the DNAUI's primary source of instruction.
We describe below six sets of protocol sections that share similar deficiencies: 1) Evidence Control, Facilities (Security), and Procedures for the Examination of Evidence; 2) Extraction, Amplification, and Laboratory Set-Up; 3) Procedures for the Preparation of Dried Bloodstains from Coagulated Whole Blood and from Anticoagulated Whole Blood; and Procedures for the Extraction of Suspected Semen Stains: Quality Control Procedures and Questioned Stain Extraction Procedure; 4) Case Documentation Policy and Case Assignment, Documentation and Review; 5) Guidelines for Control Samples and Interpretation of Control Samples; and 6) Organization and Management and Authority and Accountability.
Forensic Standard 6.1 requires that laboratories have a facility that is designed to provide security and minimize contamination. This includes, as specified in subsection 6.1.1, controlling and limiting the access to the laboratory; and, as specified in subsection 6.1.2, separating by time or space evidence examinations, DNA extraction, PCR setup, and PCR amplification. Further, Forensic Standard 7.1 requires that DNA laboratories have and follow a documented evidence control system to ensure the integrity of physical evidence, including (as specified in subsection 7.1.4) secure areas for evidence storage.
While the DNAUI protocols address compliance with these standards in general terms, certain sections of the protocols lack the level of detail necessary to ensure that staff members understand and comply with management expectations regarding evidence control and security. We identified four sections as problematic: the Evidence Control Policy of the FBI Laboratory Division Quality Assurance Manual; the Evidence Control and Facilities (Security) sections of the DNA Analysis Unit I Quality Assurance Manual; and the Procedures for the Examination of Evidence section of the FBI Laboratory Division Caseworking Procedures Manual.
For example, the Evidence Control Policy requires in Section 3.2 that "Evidence will be labeled, stored, secured, and/or sealed to prevent loss, cross-transfer, contamination, or deleterious change." Additional guidance is not provided regarding how staff members should implement this protocol with respect to particular kinds of evidence. While this information may not be included within a laboratory-wide policy such as this one, we expected to find, and did not, a reference to other FBI manuals or protocols where more specific guidance can be obtained. Similar language is found in Section 3.5, which requires without elaboration the use of "universal precautions . . . to ensure the health and safety of personnel."
The Evidence Control Section of the DNA Analysis Unit I Quality Assurance Manual also includes vague provisions. For example, Section 7.6.1 states that "DNAUI will utilize documented standard operating procedures which minimize potential sample loss, contamination and/or deleterious change to the evidence." It does not, however, identify what those procedures are or reference another source where they are specified. Section 7.6.2 requires that "an attempt to limit the consumption of evidence should be made, when possible," and although three subpoints are provided explaining how storage should be used to limit consuming an evidence sample, the protocol does not clarify what is meant by "when possible" or address methods other than storage that can be used to ensure that a remnant of the evidence remains for future testing.
In addition, the proper handling of evidence during examinations is not addressed in the Evidence Control section. That section contains a discussion of chain-of-custody issues and the proper transfer, storage, and labeling of evidence, all of which are crucial to ensuring evidence integrity. Yet, if evidence is mishandled during the course of the examination itself, the benefits obtained from these precautions will be lost. While we were able to locate some examination evidence handling information in the Short Tandem Repeat Analysis Protocol, no reference was made to it in the evidence control policies, and the limited information provided in the STR protocol is no substitute for comprehensive guidance. We anticipated, for example, that the DNAUI protocols would specify that Serologists with workbenches in close proximity to each other should not have case evidence open at the same time or that some type of "sign-up" sheet would be used to ensure that they do not use adjoining areas concurrently. Also, we expected to find a clear description of what it means to "separate" known and unknown DNA samples. Separation could mean that staff members clean between processing samples but use the same area, or alternatively that a different ventilation hood is used for the known and unknown samples. However, no such guidance on either issue is provided.
We also identified in the Procedures for the Examination of Evidence similar broad statements as those found in the Evidence Control Policy.87 For example, although Section 4.1.3 directs staff members to "follow established unit precautions to preserve the integrity of the evidence," we were unable to locate a listing or explanation of the referenced "unit precautions." Also, Section 4.5.1 directs staff members to place evidence in a limited access secured area when they must leave it unattended and "protect the exposed areas of the evidence from loss, cross-transfer, and/or contamination." However, no explanation is provided regarding how staff members should implement this requirement, or what is intended by terms such as "protect" or "exposed areas." During an OIG tour of the DNAUI, staff members explained that evidence items under active examination can be left out while staff take breaks and that the evidence is marked with an "Evidence - Do Not Disturb" sign. Leaving evidence exposed poses unnecessary risks of evidence contamination, and the Procedures for the Examination of Evidence should be amended to require the proper storage of evidence when it is not being examined.88
In the Facilities (Security) Section of the DNA Analysis Unit I Quality Assurance Manual, the discussion of facility access limitations is not clear. For example, Section 6.1 specifies that "access to these areas [specific rooms that comprise DNAUI space are listed] are controlled by DNAUI personnel." However, no further explanation is provided for how DNAUI personnel are expected to accomplish this access control. Further, the Section states that "non-DNAUI personnel are permitted entry during normal working hours for purposes relative to laboratory operations." Yet, no further explanation is provided concerning what restrictions should be imposed on non-DNAUI personnel (e.g., whether visitors must be escorted and in what areas they are permitted access). The Section seems to assume that DNAUI personnel already are aware of existing physical access restrictions, for the building and for the laboratory space, that serve as components of the Unit-specific access control. Instead, access control and physical security should be described (and other relevant sources referenced) in the DNA Analysis Unit I Quality Assurance Manual, and those descriptions should not assume an understanding of other physical access limitations.89
The overly broad or vague language used in the above evidence-control related protocols creates a number of problems. Staff members may not fully understand how they are expected to avoid loss, cross-transfer, or contamination of evidence, and may not realize that they are putting evidence at risk through the handling methods they have adopted. One item of evidence might be allowed to come too close to another item, and hairs, blood flakes, or fibers that become airborne might inadvertently be transferred to other evidence, possibly causing the analysis results to reflect improperly a connection between the cases or a DNA profile of someone not actually involved in the case.
Further, leaving evidence exposed and unattended poses a risk to the Examiner's ability to attest that team members maintained control over it. The importance assigned by DNAUI management to controlling access to evidence is apparent from other DNAUI protocols: 1) as a means of verifying limited access, evidence not under examination is stored sealed with tamper-evident material; and 2) evidence not under examination is secured in locked refrigerators, freezers, and bulky-storage space. It is difficult to reconcile these policies with protocols that contemplate that evidence will be left unattended and that fail to provide clear guidance for how it is to be protected.
Finally, there is a risk that evidence that is left unattended could suffer damage through an unforeseen event, such as fire-alarm sprinklers being set off in an emergency, or a plumbing leak that floods and compromises space below it. If that evidence were in a locked cabinet or refrigerator/freezer, it would be further protected in such events.
In light of these potential risks, we conducted fieldwork to determine whether other factors diminished the risks we had detected during our document review. At the time of our fieldwork, the DNAUI had relocated from FBI Headquarters to a new laboratory building at Quantico, Virginia. We found that the change of facilities mitigated to some extent the impact of the lack of detail in several of the facility-related protocols. Specifically:
Our fieldwork also included interviewing staff members and management to determine whether their work practices offset any of the vulnerabilities created by weaknesses in the evidence control procedures and protocols.
We determined that there are several practices, primarily falling under the heading of cleaning and decontamination, that mitigate to some extent the protocol limitations. For example, we were informed by all staff members involved in evidence examinations that it is DNAUI practice to cover the work surface with paper prior to laying out an evidence item for examination. Such a step reduces the possibility for cross-transfer, since the paper that surrounds the item serves as a buffer between the item being examined by one member and another in close proximity. Also, we were informed that if an item stretches beyond the parameters of the paper, then that portion of the work surface is decontaminated before moving on to another item. Further, staff members stated that work surfaces are decontaminated between cases, and utensils used in the examination of an item are decontaminated between items. Almost all staff members and management personnel interviewed regarding evidence handling issues reported working on one item at a time and one case at a time, separating evidence samples from known samples, using and changing gloves, cleaning and/or decontaminating work benches and hoods regularly, and changing work surfaces between items.91 Also, the responses of staff members that were able to identify where in the protocols these requirements are found, as well as the responses provided by management, were consistent as to which protocols serve as the source of guidance on general evidence handling, cleaning, and decontamination.
However, not all of the staff members interviewed were able to identify where guidance on these topics could be located in the protocols. In addition, in spite of the consistency of the responses among those that did know where the guidance could be located, their responses indicated in other ways the need for greater clarity and detail in the written procedures and protocols. For example, several interviewees commented on the lack of general evidence handling guidance for serologists (additional deficiencies found in the Procedures for the Serological Identification of Biological Substances on Evidentiary Materials are addressed in the next section). In addition, when questioned regarding the separation of known and unknown samples, interview responses lacked a precise indication of the period of time or amount of space that meets the requirement for "separation." Consequently, it was not evident from the responses whether staff members have a clear understanding of the requirement to separate known and unknown samples.
Further, of all the staff members and management interviews we conducted, only one Laboratory employee, an Examiner, cited as a source of guidance the FBI Laboratory Division Caseworking Procedures Manual. This Manual provides the most detailed guidance available to DNAUI staff members on the subjects of inventorying, identifying, and examining evidence. In addition, it is the only protocol that addresses the issue of leaving evidence unattended during examinations, and the requirements (even though they are overly broad) for how that evidence is to be handled. One staff member, a senior PCR Biologist, claimed to have never seen the Manual before.
Many of the interviewees commented on the lack of written guidance concerning the routing sequence when evidence needs to be distributed to multiple units. These interviewees indicated that they know there is a definite order that evidence needs to follow to preserve each unit's ability to conduct testing. For example, the Latent Fingerprints Unit must test before the DNAUI tests, and the item cannot be stored in a refrigerator or freezer prior to the latent testing. One Examiner commented on the importance of this issue, emphasizing that the Serologists must be very conscious of the items that might need testing by other units, and work to preserve the evidence that is needed for this testing. The Examiner explained that when there is a question about how to route evidence, Serologists are expected to obtain input from the other unit. Yet, this guidance is not part of the DNAUI's serology procedures.
We also detected variance in responses regarding particular procedures in the protocols. While not every variance translates into a problem, diverging interpretations by staff members could indicate that they do not fully understand protocol requirements or are given a measure of flexibility that is not reflected in the protocol. The following examples illustrate these points:
During our fieldwork at the FBI's new laboratory, we inquired about facility security and were informed that access to DNAUI space is limited to DNAUI staff members, either by regular keys or by card keys. Although these practices seem sufficient, they are not evident from the text of the relevant protocol. In addition, after receiving interview comments that showed that some Laboratory personnel did not understand the importance of the protocols, we inquired whether staff members are required to certify that they have read and understand protocol revisions. According to the Unit Chief, staff members are required to attest to the receipt of new versions of the protocols. For notifications of changes that are made between revisions, no certification of receipt is required. The Unit Chief added that during training both trainees and trainers must attest to the trainee's understanding of the importance of adherence to protocols.
In light of the foregoing, we concluded that the potential risks posed by the lack of detail in the protocols described above, particularly the risks associated with facility limitations, have been mitigated somewhat by the following: 1) the DNAUI has moved to a new facility that no longer has the limitations we noted in the previous facility; 2) interview responses identified several methods and practices that exceed protocol requirements and that appear to protect the evidence; and 3) interview responses were largely consistent and in agreement with the protocols.
However, several interview responses we received revealed that some DNAUI staff members have an incomplete or inaccurate understanding of evidence handling and control requirements, and two persons we interviewed indicated that the protocols are not essential to their daily activities. We believe that staff members who do not understand or appreciate the importance of evidence handling and control procedures are susceptible to inadvertent noncompliance with the applicable protocols.
We recommend that DNAUI management:
Forensic Standard 9.1 requires that DNA laboratories follow written analytical procedures, including a standard operating protocol for each analytical technique employed.93 Forensic Standard 9.1.2 requires that the procedures meeting these requirements address reagents, sample preparation, extraction, equipment, controls, and data interpretation. While the DNAUI meets these standards with the Short Tandem Repeat Analysis Protocol, we identified important information that is missing from its Extraction, Amplification, and Laboratory Set-up sections.
First, both the Extraction and Amplification sections fail to specify the requirements for the separation of known and unknown samples and of high- and low-quantity DNA samples. We believe that the Extraction section should specifically prohibit having two sample tubes open at once, and the Amplification section should require control samples to be processed last. The Extraction section also contains provisions that are overly broad: the incubation time given in Section 4.1.3 is listed as between 2 and 24 hours, and there are provisions in Sections 4.3-9 for staff members to perform additional organic extractions, if needed, without information provided regarding when and why those additional extractions would be appropriate. Finally, Amplification Section 6.7 fails to describe clearly the order in which sample tubes should be set up.
We recognize that some of the missing information cited in the previous paragraph is located within the Laboratory Set-up section, specifically the requirement for the separation of known and unknown samples and of high- and low-quantity DNA samples. The Laboratory Set-up section also clearly requires that the negative control samples be processed last as part of the amplification set-up, which is important for those samples to indicate contamination effectively. However, because the protocols function in part as a reference manual, the listing of these requirements in the Laboratory Set-up section does not diminish the need for them also to be presented in the protocol sections that address the stages in the analysis process (i.e., extraction and amplification) where an understanding of the requirements is most crucial. An employee who seeks guidance on an analytical step should find complete information in the relevant sections of the protocol and should not have to search elsewhere.
Moreover, the requirements described above that are included in the Laboratory Set-up section do not provide the level of specificity and clarity necessary for staff members to understand what is intended by the "separation" of the types of samples. For example, special precaution 3 under Section 12.2 states that "[i]t is important that the DNA extraction of questioned samples be performed at a separate time from the DNA extraction of known samples." Yet, the protocol does not explain what is meant by "separate time." It should state that all the unknown samples must be processed, capped, and removed from the immediate work-area before staff members begin to process the known samples. Also, special precaution 4 under Section 12.1 directs staff members to "[w]henever possible, extract samples containing high levels of DNA (whole blood) separately from samples containing low levels of DNA (stamps, small bloodstains, etc.) to minimize the potential for sample-to-sample contamination." Unfortunately, the qualifier "whenever possible" renders the protocol vague and open to differing interpretation.
Due to the importance of separating known and unknown samples, the risks posed by the lack of detail in the protocols described above are substantial. Without the requirement to process, cap, and set aside the unknown samples before processing the known samples, the risk of cross-contamination increases. In addition, without a clear requirement that separation be maintained during extraction and amplification, samples may be compromised later in the DNA analysis process. The implications of this particular aspect of a DNA laboratory's internal controls are far-reaching for the evidence tested and the presentation of the analysis results in court: adequate separation of known and unknown samples enables examiners to testify that the connection made between a suspect and the crime scene evidence analyzed is not the result of cross-contamination between the known sample from the suspected perpetrator(s) and the evidence items. If the known samples were to contaminate the evidence, the analysis results for the evidence profile would reflect the profile from a known sample as well as the profile obtained from the DNA present on the evidence item. In addition, the strength of the DNA in the known sample could serve to "drown out" the results of the DNA in the evidence, causing the resultant profile to reflect only faintly the DNA present in the evidence. Further, if the profile from a known sample were to appear in the analysis results for an evidence sample, an Examiner might wrongfully conclude that the contributor of that known sample is the potential source of the DNA on the evidence in the case being investigated.
Lastly, aside from sample separation, the lack of specific guidance in the protocols regarding incubation time and the use of additional organic extractions leaves Unit staff members at risk of deviating inadvertently from management expectations in those areas.
In our interviews with DNAUI staff members regarding extraction, amplification, and laboratory set-up procedures, we asked them to comment on: 1) the separation of sample types and the separation of the stages in the analysis process; 2) the exact incubation time used by staff members; 3) the use of the provision that governs additional organic extractions; and 4) the details of amplification set-up. Responses were virtually identical from the PCR Biologists and Laboratory management regarding the incubation time, the circumstances under which they would perform additional organic extractions, the separation of the stages in the analysis process, and the details of amplification set-up. In addition, written guidance had been disseminated to staff members restricting the incubation time to a more precise period.
Responses also were very similar regarding the separation of known and unknown samples during analysis. However, as previously described at page 86, the interview responses we received regarding sample separation did not reveal that staff members share a common understanding of the meaning of "separation," which would mitigate the risk resulting from the lack of clarity in the protocols. In addition, staff members and management generally indicated that the evidence and known samples are currently separated only through the completion of the extraction of the DNA from the samples. From that point forward the sample tubes from evidence and known sources can be on the same tray and can be set up for amplification at the same time. While such a policy limits the risk of contamination during the extraction process, it does not address possible contamination during amplification set-up. Therefore, evidence samples are still at risk for contamination prior to amplification.
With respect to separation of the analytical stages themselves, one area of vagueness surfaced when we interviewed the PCR Biologists about the amplification set-up process. Interviewees acknowledged that it is clear in the protocols that DNA extraction, PCR set-up, and amplification are to be conducted separately. Further, there is a requirement to have dedicated equipment for the pre-amplification areas of the Laboratory isolated from the dedicated equipment used in the amplification areas. We confirmed that these requirements are explained and referred to throughout the Laboratory Set-up section of the Short Tandem Repeat Analysis Protocol. However, we noted from the interview responses that the PCR Biologists have a practice of using a pre-amplification tray to carry tubes to the amplification room. The tubes are then placed into the thermal cycler and the "transport tray" returned to the pre-amplification area. Since the protocols do not address this practice or provide guidelines regarding the cleaning or decontamination steps that should be completed before that tray is put back in use, we could not compare the interviewee responses to written requirements. However, recognizing that some cleaning of the tray is necessary prior to future use, we inquired further with the PCR Biologists we had previously interviewed and were told that they understand the importance of cleaning these trays, and that they believe this to be true for all the PCR Biologists. Each of the interviewees described the cleaning method they use, whether a bleach wash, a UV light source, or both. But the PCR Biologists acknowledged that there is no specific guidance on this aspect of the process in the protocols.
Finally, the DNAUI Chief explained that, in light of the foregoing risks, the Unit is making arrangements to have the known and unknown samples processed in different locations within the Unit. Thus, different staff members, equipment, and space will be used to analyze the known and unknown samples, mitigating any risk of cross-contamination due to the lack of separation.
Therefore, we concluded that the lack of guidance for Biologists on the use of transport trays was mitigated somewhat by Biologist cleaning practices. However, we find unmitigated the risks posed by: 1) the lack of clear delineation in the protocols or staff responses for what constitutes adequate separation of known and unknown samples; and 2) the failure to maintain that separation through amplification. While we acknowledge that DNAUI management's proposed solution of analyzing known and unknown samples in different sections of the Laboratory would address these risks, we base our conclusions and resultant recommendations on the protocols in place at the time of our review.
We recommend that:
Our analysis of the Procedures for the Serological Identification of Biological Substances on Evidentiary Materials, which includes bloodstains, semen, quality control, and stain extraction protocols, revealed that it fails to provide adequate guidance on various evidence handling procedures. The following examples illustrate information gaps that DNAUI Serologists fill using unwritten standards.
A risk posed by the failure to establish comprehensive guidelines on the amount of evidence to use in testing (e.g., items 1 & 2 above for semen protocols) is that DNAUI staff members inadvertently may use too much of the available body fluid stain or exhaust the supply altogether, making future testing impossible. In circumstances where the first analysis run proves problematic, such waste could prevent acquisition of scientifically valid testing results since staff would not later have the option of reanalyzing a sufficient quantity of the evidence. In addition, the lack of general evidence handling guidance in these sections (e.g., items 3 & 4 above for semen protocols, and use of cotton sheeting in bloodstain protocol) poses an unnecessary risk of contamination, possibly hindering the production of a usable DNA profile. Further, the lack of a clear description of what should occur if a stain extraction procedure detects no semen (e.g., item 5 above for semen protocols), presents the risk that the analysis process could be prematurely halted rather than taking additional steps to determine if the evidence sample contains DNA. Further, although in isolation the remaining information missing from the protocols may have only a minor impact on DNAUI operations, the cumulative effect of failing to provide comprehensive guidance in the protocols is to allow a significant portion of the Unit's operations to be determined by the idiosyncrasies of individual staff member preferences. This in turn conveys the message that the written procedures and protocols are a peripheral source of instruction, and puts the DNAUI at greater risk for protocol noncompliance.
During our fieldwork we interviewed DNAUI Serologists and Laboratory management regarding various information gaps in the serology protocols. Specifically, we asked them to describe: 1) the circumstances in which they would or would not perform each of the serology tests; 2) the factors they consider to determine how much evidence to test and how much to forward to the PCR Biologist; 3) their understanding of the usable life of the semen control; and 4) what steps are taken after testing if they obtain a negative result (indicating that no semen was detected). Serologist responses were generally similar on each of these issues and to the responses provided by management, indicating that staff members have acquired a clear understanding through training or peer input of management expectations.
Serologists offered mixed responses on only one issue: the useable life of the semen control. While they guessed what the expiration date might be (generally their guesses of one year were correct), they also stated that they look for a strong result from the control. If they do not obtain one, they simply retest a new batch of control sample.95 Further, they commented that the semen control is used in significant enough volume that the expiration date is not an issue (each batch is used before the expiration date would be reached).
We reviewed the logs documenting the reagents used and confirmed that the last three batches of semen control were depleted prior to each batch's 1 year expiration date. In addition, we confirmed from the protocol that an expiration date is provided for staff members, but not within the Procedures for the Extraction of Suspected Semen Stains section of the protocol. Instead, the useable life of semen control samples is set forth in the Procedure for the Presumptive Identification of Semen. Consequently, we concluded that, while Serologists should be reminded of the control expiration dates, the Serologists' unfamiliarity with that information does not pose a significant risk to the proper analysis of evidence samples.
In addition, we determined from the interview responses that staff member practices include various helpful internal controls that are not reflected in the protocols. Specifically, one member of management stated that the serology procedures do not reveal that a team's PCR Biologist also looks at and describes the evidence that the team's Serologist receives, and those descriptions can be compared for consistency. Staff member interviews also described a practice of performing a "general swabbing" of an item to ensure that no possible sources of DNA on that item have been missed. OIG team scientists stated that this is a valuable practice to ensure that staff members exhaust all options in finding potentially probative DNA sources.
Thus, our field work demonstrated that staff members generally possess a clear and consistent understanding of management expectations regarding serology procedures. We therefore conclude that the risk posed by the incomplete protocols has been mitigated in large part by communication of the necessary information through other means, such as training and peer guidance. Despite this fact, we believe that the better practice is for the DNAUI to revise its serology protocols to provide comprehensive guidance on all serology procedures in use in the Unit. The staff members and managers we interviewed agreed that many of the details they described about serology procedures are not found in the serology protocols. Further, one of the Serologists stated that the methods that staff members are supposed to employ to navigate an item through the serology process seems to be addressed only through training, and that the work of the Serologists is shaped by the preferences and idiosyncrasies of the person who trained them. One management interviewee acknowledged that the serology procedures do not contain information on administrative processes, such as proper storage of evidence, even though Examiners frequently are cross-examined on this topic in court.
Forensic Standard 11.1 requires that DNA laboratories follow written procedures for taking and maintaining case notes to support the conclusions drawn in laboratory reports. In addition, Forensic Standard 11.1.1 requires that DNA laboratories maintain, in a case report, all documentation generated by Examiners related to case analyses. While the DNAUI has implemented procedures and protocols for these broad standards, we determined that two of the protocol sections we reviewed lacked the kind of detailed guidance necessary to ensure that all staff members understand and comply with management expectations for case documentation and review. Specifically, these sections are the Case Documentation Policy section found within the FBI Laboratory Division Quality Assurance Manual, and the Case Assignment, Documentation and Review section found in the DNA Analysis Unit I Quality Assurance Manual.
Although the Case Documentation Policy section is a broad protocol that applies to all units within the Laboratory, we expected to find more detailed information on case documentation procedures. For example, the section currently fails to list the required contents of case files, including the basic forms and worksheets universal to all the units. In addition, it lacks a reference to the more detailed file content requirements found in each unit's case documentation and review protocol. Further, the section should contain general guidance on notetaking methods, with a reference to any additional detailed information contained in unit-specific protocols. See Chapter Five, Section II.B.3 for additional information on notetaking.
We identified similar deficiencies in the Case Assignment, Documentation and Review section in the DNA Analysis Unit I Quality Assurance Manual. That section lacks a detailed description of the requirements for a complete case file review, and fails to identify procedures to ensure that documentation of each key item within the case file is accounted for. Instead, we found general and overly broad guidance regarding review procedures. For example, Section 22.214.171.124 requires that case file reviews include an evaluation of all data, lumigrams, interpretations, conclusions, and other supporting materials contained in the case file packet. Although five subparts are included in this section that describe the type of information that is subject to these reviews, their terms are overly broad and open to interpretation. For example, subsection 126.96.36.199.4 requires a confirmation that "all conclusions reached by the examiner are consistent with the documented data and within the limits of the discipline." In addition, the section does not contain a precise description of notetaking methods and requirements for DNAUI staff members so that they are clear on how and when they should be taking notes. We did not find this information anywhere in the protocols. See Chapter Five, Section II.B.3 for additional information on notetaking.
Finally, the Case Assignment, Documentation and Review section fails to specify the procedures that should be followed to review and confirm case evidence profiles for entry into CODIS. A review of approximately 150 case files during OIG audit work conducted in April 2002 (prior to the OIG being notified of Blake's misconduct) provided an opportunity to confirm that the DNAUI has such procedures, and our examination of their use of those procedures found that the profiles were being processed correctly. See Chapter Five, Section I.D for further detail on this case file review. However, no information on these procedures was reflected in this Case Assignment, Documentation and Review protocol.
The risks regarding the lack of specificity in these protocol sections primarily are that staff members will not understand, and therefore may not comply with, management expectations for case file documentation and review. These risks are heightened in environments such as the DNAUI's, where work is performed by teams and individual team members must rely upon the quality and thoroughness of their fellow team members' case file documentation for the completion of their own work. Therefore, without detailed guidance for case file documentation and review, the DNAUI is at risk of not having proper verification in each case that all necessary testing procedures were completed as required. For example, as the Blake matter reveals, had DNAUI protocols required the inclusion in the case file and review of GeneScan® data, Blake's supervisors readily could have detected her failure to complete the negative controls. Instead, Blake's misconduct escaped detection for more than two years. In addition, without a checklist to assist the review, a technically-qualified DNA scientist could miss an important part of the review process or fail to notice the omission of a vital document or piece of information. For example, an Examiner might be called away suddenly to assess analysis difficulties a staff member is experiencing and, upon returning to his or her case-file reviews, fail to remember what material had not been examined and thus overlook important information.
To determine the extent to which the vulnerabilities we identified were mitigated or exacerbated by staff member work practices, we interviewed DNAUI staff members and management regarding their understanding of the requirements for case documentation and review. From those interviews we determined that the interviewees' comprehension of case file documentation requirements was largely consistent, both with other interviewees as well as with the general mandates provided in the protocols. The amount of detail in the answers varied among the interviewees; however, these variations were consistent with the duties and position of the interviewees. Most interviewees knew where case file documentation requirements were addressed in the protocols, and their answers often expanded upon the information given in the protocols.
Because our interviews with staff members focused on their duties that implement the protocols under scrutiny in our review, we interviewed only Unit management and Examiners regarding case file review. We found that their answers typically were similar, with the Examiners providing the level of detail that is consistent with their job responsibilities. We also found that the interviewee responses were consistent with the written case review protocol. We again noted that the answers, particularly from the Examiners, went beyond what is specified in the protocol, revealing that, while the protocol is lacking in detail, staff members appear to have gained an understanding of what constitutes a proper case file review through other means, such as training or peer guidance. Further, the OIG team scientists detected no additional risks posed by the methods attested to by staff members or management in the interview responses.
In conclusion, our fieldwork results revealed that staff members appear to understand management expectations regarding case file documentation and review, mitigating the risk posed by the lack of detail in the protocols. However, until the protocols adequately address the information gaps we previously identified, the risk of undetected noncompliance with management expectations will remain significant.
We recommend that DNAUI management:
Forensic Standard 9.4 requires that forensic DNA laboratories monitor their analytical procedures using appropriate controls and standards. Forensic Standard 9.4.2 explains that, for PCR casework analysis, those controls and standards must include quantification standards, positive and negative amplification controls, reagent blanks, and allelic ladders. Further, Forensic Standard 9.6 requires that forensic DNA laboratories have and follow written guidelines for the interpretation of data, including (in Forensic Standard 9.6.1) verifying that all control results are within established tolerance limits. Although we found that the DNAUI analytical protocols require and describe the use of these various standards and controls, as well as provide general guidelines for their interpretation, we identified certain information that is missing from the Guidelines for Control Samples and Interpretation of Control Samples sections in the Short Tandem Repeat Analysis Protocol that should be included to ensure that staff members have a clear and consistent understanding of control sample requirements.
First, both the Guidelines for Control Samples and Interpretation of Control Samples sections lack comprehensive guidance regarding the material staff members should review when they examine control results. In addition, both sections fail to differentiate the review responsibilities of the PCR Biologists and Examiners. Such guidance could include a checklist or summary sheet that would enable reviewers to ensure the completeness of their examinations. Further, the guidance should describe the circumstances in which a control result would cause an analysis run to "fail" (meaning that those samples must be re-analyzed). The sections seem to assume that DNAUI staff members already understand fully from another source what it means to review the control results and know exactly what to do.
In addition, we questioned language in Section 10.3.3 of the Interpretation of Control Samples that allows DNAUI Examiners to use the results of an analysis to exclude a suspect in circumstances where the positive control has failed. The relevant provision states:
If FSB [the DNAUI name for a positive control sample] does not exhibit the STR typing results listed above [the correct results are shown in a table above this statement], the following steps must be taken. 1) If there appears to be an injection or electrophoretic problem, reinject the FSB with a ladder. 2) If re-injection of the FSB does not resolve the problem, and may be due to amplification issues, all samples set-up and amplified with this control will be considered inconclusive for matching purposes, but can be used for purposes of exclusion. If sufficient DNA remains of samples co-amplified with a failed control, then it is appropriate to re-amplify them.
We questioned this provision primarily because it fails to identify precisely when staff members should apply it. Limited information is provided regarding the additional steps that might be attempted before reporting results. Further, it is not clear what a report would state about the results in situations where this provision is employed. In circumstances where additional analysis has the potential to generate a dispositive result, the lack of detailed guidance in this protocol could result in the unnecessary provision of inconclusive information to law enforcement agencies.
In our view, the most significant vulnerability that results from the kinds of missing guidance on control results described above is that, similar to the previous protocol sections we have discussed that lack information, DNAUI management cannot ensure that staff members will know and comply consistently with their expectations. Also, when protocols are not comprehensive, staff members may become dismissive of them, enhancing the potential for inadvertent protocol noncompliance.
It is important to note that there does not appear to be a significant risk that testing results will be used improperly, given the requirement within the DNAUI (consistent with Forensic Standard 12.1) that all cases and analysis results be reviewed by a technically-qualified peer reviewer as well as by an administrative reviewer. In other words, if there is a misunderstanding on the part of a PCR Biologist and a supervising Examiner regarding the scrutiny that is to be applied to the control results, that misunderstanding would most likely be detected by the technical reviewer or the administrative reviewer (who, in the case of the DNAUI, is also the technical manager).
To determine whether staff members' work practices serve to mitigate these risks, we interviewed PCR Biologists, Examiners, and management regarding: 1) how the responsibility to review control results is divided between the Biologists and Examiners; 2) what information staff members look for when they review the control results; 3) under what circumstances an analysis of samples would fail because of the control results; and 4) the rationale behind the policy of using an analysis of samples to exclude a suspect even though the positive control failed. Staff members and management responses on these issues revealed a high degree of consistency. Regarding the fourth issue, DNAUI management explained during our interviews that the practice of using a DNA analysis to exclude a suspect even though the positive control failed is employed only if the sample results are good but the positive control has some malfunction that causes it to fail applicable quality requirements, no remaining DNA exists for another test, and where the results clearly indicate that the suspect does not match the evidence samples. DNAUI managers stated that they believe DNAUI staff members understand these limitations and how they should represent pertinent results in a report. They further explained that this policy is based upon the belief that it would be inappropriate not to make available results that exclude a suspected perpetrator simply because of a technical issue on the positive control. While we understand the rationale for the policy, we disagree that the parameters for the use of the policy are communicated fully. Information about the policy's usage is not represented in the protocol with the same level of clarity and comprehensiveness that was communicated to us through the interview responses.
The information we received during our interviews of staff members and management regarding the four issues above is not found in the protocols, but rather is communicated through other means, such as training and peer guidance. The high degree of agreement and shared understanding among DNAUI staff members serves to mitigate the potential risk to proper scrutiny of control results posed by the incomplete protocols.
Forensic Standard 4.1(c) requires that forensic DNA laboratories specify and document the responsibilities, authority, and interrelation of all personnel who manage, perform, or verify work affecting the validity of DNA analyses. The DNA Analysis Unit I Quality Assurance Manual includes sections entitled Organization and Management and Authority and Accountability that, ostensibly, should satisfy the requirements of Standard 4.1(c). However, upon examination, we discovered that these sections do not provide an adequate description of the interrelation of the various members in the DNAUI team structure. We also did not find a clear indication that the Unit Chief, as technical leader, has the authority to halt operations if a significant problem is detected. Nor did we find a clear delineation of the responsibilities of each team member when responding to problems or improper staff member actions.
The lack of specificity in this protocol increases the likelihood that staff members will misunderstand lines of authority and accountability, particularly with teams that have been allowed to vary their operations from one another. As explained in Chapter Five, Section II.C.1, staff members cited many examples of teams adopting their own methods when the protocols are not specific, including defining the division of authority and responsibilities of team members. Although poor communication resulting from deficiencies in this protocol would not necessarily jeopardize analysis results, it could hinder the efficient execution of laboratory duties. Further, misunderstandings could cause a delay in the response to significant problems if staff members do not understand the lines of authority.
We conducted interviews to determine if staff members and management share an understanding of roles and responsibilities within teams and with respect to problem resolution. From the responses received we determined that, while staff members and management had the same general understanding, certain roles vary according to the team Examiner's preference. All interviewees agreed that team member roles and responsibilities, including those pertaining to problem resolution, are not clearly delineated in the protocols, although a few stated that position descriptions and performance plans provided some additional guidance. In addition, Laboratory management, Unit management, and staff member interviewees noted the lack of guidance concerning how staff members and management should respond in situations involving employee misconduct, such as the discovery of Blake's actions.
Due to the consistency of the interviewee answers from both staff members and management, we concluded that a clear understanding of general team member roles and responsibilities is provided through means other than the protocols, such as training and peer guidance. However, problem resolution guidance is lacking. Consequently, we concluded that the risk posed by the lack of comprehensive guidance in the protocols on team member roles is mitigated by the DNAUI's provision of that information to staff members through other means, while the risks associated with the lack of guidance on problem resolution remain unmitigated.
In addition to protocols that fail to identify in sufficient detail the procedures that DNAUI staff members should follow when they perform DNA analysis, we also identified protocols that suffer from a related defect: the failure to specify the decision criteria staff members should employ when their duties require them to exercise discretion in the testing process. See Chart at page 80. Greater risk of abuse and error is present when procedures call upon the proper exercise of discretion.
For example, the amplification set-up process requires DNAUI staff members to complete a series of objective steps where precise amounts of various substances are added to the amplification tubes. In contrast, Serologists are faced with a complex decision when, after completing preliminary testing and obtaining a negative result, they must determine what step to take next. Serologists must decide whether the presumptive test results should be followed or disregarded. In making that decision, they must consider, for example, whether other factors may have influenced the results, and whether a negative result automatically means that there is no detectable DNA on the evidence item. These questions and others like them must be answered and judgment applied to decide how the testing process should proceed. If staff members are not equipped with sufficient guidance to answer these questions, they could prematurely halt the serology process when a probative DNA result might otherwise have been obtained.
Consequently, in our view, the risks inherent in such decision-making should be offset by the provision to staff members of adequate evaluative tools and guidance to ensure that they are thorough and consistent in their consideration of options at each "crossroad" in the DNA testing process. We failed to find this type of guidance in five sections that address serology testing and electrophoresis data interpretation: the Procedures for the Preparation of Dried Bloodstains from Coagulated Whole Blood and from Anticoagulated Whole Blood; Procedures for the Extraction of Suspected Semen Stains: Questioned Stain Extraction Procedure from the Procedures for the Serological Identification of Biological Substances on Evidentiary Materials;96 and the GeneScan Analysis and Interpretation of Control Samples sections from the Short Tandem Repeat Analysis Protocol.
To understand how staff members exercise discretion under these protocols, we interviewed Serologists and Unit management regarding: 1) the circumstances in which they do or do not perform each of the serology tests; and 2) the procedures they follow if they obtain a negative result (indicating that no DNA was detected). Serologist responses were generally very similar to each other and to the responses provided by management. For example, Serologists provided similar descriptions of the various factors that they consider when deciding how to process an evidence item - factors that are not explained in the protocols, and Serologists and managers (including the Examiners) provided comparable descriptions of the Examiners' role in serology decisions.
We also interviewed Examiners regarding the discretion they have when interpreting electrophoresis results. Specifically, we asked them to describe: 1) their responsibilities in the DNA analysis process; 2) what they specifically look for when reviewing GeneScan® data and control results; and 3) the circumstances in which they would decide to fail an analysis run based upon the control results. Their responses were generally consistent with one another and conveyed a level of detail not found in the protocols.
In light of the foregoing, we concluded that the work practices of DNAUI staff members serve to mitigate the risk posed by the lack of adequate evaluative tools and guidance in the protocols.
Forensic Standard 11.1 requires that forensic DNA laboratories follow written procedures for taking and maintaining case notes to support the conclusions drawn in laboratory reports. Further, Forensic Standard 11.1.1 requires that forensic DNA laboratories maintain, in a case record, all documentation generated by examiners related to case analyses. The DNAUI's protocols refer to these requirements and address, in a limited way, the documentation that should be present in a case file. However, we did not find comprehensive guidance on notetaking methods in three sections of the protocols where manual notetaking is identified as a significant part of staff member responsibilities. In addition, the sections lacked an explicit requirement for staff members to complete their notes contemporaneously with their work. The three sections are: 1) the Case Documentation Policy within the FBI Laboratory Division Quality Assurance Manual; 2) the Evidence Control section within the DNA Analysis Unit I Quality Assurance Manual; and 3) the Procedures for the Examination of Evidence within the FBI Laboratory Division Caseworking Procedures Manual.
The team structure in DNAUI makes it especially important that all staff members have a comprehensive and consistent understanding of how to record information as they complete their work. The case file documentation created by the Serologists and PCR Biologists serves a crucial role in communicating to the Examiner the results of the DNA analyses they have performed. The Examiner draws conclusions from this work and often testifies in court based in part on the documentation contained in the case file.
In addition, contemporaneous documentation is important to ensure that the case file accurately reflects the work performed on each evidence item that is tested. If staff members are allowed to delay recording observations and test results until after they have examined all the items for a case or have completed all of their work for the day, their documentation may not be fully accurate. Also, staff members may be unduly influenced by protocol requirements when relying on memory, and document what they know should have occurred when their recollection is vague. Such a situation could lead to difficulties when trouble-shooting testing problems. For example, a weak and unusable testing result might be caused by a sample with low quantities of DNA or by a technical problem in the analysis process. An Examiner reviewing such results may not be able to pinpoint how to generate a better outcome if he or she is provided with an incomplete record from a staff member who is documenting from memory.
Because we could not find any requirement for contemporaneous documentation or comprehensive guidance on how staff members should take notes, we asked staff members and management what they believe the requirements are on this subject. We interviewed ten DNAUI staff and three members of DNAUI management.
DNAUI management cited a specific section of the protocols as the source of guidance on notetaking. However, the five staff members who cited the same section stated that the guidance in the protocols is very general and does not fully address the subject. Two staff members stated that notetaking is addressed in the protocols but did not cite a specific section. Three of the ten staff members stated that they did not think the protocols addressed the subject at all. Six of the ten staff members stated that documentation methods are learned during training.
Unit management stated that notes are taken contemporaneously as testing is performed. Staff members generally indicated that they take handwritten notes as they work and then transcribe the notes into the computer at a later time (typically on the day they are written). Further, staff members did not always give the same answers regarding the time when they take notes: one Serologist stated that staff members will process multiple items and then type up notes; another Serologist explained that staff members create notes immediately after processing each item; and a PCR Biologist stated that staff members would not "typically" put off transcribing their notes until the following day (indicating that there might be times when that does occur).
We also noted from interview responses that for those staff members who are taking contemporaneous notes on their computer during their work, there are no policies in place that require a protective covering (such as plastic wrap) to be used and changed at appropriate intervals to prevent contamination or cross-transfer as the staff person moves from handling an evidence item to typing on the computer keyboard. This issue is of greater concern now that the DNAUI has moved to its new facility. During our tour of the new Laboratory, the Unit Chief explained that the serology and PCR Biologist areas are equipped with a computer for each workstation to permit the immediate transfer of examination and analysis notes into the computer. He further stated that with the addition of these workstations staff members are expected to complete their notes contemporaneously with their work.
Given the disparities in staff member answers, we concluded that Laboratory management has not clearly articulated standards to govern notetaking, including handwritten notes that are later transferred to the computer. Further, it is evident from the Unit Chief's responses that while contemporaneous documentation previously was a goal, it now is an expectation in the new facility. No written requirement has been published for staff members, however, setting forth comprehensive guidance on notetaking methods.
We recommend that Laboratory and DNAUI management:
Our review of the DNAUI's protocols revealed that, at the time of our examination, four document sections had not yet been updated to reflect a new policy that had been implemented in the Unit as a consequence of Blake's misconduct. The new policy requires Examiners to review GeneScan® data for all samples that show no DNA peaks on the Genotyper® data, and should be reflected in: 1) the Case Assignment, Documentation and Review section of the DNA Analysis Unit I Quality Assurance Manual, and 2) the Amplification, STR Typing: Setting up a Run, and GeneScan Analysis sections of the Short Tandem Repeat Analysis Protocol. These sections of the protocols reflect the previous requirement for review of only the Genotyper® data by the Examiners, a practice that allowed Blake's actions to escape detection.
The risk posed by having outdated protocols is that some staff members might not be aware of new requirements and inadvertently rely upon standards that have been superseded. Unit management stated that their reason for delaying the updating of the protocol is two-fold: 1) they anticipated that the OIG vulnerability assessment would result in additional changes to their protocols, and 2) because the protocol revision process is lengthy and time consuming, they wanted to wait and make all the revisions at one time.
We concluded from our fieldwork interviews that staff members involved in data review are aware of the new GeneScan® data review policy and understand that adherence to its terms is required. Further, in discussions on this subject with Unit management, they informed us that the policy change had been executed through an "electronic communication" in April 2002, a written format that is used to notify staff members of protocol changes that are implemented between formal revisions. OIG staff members were provided a copy of the electronic communication and confirmed that it describes the new data review requirements.
In conclusion, the OIG team found 31 vulnerable sections in the 5 key manuals that govern the work of the DNAUI. In response to these findings, we examined the work practices of DNAUI staff members and management to determine whether those practices mitigated the vulnerabilities detected in the protocols. We found that the vulnerability risks posed by weaknesses in the protocols were mitigated to some extent by the work habits of the Unit's employees.
Although we did not conduct case file reviews throughout DNAUI, we did not identify through our interviews and fieldwork any instances of misconduct of the sort committed by Jacqueline Blake. Staff member and management responses conveyed a level of detail and consistency that reassured us that DNAUI employees are obtaining a clear understanding of the Unit's requirements through means other than the protocols, such as training or peer guidance. However, staff practice did not mitigate all of the protocol weaknesses, and interviewee responses confirmed the remaining protocol vulnerabilities that we detected. Below we summarize whether the vulnerabilities identified in the document sections were mitigated or confirmed in accordance with the practice information provided by DNAUI staff members and management.
|Protocol Vulnerability Status
[Image not available electronically]
Our recommendations to DNAUI management detailing the remedial actions that we believe are necessary to correct the identified protocol vulnerabilities are set forth in Section C below. We believe that until the DNAUI completes the actions prescribed in those recommendations, the DNAUI needlessly will remain subject to an increased risk of employee error and inadvertent protocol noncompliance.
While we did not conduct an assessment of DNAUII's protocols, we believe that our conclusions and recommendations regarding the DNAUI would also benefit the DNAUII in the following ways: 1) the changes made to the DNAUI's Short Tandem Repeat Protocol will improve STR work performed by or for DNAUII personnel; and 2) case file documentation and evidence storage and handling in the DNAUII will be improved by changes made to Laboratory-wide protocols, such as the FBI Laboratory Division Quality Assurance Manual and the FBI Laboratory Division Caseworking Procedures Manual.
However, based upon the extent of the vulnerabilities identified within DNAUI's protocols, we believe that the risk exists that DNAUII's protocols contain vulnerabilities of a similar nature, vulnerabilities that will not be remedied completely by the improvements made as a result of our preceding recommendations. Therefore, we make the following recommendation.
As explained earlier, the second type of vulnerability we identified during our review - practice vulnerabilities - was detected as we completed our fieldwork on the protocol vulnerabilities. These weaknesses result from the manner in which the DNAUI implements its protocols, leaving the Unit more susceptible to undetected inadvertent or willful protocol noncompliance. Our analysis is based largely upon the perceptions of DNAUI staff members, since each of the vulnerabilities was identified by multiple staff members and/or management during our interviews with them. Of special concern are the following: 1) variations in team operations; 2) the "oral tradition" of DNAUI guidelines and training; and 3) communication and operational inefficiencies. We describe each below.
During our interviews with DNAUI staff members, we regularly received comments regarding the degree of variation that exists in the operations of the DNAUI teams. For example, a PCR Biologist explained that various aspects of the work performed by staff members are not standardized. The Biologist further explained that while everyone follows the protocols (e.g., how samples are numbered, how to check in cases), the protocols are general and leave a lot of room for flexibility. The Biologist stated that, consequently, the teams often function very differently. The Biologist stated that this can be beneficial for someone who wants to devise and follow his own procedures, but when issues arise in the Unit, those variances also can pose a problem.
Another PCR Biologist provided similar comments, stating that even though the protocols are followed very strictly, there is a broad range of interpretative leeway for anything that the protocols do not specify. According to this Biologist, the practices of Biologists on the DNAUI teams vary, which can pose a problem if a Biologist ever has to do work for another team, since the supervising Examiner may not be comfortable with the Biologist's methods. In addition, the Biologist explained that since the Examiners testify based upon how their team operates, their testimony might not be precise if a Biologist from another team has provided assistance and employed another team's methods. The Biologist stated that there needs to be more uniformity between teams. The Biologist also explained that Examiners are probably unaware of the extent of these variations.
A Serologist commented that, due to the nature of the "oral tradition" of training (covered under the next section), many staff members have developed their own style, methods, and preferences for how they perform their work. Another Serologist referred to the variations in team operations and explained that they make it hard to train new staff members and to learn what is required.
Multiple staff members commented about the flexibilities afforded to teams due to the lack of detailed guidance on specific topics in the written procedures and protocols. Examples cited by staff members were: 1) how forms are filled out; 2) how case file documentation is transferred to the Examiner; 3) how responsibilities are divided between the PCR Biologist and the Examiner; 4) how much Examiners interact with their team members on decisions; 5) the work flow of the serology procedures; 6) notetaking; and 7) inter-Unit transfer of evidence.
One member of DNAUI management acknowledged that it would be advisable to standardize guidelines for team operations. He said that while he thought that the lack of uniformity had not yet caused a problem for the DNAUI, it could in the future. He stated, however, that there would probably be resistance to standardizing team operations. He stated that even though there may be better ways to do things, staff members are accustomed to their own procedures and likely would resist standardization because they would not want to sacrifice their autonomy.
These interview comments highlight the need to ensure that protocols are comprehensive and address all aspects of the Unit's operations. See Chapter Five, Section II.B.1 (describing protocols that lack sufficient detail). As many interviewees mentioned, practice variations exist because the written guidance is silent on many subjects.
During our interviews with DNAUI staff members and management, we were informed that the Unit's training curriculum consists largely of individual discussions with a mentor and presentations given by various experienced staff members. Laboratory management was unable to furnish us with a single, comprehensive curriculum, though we were provided training program manuals for Biologists and Examiners, and PowerPoint slides used during training presentations. Also, we saw evidence in the training records that these presentations often relied upon other training materials, such as handouts and checklists. However, none of these materials has been collected and incorporated into a larger training program with a defined curriculum.
According to DNAUI staff members, this diffuse approach to training is founded upon the Unit's "oral tradition," since verbal instruction is the primary means of conveying training information. During interviews regarding Unit operations, several staff members explained their perspective on why the oral tradition, as a training philosophy, increases the Unit's vulnerability to inadvertent and willful noncompliance with applicable protocols.
For example, two staff members cited training as a key weakness in the Unit. They explained that when the Unit was created, training was better because everyone was "starting fresh and learning the same thing." However, over time people began to teach their own preferences as the only way to complete the Unit's work. In the view of these two individuals, the result is a staff that does not have an understanding of the "big picture" and that performs work in noticeably different ways. One of the staff members, a Serologist, explained that the weaknesses in training are exacerbated by the fact that, because many staff members have biology degrees, much is assumed about their basic understanding of laboratory operations, which may be unwarranted. The other staff member, a PCR Biologist, stated that some training improvements were being implemented under the current Unit leadership, and cited as examples the development of a written exam to assess candidate qualifications and increased stringency in the qualification requirements.
An Examiner who has served as the training coordinator for the DNAUI acknowledged that the oral tradition concerns him since it fosters "protocol drift" - something staff members described as the use of personalized testing procedures that deviate from and/or add to the letter of the protocols, though without jeopardizing the integrity of the testing results.97 The Examiner identified several checks and balances on the training process that he felt counteracted the risks associated with an oral tradition. These included the constant involvement during training of a trainer/mentor and, following training, the oversight of an Examiner during casework duties who would notice whether something had been overlooked or improperly communicated during training. He added that key policies are reiterated throughout training, and that there are multiple ways to check to see if a person understands his or her duties. Yet, the Examiner acknowledged that in spite of these "checks and balances," protocol drift still occurs.
We note two considerations regarding the Examiner's comments. First, each of the checks on staff member behavior he identified requires another DNAUI staff member, rather than a written document in conjunction with staff guidance, to be the source of information and direction for the Unit's employees. Second, based upon the division of duties within DNAUI teams, particularly the division of duties and roles described by staff members, we are not convinced that the Examiners are sufficiently involved in the day-to-day activities of their team members to serve as one of the "checks and balances." This was illustrated by the fact that Blake's Examiner was unaware of her misconduct until Blake's colleague noticed her omissions.
In addition, when the distinction between staff member preference and protocols is unclear, trainees are left to draw their own conclusions regarding proper testing methods. In our view, such an environment leaves the Unit vulnerable to inadvertent noncompliance with Unit requirements, since staff members may choose to alter their methods in ways that unwittingly contradict Unit requirements.
A DNAUI Serologist, who also is involved in training activities, stated that management's failure to communicate the reasons behind the Unit's work requirements is a weakness and needs to be addressed during training. A PCR Biologist reiterated the point, stating that training has been a weakness in the Unit and that part of the problem is the failure to teach staff members why they are asked to do their work a certain way - the importance and history behind what they are doing - not just the actual procedural steps to perform.
In conclusion, moving away from training that is based on the DNAUI's "oral tradition" will help to ensure that the other recommendations we present in this report reduce the DNAUI's vulnerability to undetected protocol non-compliance. A well-documented and comprehensive training curriculum should reinforce application of the revisions we have outlined to the Unit's protocols.
We determined from our review of DNAUI operations and our analysis of interview responses that there are various communication and operational inefficiencies within the DNAUI.
During our interviews with DNAUI staff members and management, we inquired about the way that information is communicated within the Unit. This issue was of particular interest to us because of the importance of communication to the proper implementation and improvement of DNAUI protocols. We asked interviewees the following questions:
We observed from interview responses that although members of upper management think that communication within the Unit, and between the Unit and Laboratory management, is functioning well, several staff members do not feel that they are kept informed about operational information and believe that communications are at times dysfunctional. Further, several comments we received indicate that some staff members do not believe that Laboratory management actively solicits and considers their input on issues that affect their work.
Management and staff members identified similar methods that are used to keep staff members updated on operational and protocol-related information, including e-mail and Unit or program meetings. However, it was clear from staff member responses that these methods are not consistently effective. Staff members explained that the dissemination of information, including protocol-related information, is erratic. Examiners made reference to this problem and said that some Examiners are better than others in passing along information, a point that also was noted by both a PCR Biologist and Serologist. One Examiner stated that changes in technical operations that affect the quality of work are always passed on to staff members, but that Examiners may not pass on administrative information. However, we question whether Examiners are as consistent as this Examiner claims in conveying operational information, given the lack of a requirement that Examiners disseminate protocol-related information promptly and accurately to those under their supervision.
In addition, one Serologist added that even when decisions and other important information are communicated to staff members, the rationale behind them often is not explained, leaving staff members unclear on the goal that management is trying to achieve.98 One Examiner said that he felt that information does not flow effectively up the hierarchy either, and identified a situation where the PCR Biologists decided to initiate a technical change in Unit procedures and failed to ensure that all the Examiners were made aware of it.
The types of communication weaknesses mentioned by staff members pose a risk to the efficiency and effectiveness of the Unit's operations and should be addressed. Of particular concern is the perception of Laboratory and Unit management that communication lines are functioning well, while many staff members describe a different perspective that questions whether they are being kept well informed of procedural changes and whether management properly considers technical input in operational matters.
During our review of protocol vulnerabilities, we observed many DNAUI operations that could be made more efficient through use of a Laboratory Information Management System (LIMS). A LIMS is a computerized system of databases that track, organize, and link the information that must be maintained to document the receipt, handling, and disposition of each case and evidence item. A LIMS allows a laboratory to:
Because of these capabilities, we believe such a system would strengthen the Unit's internal controls and allow DNAUI staff members to be more efficient in their duties. As revealed in their interview responses, Laboratory and Unit management, as well as staff members, share this assessment and believe that acquisition of a LIMS is a priority for the Laboratory. We were informed that Laboratory management began to lay the groundwork for the implementation of a LIMS in 2002. During our March 2003 fieldwork, we met with personnel involved in LIMS development at the Laboratory and reviewed available documentation concerning the progress of implementation. From this information, we determined that the Laboratory had completed the bid process for a contractor who would design and implement the LIMS, and that all Unit Chiefs had been involved in determining the capabilities that the LIMS would need in order to suit the activities of their Unit.99 In October 2003, we were informed that the LIMS had been procured and that the system should be on-line in December 2003 and functioning fully in the Laboratory by approximately March 2004. The Laboratory Director told us in March 2004 that he expected the LIMS to be fully operational this fiscal year, and that the Laboratory was waiting on security clearances for the staff of the LIMS contractor before commencing implementation of the system.
We recommend that the Laboratory's LIMS work remain one of its top priorities. Specifically, successful implementation requires that all appropriate personnel have ready access to the system, have received adequate training, and are afforded the resources needed to convert their current methods and operations to those that will maximize the capabilities of the LIMS. Further, we recommend that Laboratory management continue to set aside sufficient resources for the LIMS to ensure that it keeps pace with the changes and developments in technology that invariably will occur over time.