The Office of the Deputy Attorney General (ODAG) provided my office the following two comments on the subject Office of the Inspector General (OIG) draft report. I am forwarding them to you for your consideration.

If you have any questions concerning this information you may contact me on (202) 514-0469.

Attachments

November 4, 2002

Terry Zobeck, Ph.D., ONDCP
314 Missouri Avenue
Herndon, VA 20170

Dear Mr. Zobeck

The Substance Abuse and Mental Health Services Administration (SAMHSA), DHHS, recognizes that the Office of National Drug Control Policy (ONDCP) is an important consumer of data produced by the Office of Applied Studies (OAS). SAMHSA is also aware that ONDCF uses SAMHSA data to assess the progress that has been made in attaining the goals incorporated in the National Drug Control Strategy. Several important goals such as reducing youth and adult drug use are measured with the data from the National Survey on Drug Use and Health (NSDUH), formerly the National Household Survey on Drug Abuse (NHSDA). It is, therefore, important that ONDCP be kept informed of technical changes in the survey and decisions that might affect the ability to generate treads based on those data. We have already advised you during a recent meeting with OMB of our concerns about comparing estimates based on data from the 2002 NSDUH with estimates based on data from surveys conducted prior to 2002. This memorandum is a response to your request for written comments on the best approach for dealing with this issue.

Three changes were made to the NSDUH at the outset of the 2002 survey. First, respondents were offered $30 to encourage them to participate. This incentive was introduced because of a decline in response rates. Second, the survey introduced new field procedures and training to increase adherence to survey protocol. Methodological studies have demonstrated that interviewers unilaterally alter field procedures, a practice that can affect the way people answer survey questions. Finally, SAMHSA was ordered by a former Secretary, DHHS, to change the name of the survey, a decision that required two years to implement. All of these changes were designed to increase the quality of the data and the accuracy of the estimates.

An experiment was conducted in 2001 to assess the probable impact of incentives on response rates and data content and analyses were employed to examine the impact of the changes in field procedures and training of field interviewers on the quality of the data. However. it was not possible to examine the consequences of introducing these changes simultaneously. Nor was it possible to study how the change in the name of the survey would either independently or in conjunction with the other modifications effect the estimates.

In analyzing data from the 2001 NSDUH, OAS found that the incentive and the change in field procedures had some effect on the estimates. Extensive studies of this matter showed these effects were small. However, with these two methodological changes in place questions were raised about how OAS could adjust estimates based on the 2002 NSDUH data to make them comparable to estimates produced from earlier surveys. The problem is further complicated by the change in the name of the survey, which also took effect in 2002. To address the problem of making comparisons, OAS met with several nationally recognized experts in the area of survey methodology in September, 2002.

A summary of this meeting was provided to you under separate cover. Briefly, the panel noted that there is no simple way to quantify the impact of the changes implemented in the 2002 NSDUH. There is no question that the changes have improved response rates, probably improved the quality of the data, and somewhat lowered the costs of the survey. However, it is also clear that the quantitative adjustments necessary to compare the results of the 2002 survey with those of earlier surveys cannot be developed with the information now available. The impact of these changes on survey results can only be measured by conducting a very large, complicated, and costly experiment, for which there is neither time nor money.

Therefore, OAS is recommending that users of the NSDUH data not try to compare estimates based on the 2002 NSDUH with estimates produced prior to 2002. OAS continues to assess the preliminary data from 2002 and the possible methodological artifacts to determine if there are NSDUH measures that might not be affected by the changes in the survey. However, based on the analyses completed at this time, it appears the changes have affected the estimates of the prevalence of illicit drug use; and comparisons across years for these particular estimates will not be valid.

SAMHSA has informed DHHS of this decision and will document the analyses and deliberations leading to it in the forthcoming 2002 NSDUH report. SAMHSA does not at this time anticipate further changes of the NSDUH in the near future.