Effectiveness of the Office for Victims of Crime Tribal Victim Assistance Program

Audit Report 06-08
February 2006
Office of the Inspector General

Appendix V
Analysis and Summary of Actions
Necessary to Close the Report


The OIG has identified several issues in the OVC response to our draft report (see Appendix IV) that we believe should be addressed. As a result, we are providing the following comments on the OVC response to the draft report.

In Appendix IV, page 87, the OVC provided the following general comment:

The “number of victims served” appears to have been given considerable weight in the ultimate determination of a program’s effectiveness (i.e., neither tribe that received a “no” on this measurement was considered effective).

The OIG disagrees with the OVC assertion that the number of victims served was given considerable weight in the ultimate determination of program effectiveness. The number of victims served was only one of eight factors considered in determining the effectiveness of the individual grant programs. In fact, the OVC response does not address other measures the OIG considered in making this assessment. For example, although both tribal victim assistance programs established by the Oglala Sioux Tribe and the Mississippi Band of Choctaw Indians could not demonstrate an increase in the number of victims served, as stated in Findings III and IV of this report, the OIG also found that these tribes:

  • were not considered effective or were considered only partially effective by tribal collaborating agencies;

  • had not established a plan to sustain its victim assistance program when the victim assistance grant funding expired; and

  • did not accurately report performance information in its progress reports.

Additionally, the Oglala Sioux Tribe’s program was never fully implemented due to frequent changes in program management. Further, the victim assistance program implemented by the Mississippi Band of Choctaw Indians only partially achieved the grant objectives outlined for its victim assistance grant and focused solely on victims on non-major domestic crimes. We considered the number of victims served along with other measures to assess program effectiveness.

Also, in Appendix IV, page 87, the OVC provided the following general comment:

Data, such as “number of victims served” gathered for internal reporting purposes is not necessarily appropriate for use as a performance measure. Although this seems to be an easy and appropriate measure for a program aimed at assisting crime victims, the outside observer is tempted to use an increase in the number of victims served as a measure of effectiveness. . . The number of victims served depends on many factors outside the control of the program in question. In fact, to the extent the law enforcement goal of reducing crime is achieved it seems problematic to use an increase in the number of victims served as a performance measure.

The OIG recognizes the challenges faced by the OVC in developing performance measures that can be used to determine the effectiveness of its tribal victim assistance program. However, the OIG disagrees with the OVC assertion that the number of victims is not appropriate for use as a performance measure. As stated on page 6 of this report, the OVC established the performance measures for its tribal victim assistance program, and one of the original measures established by the OVC was the number of victims served.36 The OIG also recognizes that performance measures can be influenced by factors outside the control of the program. As stated previously, the OIG did not consider the number of victims served to be the only factor in determining whether a program was effective. However, in our judgment, the number of victims served is an indicator of program effectiveness that should be utilized by the OVC. In addition, we believe the OVC should follow up with tribal grantees demonstrating a reduction in the number of victims served to determine whether it is related to poor program performance or some factor outside the control of the grantee program.

Finally, in Appendix IV, page 87, the OVC provided the following general comment:

. . . the Report notes that OVC transferred approximately $450,000 to the National Institute of Justice (NIJ) for an assessment of whether TVA grantee programs could be evaluated, the Report appears to question the significance of evaluating two grants valued under $200,000. TVA grantees operate in a complex environment, therefore, it is not surprising that NIJ limited the cost of the study to two tribes with an expectation that the lessons learned would be useful for other programs.

As stated on page 15 of this report, the OIG expressed concern over expending $425,200 to evaluate grant programs with funding totaling less than $200,000 because it may not be the most effective use of limited victim assistance resources. Our concern was not that NIJ limited the cost of the study to two tribes. Rather our concern was that it may not be cost effective to spend more than twice the amount of funding awarded to evaluate a limited number of grantees, especially since NIJ officials stated that the evaluations cannot be used to determine the effectiveness of the OVC tribal victim assistance program as a whole.

In Appendix IV, pages 88 through 90, the OVC provided responses to the OIG recommendations, which we analyze in turn:

  1. Establish long-term and annual performance goals for its tribal victim assistance program.

    2. Unresolved. This recommendation can be resolved when the OVC provides a corrective action plan that addresses the recommendation to establish long-term and annual performance goals for its tribal victim assistance program. In Appendix IV, page 88, the OVC provided the following response to recommendation 1.

    In compliance with the Government Performance and Results Act, OVC requires TVA grantees collect and report data on the results of their individual programs. OVC will continue to work with grantees to establish goals that are specific to their community’s needs and the long-term sustainability of victim services.

    The OVC response to the recommendation states that the OVC will continue to work with grantees to establish goals that are specific to the community’s needs and the long-term sustainability of victim services; however, the intent of the recommendation was to ensure that the OVC establish long-term and annual performance goals for the program as a whole, not for individual grantees. As a result, the OVC’s response does not adequately address the recommendation.

    As stated on pages 13 and 14 of our report, we evaluated the OVC tribal victim assistance program structure and design to determine whether the programs incorporate adequate strategic planning, which is essential in evaluating program effectiveness. We concluded that the OVC tribal victim assistance program structure and design does not incorporate any strategic planning. Specifically,

    • Programs should have specific long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the programs. We found that no long-term performance goals were established for the OVC tribal victim assistance program.

    • Programs should have annual performance goals that demonstrate progress toward achieving long-term goals. Annual performance goals enable program management to detect deficiencies in program performance and develop corrective actions in a timely manner. We found that no annual performance goals were established for the OVC tribal victim assistance program.

    Therefore, based on the findings detailed above, the OVC response does not adequately address the establishment of long-term and annual performance goals for the tribal victim assistance program as a whole.

  2. Ensure that resource-allocation decisions reflect program effectiveness.

    3. Unresolved. This recommendation can be resolved when the OVC provides a corrective action plan that addresses the recommendation to ensure that resource-allocation decisions reflect program effectiveness. In Appendix IV, page 88, the OVC provided the following response to recommendation 2.

    The TVA grants are made based on a three-year proposal. Although the initial award is based on need, the accomplishments of TVA grantees toward long-term goals are considered when making resource-allocation decisions. . .

    The OVC response to the recommendation states that although the initial award is based on need, the accomplishments of TVA grantees toward long‑term goals are considered when making resource-allocation decisions. However, the intent of the recommendation was to address the fact that the OVC was not required to provide performance information with its budget requests for the TVA program; as a result, program funding decisions were not tied to program effectiveness. Therefore, the OVC’s response does not adequately address the recommendation.

    As stated on page 14 of our report, we evaluated the OVC tribal victim assistance program structure and design to determine whether the programs incorporate adequate strategic planning, which is essential in evaluating program effectiveness. We found that the OVC tribal victim assistance program structure and design does not incorporate any strategic planning. Specifically,

    • Budget requests should be tied to the accomplishment of annual and long‑term performance goals. It is also essential that program performance and budget planning processes are integrated so that resource-allocation decisions reflect program effectiveness. We found that the OVC was not required by OJP to provide performance information with budget requests for its tribal victim assistance program. Additionally, since annual and long-term performance goals were not established, funding could not be tied to program effectiveness.

    Therefore, based on the findings detailed above, the OVC response does not adequately address the recommendation to ensure resource‑allocation decisions for the TVA program as a whole are based on program effectiveness, not funding for individual grantees.

  3. Provide tribal grantees with definitions of terms used for the required performance measures and guidance on tabulating the performance information reported.

    4. Resolved. This recommendation can be closed when we receive documentation supporting that the OVC has provided grantees with definitions of terms used for the required performance measures and guidance on tabulating the performance information reported. To adequately address this recommendation the OVC needs to develop written policies and procedures to provide grantees with definitions of terms used for the required performance measures and guidance on tabulating the performance information reported.

  4. Establish a standardized progress report that captures required performance measure information.

    5. Resolved. This recommendation can be closed when we receive documentation supporting that the OVC has developed and implemented procedures to ensure that consistent and accurate performance measure information is included in the standard progress report.

  5. Ensure that progress reports include required performance measure data.

    6. Resolved. This recommendation can be closed when we receive documentation supporting that the OVC coordinated with the TVA training and technical assistance provider to increase efforts to ensure that progress reports include required performance measure data. To adequately address this recommendation the OVC needs to develop written policies and procedures to follow up with grantees who failed to include all required performance measure data in their progress reports.

  6. Summarize the performance information reported by tribal grantees to report on the effectiveness of its tribal victim assistance program as a whole.

    7. Resolved. This recommendation can be closed when we receive documentation supporting that the OVC has developed and implemented a formalized annual reporting process for analyzing the performance information reported by tribal grantees in order to report on the effectiveness of its tribal victim assistance program.

  7. Utilize the performance information reported by tribal grantees to evaluate the effectiveness of individual grantee tribal victim assistance programs, and to follow up with tribal grantees demonstrating poor program performance.

    8. Resolved. This recommendation can be closed when we receive documentation supporting that the OVC developed and implemented procedures to improve the programmatic monitoring process to ensure that performance information reported by tribal grantees is utilized to evaluate the effectiveness of individual grantee tribal victim assistance programs, and to follow up with tribal grantees demonstrating poor performance.


Footnotes

  1. In the 2005 TVA solicitation, the performance measures were changed to: (1) percent of increase in the number of victim services provided, (2) percent of increase in the number of victim services training workshops provided, and (3) percent of increase in the number of victim compensation claims submitted.


« Previous Table of Contents Next »