Return to the USDOJ/OIG Home Page
Return to the Table of Contents

The Immigration and Naturalization Service's
Automated I-94 System

Report Number 01-18
August 6, 2001


FINDINGS AND RECOMMENDATIONS

  1. AUTOMATED I-94 SYSTEM PERFORMANCE NOT MEASURED

    Although the INS has spent $31.2 million to develop and deploy the Automated I-94 System, it has not developed performance measures for the system. The INS did not complete many of the steps necessary to measure the performance of the system, including: (1) the conversion of the project's intended purpose into measurable goals, (2) the collection of baseline information, and (3) a cost- benefit analysis. In addition, the INS's project plan did not include cost and schedule data. As a result, the INS does not have evidence that the system meets its intended goals, and INS management does not have information necessary to determine whether the project is progressing as expected. In our judgment, these deficiencies are symptomatic of the broad-based weaknesses previously identified by the OIG and the GAO.

    Performance Measures Not Established

    According to the INS SDLC, performance measurement is an essential element in developing effective information technology systems. For each goal in its strategic plan, the INS established a set of performance measures. These measures enable the INS to assess how effective each of its projects is in improving operations. For each project plan, as part of the economic analysis, the INS estimates the performance levels it expects to reach as a result of the planned improvements. As a project's improvements are implemented, actual results are compared with the estimated gains to determine the success of the effort. The project plan is the crucial document of the information system life cycle. According to the SDLC, the project plan is subject to approval by the INS program management and should be produced in the planning phase. The plan should also be updated, expanded, and refined continually throughout the life cycle.

    The INS Automated I-94 System project plan provided to us did not include any performance measures. The 1998 plan, which included both the systems for air and land ports of entry, listed the project's goals as reducing the cost of collecting I-94 data, expediting the entry of I-94 data to the NIIS database, improving the quality of the data in the NIIS database, and simplifying the I-94 data collection process for passengers, airlines, and the INS. However, the plan did not define the basis for determining whether or not the system met the stated goals. Information on performance is necessary for in-depth reviews of a project to ensure that the project delivers the intended benefits. If the project does not deliver the intended benefits, performance data allows management to correct problems before significant money has been spent.

    The INS commissioned two studies to assess the performance of the Automated I-94 System. The first, conducted in 1997, was to study whether the system met its major objectives to:

    Based on data collected over six days, the study concluded that it "appeared" that the Automated I-94 System improved the accuracy of data for US Airways flight 0015 from Munich, Germany to Philadelphia. Based on data collected from flight 0015 over 110 days, the study concluded that it "appeared" that the automated system improved the timeliness of arrival and departure data. The study also concluded that the system reduced data entry costs and that there was no "perceptible difference" in the processing times of the manual system and the automated system.

    The study noted that US Airways flight 0011 from Madrid, Spain to Philadelphia was very similar to flight 0015 in passenger composition and the level of difficulty. The study called for the Madrid flight to be used as a control flight to compare and quantify the differences between the manual and automated systems. However, we found no evidence that such a follow-up study was conducted.

    The second study, conducted in March 2000, collected data on the inspection time for both the automated and manual systems. Data on the automated process was collected for 90 minutes. The average inspection took 100 seconds. Data on the manual system was collected over 67 minutes, taking an average of 60 seconds per inspection.

    In our judgment, neither study provides reliable data on the performance of the Automated I-94 System. In the first study, the analysis of data accuracy and timeliness does not include any baseline data. The first study also does not include any cost data on the automated process, noting that cost for processing the automated "Arrival cards is yet to be determined," but concludes that "it should be equal to or less than the current Departure card cost." The first study did not collect any data on the amount of time required for either the automated or the manual inspection process. The study concluded that the difference between the two processes was imperceptible based solely on observation. After reviewing the study, we cannot make the same conclusions.

    We have the following concerns about the second study. The sample size was not the same for the automated and manual inspection processes. The impact of a referral to secondary inspection was not considered. The difference in the number of people inspected during a given inspection was not held constant.

    Stated Goals Not Converted to Measurable Objectives

    For each project plan, the INS estimates the performance levels it expects to reach as a result of the planned improvements. The project plan is the crucial document of the information system life cycle. It is first produced in the planning phase and should be updated, expanded, and refined continually throughout the life cycle. The project plan should cover project scheduling, staffing, resources, adjustments to the life-cycle structure, selection of tools and methodologies, identification of applicable reviews and approvals, configuration management methods, and other related topics.

    The INS did not translate the goals stated in its project plan to measurable objectives. For example, one stated goal was to simplify the data collection process for air passengers. The project plan provides no explanation of how the INS will know whether or not it achieved this goal. Because the INS has not established measurable objectives to determine whether the system is achieving its goals, neither the INS nor we could measure the performance of the system. According to the INS's FY 2000 Annual Performance Plan, evaluating programs is vital to identifying and overcoming weaknesses before the programs are more widely implemented. The plan listed both automated exit/entry control and port-of-entry facilitation as areas where evaluation was critical.

    Baseline Information Not Gathered

    The project plan did not include baseline information on the then current level of performance in the areas included as goals: cost reduction, data timeliness, data quality, and process simplification. Measuring progress is not possible without baseline information. Establishing a baseline should be one of the initial tasks in managing the performance of an information technology project. Without baseline information, the INS will not be able to determine what progress towards the stated goals of I-94 automation are attributable to the project and what progress is attributable to other factors such as process improvements. Because the INS did not establish the baseline level of performance, it will be difficult for the INS to measure the performance of the system in meeting its intended goals.

    Cost-Benefit Analysis Not Performed

    According to the SDLC, cost-benefit analysis is a vital management tool for linking function and budget, and supporting ongoing management oversight. A preliminary assessment of costs and benefits should be made during the strategic planning process and should be refined and updated as appropriate throughout the remainder of the life cycle. At each phase, information should be gathered and decisions should be made that enable the project team to make increasingly accurate projections of the total costs and benefits of the system over its projected life. Subsequently, the cost-benefit analysis should be updated at the end of each phase. Reasons for updating a cost-benefit analysis include such factors as significant changes in projected costs and benefits; major changes in requirements, including legislative or regulatory changes; or empirical data based on performance measurement gained through prototype or pilot experience.

    The level of detail in the cost-benefit analysis should be commensurate with the size of the investment. 6 When a system provides services to the public, managers should quantify the performance of the system through systematic measurement of the outputs. Agencies should seek to maximize return on investment over the information system life cycle by establishing and evaluating systematic performance measures. These performance measures should include the following: effectiveness of program delivery, efficiency of program administration, and reduction in burden, including information collection imposed on the public. The revised cost-benefit analysis at each phase of the SDLC provides up-to-date information to ensure the continued viability of an information system before and during implementation.

    The INS did not complete a cost-benefit analysis for the Automated I-94 System as developed. According to the project plan, full implementation of the I-94 automation was based on a successful pilot test in Philadelphia. (See previous discussion of our concerns about the data used to determine the success of the Philadelphia pilot.) The INS provided us with a draft cost-benefit analysis for I-94 automation; however, the magnetic stripe technology selected by the INS was not one of the alternatives considered in the cost-benefit analysis. Because the INS has not completed a cost-benefit analysis for the Automated I-94 System as developed, it lacks the data to decide whether the system is meeting the INS's needs.

    The cost-benefit analysis provided to us was a draft version dated January 1996. The analysis considered three feasible alternatives: (1) continuing the current manual process; (2) optically scanning I-94 arrival and departure records and validating the scanned images at the port of entry; and (3) optically scanning I-94 arrival and departure records at the port of entry and electronically sending the records to an off-site image processing facility. The analysis recommended continuing the manual process.

    Cost and Schedule Data Not Available

    In January 1998, the INS issued a final project plan for the Automated I-94 System. According to the SDLC, project plans should include cost and schedule information. The final project plan for the system does not include a budget and schedule. The INS project manager did not know why the project plan did not include a budget and schedule because he was not the project manager at the time the project plan was developed. Without a budget and a schedule, the INS could not and cannot properly manage the performance of the Automated I-94 system.

    Conclusion

    The INS has a substantial history of difficulties in managing its automation initiatives. In a 1999 follow-up on prior audit work, we noted that the INS did not adequately manage its automation programs despite the substantial investment it had made in such programs. We stated that "(1) estimated completion dates for some automation projects have been delayed without explanations for the delays, (2) costs continue to spiral upward with no justification for how the funds are spent, and (3) projects are nearing completion with no assurance that they will meet performance and functional requirements." In December 2000, the GAO noted that the "INS has limited capability to effectively manage its planned and ongoing IT [information technology] investments." In our judgment, the deficiencies noted in the Automated I-94 System are symptomatic of the broad-based weaknesses previously reported in the INS's management of automation projects and may be indicative of the challenges the INS will face when it expands the Automated I-94 System to land and sea ports of entry.

    Recommendations

    We recommend that the Commissioner of the INS:

    1. Prior to making any further expenditures on the Automated I-94 System, conduct a cost-benefit analysis that includes the most recent cost and benefit data, changes in requirements, and any data on performance.

    2. Use the results of the cost-benefit analysis to determine whether the project should proceed. If the results of the cost-benefit analysis indicate that the costs outweigh the benefits, notify Congress that the Automated I-94 System cannot function as required and request relief from the relevant portions of the Immigration and Naturalization Service Data Management Improvement Act.

    3. Develop and implement performance measures before further implementing the Automated I-94 System.

  2. AUTOMATED I-94 SYSTEM RISKS NOT ADEQUATELY MANAGED

    The INS did not adequately manage the risks associated with the Automated I-94 System. The project plan recognized a number of risks associated with the system; however, the INS did not develop a risk management plan as required. One risk, the lack of air carrier participation, has halted the deployment of the system as currently configured. As a result, the INS plans to modify the system so that voluntary air carrier participation is not necessary. Recent INS projections estimate that an additional $57 million will be needed to fully implement the system.

    No Risk Management Plan

    According to the INS SDLC, risk management is: (1) the assessment of a project's potential outcomes, including the likelihood of unsuccessful outcomes; and (2) the process of accepting, transferring, or mitigating risk. A risk management plan should document and identify project risks and then analyze, assess, and prioritize those risks. The plan is a control mechanism to monitor and direct all risk mitigation activities. The tracking of risks in a risk identification list is a critical facet of successful system development management. The risk identification list should be used from the beginning of the project and is a major source of input for the risk assessment activity. According to the SDLC Manual, risk categories include:

    A risk management plan should score each risk according to the probability and the resulting impact. The impact of a risk may vary from an inconvenience to a failure of the project. Risk items with high rankings should be reviewed and a determination made whether the risks will be accepted, transferred, or mitigated. If a risk is to be accepted, two options are available: contingency planning and no action. A contingency plan is a back-up plan to minimize the effects of a risk. When no action is taken and the risk is accepted, the project team has to accept responsibility if the risk occurs.

    Like other project activities, reducing the effects of risk requires effort, resources, and time. Efforts to reduce risk need to be incorporated into the budget, schedule, and other components of the project plan. The project plan should be updated to ensure the planning and execution of efforts to reduce risk. If risks have been accepted, the project plan should refer to the contingency plan.

    The INS did not develop a risk management plan as required by the SDLC. However, the project plan for the Automated I-94 System contained a list of some risk areas that could have an adverse impact on the delivery of the system. One risk area identified in the project plan, the lack of air carrier participation, has halted the deployment of the system as currently configured. The lack of air carrier participation and data integrity (a risk not discussed in the project plan) are addressed below.

    Low Airline Participation Hampers Implementation

    Currently, only US Airways and TWA have agreed to participate with the INS to implement the Automated I-94 System. In its project plan, the INS recognized that airline participation was a risk; however, it is not clear what steps the INS took to mitigate this risk. The Department's FY 1999 Annual Accountability report indicated that the INS would have to rely heavily on the cooperation and participation of airlines. One airline, Northwest Airlines, agreed to test the Automated I-94 System but declined permanent participation. In a memorandum to the International Air Transport Association, Northwest Airlines listed the following concerns.

    Managers from the INS acknowledged Northwest Airlines' concerns with the Automated I-94 System and said that the INS was willing to address the concerns of other non-participating airlines. At the conclusion of our audit, INS officials told us that modifications currently under consideration are intended to eliminate the need for relying entirely upon carrier participation. However, the INS officials provided no details on the nature of these modifications to allow us to determine the feasibility of operating the system without voluntary air carrier participation. The INS also told us it was concerned about meeting the time frames for implementing an automated system as directed by the Immigration and Naturalization Service Data Management Improvement Act of 2000.

    Full implementation of the Automated I-94 System as currently configured has ceased because most carriers have repeatedly resisted participation. Consequently, the INS is currently considering ways to redesign the system so that it does not rely on carrier participation. The INS admits that it did not plan for the possibility of poor carrier participation. Specifically, the INS did not realize the impact that air carrier alliances would have on the willingness of airlines to participate. In our judgment, the INS should address both the concerns of Northwest and other non-participating airlines before proceeding with the modifications to the system.

    Departure Data Lacks Integrity

    The INS has had a long-standing problem collecting I-94s when nonimmigrants depart the United States. Staff at the INS believe that the unrecorded departures are caused by the failure of airlines to collect the departure records and turn them over to the INS and nonimmigrants departing through land borders. As noted in the introduction, the Immigration and Naturalization Service Data Management Improvement Act requires the INS to develop an Integrated Entry and Exit System at air, land, and sea ports of entry. With regard to air ports of entry, the manual I-94 system cannot be used to determine the number of departure records that should be forwarded to the INS for a given flight because airline manifests are required to separate the passengers into U.S. citizens and non-U.S. citizens; however, two groups of non-U.S. citizens are not required to submit an I-94: Canadians and lawfully admitted permanent residents. As a result, the INS does not know whether the airlines are submitting the correct number of I?94s. According to an INS official, the INS is aware of the weakness and plans to address it by modifying its regulations to: (1) require I-94s from Canadians and lawfully admitted permanent residents, and (2) fine airlines for not submitting the correct number of departure I-94s.

    Conclusion

    In our judgment, the inadequate risk management we found in the Automated I-94 System is also symptomatic of the broad-based weaknesses previously reported about the INS's management of automation projects.

    Recommendations

    We recommend that the Commissioner of the INS:

    1. Develop a risk management plan before making further expenditures for the Automated I-94 System.

    2. Address the concerns of non-participating airlines before proceeding with the modifications to the Automated I-94 System.

    3. Develop a method for determining the number of departure I-94s that should be submitted for each flight.


    Footnote

    1. According to the INS SDLC Manual, a major project is one that is anticipated to cost more than $10 million.