Source:
OIG developed using WAM data for the median week in FY 1994 at an airport in the
Northwest. Legend: 1F equals one flight arriving during the hourly period reported. 2F equals 2 flights.
Figure 1
FINDINGS AND RECOMMENDATIONS
I. PROJECTIONS.
INS has no assurance that WAM will determine the optimum number of inspectors needed at a port of entry. This is because WAM projects inspectors needed above the inspectors entered in the ports' shift schedules but will not detect overstaffed work shifts or project staff decreases when needed. As a result, INS must rely on procedures outside the simulations because WAM does not detect anomalies.
From our testing of WAM, Version 2.2 and the unreleased Version 3.2, WAM will not necessarily determine an optimum number of inspectors needed to staff a port of entry. Rather, WAM computes the number of additional inspectors needed in total and on an hourly basis over and above the staff entered in the inspectors' shift schedules. Further, WAM does not detect overstaffed work shifts or project staff decreases when required. As a result, WAM's projection of additional inspectors needed at a port of entry cannot be relied upon by itself.
In our judgment, WAM should have been designed to determine the optimum inspector staffing levels at ports of entry. Headquarters Inspections indicated the model was designed to operate under the assumption that the POEs knew best how to schedule inspectors for the workload. WAM needed inspector shift schedules entered in the model as a starting point to begin the analysis. Otherwise, without entering shift schedules, WAM could take many hours to determine staffing projections for a single large port.
An INS official also told us that an objective means (computer modeling) to determine optimum inspector staffing levels was necessary to avoid the "squeaky wheel" method of staffing. INS officials stated that a mathematical model seemed desirable because all staffing decisions would be based on the facts related to the workload.
We chose an existing airport in the Northwest from our sample of 41 ports, to demonstrate that WAM will not determine the optimum number of inspectors needed at a port. Using WAM, Version 2.2, we entered the actual data submitted by that port using their FY 1994 workload and staffing data. The inspection workload for that port was 5-7 international flights per day, 7 days per week. The flights arrived every few hours or so, between 8:30 a.m. and 10:30 p.m. The port had 21 permanent full-time (PMFA) inspectors and 4 other-than-full-time (OTFP) inspectors assigned for that workload. After the output optimization run was performed, WAM projected that the port needed 2 additional other-than-full-time inspectors using the shift formula (see Appendix III, Figure 3). However, using the total hours formula, WAM indicated that no other-than-full-time inspectors were needed. In this specific instance, INS told us that they would not add any additional inspectors to the port because there were insufficient total hours to justify any additional staff. Therefore, depending on the formula used, WAM indicated a total of 25 or 27 inspectors were needed.
To show that WAM will not optimize staffing, we began with the same workload as described above but ran WAM with initial staffing of only one inspector for 1 hour per week and reduced the authorized staff to zero. This procedure forced WAM to calculate the minimum number of inspectors needed by adding inspectors to booths until the inspection staffing was adequate to process the workload on an hour-by-hour basis. When the output optimization run was completed, WAM indicated the need for only 6 full-time inspectors and 3 other-than-full-time inspectors, or a total of 9 inspectors would be sufficient to handle the workload on an hour-by-hour basis (see Appendix III, Figure 4). Considering this example with a projection of 9 inspectors and the preceding example with at least 25, WAM clearly is not optimizing staffing levels.
In addition to WAM not determining an optimum number of inspectors, WAM's projection of 9 inspectors could only be correct if Immigration Inspectors were willing to work no less than 10 hours per day, 7 days per week, year-round. WAM projects the absolute minimum number of additional inspectors needed on an hour-by-hour basis to process the workload. WAM does not take into account the fact that most inspectors work in 8-10 hour shifts, four to five contiguous days per week. Additionally, WAM does not take into account the normal overhead associated with employing staff, such as annual and sick leave, holidays, training, etc. INS factors these in later in a separate process (see Appendix II, page 25, concerning the Consolidated Staffing Requirements and Allocation Model 1). Thus, starting from a minimum shift schedule, WAM will only project bare bones staffing that would not work in actual situations.
To radically demonstrate that WAM does not detect overstaffing and also to illustrate
that WAM does not perform comparisons of the number of inspectors included in the shift
schedules to the maximum capacity of the port, we entered 320 inspectors working 12,000
hours to the shift schedules. The maximum capacity of the port was 21 booths for primary
inspection and 4 positions for secondary inspection. Again, we used the same inspection
workload of 5-7 international flights per day, 7 days per week at the same port. Our
hypothetical work shifts were overlapping and included 80 inspectors per shift, 40 for
primary inspection and 40 for secondary (see Appendix III, Figure 5). We did not assign
any staff to Saturday or Sunday, even though inspection workload existed. After the
optimization process was run, WAM did not detect the excessively overstaffed situation,
project staffing decreases, or indicate that staffing in the shift schedules greatly
exceeded the maximum capacity of the port. In fact, WAM recommended the addition of 2 more
other-than-full-time inspectors to handle the Saturday and Sunday inspection workload.
In our judgment, WAM would be of greatest utility if it could develop efficient staffing
projections based upon workload. The model should optimize the number of inspectors needed
based upon a given workload, and the optimizations should not be excessively dependent
upon the entry of shift schedules by the port directors. Undoubtedly, INS's procedures
outside of WAM simulations, such as manually comparing scheduled hours to authorized
staff, would have detected such gross anomalies as we used for demonstration purposes.
However, manual detection does not equal optimization.
On June 10, 1996, INS authorized a new task and contracted with the WAM contractor to generate optimum hourly shift schedules that will be compared to the POEs' actual shift schedules entered in the model. The differences can then be analyzed to detect overstaffed work shifts and ports. If the planned WAM Scheduling Report is implemented as described, the report can be a useful tool to assist POE and INS management with improvements in inspector shift scheduling and reduction of overtime costs. However, procedures defined in the new task will not determine the optimum number of inspectors needed at a port or determine optimum work shifts, because the new task will only optimize on an hour-by-hour basis.
Recommendation
We recommend that the Commissioner, INS:
1. Direct a reprogramming of WAM to determine the optimum number of inspectors needed
at any port of entry.
The above recommendation is closed. See Appendices V and VI for resolution activity.
II. REPORTS
WAM output reports could present the data in a more useful format to assist Headquarters Inspections and port directors with managing and staffing their ports efficiently. Output reports comparing workload to shift schedules, queue times, and so forth could be developed to assist Headquarters Inspections with their assessments of the validity of the POEs' input data and the projections generated by WAM.
The output reports in WAM were designed primarily for Headquarters Inspections use. However, the use of those reports was somewhat limited because the output reports do not display the workload in comparison to the staffing. WAM uses the workload data but the data is converted into the number of additional inspectors needed to process the workload.
WAM's output consists of a 10-20 page report with data tabulated individually by an air, land, or a combination POE. Output data for a POE includes the number of inspectors assigned; total scheduled inspection hours; maximum number of lanes or booths; numbers inspected, by type; secondary referrals; average and maximum times in the queue; primary and secondary duty hours; scheduled overtime hours; hours at remote assignments; facilitation processing times 2 in percentages; and a projection of the number of additional inspectors needed in primary and secondary inspection in total and on an hourly basis.
We believe the current output reports, although voluminous, could present the data in a more useful format. The following information could be presented to assist Headquarters Inspections with managing and staffing their ports efficiently.
• Output reports should display the average wait times in the queue, on an hourly basis, by each day of the week. Displaying queue times on an hourly basis would assist INS with validating WAM and improve inspector scheduling efficiency.
• Output reports should display actual primary and secondary inspection shift schedules separately. In the unreleased Version 3.2 of WAM, primary and secondary inspection shift schedules are combined. Separating shift schedules would allow INS to see how and when inspectors were assigned to process the workload.
• Output reports should display, in grid form, the primary and the secondary workload (number of inspections and conveyances, 3 mean inspection processing times, and the numbers and types of booths open) on an hourly basis, by each day of the week (see Figure 1 below). Displaying the workload along with the inspectors used to process the workload allows INS to visually identify over/under staffed work shifts.
As an example of a visual comparison, the following figure shows an airport from our
sample of 41 ports that was located in the Northwest. The actual workload, in numbers of
flights, and the number of inspectors assigned by the shift schedules that were used to
process the workload, are displayed. The time period was the median week from FY 1994
workload. There were 5-7 international flights per day, 7 days per week. The earliest
flight arrived about 8:30 a.m. and the last flight arrived about 10:30 p.m.
Source:
OIG developed using WAM data for the median week in FY 1994 at an airport in the
Northwest.
Legend: 1F equals one flight arriving during the hourly period reported. 2F equals 2
flights.
Figure 1
The above figure clearly shows that there can be staff assigned when there was no workload (Sunday through Saturday from 6:00 a.m. to 8:00 a.m.), and workload can occur when no staff were assigned (Sunday afternoon and Saturday evening) to conduct the inspections. WAM does not display the workload to show a visual comparison to the assigned workforce or print the other reports described above. Without displaying the workload for visual comparisons, inefficiencies are much more difficult to detect. The information necessary for the above suggested reports are already contained in the WAM data bases. Per the WAM contractor, developing output reports, such as Figure 1 above, are relatively easy.
The WAM contractor is currently working to improve the model to provide additional
management information. The WAM contractor was issued a task, Task Order 96-ITP-205,
issued June 10, 1996, to convert the hourly output projection data into optimum hourly
shift schedules for use at Headquarters Inspections and the ports. If the planned output
reports are developed with headquarters and port management in mind, the output reports
can be used as a staff cost reduction tool if the optimized data is appropriately
displayed.
Recommendations
We recommend that the Commissioner, INS, require that WAM:
2. Generate output reports that display the workload (number of conveyances and inspections); the number of lanes/booths open; and queue and mean processing times on an hourly basis, 7 days each week. For visual comparison purposes, the reports should display the data in the same format as that currently presented for the primary and secondary staffing projections.
3. Generate detailed and summary output reports that Headquarters Inspections and port directors can use to assist with adequate staffing for the workload, preparing inspector shift schedules, and monitoring and controlling use of overtime.
Recommendation 2 is resolved, but not closed. Recommendation 3 is closed. See appendices V and VI for resolution activity and the actions necessary to close Recommendation 2.
III. VALIDATION
National standard inspection processing times were developed by summarizing processing times developed by POEs. However, the ports' inspection processing times were estimated and not documented or verifiable. In addition, although INS officials claim that WAM projections have been validated, we found no verifiable documentation. As a result, we cannot give any assurances as to the accuracy of WAM's projections.
Validation of Inspection Processing Times
The INS utilizes the WAM model to simulate the primary and secondary inspection process using national standard inspection processing times. These times are developed from data submitted annually by POEs.
The process for the development of port inspection processing times required that each port determine their processing times and enter that, along with a 3-week sample of inspection workload, into computerized WAM data sets. The completed data sets were sent to the WAM contractor who combined and statistically analyzed the port developed processing times. The resulting analysis was reviewed by Headquarters Inspections in order to develop the national standard inspection processing times. These national standards were used by WAM to project staffing for POEs. The WAM contractor explained that the inspection processing times, used by the model for any single inspection, were a random distribution around the national standard times. As a result, the model was placing all POEs of the same type on an equal basis.
According to INS Headquarters Inspections, POE management and the WAM contractor should perform tests of actual inspection times, and if available, review exception reports contained in the Treasury Enforcement Communications System (TECS) 4 in order to maintain a good estimate of the ports' inspection processing times. However, there was no documentation to indicate whether the contractor or POE management followed these procedures. In addition, we found no supporting documentation for the processing times at any of the 41 ports in our sample. Three port officials indicated that their processing times were estimates and one port director said that its times were based upon historical experience. Without supporting documentation, we could not verify or validate for accuracy any of the port inspection processing times.
When we asked the WAM contractor if they thought the estimated inspection processing times were accurate, they said that actual time studies of inspections, if performed at a sample of ports, would probably show estimated inspection processing times within plus or minus 10 percent of actual times.
We acknowledge that there was considerable experience involved in the effort of developing standards. However, we believe that without recording precise inspection start/stop times using electronic, manual, or mechanical methods by the type of inspection, the current port-developed estimated processing times cannot be relied upon with a high degree of confidence until they are validated. Further, projections of additional inspectors needed at a port may be in error, because WAM's projections are based on the inspection processing times.
Validation of WAM
INS officials told us that the WAM was validated exhaustively during WAM's development process, about 5 years ago, when over 60 ports were visited. They indicated that data, such as inspection processing times, was collected. Each port type was modeled with this data, as part of the model's verification and validation (V&V). Results of the simulation runs were also compared to actual queue times and lengths of inspection experienced at both air and land POEs. In addition, predicted levels of regular and overtime pay hours were compared to actual experience at sea POEs as part of the V&V process. However, after a review of the site visit documents, we found no discussion of the V&V process.
The WAM contractor said that a booth utilization rate, a measure of the amount of time an inspector was actually performing inspections, was developed during the WAM simulations but was not displayed in the output. The contractor suggested that comparisons of the WAM generated rate to an actual, measured booth utilization rate would be a good method for validation of the model. We agree. Also, in our judgment, comparing predicted queues and overtime hours to actual results seems to be an easy and inexpensive method to validate the model's accuracy. More of these procedures should be performed and the results documented.
A draft write-up of the INS's planned "Performance Indicator Management
System" identifies new techniques and procedures that should improve the capability
of Headquarters Inspections to test WAM's reliability. A few examples mentioned in that
planning document that should eventually lead to greater accuracy of some data with which
to base WAM projections include: cameras focused on geographical points at which the queue
time generally exceeds 30 minutes, depending upon the number of vehicle lanes open; a
field labeled "physical queue time" added to the G-22; 5
and passive transponders to measure the time of air passengers passing through the
inspection process. With that data, the model could be validated by comparing WAM's
simulated queue time with the POE actual queue.
Recommendations
We recommend that the Commissioner, INS:
4. Direct Headquarters Inspections to validate, on a test basis, port of entry developed inspection processing times.
5. Direct Headquarters Inspections to validate WAM projections by periodic, documented testing of WAM generated queue measures, booth utilization rates, and predicted levels of regular and overtime hours against actual inspection operations.
The above recommendations are resolved, but not closed. See Appendices V and VI for resolution activity and the actions necessary to close the recommendations.
IV. LAND BORDER PORTS' WORKLOAD ACCURACY.
Vehicle and pedestrian counts at some land border ports were either estimated or had to be prorated to an hourly basis using a profile based upon judgment. As a result, WAM's staffing projections of additional inspectors needed may be inaccurate because projections are based on the hourly vehicle and pedestrian counts.
As with any mathematical model, output data are affected by the accuracy of the input data. Specifically in WAM, the accuracy of the vehicle and pedestrian counts at land border ports can affect the projections of needed inspectors. More than 400 million persons were inspected at these ports, accounting for approximately 87 percent of all inspections in FY 1994, the latest year available. Yet, we found that few of the ports in our sample used actual counts. Most used some form of estimation based on port personnel experience or profiles. 6 Land border port traffic is inherently more difficult to count as opposed to air traffic, because at airports, exact counts of passengers are readily available from airline manifests.
Based on interviews of officials at the 17 land border ports in our sample, only 1 port had actual counts of vehicular traffic for each hour, as needed by WAM. Six of the ports had actual daily counts that were then allocated on an hourly basis through means of a documented profile. The remaining ten, however, either based their counts totally on the personal estimates of port officials or allocated a known daily count using personal estimates. WAM needs hourly counts as input, because the model is designed to use that data to determine the inspectors needed on an hourly basis for each day of the week.
As a solution to obtaining accurate hour-by-hour vehicle counts, INS Headquarters Inspections staff indicated that the Treasury Enforcement Communications System (TECS) can be queried by INS port officials to obtain the number of vehicle license plates entered in the system. TECS records these entries on an hourly basis. However, some INS officials at the land border ports stated that TECS counts in total were not an accurate source for vehicle counts because of system down-time and human error. However, some INS port and Headquarters Inspections officials said that TECS counts, converted into percentages of vehicle traffic by hour, were helpful to assist with hourly profiling procedures.
For pedestrian counts, the situation was similar to vehicles. Based on interviews of officials at the eight land border ports in our sample with pedestrian traffic, six ports developed the hourly traffic counts based on officials' personal judgment. One port allocated an estimated daily count on a hourly basis by means of a documented profile. Only one port had actual hourly counts for pedestrians.
INS is considering new methods to generate better counts of inspections. As stated in a draft report on its Performance Indicator Management System, INS is considering using advanced technology, such as electronic devices, to count persons inspected in pedestrian lanes. This type of procedure should increase the accuracy of workload data.
Although it is axiomatic that bad data leads to bad results, good data is not
necessarily synonymous with absolute accuracy. The key is how sensitive a given model is
to data accuracy. Given the vast number of inspections that one full-time inspector can do
per year, WAM may not need pinpoint accuracy of workload to make projections of needed
inspectors. A Headquarters Inspections official told us that the sensitivity of the model
to data accuracy was studied, but only during WAM's development phase. No similar studies
have been performed on the subsequent versions of WAM. The official agreed that such
analyses should be done again.
Recommendations
We recommend that the Commissioner, INS:
6. Develop more accurate counting techniques, such as TECS-based profiles or advanced technology, if the sensitivity analyses show the need for better data from land ports.
7. Have sensitivity analyses performed on WAM to determine the accuracy needed for proper results.
The above recommendations are resolved, but not closed. See Appendices V and VI for resolution activity and the actions necessary to close the recommendations.
1 A set of spreadsheets to summarize WAM projections, authorize
positions, and add indirect time and administrative overhead.
2 Facilitation is a measurement to determine what percentage of passengers on international flights were inspected within 45 minutes of an aircraft's arrival.
3 A plane, ship, bus, train, or vehicle.
4 TECS is the lookout system used for primary inspections at POEs. As each vehicle enters the booth for inspection, the vehicle's license number is entered into the TECS data base.
5 A form to collect workload statistics on a monthly basis.
6 A profile shows which hours have heavier or lighter traffic (e.g., on the average, traffic at 1 p.m. is twice as heavy as 2 p.m.). Thus, by knowing total daily traffic, one can allocate traffic on a hourly basis by means of a profile. Support for profile validity can range from heavily documented evidence to rules-of-thumb based on personal experience.
#####