DIRECTIVE	:	UNEMPLOYMENT INSURANCE PROGRAM LETTER NO. 41-92
TO	:	ALL STATE EMPLOYMENT SECURITY AGENCIES
FROM	:	DONALD J. KULICK Administrator for Regional Management
SUBJECT	:	The Implementation of Alternative Verification Methods in the UI Benefits Quality Control Program

1. Purpose. 

   (a)  To summarize the findings of the Telephone Pilot Project, and (b)  to indicate the process and schedule for implementing alternative methods in the Benefits Quality Control (BQC) Program.

2. References. None.

3. Background. Benefits QC, following the model of the Random Audit program, draws small samples of paid claims each week and investigates all aspects thoroughly by in-person interviews with claimants, employers and third parties. The systematic random sampling and rigorous standardized method for conducting the verification enable the State and Department of Labor to infer reliably the extent of mispayments in the universe of payments, and their causes and responsibilities. This rigorous investigative methodology has its price: the average BQC case takes about 13 hours of investigator's time, plus considerable travel expense.

   Since Random Audit, there has been interest in verifying UI payment accuracy by telephone. No one doubted telephone verifications would be cheaper, but their potential effectiveness* was questioned. When QC formulated its initial plan for subjects to be pilot tested, "alternative methods" was given the highest priority.

   In 1986-87 Idaho conducted a controlled test of telephone interviewing. Overall, it showed that there was no statistically significant difference between the telephone and in-person methods in detecting cases with errors. The input time per telephone case was about half that for an in-person case. However, there were questions about some of the findings, and the average elapsed time to complete a case by telephone was significantly greater than in person (apparently it was easier to defer telephone cases). The latter point is important because of the danger that QC case quality may decline if the case becomes "cold."

   In view of these results, and because Idaho differed in many aspects from other States, it was thought prudent to replicate the test in other, more varied environments.

4. The 1990 Pilot Test. 

   1. Research Questions. With the Idaho results in mind, this pilot was designed to answer four main research questions:

      (1)  Is telephone verification less costly than in-person? If so, by how much?

      (2)  Is longer time lapse inherent in telephone cases?

      (3)  How does the accuracy (completeness or thoroughness) of telephone verifications compare with in-person, both overall and by aspect of the verification process? (i.e., are there areas of bias or weakness?)

      (4)  Are there other implications of telephone verifications, such as QC Investigators' morale/image of the job; relationships with UI local offices, employers, and claimants; and the credibility of QC data with potential data users?

   2. Description.  Four States--California, Colorado, Florida, and Maryland--participated in the pilot. They, were selected either because they had volunteered earlier to replicate the Idaho study, or because they gave geographical and programmatic balance to the study. Colorado and Maryland have centrally stationed QC staff; Florida's staff are all outstationed. In these States, all staff did both telephone and in-person investigations. California QC staff are also outstationed; for the pilot it created a small unit in Sacramento which did all telephone investigations, while all in-person contacts were made by outstationed staff. Beginning in early February, cases became part of the pilot, and this continued through the last batch of 1991. The States randomly divided their weekly QC case draw between telephone investigation and in-person investigations. In. return for some additional staff, the States raised their total case load, so that pilot cases were as follows:

      	Base QC Caseload	Telephone Cases	In-Person Cases	Total Pilot Cases
      California	2000	733	1467	2,200
      Colorado	1000	550	367	917*
      Florida	1000	642	642	1,284
      Maryland	1000	550	550	1,130

      *Colorado simultaneously conducted a QIP study, for which it was allowed to reduce its sample size.

      The above cases were assigned and investigated over an 11-month period, versus the base QC workload which is for a full year.

      The following protocol was used for cases assigned for telephone investigation: cases were to be investigated and completed entirely by telephone if possible. If this proved impossible after exhaustive attempts, the QCI could ask the supervisor's approval to complete the case as a "mixed mode" case. In such cases, they were to seek the information first by mail (or by fax) and only then in person. There were 2 exceptions to this protocol. (1) For all States, base period wage information obtained by fax was considered to have been received by telephone. (2) In exchange for investigating an additional 200 cases by telephone, California was allowed to split its telephone sample. Two thirds were "pure telephone," i.e., if information could not be verified by phone it was considered unverifiable. The other third were "mixed mode" cases, completed using the protocol outlined above. Unlike Idaho, no telephone cases were reinvestigated in person. Investigators disliked reverification work because the parties to the case frequently complained about being contacted again. The technical support contractor advised that clear inferences could be drawn without it.

      All the standard QC Data Collection Instrument (DCI) items were recorded for both telephone and in-person cases. In addition, pilot staff were asked to record considerable information on the time and costs of completing the cases by the various modes. Most of this was case-specific ("job ticket"). For one week in each of the four calendar quarters of the pilot, however, they accounted for all their time used throughout the day on a "time ladder." Timekeeping was undoubtedly the greatest burden the pilot imposed.

      A technical contractor, Abt Associates, Inc., assisted in designing and managing the pilot, and conducted the evaluation. Its final evaluation report was received on November 15, 1991. The following findings rely heavily on that report, but also involve subsequent analyses of the pilot data.

   3. What the Pilot Found, by Research Issues

      (1)  How Input Costs Compare.  Data from the case-specific "job tickets" indicate that it takes about half as much staff time to do QC by phone as in person. This finding is virtually identical to results from the Idaho phone pilot. Most of the average 5.8 hours per case saved is travel time; but less time is spent doing the actual verification by phone as well. By phases of the investigation, the savings are 2.1 hours for Base Period Wage and Separation investigation, 1.3 hours for Work Search, 1.8 hours for the Claimant Interview, and 0.6 hours for all other aspects.

      The study's best estimate of the time to complete a QC case in person, arrived at by combining job ticket and time ladder data, is approximately 13 hours. It estimates that if all cases were done by telephone, a typical QCI could handle 3 cases per week instead of 2, and QC staff could be reduced by about 3 per 1,000 cases investigated. The projected savings would be about 200 staff nationwide.

      In addition, the study estimates that savings in travel costs (mileage alone) will range from $25,000 to $54,000 per 1,000 cases investigated, depending on the State. These savings could be partially offset by increases in telephone charges of between $5,000 and $15,000 per 1,000 cases.

      The pilot also did not attempt to measure all costs of doing QC bY telephone. Full telephone verifications often require local office staff to assemble and copy UI documents and employers to assemble and read or fax wage data, tasks QCIs often perform when on site.

      (2)  Elapsed Times to Complete Cases.  In three of the four pilot States, times were the same. Times were significantly longer in California, which had a centralized phone unit doing 4 cases per investigator per week. California's study director believes, however, that longer time lapse was a management issue, and is not inherent in the mode.

      (3)  Relative Thoroughness of Telephone and In-Person Investigations.  Overall, the in-person mode was consistently more thorough at finding cases with errors, but only in Colorado was the difference statistically significant. Only in Florida did in-person not produce a significantly higher dollar overpayment rate, however. For California, Colorado and Maryland, the telephone overpayment error rate averaged about 60% of the dollar overpayment rate yielded by the in-person mode; it was 70% overall.

      In-person investigations yielded slightly greater proportions of cases with errors; more cases with multiple errors; and in those cases with errors, higher average dollar amounts. The findings on dollar underpayments are mixed, perhaps because the underpayment (UP) error rate is much smaller and average dollars underpaid, due mostly to base period wage errors, are small. The results are summarized in Table I.

      Table 1
      
      Case and Dollar Error Rates by Method of Investigation
      Figures are Percents

      Cases w/ Errors	California		Colorado		Florida		Maryland		Average*
      	Tel	In-P	Tel	In-P	Tel	In-P	Tel	In-P	Tel	In-P
      OP	10.6	12.2	14.4	19.9*	25.3	26.3	11.4	14.2	15.4	18.2
      UP	4.6	5.6	4.3	4.2	19.5	21.7	4.4	3.0	8.2	8.6
      Dollars In Error
      OP	4.4	7.2	7.8	12.7*	13.7	14.4	7.0	12.6	8.2	11.7
      UP	0.6	0.5	0.3	0.5	1.1	1.3	9.4	0.5	0.6	0.7

      * Each State weighted equally.

      The QC database permits disaggregating the overall overpayment data in two particularly useful ways: by Cause (where in the UI eligibility determination process the overpayment occurred) and Primary Detection Point (where in the QC investigation process the error was detected). When the overpayment errors are broken down by Cause and Detection Point, the basic findings are: (1) case error and dollar error rates through in-person investigations tend to be higher that those by telephone across all States; (2) most differences are not statistically significant; and (3) both in-person and telephone modes tend-to show the same rankings of the type of errors. That is, if in-person investigations show that work search errors are more prominent than separation errors, the telephone results tend to show the same thing. Telephone investigations tend to reflect the same pattern of errors in the payment universe as in-person investigations, but they tend to give a lower estimate of the levels of these errors.

      When the error rates, by Cause and Detection Point, detected by the two modes are put onto a consistent basis (dollars overpaid as a percent of benefits paid) and averaged across the pilot States, the following relative detection powers are suggested:

      By Cause	Telephone/In-Person. %
      Base Period Wages	123
      Separations	73
      Work Search	65
      Benefit Year Earnings	85
      Other	47
      ALL	71
      By Detection Point
      Claimant Interview	48
      BP Wage/Sep Inv.	77
      Work Search Verif.	54
      3rd Parties	0
      UI Records	100
      ES/JS Records	99
      Union Verif.	0
      All	70

      These calculations suggest that the telephone is quite effective for doing wage verifications (perhaps because of the high use of fax) and Benefit Year earnings verifications (these often involve wage verifications). It is relatively ineffective for such components as the claimant interview and verifying work search. In total, the four-State average shows it 70% effective; if the results for Florida are excluded, the rate falls to 60°l0.

      (4)  Other Implications of Doing Telephone Verifications

      1. Investigators' Morale/Image of the Job.  The consensus of QCls was that an exclusively telephone QC operation (implying more cases per week) would be bad for morale and make it hard to retain/attract best qualified staff. A proper mix, however, (based on their judgment) would improve conditions.

      2. Relationships with Ul Local Offices (LO's):  Phone verifications impose costs on LO's to obtain and copy documents. This created friction in some States (e.g., California), but not in others.

      3. Relationships in Employer Community.  As with LO's, phone operations burden employers, especially for some base period wage verifications. QC staff in one State reported significant employer complaints about it.

      4. Relationships with Claimants.  Claimants preferred being interviewed by telephone; QCIs preferred in-person interviewing. Both felt more comfortable being on their own "turf," and QCls believed they get many clues from observing the claimant.

      5. Credibility with Data Users.  How investigations were done generally did not seem to matter to users. Florida, California and Colorado do many operations, e.g. , appeals hearings, entirely by phone. Telephone findings created a problem for fraud cases in Maryland, where an Administrative Law Judge ruled phone information "hearsay."

      6. Use of Time, Organization, etc.  Completely telephone-based investigations would require different patterns of office coverage and more teamwork among QCls.

      (5)  What Aids. Hinders Telephone Investigations.

      1. Investigators agreed that the fax machine is essential. Useful are headsets, criss-cross directories, answering machines; possibly sophisticated telephone tracking and routing equipment.

      2. Training in skip-tracing by phone, time management, and customer service skills would be helpful.

      3. Use of phone is inhibited by suspicious attitudes of employers and claimants, and poor working relationships with local office staff (on whom some of the burden of phone verification can fall). Some pilot States used introductory letters to ease responses from employers; there was no consensus about their overall value.

      (6)  Study Recommendations. Based on the technical contractor's field experience and analysis of the evaluation data, but not reflecting any policy considerations, the report contains the following recommendations:

      1. Make telephone the lead method or "primary mode" of investigation for all portions of the investigation.

      2. Give investigators flexibility to switch to in-person mode when necessary to investigate cases thoroughly.

      3. Use savings to allow QCIs to spend time doing analysis and PI studies to fight the tedium of a basically-phone operation.

      4. Give States sufficient lead time (for training, reorganization, equipment purchases, redesign offorms, changing job descriptions) to implement phone methods.

      5. Provide resources to purchase necessary supplementary equipment (faxes, multi-line phones, answering machines, headsets, criss-cross directories, etc.).