DIRECTIVE	:	UNEMPLOYMENT INSURANCE PROGRAM LETTER NO. 27-96
TO	:	ALL STATE EMPLOYMENT SECURITY AGENCIES
FROM	:	MARY ANN WYRSCH Director Unemployment Insurance Service
SUBJECT	:	Pilot Test to Determine the Accuracy of Benefit Denials

1. Purpose. To announce a pilot test of an approach for measuring the accuracy of unemployment insurance (UI) benefit denials, and to solicit volunteers for the pilot.

2. References. UIPL 41-95 (August 24, 1995), "Draft Narrative Describing the System for Enhancing Unemployment Insurance (UI) Performance: the 'UI Performs' System."; and UIPL 15-96, (April 2, 1996), "Proposal to Modify the Benefits Quality Control Program."

3. Background. Beginning in 1993, a team of senior State Employment Security Agency (SESA) managers and Unemployment Insurance Service staff known as the Performance Enhancement Workgroup developed the outlines of a comprehensive new closed-loop system for helping ensure continuous improvement in UI operational performance. This new system, called UI Performs, is described in UIPL 41-95, issued August 21, 1995, which solicited comments from SESAs. The UI Performs design is now being refined in light of responses to UIPL 41-95; the system will be phased in over 3-4 years as the various components are completed.

   When complete and fully operational, UI Performs will comprise a renewed affirmation of long-standing principles for Federal-State cooperation; a redesigned process for operational planning and the execution of improvement activities; various mechanisms to ensure that performance in key areas at least exceeds certain minimum levels; and the redesign of many performance measurements plus the regular validation of key performance measures. The measures were selected to enable the UI system's success to be judged by how well it serves its ultimate customers--claimants and employers. Most of the data used to contruct existing and new UI Performs measures are already in State databases; UI Performs should not appreciably increase measurement effort overall.

   The measurement redesign aspect embraces the results of three initiatives which predated UI Performs: Revenue Quality Control, which developed new measures for tax performance to replace those previously gathered under the Quality Appraisal (QA) system; the Performance Measurement Review project, which updated QA benefit payment timeliness and quality measures; and data validation. The team designing UI Performs also proposed substantial changes in the Benefits Quality Control (BQC) system for measuring benefit payment accuracy, including the assessment of the accuracy of decisions to deny eligibility.

   This issuance concerns the pilot test of a system to assess the accuracy of decisions to deny benefits. An approach to such measurement was pilot tested shortly after BQC began, but was never implemented. In 1987, the Department of Labor (DOL) pilot tested using the BQC methodology to measure the accuracy of determinations to deny benefit claims. Five States--Louisiana, Iowa, Pennsylvania, South Carolina, and Washington--conducted the pilot. Appendix I of the attached paper summarizes the findings of this pilot. Although most pilot States showed relatively high rates of error in their denial determinations, resource considerations and other priorities precluded the DOL from expanding the pilot effort or building it into the BQC program.

   Interest in the question of the accuracy of denial decisions has continued. The Vice President's National Performance Review said the DOL should consider to what extent the BQC performance measurement should include "decisions to deny claims." The joint Federal-State Performance Enhancement workgroup recommended adding measurement of denials to paid claims as part of a comprehensive payment accuracy measure for the UI Performs system. Various UI system stakeholders, including the National Employment Law Project and the DOL's Office of Inspector General, have also advocated assessing denials accuracy.

   Before implementing such a measure, it seems prudent to conduct a pilot test to ascertain the extent of current errors, costs and approaches to measurement, and technical difficulties.

4. Plan for the Pilot. The attached planning paper summarizes the overall design for the pilot. The DOL envisions a five-State pilot, ideally to include at least one alternative base-period State. A cooperative agreement will be used to specify State and Departmental responsibilities and deliverables and provide funding for certain additional State costs. Following the preferences of stakeholders and the model of the 1987 pilot, this pilot will measure accuracy using the sampling-and-verification approach used in the BQC program; the accuracy of all types of denials--monetary, separation and continuing eligibility--will be examined in this way. However, it will also examine whether some variant of the Quality Performance Index (QPI) instrument might be a satisfactory alternative for assessing the accuracy of nonmonetary denials.

   Samples will be drawn and investigated for up to 12 months to give a balanced picture of denials activity and accuracy. State selection will be made in the Summer of 1996 with the object of commencing sampling in Spring 1997. The starting date will depend on several factors, among them being the time required to accomplish the necessary programming and related ADP preparation, to refine the design, and to secure clearances.

5. Responsibilities and Resources Under the Pilot. The responsibilties for the pilot will be divided as follows.

   1. Departmental Responsibilities. The DOL will assume overall direction for the pilot and has produced a basic design. In addition it will:

      fund States' additional pilot costs through a combination of a cooperative agreement and basic UI Performs allocations;

      develop specifications for the software needed on State mainframes to develop transactions files defining the universes of the three types of denials;

      develop specifications for the software program for selecting the samples;

      provide data entry and data management software based on existing BQC software for the SESA's UI SUN workstations;

      monitor the pilot; and

      evaluate the pilot's process and impact.

      The DOL has contracted with PRAMM Consulting Group to conduct the evaluation. PRAMM Consulting Group will also assist in refining the design, providing training and technical assistance, and managing the project.

   2. State Responsibilities. Pilot States will be responsible for:

      identifying a pilot project coordinator who will participate in pilot meetings, assist in refining the design, develop detailed specifications for the pilot and manage the pilot operations, and participate in a debriefing at the end of the project;

      provide UI-qualified investigators for the project and ensure they participate in training for the pilot;

      gather and record relevant time and cost data;

      gather and record both types of accuracy verification data:

      * BQC-type accuracy assessments of all monetary and nonmonetary denials;

      * QPI-type assessments of nonmonetary denials;

      enter all data using the database software provided; and

      mainframe programming (building and maintaining the transaction files from which denials samples will be drawn; installing, testing and maintaining the COBOL or COBOL II sample-selection program).

      Because of their currency with the programming languages (such as COBOL or COBOL II or alternatives) most efficient for sample selection, it is anticipated that two States (one COBOL, another COBOL II or alternative) will write the sample selection program to DOL specifications, as an additionally-funded feature of their pilot agreement.

   3. Costs, Agreements and Resources. The pilot will entail additional costs for the participating States. The main ones are anticipated to be the following:

      the salaries and expenses of the coordinator;

      the costs of additional travel by the pilot coordinator for refining the design, training, and debriefing;

      the salaries and expenses of the investigators;

      the costs of mainframe programming; and

      writing the sample selection program (two States only).

      Following a customary model, the pilots will be conducted according to a simple cooperative agreement between the DOL and the State. Through the agreement the DOL will provide resources reasonably calculated to defray all but salaries and expenses of the investigators. Funding for two investigators plus travel will come through the new UI Performs allocation.

6. Application and Selection. All States interested in participating in a pilot to determine the accuracy with which they are denying benefit claims are encouraged to apply. In applying, States must indicate:

   the name and qualifications of their proposed candidate for pilot coordinator;

   their ability to identify the universes of monetary and nonmonetary denial actions from automated records for the transaction files;

   their ability to build transactions files and implement COBOL programs, and maintain both;

   their interest and capability to write BQC-type (COBOL) sample selection programs;

   their commitment to make UI-qualified investigators available for the pilot; and

   whether their UI laws have features which might affect the difficulty or extent of denials, e.g., the existence of an alternative base-period law.

   In selecting applicants, the DOL will attempt to achieve a reasonable balance among geography, size of States, and law features.

7. Action Required. SESA Administrators are requested to share these materials with appropriate staff. States interested in participating in the pilot should prepare an application covering items indicated in section 6 above. Applications should be sent by <insert 45 calendar days from date of issuance>, 1996, to the appropriate Regional Office (RO).

8. Inquiries and Comments. Direct questions and comments to your RO.

9. Attachment. Paper, "Measuring the Accuracy of Decisions to Deny UI Claims: A Pilot Project."