U.S. DEPARTMENT OF LABOR
Employment and Training Administration
Washington, D. C. 20210

CLASSIFICATION

UI

CORRESPONDENCE SYMBOL

TEUDPR

ISSUE DATE

November 9, 1999

RESCISSIONS

None

EXPIRATION DATE

November 30, 2000

DIRECTIVE

:

UNEMPLOYMENT INSURANCE PROGRAM LETTER NO. 06-00
 

TO

:

ALL REGIONAL ADMINISTRATORS
 

FROM

:

GRACE A. KILBANE
Director
Unemployment Insurance Service
 

SUBJECT

:

Results of the Denied Claim Accuracy (DCA) Pilot Project

1.Purpose. To summarize the Denied Claim Accuracy Pilot Project and its findings, indicate how to obtain copies of the Evaluation Report on the project prepared by PRAMM Consulting Group, Inc., and outline plans to implement a nationwide denials accuracy measurement program.

2.Reference. UIPL 27-96, "Pilot Test to Determine the Accuracy of Benefit Denials" (July 12, 1996); UIPL 15-96, "Proposal to Modify the Benefits Quality Control Program" (April 2, 1996).

3.Background. As a result of responses received to its solicitation in UIPL 27-96, July 12, 1996, the Department selected Nebraska, New Jersey, South Carolina, Wisconsin and West Virginia to participate in a pilot test to determine the accuracy of benefit denials using the Benefit Accuracy Measurement (BAM) methodology. The Department conducted a similar 5-State pilot in 1986-87, in which South Carolina also participated. This second pilot used one of the three models tested in the original pilot (by Louisiana). In this pilot, each State agreed to draw weekly cross-sectional random samples of formal denials at the monetary, separation, and weekly eligibility (i.e., nonseparation) determination levels. States agreed to select and verify the accuracy of 200 of each type of denial over the 12 months of the pilot. State sampling began in the 36th week of calendar 1997 and concluded between the 33rd and 35th weeks of 1998. Each denial decision was investigated according to the verification protocol used by BAM to determine the accuracy of paid claims. This means not only reviewing all pertinent agency records, but also contacting claimants, employers, and any relevant third parties to establish facts that pertain to the sampled denial decision. In addition, other agency staff with expertise in applying the Benefit Timeliness and Quality program's Quality Review instrument also reviewed every nonmonetary--separation and non-separation--denial. This was to permit a comparison of findings from the two review methods. Investigative findings for each completed denial were coded into the State's UI database which the Department retrieved electronically. States gathered cost data by coding pilot participants' time into their State cost accounting systems instead of completing special time sheets or time ladders.

For its part, the Department supplied each pilot State with funding for two positions to conduct the investigations, and a supervisor/coordinator position for the duration of the pilot. South Carolina and Wisconsin received additional funding to jointly develop a COBOL program to select the denials samples, which all pilot States used. The Department provided the database software, a simplified version of the BAM software, for data entry and case management. The pilot featured an initial training session in Washington, DC, a mid-session meeting hosted by South Carolina, and a closeout meeting hosted by New Jersey.

4.Pilot Findings. The final report on the Pilot, prepared by PRAMM Consulting Group, Inc., was delivered in May, 1999. Its key findings are:

    a.Error Rates on Denials. The report computed error rates, summarized in Table 1 below, on the number of cases in error. This contrasts with the way Benefit Accuracy Measurement (BAM) errors are typically shown. The BAM error rate is usually computed as dollars overpaid or underpaid as the percentage of dollars paid, although BAM can compute a case-error rate if desired. The report emphasizes that dollar impacts of erroneous denials can only be estimated. The table shows the average error rate, with the range across the pilot States in parentheses below it.

    Total error rates averaged between about 9 and 16 percent, depending on the type of denial:

Monetary denials. The total error rate averaged 16 percent, but agencies were in the process of correcting about one third of them at the time the investigations occurred. None were corrected by appeals or re-determinations; few States allow monetary determinations to be appealed.

Separation denials averaged 8.7%. Appeals eventually corrected one fifth of these errors and internal agency processes a tenth so that the adjusted error rate was about 6.4%.

Non-separation denials initially averaged 15%, but roughly equal combinations of appeals and other corrections reduced that to about 13%.

    These results are broadly consistent with the findings of the 1986-87 pilot, although exact comparisons cannot be made because 4 of the 5 pilot States were different. (South Carolina had about the same monetary and non-separation errors in both pilots; separation errors were lower in the second pilot.)

Table 1

Total and Adjusted Error Rates on Denials, by Type

1997-98 Pilot and 1986-87 Pilot. Range in ( ).

Total Error

Rate

Error Rate After Adjustment for:

Redets & Appeals

Agency Resolution

All Factors

1997-98 Pilot

Monetary 16.0%

(10-23%)

 16.0%

(10-23%)

 11.2%

(8-16%)

 11.2%

(8-16%)

Separation 8.7%

( 3-20%)

 6.8%

( 3-17%)

 8.0%

(3-19%)

 6.4%

(3-16%)

Non-separation 15.0%

( 7-22%)

 14.1%

( 6-20%)

 13.8%

(6-18%)

 12.9%

(6-17%)

1986-87 Pilot

Monetary 23%

(10-36%)

 16%

( 7-23%)

NA

NA

Separation 15%

( 5-29%)

 9%

( 2-25%)

NA

NA

Non-separation 14%

( 7-23%)

 11%

( 6-21%)

NA

NA

Only a small fraction of all decisions results in a denial; the corrected error rates represented 1.0% of the pilot States' monetary determinations, 1.1% of their separations, and 0.3% of non-separations. Nevertheless, if these adjusted error rates are projected to national activity levels for CY 1998, they imply a total of nearly 600,000 uncorrected denials (Table 2). Additionally, the 130,000 initial denials that were eventually corrected by continuing agency processes or appeal still resulted in costs to both the claimants (delayed benefits) and agency (rework).

Table 2

National Estimates of Number of Claimants Whose Benefits Would Have Been Denied or Delayed

Type of Denial Initially Denied Delayed (Corrected after Initial Denial) Ultimately Denied
Monetary 191,000 60,000 131,000
Separation 162,000 21,000 141,000
Non-separation 359,000 49,000 310,000

Table 3 compares the pilot States' case error rates for denials with their rates for allowed claims (taken from BAM data for the period of the pilot). Even the adjusted error rates for monetary and non-separation denials are considerably higher than errors made on decisions to approve. The error rate for claims that were paid erroneously due to separation issues was higher for the pilot States.

Table 3

Case Error Rates on Denials and Approval Decisions

Type of Decision

Denials

Approval

Initial

Adjusted

Monetary 16.0 11.2 7.8
Separations 8.7 6.4 11.2
Non-separation 15.0 12.9 6.4

    b.The Benefit Quality Review vs. the BAM Approach for Assessing Nonmonetary Denials. The Department can assess the quality of nonmonetary determinations in either of two ways.

(1) Since the 1970s it has assessed a broad measure of the quality of nonmonetary adjudications by applying a Quality Performance Indicator (QPI) instrument to data in existing agency records for a random sample of positive and negative determinations. Now called the Benefits Quality Review, its concept of "quality" includes not only accuracy but also relevant dimensions of correct process. Thus, even if the decision to deny is correct, a denial case can fail the QPI review if its process was faulty or the written determination unclear. (The converse is also true.) Using only agency records makes this review quite inexpensive.

(2) The BAM de novo review approach used in this DCA pilot focuses primarily on whether the claim was accurately denied according to existing State law and policy. However, its concept of accuracy is broader than the QPI: whether the decision was the one that would have been made if all relevant facts were known, not merely what had been obtained by the adjudicator and was contained in agency records. Re-contacting all interested parties to obtain all relevant facts makes the review much more costly than the QPI but is needed to give a complete picture of whether the case should have been denied.

    In this pilot, the States compared the two approaches by independently subjecting all nonmonetary denials to both reviews.

Table 4 summarizes the findings. It shows that

  • The QPI review finds many more nonmonetary determinations had inadequate quality than DCA judged to be inaccurate;

    • Over three times as many separation decisions (287/81) and

    Nearly twice as many non-separation decisions (239/143);

  • Due to its narrower concept of accuracy, the QPI review frequently fails to detect inaccurate denial decisions. The following percentages of denials which DCA found inaccurate passed QPI:

    • 33 percent of the separations (27/81)

    45 percent of the non-separations (65/143)

    96% of separations passing QPI (603/630) were accurate, vs. 91% (836/917) of all separation denials;

    90% of nonsep denials passing QPI (607/672) were accurate, vs. 85% (678/911) of all nonsep denials

    DCA found only 19% of separations (54/287) failing QPI to be inaccurate, and

    Only 33% of non-separations (78/239).

    Table 4

    BAM vs.QPI Methodology Results for Separation and Non-separation Denials

    in the 5 Pilot States

    Separation Denials Nonseparation Denials
    Fail QPI Pass QPI Total Fail QPI Pass QPI

    Total

    BAM: Inaccurate

    54

    27

    81

    78

    65

    143

    BAM: Accurate

    233

    603

    836

    161

    607

    768

    Total

    287

    630

    917

    239

    672

    911

    The contractor's report concludes that "The clear conclusion is that the accuracy of nonmonetary denials is best estimated using the BQC [BAM] reassessment and reverification method."

      c. Costs. The pilot suggests that after allowance is made for supervisory review, 2 to 3 staff years, not the two provided in the pilot, are needed to investigate 200 of each kind of denials in a year. The evaluation contractor thus recommends, and the Department agrees, that if no more than 2 positions can be provided for denials investigations, the sample size should be reduced to 150 of each type of denied claim.

      d. Investigative Procedure.

    The States found no difficulty in investigating denials using the BAM methodology. The pilot included interstate cases for the first time. States had no difficulty completing these investigations using mail, phone and fax. The report recommends that all interstate payment cases be included in BAM.

    This pilot confirmed the first pilot finding that contacting denied claimants was much more difficult than those in payment status. Four States completed between 43 and 79 percent of claimant interviews for monetary denials, and 68 to 83 percent for nonmonetary denials. The fifth State, however, completed over 90% of all claimant contacts. The contractor's report recommends setting standards for reasonable attempts to contact claimants and disseminating the best contact practices to all SESAs.

    The difficulty of contacting claimants tended to stretch out completion timeliness, as well. The contractor recommended that timeliness standards be set to help ensure timely completion of cases, although the standards may differ from the timeliness criteria used for paid claims.

    One third of the monetary denials which DCA determined to be erroneous at the time of sampling were in the process of being corrected by Agency processes. The contractor's report thus recommends that for the nationwide program, "sampling of monetary denials should be delayed for up to 14 days so that wage-request processes are completed and most of the denials in the sample are 'true' denials", i.e., claims denied after all wage credits have been reported.

      e. Monetary Impacts of Erroneous Denials. The report finds, as did the report on the previous pilot, that monetary impacts of erroneous denials cannot be directly measured. In addition, there are both conceptual and practical problems with using statistical models to estimate those impacts. However, the report is optimistic that with additional work it may be possible to produce estimates that could serve as "reasonable guides for policy."

    5.Availability of the Report.The final contractor's evaluation report, "Denied Claims Accuracy Pilot Project" by PRAMM Consulting Group, Inc. (May 1999), is available on the ITSC web site at http://www.itsc.state.md.us/data_stats/cover.htm

    A hard copy may be obtained from Burman Skrable, Division of Performance Review, Unemployment Insurance Service, U.S. Department of Labor, Room S-4522, 200 Constitution Avenue, NW, Washington, DC 20210, telephone (202) 219-5223, ext. 140; e-mail, bskrable@doleta.gov.

    6.National Implementation. The Department is now making plans for national implementation of the Denied Claim Accuracy program. The chief design issue is whether national implementation should include samples of separation and non-separation denials or whether the QPI review is sufficient to assess the quality of nonmonetary denials. The policy question is whether the DCA approach adds sufficient information to warrant its greater cost. The national program would not be implemented before October 2000, due to the time needed to obtain the necessary data collection approvals from the Office of Management and Budget, complete software and handbook development, and provide any needed training. These issues will be discussed as part of a workshop on the Denials pilot at the St. Louis UI Directors' meeting in November.

    States can voluntarily begin sampling denied claims before national implementation as a UI PERFORMS continuous improvement initiative. The Department has provided all States with the sample selection program and case management software on the State SUN computer, which allows them to sample and enter investigation data on the I database. States that have not yet installed the BAM COBOL population edit and sample selection program are reminded that they must test this program and send the test results to the Department for approval before installing the program as production software. States are also reminded that the Denied Claims Accuracy software installed on the States' Sun computers was designed to support the pilot study only, and does not contain all the functions of the current BAM software. Resolution of any problems that States encounter using the DCA software will depend on the resources available to the Division of Information Technology.

    7.Action Required. SESA administrators are requested to (a) provide this information to interested staff; and (b) encourage staff to attend the workshop on denials accuracy measurement at the UI Directors' meeting in St. Louis.

    8.Inquiries. Please refer inquiries to the appropriate Regional Office.