DIRECTIVE	:	UNEMPLOYMENT INSURANCE PROGRAM LETTER NO. 35-96
TO	:	ALL STATE EMPLOYMENT SECURITY AGENCIES
FROM	:	MARY ANN WYRSCH Director Unemployment Insurance Service
SUBJECT	:	Additional Guidance Concerning Sampling for Quality Measures for Unemployment Insurance (UI) Benefits Operations

1. Purpose. To provide policy on the sampling procedures to be followed by the States for the Benefits Quality measures.

2. References. UIPL 10-96, Implementation of New Time Lapse and Quality Measures for UI Benefits Operations and ET Handbook No. 401, 2nd Edition, Change 7.

3. Background. Implementation of UI Bene its Quality measures, effective July 1, 1996, was announced in ET Handbook No. 401, 2nd Edition, Change 7 and UIPL 10-96. In June 1996, the Department announced that although the implementation of several new and revised reports will be delayed until January 1, 1997, the implementation of ETA 9056, Nonmonetary Determinations Quality Review, and ETA 9057, Lower Authority Appeals Quality Review, will proceed as planned. The Department also announced that some minor clarifications of the sampling instructions for the Benefits Quality measures would be issued.

4. Policy.

   1. Sample Sizes.

      Appendix A of ET Handbook No. 401, 2nd Edition, Change 7 (page A-5), states that the sample sizes for the nonmonetary determinations and lower authority appeals quality reviews are determined by the total number of nonmonetary determinations and lower authority appeals reported by the State in the preceding calendar year (CY). Attachment A of UIPL 10-96 (page 35) states that the sample sizes will be based on the number of nonmonetary determinations and lower authority appeals reported in the prior fiscal year. To resolve this conflict, sample sizes will be based on the calendar year totals for the preceding year.

      The sampling frames for the nonmonetary determinations quality samples are the universe data in the ETA 9052 Nonmonetary Determinations Time Lapse, Detection Date report. The sampling frames for the lower authority appeals quality samples are the universe data in the ETA 9054 Appeals Time Lapse report. Because these reports will not be implemented until January 1, 1997, data for the first complete CY will not be available until January 1998. The following guidelines will be used to determine State sample sizes for both nonmonetary determinations and lower authority appeals beginning with the third quarter of CY 1996 through the fourth quarter of CY 1997.

      1. Nonmonetary Determinations

         As noted above, the number of cases to sample for the nonmonetary determinations quality review depends on the number of nonmonetary determinations reported by the State in the preceding CY. For the quarterly samples which will be selected beginning in October 1996 from the July-September 1996 universe of determinations and in January 1997 from the October-December 1996 universe of determinations, the sample size will be determined by the number of nonmonetary determinations reported in the quarterly ETA 207 reports for CY 1995. The total number of nonmonetary determinations will be the sum of line 101 column 2 and line 101 column 4.

         If the sum of these columns equals 100,000 or more for CY 1995, the State will select 50 separation determinations and 50 nonseparation determinations for the quarterly samples selected from the July-September 1996 and October-December 1996 transaction files (sampling frames). States with fewer than 100,000 nonmonetary determinations will select 30 separation and 30 nonseparation cases per quarter from the July-September 1996 and October-December 1996 transaction files. State sample allocations for the July-September 1996 and October-December 1996 nonmonetary determinations samples are provided in Attachment A to this UIPL.

         For the four quarterly samples selected from the nonmonetary determinations transactions files created during CY 1997, the total number of nonmonetary determinations reported in the ETA 207 report during CY 1996 will determine the State sample sizes.

         For quarterly samples selected from the nonmonetary transactions files beginning in the first quarter of CY 1998, the total nonmonetary determinations reported in the ETA 9052 Nonmonetary Determinations Time Lapse reports for CY 1997 will be used to determine the State sample sizes.

         Note: Use of the ETA 207 report is limited to determining the size of the nonmonetary determinations quality samples. As stated in the sampling instructions for the nonmonetary determinations quality review in Change 7 to ET Handbook 401 (page V-7-3), "The sample universe is based on the time lapse data reported on the ETA 9052 for each month in the review quarter." The universe of nonmonetary determinations, as currently defined for the ETA 207 report, does not include all of the determinations which are to be included in the sampling frames for the nonmonetary determinations quality review.

      2. Lower Authority Appeals

         The number of cases to sample for the lower authority appeals quality review depends on the number of lower authority appeals reported by the State in the preceding CY. For the quarterly samples which will be selected beginning in October 1996 from the July-September 1996 universe of lower authority appeals and in January 1997 from the October-December 1996 universe of lower authority appeals, the sample size will be determined by the number of lower authority appeals reported on line 100, column 1, in the monthly ETA 5130 reports for CY 1995.

         If the CY 1995 total equals 40,000 or more, the State will select 40 lower authority appeals per quarter from the July-September 1996 and October-December 1996 transaction files (sampling frames). States with fewer than 40,000 lower authority appeals will select 20 lower authority appeals from the July-September 1996 and October-December 1996 transaction files. State sample allocations for the July-September 1996 and October-December 1996 lower authority appeals samples are provided in Attachment B to this UIPL.

         For the four quarterly samples selected from the lower authority appeals transactions files created during CY 1997, the total number of lower authority appeals reported in the ETA 5130 reports during CY 1996 will determine the State sample sizes.

         For quarterly samples selected from the lower authority appeals reported beginning in the first quarter of CY 1998, the total lower authority appeals reported in the ETA 9054 Appeals Time Lapse reports for CY 1997 will be used to determine the State sample sizes.

   2. Sorting the Sampling Frame.

      1. Nonmonetary Determinations

         Appendix A of ET Handbook 401, 2nd edition, change 7 (page A-5), instructs the State Employment Security Agencies (SESAs) to sort the universe file by separation (codes 10-29) and nonseparation (codes 30-99) issues. However, no other sort criteria are specified. The goal of increased statistical validity is supported by the use of systematic random sampling techniques to obtain samples. This implies that any sample pulled will reflect the way in which the sampling frame is sorted. This section will provide guidance to replace/supplement ET Handbook 401.

         In order to insure consistency among the States, the primary sort key of the nonmonetary determinations sampling frames should be Issue Code (data element 2, skeleton data element 2) (ascending). The secondary sort key should be Date Determination Issued (data element 25, skeleton data element 5) (ascending). Thus, the first record in the sampling frame file for separations will be the determination with issue code 10 (quit) with the earliest date of issue, and the last record in the file will be the determination with issue code 29 (other separation issue) with the latest date of issue. Similarly, the first record in the sampling frame file for nonseparations will be the determination with issue code 30 (able and available) with the earliest date of issue, and the last record in the file will be the determination with the highest issue code up to 99 (multiclaimant other) with the latest date of issue.

      2. Lower Authority Appeals

         Appendix A of ET Handbook 401, 2nd edition, change 7 (page A-5), provides no instructions to the States on sorting the universe file of lower authority appeals. In order to insure consistency among the States, the primary sort key of the lower authority appeals sampling frame should be Date Decision Issued (data element 35, skeleton data element 2) (ascending).

   3. Randomization.

      UIPL 10-96 (Attachment A, page 35) gives States the option of randomizing the transaction files and selecting the first 30 or 50 nonmonetary determinations (or 20 or 40 lower authority appeals records) for the sample. States may chose to use this method as an alternative to systematic sample selection.

   4. Sample Cases Excluded from Quality Scoring.

      If the sampling frame contains records that do not fit the definition for the target population (universe), there is a chance that these cases could be selected for the sample. States should undertake every effort to eliminate these unwanted listings from the sampling frame before the sample is selected. If, however, the sampling frame contains listings that do not meet the definition but are selected for the quality samples, they must be identified so that they are excluded from calculation of the State's quality score.

      Sample cases which meet the definition for the target population but cannot be evaluated because case materials are missing must also be identified so that they are excluded from calculation of the State's quality score.

      1. Nonmonetary Determinations

         If a case is selected for the nonmonetary determination quality sample that does not meet the definition for inclusion in the population, the case should be coded as follows:

         If the determination does not meet the definition of a reportable action, as defined in ET Handbook 401, 2nd edition, change 7, enter code 'N' in item 4 (Correct Issue Code?) and enter code '00' in item 5 (Corrected Issue Code) of the data collection instrument. The remaining data fields will be left blank. Several examples of nonmonetary determinations which are not reported in the ETA 207 definitions are listed on pages I-4-4 and I-4-5 of ET Handbook 401, 2nd edition, change 7.

         Nonmonetary redeterminations, which are reported in the ETA 207 report, are excluded from the ETA 9052 report, which serves as the sampling frame for the nonmonetary determinations quality review. If a nonmonetary redetermination is selected in the sample, enter code 'N' in item 4 (Correct Issue Code?) and enter code '01' in item 5 (Corrected Issue Code) of the data collection instrument. The remaining data fields will be left blank.

         If the determination does meet the definition of a reportable action, as defined in ET Handbook 401, 2nd edition, change 7, but cannot be evaluated for quality because case materials cannot be located, enter code 'N' in item 3 of the data collection instrument. The remaining data fields will be left blank.

      2. Lower Authority Appeals

         If the lower authority appeal meets the definition in ET Handbook 401, 2nd edition, change 7, but cannot be evaluated for quality because of an inaudible or missing tape and/or missing case materials, enter the appropriate code in item 37 on the ETA 9057 State evaluation score sheet. The remaining data fields will be left blank.

   5. Sampling Method.

      Appendix A of ET Handbook 401, 2nd edition, change 7, offers the States several alternatives to select the nonmonetary determinations and lower authority appeals quality samples:

      Selection using two COBOL programs that were developed for the UI Revenue Quality Control (RQC) program, PICKNMBR and SAMPS0nn, which need to be modified for the Benefits Quality samples.

      Use of a commercial software package such as Easytrieve or SAS.

      Manual sample selection.

      States may choose any of these methods to select the Benefits Quality samples. The following information is provided to assist States in implementing these methods and to insure the validity of the results.

      States can run the PICKNMBR program using codes in the control record which are valid for RQC. However, the hard copy output report created by PICKNMBR will reflect these RQC codes. For example, Record Type code "CS011" and Transaction Type "1" will produce a report with the heading, "STATUS DETERMINATION - NEW". The State will have to distinguish between the three Benefit Quality samples -- separations, nonseparations, and lower authority appeals -- by, for example, using the three different Transactions Type codes that PICKNMBR accepts.

      Alternatively, a State can modify the sections of the program which control the printing of the output report to properly identify the appropriate Benefits Quality sample.

      The following information must be used in the control record for PICKNMBR.

      Number to be Sampled - Will equal 30 or 50 for nonmonetary determinations; will equal 20 or 40 for lower authority appeals.

      Record Count - Enter the number of records in the transaction file from which the sample is selected (sampling frame).

      There is one significant limitation of the PICKNMBR COBOL program developed for UI RQC sampling. The control record for this program uses a three-digit random number to designate the first case to be selected for the sample from the skip interval. Because only 1000 initial selections are possible with a three-digit random number, for skip intervals (K) greater than 1000, K-1000 records will have no chance of being selected in the initial selection. States with large numbers of nonmonetary determinations or lower authority appeals, which result in skip intervals greater than 1000, will not be able to use this program unless it is modified to expand the random number field to four places.

      If a State chooses not to modify the PICKNMBR program to accommodate the larger random numbers, the State will have to write a routine using a commercial software package (such as Easytrieve, SAS or SPSS) or use a manual selection routine. This routine must use the formulas on pages A-8 and A-9 in Appendix A of ET Handbook 401, 2nd Edition, Change 7, or another algorithm which produces a statistically valid random sample.

      Appendix A of change 7 to ET Handbook 401 references the RQC SAMPS0nn programs, specifically the RQC Status Determination sample selection program. While this program can serve as a general model for the development of a sample selection program specific to the Benefits Quality program, the RQC SAMPS0nn programs would require extensive modifications for use in selecting nonmonetary determinations or lower authority appeals samples. States are discouraged from undertaking this task.

      States are reminded that, as stated in change 7 to ETA Handbook 401 (2nd Edition), "States are responsible for creating the programs/utilities necessary to extract the data elements for the universe files for each [benefit quality] sample." The Department cannot provide programming resources to write or modify sample selection routines.

   6. Random Numbers.

      Appendix A of ET Handbook 401, 2nd edition, change 7 (page A-7), advises States to obtain random numbers from "a list of random numbers supplied in any statistics handbook". Random numbers are presented differently in the various statistics texts. For example, some texts list two, three or four-digit random integers. If the State chooses to use the sample selection formula described on pages A-8 and A-9, these integers must be converted to decimals, because the formulas require a decimal between 0 and 1. On the other hand, the PICKNMBR program accepts an integer input and converts this number to a decimal.

      As discussed in section 4 (e), the random number must retain a sufficient number of places to give each record in the universe as chance of being selected. For example, if the skip interval is 1000 and the random number retains only one decimal place (.1, .2, etc.) only records 100, 200, etc. have a chance of being chosen.

      To simplify the procedure for the States and insure that the random numbers will meet the requirements of the proposed sampling methodology, the following random numbers are provided for the quarterly samples through CY 1997. These random numbers are presented as decimals carried out to four places. States that do not need to modify the PICKNMBR program to accommodate large skip intervals, as discussed in section 4 (e), can enter the first three places of the random numbers as integers in the PICKNMBR control record. For example, the random number for the separations determinations sample for the July-September 1996 quarter should be entered in the control record as 419.

      Table of Random Numbers for Benefits Quality Samples

      Quarterly Sample For:	Separation Determinations	Nonseparation Determinations	Lower Authority Appeals
      July-September 1996	.4195	.4038	.6518
      October-December 1996	.9201	.1965	.3014
      January-March 1997	.5941	.1115	.2040
      April-June 1997	.4336	.6795	.0701
      July-September 1997	.7983	.3968	.6027
      October-December 1997	.1010	.2031	.6944

5. Action Required. SESA administrators are requested to provide copies of this UIPL to appropriate staff.

6. Inquiries. Direct inquiries to the appropriate Regional Office or to Andrew Spisak, at the National Office, on 202-219-5922.

Note: Click here for Attachment A--Nonmonetary Determination Quality Samples State Sample Allocations

Click here for Attachment B--Lower Authority Appeals Quality Samples State Sample Allocations