1. Purpose. To inform the workforce investment system of the data validation initiative for employment and training programs.
2. Background. In 2001, the President announced a Management Agenda to improve the management and performance of the federal government. One of the five government-wide goals, budget and performance integration, emphasizes the importance of complete information for program monitoring and improving program results. A performance audit conducted by the Department's Office of Inspector General found that the accuracy of state-reported performance outcomes cannot be assured due to insufficient local, state, and federal oversight. To address these concerns, and to ensure the accuracy of data collected and reported on the workforce investment system, the Employment and Training Administration (ETA) is expanding efforts to conduct data validation.
ETA has already begun implementation of data validation for the Unemployment Insurance program and is now developing a validation process for the Agency's employment and training programs. Initially, this new data validation initiative will encompass the following programs: WIA Title IB, Employment Service, Trade Adjustment Assistance, Migrant and Seasonal Farmworkers, Senior Community Service Employment Program, and Native American Employment and Training.
3. Data Validation Initiative. The data validation initiative covers both the accuracy of aggregate reports submitted to ETA on program activity and performance outcomes and the accuracy of individual data elements. The data validation process consists of two parts:
- Report validation ensures that aggregate reports are calculated correctly by processing an entire file of participant records into validation counts and comparing the validation counts to those reported by the state or grantee.
- Data element validation ensures that the data elements in participant records used to calculate aggregate reports are accurate by reviewing samples of participant records against source documentation to ensure compliance with federal definitions.
- Data Validation Tools
ETA has developed standardized software and instructional handbooks that states and grantees can use to perform data validation.
- For report validation, the standardized software will validate the accuracy of aggregate reports that are generated by state or grantee software and will produce an error rate for each reported count.
The standardized software can also be used to generate aggregate reports submitted to ETA. States or grantees that submit aggregate reports using the software provided by ETA will not be required to perform report validation.
- For data element validation, the standardized software generates a sample of the participant records and data elements the state or grantee will validate. The software produces worksheets on which the validator records information after checking the source documentation in the sampled case files. The software calculates error rates for each data element, with confidence intervals of 3.5% for large states/grantees and 4% for small states/grantees.
Data validation software and handbooks for each program will be posted on ETA's Web site at http://www.uses.doleta.gov/dv/ as these tools are developed.
Data Validation Framework
As part of the data validation initiative, ETA is developing a policy framework for conducting validation that will include the following areas:
- Frequency of Validation. Ensuring the accuracy of data necessitates conducting validation on a periodic basis. ETA may propose that validation be conducted annually to correspond with current reporting cycles and to avoid over burdening states and grantees.
- Accuracy Standards. For the data validation initiative to be effective and to allow for continuous improvement, acceptable levels for the accuracy of reports and data elements will need to be established. In setting accuracy standards, ETA expects to pursue a phased approach in which standards are not set immediately. Initial implementation of data validation will focus on detecting and resolving any issues with data and reporting systems. Accuracy standards will be established over time.
- Corrective Action. Once accuracy standards are established, states and grantees should be held accountable for meeting those standards and be required to address any issues with data accuracy.
ETA will release a policy on data validation in the coming months.
Data Validation Implementation
ETA will provide the necessary training and technical assistance to states and grantees to implement data validation. ETA is currently working with pilot states and has refined the validation software and handbooks based on their experiences. The Employment Service has already begun a voluntary report validation effort.
ETA expects to implement data validation in two phases:
- States begin validating data for WIA Title IB, Trade Adjustment Assistance, and the Employment Service by the end of CY2003. States would finish validation for these programs and submit output reports containing error rates to ETA in spring 2004.
- Grantee programs, including Migrant and Seasonal Farmworkers, Senior Community Service Employment Program, and Native American Employment and Training, begin validating data in spring 2004.
4. Action Required. Please make this information available to appropriate program staff.
5. Inquiries. Questions regarding the data validation initiative should be directed to the appropriate regional office or grant administrator.