About the Report

Download Report

The report documents and explores the strengths and drawbacks of data sources commonly used to produce impact estimates for evaluations of workforce development programs. Specifically, researchers use information from the Workforce Investment Act (WIA) Adult and Dislocated Worker Programs Gold Standard Evaluation to examine three data sources used to evaluate the impacts of access to services provided by the public workforce system’s Adult and Dislocated Worker programs, two of the largest, publicly-funded workforce development programs in the nation. The three sources we consider are: (1) evaluation-administered survey data for the study sample; (2) administrative data from the Administration for Children & Families’ National Directory of New Hires; and (3) administrative tax data collected by the Internal Revenue Service, a bureau of the Department of the Treasury. Researchers compare earnings levels and impact estimates based on all three sources to gather insights about the strengths, drawbacks, and tradeoffs inherent in pursuing each data source to answer research questions.

Key Takeaways

• Estimates from each data source showed that intensive workforce development services provided through the Adult and Dislocated Worker program improved earnings. However, the magnitude differed: analysis using administrative data from the NDNH showed smaller impacts than survey data. Tax data also showed smaller impacts than survey data but showed larger impacts than the NDNH. This result is consistent with previous studies.
• Survey and administrative data sources both have strengths and drawbacks. Researchers should be cautious when limiting a study to one type of data source.
  • Survey data are subject to recall error. Respondents tended to overreport earnings per job and underreport the number of jobs held early in the follow-up period. Survey data from later in the follow-up period did not appear to be subject to the same level of over- or underreporting.
  • Administrative data may not capture all work types. Many survey respondents engaged in informal work, alternative work arrangements, day labor, self-employment, that the NDNH did not capture. This study indicated that the NDNH may overlook between one-tenth and one-third of jobs that survey respondents held.

Citation

Mastri, A., Rotz, D., Schwartz, D., Hanno, E.S. (2018). Mathematica Policy Research. Comparing Job Training Impact Estimates using Survey and Administrative Data. Chief Evaluation Office, U.S. Department of Labor.

Download Report   View Study Profile

The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy and CEO’s research development process.