About the Brief

Download Report

The brief reports results of an exploration of survey non-bias using data collected for the evaluation of the Ready to Work (RTW) Partnership Grant program. Additional detail on the RTW program and its evaluation are also provided.

Download this Summary (PDF)

Key Takeaways

• Using administrative and survey data collected as part of the Ready to Work (RTW) Evaluation, this study finds no clear evidence that non-response to the follow-up survey generated “survey non-response bias” in estimated impacts on two outcomes measured for the full study sample in administrative data.
• Namely, for earnings and employment, calculating impacts on survey respondents provided estimated impacts that were not statistically significantly different than impacts estimated on the full sample (respondents and non-respondents). Because the RTW Evaluation was an experimental impact study, estimates of impact using the full sample provides a strong estimate of the true impact of the program.
• Although evaluations normally apply weights based on baseline characteristics when estimating impacts with survey-based outcomes, those weights may not address survey non-response bias, or provide impact estimates that are more similar to those for the full sample, if the outcomes for survey respondents are different from the outcomes for non-respondents.
• For earnings and employment, in the RTW Evaluation applying non-response weights did not substantially decrease the difference in estimated impacts between survey respondents and the full sample, although there remained no clear evidence of bias in the weighted estimate of impact.
• A lack of evidence of survey non-response bias in impacts estimated on RTW outcomes measured in administrative data (where data availability for both survey respondents and non-respondents allows a test for bias), suggests there is no clear reason to be concerned about non-response bias in impacts estimated on outcomes measured in RTW survey data (where lack of data for non-respondents does not allow a direct test).

Citation

Herr, J. L. and Klerman, J.A. (2022). Abt Associates. Survey Non-Response Bias in the Evaluation of the Ready to Work Partnership Grant Program. Chief Evaluation Office, U.S. Department of Labor.

Download Report   View Study Profile

This study was sponsored by the Employment and Training Administration, Office of Policy Development and Research, Division of Research and Evaluation, and was produced outside of CEO’s standard research development process.