Survey Non-Response Bias in the Evaluation of the Ready to Work Partnership Grant Program
Survey Non-Response Bias in the Evaluation of the Ready to Work Partnership Grant Program
Publication Info
Description
In 2014, the Department of Labor, Employment and Training Administration awarded four-year grants totaling $170 million to 24 grantees for the Ready to Work Partnership (RTW) Grant Program. RTW partnerships of workforce agencies, training providers, employers, and other local organizations established programs to prepare workers for employment, particularly in occupations and industries being filled by foreign workers through the H-1B visa program. Grant funds were used to provide a range of customized services to the long-term unemployed. In addition to the impact evaluation, the study produced several brief reports, including an analysis of survey non-response bias of participants.
Evaluations often field surveys to gather data on outcomes, such as receipt of training and others, not generally available in administrative data. Given the need to collect non-administrative data, researchers acknowledge that even well-conducted surveys with good contact information and intensive follow-up often have response rates below 80 percent. The potential for differences in outcomes between respondents and non-respondents is called non-response bias. However, when an evaluation also has administrative data on outcomes which is available for the full sample, it can explore the magnitude of the survey non-response bias associated with the survey. Using administrative and survey data collected as part of the Ready to Work (RTW) Evaluation, this study finds no clear evidence that non-response to the follow-up survey generated "survey non-response bias" in estimated impacts on two outcomes measured for the full study sample in administrative data.
- For earnings and employment, calculating impacts on survey respondents provided estimated impacts that were not statistically significantly different than impacts estimated on the full sample (respondents and non-respondents). The estimates of impact using the full sample provide a strong estimate of the true impact of the program because the RTW Evaluation was an experimental impact study.
- For earnings and employment, applying non-response weights did not substantially decrease the difference in estimated impacts between survey respondents and the full sample, although there remained no clear evidence of bias in the weighted estimate of impact.
- A lack of evidence of survey non-response bias in impacts estimated on RTW outcomes measured in administrative data (where data availability for both survey respondents and non-respondents allows a test for bias), suggests there is no clear reason to be concerned about non-response bias in impacts estimated on outcomes measured in RTW survey data (where lack of data for non-respondents does not allow a direct test).