About the Report

Download Report

The U.S. Department of Labor (DOL) sponsored a process study of the two Pay for Success (PFS) pilots, conducted by Abt Associates, to document project implementation and provide information on the PFS approach to policymakers and program administrators. This is the process study’s report. An earlier interim report documented the development of the pilots and their initial implementation. This final report updates information about the implementation of the pilots and includes observations based on the entirety of the DOL supported portion, or first phase, of the PFS grants. Specifically, this report documents the services provided to participants through the PFS interventions, the implementation and management of the RCT evaluations, and the evaluation results, including whether target outcomes were met, for the first phase of the pilots.

Download this Summary (PDF)

Research Questions

• How were the PFS pilots designed, and what were the national and local contexts in which they were developed?
• How did grantees identify, recruit, and maintain partnership organizations for their PFS pilots?
• How is the PFS initiative managed and operated?
• How was the PFS financing structure developed and operated?
• How does the service delivery operate and what services were provided to participants?
• Were the target outcomes achieved and payment to investors made?
• What are the challenges and promising practices experienced by the PFS projects?

Key Takeaways

• Project partners were optimistic about achieving outcome targets in the second phase, in spite of not achieving target outcomes in the first phase. Both projects brought on required partners, implemented the service model, and conducted rigorous evaluations. Neither project reached the pre-specified outcome targets, so they did not request payment from DOL.
• Conducting a randomized control trial (RCT) was reported to be one of the most difficult aspects of the PFS project. Each project experienced difficulties with some areas of implementing an RCT, including obtaining adequate sample size, high levels of participation in program services among the treatment group, a strong treatment-control contrast, and high quality and accessible data.
• The PFS sponsor was suggested to be specific about the required reporting on the RCT methodology and factors that may have affected the interpretation of results. Based on the experience the DOL pilot projects, it is important to provide detailed requirements related to how final results should be presented and what issues will be addressed. Other suggested considerations include being transparent about evaluation plans and progress through the life of the project, and having the sponsor clearly outline the responsibilities of and scope of work for the validator.
• The two PFS pilot projects required substantial time and resource investment by the key partners. Both pilots were able to successfully develop and manage the projects, but partners reiterated that doing so took a great deal of time. Staff changes were especially challenging over the course of the grants. Both states reported that they invested substantial staff time in developing, managing and overseeing the project.

Citation

Kean, S., Schwartz, D., Sarna, M., Martinson, K. (2020). Abt Associates. Process Study of the U.S. Department of Labor’s “Pay for Success” Pilots in Two States: Final Report”. Employment and Training Administration, U.S. Department of Labor.

Download Report   View Study Profile

This study was sponsored by the Employment and Training Administration, Office of Policy Development and Research, Division of Research and Evaluation, and was produced outside of CEO’s standard research development process.