PY 2022 Performance Assessment
U.S. Department of Labor, Employment and Training Administration (ETA)
U.S. Department of Education, Office of Career, Technical, and Adult Education (OCTAE)
U.S. Department of Education, Rehabilitation Services Administration (RSA)
January 08, 2024
Performance Results
This report assesses state performance based on state reported performance indicator results for PY 2022 compared to the adjusted negotiated levels of performance. The negotiated levels are adjusted based on the estimates from the statistical adjustment models and the actual characteristics of the participants served and economic conditions within the state in PY 2022. This assessment focuses specifically on those indicators and scores that are being formally assessed in 2022.
Some things to note about the PY 2022 Assessments
The Departments of Labor and Education issued a notice indicating which primary indicators of performance will be assessed for PY 2022. The notice was published as Training and Employment Notice (TEN) 04-23, Program Memorandum 23-3, and FAQ 23-01 for each agency.
For PY 2022, the performance indicators being formally assessed are:
For the WIOA title I and IV Programs (WIOA Adult, WIOA Dislocated Worker, WIOA Youth, Vocational Rehabilitation):
- Employment Rate 2nd Quarter after Exit
- Median Earnings 2nd Quarter after Exit
- Employment Rate 4th Quarter after Exit
- Credential Attainment Rate
For the WIOA title III program (Wagner-Peyser Act Employment Service):
- Employment Rate 2nd Quarter after Exit
- Median Earnings 2nd Quarter after Exit
- Employment Rate 4th Quarter after Exit
- Overall Program Score
Performance failure
For PY 2022, there are two ways that a state can have a performance failure:
- Less than 50% for an individual indicator score (i.e., the score of a specific indicator within a specific program).
- Less than 90% for an overall program score (i.e., the average of all individual indicator scores within a program)
The figures below provide an overview of the performance results for states in PY 2022. Any state with at least one individual indicator score failure is shown in purple. States that have a failure on any individual indicator will receive technical assistance from the Departments of Labor and Education.
Overall Performance
The figure below indicates which states did not meet their adjusted levels of performance (performance failure) in PY 2022. A failure for any type of performance score would result in that state having a performance failure.
Result Type: Overall Performance Scores
The figures below have the overall state indicator and overall state program scores. The overall indicator score is the average of a particular individual indicator score across programs. The overall program score is the average of all individual indicators within a program. States have a performance failure if an overall score is below 90%.
There is one figure for each program that can be assessed in PY 2022.
Result Type: Individual Indicator Scores
The figures below have the individual indicator scores. Each tab has all the individual indicators within that program. States have a performance failure if an individual indicator score is below 50%.
These data tables list the values for each of the following data points, which are used to derive the performance scores. The definitions are as follows:
- Estimate0 - the pre-PY prediction for PY 2022. It is calculated using the model estimates and pre-PY values for the model variables.
- Estimate1 - the post-PY prediction for PY 2022. It is calculated using the model estimates and actual-PY values for the model variables.
- Negotiated Level - the agreed upon target level of performance for PY 2022 prior to the start of PY 2022.
- Adjustment Factor the total amount that the original negotiated level was adjusted. It is calculated by taking the difference of Estimate0 and Estimate1.
- Adjusted Level - the final target that actual PY 2022 performance is measured against. It is calculated by applying the Adjustment Factor to the Negotiated Level.
- Actual Level - the actual reported performance result for the performance indicator in PY 2022.
- Score - the final performance score. It is a percent calculated by dividing the Actual Level by the Adjusted Level. Scores over 100 indicate that the state exceeded their adjusted level, while scores less than 100 indicate the state’s outcome was below their adjusted level.
Click the button to download the full target level data shown in the tables below for all states, programs, and indicators.
This table shows the overall program score data for each program.
This table shows the individual indicator performance data for the Adult program indicators.
This table shows the individual indicator performance data for the Dislocated Worker program indicators.
This table shows the individual indicator performance data for the Youth program indicators.
This table shows the individual indicator performance data for the Wagner-Peyser program indicators.
This table shows the individual indicator performance data for the Vocational Rehabilitation program indicators.
The analysis below gives an overview of how well the models performed in adjusting and assessing performance in PY 2022. This section includes some aspects of the overall performance of the models and highlights the model variables which had a significant impact.
Overall Model Performance
The figures below provide some information on the how well the models performed using PY 2022 data. While there are a few outlier results, these seem to be driven by state performance rather than the performance of the model. Click on the tabs below to see the figures for each of the program indicator models.
The distribution plots show how the adjustment process (using the statistical adjustment model) affected the performance scores for states. The blue shows the distribution of the performance scores for states in the hypothetical scenario of not using the statistical adjustment model to adjust negotiated levels. In this scenario, actual performance is compared to the original negotiated level. The green shows the distribution of performance scores for states when comparing a state’s actual performance to the adjusted level of performance. Overall, the scores using the model adjustments are more in line with performance expectations. In most cases, there are less high and low outlier scores. In addition, the higher peak for the performance scores (i.e., the green) also indicates less variance with a larger concentration of scores around 100%.
Below the distribution plots are scatter plots that show how the model predictions using the actual PY 2022 data (i.e., Estimate1) compare to the actual performance results reported for each program indicator. These plots highlight that there is some variance in the performance of different program indicator models. Overall, the better performing models have smaller residuals (i.e., points that are closer to the line which would indicate that the prediction matches the actual performance) and those residuals are evenly distributed (i.e., the points are evenly spread around the line).
Plots of the Employment Rate 2nd Quarter after Exit indicator models for each program.
Plots of the Median Earnings 2nd Quarter after Exit indicator models for each program.
Plots of the Employment Rate 4th Quarter after Exit indicator models for each program.
Plots of the Credential Attainment Rate indicator models for each program.
Model Variables with Significant Impact in PY 2022
There were some model variables that had a larger impact in determining the adjustment factor and resulting performance score for each program indicator model. The reason that a variable has a large impact can vary, but includes factors such as: the relative importance that element has on a particular performance outcome and the variance in state reporting the data on particular variables. The bar plots below show the variables with the largest average adjustment for each model.
Plots of the variables in the Employment Rate 2nd Quarter after Exit indicator models for each program.
Plots of the variables in the Median Earnings 2nd Quarter after Exit indicator models for each program.
Plots of the variables in the Employment Rate 4th Quarter after Exit indicator models for each program.
Plots of the variables in the Credential Attainment Rate indicator models for each program.
