PY 2021 Performance Assessment
U.S. Department of Labor, Employment and Training Administration (ETA)
December 12, 2022
Performance Results
This report assesses state performance based on state reported performance indicator results for PY 2021 compared to the adjusted negotiated levels of performance. The negotiated levels are adjusted based on the estimates from the statistical adjustment models and the actual characteristics of the participants served and economic conditions within the state in PY 2021. This assessment focuses specifically on those indicators and scores that are being formally assessed in 2021.
Some things to note about the PY 2021 Assessments
- The Department of Labor evaluated the statistical adjustment model’s effectiveness in accounting for the impacts of the COVID-19 pandemic, and found that the model sufficiently adjusted for these impacts with respect to the measures being assessed here. The model did however consistently overestimate Employment Rate 2nd Quarter after Exit results, indicating there was variation in the state results that was not represented in the model. The Department of Labor will continue to refine the model with additional data and improved methodologies and work with the Department of Education to ensure fair and objective performance assessments.
- The Departments of Labor and Education issued a notice indicating which primary indicators of performance will be assessed for PY 2020 and PY 2021. The Department of Labor published this as Training and Employment Notice (TEN) 14-21.
- For PY 2021, the only performance indicators being formally assessed are Employment Rate 2nd Quarter after Exit and Median Earnings 2nd Quarter after Exit and the only programs being formally assessed are the WIOA title I (WIOA Adult, WIOA Dislocated Worker, and WIOA Youth) and title III (Wagner-Peyser Act Employment Service) programs.
Performance failure
For PY 2021, there is one way a state can have a performance failure:
- Less than 50% for an individual indicator score (i.e., the score of a specific indicator within a specific program).
The figures below provide an overview of the performance results for states in PY 2021. Any state with at least one individual indicator score failure is shown in purple. States that have a failure on any individual indicator will receive technical assistance from the Department of Labor.
Result Type: Individual Indicator Scores
The figures below have the individual indicator scores. Each tab has all the individual indicators within that program. States have a performance failure if an individual indicator score is below 50%.
These data tables list the values for each of the following data points, which are used to derive the performance scores. For PY 2020 and PY 2021 only individual indicator scores are assessed. The definitions are as follows:
- Estimate0 - the pre-PY prediction for PY 2021. It is calculated using the model estimates and pre-PY values for the model variables.
- Estimate1 - the post-PY prediction for PY 2021. It is calculated using the model estimates and actual-PY values for the model variables.
- Negotiated Level - the agreed upon target level of performance for PY 2021 prior to the start of PY 2021.
- Adjustment Factor the total amount that the original negotiated level was adjusted. It is calculated by taking the difference of Estimate0 and Estimate1.
- Adjusted Level - the final target that actual PY 2021 performance is measured against. It is calculated by applying the Adjustment Factor to the Negotiated Level.
- Actual Level - the actual reported performance result for the performance indicator in PY 2021.
- Score - the final performance score. It is a percent calculated by dividing the Actual Level by the Adjusted Level. Scores over 100 indicate that the state exceeded their adjusted level, while scores less than 100 indicate the state’s outcome was below their adjusted level.
Click the button to download the full target level data shown in the tables below for all states, programs, and indicators.
This table shows the individual indicator performance data for the Adult program indicators.
This table shows the individual indicator performance data for the Dislocated Worker program indicators.
This table shows the individual indicator performance data for the Youth program indicators.
This table shows the individual indicator performance data for the Wagner-Peyser program indicators.
The analysis below gives an overview of how well the models performed in adjusting and assessing performance in PY 2021. This section includes some aspects of the overall performance of the models and highlights the model variables which had a significant impact.
Overall Model Performance
The figures below provide some information on the how well the models performed using PY 2021 data. Overall, there is some variance in the performance of the different models which was primarily driven by issues discussed in the About section of this report. Click on the tabs below to see the figures for each of the program indicator models.
The distribution plots show how the adjustment process (using the statistical adjustment model) affected the performance scores for states. The blue shows the distribution of the performance scores for states in the hypothetical scenario of not using the statistical adjustment model to adjust negotiated levels. In this scenario, actual performance is compared to the original negotiated level. The green shows the distribution of performance scores for states when comparing a state’s actual performance to the adjusted level of performance. Overall, the scores using the model adjustments are more in line with performance expectations. In most cases, there are less high and low outlier scores. In addition, the higher peak for the performance scores (i.e., the green) also indicates less variance with a larger concentration of scores around 100%.
Below the distribution plots are scatter plots that show how the model predictions using the actual PY 2021 data (i.e., Estimate1) compare to the actual performance results reported for each program indicator. These plots highlight that there is some variance in the performance of different program indicator models. Overall, the better performing models have smaller residuals (i.e., points that are closer to the line which would indicate that the prediction matches the actual performance) and those residuals are evenly distributed (i.e., the points are evenly spread around the line).
Plots of the Employment Rate 2nd Quarter after Exit indicator models for each program.
Plots of the Median Earnings 2nd Quarter after Exit indicator models for each program.
Model Variables with Significant Impact in PY 2021
There were some model variables that had a larger impact in determining the adjustment factor and resulting performance score for each program indicator model. The reason that a variable has a large impact can vary, but includes factors such as: the relative importance that element has on a particular performance outcome and the variance in state reporting the data on particular variables. The bar plots below show the variables with the largest average adjustment for each model.
Plots of the variables in the Employment Rate 2nd Quarter after Exit indicator models for each program.
Plots of the variables in the Median Earnings 2nd Quarter after Exit indicator models for each program.
