About the Report

Download Report

The evidence capacity assessment included the 16 U.S. Department of Labor (DOL) agencies in the Department’s Strategic Plan. It reflects data collected through a survey of targeted DOL staff, focus groups with selected DOL staff, and a review of selected evidence documents. The research team used a strengths-based approach that recognizes and leverages DOL’s and agencies’ progress and embodies a continuous improvement mindset. DOL’s Chief Evaluation Office (CEO) and the research team collaborated to develop this summary of key survey and focus group findings that seem particularly actionable. In reviewing the findings, it is important to note that they are not representative of the Department as a whole. The survey and focus group findings have several important limitations: (1) the survey used a convenience sample of GS-13s and above even though staff in other grade levels also use and produce evidence; however, future surveys may include a broader sample of GS levels; (2) the survey targeted staff in occupations for which evidence use and production were assumed to play larger roles compared to other occupations; however, future surveys may include a broader list of relevant occupations; (3) the total number of staff in the survey was 3,446, but only a quarter responded to the survey; and (4) the focus groups included 125 purposively selected staff from 15 agencies; however, future focus groups may be expanded to provide additional perspectives on evidence use and production.

Key Takeaways

• The Department and its agencies are building on strengths that include a strong sense of mission and strategic goals; thoughtful visions of improved evidence use; and strong leadership support for evidence-based decision making.
• Data access is a multifaceted constraint cited across agencies. Agency staff need better data systems and processes to reduce costs of accessing and merging existing data, the ability to access and merge data from other agencies and access data sets from external parties, and access more granular data collection.
• Data visualization software can be game-changing in helping agencies use data, but several agencies indicated a lack of necessary tools or the training for their use. Agency staff need data visualization tools and training to generate customized, just-in-time analyses.
• Several challenges interfere with agencies’ ability to gather and use evidence that is aligned to their decision-making needs. For example, agencies struggle with evaluation results becoming available after program, policy, or context changes make their findings less relevant.
• Competing demands, understaffed agencies, and time-consuming processes strain the ability of agency staff to focus on efficient use and production of evidence.
• Agency leadership sets the stage for a strong culture of evidence use and production. Staff generally feel supported by their supervisors and agencies in using and producing evidence but desire greater and more consistent support over time. For example, there is a need for more consistent championing of evidence strategies across political cycles.

Citation

Corea, C., Amin, S., Ahlstrand, A., Cadima, J., Stockham, J., Rege, G. (2022). American Institutes of Research. Supporting an Evidence-Driven Culture: U.S. Department of Labor’s Evidence Capacity Assessment. Chief Evaluation Office, U.S. Department of Labor.

Download Report   View Study Profile

The Department of Labor’s (DOL) Chief Evaluation Office (CEO) sponsors independent evaluations and research, primarily conducted by external, third-party contractors in accordance with the Department of Labor Evaluation Policy. CEO’s research development process includes extensive technical review at the design, data collection and analysis stage, including: external contractor review and OMB review and approval of data collection methods and instruments per the Paperwork Reduction Act (PRA), Institutional Review Board (IRB) review to ensure studies adhere to the highest ethical standards, review by academic peers (e.g., Technical Working Groups), and inputs from relevant DOL agency and program officials and CEO technical staff. Final reports undergo an additional independent expert technical review and a review for Section 508 compliance prior to publication. The resulting reports represent findings from this independent research and do not represent DOL positions or policies.