Supporting Policy Implementation: Methods and Tools for Assessing Evaluation Efforts

Objective(s): To describe the methods, tools, and prospective impacts of using these tools on policy implementation.

Data Sources: We conducted content analysis of the PACT Act legislation text, and an environmental scan of literature, internet, official communication searches, and stakeholder meetings, to identify relevant PACT Act evaluations.

Study Selection: The study team identified over 60 potentially relevant evaluations to the PACT Act.

Data Extraction: We created a toolkit consisting of three key tools to aid stakeholders in decision-making: the Evaluation Monitoring Dashboard, Evaluation Assessment Tool, and Evaluation Opportunity Assessment Tool. The Monitoring Dashboard is a repository for stakeholders to use to monitor existing evaluations that were identified from the qualitative analysis and ongoing surveillance. The Evaluation Assessment Tool comprises key quality domains, sourced from the Office of Management and Budget’s evaluation standards. This tool summarizes major findings from ongoing and completed evaluations and assesses quality and completeness to identify additional opportunities for evaluation. The Evaluation Opportunity Assessment Tool explores novel evaluation ideas, their relevance, potential policy impacts, and feasibility.

Data Synthesis: In piloting these tools, we reviewed 25 existing evaluations and 43 stakeholder-nominated evaluation ideas. Formative evaluation of the process and tools elicited positive feedback from stakeholders.

Conclusions: A structured and replicable process enhances leaders’ ability to track and act upon evaluation activities in a large, complex federal agency tasked with implementing legislation mandating vast and diverse change. Using a participatory approach, where key stakeholders play an integral role in structure, process and tool development, resulted in a novel approach to identify, monitor, and explore evaluations relevant to the PACT Act.

Author(s) Disclosures: none