Evaluating the Joint Action ImpleMENTAL pilot implementation using the Theory of Change framework

Introduction: Mental health problems represent significant public health challenges. In response, one of the two best practices selected to be piloted for implementation in the current EU Joint Action (JA ImpleMENTAL) on mental health (2021-2024) is the Mental Health Reform in Belgium (BMHR). Out of 14 participating Member States (MS), 11 are pilot-implementing aspects of the BMHR. The pilots require an evaluation framework that is beneficial in telling the story of pilot implementation, despite contextual differences across MS, and can set the scene for upscaling after the Joint Action. Therefore, the Theory of Change (ToC) methodology was chosen. This presentation will focus on the application of this approach to evaluation across MS. Method: Individual country-level ToC models have been developed in co-creation with the implementers from each of the 11 MS, via several workshops. Clusters were developed to categorise different elements of the best practices (e.g., “Building, maintaining and sustaining inter-sectoral collaboration/network”).  Currently, a logic framework, containing indicators that aim to evaluate pilot implementation, is being co-created with MS and subsequently used for data collection. Results: Using the ToC as an evaluation framework was new to the majority of MS, but its usability was evaluated positively (4,27/5). During the presentation, cluster selection and preliminary findings from reported output realisation will be presented, together with identified limitations of the evaluation approach. Discussion: Ultimately, the ToC framework will act as a tool for implementers to measure their pilot implementation and relay results to decision-makers. The model can determine where adjustments to implementation strategies, or outputs need to be made to better achieve outcomes and (eventually) the desired impact on a regional/country level. This unique opportunity to synthesise and compare ToC models across many countries is useful for both policy and for directing future health services research.