As Member States work towards finalising their energy and climate plans by June 2024, there is a pressing need to intensify efforts to achieve the strict energy and climate objectives by 2030. To meet these targets, policies and measures should be designed and implemented in an effective, efficient, coherent, and relevant manner.

Pre-implementation or ex-ante evaluations play a crucial role in enabling policymakers to understand and compare the potential effects of these planned policies. However, once a policy is implemented, ex-post evaluations offer valuable insights into its performance, guiding future decision-making.

In this position paper, we elaborate on the importance of ex-post evaluations, emphasising their role in fostering evidence-based and effective policy interventions. Drawing on insights from the Horizon project PATTERN and the European Topic Centre on Climate Change Mitigation, we shed light on current evaluation practices in European climate policy, identifying gaps and opportunities for improvement. Furthermore, we explore the myriad challenges policymakers encounter in evaluating environmental and climate policies, offering recommendations to enhance evaluation practices and bolster informed decision-making.

This position paper came about thanks to the expertise of our VITO/EnergyVille researchers Nele Renders, Tom Dauwe, Kelsey van Maris and Annabel Vella.

Current evaluation practices in European climate policy

The critical role of evaluation  

While policy evaluation is widely recognised and supported by substantial scientific research, there remains a significant gap in evaluating climate and energy policies across European countries, including Belgium. Evaluations play a critical role under the Governance Regulation to meet the strict 2030 climate and energy targets. Thus, designing and implementing policies and measures in a way that is efficient, effective, coherent, and relevant is essential to achieve these ambitious goals. 

Lack of ex-post evaluation reporting

Linking ex-ante and ex-post evaluations leads to more evidence-based and effective policy interventions, as ex-post evaluations rely on real-world data and experiences rather than assumptions and projections1. Therefore, a well-conducted ex-post evaluation is crucial for informed policymaking. 

But, in order to understand the significance of ex-post evaluations, it is essential to first clarify what constitutes an ex-post evaluation and its role in the policy cycle:  

Despite the critical role of ex-post evaluation, there is a notable absence of reporting on ex-post assessments within the climate policy field, encompassing factors such as emission reductions, costs and benefits, and progress indicators. Within the framework of the European Topic Centre on Climate Change Mitigation (ETC CM), VITO/EnergyVille conducts analyses of Policies and Measures (PaMs) reporting5 from European Member States as part of the National Energy and Climate Progress reports6. 

In the year 2023, only seven EU Member States reported ex-post quantified data for at least one policy or measure, including Belgium, Greece, Finland, France, Ireland, Poland, and Slovenia. This reporting covered 142 policies and measures and a total of 448 quantified emission reductions in 20237, compared to 167 policies and measures and 640 quantified emission savings in 2021.

Insights from the PATTERN project

In the Horizon project PATTERN, VITO/EnergyVille conducted a meta-analysis focusing on ex-post and ex-ante evaluations, identifying 58 studies in Europe dedicated to environmental and climate policies. However, these existing ex-post studies do not cover all sectors and policy instruments uniformly. While sectors like energy, buildings, and industry receive considerable attention, others such as agriculture, forestry, and waste are relatively understudied. Notably, economic and fiscal policies receive more assessment in ex-post evaluations compared to other policy types like information, voluntary measures, and regulatory measures. 

In terms of bridging ex-post and ex-ante assessment, current practices demonstrate limited interaction, with ex-post evaluations rarely informing ex-ante evaluations comprehensively. Among the limited studies establishing a consistent information flow between these assessments, the focus tends to center on data collection and monitoring, neglecting crucial aspects such as defining intervention logic, engaging stakeholders, and refining methodologies. 

European Court of Auditors’ concerns and recommendations 

Furthermore, the European Court of Auditors (ECA) has highlighted the overall deficiency in ex-post evaluation by European Member States8. In their assessment of the EU’s progress towards its 2030 energy and climate targets, the ECA states in their analysis:  

               “… We found that the Commission has sparse information on the Member States’ policies aimed at achieving their climate targets. … Data on costs only existed for limited number of policies and measures that had been implemented. The same limitations apply to data on in terms of greenhouse gas emission reductions.” 

As a result, the ECA recommends – already for a second time – to the Commission “to provide more transparency on the performance of the EU and its Member States on climate and energy action”.

What evaluation challenges are policymakers facing?

Variation in evaluation practices  

Policymakers grapple with numerous challenges when assessing environmental and climate policies, each presenting distinct complexities9 10. One fundamental challenge is the prevalent ad-hoc approach to policy evaluation across many countries in Europe. There is considerable variation in practices among EU Member States, with some lacking formal definitions of evaluation.  

The absence of a clear and comprehensive intervention logic in policy evaluation processes complicates understanding the intended mechanisms and effects of policies. This hampers efforts to trace how policies translate inputs and activities into desired outcomes, hindering the assessment of effectiveness and the design of improvements. 

Issues with data availability and reliability

A pervasive challenge in evaluation is the lack of monitoring data, which can lead to costly and time-consuming data gathering efforts and reliance on generic data sources, resulting in evidential gaps. Difficulties in obtaining representative samples for surveys and selecting appropriate control groups further undermine the reliability of evaluations.  

Additionally, establishing cause-effect relations (input/outcomes) is often challenging due to parallel policies or interventions leading to similar effects. The intricate web of existing measures within which policies are implemented creates interactions that can affect their effectiveness, necessitating a nuanced understanding of policy interactions—whether they overlap, reinforce, or operate independently. 

Issues with understanding behavioral effects

In addition, behavioral responses to policy measures pose complexities to ex-post evaluations. Rebound effects, where efficiency improvements paradoxically increase overall consumption, complicate the assessment of policies aimed at reducing resource use or greenhouse gas emissions. Quantifying the rebound effect introduces uncertainties given its variability across sectors and policies.  

Moreover, policies providing financial support may inadvertently incentivise free-rider behavior, wherein economic agents utilise the financial support despite having intended to make the same investment or behavioral change without it. Consequently, free-rider effects reduce intervention effectiveness and efficiency. Distinguishing between policy-induced changes and independent actions is challenging, further complicating ex-post evaluations. 

Enhancing policy evaluations - Our recommendations

Policy evaluations serve as crucial learning tools, shedding light on why policies succeeded or encountered barriers to effectiveness and efficiency11 12.

In this section, we propose a series of five recommendations aimed at enhancing policy evaluations, ensuring they become integral components of the policymaking process.  

1. Make policy evaluation integral to the policy cycle. 

In many countries, policy evaluations often follow an ad-how approach. Therefore, it is imperative to institutionalise policy evaluations within policymaking processes. This entails establishing legal, institutional, and procedural frameworks that delineate the responsibilities of governmental agencies and ministries, set agenda priorities, and define evaluation objectives, methodologies, and quality standards.  

2. Recognise the role of intervention logic.  

Understanding the logic behind interventions helps identify factors influencing policy effectiveness and mitigate unintended consequences. Intervention logic, typically represented through narratives and diagrams, structures and communicates retroactive policy assessments. The European Environment Agency (EEA) underscores its significance in delineating cause-and-effect relationships within interventions13. Stakeholder participation enriches this process, clarifying the chain of objectives, inputs, outputs, and impacts. Moreover, intervention logic aids in determining evaluation criteria, required evidence and methodologies, ensuring comprehensive and effective evaluations. 

3. Mitigate optimism bias.  

When estimating costs and benefits, it is crucial to account for optimism bias to avoid underestimating the costs of policy interventions. This is particularly important in situations where data availability is limited. In such cases, historical evidence of cost overruns in similar projects or interventions can be used to inform adjustments. This involves identifying a reference class of comparable interventions to benchmark expected costs and benefits, establishing a range of optimism bias based on historical discrepancies, and adjusting the evaluation estimates. This proactive approach lays a solid foundation for conducting realistic and comprehensive evaluations. 

4. Address policy interactions.

In practice, policy interventions often interact with each other, potentially enhancing overall effectiveness and efficiency. Recognising these interactions’ potential to amplify results, it may be preferable to evaluate groups of policies and measures collectively rather than individually. Deciding whether to evaluate policies together or separately depends on contextual factors such as data availability and the objective of understanding individual interventions’ roles. Evaluating policy groups can lead to more coherent outcomes, reduce the risk of double counting, and integrate interaction impacts. A policy interaction matrix can be a useful tool for analysing these interactions and assessing whether they are neutral, overlapping or reinforcing, thereby guiding more informed evaluation strategies.  

5. Enhance evaluation robustness through uncertainty analyses.

The robustness of an evaluation depends on how well its design, data collection methods, and data analysis lead to replicable answers to evaluation questions. To achieve this, it is essential to minimise uncertainty and ensure transparent reporting. Triangulation, which integrates multiple information sources and methodologies, can validate findings and explore any discrepancies. Additionally, the use of sensitivity analysis and scenario analysis are common approaches to address uncertainty sources. Both analyses assess the responsiveness of policy outcomes to changes in uncertain parameters, to demonstrate the range of uncertainty in the parameters and the resulting effects.  

In general, it is important to note the good practice of being as transparent as possible when reporting on assumptions and sources of uncertainty.  

Conclusion

In conclusion, robust policy evaluations stand as indispensable tools in shaping effective climate and energy policies. By identifying key challenges and offering practical recommendations, we aim to deepen understanding of the complexities involved in evaluating environmental policies.  

To drive meaningful progress towards climate and energy targets, policymakers should prioritise several key actions: institutionalising evaluation processes, recognising intervention logic, mitigating optimism bias, addressing policy interactions, and enhancing evaluation robustness.  

Taking proactive steps in implementing these recommendations will not only facilitate evidence-based decision-making, but also bring us closer to achieving our climate and energy targets.  

Footnotes

  1. PATTERN: Pattern-Heu – Providing operational economic appraisal methods and practices for informed decision-making on climate and environmental policies
  2. European Commission, Better regulation: guidelines and toolbox:  https://commission.europa.eu/law/law-making-process/planning-and-proposing-law/better-regulation/better-regulation-guidelines-and-toolbox_en
  3. HM Treasury, The Magenta Book (2011): https://www.gov.uk/government/publications/the-magenta-book.
  4. European Commission, Better Regulation: guidelines and Toolbox: https://commission.europa.eu/law/law-making-process/planning-and-proposing-law/better-regulation/better-regulation-guidelines-and-toolbox_en
  5. Eionet, ETC CM report 2023/06: Overview of reported integrated national climate and energy policies and measures in Europe in 2023:  https://www.eionet.europa.eu/etcs/etc-cm/products/etc-cm-report-2023-06
  6. Article 17 of Regulation (EU) 2018/1999 on the Governance of the Energy Union and Climate Action.
  7. A Member State can report emission savings for multiple past years for a single or a group of single policies and measures.
  8.  European Court of Auditors, Special report 18/2023: EU climate and energy targets – 2020 targets achieved, but little indication that actions to reach the 2030 targets will be sufficient: Report | European Court of Auditors (europa.eu)
  9. Eionet, Guidance document for ex-post evaluation of climate policies in Effort Sharing sectors: Guidance document for ex-post evaluation of climate policies in Effort Sharing sectors — Eionet Portal (europa.eu)
  10. Eionet, ETC CM report 2023/06: Overview of reported integrated national climate and energy policies and measures in Europe in 2023: ETC CM report 2023/06: Overview of reported integrated national climate and energy policies and measures in Europe in 2023 — Eionet Portal (europa.eu)
  11. Eionet, Guidance document for ex-post evaluation of climate policies in Effort Sharing sectors: Guidance document for ex-post evaluation of climate policies in Effort Sharing sectors — Eionet Portal (europa.eu)
  12. OECD, Improving Governance with Policy Evaluation: Improving Governance with Policy Evaluation : Lessons From Country Experiences | OECD Public Governance Reviews | OECD iLibrary (oecd-ilibrary.org)
  13. EEA Report, Environment and climate policy evaluation: Environment and climate policy evaluation — European Environment Agency (europa.eu)

Want to know more?

For deeper insights into our ongoing research and expertise in this field, feel free to reach out to our expert, Nele Renders. 

Contact:
Nele Renders
+32 14 33 58 35