By Kate Mattocks
Governments around the world undertake policy experiments – temporary, often micro-level interventions – to try new things and ‘learn what works.’ But what makes an experiment successful? This is the question I explore in my recent article published in Policy & Politics.
Discussions of success are surprisingly absent from the literature. We might think of success as a positive hypothesis: i.e. achieving an expected result. But this doesn’t capture all of the possible outcomes of experiments, and it also doesn’t consider the process of carrying them out.
In my article I make two main points. The first is that policy experiments can’t be evaluated purely based on their outcomes alone. How experiments are created and carried out matter: the broader political and institutional context needs to be taken into consideration. In other words, “processes are as important as the generated outputs” (Luederitz et al. 2017: 69).
The second point is that experiments should ultimately aim to further policy learning. Learning is crucial to experimentation (Ansell and Bartenberger 2016): those that are experimenting are aiming to produce evidence to inform subsequent decision-making, via a trial-and-error approach. More specifically, learning characterised by open, inclusive discussion – reflexive learning (Dunlop and Radaelli 2018) – is an ideal type of learning for experiments, because it is open-ended, collaborative, and allows for preference shifts. Deliberation involves listening to others, discussing options, and reasoning. It does not just “happen” by default in any group setting (Niemeyer et al. 2024), but must be consciously designed and planned for.
In my article, I outline four criteria for successful experiments based on these two arguments. The criteria were created using insights from existing literature, as well as my own empirical study on experimentation in Canadian arts and cultural policy.
- Basic characteristics: experiments should define a problem, identify an intervention and hypothesis, and should produce evidence that can be measured against a baseline.
- Leadership and Resources: leadership must champion experimentation, and there must be sufficient resources to support it.
- Procedural Elements: experiments should be legitimate, transparent, and valid.
- Evaluation: a ‘light-touch’ approach to evaluation is ideal
Importantly, the criteria aren’t a zero-sum, crude ‘success or not’ tool. Rather, their aim is to provoke critical thought and provide guidance on how to run experiments. Future research can test these criteria and further expand on our knowledge of each category.
…
You can read the original research in Policy & Politics at
Mattocks, K. (2025). What is successful policy experimentation?. Policy & Politics (published online ahead of print 2025), available from: < https://doi.org/10.1332/03055736Y2025D000000065>
If you enjoyed this blog post, you may also be interested in reading
Dunlop, C. A., Radaelli, C. M., Wayenberg, E., and Zaki, B. L. (2024). Policy learning and policy innovation: interactions and intersections. Policy & Politics 52, 4, 547-563, available from: < https://doi.org/10.1332/03055736Y2024D000000049>
Goyal, N., and Howlett, M. (2024). Types of learning and varieties of innovation: how does policy learning enable policy innovation?. Policy & Politics 52, 4, 564-585, available from: < https://doi.org/10.1332/030557321X16841388707452>
Trein, P., and Vagionaki, T. (2024). Why policy failure is a prerequisite for innovation in the public sector. Policy & Politics 52, 4, 586-605, available from: < https://doi.org/10.1332/03055736Y2023D000000012>
Policy & Politics Journal Blog 