Policy & Politics Journal Blog

by Clementine Hill O’Connor, Katherine Smith & Ellen Stewart

Balancing evidence with public preferences – a pressing policy dilemma? 

How can policy organisations balance competing (and sometimes conflicting) imperatives to strengthen the role of evidence in policy, with simultaneous calls to better engage diverse publics? Academic research has much to say about both the value of evidence for policymaking and there are multiple studies examining evidence use in policy and assessing efforts to increase (or improve) the policymakers’ engagement with evidence. Academics have also been involved in developing a wide range of methods through which publics can be involved in policymaking. Perhaps surprisingly, these contributions are rarely connected. So, despite sharing a fundamental concern with the basis on which policy is made and a (sometimes implicit) claim to improve policy, these two areas of academic work are rarely connected. This is important because this disconnect creates real world challenges for people working in policy settings. This disconnect is the focus of our recent research published in Policy & Politics entitled Integrating Evidence and Public Engagement in Policy Work: An empirical examination of three UK policy organisations.

To expand on our research. We conducted research in three policy organisations – the Scottish Government, Greater Manchester Combined Authority, and Sheffield City Council – interviewing 51 staff involved in data analysis for policy and in constructing policies to tackle inequalities. We explored how people described the role of evidence and data in policy, how they sought to understand and engage with publics (including examining types of public engagement processes currently in use), and how these different strands of policy work connected.   

The interview data suggest that, in all three organisations, ‘evidence’ and ‘data’ were primarily thought about in quantitative terms: methods, data and analysis. Further, ‘evidence’ was commonly used to refer to externally produced analysis of quantitative data, while ‘data’ was used to talk about quantitative indicators being internally monitored, e.g. via ‘data dashboards’. There was also a reliance on engaging with public views via surveys which focus on the quantity of responses and can lack depth of insight.. While this aligns well with the increasing quantification of policy governance, our interview data also capture dissatisfaction with the insights provided by surveys and a desire for more qualitative, in-depth insights about public views and the need to understand the ‘lived experience’ of those impacted by policy decisions.. Yet, even where these, more qualitative insights, were gathered, there seemed to be a dearth of options for bringing these different types of data together. 

We did not identify any big differences between the organisations in how to approach this, but we did find variation in individual preferences for navigating this issue, which led to us identifying four clusters of approaches:  

1. Positioning public engagement as foundational  

2. Employing public engagement, evidence and data strategically to support desired policy progress 

3. Treating quantified evidence and data as foundational 

4. Working to try to integrate quantified data and evidence with qualitative insights about public views and experiences 

The majority of people we interviewed were within cluster 4. The following quote is a good illustration of the views of people within this cluster:  

‘I think we need to fundamentally move away from just looking at data to really building in qualitative evidence as well, building in those stories, people’s experiences and really understand why those data looks the way it does [sic] and actually start using a combination of both for that and I think that’s the place where we need to be.’ (Interviewee 26, Scotland) 

Whilst interviewees identified examples of trying to undertake this type of integration, it was clear that it was challenging and there were few, if any, approaches that seemed to fully achieve the kind of integrated insights they were seeking.  

Integrating these different forms of knowledge is not only practically challenging, but is complicated by the fact that (as alluded to by the gap between the scholarship on evidence use and public engagement) there are foundational differences in what each seeks to show, and how their robustness might be judged. Nonetheless, we argue that it is an important and potentially impactful avenue for future research to understand and bridge these different insights and perhaps even begin to develop mechanisms to support the better integration of these different forms of knowledge. 

You can read the original research in Policy & Politics:

Hill O’Connor, C., Smith, K., & Stewart, E. (2023). Integrating evidence and public engagement in policy work: an empirical examination of three UK policy organisations, Policy & Politics (published online ahead of print 2023) https://doi.org/10.1332/030557321X16698031794569

The views and opinions expressed on this blog site are solely those of the original blog post authors and other contributors. These views and opinions do not necessarily represent those of Policy & Politics, the Policy Press and/or any/all contributors to this site.

If you enjoyed this blog post, you may also be interested to read:

Christensen, J. (2021). Expert knowledge and policymaking: a multi-disciplinary research agenda, Policy & Politics, 49(3), 455-471 from https://doi.org/10.1332/030557320X15898190680037

Koga, N. M., Loureiro, M., de Moura Palotti, P. L., da Silva Lins, R., Gontyjo do Couto, B., & Nogueira Lima, S. (2022). Analysing the information sources Brazilian bureaucrats use as evidence in everyday policymaking, Policy & Politics, 50(4), 483-506 from https://doi.org/10.1332/030557321X16588356122629

MacKillop, E., Quarmby, S., & Downe, J. (2020). Does knowledge brokering facilitate evidence-based policy? A review of existing knowledge and an agenda for future research, Policy & Politics, 48(2), 335-353. Retrieved Mar 6, 2023, from https://doi.org/10.1332/030557319X15740848311069

Sayer, P. (2020). A new epistemology of evidence-based policy, Policy & Politics, 48(2), 241-258. Retrieved Mar 6, 2023, from https://doi.org/10.1332/030557319X15657389008311