Fellowship report now available: “Applying Design Approaches to Policy Making: Discovering Policy Lab”

During 2014-2015 I spent three days a week embedded as an academic researcher in Policy Lab, a team in the Cabinet Office working closely with government departments and with the UK Civil Service’s policy profession. This was enabled by a fellowship from the Arts and Humanities Research Council while I was principal research fellow at the University of Brighton, building on other work including Mapping Social Design Research and Practice. Policy Lab is one example of an innovation team inside government, a growing network and set of practices.

The report summarises findings from the research informed by literature in design studies and organisation studies. It uses a format inspired by graphic novels in order to open up the work of interpretation about the role of design approaches in policy making and government.

Some excerpts from the report are below. If you want to (re)use them, please note this work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. Please use this citation: Kimbell, L. and Macdonald. H. (2015). Applying Design Approaches to Policy Making: Discovering Policy Lab. University of Brighton.

Click to download report (6MB, PDF)

abstract

Finding1

exploring

finding2

abduction

finding3

ethnog

proto

finding4

agonisticlab_key_insightsPolicy Exploration Framework

Advertisements

Draft findings from fellowship

Research question

What is the difference that Policy Lab’s approach makes to policy making?

1 What the Lab approach is/does

Lab’s approach problematises policy making – it’s not just exploring new tools, techniques and new data. Policy Lab connects/reassembles/tweens actualities and potentialities, problems and solutions, thinking and doing, inside and outside.

The key characteristics of this approach are that it is based in:

  • Abductive discovery, through which insights, guesses, framings and concepts emerge eg ethnographic research, co-design, prototyping in the fuzzy front end of policy making.
  • Collective inquiry – through which problems and solutions co-evolve, which is participatory, and through which constituents of an issue are identified and recognised, and solutions are tested eg prototyping.
  • Recombining experiences, resources and policies – the constituents of an issue – into new (temporary) configurations.


2 What Lab approach results in – its impact which we can seek proxy measures for

Project level – Relating to the policy area

  • New insights, guesses, framings
  • Plausible concepts for artifact-experience bundles
  • Prototyped proofs of concept – “proto policies”
  • An issue team/public engaged in a collective inquiry engaging with a more ordered problem

Capabilities in within the policy profession and wider ecosystem

  • Reordered relations between actors in an issue (inside and outside an issue)
  • Reordered relations between actors and evidence
  • Reordered timings
  • Ability to set up and participate effectively in collective inquiries and early-stage abductive exploration
  • Awareness of the interdependencies between experiences, resources and policies

Phases of the research: Phase 1 – Infrastructuring

Part 2 of some retrospective sensemaking of my research fellowship within the Policy Lab team in the Cabinet Office.

Phase 1 Infrastructuring: January – early May 2015

Working with Hannah Rutter from Policy Lab to develop a framework to make sense of Lab
Working with Hannah Rutter from Policy Lab to develop a framework to make sense of Lab

 

With my newly-gained, temporary insider status and confidence – enabled by the security pass which allowed me into some government buildings without an escort and by my emerging understanding of the civil service’s policy making environment, the first few months of 2015 gave me deep access to new developments in Policy Lab’s world. As well as continuing to deliver many one-hour or longer taster workshops to departmental policy officials, Policy Lab took shape through its demonstrator projects lasting over several months and ongoing discussions about its future in the context of a countdown to the general election.

One of Lab’s five demonstrator projects, with the Department of Work and Pensions (DWP) and Department of Health (DH) on health outcomes, kicked off in late January after two months of what Pelle Ehn calls “infrastructuring”[1] – the briefings, proposals, meetings, emails, commitments and contracting that construct a project. In previous projects I had been more of an observer. In this project I took a more active role at the beginning, for example helping Lab’s project lead Cat Drew and the rest of the team design, facilitate and make sense of the policy “sprint” workshop. Unlike the earlier projects in which Lab and the government department subcontracted chunks of the project to specialists in ethnographic research and design, in the health outcomes project Policy Lab directly brought together and mediated between experts. They worked in close collaboration with one another and with staff from the two departments including policy makers, analysts and some of their advisers and other stakeholders. The two-and-a-half day sprint staged the project from the outset as a collective inquiry by articulating and iterating a goal, defining research questions and approaches, and building a shared, although provisional understanding of the issue.

Other demonstrator projects with HM Revenue and Customers (HMRC) on National Insurance numbers and young people, and with the Department for Education (DfE) on childcare, moved forward with combinations of ethnographically-informed research and analysis, design and prototyping. I participated in workshops in which Policy Lab and the wider project-specific teams shared research insights, supported collaborative design by civil servants and other stakeholders. I also participated in review meetings and sometimes helped edit or produce documents at key points in a project lifecycle. In one project I took a direct role as the lead for Policy Lab, on a consultancy basis. This project was for the small team serving the civil service’s Heads of Policy Profession Board with the goal of exploring and developing proposals for assessing and accrediting the capabilities of people working in the policy profession. I’ll discuss the ethical, political, and methodological implications of doing this alongside my fellowship in a separate post.

Projects that were more or less completed such as with the Home Office on digital policing, and the Ministry of Justice (MOJ) on family mediation, were still part of Policy Lab’s world, surfacing in team discussions about next steps and demonstrating Lab’s impact. The challenge for each was how to take forward what the Policy Lab project had produced but without ownership – as the policy areas lived in departments, not in the Cabinet Office – and without much resource in terms of time or money, nor yet much visible commitment from senior leaders in the civil service or ministers.

I became more aware of the importance of formal governance structures and processes in this civil service world. Crafting Policy Lab’s demonstrator projects involves setting up “boards” chaired by Paul Maltby, the director of the Government Innovation Group in which Policy Lab is embedded. These involve the Policy Lab lead and the policy officials leading the policy area but, crucially, involve senior civil servants from the departments involved. Punctuating the project journey, these boards invite senior people to review Policy Lab’s work including the research insights and emerging concepts and decide how to move forward and making commitments to one another, often across departmental boundaries.

In early February – in a very short space of time – the Open Policy Making team and Policy Lab organised 19 events for over 500 people with the title Open Policy 2015. Many of these were practical workshops and taster sessions for civil servants to try out tools and techniques including user research, behavioural insights, agile approaches such as hackdays, and working with stakeholders. While these approaches are not new to some policy teams, these were opportunities for participants to hear an experienced civil servant or external speaker share experiences of using a particular approach and then try aspects of it out. I attended some events such as Policy Lab’s prototyping workshop and also organised one for Policy Lab, which was discursive rather than practical, on ethnography in policy making.
As it got closer to the first anniversary of Policy Lab being set up – initially for one year in April 2014 – discussions among the team and with senior stakeholders focused on making sense of what Lab had been doing, and demonstrating its impact. This went in parallel with constructing future projects with departments and articulating options for senior civil servants to consider about its purpose, resourcing, and expected outcomes. Various ways of framing Policy Lab were discussed, with a recurring themes of experimentation, engagement and evidence and what it takes to make a project “land”.

My research at this time was guided still by Bent Flyvberg’s Making Social Science Matter, as well as by Jesper Christiansen’s PhD thesis entitled The Irrealities of Pubic Innovation based on his research/work at MindLab. Researcher Ben Williamson’s blog, the anthrodesign mailing list and the twitter hashtag #psilabs were also useful. I continued taking lots of notes and photos, doing some interviews, but decided against using video to gather data.

The UK general election date of May 7 marked an end point to this phase of the research. Owing to the relationship between policy makers and ministers, as well as to the particular uncertainty around who might win that election, the months leading up to the election had a particular intensity and urgency. Civil servants talked about the pressure of getting things done before “Purdah”, the name given to the period of time after Parliament is dissolved and before a new government is formed, when the civil service is not supposed to favour any political party. Although the civil servants in Policy Lab did not work directly with ministers at that point, this urgency to get things done shaped the working culture and expectations about the timeframes within which some of its projects with departments had to produce results.

As the civil service entered Purdah, it seemed ironic that parts of the civil service advocating and practicing open government decided not to have any online digital engagement during this time – even though some government departments did. For example the OPM team and Policy Lab were advised not to tweet or blog. With an inside/outside role, I changed some of my own online behaviours during this time too.

This phase of my research was still about building, connecting and expanding rather than making sense. Writing up a couple of blog posts for the OPM blog (the links are above) and doing a couple of keynotes and talks to early career researchers forced me to try to locate and digest the research to date. I found it very hard. In my application for the fellowship I had said I would co-design an evaluation framework for Policy Lab and made various efforts to do so, working closely with the team of Andrea Siodmok, Beatrice Andrews, Hannah Rutter, Cat Drew and Cabinet Office intern (and doctor) Lisa Graham. But I was still in the mess of being-in-the-work, trying to understand what Policy Lab was doing in its various emergent forms in a context of massive uncertainty and ambiguity. It began to get clearer – to me at least – that Policy Lab and its publics might benefit from an account of what it was doing – the difference it made to policy making – which needed to precede any framework.

[1] Ehn P (2008) Participation in design things. In: PDC ’08: Proceedings of the tenth conference on participatory design, Bloomington, Indiana, 30 September–4 October 2008. New York: ACM Press, pp. 92–101.

Abductive policy making. Draft section of report from research fellowship

Image from Policy Lab/Strange Telemetry project with Government Office for Science
Image from Policy Lab/Strange Telemetry project with Government Office for Science

Between discovery and justification

The aim of this section is to help clarify what kinds of experiments are going on in the work of Policy Lab. This section draws on the work of philosopher Charles Sanders Peirce who developed the term abduction. First it will describe what abduction is, and how abduction relates to the other kinds of inference, deduction and induction, which are well-established in policy development. It will then discuss Policy Lab’s work through the lens of abduction and show how policy experimentation via abductive reasoning intersects with the other logics shaping policy making.

Deduction and induction

The term abduction is associated with the philosopher CS Peirce, who explored the term over some decades[1]. Like other Pragmatists such as Dewey, Peirce’s approach has an orientation toward our experiences of what happens in practice, rather than proposing an idealized analysis. To explain why Peirce’s work is a useful contribution to understanding Policy Lab’s approach requires a brief detour into the two other logics through which reasoning is usually understood to proceed.

Deduction is the process of taking a principle (a rule) and then inferring a result in a particular case. For example:

Rule: People living in the Midlands are friendly.

Case: These people are from the Midlands.

Result: These people are friendly.

Deductions offer reliability if the initial statements are true, but Peirce argued they do not generate anything new.

Induction starts with surveying data (the case) and generalising across many observations (the result) to identify a pattern (the rule). For example:

Case: These people are from the Midlands.

Result: These people are friendly.

Rule: People from the Midlands are friendly.

Inductions indicate probability about patterns in the data. They suggest that something is the case. Depending on the research methods used, they offer some kind of validity. As with deduction, Peirce argued inductive reasoning does not generate new concepts or knowledge.

In the context of experimental research, knowledge building typically proceeds by developing a hypothesis based on the stock of existing knowledge via deduction and then seeking confirmation by induction if it holds in a particular case.

The logics of policy making

Much of the evidence used to inform policy making uses mixed methods based on deductive or inductive reasoning in various combinations. Neither are right or wrong – they do different things and offer different kinds of validity to different audiences, to allow policy officials and ministers to reach decisions. But in the culture of policy making, the deductive logic offers the allure of offering definitive evidence. For one civil servant, “Trials are the gold standard for policy making”[2] because they are able to prove whether something is true or not, providing sound evidence that decision makers want about whether to go ahead with a policy.

Deductive reasoning underpins work in the natural and physical sciences and also shapes research in the social sciences. For example the methodological approach “Test, Learn Adapt” advocated by the Behavioural Insights Team is grounded in deductive logic [3]. BIT helps civil servants design and construct trials of policy by demonstrating whether an intervention will achieve intended outcomes, informed by existing knowledge about human behavior. Randomised control trials (RCTs) are one way to test systematically, in a particular case, whether a hypothesis underpinning an intervention is true or not. In policy terms, RCTs can prove whether a proposed intervention will lead to the desired change in a particular case. It generates statistically valid data about changes to variables associated with the outcome that result from the intervention.

Inductive reasoning is also very familiar within the policy environment. It underpins much of the research in the social sciences and the humanities. Inductive research does not have to use qualitative data but it is strongly associated with it. Researchers specialising in research methods working within this logic make efforts to show to what extent their findings have validity. They make careful claims about whether they can show links between cause and effect and discuss the extent to which their findings are generalisable to other contexts.

Abduction

But where do hypotheses and new ideas come from in the first place? What happens in the context of massive uncertainty, when there is very little data, or much of it is in disagreement? What if you have a desired outcome that you want to achieve but are not sure of the constituent elements that might help you achieve it or how they relate? How do researchers get to the point that they are able to isolate an outcome variable which could be tested through a trial?

Recognising a gap in the philosophy of science, Peirce developed the idea of abduction as a logic of discovery within scientific inquiry, in contrast to the logic of justification associated with deductive reasoning. He argued that philosophers of science had paid insufficient attention to where ideas come from. Informed by ancient Greek thought, he developed the concept of abduction to explain how new concepts and hypothesis are created.

Abduction takes a result and a rule, and then jumps to making an interference that links the two. For example:

Rule: People from the Midlands are friendly.

Result: These people are friendly.

Case: These people are from the Midlands.

In abduction, we link things together in new ways. We can’t say if the interference is true or not, as is the case with deduction. Nor can we say it has strong validity because of the observations we made, as with induction. But with abductive reasoning, what we do get is a new insight or concept that we can explore further with the other two logics.

Hypotheses are not out there waiting to be discovered. Instead, Peirce argued, they are the outputs of a process of sensemaking. As we make observations through our own experience of the world, we compare these to the existing stock of knowledge. We may find something surprising that we can’t account for, resulting in a tentative guess – an embryonic hypothesis. Social researcher Jo Reichertz explains, “Something unintelligible is discovered in the data and, on the basis of the mental design of a new rule, the rule is discovered or invented and, simultaneously, it becomes clear what the case is.”[4] In contrast to deduction, which offers reliability, abduction offers possibility by generating something new, which can then be explored further through induction and deduction.

With the concept of abduction, Peirce was able reconnect creativity in science with the well-established idea that scientific reasoning can prove things. Other researchers such as Karl Popper separated the logic of discovery from the logic of justification. They focused on how science can make more reliable truth claims, paying less attention to how novel concepts are generated [5]. In Peirce’s own words, “Abduction is the process of forming an explanatory hypothesis. It is the only logical operation that introduces new ideas, for induction does nothing but determine a value, and deduction merely evolves the necessary consequences of a pure hypothesis.”[6] Table 1 shows the differences between the three logics, based on Peirce’s work. In his view there is an order: abduction precedes deduction and induction [7].

Table 1: Peirce’s ordering of the logics of scientific inquiry [7]

Table 1: Peirce’s ordering of the logics of scientific inquiry, developed from Hansen 2008.

Producing plausible, provisional results

Abduction reasons from effects to causes with incomplete data. It constructs plausible guesses and insights, shaped by our existing stocks of knowledge and in responses to effects gathered through observations or experiences. For management researcher Hans Hansen, “We take disparate elements and place them into relationships that are meaningful for us. Abduction generates hypothesis (sic) in the absence of any existing construct to interpret observation.”[8] Abduction shows something may be, but does not prove it, whereas deduction shows something is true in a particular case. Abductive inferences are plausible but are not justified by the structure of the argument. But they are plausible enough to move a project forward[9].

Abduction results in a new order that takes surprising observations and offers a way to make sense of them – for now – which is still productive. However for social researcher Jo Reichertz,

“The search for order is never definitively complete and is always undertaken provisionally. So long as the new order is helpful in the completion of a task it is allowed to remain in force: if its value is limited, distinctions must be made; if it shows itself to be useless, it is abandoned. In this sense, abductively discovered orders are neither (preferred) constructions nor (valid) reconstructions, but usable (re-) constructions.”[10]

At first glance there is a relationship here with the work of Karl Popper. He argued that science proceeds by hypotheses being challenged or upheld by subsequent research, a process he called falsification. But abductive interferences are never as firm as hypotheses in the first place. They are provisional, plausible constructs that are usable – they move a process of inquiry along – but do not offer a truth claim.

Participants in Policy Lab workshop discussing their hopes and fears in relation to a policy area
Participants in Policy Lab workshop discussing their hopes and fears in relation to a policy area

Creating the conditions for abduction

When developing his theory of abduction, Peirce discussed the conditions that gave rise it to it, which can be seen as broader principles for enabling the generation of new ideas. Rather than arguing that coming up with new insights is simply the result of chance, he identified strategies or enabling conditions for making it more systematic[11].

Peirce developed his thoughts on abduction as a result of a personal experience[12]. This was when a Tiffany watch was stolen from him after he left it behind by accident on a ship to New York. Initially he had no idea who was responsible[13]. In his account of the stolen watch, Peirce asked the captain of the ship to line up all the crew for him to talk to. At first he found himself unable to work out who might have taken his watch. Reflecting on what happened, Peirce described how this experience prompted him to reconsider how knowledge is generated.

The first condition conducive to the presence of developing an abductive inference is genuine doubt, uncertainty or great pressure to act[14]. The Tiffany watch was a gift, and a valuable one. But the fear motivating Peirce was not fear of its loss, but of professional disgrace for not being able at first to work out who might be guilty.

The second condition conducive to abduction Peirce identified was to let his mind wander with no specific goal – what he called “musement”. Instead of trying to use deductive logic to work out who had stolen his watch, Peirce gave into a state that was not controlled by his conscious mind. As a result, he concluded that his consciously working mind, which usually relied on logical rules, was outmaneovoured[15].

The third condition to make abduction more likely was to decide to act, even if the direction seemed arbitrary. As he tried to work out who had stolen the watch by walking up and down the lined up crew of the ship, Peirce concluded he must fasten on someone even through it would be almost a random choice[16]. The guess that Peirce made turned out later to be true and eventually he got the watch back. Management researcher Hans Hansen summarises, “At the point of being surprised by a surprising fact, if we can make a guess, any guess, we can make progress.”[17]

Recent interest in abduction

Researchers and practitioners in several domains are using abduction to help them distinguish between different kinds of research activity and practical experimentation. In business, Roger Martin[18] argued that managers need to use abductive as well as deductive and inductive reasoning as tools to achieve competitive advantage. In design[19], researchers have used the theory of abduction to explain how designers come up with new concepts. In social research, especially in fields such as nursing, researchers have turned to abduction to better understand how themes, codes and categories emerge during research[20]. In artificial intelligence and data science, there is a longstanding interest in abduction and induction and how they relate to one other in algorithmic machine learning[21].

Participants in a Policy Lab prototyping workshop
Participants in a Policy Lab prototyping workshop

Abduction in policy making

Drawing these argument together, the concept of abduction helps those involved in policy experimentation distinguish between the logic of discovery and the logic of justification. What Peirce’s ideas do is highlight the often invisible work that goes on during what we might call the “fuzzy front end” of policy making[22].

This discussion highlights the different kinds of reasoning produce different results at different phases of the policy making cycle. They are not directly comparable and further, if Peirce is right, then there are interdependencies between them. Peirce’s view is that there is a sequence which starts with abduction. The exploratory insights and guesses produced through abductive reasoning with limited data but with are nonetheless plausible can then be further developed through deductive and inductive reasoning. Abduction produces the provisional insights and guesses linking things together in new ways, that become hypotheses that be tested through experimentation and other research based in deductive and inductive logics. Deductive research can answer if a policy intervention works or not; inductive research helps explain why it does[23]; but abductive reasoning enables the discovery of insights and guesses when there is not yet theory or evidence but a desired policy outcome.

This highlights the mostly invisible work that policy makers do, where they are required to rapidly gather and assess evidence and come up with options for ministers. Looking at this through the lens of Peirce’s work, we might pay more attention to this early, hypothesis-free, exploratory phase. Trials test policy interventions in particular kinds of case when there is a hypothesis to test. In contrast design and prototyping support policy inventiveness by making plausible links between elements of an issue to achieve an intended outcome. If RCTs are a robust way establishing if a policy is working[24], the abductive cycle of generating and exploring insights and guesses is the best way of developing and iterating plausible early-stage policy ideas.

This is where Policy Lab’s approach comes in. It is rooted in the practical experimentation of design going through cycles of rapid insight generation, idea generation and exploration via prototyping. Working within this tradition, Policy Lab foregrounds the abductive early stage of policy making – where “early stage” includes revisiting persistent, complex policy issues. Alongside data science, and agile approaches also used in government, Policy Lab reveals and supports the often neglected abductive work that policy makers do, but opens it up to a wider group of participants and new sources of evidence and inspiration which the next section will illustrate.

References (not complete or checked)

[1] Scholars make a distinction between his earlier work using the term and his later work, which is what I draw on here. See Roozenburg, N. (1993). On the pattern of reasoning in innovative design. Design Issues 14(1): 4-18.

[2] The speaker was a participant in a workshop organised by the What Works network held at the Institute for Government in June 2015.

[3] Haynes, Laura, Service, Owain, Goldacre, Ben and Torgerson, David. (2012). Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trial. London: Cabinet Office.

[4] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 1(13). Italics in original.

[5] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 1(13).

[6] CS Peirce cited in Hansen, Hans. (2008). Abduction. In Barry, David and Hansen, Hans (eds). The Sage Handbook of New Approaches in Management and Organization. p.456

[7] Table developed from Table 3.5.1 in Hansen, Hans. (2008). Abduction. In Barry, David and Hansen, Hans (eds). The Sage Handbook of New Approaches in Management and Organization. p.457.

[8] Hansen, Hans. (2008). Abduction. In Barry, David and Hansen, Hans (eds). The Sage Handbook of New Approaches in Management and Organization. p.457

[9] This is the underpinning to lean start up and agile software development.

[10] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 11(1). Article 13. http://nbn-resolving.de/urn:nbn:de:0114-fqs1001135.

[11] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 11(1). Article 13. http://nbn-resolving.de/urn:nbn:de:0114-fqs1001135.

[12] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 11(1). Article 13. http://nbn-resolving.de/urn:nbn:de:0114-fqs1001135.

[13] Boje, D. (2014). Storytelling Organizational Practices: Managing in the Quantum Age. Abingdon: Routledge. p.39-40.

[14] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 11(1). Article 13. http://nbn-resolving.de/urn:nbn:de:0114-fqs1001135.

[15] Reichertz, Jo. (2010). Abduction: The Logic of Discovery of Grounded Theory. Forum: Qualitative Social Research. 11(1). Article 13. http://nbn-resolving.de/urn:nbn:de:0114-fqs1001135.

[16] Boje, D. (2014). Storytelling Organizational Practices: Managing in the Quantum Age. Abingdon: Routledge. p.39-40.

[17] Hansen ibid p.461.

[18] Martin, Roger. (2009). The Design of Business: Why Design Thinking is the Next Competitive Advantage. Harvard Business Press.

[19] See for example, Roozenburg op cit; Kolko, John. (2010). Abductive Thinking and Sensemaking: The Drivers of Design Synthesis. Design Issues, 26(1); and Dorst, K. (2015). Frame Innovation: Create New Thinking By Design. MIT Press.

[20] See for example Tavory, Iddo and Timmermans, Stefan (2014). Abductive Analysis: Theorising Qualitative Research. Chicago University Press.

[21] See for example Flach, Peter and Kakas, Antonis (eds). (2000). Abduction and Induction. Essays on Their Relation and Integration. Springer.

[22] Fuzzy front end is a term describing the early stage of product development, introduced in Khurana, Anil and Rosenthal, S. (1997). Integrating the Fuzzy Front End of New Product Develpoment. Sloan Management Review, Winter. pp. 103-120.

[23] https://www.ipsos-mori.com/Assets/Docs/Scotland/Approach%20Summer%202012/SRI_Scotland_Newsletter_Summer2011_Test_Learn_Adapt.pdf

[24] Haynes, Laura, Service, Owain, Goldacre, Ben and Torgerson, David. (2012). Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trial. London: Cabinet Office. p.4.