Ethnography in policymaking: Barriers and opportunities

Originally published on the Cabinet Office Open Policy blog on 27 March, this is my write up of an event I chaired during Open Policy 2015.

Example of ethnographically-informed research output from a Policy Lab/Department of Health/Department of Work and Pensions project about people in employment with health conditions: “Due to [the employer] not helping (they didn’t provide sick pay), I got into extreme financial difficulty. Benefits weren’t enough to cover everything and everything went wrong. The empty fridge represents no money as I couldn’t feed myself sometimes.” Picture and story from Katrina, circling in and out of work with a physical health condition, Sussex (name and details anonymised)
Example of ethnographically-informed research output from a Policy Lab/Department of Health/Department of Work and Pensions project about people in employment with health conditions: “Due to [the employer] not helping (they didn’t provide sick pay), I got into extreme financial difficulty. Benefits weren’t enough to cover everything and everything went wrong. The empty fridge represents no money as I couldn’t feed myself sometimes.” Picture and story from Katrina, circling in and out of work with a physical health condition, Sussex (name and details anonymised)
Increasingly on the agenda of policymakers is a need to understand the needs, capacities and perspectives of citizens, service users, beneficiaries and front line staff so that policies are fit for purpose and deliverable and public services are better designed. Ethnography is seen as one way to achieve this. Based on a methodology originally associated with anthropology, ethnographic approaches are now found within product development, innovation, strategy, marketing, and research and development in a wide range of organisations from Intel and Amazon to start-ups to central and local government. In the UK, the Government Digital Service (GDS) and government departments, as well as others such as Nesta and the King’s Fund have promoted ethnographically-informed approaches to doing user research.

But the value of ethnography is not simply that it’s a method for understanding people in the context of their own lives, although it does offer that. The real potential for ethnography in policymaking is to help reframe government’s understanding of its purposes and how the world in which it exists and which it shapes is changing. This insight emerged from a panel discussion organised by Policy Lab in the grand surroundings of the Churchill Room during Open Policy 2015, during which three people with different perspectives reflected on the opportunities and barriers for ethnography in government. An audience of over 50 people, the majority of whom were civil servants, gained a valuable overview from leading practitioners applying ethnographic approaches to contemporary organisational and social issues.

The first speaker, Dr Simon Roberts of Stripe Partners, set the scene, informed by his extensive knowledge of applied ethnography from consulting work and gained by twice co-chairing the international Ethnographic Praxis in Industry conference. He summarised what makes ethnographic research distinctive and illuminated the ongoing challenge of understanding its value and impact.

Next up was Lisa Rudnick from Interpeace, previously at the UN Institute for Disarmament Research, where she co-led the development of the Evidence-based Program Design tool and the Security Needs Assessment Protocol with Derek Miller, now of the Boston-based consultancy, The Policy Lab. Sharing perspectives gained from conducting ethnographic research in post-conflict contexts such as Nepal conducted with Ruth Edmonds,  she highlighted the importance of not just using the parts of ethnography that generate local descriptions, but of engaging ethnographic analysis of those descriptions as well as the basis from which to design, or implement, policy for shaping local action on the ground.

Example of ethnographically-informed research output from a Policy Lab/Department of Health/Department of Work and Pensions project about people in employment with health conditions: Photo taken by interviewee Kelly to highlight the amount of work required to manage health conditions at her place of work. (Details changed)
Example of ethnographically-informed research output from a Policy Lab/Department of Health/Department of Work and Pensions project about people in employment with health conditions: Photo taken by interviewee Kelly to highlight the amount of work required to manage health conditions at her place of work. (Details changed)


The third speaker, Rupert Gill, is a policymaker within the Department of Work and Pensions, currently using ethnography as well as data science approaches on a joint project with the Department of Health and Policy Lab. He shared some of the challenges civil servants face when trying to make use of ethnography in a policymaking culture which values particular kinds of argument and evidence.

What ethnographic research is

  • Today’s ethnographic research in organisations exists on a spectrum from hypothesis-free, exploratory research over several months into topics such as “ageing and mobility”, to targeted requirements gathering over a few days to inform the design of a service.
  • As a kind of qualitative research, ethnography investigates worldviews, socio-cultural structures and the practices that shape behaviours. It’s not just finding out what people think, listening to what they say or watching what they do.
  • Ethnographic research makes a commitment to being there with people in their worlds – which these days includes people’s digital lives. Its emblematic method since anthropologist Malinowski went to the Trobriand Islands a century ago is participant observation.

What ethnographic research produces

  • Ethnography is not just descriptive fieldwork. It’s a theory-building endeavour that makes use of research from across the social sciences. While the data might include stories about people’s lives, in their own language and categories, or observations about what people do, what is just as important is the interpretive analysis of that data. Or as Lisa Rudnick put it, “It’s not the story that matters for policymaking. The value is in what makes the story make sense.”
  • As a result, the output of ethnography is informed by people’s stories, and generates insights derived from people’s day to day experiences, but is better understood as an analysis of a social world within which people exist within and have relationships with others including organisations, governments and places.

The opportunity for using ethnography for government

  • The value of ethnographic research is how it creates (re)framings of a social world and helps an organisation understand what it exists for. Reflecting on the impact on technology firms such as Intel, which have made extensive use of ethnography over nearly two decades, Simon Roberts argued, “Ethnography has created a space and a possibility for organisations to reshape their understandings of the world and their understandings of how they have those understandings.”
  • The opportunity is for government to address the complexity of society by understanding people better in the context of their lives, and then changing the focus of policy responses, especially when things are changing. Rupert Gill said there was an appetite for this within the civil service. “We hope to get insights we wouldn’t get elsewhere and use them to create interventions we wouldn’t otherwise have thought of.”

Barriers and challenges

  • The culture within which policy making takes place is dominated by the need to produce evidence that is statistically valid, and not “policy by anecdote”. The small sample sizes associated with ethnographic research may not be seen as valid in this context. Minsters who have to give an account to parliament about their policies feel more confident about analysis from large data sets. But there is a contradiction here, in that ministers also get first hand access to, and are influenced by, stories from their constituents – a kind of field data with very small samples sizes.
  • What’s needed is to combine quantitative data with other approaches, recognising what each brings. “It’s real depth that we need and we can’t get this from numbers,” argued Rupert Gill.
  • Within the UN and peacebuilding contexts in which she works, Lisa Rudnick shared how the approach she co-developed (with Derek Miller) makes managers accountable for the data they use (or don’t) to shape their decisions. Like the policy tests being used in the civil service, this involves asking managers considering a proposal if they have the right kind information for the question at hand, enough information and whether it’s reliable. Posing such questions makes any commitment to action rest on research findings, not on data points or methods.


Policy Lab, GDS and government departments continue to explore ethnographic approaches in practical projects in policy making. If you are a civil servant, look out for guidance on the Open Policy Making Toolkit, Civil Service Learning short courses and Policy Lab workshops to try using the approach yourself.

Introducing sprints to policymaking

Introduction

This post discusses what agile approaches can bring to policymaking. It is based on my recent participant observation in a “policy sprint” organized by the Policy Lab in which I am embedded as an academic researcher. The sprint took the form of a two-day workshop held in Whitehall, followed the next day by a half-day stakeholder workshop, during the early phase of a larger, ongoing project involving Policy Lab, the Department of Work and Pensions (DWP) and the Department of Health (DH).

The focus of the project was better supporting people in relation to their health and employment outcomes. The activities involved a co-located, cross-disciplinary and cross-government group of people including policymakers and specialists skilled in ethnography, data science and design research, in exploring a policy issue and working out collectively what research and design activities were needed to move the project on.

Inspired by and drawing on the “sprints” used in agile software development, this event brought a new way of working to policymaking. A recent blog post by Lisa Ollerhead of the Cabinet Office Open Policy Making team, drawing on her experience of GovCamp, in which she adapted the principles of the agile manifesto discussed what agile approaches might bring to policymaking. She concluded that one significant challenge is establishing what the deliverable or output of a sprint is.

As part of its remit to try out new tools and technique for policymaking, Policy Lab decided to use an agile-inspired approach within one of its demonstrator projects to see how these principles could be adapted to the challenge of developing policies, rather than software. A blog post published right after the event by Cat Drew of Policy Lab, who led the sprint, summarised what happened in the January workshop and how it worked.

The purpose of this post is to analyse what the sprint did for the project. The data on which it draws are notes and photographs from my participant observation, emails and documents, as well as interviews and responses to a survey distributed to participants after the event.

Background – agile, lean, design thinking for policymaking

The concepts and activities used by Policy Lab to shape its policy sprint are not new. Within government, many of the principles and activities associated with agile software development inform the work of the UK Government Digital Service (GDS). For example the GDS approach to designing government services that are “digital by default” is shared here, including a discussion of using sprints. Some of GDS’s terminology and approaches are also found in government departments, especially that have in-house digital teams. The space that Policy Lab is exploring bringing this approach to the early stage of making policy, not building digital services.

Internationally there are other examples of bringing design and digital approaches to policymaking, which share lineages with agile software development. The Helsinki Design Lab, part of Finish innovation agency Sitra which was active between 2008-2011 organized “studios”, typically week-long collective inquiries into an issue such as ageing in Finland, the outputs of which included a roadmap of strategic improvements (see Steinberg 2014). These approaches are closely related too to design-led innovation. For example, innovation consultancy IDEO organizes “deep dives” into issues such as the one on the shopping cart filmed by ABC television in 1999, which involved multi-disciplinary teams collectively inquiring into a problem area, generating ideas and creating rough prototypes of potential solutions.

Related principles such as looking at the bigger picture and systems thinking are also found in other UK policy initiatives like the Total Place programme run by the Leadership Centre for Local Government. The Troubled Families programme takes a “whole person” perspective on families and their interactions with public services.

There are also overlaps with lean start-up, popularized by books by Steve Blank and Bob Dorf (2012) and Eric Ries (2011). Key concepts here include rapid experimental cycles in which entrepreneurs articulate their hypotheses, build software in chunks known as “minimum viable products”, test them with customers and then, depending on the results, “pivot” by switching to a different product or business model if necessary.

Within agile software development, a sprint is usually understood as a time-limited set of activities which result in delivering working software. Used more loosely in the context of policymaking, here this policy sprint was an intense burst of activity that focussed on the early stage of a project – called “discovery” in software development, which will be followed by other short, intense bursts of co-design and prototyping over the months to come.

Principles in the policy sprint

In the preparation, design and facilitation of its policy sprint, Policy Lab recombined principles borrowed from agile, design and lean start-up as follows:

  • Recognition that a project’s context cannot be fully understood or defined. Instead, the focus is on enabling a team to form and to work together to achieve something meaningful in a short time frame. Instead of trying to do a definitive analysis, the emphasis is on generating provisional accounts of an issue, and on an orientation towards repeated cycles of collective learning, idea generation, prototyping and synthesising.
  • Staging a collective enquiry into the issue. The sprint event recognised that there is already considerable data and analysis within the two departments involved and beyond. The design and facilitation of the workshop enabled a range of participants from different backgrounds collectively identifying “what we know” and “what we don’t know” to synthesise current knowledge and identify gaps relevant to the project at hand – albeit without being fully comprehensive – in order to move forward the project’s goals.
  • An orientation towards cycles of practical work. The development of new products or software should be based on trying things out with real customers (lean start-up) or via cycles of iteration based on defining user needs, delivering software and testing it with users (agile). These partial, temporary placeholders for solutions are termed minimal viable products (in lean start-up), which is the minimum set of functionality that is of value to end users. Agile software developers use the terminology of alphas and betas, which suggests evolutionary development. The policy sprint was positioned as the start of a project structured with fast learning cycles.
  • A holistic approach to understanding the experiences of the target population and those around them. Lean start-up aims to identify customers’ needs and test hypotheses by engaging them directly. Agile starts by identifying user needs and builds software solutions for them. However the context of policymaking is systemic: there is no single customer and nor should the focus just be on only service users. Instead a holistic agile approach for policymaking acknowledges citizens and service users, but also family members, other members of the public such as neighbours, front line staff and managers in delivery agencies, volunteers, but also ministers and other actors in the wider political and policy environment. For example in the case of this project, the sprint brought into focus the role of GPs, HR personnel and line managers as well as family members, friends and colleagues shaping the experience of people becoming ill.
  • Less emphasis on producing definitive documentation, and more emphasis on co-articulating just-in-time knowledge. The emblematic artefact associated with design thinking and agile software is the post-it note. Along with whiteboards, flipcharts and other such low-fi organizational technologies, post-it notes emphasise the provisionality of ideas and knowledge because they are easy to write on, stick up, move around, edit, or remove. Further, the collective activity of writing things down on post-its and sharing and clustering them on a wall makes collective knowledge and ideas visible to everyone – at least those who are in the room and able to see. Collections of post it notes stuck on to flipchart paper, which are then photographed and shared offer repeated snapshots of the current status of a project. In contrast to the highly polished, formal reports produced by government, such arrangements of small fragments of ideas and knowledge highlight policy making as a work-in-progress.

References

Blank, S. and Dorf, B. 2012. The Start-Up Owners Manual: ‪The Step-by-step Guide for Building a Great Company, Volume 1.

Steinberg, M. 2014. Strategic Design and the Art of Public Sector Innovation. In Bason, C. (ed). Design for Policy.

Reis, E. 2011. T‪he Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses.

Report on Policy Lab prototyping workshop during Open Policy 2015

This is an edited version of a post first published on the Cabinet Office Open Policy Making blog

A room full of people making Play-Doh shapes of ducks and monkeys might not seem an obvious way to start a conversation about prototyping in government. But asking people to do this as part of a workshop for policy makers allowed the speaker, Andrea Siodmok, head of Policy Lab, to surface at once three of the key principles of prototyping. The workshop for civil servants was part of Open Policy 2015, a series of events across UK government departments in February 2015.

Policy Lab’s workshop highlighted three principles. First it’s about making ideas tangible. Second, it makes them shareable and discussable, so that they can then be improved upon. Third, it is a collective activity that many people can contribute to. This fast-paced session kept these principles in view as it enabled people to explore what prototyping in government might mean at the earliest stages of policymaking.

The UK Government Digital Service has championed within the Civil Service both understanding of the value of prototyping and the skills and processes to do it in the context of digital services. Nesta has promoted the approach through its prototyping framework. The Design Council has worked with several government departments to generate and prototype service improvements. The Cabinet Office Open Policy Making team argues for the importance of early stage experimentation in policymaking. Policy Lab’s demonstrator projects with the Home Office, HMRC, Department of Health and Department of Work and Pensions, some of which are described in posts on the Open Policy blog, are working out how prototyping can take place in policymaking and what impact it has.

Lab prototyping materials

As a result, in government the idea of prototyping is now spreading beyond digital services. I’ve heard policymakers refer to their policies as “alpha” and “beta” versions.

This is where Lab’s Open Policy 2015 workshop came in. Participants included policymakers from areas such as tax and pensions as well as the Scottish Government and UK Trade and Investment intrigued to find out more. Sharing examples of work from previous roles in local government, Andrea showed how practical, low-cost prototyping helped reduce time to delivery, reduce wastage and costs from rolling out ideas that don’t work, and improve the likelihood of take-up of a policy.

Mixed team of policy makers prototyping their idea

Having excited people about the possibilities in the first hour, the second hour was even more hands on. Andrea set a challenge – but gave people an opportunity to work on their own policy area if they wanted to (none did). The challenge was to respond to the brief “What if visiting a GP was based on the idea of ‘The patient will see you now’? How would this change the whole primary healthcare system?”

Participants then self-organised into three groups, found somewhere to work in the basement conference room where the workshop took place, and got stuck in. They helped themselves to craft materials arranged on a table as well as flipchart paper, post it notes and pens. Policy Lab staff were on hand to help with practical advice and encouragement, for example suggesting looking at three phases – making appointments/arriving, seeing the doctor, and after the visit, and focussing in on one persona representing a particular user segment and their needs. Policy Lab also gave each team one ‘challenge card’ to provide an extra focus, for example “What if the service was people-powered?”

Team of policy makers making their ideas tangible

Over 45 minutes participants moved from analysing what did not work in the current patient experience of GP surgeries, to generating ideas about how to redesign the service based on the principle of ‘the patient will see you now’, and giving it physical form. What was striking was how people who had never met before managed to work together to explore a complex area, generate and realise and share their ideas in an open, creative way.

The workshop ended with a two-minute presentation by each team – all of which involved some degree of role play and performance, bringing their ideas to life and helping the rest of us understand how patients would interact with the new service. Their service concepts were rough-and-ready but complex, combining organisational processes, people’s behaviours and interactions, established technologies (such as buzzers, digital devices or wifi) used in new ways, as well as communications and spatial design.

Policy making team collaborating via prototyping

The takeaways from this workshop were

  • A sense that prototyping is not a technical jargon, but an approach to exploring ideas and taking action to move a project on
  • Understanding that prototypes can take many forms, from physical models, to role play or digital mock ups and that prototyping involves interacting with things as well as telling stories
  • Recognition that users of government services experience policy through their interactions with places and digital and material things
  • The benefits of working collaboratively with people from different backgrounds and perspectives to generate and co-design ideas together
  • The appetite among some policymakers to try out new things and work in a more open, generative way. Policy Lab is currently developing materials for policymakers to be shared via the Open Policy toolkit and also specialist courses for Civil Service Learning.

Prototype of new primary healthcare service