Death by a thousand roundtables

Most policy work is pointless. It doesn’t need to be.

For this piece, I’ve teamed up with Anastasia Bektimirova, a researcher focusing on technology and science policy. Anastasia writes in a personal capacity, so nothing in this piece should be seen as representing the views of her current or past employers. You can keep up with her work on X at @anastasiabekt. 

I have seen the dark universe yawning, where the black planets roll without aim, where they roll in their horror unheeded, without knowledge or lustre or name.

Introduction

The cheap coffee, the untouched orange juice, the collection of half-stale pastries, the same two people hijacking the discussion, and the dawning sense that the Earth is rotating on its axis. These are the hallmarks of the dreaded Westminster roundtable. As friends who have routinely suffered through these events, obtaining nothing from the experience, we’ve often struggled to work out what’s in it for their conveners or sponsors. 

We believe that this speaks to a wider malaise in the world of policy. Fuelled by a flood of money from technology companies with vast budgets, we believe that the quality of policy work has been negatively correlated with quantity. As well as being wasteful, it leads to an impoverished standard of debate on issues that matter.

In some group therapy dressed up as a think piece, we canter through the bad types of work that clog up timetables, inboxes, as well as how organisations fritter away their potential influence. And in the spirit of being constructive, we suggest some potential improvements.

While these observations are written with technology policy in mind, we suspect they have broader applicability.

Genres of work

“Convening a conversation”

A favourite of in-house policy teams, this work usually consists of subjecting start-up founders and a loosely defined set of ‘civil society’ representatives to a series of interminable discussions about a set of ‘principles’.

These principles are usually sufficiently vague that it’s basically impossible to object to them without sounding mental (“I hate collaboration”, “innovation sucks”, “AI shouldn’t be developed responsibly”). This renders them practically useless. But the company hopes that by including a constellation of logos on their reports, they can signpost their broad coalition of support when they’re arguing with regulators. 

The problem with this genre of work is that the trade-offs don’t balance out. The ‘research’ is blatantly self-interested, but doesn’t compensate for it by being interesting. Because it’s desperately trying to hide that it’s an advocacy tool, it stops short of surfacing good policy ideas. The only winners are the people running the work, who can signal how busy they are to their bosses.

Rating: 0/5 - no one understands what you do for a living. You are ridiculed in the street and at dinner parties. 

“The government is already doing it … and we agree”

Common among think tanks desperate to appear relevant, this genre involves proposing initiatives or policies that are, in fact, already being implemented by the government. 

The authors begin by identifying an existing government programme or policy, before repackaging it with a few minor tweaks. Look out for vague calls for “expansion”, “enhancement”, “revitalisation”, or, when the authors are really phoning it in, “delivering on current plans”. While this means a government minister will probably speak at your event, it’s a waste of time and may divert attention away from genuine problems. Once you recognise this trope, you’ll see it everywhere.

Rating: 0.5/5 - could have been a tweet. If you want to suck up to ministers professionally, become a SpAd.

“AI will cost 100 million jobs by 2040 and I am sagittarius rising”

You refer to yourself as a ‘thought leader’ on LinkedIn, but you’re just making it up. The ‘implausibly precise future prediction’ is a cousin of the bad economic impact assessment, but it’s even worse. At least the impact assessment is usually trying to answer a theoretically answerable question. 

Typically, this work is produced by either an academic with a book to sell or a new grad at McKinsey with a PowerPoint to make. Because this work provides certainty and specificity, it acts as a frequent source of zombie statistics that clog up corporate presentations and ministerial remarks, years after anyone remembers how they were invented calculated. For example, we’ve seen a Microsoft claim that AI could add £550B to the UK economy by 2035 bandied around various corporate presentations. The source? A ‘study’ where the authors asked GPT-4… #science

Rating: 1/5 - horoscopes for people with PPE degrees.

“We asked people with a vested interest if they like free stuff”

This usually boils down to surveying the potential beneficiaries of a tax change or spending decision. Unsurprisingly, said stakeholders are usually in favour of more tax breaks or a greater share of public spending. In Alex’s recent piece on UK VC policy, this was often one of the government’s preferred ways of measuring the success of its interventions in the venture market. 

The issue is not that this data is useless, it’s just woefully incomplete. Setting aside their self-interest, while industry representatives will know their pain points and can point to domain-specific subtleties that officials miss, they’re often not well-versed in the machinery of government. We have both endured roundtables where this was apparent. 

Industry representatives are also unlikely to consider (or care about) the implications for others of policies that favour them (e.g. funding being diverted). But these studies skip over this and don’t provide any kind of independent economic analysis. 

Unfortunately, this style of policy work is popular because it suits everyone involved. Consultancies and lobby groups can advocate for their clients, while governments like producing documents that say their policies are being received well by other people.

Rating: 1.5/5 - while it might surface the odd useful nugget of information, this work is usually low value and the enemy of good policy-making.

“We’ll bring the raw pie, hope you have a good oven”

The preserve of the well-meaning but naive. They wrongly approach the policy-making process like it’s a seminar and throw out some half-baked ideas or underdeveloped concepts. They then expect policymakers to do the hard work of turning these into actionable ideas through some kind of undefined co-creation process. 

The end product is filled with lofty goals and grand visions, but short on practical details. Think calls for “a complete overhaul of the higher education funding model”, “greater cooperation between the private and public sectors”, or “making it easier to attract the best and brightest to the UK”. These proposals often ignore real-world constraints such as budgets or political feasibility. 

Rating: 1.5/5 - the policy equivalent of “someone should cure cancer”.

“The people we paid say our work is valuable”

The sketchy economic impact assessment is another favourite of consultancies. It typically involves taking either an individual company (that’s usually in difficult regulatory negotiations) or a whole sector and calculating its multi-billion dollar contribution to the economy. 

This usually involves making some … generous assumptions. Tricks include using iffy self-report survey data or choosing the sunniest possible macroeconomic assumptions and working on the basis they’ll never change. The other is simply just to define the ‘sector’ absurdly broadly. For example, lobbyists often claim that the creative industries’ make a £126B contribution to the UK economy every year. They don’t mention that 56% of this number comes from IT and software. 

While the numbers reported are often inflated, the one advantage this genre has is that its authors are forced to share their workings. You can theoretically semi-hijack their model and re-run it using more reasonable assumptions. 

2.5/5 - they’ve given you some of the ingredients to build a spreadsheet. It’s better than a set of principles.

“We gave them a recipe and told them where the ingredients were”

“Implementation” is not simply code for “someone else’s problem”. Imagine you’ve already been given a green light to implement your policy idea. What’s next? What concrete steps do you need to take? Where does the money come from? Which levers of the government do you need to pull? Those who craft their policy recommendations with that in mind are less likely to immediately have them filed under “nice idea, but utterly impractical”.

The end products don’t just say “improve healthcare”. Instead, like the TBI approached their work on the National Data Trust and Digital Health Record, they focus on specific solutions and don’t shy away from getting their hands dirty by developing their technical design. Or take, for example, Reform’s work proposing to establish the Government Data and AI Service, with 19 recommendations on its setup alone.

Rating: 4/5 - answers all those stubborn “how” questions that usually consign policy ideas to the graveyard of good intentions. The challenge lies in delivering this at the right time, given the political, fiscal, and technical realities of the moment. Objectively good, innovative solutions can be ahead of their time.

“We actually established the ground truth on something contentious”

The more time you spend reading policy work or kicking around Westminster, the more apparent it becomes that a lot of it is based on, well, vibes. If enough credentialed people apparently repeat a statement enough, it becomes accepted. This means bad data or arguments from partisans can become accepted as fact, simply because no one has bothered to check the ground truth. 

For example, we see apocalyptic claims made about the prevalence and impact of online misinformation based on comically weak evidence, which have potentially sweeping regulatory ramifications. 

Plugging one of these gaps and using the data as a basis for a set of informed recommendations is potentially impactful. Air Street Capital, Alex’s employer, took this approach towards European universities’ treatment of spinouts. Universities had long claimed that their rapacious approach to founders, forcing them to sign away large equity stakes or extractive royalties agreements to commercialise their work, was exaggerated by critics. 

Air Street surveyed hundreds of founders who had been through the process, obtaining their deal terms and account of the process, showing up the gap between the universities’ claims and the facts on the ground. This also provided an insight into international best practice and suggested a potential model for governments to follow, which the UK government largely accepted following an independent review. 

Rating: 4/5 - particularly effective if you’re campaigning as an outsider, it can make it significantly harder for your opponents to claim you’re a fringe voice or arguing based on anecdote. It does, however, require an ability to select a good question, willingness to invest in sourcing the data, and the patience to accept that it may take a while to get to the top of the agenda. 

“We bothered to check if our brilliant plan is feasible” 

Policy ideas might stand a better chance of making it off the page and into the real world if implementers are consulted first. Groundbreaking.

No matter how intellectually rigorous, not all ideas are usable. At the same time, there is a wealth of insiders, such as former ministers and civil servants, who understand the government’s institutional DNA and can help make policy proposals more actionable. While you need to balance institutional knowledge with challenge, bringing people with you is more likely to result in high-impact follow-up work.

When at Onward, Anastasia saw the effectiveness of this approach first hand. With insiders, including former ministers, on its steering group, the team received their steers on research. This was helping produce practical policy ideas that the government was receptive to. One example is Wired for Success, where her team, among other things, recommended the measures to make the notoriously lengthy Treasury business case procedures more streamlined for certain science and technology projects. These were so lengthy that the business cases would become out-of-date and would have to be resubmitted.

The report caught the eye of Treasury and DSIT ministers, leading to a government-commissioned review on the issue. It was endorsed by the DSIT Permanent Secretary, and all recommendations were accepted.

Rating: 4.5/5 - sometimes insiders can be too wedded to the status quo, shooting down ideas because “that's not how we do things”. But on balance, their reality checks are a net positive. It's the difference between proposing policies for an imaginary government and ones that might actually see the light of day.

Beyond methodology

Policy work often fails because of bad design, timid recommendations, or a lack of realism. But that’s only one half of the picture. The other is moving beyond the piece of research. Too often, companies or think tanks publish a PDF, hold a roundtable, secure a desultory media or newsletter reference, and decide it’s mission accomplished. And the ideas they front up are too generic, too vague, or too inoffensive to matter, because they seek consensus rather than division. This means that even well-designed work accomplishes nothing. Policy isn’t a more secure or better compensated branch of academia. Stacking up publication credits doesn’t come with any rewards. 

There are essentially two models for moving beyond the illusory version of ‘influence’ we often see. At the risk of being crude, you can divide them into ‘insider’ versus ‘outsider’.

The ‘insider’ model boils down to having a forensic knowledge of i) who is actually relevant or wields influence over a certain policy area, ii) understanding their motivations, pressures, and room for manoeuvre, and iii) how to win them over. At its most effective, it leads to individual groups or research organisations being able to shape entire areas of policy. The Tony Blair Institute for Global Change or Labour Together with this government or Onward with its predecessor spring to mind.

The ‘outsider’ approach involves using policy work as a battering ram to shove an issue up the agenda, whether those in the system like it or not. This requires i) a willingness to be confrontational publicly, ii) a thick skin, and iii) an ability to leverage media effectively. As this approach involves a degree of bridge-burning that is culturally alien to most people in policy, it’s less popular. But it can be surprisingly effective. Many institutions (especially in the UK) are so unused to being challenged robustly in public, they don’t really know how to respond to it. 

If your approach can’t be categorised as one or the other, the chances are that it’s not achieving much. This is the reality of most policy work: patchy engagement with peripheral ‘stakeholders’ to build ‘mood music’, matched with cautious external communications designed to avoid offence. 

The thing successful approaches have in common is persistence and discipline. Making progress can take months or even years. Good ‘insiders’ have to navigate their way around a morass of intermediaries, middlemen and associated hangers-on who clog up the corridors of power and will persistently want to have coffee with them. Good ‘outsiders’ need to know which arguments are worth having, when to accept a compromise, and when to push for more.

Disclaimer: These are my views and my views alone. They aren’t those of my employer (Air Street Capital), my local newsagent, the guy who sells me wine, or anyone else. I’m not an expert in anything, I get a lot of things wrong, and change my mind. Don’t say you weren’t warned. Any resemblance to real-life roundtable organisers, living or dead, is purely coincidental. Or is it?