Data gaps are under-reported, because it’s hard to write about data that doesn’t exist.

As we've written about before, newspapers publish endless stories on house prices, where there's lots of data - but few on rental costs, even though millions of people rent. That’s partly because the Office for National Statistics doesn't collect much data on rentals.

To tackle this problem, I’ve been thinking about how to map data gaps, and make them more visible.

And I think the best way is actually to think about questions, instead of data. What are the important questions that the government can’t answer?

Obviously, 'important' is subjective! But one source of clearly important questions is Parliamentary written questions, which are the formal questions that MPs and peers ask the government. Where the government doesn't have the data to answer them, it has to say so.

So this post introduces new research: a data analysis of 200,000 Parliamentary written questions, and what they tell us about the UK's missing numbers.

Our modest goal: to find the UK’s biggest data gaps.

What we did

Building on some previous research of ours, we strapped on our coding hats 🪖, and did the following:

• First, we scraped all the written questions in Parliament from December 2019 to February 2023, from TheyWorkForYou, which gaves us about 200,000 questions.
• Next, we flagged questions asking for quantitative information, with phrases like “how many” or “how much” - which showed that about a fifth of questions wanted data, just under 40,000.
• Then we flagged questions where the government apparently said the data was "not held", "not collected", etc. About a quarter of quantitative questions were answered like this.

And we ended up with a dataset of around 10,000 questions where MPs apparently both (i) asked for data, and (ii) were told it was not available. So: missing numbers.

Then we spot-checked the questions to check our method. It wasn't perfect, but it was very decent. (It helps that Parliament uses formal, consistent language.) You can download the full dataset here.

Sometimes, MPs ask about strange things, like jobs for clowns. But most are extremely serious, covering the issues that affect MP's constituents. And overall, they tell us what MPs need to know.

Data gaps by department

Firstly, we looked at how often each government department said that data wasn't available. (See the code.) And there were were huge differences:

• At the Department of Health & Social Care, around 40% of quantitative requests were unanswered (though we can cut them some slack, as this was during the Covid pandemic).
• At the Home Office and the Department for Work & Pensions, around a third were; at the Ministry of Justice the proportion of unanswered quantitative requests was 30%, and the Department for Education  27%.
• But the proportion was much lower at other big departments - almost all others were below 20%.

Of course, we need to be cautious here, as the numbers are approximate. Without reading each question, we can't be sure that we've tagged it correctly, or if the MP was asking something impossible. It's probably most useful to consider the differences between departments.

Given that, it's not surprising that the health, benefits, justice and education departments would get requests for data, since they run massive operational services that affect people's lives. (The Foreign Office, by contrast, largely seems to get asked about wine.) It's more surprising that they seem to struggle to answer them more than other departments.

Now let’s dive into what these unanswered questions were about.

The topics with the biggest data gaps

Each question scraped has a title. We can use this to see which topics were least likely to get an answer.

Other than Covid-related topics, the major topics with the highest proportion of unanswered questions were:

1. Benefits - grouping together benefits like Universal Credit and PIP
2. Asylum, refugees and migrants
3. Child maintenance
4. Energy meters
5. Armed forces housing

This seems plausible. The DWP Select Committee has repeatedly criticised the government for the lack of visibility over the benefits system; the statistics regulator has expressed concerns about the use of asylum statistics, while the National Audit Office has noted gaps in the data available on smart meters.

We also used GPT-4 to try tagging questions, which worked quite well. We used it to tag questions to the Department of Health & Social Care. This helped us identify major clusters of unanswered questions in these areas.

In healthcare, MPs often struggled to get basic prevalence information, whether:

• the number of people diagnosed with particular conditions, like silicosis
• diagnoses broken down by characteristics, like the number of women with meningitis, or region, like autism in the East of England
• or diagnoses for particular (important) groups, like prisoners with mental health conditions.

Also, funding is a topic it's surprisingly difficult to get information about, e.g.

• funding for particular conditions, like endometriosis, or
• funding on social care by local authority
• funding at a local level, especially per hospital, which MPs often care about.

Following on from this, hospital-level information in general often seems to be poor, e.g.:

• how many A&E visits there are per hospital
• what waiting times are per hospital.

And finally, workforce is a huge one, with topics like:

• vacancies - how many current GP vacancies are there?
• retention - how many dentists are still working 5 years after qualifying?
• skills - how many specialists are there with particular skills, like Parkinson's nurses or motor neurone nurses?

You can see the tagged questions here - there are many more examples under each topic.

This gets really worrying when you look at the dataset over time. It's immediately clear that MPs often ask the same thing over and over again - yet the information doesn't seem to improve.

What next?

We think statistics producers should be monitoring Parliamentary questions, to tell them where data needs to be better. After all, MPs deserve answers to their questions, and so do we all.

If you can help us make this happen, we'd love to talk.

If you're interested in this research - or even better, if you can fund us to do more of it! - please do get in touch.

Samuel Eni was stopped by police last August while at Paddington station. He wasn't suspected of any crime: the officers told him he'd been 'flagged' and asked him to account for his presence, under something called 'stop and account'.

Samuel filmed the encounter and put it on Twitter, where it promptly went viral (low-quality copy here). Watching the video, it's clearly an intrusive experience. He's just catching a train, and suddenly two uniformed officers are asking him to justify his presence.

So it's surprising that police don't have to record data on 'stop and account'. (And many people don't know what it is - I didn't till I read Samuel's story. It's like stop and search, but without the search: you'll be stopped, and asked who you are and what you're doing.)

This wasn't always true. Police officers once had to record these stops, and the ethnicity of the person stopped, and the data showed there were millions of stops each year. But in 2010, the requirement was quietly dropped. Now only three forces still collect data on these stops.

As the Prime Minister controversially plans to expand police stop and search powers, this is the story of how 'stop and account' became a missing number.

Stop and account: a brief history

Until 1981, Britain's infamous 'sus laws' allowed the police to search anyone they vaguely suspected, more or less at whim. The sus laws were replaced by modern stop and search law in the 1980s.

Meanwhile, stop and account continued as an informal practice, with no requirement for reasonable suspicion, and not covered by law. (This is because stops and account is technically voluntary: in theory, you can walk away, although in practice that may be difficult.)

In 1999, following the murder of Stephen Lawrence, the report of the Macpherson Inquiry warned that excessive stops were a major problem:

If there was one area of complaint which was universal it was the issue of "stop and search"... conditioned by... experiences of being stopped under traffic legislation, drugs legislation and so called 'voluntary' stops.

Macpherson was clear that while stops were an important part of policing, they were also highly intrusive and easily abused. So he recommended more data was needed on all stops, including the ethnicity of the people stopped:

It is essential to obtain a true picture of the interactions between the police and minority ethnic communities... All "stops" need to be recorded, and related self-defined "ethnic data" compiled.

The government agreed, and by 2005, all the UK's police forces were recording data on stop and account. The new data showed that 'stop and account' was actually very widely used, at least in certain areas: there were 2.2 million stop and account stops in 2008-9.

2010: Stop and account data becomes 'optional'

However, Macpherson's recommendations for stop and account data were never formalised in law - because stop and account is not a legal power.

So it was easy for a subsequent government to undo the reforms. In 2010, the Police Code of Practice was amended to no longer require forces to collect stop and account, on the grounds that it would "produce significant savings in police bureaucracy". No public consultation was held.

Most forces immediately ended the recording of stops: by 2012, according to the StopWatch campaign group, only 10 out of 43 forces were still recording data.

One exception was the Metropolitan Police in London. However, last year, the Met also quietly stopped collecting data. Again, there was no consultation, but you can see the data silently vanishes from the Met's public stop data dashboard in January 2019 (click on "Stop and Account by SDE").

2019: How many forces still record data?

To see what the situation is now, I sent Freedom of Information requests to 44 police forces in the UK asking how many 'stop and account' stops they'd carried out, and the ethnicity of the people stopped, in the past three years.

Forty-one forces said they were not recording stops: only three (Gwent, North Yorkshire and Northumbria) provided data. But even this data is patchy: the numbers jump around implausibly, or significant numbers of people have their ethnicity recorded as 'Unknown'.

So in short, only three police forces in the UK collect any data on stop and account, even though millions of these stops are probably still happening each year.

Why data matters

I spoke to Katrina Ffrench, CEO of StopWatch, which campaigned against the 2010 reforms, about why stop and account data matters. She said:

Any stop means that people feel their day has been interfered with. Figures allow us to understand if police time is well-used, and whether ethnic minorities are having more contact with police.

She notes that from stop and search data, we know that at the time of the Macpherson Inquiry in 1999, black people were around four times more likely to be searched than white people. Now, black people are eight times more likely to be searched. But we cannot know if the same applies to stop and account, or if particular forces are using these stops disproportionately.

Katrina also says that without data, it's easy for the authorities to dismiss people who do report excessive stops:

Data allows lived experiences to be validated. If there's no data, then people are often told their experiences are anecdotal.

A case study in missing data

The story of stop and accounts highlights two themes I'm noticing often as I write Missing Numbers:

1. Missing numbers are often used as a way to dismiss experience as 'anecdotal' - by the very same authorities who choose not to collect data.
2. It's easy to scrap data if it's not required by law. In 2010, stop and account data simply vanishes. This data was introduced by one of the most significant public inquiries in modern history; tracks stops that are widely used; and tracks events that are highly unpleasant and intrusive to experience. Yet despite this, it was withdrawn with no public consultation and almost no reporting.

By the way, it's important to know that if you are stopped under 'stop and account', you do have the right not to answer questions. Here's the BBC (featuring Katrina Ffrench) explaining your rights if you are stopped.

Photo credit: City of Toronto Archives.

Looking at my map of house price by square metre recently made me wonder why government collects good data on house sale prices, but almost nothing on what it costs to rent.

So I went digging to find out why not. Surprisingly, it turns out the government has known for some time that rental costs are important missing numbers. But despite an official call for better data back in 2017, nothing has changed.

And the reason why illuminated a big problem with the UK’s statistical system.

Renters: the missing millions

When it comes to house sales, we have quite good numbers. The Land Registry releases both an overall house price index, and data on individual sales. This is one reason why there are endless news stories about house prices.

But when it comes to rentals, there’s no equivalent data, even though one in five households now rents from a private landlord.

Worse, there are two incompatible official datasets on what it costs to rent a house in England:

1. The Private Rental Market Statistics series: this tracks the absolute mean rent by local authority.
2. The Index of Private Housing Rental Prices: this measures the relative change in rents, by high-level region.

But the first series can’t be compared over time, while the second series is a relative index, not an absolute one. As a result, we don't know:

• How have rental costs changed over time, in an individual local authority?
• What are tenants really paying to rent a particular type of property, in a particular location?
• How often do landlords put up rent on a typical property?

Bad news for everyone

This makes the market inefficient: landlords have to pay lettings agents to find out the market rents for their property.

And even professional analysts struggle to understand the market. While some commercial datasets exist, they’re incomplete; measure flow rather than stock; and expensive.

I spoke to campaigners Generation Rent and housing market researchers Residential Analysts, and both mentioned rental price data immediately as a big problem.

Housing market expert Neal Hudson has written about how this affects local authorities (pdf):

Limited publicly available rental data is a big challenge, particularly for the private rented sector. It hampers our ability to measure how much renting households actually pay and their affordability at a local level.

I also think this lack of data, and consequently news coverage, is one reason renters don’t have the political clout of homebuyers: they’re just less visible.

Why is the data missing?

For me, this is the most surprising part of the story.

The Office for Statistics Regulation (OSR) exists to ensure that UK statistics are high-quality. It called nearly 2 years ago for better rental market data - but since then, nothing has changed, because the OSR lacks powers to match its responsibilities.

Back in November 2017, the OSR interviewed housing data users - from businesses to academics to campaign groups like Shelter - and published a systemic review of housing statistics. The majority of users asked for better rental statistics:

The majority of users we spoke to cited frustration at the lack of robust, timely and insightful statistics about the private rented sector… The dispersed nature of existing information about the sector hinders users in developing an understanding of changing rental patterns.

In particular, users wanted better data on real rents:

[Users need] robust statistics about actual rent prices for all four countries that can be compared over time and across geography; and can distinguish between new and existing lets.

In response, a group of government statisticians published an Action Plan. But their Plan committed to just one Action: the ONS would er, write an article about rental statistics.

This article duly appeared, and said that the ONS struggled to produce better data because it relied on data held by another part of government: the Valuation Office Agency (VOA). But the ONS promised to ask the VOA for better access.

Eighteen months later, only two things have happened:

1. ONS has published a one-off analysis of data from tenancy deposit schemes, but this has little to say about rental costs.
2. There is no 2. None of the price series have improved. I asked the ONS about what progress it had made on access to VOA data, and it said only: “Discussions are still ongoing”.

So although the OSR has done its job, nothing more has changed. The levers of the UK statistical system are broken.

What needs to change?

Statistical governance reform may seem like the definition of “dry”, but this is a huge political deal.

Renters are multiplying. Users are calling for better data about their experience. But UK statistics hasn’t responded to those needs.

So what needs to be different?

1. On rental data, we need the ONS to accept that it has a social responsibility to publish meaningful data about rental prices, which now affect more than a fifth of the population. If important data is locked behind intellectual property agreements, the ONS must enlist political support to unlock it.
2. More generally, the regulators at the OSR should be given greater power to enforce change. For now, the OSR should publicly reprimand statistics producers who fail to follow its guidance.

Change is in the air!

I was happy to see the Public Administration Select Committee calling for some of these bigger changes in their new report on the governance of UK statistics.

Excitingly too, the very first recommendation in the report is on data gaps:

Those producing official statistics do not understand all of today’s users and potential users of statistics and how statistics are used... Producers are, therefore, not able to close statistical gaps...We recommend that UKSA should lead cross-government research... to establish where data gaps persist.

Missing Numbers is catching on! More on the Committee’s report soon.

Photo credit: Library of Congress

Since 2010, the vast majority of government spending has been public. UK local authorities must publish all spending over £500, and Whitehall departments must publish all spending over £25,000. The Government says boldly: "The public should be able to see where their money goes and what it delivers".

So I'm surprised to learn that another public body, the Bank of England, has recently bought corporate debt worth more than £9 billion, but never published details of its spending - without having a clear reason.

This seems like a strangely inconsistent transparency policy, not to mention a much larger gap in democratic oversight.

Background: the missing £9 billion

After the financial crash in 2008, the Bank of England began 'quantitative easing', buying up UK government debt to stimulate the economy. After Brexit in 2016, the Bank added corporate debt to the policy, hoping this would make it cheaper for companies to borrow and free up investors' cash for riskier assets.

Under the Corporate Bond Purchase Scheme, the Bank has splashed out on £9 billion of investment-grade sterling corporate debt. But where exactly did that £9 billion go: what does the Bank now own?

This is a missing number. We do not know.

I'll say that again, more slowly: even though we make local authorities publish truly microscopic spending decisions, we do not require the Bank to share spending decisions that are millions of times larger.

The Bank does publish a list (Excel) of the 144 companies that were eligible for the scheme - but not the amount of each company's debt actually purchased.

Why this data is missing

A Bank of England report on the scheme explains why it did not share purchasing decisions at the time:

It was thought that by disclosing less information, the Bank would reduce market distortions that might have arisen as a result of the publication of information on individual bond pricing and allocations. Moreover, it was thought possible that the publication of the prices paid in auctions might effectively render the Bank a price setter.

In short, the Bank feared distorting the market, which seems reasonable. However, since all the purchases under the scheme were complete by April 2017, the risk of market distortion is surely now long past.

So I asked the Bank if it would now publish details of its corporate debt holdings. But the Bank said it did not want to comment.

The problem with these missing numbers

This is probably just classic civil service caution from the Bank. But I think the decision not to publish should be debated more publicly.

This is for the following reasons:

• The list of eligible bonds includes the Daily Mail & General Trust. It seems like basic democratic oversight to know whether our central bank owns the debt of large media companies.
• The list also includes overseas companies, such as Apple, Wal-Mart and Deutsche Telekom, which "make a material contribution to the UK economy". But some of these companies compete with UK firms, and some have historically paid little UK tax. The nature of the "material contribution" should be discussed.
• The list also includes fossil fuel companies, like Shell and BP. As other parts of government divest from fossil fuels, we should have a public conversation about whether it's still appropriate for the Bank to hold these companies' debt.
• Governor Mark Carney has made public commitments to improve the Bank's transparency. This decision doesn't seem in line with the Bank's public position.

Things are better in Europe (slightly)

In 2017, MEPs asked the European Central Bank to improve the transparency of its equivalent corporate bond purchasing programme. The MEP Ramon Tremosa said:

A more transparent ECB will be stronger in its monetary policy. More transparency in the corporate debt purchases is fundamental to ensure that the single market and state aid rules are not undermined through the back door.

The ECB eventually complied, up to a point. The coupon and expiry date of each bond holding is in this Excel file, though not the total amount.

What should happen

Even if the BoE is justified in not sharing the data with the public, an issue with this level of democratic consequence needs democratic discussion. But this form of quantitative easing has never received significant attention in Parliament.

At the very least, a Parliamentary committee should be given access to the Bank's historic purchasing data, publicly debate the issue, and decide that no extra transparency is needed.

The Bank plans to restart reinvesting in corporate debt in the second half of 2019. So now is a good time to begin the debate.

Photo credit: Library of Congress

So far, this blog's mostly been about data that government has never measured. But there's another kind of missing number: official statistics and other data that government has chosen to stop measuring.

This matters because discontinuing a dataset is a profoundly political act. And right now, we don't have any way to know that it happens.

In 2011, the Government decided to stop collecting data on stop and account (a type of police stop and search). And since 2010, changes and cuts to the data on Sure Start centres have made it hard to find out how many have closed.

These are just examples I happen to have heard of: there are probably lots more under the radar.

So I've decided to track this, and I would like your help.

The UK doesn't know what it's stopped collecting

Trying to get a clearer picture of discontinued datasets and statistics, I FOI'd the UK's Office for National Statistics about what it has stopped publishing recently.

To my surprise, the ONS said it didn't know:

You asked...during the past 5 years, what statistical bulletins/datasets have been discontinued by the ONS? We have conducted a search of our systems and files and unfortunately we do not hold the information you require.

I'm kind of amazed that even the UK's national statistics body doesn't know what data it's stopped collecting itself. And no part of government seems to be keeping a broader list.

So I've set up graveyard.missingnumbers.org to build up a picture of this more broadly across government.

A website to keep track of discontinued data

The purpose of graveyard.missingnumbers.org is to track UK government datasets that have been cut. It couldn't be more minimal: just a URL for each dataset, any information published on why it was discontinued, and the source.

So far, it's mostly datasets that were collected by local government, simply because that's what I've been able to find.

But I already know about more than 90 datasets cut by local authorities since 2010. These fall into three broad categories:

• Data about funding schemes that were expiring, or cut by the coalition government
• Data that was no longer needed because it was collected elsewhere
• Datasets that look useful, but might have revealed unpalatable things about the consequence of austerity.

Can you help?

I've now FOI'd all the big Whitehall departments to see if they know what Official Statistics they've cut since 2010. I'll report back here.

But I'd also like to know:

1. Any examples of government datasets or statistics that we're no longer measuring
2. Evidence and stories about the datasets or statistics already on graveyard.missingnumbers.org - who decided to stop collecting them, and why.

Can you help? Email me anna@anna.ps, or tweet @darkgreener.

Photo credit: Library of Congress

Back in 2014, the Home Office introduced the Right to Rent scheme, which requires landlords to inspect tenants’ passports to check they are not illegal immigrants.

But earlier this year, the High Court slapped a halt on plans to expand the scheme, finding that it caused landlords to discriminate against foreign tenants, and breached human rights law.

Amazingly, the High Court’s judgement was based not on official data (there was none), but on data collected by JCWI, a tiny charity. I’m writing about it because it’s a powerful and unusual example of a third party filling in missing numbers.

The judge also berated the Government for not collecting better data on whether passport checks actually deter illegal immigrants, in a tone I can only describe as judgemental:

The nail in the coffin of justification is that, on the evidence I have seen… the [Home Office] has put in place no reliable system for evaluating the efficacy of the Scheme.

So how does a 10-person charity collect better data than the Home Office, with all the powers of the state?

The Government’s numbers on Right to Rent: inadequate

Checking someone’s immigration status is complicated, and getting it wrong gets you 5 years of jail time. So the scheme obviously creates incentives to ignore tenants who look or sound foreign.

Many people pointed out this out the instant the scheme was proposed. So in its first pilot of the Right to Rent scheme, the Home Office ran a small study, staging a few hundred encounters with landlords, and comparing the response rates to white and BME tenants.

The results of the study found “no major differences in tenants’ access to accommodation”. And so the Home Office rolled out the scheme across England.

But how meaningful was the Government’s data?

How a charity filled in the gaps

The Joint Council for the Welfare of Immigrants (JCWI), a small charity, argued that the research was inadequate:

• The numbers were small and the data was oddly aggregated
• The methodology and aims of the study were unclear
• The scenarios used were limited and unrepresentative

I agree: I’ve read the Home Office study, and it’s essentially impossible to figure out how it was meant to work.

So JCWI collected its own data. It sent 1,800 emails to real rental adverts, from different ‘tenants’ with names like Colin, Harinder and Ramesh. It found clear evidence that landlords were less likely to reply to foreign-sounding names.

Then JCWI went to court.

What the courts said

At the High Court in March 2019, JCWI and others brought a Judicial Review of plans to expand the policy.

The judge agreed: the Home Office’s study was inadequate, and JCWI’s data showed that the scheme breached human rights law.

The written judgement notes that the Home Office collects no data on whether the Right to Rent scheme (which is still going in England) even works:  

The Defendant was not collecting data that would allow it to measure discrimination resulting from the Scheme, the cost-effectiveness of the Scheme, whether the Scheme was resulting in migrants’ voluntarily leaving the UK or increasing the propensity of rogue landlords or the impact of the Scheme on agents and landlords.

As a reminder: the Home Office has an annual budget of £8.9 billion. JCWI has an annual budget of about £1 million. But still it somehow managed to collect better data.

Collecting data to force change

JCWI is the first to admit that its data isn’t perfect, and the Home Office could easily run definitive studies to confirm or disprove the outcome - if it wanted to.

But as Chai Patel, legal director at JCWI, told me: “The Government made no effort beyond the pilot to monitor the effects of the scheme. I think it's because they know they wouldn't like what they found.”

He highlights other examples - hospitals must now charge migrants for treatment, but no data is recorded on how much the extra admin costs the NHS, so it may well cost more than it saves.

Chai says JCWI collected its data deliberately to force the Judicial Review, and adds: “I think this is something that should be done much more, in different areas of policy”. I agree!

Thanks to Dave Whiteland for alerting me to this story!
Photo credit: Library of Congress

I need your help

Thanks for reading! I’m keen to collect more ideas about important missing datasets, on any policy area in any part of the British state. If you’ve got any hints or tips, please drop me a line at anna@anna.ps, or tweet @darkgreener.

This week: Missing Numbers takes a look at the missing numbers behind ‘land options’. These are private agreements that housebuilders and other firms use to control land. But we have no numbers on how widely they’re being used - even though the Government promised two years ago to make them more transparent.

Land options: what on earth are they?

If you’re a housing developer, the first thing you need is land to build on. There are two ways you can get it:

1. Buy it. But this costs money. And if the demand for houses falls in future, you own land that you’ll have to sell at a loss.
2. Sign a private ‘option agreement’ with a landowner. This is safer, because you don’t pay the whole cost of the land, you just give the landowner a deposit. In return, you get a guaranteed option to buy the land in future if you want.

In category 1, we know roughly what each housing developer owns directly, because they have to register it at Land Registry. We even have some idea where it is - last year I mapped the land that the housebuilders own for Who Owns England.

But when it comes to option agreements, nobody knows:

• how many options are out there, and who controls them
• how much land is tied up under option agreements, and where it is.

And currently there’s no way anyone could know, because these options are often private contracts, kept in a lawyer’s office rather than at Land Registry. As Shelter explained in 2014:

Developers also hold ‘strategic land banks’ – sites without planning permission that they may wish to promote one day. These sites are… often held under option – meaning that they are not recorded as being in the developer’s ownership at all, and that there is no public record of where or how large such strategic land banks are.

Why does it matter?

Britain is in desperate need of new housing, and not enough houses are being built.

New houses can only be built if there’s a plentiful supply of land for sale. But businesses that control land have incentives to release it slowly, so every new house is nice and expensive. This is called ‘land banking’: there’s endless disagreement about whether it goes on, multiple reviews have been held, and the argument is almost entirely data-free.

As citizens, we want our government to stop companies keeping land off the market intentionally. But the hidden nature of options makes it impossible to build political support for change. After all, if there’s no data on the problem, it’s hard to lobby for a solution.

Option agreements also make it harder for local authorities to plan housing. They make it harder for smaller builders to enter the intensely concentrated housebuilding sector. They even make life harder for landowners, because they don’t know the fair market price for their land.

All these problems are well-known: for years, from the Lyons Housing Review to the National Housing Federation and the new Centre for Housing Evidence, armies of experts have raised these concerns.

Across the political spectrum, thoughtful analysts agree - here's centrist-Tory think-tank Onward:

Government should legislate to require all options on land to be centrally registered... With a central register of options, smaller builders would be able to see new opportunities, find out who controls which land, and, if transparency revealed a lack of competition in local areas, further action could be taken to address it.

How big is the problem?

By definition we don’t know, but Shelter quote an Office of Fair Trading estimate from 2008 that around 80% of developers’ land is held in ‘strategic land banks’. Shelter also found that at least 400,000 plots were recently held in strategic land banks. We don’t know how many are controlled by options, but this Who Owns England analysis suggests it’s a lot.

I have read the whole of the Office of Fair Trading report - no wonder I get invited to so many parties - and its estimate is based on survey data. Even back in 2008, the report was flagging up the lack of data on options as a problem.

What’s the Government doing?

Good news! (See, this blog isn’t all doom and gloom.) In 2017, prompted by the work from Shelter and others, the Government promised to take action on options in its Housing White Paper:

The Government will consult on improving the transparency of contractual arrangements used to control land. Following consultation, any necessary legislation will be introduced at the earliest opportunity.

Bad news! Two years later, the Government not only hasn’t put forward any new legislation, it hasn’t opened a consultation.

Some might blame Brexit for the delay - but MHCLG has found time to hold four consultations on other, arguably less important proposals in the Housing White Paper.

Even Government ministers are impatient with the lack of progress. In June 2018, Oliver Letwin MP wrote in his ‘land banking’ review, by necessity largely interview-based:

We have been somewhat dismayed by the paucity of publicly available data... I recognise that the Government has commissioned work to make options and other agreements on land transparent by ensuring that they are recorded at the Land Registry; I urge Ministers to expedite this work so far as possible.

That’s Whitehall-speak for “Get on with it”. But still, nothing has happened.

What needs to change?

It’s simple: we need an open, compulsory register of options over land, so we have evidence on how many options exist, who controls them, and where they are.

Only then can local authorities plan the housing we need, only then will smaller builders have equal access to information, and only then will we have a truly transparent land market.

This might all seem a bit abstract, but given how deeply the land market is tied to housing, and the daily misery caused by the housing crisis, these are really vital missing numbers.

I’ve asked my MP to ask in Parliament what action is being taken on the White Paper’s commitment: if you feel so moved, please ask your MP too.

UPDATE 7 June 2019: In a written question, David Drew asked the government what’s happened to the consultation. The answer from Kit Malthouse is not very enlightening.

I need your help

Thanks for reading! I’m really keen to collect more ideas about important missing datasets, on any policy area in any part of the British state. If you’ve got any hints or tips, please drop me a line at anna@anna.ps, or tweet @darkgreener.

Photo credit: Library of Congress

This is a post from Missing Numbers, a new blog about the gaps in government data by Anna Powell-Smith. Learn more about this project, or follow it on Twitter.

One of government's most important jobs is to protect consumers from mistreatment. But how much data are different industries required to gather and publish about misbehaving firms?

I thought I'd compare how different industries do things. First up, lawyers.

Why complaints matter

Most consumer-facing industries have a regulator or “ombudsman”. If your bank loses your money, then ignores your complaints - the Financial Ombudsman will help.

Some regulators also publish data, like the total complaints they receive per year, or per firm. The Financial Ombudsman publishes this, and each year there are headlines like “Bank of Scotland receives most complaints - again” (ouch).

But some regulators publish zilch. Zero. So I’ve started to look at how the situation compares across industries. Which regulators publish data on complaints and investigations, and which don't? And why?

I’m looking at everything from pensions to plumbers, but have started with the legal profession: barristers, solicitors and conveyancers.

And finally, I've written what I think they should be doing better.

1. Complaints about poor service or fees: the Legal Ombudsman

Complaints about poor service and high fees go to the Legal Ombudsman, for all kinds of lawyer. Here, the data's okay, but it's not the most serious type of complaint.

• What data do they publish? General data on total complaints by area of law. Decisions data showing total complaints made/upheld per firm, in past year.
• Historic data? Nope, just one year.
• What's missing? Only records raw number of complaints, which doesn't tell you much unless you know how many cases the firm handled.

More serious complaints go to the professional bodies, who investigate them along with internal reports of misdoing.

2. Complaints about dishonesty or negligence: the professional bodies

For serious complaints about professional standards, there's a major difference between what barristers, solicitors and conveyancers publish.

a. Barristers: the Bar Standards Board

• What data do they publish? A decent searchable list of decisions about individual barristers
• Historic data? Kept online for 2 years, or indefinitely for suspension or disbarring
• What's missing? Does not include their chambers. No aggregate data (total number of hearings per year etc).

b. Solicitors: the Solicitors Regulation Authority

• What data do they publish? A decent searchable list of decisions about individual solicitors
• Historic data? Kept online for 3 years
• What's missing? Does not include their firms. No aggregate data.

c. Conveyancers: the CLC

• What data do they publish? Essentially nothing. “All complaints are recorded” says the information for customers (pdf), weakly. We don't know the number of complaints, the firms who have been investigated, or the actions taken.
• Historic data? None
• What's missing? Everything!

Conclusions

No wonder conveyancers have such a poor reputation! How does data on barristers and solicitors come to be so much better than data on conveyancers?

I've written to the CLC to ask them if they plan to make more data available, but haven't heard back yet.

This is what I think all Ombudsmen and regulators should be required to release:

• For each firm, the proportion of customers who complained: The raw number of people who complained per firm is pretty meaningless, if you don’t know whether it’s 0.001% of customers or 10%, and can't compare it with other firms. (Sometimes this might be market-sensitive data, but there are ways around that.)
• For each firm, the proportion of complaints upheld: To see which complaints were valid.
• Past data: To see trends over time.
• As much raw data as is compatible with the complainant’s privacy: To allow more meaningful analysis.

One odd thing I've spotted is that there seems to be little legal requirement on any of these regulators to release data. The Legal Services Act 2007 founded the Legal Ombudsman, but has nothing to say about data.

But perhaps other industries are different. (After all, the law is written by lawyers.) Next time: Finance! This is a heavily regulated industry, so it'll be interesting to see if there are similar blind spots.

I need your help

I’ve got several blog posts on different missing numbers lined up, but I’m really keen to collect more ideas, on any policy area in any part of the British state. If you’ve got any hints or tips, please drop me a line at anna@anna.ps, or tweet @darkgreener.

This is a post from Missing Numbers, a new blog about the gaps in government data by Anna Powell-Smith. Learn more about this project.

If you’ve read the news in Britain in the past decade, you’ve probably heard of Universal Credit, though you may not be too sure what it is. It’s the current Government’s main welfare policy, which is a fancy way of saying that it is the mechanism through which the government ensures that people who can’t work don’t actually starve or become homeless.

Universal Credit (UC) is a big change from what came before, and many others have written about whether they believe it’s working. But that’s not the point of this blog, Missing Numbers.

Here, I focus on whether the government actually collects the data required to know if this policy is succeeding. What data would the government need, in order to know whether or not UC was working?

Let’s start by looking at the Government’s official aims for the policy. These are:

1. Make the benefit system fairer and more affordable
2. Reduce poverty, worklessness and welfare dependency
3. Reduce levels of fraud and error.

Of course, there’s lots more data it could track to make sure the policy is being delivered successfully, and not having unwanted side-effects - I’ll write about that data in future.

But in short, the official, measurable aims are:

1. Reduce the cost of the benefit system
2. Reduce unemployment
3. Reduce fraud and error.

In this post, I just want to look on point 1: what data the Government is collecting about the savings from UC. (I've been reading Cass Sunstein's excellent book on cost/benefit analysis, which argues that this is the most important data any government can collect.)

What data exists on whether Universal Credit reduces the cost of the benefit system?

In 2012, as UC was first being developed, the Department for Work and Pensions (DWP) laid out its plans for monitoring costs and benefits in its UC Evaluation Framework. It said it would look at five questions:

How much does Universal Cost [sic] to deliver? When do the different benefits and costs occur and what is the difference between the short-run and the long-run steady state costs and benefits? How are the benefits and costs distributed between different groups? What is the monetary value of the various impacts identified under other evaluation themes? What is the overall net value of Universal Credit?

It said these would be tracked via “administrative data”, “programme finance information”, and “outcomes from impact assessments”. But it didn’t go into more detail about what data it would use, or explain how it would use this data to answer the questions.

In 2016, DWP published an updated Evaluation Framework. It did not answer any of the questions above. Instead, on cost/benefit analysis, it said only:

We are continuing to undertake a programme of economic analysis.

And the section on ‘future work’ did not mention cost/benefit analysis at all.

In June 2018, under pressure, DWP made public its Business Case for UC. This estimated that UC would save £8 billion per year once fully rolled out. The Business Case is long, and it would be painful to unpick how data-driven it is, but luckily we don’t have to do that, because the National Audit Office has done it for us.

The National Audit Office's verdict: the data's not strong enough

In its report on Universal Credit in June 2018, the NAO stated the cost/benefit data in the Business Case was not strong enough to draw any firm conclusions:

We cannot be certain that Universal Credit will ever be cheaper to administer than the benefits it replaces… We cannot judge the value for money on the current state of programme management alone.

In the NAO's view, this was because three key data inputs to the cost/benefit calculations were unreliable.

Firstly, because the Government’s savings estimates do not include the extra costs that UC creates for local authorities, which the NAO said DWP does not measure...

The Department...  has not sought to systematically collect data on wider costs. It will therefore have no means to assess the full monetary impact that Universal Credit is having.

...and moreover has refused to measure:

The Department has said it will not measure these additional costs until local organisations can demonstrate they are being caused by Universal Credit.

Secondly, because the savings estimates assumed that UC would help many more people find employment - but the data on which this was based is limited and unrepresentative:

It is not known whether the employment impact identified by early evaluation can be replicated across the programme... These studies... covered claimants with relatively simple needs.

(Full Fact has covered this issue more in typically excellent fashion.)

And finally, because DWP's cost/benefit analysis included savings from reduced fraud and error, but the data here also wasn't yet robust:

The Department does not know whether Universal Credit is reducing fraud and error... The Department intends to develop a fully automated risk analysis and intelligence system on fraud and error. But it has not developed this sufficiently to understand and assess fraud and error.

Changes since then

In the House of Commons in October 2018, Ruth George MP asked the Minister for DWP to respond to the NAO’s report:

To ask the Secretary of State for Work and Pensions, what assessment she has made of the effect of the conclusions of the report, Rolling Out Universal Credit, published by the National Audit Office in June 2018...

Minister Alok Sharma supplied a “holding answer”, which is what the Government gives when it does not have time to write a proper answer:

The Government’s conclusion is clear that Universal Credit represents value for money.

It doesn’t look as though the “holding answer” has since been updated with an actual answer.

In October 2018, the Public Accounts Committee recommended (PDF) that the Government should supply better data for its claims:

In future the Department must make sure that all claims for Universal Credit are supported by empirical evidence rather than theoretical models.

The Government's response (PDF) merely said:

The Government agrees with the Committee’s recommendation... The Department intends to regularly update Parliament on progress as data becomes available.

I have searched Hansard, but cannot find any updates (though I did find the DWP minister in February 2019 citing the same employment research that the NAO said was inadequate). Meanwhile, I can find no new published data or research from DWP on cost savings.

So in summary: according to the most authoritative sources I can find, Government either isn't collecting meaningful data on the cost savings of Universal Credit. Or, it is collecting this data but is inexplicably failing to publish it.

Cost savings are the primary stated goal of this programme, and so that makes this officially our first Missing Number.

I need your help

Thanks for reading! I’m really keen to collect more ideas about important missing datasets, on any policy area in any part of the British state. If you’ve got any hints or tips, please drop me a line at anna@anna.ps, or tweet @darkgreener.

Photo credit: Library of Congress

Today I’m launching a new project, Missing Numbers.

Missing Numbers is a blog about the data that the government should collect and measure in the UK, but doesn’t.

I’m a data analyst and programmer who cares about social justice. Over the years, I’ve worked on many projects using data to create pressure for change. Here are a few:

• I was on the founding team at the Evidence-Based Medicine DataLab at the University of Oxford, which among other things monitored government bodies, universities and companies failing to publish clinical trial results.
• I’ve worked with Private Eye to reveal the full extent of foreign ownership of UK property, which helped push this issue up the political agenda.
• As tech lead on Who Owns England, among other projects I helped reveal thousands of empty homes in Kensington & Chelsea.

Most of these projects were lucky accidents, where we obtained data via accidental release, leaks, or by laboriously scraping and joining datasets. Almost none used official data releases.

Over time, I started to notice a pattern. Across lots of different policy areas, it was impossible for governments to make good decisions because of a basic lack of data. There was always critical data that the state either didn’t collect at all, or collected so badly that it made change impossible.

Eventually, I decided that the power to not collect data is one of the most important and little-understood sources of power that governments have. This is why I’m writing Missing Numbers: to encourage others to ask “is this lack of data a deliberate ploy to get away with something”?

By refusing to amass knowledge in the first place, decision-makers exert power over over the rest of us. It’s time that this power was revealed, so we can have better conversations about what we need to know to run this country successfully.

A typical example

The government records and publishes data on how often each NHS hospital receives formal complaints. This is very helpful, because it means patients and the people who care for them can spot hospitals whose performance is worrying.

But the government simply doesn't record data, even internally, on how often formal complaints are made about each Jobcentre. (That FOI response is from 2015, but I've confirmed it's still true in 2019.) So it is impossible for it to know if some Jobcentres are being seriously mismanaged.

I’ll be diving into this topic more, but...

I need your help

I’ve got several blog posts on different missing numbers lined up, but I’m really keen to collect more ideas, on any policy area in any part of the British state. If you’ve got any hints or tips, please drop me a line at anna@anna.ps, or tweet @darkgreener.

Image credit: Library of Congress