DEBIAS data story · Published in Royal Society Open Science
Who is missing from the map?
The dots line up. The local coverage still varies.
Mobile-app counts can closely track census population while producing very different observed records per resident across places. A high correlation is a useful first check—not proof of representativeness.
Aggregate active-account estimates—not identified people. This comparison can show differences in aggregate coverage. It cannot identify missing people; the estimates are not verified unique-resident counts or individual inclusion probabilities.
Meta active-account example · ≈102,000 residents each · illustrative extreme from 331 areas
Watford
102,246 residents
2.37 estimates per 100
Meta estimate: 2,419
6.7×
different local rates
North East Derbyshire
102,001 residents
15.89 estimates per 100
Meta estimate: 16,204
Meta average nighttime active-account estimates · 2021 Census · England and Wales
Larger places generally produce larger counts. Pearson r = .91 across 331 local authorities.
2 · Counts
The counts still line up.
Across all 331 areas, census population and Meta’s active-account estimate have a Pearson correlation of r = .91. Larger places generally generate more records. This is where many checks stop.
Correlation shows that the counts move together. It does not show that every place is observed at the same rate.
The standard check looks strong.
3 · Rates
Put every place on the same scale.
Per 100 census residents, all 331 local rates spread around a fitted Meta rate of 8.09. The middle 90% spans 4.61–12.31 estimates per 100 residents—a 2.7× range.
The pair is an extreme. The full distribution shows that local variation is ordinary, not exceptional.
Dividing by population reveals the spread.
4 · Map
Put the differences back in place.
The map shows each area’s local rate relative to the fitted Meta rate. Teal means fewer estimated accounts per resident than fitted; coral means more.
More accounts do not mean better representation. The pattern is local—not a simple regional divide.
Local departure from the fitted Meta rate.
Act II · Different sources, different patterns
The same place looks different through different data.
The Meta map is one lens. Across four datasets, the same area can land on opposite sides of its source’s fitted population rate—even though every source has a strong count correlation with census population.
Each source has its own fitted baseline. The comparison diagnoses relative observed-identifier coverage; it is not a representativeness score or a percentage of residents included. The four measures and observation dates are not interchangeable.
What differs between the four sources?
Twitter/X uses inferred monthly home locations of active accounts and Meta uses average nighttime active-account estimates for March 2021. Multi-app1 uses inferred home locations of qualifying devices from the first week of April; Multi-app2 uses inferred home locations from preprocessed multi-application GPS data for November. The fitted index normalises scale; it does not make the sources equivalent.
The pair changes position when the source changes. One times marks each source’s own fitted proportional rate.
5 · Four sources
The same pair switches sides.
Watford is below Meta’s fitted rate but above the other three source baselines. North East Derbyshire is above Meta and Multi-app2, but below Twitter/X and Multi-app1.
The places have not changed. The source has.
Relative to each source’s fitted proportional rate
Source
Watford
North East Derbyshire
Twitter/X
1.09× 9% above
0.49× 51% below
Meta
0.29× 71% below
1.96× 96% above
Multi-app1
1.44× 44% above
0.83× 17% below
Multi-app2
1.27× 27% above
1.21× 21% above
Values compare local observed identifiers per resident with a separate fitted rate for each source. They are not percentages of residents represented.
6 · 331 areas
91% change position when the source changes.
Using the paper’s fitted proportional baseline, 300 of 331 local authorities sit above their source-specific fitted rate in at least one dataset and below it in another.
The pair is memorable. The England-and-Wales pattern shows it is not an edge case.
300/ 331
Calculated from the released LAD-level data. The 91% describes cross-source position relative to fitted rates—not the percentage of people represented.
7 · Why one fix fails
The area-level signals change with the source.
The local characteristics that help explain coverage bias change from source to source—across demography, socioeconomic conditions, resource accessibility, mobility and geography. These describe places, not users.
Website re-render of the accepted-revision main-model outputs—not a reproduction of the paper's radial figure.
Selected source: Twitter/X
Choose a source. Labels mark area characteristics above 0.5 on that source's relative-importance scale. The cutoff is not a significance test or a measure of how biased a source is.
A
Demographic context
Ten demographic characteristics of local areas.B
Socioeconomic context
Eleven socioeconomic characteristics of local areas.C
Resource accessibility
Four Census household proxies—not direct measures of internet or device access.D
Mobility & geography
Two model domains combined in one display group.
1 of 4 · Demographic context
Area characteristics—not user attributes. Radius is relative mean absolute SHAP importance within each source: 1 marks that source's highest-scoring model feature and 0 its lowest, not “no effect.” Use the switcher to compare which features rank relatively high within each source. Do not compare polygon area or raw mean absolute SHAP magnitudes across sources; the models are fitted separately. Importance does not show direction, causality, population shares or group-specific inclusion rates.
Higher managerial, administrative and professional occupations
0.000034953457907249684
0.18290435846
7
Socioeconomic
Lower managerial, administrative and professional occupations
0.000059920262322560554
0.313550583978
4
Resource accessibility
Households without central heating
0.000020664578218914
0.108133548103
13
Resource accessibility
Households without a car or van
0.00012899871064449975
0.675024098481
2
Resource accessibility
Households owned
0.000019936567481754933
0.104324015519
14
Resource accessibility
Households not deprived in any dimension
0.000008992498082396038
0.047055919248
19
Mobility & geography
Population density
0.00003367524472512583
0.176215728034
8
Mobility & geography
Rural population
0
0
20
Mobility & geography
Population born outside the UK
0
0
20
Mobility & geography
Employed residents working mainly at or from home
0.000012688271766345006
0.066395153623
16
Mobility & geography
Population resident in the UK for less than two years
0
0
20
What stands out changes. So does the shape of the relationship.
The relationships bend, reverse and jump.
The radial fingerprints rank relative feature importance. These three accepted empirical examples show how individual area-level relationships behave—and why a straight-line correction can miss the local pattern.
The association changes direction across the density range.X standardised · Y SHAP contributionAccepted empirical exampleS-SHAPE
Demographic context
Twitter/XArea share aged 20–29
Low, middle and high area values follow different phases.X standardised · Y SHAP contributionAccepted empirical exampleTHRESHOLD
Socioeconomic context
Multi-app1Area share with Level 4 qualifications
The association changes abruptly around a step.X standardised · Y SHAP contribution
How to read the two model views: the radial fingerprints show relative mean absolute SHAP importance within a source; the dependence plots show the shape and direction of individual associations with predicted coverage bias. Each point in the dependence plots is one local authority, not a person or group. These are feature-level examples, not universal source signatures, and neither view identifies inclusion rates or causal effects.
Representativeness must be diagnosed source by source.
Different datasets place the same areas on different sides of their fitted rates, and their biases relate to local characteristics in different, often nonlinear ways. A correction validated for one source should not be assumed to transfer to another.
Measure before you infer. Validate before you adjust.
A framework for diagnosing aggregate coverage bias.
Carmen Cabrera and Francisco Rowe of the University of Liverpool’s Geographic Data Science Lab compared four mobile-app datasets with the 2021 Census across 331 local authority areas in England and Wales. Their published Royal Society Open Science study, Making hidden biases visible in population location data from mobile phones, combines proportional diagnostics and explainable machine learning to examine where aggregate coverage differs and which area characteristics are associated with that variation.
The analysis diagnoses area-level patterns; it cannot identify missing individuals.
Analysis uses released local-authority counts, Meta’s average nighttime active-account estimates for March 2021, 2021 Census resident populations and official 2021 local-authority boundaries. The public map displays local rate minus fitted rate; this has the opposite sign to the paper’s residual-bias convention.
