DEBIAS data story · Published in Royal Society Open Science
Who is missing from the map?
Rapidly unfolding social challenges such as natural disasters, conflicts and epidemics require timely and granular data to understand population changes and appropriately inform planning and decision-making.
However, existing data platforms, including censuses and surveys, are often slow and expensive. Passively generated data via digital technology, including mobile phone apps, have demonstrated great utility in supporting responses to many urgent social challenges, such as the COVID-19 pandemic, the war in Ukraine and flood-induced disaster management in Pakistan. They can provide temporally and geographically granular population data in near-real time.
Yet, digital-trace data can be biased and provide a distorted representation of local populations. Generally, the correlation between the geographical distributions of resident and active-user populations is assessed, but this does not offer an assessment of bias or representativeness.
High correlation may exist. But local coverage may still vary and local populations remain poorly represented.
Digital-trace counts provide a useful signal but they do not represent the local populations. Comparing active-user counts from Meta illustrates wide variation in coverage for places with similar population counts.
Places can have similar resident population counts but have widely different coverage levels in digital-trace data. Using data from Meta, we illustrate how population coverage varies across UK local authorities. Comparing Watford and North East Derbyshire illustrate how two places with similar resident population counts can have a 6.7 time difference in population coverage.
2 / 8 · Counts
Population counts still line up.
Across local authorities, census populations and Meta’s active-user population estimates have a Pearson correlation of r = .91. More populated places generally have more accounts. Yet, this high correlation does not quantify limited coverage or representation.
Local variations in coverage rates reveal which populations are under- or over-represented in digital-trace data.
3 / 8 · Population coverage rates
Putting places on the same scale
Expressing Meta coverage as user accounts per 100 census residents, local variations become clearer. They vary widely around Meta’s fitted rate of 8.09 per 100 residents across local authorities.
The middle 90% of areas spans 4.61 to 12.31 accounts per 100 residents, equivalent to a 2.7x range. Such level of variation is typical in digital-trace data from various sources.
Dividing by population reveals the spread.
4 / 8 · Map
Mapping population coverage rates
Mapping coverage rates relative to the fitted Meta rate reveals under- (darker teal colours) and over-represented (darker coral colours) places.
Coverage rates vary systematically but differently with demographic, socioeconomic, accessibility, mobility and geographical features of local areas. Analysing these relationships helps more precisely identify which population groups may be under- or overrepresented in digital-trace data.
Difference between local and fitted rate · estimates per 100
Different data sources, different patterns
The same place is portrayed differently through different data.
Meta is one data source and offers one representation of local populations. But there are more sources. Each may display a high correlation score but offers its own distorted representation of local populations. We investigate four sources.
Each data source has its own fitted coverage rate. Still, the same area can land on opposite sides of a fitted coverage rate depending on the digital-trace data source used to represent their local populations.
What differs between the four sources?
Twitter/X uses inferred monthly home locations of active accounts and Meta uses active-user population estimates for March 2021. Multi-app1 uses inferred home locations of qualifying devices from the first week of April; Multi-app2 uses inferred home locations from preprocessed multi-application GPS data for November. The fitted index normalises scale; it does not make the sources equivalent.
Population coverage for the same area can vary widely across data sources
Focusing on Watford and North East Derbyshire shows large differences in coverage by data source. For Watford, the fitted coverage rate is below for Meta, but above for Twitter/X, Multi-app1 and Multi-app2. For North East Derbyshire, it is above for Meta and Multi-app2, but below for Twitter/X and Multi-app1.
The places have not changed. The source has.
Relative to each source’s fitted population coverage rate
Source
Watford
North East Derbyshire
Twitter/X
1.09× 9% above
0.49× 51% below
Meta
0.29× 71% below
1.96× 96% above
Multi-app1
1.44× 44% above
0.83× 17% below
Multi-app2
1.27× 27% above
1.21× 21% above
91% of local authorities change position
300 of 331 local authority areas display higher or lower coverage rates than the fitted or expected rate in at least one data source. This indicates that patterns of under- or over-representation across areas are the norm, rather than an exception.
300/ 331
300 of 331 authorities change sides across the four source-specific fitted rates from being over- to under-represented or vice versa.
6 / 8 · Understand the influence of local attributes
Understanding which populations are under- or over-represented.
Coverage rates vary systematically with local population attributes. By analysing this variation, we can identify which population groups are likely under- or over-represented in digital-trace data. We investigate differences across our four data sources: Twitter/X, Meta, Multi-App1 and Multi-App2.
We use a machine-learning based model (XGBoost) to analyse which place-based attributes are the most important predictors of low or high population coverage bias. These attributes identify the population groups which are under- or over-represented in a particular digital-trace dataset.
Selected source: Twitter/X
Choose a digital-trace data source. The radial plots below identify the most important place-based attributes associated with variability in population coverage rates. For example, the share of the population aged 20–29 and share of households without a car are the most important predictors of low population coverage rates.
We find that the associations between local attributes and population coverage rates can flatten, reverse direction or change shape across the range of observed records. Machine learning is employed to capture these non-linear relationships.
A
Demographic context
Ten demographic characteristics of local areas.B
Socioeconomic context
Eleven socioeconomic characteristics of local areas.C
Resource accessibility
Four Census household proxies—not direct measures of internet or device access.D
Higher managerial, administrative and professional occupations
0.000034953457907249684
0.18290435846
7
Socioeconomic
Lower managerial, administrative and professional occupations
0.000059920262322560554
0.313550583978
4
Resource accessibility
Households without central heating
0.000020664578218914
0.108133548103
13
Resource accessibility
Households without a car or van
0.00012899871064449975
0.675024098481
2
Resource accessibility
Households owned
0.000019936567481754933
0.104324015519
14
Resource accessibility
Households not deprived in any dimension
0.000008992498082396038
0.047055919248
19
Mobility & geography
Population density
0.00003367524472512583
0.176215728034
8
Mobility & geography
Rural population
0
0
20
Mobility & geography
Population born outside the UK
0
0
20
Mobility & geography
Employed residents working mainly at or from home
0.000012688271766345006
0.066395153623
16
Mobility & geography
Population resident in the UK for less than two years
0
0
20
Local attributes do not relate to coverage in a simple, linear way
The illustrative examples below show the shape of the relationship between key place-based attributes and population coverage rates for different data sources. Radial plots indicate which place-based attributes are important predictors of low coverage rates, but they do not tell us how these attributes influence coverage. For example, the radial plots show that the percentage of people aged 20–29 is an important predictor of low coverage rates in Twitter/X. The middle plot below shows how this relationship unfolds: as the percentage of people aged 20–29 increases, population coverage in Twitter/X data also tends to increase; that is, poor coverage becomes smaller. Here, the y-axis represents the inverse of population coverage rates.
We find that the associations between local attributes and population coverage rates can flatten, reverse direction or change shape across the range of observed records. Machine learning is employed to capture these non-linear relationships.
Digital-trace population estimates are valuable, but their representativeness must be assessed and adjusted.
Across four digital-trace data sources, we find substantial local differences in population coverage. These differences vary across places and data sources, and they are associated with demographic, socioeconomic, resource accessibility, mobility and geographical attributes of local populations. These variations reflect biases resulting from the under- and over-representation of certain population groups in the digital platforms used to collect the data. Assessing the representation of digital-trace data is therefore required to define appropriate ways of adjusting them and using them for reliable population inference.
