Data acquisition#
Navigation: ← Previous: Introduction | Next: Direct Estimates →
All inputs are open Nomis / ONS tables. This notebook documents the API calls and loads the bundled panel used by later chapters.
import sys
from pathlib import Path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.express as px
from IPython.display import HTML, display
import warnings
warnings.filterwarnings('ignore')
plt.style.use('seaborn-v0_8-whitegrid')
PROJ_DIR = Path('.').resolve()
if not (PROJ_DIR / 'sae_utils.py').exists():
PROJ_DIR = Path('projects/small-area-estimation').resolve()
if str(PROJ_DIR) not in sys.path:
sys.path.insert(0, str(PROJ_DIR))
from sae_utils import (
load_panel, modelling_frame, design_matrix, eblup_fh, fh_reml_sigma_u,
jackknife_mse, gibbs_fh, summarise_theta, posterior_predictive_y,
rhat_split, coverage_rate, project_data_dir,
)
def display_plotly(fig):
"""Embed Plotly with CDN JS — fig.show() is blank in Jupyter Book HTML."""
display(HTML(fig.to_html(include_plotlyjs='cdn', full_html=False)))
FIG_DIR = PROJ_DIR / 'figures'
FIG_DIR.mkdir(exist_ok=True)
DATA_DIR = project_data_dir()
PANEL = load_panel()
MODEL = modelling_frame(PANEL)
print(f'Panel: {len(PANEL)} local authorities; modelling subset: {len(MODEL)}')
print(f'Period (APS / ONS model-based): {PANEL["direct_period"].dropna().iloc[0]}')
Panel: 348 local authorities; modelling subset: 136
Period (APS / ONS model-based): Apr 2025-Mar 2026
Sources and licence#
Source |
Nomis id |
Role |
|---|---|---|
Annual Population Survey, unemployment rate aged 16–64 |
|
Direct estimator \(y_i\), 95% CI → \(\psi_i\) |
APS economic inactivity rate aged 16–64 |
|
Optional auxiliary covariate |
Model-based estimates of unemployment |
|
ONS production benchmark |
Claimant count as % of residents aged 16–64 |
|
Administrative covariate \(x_i\) |
LAD December 2023 BGC boundaries |
ONS Open Geography |
Choropleths |
Geography: TYPE424 — local authority districts / unitaries as of April 2023. The APS and model-based series used here are the 12 months Apr 2025–Mar 2026; the claimant snapshot is March 2026 (end of that window).
Attribution. Contains public sector information licensed under the Open Government Licence v3.0. Source: Office for National Statistics and Nomis.
Nomis REST shape#
Nomis is a SDMX-style API. A typical CSV pull is
https://www.nomisweb.co.uk/api/v01/dataset/{id}.data.csv?geography=TYPE424&...&time=latest
APS percentages expose four MEASURES: the rate (20599), numerator, denominator, and CI half-width (21003). Model-based estimates expose value (20100) and confidence (20701). Notebooks do not hit Nomis at build time; they read data/la_sae_panel.csv. Refresh with:
python projects/small-area-estimation/_build_data.py
print('Bundled files:')
for p in sorted(DATA_DIR.glob('*')):
if p.is_file():
print(f' {p.name:28s} {p.stat().st_size/1024:7.1f} KB')
print('\nPanel columns:', list(PANEL.columns))
print(PANEL.head(8).to_string(index=False))
Bundled files:
ATTRIBUTION.txt 0.3 KB
README.md 1.5 KB
la_boundaries.geojson 1253.7 KB
la_sae_panel.csv 46.3 KB
Panel columns: ['GEOGRAPHY_CODE', 'GEOGRAPHY_NAME', 'country', 'direct_period', 'direct_rate', 'direct_rate_ci', 'direct_rate_numerator', 'direct_rate_denominator', 'psi', 'cv', 'claimant_rate', 'claimant_period', 'inactivity_rate', 'ons_mb_rate', 'ons_mb_rate_ci', 'ons_mb_period', 'in_model']
GEOGRAPHY_CODE GEOGRAPHY_NAME country direct_period direct_rate direct_rate_ci direct_rate_numerator direct_rate_denominator psi cv claimant_rate claimant_period inactivity_rate ons_mb_rate ons_mb_rate_ci ons_mb_period in_model
E07000223 Adur England Apr 2025-Mar 2026 NaN NaN NaN 30200.0 NaN NaN 3.2 March 2026 18.5 3.1 1.4 Apr 2025-Mar 2026 False
E07000032 Amber Valley England Apr 2025-Mar 2026 NaN NaN NaN 62700.0 NaN NaN 2.6 March 2026 20.7 2.9 1.2 Apr 2025-Mar 2026 False
E07000224 Arun England Apr 2025-Mar 2026 NaN NaN NaN 72000.0 NaN NaN 3.8 March 2026 20.5 3.1 1.2 Apr 2025-Mar 2026 False
E07000170 Ashfield England Apr 2025-Mar 2026 2.6 NaN 1600.0 61600.0 NaN NaN 3.8 March 2026 24.9 4.3 1.7 Apr 2025-Mar 2026 False
E07000105 Ashford England Apr 2025-Mar 2026 2.5 NaN 1500.0 60600.0 NaN NaN 3.0 March 2026 23.4 3.8 1.5 Apr 2025-Mar 2026 False
E07000200 Babergh England Apr 2025-Mar 2026 4.9 NaN 2000.0 40500.0 NaN NaN 2.5 March 2026 20.7 3.8 1.5 Apr 2025-Mar 2026 False
E09000002 Barking and Dagenham England Apr 2025-Mar 2026 6.1 3.0 6600.0 108700.0 2.342774 0.250920 7.5 March 2026 26.3 8.2 2.4 Apr 2025-Mar 2026 True
E09000003 Barnet England Apr 2025-Mar 2026 5.5 2.7 11200.0 202700.0 1.897647 0.250464 5.0 March 2026 22.8 5.9 1.8 Apr 2025-Mar 2026 True
Coverage and suppression#
Nomis flags unpublished APS cells (OBS_STATUS other than A). After the pivot, suppression appears as missing direct_rate or missing CI. The Fay–Herriot fit below uses only LAs with a published rate and CI, plus a claimant rate and an ONS model-based figure — the in_model flag.
summary = (
PANEL.groupby('country')
.agg(
n=('GEOGRAPHY_CODE', 'size'),
with_direct=('direct_rate', 'count'),
with_ci=('psi', 'count'),
in_model=('in_model', 'sum'),
mean_direct=('direct_rate', 'mean'),
mean_ons=('ons_mb_rate', 'mean'),
mean_claimant=('claimant_rate', 'mean'),
)
.round(2)
)
summary
| n | with_direct | with_ci | in_model | mean_direct | mean_ons | mean_claimant | |
|---|---|---|---|---|---|---|---|
| country | |||||||
| England | 294 | 233 | 113 | 113 | 4.94 | 4.21 | 3.55 |
| Scotland | 32 | 29 | 12 | 12 | 3.89 | 3.79 | 2.85 |
| Wales | 22 | 22 | 11 | 11 | 4.37 | 4.20 | 3.31 |
fig = px.scatter(
PANEL.dropna(subset=['direct_rate', 'ons_mb_rate']),
x='direct_rate', y='ons_mb_rate', color='country',
hover_name='GEOGRAPHY_NAME',
trendline='ols',
labels={'direct_rate': 'APS direct rate (%)', 'ons_mb_rate': 'ONS model-based rate (%)'},
title='Direct APS vs ONS model-based unemployment rate',
)
lims = [0, max(PANEL['direct_rate'].max(), PANEL['ons_mb_rate'].max())]
fig.add_trace(go.Scatter(x=lims, y=lims, mode='lines', name='y = x',
line=dict(dash='dash', color='grey')))
display_plotly(fig)
Constructing \(\psi_i\)#
If Nomis publishes a 95% CI half-width \(c_i\) on the percentage scale,
This assumes a symmetric Wald interval. Design-based intervals for proportions can be asymmetric at the boundary; we drop areas without a published \(c_i\) rather than inventing a generalised variance function, so the likelihood in later chapters is faithful to the published errors.
m = MODEL.copy()
print(f'Modelling subset n={len(m)}')
print(m[['GEOGRAPHY_NAME','country','direct_rate','direct_rate_ci','psi','cv',
'claimant_rate','ons_mb_rate']].head(10).to_string(index=False))
print('\nCV quintiles:', m['cv'].quantile([0.2, 0.4, 0.6, 0.8]).round(3).to_dict())
Modelling subset n=136
GEOGRAPHY_NAME country direct_rate direct_rate_ci psi cv claimant_rate ons_mb_rate
Barking and Dagenham England 6.1 3.0 2.342774 0.250920 7.5 8.2
Barnet England 5.5 2.7 1.897647 0.250464 5.0 5.9
Barnsley England 2.6 1.5 0.585693 0.294349 3.3 3.8
Bath and North East Somerset England 4.6 2.0 1.041233 0.221828 1.9 3.9
Bexley England 7.3 3.7 3.563619 0.258597 3.4 5.5
Birmingham England 8.5 2.1 1.147959 0.126050 9.9 8.1
Blackburn with Darwen England 4.3 2.0 1.041233 0.237304 5.3 5.3
Blackpool England 4.2 2.0 1.041233 0.242954 6.2 4.7
Bolton England 3.7 2.0 1.041233 0.275786 5.2 4.7
Bournemouth, Christchurch and Poole England 3.9 1.4 0.510204 0.183150 3.5 3.6
CV quintiles: {0.2: 0.196, 0.4: 0.224, 0.6: 0.248, 0.8: 0.277}
Key takeaways#
Four open Nomis tables plus ONS boundaries are sufficient to fit and validate an area-level SAE model — no Accredited Researcher access.
Suppression is a feature of the problem: many LAs have \(y_i\) but no CI, and some have neither.
\(\psi_i\) comes from published CIs, not from a binomial approximation to weighted counts (those counts are population estimates, not sample sizes).
Navigation: ← Previous: Introduction | Next: Direct Estimates →