HadUK‐Grid—A new UK dataset of gridded climate observations

HadUK‐Grid is a new dataset of gridded climate observations for the UK produced by the Met Office Hadley Centre for Climate Science and Services. The dataset interpolates in situ observations to a regular grid using methods developed in a previous equivalent dataset that had been made available to users since 2002 through the UK Climate Projections project (UKCIP02, UKCP09). The new dataset differs from the existing one in a number of key respects: higher spatial resolution, longer time series for some variables, improved consistency with regard to the pre‐processing of station observations, the use of publicly‐accessible ancillary data sources, a revised calculation sequence for some variables and improved version control. This makes for a dataset that is more internally consistent, more traceable and more reproducible. The result is a dataset of key UK climate variables of up to 1 km resolution from 1862 for monthly rainfall, 1884 for monthly temperature, 1891 for daily rainfall, 1929 for monthly sunshine and a wider set of variables with start dates from the 1960s to support the need for national climate monitoring and climate research.

model calibration, plus many other applications by the hundreds of users that have downloaded the data. While HadUK-Grid is a new dataset, it is an iteration of an existing dataset, therefore for many users it is important to present HadUK-Grid in relation to what we shall refer to here as the 'legacy dataset'.
The need for a new dataset has been driven by a number of factors. Firstly, the spatial resolution of climate models has increased over the last decade, such that there is now a need for higher resolution observational data to match. Secondly, there has been a shift towards greater openness and traceability in the datasets used in climate change applications. Thirdly, as a result of the way in which the original dataset has been built up, some inconsistencies had crept in that needed resolving. The new dataset has been called HadUK-Grid in order to align it with other observational datasets produced by the Met Office Hadley Centre (e.g. HadCRUT, HadISD, HadISST https ://catal ogue.ceda. ac.uk/uuid/ce252 c81a7 bd471 78340 55e31 716b265).

| Daily and monthly data
The gridded datasets are created by interpolating in situ land surface, station observations, such as maximum air temperature or precipitation amount, from conventional climate observing stations. The majority of the station data have been extracted from the Met Office's Integrated Data Archive System (MIDAS) (Met Office, 2012). We made use of a preprocessed version of the database in which a series of business rules have already been applied to the raw observations to create daily or monthly values. These business rules handle not only the aggregation of the data (e.g. to create monthly values from daily observations) but also address complexity associated with the way in which data are stored in MIDAS due to variations in observing practice and instrumentation (e.g. selecting the primary instrumentation where a site may have multi-instrument set up). In practice, the vast majority of the data are unaffected by these rules, that is the impact on the final result is small. For traceability, we have kept a complete copy of the station values used in the gridding (after the business rules have been applied).
Recent data recovery activities have added several million additional daily and monthly observations of rainfall and temperature to the station archives from up to 190 stations for the period 1862-1958. For comparison, the total number of values used to generate the entire HadUK-Grid dataset is approximately 100 million. The full set of variables, their definitions and the periods of time used for each are summarized in Table 1. The number of stations available through time is summarized in Figure 1 showing a peak in the number of observing stations available in the digital archives in the late 20th Century, although there is still potential for further data recovery activities to increase the availability of data before 1960. Indicative numbers of stations available through the 1981-2010 period are provided in Table 2. The extension of some of these core gridded datasets back to the late 19th century provides an invaluable longer term context for the interpretation of the time series and variability of the UK's climate.

| Long-term averages
We also extracted station values of 30-year climatological means for reference periods 1961-1990 and 1981-2010. These long-term averages (LTAs) had been calculated previously as part of a separate activity. The calculation method is described in Perry and Hollis (2005a). It involves filling gaps in the time series of monthly station data with estimated values before computing the 30-year mean for each calendar month. The estimation process makes use of regression relationships between each target station and a number of well-correlated neighbours with overlapping records.
The majority of the input station data used for the gridding of most variables are available through the published MIDAS archives, particularly the MIDAS-open dataset (Met Office, 2019). However, a full set of station data values used to create HadUK-Grid, including the application of the pre-processing step described above, are currently only available on request from the authors under research only licence terms due to the inclusion of data from multiple originating agencies. A review of ongoing data recovery activities will be covered in a separate paper.

| Quality control
Much of the data in the MIDAS database has already been through the Met Office's formal quality control process (including range checks, internal consistency checks and nearneighbour checks). However, such processes are inevitably imperfect, and quality control of the input station data is still an essential component of the gridding process. It was not possible, within the timescales of the project, to develop a new QC system that would be both highly automated and robust for the more than 100,000 grids being produced. The legacy dataset was generated over many years with rigorous manual quality control applied to the data. In order to benefit from this embedded knowledge a simple quality check process was established using the existing data as a baseline reference dataset. A second final range-check procedure was applied to remove any other outliers.
Each value in our set of station data was compared with a colocated value taken from the corresponding legacy grid. Note: Start years reference the availability of daily and monthly series. Seasonal and annual series are available from the same start date as monthly.

F I G U R E 1 Number of stations used
for gridding-(a) monthly variables, (b) daily and monthly rainfall. Reproduced from Kendon et al. (2019) Scatter plots were used to examine the level of agreement and to determine thresholds for suspect data. If there was a large difference between an observation and the legacy grid then either the value had previously been excluded from the legacy gridding, or in the case of new data it is a poor fit to our expectation for that variable on that date. Figure 2 shows one of the scatter plot comparisons used to inspect the differences, in this case for monthly maximum temperature. Differences of <0.5°C, which account for most of the data, are not shown. The orange lines show a threshold of 1.35°C used to identify outlier values-points with larger differences (i.e. above the upper line or below the lower line) were excluded from the gridding process. Table S1 in the supplementary material shows the thresholds used. For most variables, the threshold is three times the RMS error of the interpolation process calculated by Perry and Hollis (2005b). In a few cases (i.e. variables for which an estimate of the interpolation error was not available), we chose a threshold based on visual inspection of the distribution of the data.
A secondary test was achieved by comparing each value against a set of absolute thresholds that represented clearly erroneous data. The thresholds used and the amount of data excluded is shown in the Table S2. Only a few additional values were excluded as a result of applying this test, reflecting that suspect data are very much the exception in the MIDAS data. In total, the combined QC process excluded <0.1% of the daily data and between 0.1% and 0.4% of the monthly data (depending on the variable). The vast majority of these were identified by the comparison with the legacy grids.
Obviously, there are some limitations to this approach: • The original QC process was imperfect and in some cases will have thrown away good data or retained bad data. We will end up replicating these decisions. • We are introducing new data into the gridding process.
A large difference may be because the legacy grid is poor rather than because the new observation is suspect. Rejecting such data could have an impact on the characterization of extremes. • Retrospective corrections to the MIDAS database may have modified some of the values that were used to create the legacy grids. The corrected value is to be preferred but a large difference from the legacy grid could mean it is rejected. • Legacy grids for years between 1961 and 2000 were masked to land areas before being archived. For a few stations very close to the coast, it was not possible to extract a value from the legacy grid (we used bilinear interpolation which requires four surrounding grid points) and so it was not possible to validate these data. • Where data recovery has enabled the new dataset to extend further back in time, the observations for the early part of the  series cannot be validated because legacy grids do not exist for the years in question. The same is true for recent years for variables that were not produced routinely, that is mean sea level pressure, relative humidity, days of snow lying, vapour pressure and wind speed. The legacy series for snow lying ends in 2013 and for the other variables in 2014.
Improved automated and traceable QC processes that are not dependent on the legacy dataset is recognized as a future development area. However, because the QC process rejects a small proportion of all data, the limitations outlined above are not considered to have a large enough effect to compromise the overall utility of this version of the gridded dataset.

| Gridding
The gridding methods are essentially unchanged from those used to generate the legacy grids. They are described in detail in Hollis (2005a, 2005b) and Perry et al. (2009) which provide further detail. The station data are interpolated to a regular grid of point locations using inverse-distance weighted (IDW) averaging. Interpolation will only work reliably if the quantity being analysed varies smoothly between data points. This is often not the case for climate data due to significant spatial trends in the data, that is correlations with topographic variables such as terrain elevation, proximity to the coast or proximity to urban areas. Therefore, some variables are transformed before the IDW interpolation. This is done by the following: • Taking the differences from, or percentages of, the longterm average (LTA). • Regression analysis to build a model of the relationship between the climate variable and one or more topographic variables. The set of ancillary topographic data used for this process are summarized in Table S3.
The spatial trends in the station data are removed using one or both of these methods (see Table S4 for further details), and the IDW interpolation is applied to the residuals of that process. The resulting field is then re-combined with the regression model and/or LTA field to obtain the final grid. The gridding process has been developed in Python 2.7 utilizing the Iris package (https ://scito ols.org.uk/iris/docs/ lates t/). The grids are produced at a native resolution of 1 km for all variables. This is higher than the 5 km resolution of the legacy dataset. The higher resolution provides users with greater flexibility to re-grid the data to different projections, geographies (e.g. river catchments) and grid resolutions. This facilitates intercomparison with climatological and hydrological models and some other existing UK climate datasets.
Note that the gridding process remains one of interpolating point observations to a set of regularly spaced point locations. Some additional micro-climate information is inferred from the regression model (e.g. to capture altitude/ terrain and land-use effects), but the choice of resolution does not reflect the effective resolution of the underlying observations. These are point estimates, not grid-box averages, and the grid points in the legacy dataset (spaced at 5 km intervals) are simply a subset of the grid points in the new dataset. Although the new grids are more detailed, the number of stations used to create them is comparable to the legacy dataset (with the exception of the additional recovered data), and there have been no changes to the gridding methods that would enable smaller-scale variations in the climate to be captured. Users should bear this in mind when using the data in their own applications. We hope to investigate the effective resolution of the UK observing network (and how this might vary over time and for different climate variables) in a future study.

| Processing sequence
The process for creating the grids is summarized in Figure 3. For the majority of monthly variables, there are two main steps: the LTA grids are created by interpolating the station LTA values, and then, the individual monthly grids are F I G U R E 3 The HadUK-Grid processing sequence created from monthly station data (using the LTA grids to convert the values to anomalies where required). For days of ground frost and days of snow lying, the sequence is reversed, that is the monthly grids are produced first, then the LTA grids are generated by averaging the monthly grids (this is necessary because the methods used to infill missing monthly station data do not work well for series that contain frequent zeros). A particular advantage of the new dataset is that the data extraction and gridding can be carried out in a single batch process, whereas the legacy dataset was processed in a series of batches covering 1961-2000, 1914-1960, 1910-1913 and subsequent monthly updates from 2001 to 2018. Generating the entire dataset in a single process eliminates the possibility of non-climatic inhomogeneities being inadvertently introduced by changes in the processing chain.

| Evaluation
Although the HadUK-Grid dataset and the legacy dataset have been produced using nearly identical gridding methods and very similar station data, there are differences worth documenting. Figure 4 illustrates differences in monthly rainfall for a grid point near Blencathra in Cumbria. HadUK-Grid is markedly wetter than the legacy dataset for the period 1960-2000. The bias is a dry bias in the legacy dataset because a different reference LTA had been used to create this portion of the legacy dataset. HadUK-Grid has removed this inhomogeneity. The graph also highlights the benefit of the extension of the data series, and how this puts the exceptional month of December 2015 into longer term context. Large differences such as that shown above for the period 1960-2000 are an exception. Table 3 provides a summary of differences between co-located grid points in HadUK-Grid v1 and the legacy dataset. The differences have been calculated across all grid points in all grids for each variable. They are well within the estimated uncertainty range of the interpolation (see the RMS errors published in Perry and Hollis, 2005b) and overall relatively small. Time series comparison of UK and Scotland mean temperature and rainfall is also provided in the Figure S1. Discontinuities in 1913Discontinuities in , 1960Discontinuities in and 2000 relate to structural inhomogeneities within the legacy dataset. Legg (2014) used the legacy dataset to examine uncertainties associated with spatial sampling. HadUK-Grid represents a single realization of the climate of the UK, and some of the structural differences HadUK-Grid has identified compared to the legacy grids motivates a need for further research into these structural uncertainties resulting from processing and methodological assumptions.
Based on the preliminary intercomparison described above, we believe the new dataset is more homogeneous than the legacy dataset (achieved largely through a more consistent approach to constructing the dataset). However, it is inevitable that some inhomogeneities will remain because: F I G U R E 4 Monthly rainfall totals (mm) for a grid point near Blencathra, Cumbria (orange-legacy dataset, blue-new dataset) • The station data have not been homogenized.
• The network of stations changes over time as sites open/ close. • The size of the network has changed over time (sparser networks are less effective at resolving localized extremes). • The QC process was not the same for the entire dataset.
• The gridding process may not compensate for systematic changes to the observing network, for example more stations located at airports.
To understand the impact of changes to the network, Simpson and McCarthy (2019) compared the rainfall grids from the legacy dataset with a version produced using a much smaller network. They concluded that the methods employed avoided systematic biases when the network was reduced. In Kendon et al. (2019), the long-term temperature and rainfall series are also shown to be consistent with independent datasets such as the Central England Temperature series (HadCET), the England and Wales Precipitation series (part of HadUKP) and coastal sea-surface temperature (SST) data. We plan to develop a new QC process that can be applied uniformly to the entire dataset. This should reduce the risk of introducing inhomogeneities due to the processing methodology. A more detailed assessment of the homogeneity of the HadUK-Grid dataset will be the focus of a future study.

AND ACCESSIBILIT Y
Version 1.0.0.0 of the HadUK-Grid dataset is available for users to download from the CEDA Archive (Met Office, 2018). The grids are packaged as CF-compliant (cfcon venti ons.org) netCDF files (Unidata, 2019). The primary dataset is a 1 × 1 km grid on the British National Grid projection (EPSG:27700).
To facilitate comparison of the observational dataset with products from the UK Climate Projections project (UKCP18, Lowe et al., 2018) we have also: • Provided the dataset at 5, 12, 25 and 60 km resolution. All the gridded datasets use the same grid projection. The re-gridding is conducted through averaging of all 1 km grid points that fall within each of the coarser resolution grid cells. • Provided area averages based on averaging the 1 km grid across a set of geographical regions to provide spatial statistics for country, administrative regions and river basins. The associated shapefiles are available from https ://github. com/ukcp-data/ukcp-spati al-files . • Provided 1 km grids of the 1981-2000 climate average (the baseline used in the UKCP18 projections). These 20-year averages were obtained by averaging or summing the 20 monthly or annual gridded datasets for each variable (in contrast to the 30-year LTAs which were produced by interpolating 30-year station averages).
All of the data are provided under an open government licence (http://www.natio nalar chives.gov.uk/doc/open-gover nment-licen ce/versi on/3/) and may be copied, adapted and exploited both commercially and non-commercially. Individual grid files provide a 3-dimensional matrix of values covering the whole of the UK on the Ordnance Survey's National Grid (https ://www.ordna ncesu rvey.co.uk/ resou rces/maps-and-geogr aphic-resou rces/the-natio nal-grid. html) for all dates for one calendar year, or one calendar month. The files are named with the convention outlined below, adopting the controlled vocabularies for metadata of the UKCP18 project, available from https ://github.com/ ukcp-data/UKCP18_CVs File naming convention: <var_id>_hadukgrid_uk_<resolution>_<frequen-cy>_<startdate>-<enddate>.nc • <var_id> is the variable identifier summarized in Table 1 • <resolution> is one of 1, 12, 25, 60 km, country, region, or river • <frequency> is one of day, mon, seas, ann (for daily, monthly, seasonal or annual data), with a suffix of −30 or −20 year for long-term averages • <startdate>-<enddate> are in the form YYYYMMDD T A B L E 3 Mean and root mean square difference between the Version 1 and legacy datasets The version numbering for the dataset follows a pattern x.y.z.θ where: • X reflects a major upgrade to the whole dataset, for example if a new gridding process was implemented. • Y reflects a minor upgrade to components of the dataset, for example if a new QC process was adopted. • Z reflects addition of the latest data, for example v1.0.0.0 includes data to 2017, v1.0.1.0 will include data for 2018 and a wider range of derived indices.
θ is reserved for identifying provisional data or versions of the dataset under development.

| DATA USE AND REUSE
HadUK-Grid v1.0.0.0 was released in December 2018. Version 1.0.1.0 will be released in mid-2019 to coincide with the release of 'State of the UK Climate 2018', the latest edition of the Met Office's annual publication describing trends in the climate of the UK (Kendon et al., 2019). This version will include data for 2018 as well as a number of new variables derived from the daily temperature and rainfall grids, for example days of air frost, days of rainfall above a threshold, degree days, etc. A further version of HadUK-Grid will be released each year to coincide with the latest 'State of the UK Climate' report (the report will be based on that version of the dataset). As such, the new dataset will change annually. As a minimum, each new version will contain additional data covering the previous calendar year. We may also take the opportunity to include further data introduced from ongoing data recovery activities, add new derived variables, or to correct any minor data quality issues that may have come to light. In some years, we may issue a more substantial update when there have been significant changes to the methodology (e.g. the QC process or the gridding techniques).

| SUMMARY
HadUK-Grid is a new collection of gridded climate variables derived from the network of UK land surface observations. The data have been interpolated from meteorological station data onto a uniform grid to provide complete and consistent coverage across the UK. The datasets cover the UK up to 1 × 1 km resolution. Users should note that this choice of resolution does not necessarily reflect the effective resolution of the underlying station network but is to facilitate a range of geographies and other resolutions to allow for comparison to data from climate projections and across country, administrative regions and river basins. The dataset spans the period from 1862 to present, but the start time is dependent on climate variable and temporal resolution. The grids are produced for daily, monthly, seasonal and annual timescales, as well as long-term averages for a set of climatological reference periods. Variables include air temperature (maximum, minimum and mean), precipitation, sunshine, mean sea level pressure, wind speed, relative humidity, vapour pressure, days of snow lying and days of ground frost.