Collecting and utilising crowdsourced data for numerical weather prediction: Propositions from the meeting held in Copenhagen, 4–5 December 2018

In December 2018, the Danish Meteorological Institute organised an international meeting on the subject of crowdsourced data in numerical weather prediction (NWP) and weather forecasting. The meeting, spanning 2 days, gathered experts on crowdsourced data from both meteorological institutes and universities from Europe and the United States. Scientific presentations highlighted a vast array of possibilities and progress being made globally. Subjects include data from vehicles, smartphones, and private weather stations. Two groups were created to discuss open questions regarding the collection and use of crowdsourced data from different observing platforms. Common challenges were identified and potential solutions were discussed. While most of the work presented was preliminary, the results shared suggested that crowdsourced observations have the potential to enhance NWP. A common platform for sharing expertise, data, and results would help crowdsourced data realise this potential.


| INTRODUCTION
Within the atmospheric sciences, "crowdsourced" data is a relatively new term. While the term crowdsourcing was initially defined by Howe (2006) as outsourcing an act to the general public, this definition is no longer restricted to traditional tasks being outsourced. Today, crowdsourcing is more than outsourcing data collection to the general public. Instead, crowdsourcing embraces new data sources, data storage, quality control and utilisation, which requires standard methods and a common terminology.
Direct and indirect observations from non-conventional sources are being investigated for use in the atmospheric sciences. Examples of data sources include Personal Weather Stations (PWSs) (Bell et al., 2013(Bell et al., , 2015Clark et al., 2018), smartphones (Kim et al., 2015;McNicholas and Mass, 2018;Price et al., 2018;Hintz et al., 2019), vehicles (Anderson et al., 2012;Mahoney and O'Sullivan, 2013) and communication networks (Zinevich et al., 2009). Muller et al. (2015) provided a comprehensive review of "crowdsourcing" efforts in the atmospheric sciences. Since this review was published, new advancements have been made with crowdsourced datasets. Some of the most recent advancements include the collection and quality-control of atmospheric pressure observations from smartphones (Kim et al., 2015(Kim et al., , 2016Madaus and Mass, 2017;McNicholas and Mass, 2018;Price et al., 2018;Hintz et al., 2019). Kim et al. (2016) was the first to apply machine-learning methods to bias correct smartphone pressure observations (SPOs). McNicholas and Mass (2018) demonstrated an efficient machine-learning approach to SPO bias correction that benefited from non-meteorological smartphone sensor data. Clark et al. (2018) examined the use of PWSs and made considerable progress in the quality control and use of such data. Examples of successful assimilation of such observations into operational Numerical Weather Prediction (NWP) models are currently few and far between. The integration of observations from PWSs into the NOAA Meteorological Assimilation Data Ingest System (NCEP, 2019) dataset is an early example. Also, in the U.S., the utility of PWSs has been of increasing interest for forecasts of severe convection (Madaus et al., 2014;Carlaw et al., 2015;Sobash and Stensrud, 2015;Gasperoni et al., 2018).
A meeting on the use of crowdsourced data in NWP and weather forecasting was held in Copenhagen 4-5th December 2018 at the Danish Meteorological Institute (DMI), with two main purposes. First, to gather experts within the topics of crowdsourcing and create a network of people working on the subject, and second to discuss common issues encountered with crowdsourced data and how these can be addressed. Researchers from both universities and meteorological institutes attended the meeting, whose experience spanned a variety of subjects, including SPOs, PWSs, vehicular data, and citizen weather reports. The first day was allocated for presentations from the participants, followed by plenary discussion. The second day was allocated for discussions, starting with a sketch of ongoing activities at Institutions and Universities. Two working groups were created who reviewed current research topics for various data sources and data formats. The purpose of this article is to document the propositions and recommendations from the meeting and to inform peers of ongoing activities.

| SCIENTIFIC PRESENTATIONS
C. McNicholas (University of Washington) discussed how measurements of atmospheric pressure could be efficiently retrieved from smartphones and subsequently bias-corrected. Results from a testbed Android app, uWx, revealed that inaccuracies in smartphone location and sensor internal filtering contributed to poor data quality. Correcting these issues facilitated the retrieval of pressure change without the need for post-processing/quality control. Using a machine learning approach, smartphone pressures were bias-corrected to account for large uncertainties in smartphone elevation (McNicholas and Mass, 2018). For each smartphone, a random forest was trained on auxiliary sensor/GPS data to predict and correct pressure errors. On average, bias correction reduced pressure errors by~80%. During post-processing, fewer than 20% of smartphone pressure were discarded. In a real-world case-study bias-corrected smartphone pressures improved analyses and 1-hour forecasts of altimeter setting, 2-minute temperature, and 2-minute dewpoint.
K. S. Hintz (DMI) first presented a study on wind measurements from smartphones, in which the surface roughness length was estimated from the measured horizontal turbulence. In another work, more than 6 million SPOs were collected over 7 weeks through a software development kit installed in a third-party mobile app. These observations were quality controlled and assimilated with 3D-Var in the DMI HARMONIE NWP system Yang et al., 2017). A decrease of bias and no change in root mean squared error was found for a simulation period of nearly 2 months. Examples showing that raw observations can depict current weather was given.
X. Yang (DMI) presented the construction idea behind the operational COntinuous Mesoscale Ensemble Prediction System (COMEPS) (Yang et al., 2017a(Yang et al., , 2017b at DMI used for a routine weather forecasts, which generates a 2.5 km grid resolution, 25 member, Rapid Update Cycle (RUC) like EPS forecast with an hourly update using time lagging. Currently, a proto-type ensemble nowcasting system applying the COMEPS approach is in development, targeting sub-hourly cycling of the high-resolution nowcasting system assimilating high frequency observation data such as radar and crowdsourced data. One of the novel system components in COMEPS is the time-lagged 3D-Var analysis on overlapped observation windows, which appears especially beneficial in nowcasting applications with variational data assimilation, as the setup appears to have better potential to address observation error correlation in time and space, as well as the issue of model spin-up in connection with frequent assimilation cycling.
A. Cress (Deutscher Wetterdienst, DWD), presented the activities of DWD concerning crowdsourced data applications and their use in the local DWD data assimilation system. Within the Fleet Weather Maps Project FloWKar, a collaboration between DWD and the German car manufacturer AUDI AG has been established, to investigate to what extent future environmental observations from vehicle sensors can be combined with existing data sources to improve nowcasting and warnings and therefore make a contribution to the security of future autonomous driving. A complete real-time weather conceptual framework has been established, focusing on the flow and processing of high resolution measurements and weather products and the development of corresponding forecasts. A fast data exchange is followed by quality control according to weather service standards and smart aggregation strategies, integrating all available data into a real-time weather map. Aiming for fast weather forecasting, a data assimilation cycle with a 5-minute update rate is necessary; therefore, an ultra-rapid data assimilation method is proposed. A real-world application employs the high resolution project observations in a 5-minute assimilation cycle for the regional operational weather model COSMO-D2, focusing on the model performance optimisation near the surface and its predictions along road sections in Germany, where the current observation network is not dense enough. First results, comparing car measurements, nearby weather stations and model analysis and forecasts were presented.
E. Mallet and S. Al Ali (Météo France) first gave a brief overlook of crowdsourcing activities at Météo-France. Those activities focus on the use of human observations from "expert" non-professional observers and "citizen" observers, and automated observations collected from PWSs, agricultural networks and connected vehicles. Then, the presentation focused on two ongoing projects: (1) The first action concerns the crowdsourcing module in Météo-France's mobile application that allows mobile users to report the observed weather and to post pictures of the sky. The module provides, without access restriction, a simple entry of almost twenty phenomena to the users, who in their turn will select the observed phenomena and report the observed weather condition on a regular basis. Based on this module, more than 10,000 observations are collected daily, and more than 40,000 in high-risk situations. Visualisation of crowdsourced data is already available to forecasters, and the next step is to feed it to operational databases in order to expand its possible uses. (2) The second action concerns the potential use of vehicle observations for meteorological applications which is the subject of a partnership between Météo-France and Continental. The aim is to infer weather (precipitation and low visibility) and road conditions (dry, wet, slick) at a particular location in time, through the analysis of vehicle data elements (temperature, wiper and headlight statuses, velocity, and the activation of ABS and ESP systems). The experimental campaign started in November 2016 and is still ongoing. The fleet consists of hundreds of vehicles, transmitting data through a connected dongle. Data filtering and quality checking routines were developed, and vehicle observations were evaluated against meteorological data. Machine learning classification algorithms were developed, using data from meteorological observation merging products as references for hydro-meteor discrimination and visibility. The preliminary results were promising and also showed the need to combine multiple parameters in order to successfully derive weather observations. K. O'Boyle (Met Office) presented how The Met Office view crowdsourcing as distinct from citizen science (see section 4). There is a long history of citizen science at the Met Office. The Weather Observations Website (WOW) (wow.metoffice.gov.uk) is the Met Office citizen science portal. WOW has global reach, and is a platform for anybody to submit, share and display their weather observations, either manually or by connecting a PWS using APIs. WOW data is being trialled in nowcasting applications, but is not yet assimilated into NWP. Investigations into other opportunistic observations are ongoing at the Met Office, including collecting data from vehicles.
M. Clark (Met Office) presented on an automated quality control and gridding process for citizen science data. There has been a focus on Met Office WOW data from PWSs to create high resolution surface analyses. Parameter values from each WOW site are constrained to have the same longterm mean as neighbouring official sites, but are otherwise allowed to vary freely, as it is assumed that shorter-term, temporary deviations are the signature of genuine small scale features which are worth retaining in the analysis. A series of case studies have shown that there is value in this approach.
S.L. Dance (University of Reading) gave an overview of the DARE: Data Assimilation for the REsilient City project. This is a UK Engineering and Physical Sciences Research Council (EPSRC) Senior Fellowship in Digital Technology for Living with Environmental Change. The vision for the project is to use "datasets of opportunity", such as CCTV and vehicle observations, alongside scientific observing networks, such as satellite data (Mason et al., 2018) to improve predictions of urban natural hazards such as flooding and high impact weather. There are many potential benefits of such data, including the availability of large numbers of inexpensive observations, in areas where there are people but there may be few sources of scientific observation data. For example, (1) air traffic management reports have potential to provide observations of temperature inversions in the boundary layer (Mirza et al., 2016(Mirza et al., , 2019. (2) In many locations around the world, the population has access to smartphones, but ground-based scientific observations are sparse. Furthermore, there are a number of issues in collecting 'datasets of opportunity' for use in assimilation. These include the need for metadata such as time and location in order to carry out the assimilation, versus data protection for the data provider, who may be a private individual. Other issues include data ownership, intermittency, heterogeneity, data provenance and large data volumes. In order to use such observations in data assimilation, there needs to be an understanding of natural variability in urban areas (where many of these data originate) and the variability that can be resolved by a prediction model (e.g., Waller et al., 2014;Janji c et al., 2017). This was discussed further in the next talk by J.A. Waller. J. A. Waller (University of Reading) presented on the potential to measure temperatures in urban areas using vehicles. Issues related to the assimilation of crowdsourced data were discussed; in particular, the need to understand the data inhomogeneity and natural variability of observation urban areas in order to understand the observation uncertainties. Collaborative work with the UK Met Office, is assessing the potential of temperature observations recorded by vehicles. The preliminary findings showed that the data collection method was not reliable for collecting large temperature data sets. Furthermore, for the initial data sets collected, it was shown that temperature measurements had a negative correlation with the speed of the vehicle. It was concluded that a new data collection technique was required, and a more detailed study was vital before the benefits of assimilating vehicle temperatures could be assessed.
D. Blaauboer (KNMI and EUMETNET) presented shortly the KNMI-activities in the domain of crowdsourcing. These include participation in the WOW project of UK Met Office, application of car data (temperature sensor, wiper data), smartphone data, damage reporting app (to report weather impacts by the public), wind data from hot air balloons. EUMETNET, the grouping of 31 European National MetServices, recognised the emerging availability and application opportunities of crowdsourced data and the Internet of Things among many of its members. Therefore EUMETNET has organised a few dedicated workshops on this subject with the aim to bring experts in this field together, to foster networking and possibly create a platform or programme in near future to develop common applications to the benefit of all.
M. Dahoui (ECMWF) presented an overview of the importance on in-situ data in global NWP. It was shown that there are data gaps in the surface observations received at ECMWF and the potential and challenges for using crowdsourced data to fill these gaps were described. Also, it was stressed that crowdsourced data can be important for verification purposes. A denser network is useful to detect small scale features and rapid changes of the atmosphere, so observations have also the potential to improve the forecast verification aspects leading to a better understanding of model performance. The usage of crowdsourced observations is however very challenging. It was suggested that data collection and pre-processing needs a collaborative effort between NWP centres through coordination of the WMO, the industry and the private sector to improve and unify standards and to agree on best practices. A common and shared use of operationally managed data hubs (such as the MetOffice Weather Observation Website) is a cost-effective solution to manage the diversity of data sources and formats. A good understanding of the error characteristics of the observations is necessary to allow proper data selection and error specification. This requires a comprehensive and standardised description of metadata. Quality control, bias correction and blacklist management require unique identification of a reporting station which makes anonymous reports of less interest to NWP data assimilation unless technological solutions are available to anonymously identify the data or perform most of the quality control and bias correction near the data origin. Legal aspects related to privacy and data usage are also essential to clarify before the operational use of such observations.

| OVERVIEW OF ACTIVITIES
During the meeting, it became clear that there are many activities on-going, with opportunities for collaboration. Table 1 list activities, status and considerations for participating institutions together with ZAMG and Met Norway who agreed to share their current activities. It is seen that especially work with data from private weather stations is an active field of research at many institutions.

| CHALLENGES AND SOLUTIONS
The presentations and discussions identified several common challenges, and some solutions were proposed during the discussion sessions. These follow below: I. Terminology is not agreed upon in the community. A common vocabulary needs to be established to facilitate future collaboration. The term "crowdsourced data" is used differently within the community, and there is no agreement what this term covers and what not. Often crowdsourced data is used as a collective term, for example, citizen-science and third-party data, which is how the term will be treated in this report, though with a recognition that a more precise definition is desirable. i. The Met Office suggested a terminology that clearly separates citizen-science data and crowdsourced data, and also attempts to define associated terms: a. Citizen-science data: Information obtained from a group of people who are invited to participate in a data collection process. b. Crowdsourced data: Information derived from a group of people without their explicit involvement in the data collection process.
c. Opportunistic data: Information derived from nonmeteorological sensors or weather sensitivities. d. Third-party data: Data collected by a third-party organisation using meteorological sensors. However, some similarities are expected between third-party data and the other groups. For example PWS observations might be classified as both thirdparty data and crowdsourced data. ii. ECMWF proposed four main categories of 'crowdsourced' data; private and third party, automated amateur weather stations, smart connected devices (mobile phones and vehicles), and human reporting of the current weather, relating each of these to the ease of utility in NWP. In the terminology proposed by the Met Office (i), there is a clear separation between citizen-science data and 'crowdsourced' data, wherein the ECMWF proposal (ii) the term 'crowdsourced' data is a collective term. It is T A B L E 1 Overview of ongoing and considered activities at each participating institute and institutes that was not present but approved to be included recommended that authors define their usage of these terms. II. Obtaining useful crowdsourced data may involve collaboration with commercial entities, such as manufacturers of PWSs or vehicles. In some cases, collaborations of this kind mark a step change in the way universities and meteorological institutes have previously operated. For professional use, crowdsourced data needs to be as unprocessed as possible when received. Working in collaboration with manufacturers may enable this. Some of the workshop participants have built successful collaborations with commercial entities, taking a "virtuous circle" approach, whereby data is provided by a manufacturer, and in return the meteorological institution provides forecast data or quality controlled observational data. It is crucial that intellectual property rights and data ownership are clear and agreed upon before starting collaborations. III. Law based restrictions on storage of personal data lead to a need to de-personalise crowdsourced data, which can lead to "black boxes". Metadata can be used to help characterise the error of crowdsourced observations, and for bias correction, but the legal constraints regarding privacy and personal data can limit the collection of such metadata. Hence, metadata vs privacy is one issue that must be considered when collecting observations. DMI have invested in legal expertise and are open to sharing the information obtained with the community. This is mainly related to the European GDPR regulation (European Union, 2018). IV. New data sources can potentially produce more observations than current NWP models can realistically handle. New methods, such as those suggested by Dr. X. Yang (DMI), will need to be considered. Tendencies of parameters are not commonly assimilated into NWP; a change in approach may be required to extract maximum value from crowdsourced observations.
i. It was discussed that data streaming could be a way of handle the amount of observations in future, such that, in operational systems, observations that come in are utilised and then thrown away. This may seem somewhat provocative to some as the NWP community are often used to store data for an extended time. However, it was agreed that data streaming could perhaps be only realistic solution currently to overcome issues with data volume. Also near-real communication could perhaps be easier to implement with a streaming approach.
ii. The scale of crowdsourced observations, any reference network, and NWP models will all be different. To make them comparable, methods to deal with multiscale comparisons are required for example, filtering or superobbing. Further, other themes seemed to be well established. There was a general agreement that crowdsourced data can provide useful observations in areas otherwise devoid of observations. It was discussed whether stationary platforms (e.g., PWS) are easier to implement in existing systems than moving platforms (e.g., vehicles, SPOs). In general, stationary platforms are believed to be easier to bias-correct than moving platforms. Also, new data sources should be seen to supplement conventional observation networks rather than a replacement, as trusted observations are required as a reference for new data sources. A nested platform of reference may be a good way of organising networks in the future, for example, SYNOPs used as a reference for the quality control of PWS data, and PWS then used as a more dense reference dataset for observations from mobile platforms.

| CONCLUSIONS AND RECOMMENDATIONS
Much of the work presented at the workshop was at an early, exploratory stage, and many questions remain unanswered. However, a general set of conclusions were drawn from the discussion. Crowdsourced observations are potentially useful for NWP, and are undoubtedly useful for verification and forecasting. Use of crowdsourced observations in nowcasting, or post-processing, is perceived to be easier and less demanding than in NWP data assimilation. There is still much work to do before crowdsourced observations can widely be ingested into NWP models.
It was agreed upon that there is a sliding scale between 'crowdsourced' or "passive" data collection, where an individual's involvement is limited, and "citizen science" or "active" data collection where the individual is explicitly involved. It is generally thought that the lesser degree of interaction required by the participant the higher the volume of data that can be collected. It is not clear if either of the two are of superior quality.
Further, the following recommendations are made. An organised community of those involved in crowdsourcing activities would be beneficial. EUMETNET would provide a good forum for this, however, such a forum should not be restricted to European countries. This forum could be a simple, independent, platform accessible via a website. Regarding vocabulary, it would be beneficial for the community to agree on common terminology related to crowdsourcing. To realise the full potential of crowdsourced data for NWP, issues of data quality, privacy, and availability will need to be addressed. Data quality could be enhanced by prioritising the collection of accurate metadata. Privacy issues should be addressed to determine if, how, and when unique identifiers can be retrieved for quality control purposes. Lastly, efforts to expand crowdsourced datasets by disseminating data operationally and working with private industry should be encouraged.