Atmospheric boundary‐layer characteristics from ceilometer measurements. Part 1: A new method to track mixed layer height and classify clouds

The use of Automatic Lidars and Ceilometers (ALC) is increasingly extended beyond monitoring cloud base height to the study of atmospheric boundary layer (ABL) dynamics. Therefore, long‐term sensor network observations require robust algorithms to automatically detect the mixed layer height (ZML). Here, a novel automatic algorithm CABAM (Characterising the Atmospheric Boundary layer based on ALC Measurements) is presented. CABAM is the first non‐proprietary mixed layer height algorithm specifically designed for the commonly deployed Vaisala CL31 ceilometer. The method tracks ZML, takes into account precipitation, classifies the ABL based on cloud cover and cloud type, and determines the relation between ZML and cloud base height. CABAM relies solely on ALC measurements. Results perform well against independent reference (AMDAR: Aircraft Meteorological Data Relay) measurements and supervised ZML detection. AMDAR‐derived temperature inversion heights allow ZML evaluation throughout the day. Very good agreement is found in the afternoon when the mixed layer height extends over the full ABL. However, during night or the morning transition the temperature inversion is more likely associated with the top of the residual layer. From comparison with SYNOP reports, the ABL classification scheme generally correctly distinguishes between convective and stratiform boundary‐layer clouds, with slightly better performance during daytime. Applied to 6 years of ALC observations in central London, Kotthaus and Grimmond ( ), a companion paper, demonstrate CABAM results are valuable to characterise the urban boundary layer over London, United Kingdom, where clouds of various types are frequent.


INTRODUCTION
The mixed layer (ML) height (Z ML ) can be identified by different physical indicators. The top of the atmospheric boundary layer (ABL) is usually marked by a clear temperature inversion, decrease in humidity and strong turbulence, so that the Richardson number is often used to detect this height based on radiosonde profiles (e.g. Piringer et al., 2007). The height of maximum turbulence can be determined from Doppler lidar observations (e.g. Barlow et al., 2011) or Sonic Detecting And Ranging (SODAR) systems (e.g. Emeis et al., 2008). However, the latter are restricted to shallow boundary layers due to their limited range. Automatic lidars and ceilometers' (ALC) compact design, low cost, and high range resolution (∼10 m) make them advantageous to many of the alternative systems. ALC aerosol-studies have explored particles dispersed within the ABL (e.g. Tsaknakis et al., 2011) and layers from Saharan dust (e.g. Knippertz and Stuut, 2014), biomass burning (Mielonen et al., 2013), and volcanic ash (e.g. Wiegner et al., 2012;Marzano et al., 2014;Nemuc et al., 2014). Developed as cloud base height (CBH) recorders, ALC provide automatic CBH estimates, with multiple cloud layers identified (Martucci et al., 2010). Given they are automatic and low maintenance, ALC networks are widespread (e.g. DWD, 2018).
To exploit ALC data for ABL metrics, automatic Z ML retrieval methods are required. Although considerable effort has been made to improve various algorithms for the detection of Z ML from ALC attenuated backscatter profiles (section 2.1), no open-source algorithm is available that utilises only ALC observations to adequately and continuously detect Z ML patterns from Vaisala CL31 ceilometers. This commonly deployed instrument has the advantage of reaching complete optical overlap at low ranges so that the whole observed profile is available for mixed layer height analysis.
Evaluation of Z ML results is demanding, given reference measurements are often scarce and all observational methods are challenged by the complex task of layer attribution (section 2.2). Radiosondes, commonly used as the reference, are often limited in temporal resolution. In regions with heavily trafficked air-space, Aircraft Meteorological Data Relay (AMDAR) observations provide an alternative. Still, careful analysis is needed when temperature inversions indicate the top of the residual layer (RL) rather than the mixed layer.
While there is common agreement on the applicability of boundary-layer aerosols as a tracer for atmospheric mixing under cloud-free conditions (Barlow, 2014), the limits of interpretation of attenuated backscatter profiles in cloudy and rainy conditions are yet to be determined. Caicedo et al. (2017) discuss the implications of ALC signal response in clouds for different Z ML -methods. To address the role of clouds in sufficient detail (Schween et al., 2014), classification of ABL by not only cloud cover but also cloud type is required (section 2.3).
Most studies exclude rainy periods based on nearby surface station measurements. However, these are likely to miss certain types of precipitation events, such as rainfall evaporating above ground level and light rainfall. de Bruine et al. (2017) explicitly show examples of Z ML detected during rainfall which highlight the risk for false layer attribution under complex conditions. For example, the evaporation layer can cause strong negative gradients in the attenuated backscatter ("dark band") that might lead to false layer attribution.
The objective of this work is to describe and evaluate a novel algorithm for the characterisation of the ABL. The CABAM ("Characterising the Atmospheric Boundary layer based on ALC Measurements") algorithm uses only ALC measurements to track Z ML and to classify the ABL by cloud cover and cloud type (relating Z ML to CBH), incorporating a novel rainfall filter. Specifically designed for use with Vaisala CL31 data, CABAM includes a module to reduce false layer detection due to near-range artefacts (Kotthaus et al., 2016).
After providing some background on state-of-the-art methodologies (section 2), the CABAM algorithm is introduced (section 3). Mixed layer height results are evaluated against temperature inversions derived from AMDAR profiles and the ABL classification scheme is compared to SYNOP reports (section 4). The summary (section 5) suggests this new, automatic tool is suitable to characterise the ABL based on long-term ALC measurements (Kotthaus and Grimmond, 2018).

Automatic mixed layer height detection
Firstly, regions or heights of potential layer boundaries are detected based on a range of indicators, such as negative vertical gradients (e.g. Schäfer et al., 2004;Münkel et al., 2007;Emeis et al., 2008), continuous wavelet transform detection (e.g. de Haij et al., 2006;Baars et al., 2008), regions of high variance (e.g. Martucci et al., 2007), or some combination of these (e.g. Lammert and Bösenberg, 2006;Martucci et al., 2007;Haeffelin et al., 2012;Poltera et al., 2017). For example, STRAT-2D (Morille et al., 2007;Haeffelin et al., 2012) uses the variance field to determine which wavelet-detected negative gradient is likely associated with Z ML , whereas pathfinderTURB (Poltera et al., 2017) combines gradient and variance field diagnostic before tracing Z ML . The "hybrid" approach of COBOLT (Geiß, 2016) uses a varying combination of gradients, variance statistics and the wavelet transform depending on solar angle. While detection based on negative gradients may be more prone to noise than the wavelet method, it has the advantage of capturing potential layers at low ranges (Di Giuseppe et al., 2012). The idealised profile method (Steyn et al., 1999) fits a theoretical profile to the observed attenuated backscatter (e.g. Eresmaa et al., 2006;Peng et al., 2017).
While some studies apply detection separately for each instantaneous time stamp, temporal tracking of layers can significantly improve consistency (e.g. Martucci et al., 2010). The recent pathfinderTURB (Poltera et al., 2017), based on the pathfinder algorithm (de Bruine et al., 2017), applies a graph theory approach to track Z ML through the course of the day, while COBOLT (Geiß et al., 2017) uses a time-height-tracking approach with moving windows.
The proprietary Vaisala BLview software (Münkel, 2016) combines the negative-gradient and profile-fit approaches but information about this method is limited. BLview is increasingly being applied to observations of the Vaisala CL31 and CL51 systems with varying performance (e.g. Lotteraner and Piringer, 2016;Tang et al., 2016). Haman et al. (2012), Wagner and Schäfer (2017) and Caicedo et al. (2017) apply post-processing to reduce false detection by BLview. Haeffelin et al. (2012) conclude, from their comparison of different mixed-layer height detection techniques applied to two ceilometer types (Vaisala CL31 and Jenoptik CHM15K), that there is consistency in location of significant vertical gradients detected. The greatest uncertainty is associated with layer attribution (e.g. distinction between ML and RL), even when simple categories are applied. This is consistent with findings of Haman et al. (2012).
Some use auxiliary information to assist layer attribution. For example, Di Giuseppe et al. (2012) use a bulk model (Stull, 1988) derived time series of surface sensible heat flux, and STRAT+ (Pal et al., 2013), the successor to the variance-based STRAT-2D (Morille et al., 2007;Haeffelin et al., 2012), uses radiosondes and turbulent flux measurements (if available).
Detection of nocturnal boundary-layer heights, in contrast to the residual layer, is a major challenge (Haeffelin et al., 2012;Lotteraner and Piringer, 2016;de Bruine et al., 2017). Whilst the strongest negative vertical gradient may be a good indicator during daytime clear-sky conditions, it is less applicable for morning or afternoon transitions (Haeffelin et al., 2012). Focusing on daytime conditions, Poltera et al. (2017) find pathfinderTURB is least accurate during afternoon transitions.
Applicability of algorithms for Z ML -detection from ALC observations also depends on the quality of the attenuated backscatter profiles analysed. This may vary with sensor type (Madonna et al., 2015), but also hardware generation, firmware version and post-processing applied (Kotthaus et al., 2016). Of the two most-widely deployed ALC, Vaisala CL31 and Lufft CHM15K, the former's weaker laser causes a lower signal-to-noise ratio (SNR). If noise levels are high, fitting an idealized profile or detecting significant vertical gradients of attenuated backscatter can be challenging (Eresmaa et al., 2012;Haeffelin et al., 2012). Smoothing increases the SNR (e.g. Markowicz et al., 2008;Haeffelin et al., 2012;Stachlewska et al., 2012) and can augment data availability (Kotthaus et al., 2016). However, use of absolute SNR thresholds can reduce data in relatively clean ABL conditions (de Bruine et al., 2017).
Although higher-power lasers significantly increase the SNR, incomplete optical overlap spanning several hundred metres prevents analysis of profiles close to the ground. While adequate overlap correction can improve applicability of measurements in this region (Hervo et al., 2016), the nocturnal mixed layer near the ground is still often undetectable (e.g. Poltera et al., 2017). As optical overlap, instrument noise, and instrument-related background are generally sensor specific, the performance of an ALC model may vary between individual instruments.
Studies using CL31 observations have successfully detected Z ML (e.g. Münkel et al., 2007;Van der Kamp and McKendry, 2010;Eresmaa et al., 2012;Sokół et al., 2014;Tang et al., 2016), typically with better performance under convective conditions when aerosols are well-dispersed. Instrument-related artefacts in the attenuated backscatter profiles may sometimes hinder automatic Z ML detection from CL31 data. Near-range artefacts (e.g. Sokół et al., 2014) are addressed by Kotthaus et al. (2016). In some cases BLview tends to assign Z ML to gradients around the 600-700 m range , although boundary-layer clouds indicate a significantly higher extent of the ML. Such biases towards certain regions in the profile might be caused by artefacts in the instrument-related background profile or height-dependent internal averaging settings (Kotthaus et al., 2016).

Evaluation of mixed layer height
Mixed layer heights derived from RS profiles are commonly evaluated against radiosonde data. Various methods (e.g. Biavati et al., 2015) are used to detect the top of the ABL often marked by a temperature inversion, decrease in humidity and strong turbulence. Applicability of radiosonde profiles for Z ML evaluation depends on meteorological conditions, surface heterogeneity, location (or spatial displacement) and temporal coverage. If the aerosol-derived Z ML coincides with the thermodynamic markers of atmospheric mixing, very good agreement is found between ALC and balloon profile values (e.g. Tang et al., 2016). Best agreement is usually reported near midday or early afternoon (Sokół et al., 2014;de Bruine et al., 2017), when the aerosol-loaded mixed layer extends over the whole ABL and hence reaches the height of the entrainment zone. Considerable disagreement between thermodynamic indicators and aerosol tracers can occur (Collaud Coen et al., 2014), e.g. during stable nocturnal stratification (Caicedo et al., 2017). When several layers are present, greater uncertainty is found in the analysis of both radiosonde and ALC profiles as, for example, the top of the residual layer may be associated with a stronger temperature inversion and stronger aerosol gradient than the top of the mixed layer (de Haij et al., 2006).
Alternatively, ALC-derived Z ML can be compared to products from stronger aerosol lidars, which may have advantages under very clean conditions with low-aerosol loading due to a higher SNR. However, algorithms are still challenged by layer attribution (Haeffelin et al., 2012). Care must be taken when comparing aerosol-derived and turbulence-derived layer boundaries (e.g. from Doppler lidar or wind profiler measurements: Collaud Coen et al., 2014;Schween et al., 2014) given the vertical distribution of aerosols is a result of previous mixing processes, and discrepancies may occur to instantaneous turbulence statistics (Pearson et al., 2010). Poltera et al. (2017) develop an "expert method" based on ALC and auxiliary observations to manually trace the daytime evolution of Z ML for one year. Estimates between different experts agree well (root-mean-square error, RMSE = 92 m), but differences exceed 500 m in a few complex cases. These expert-based reference data are used for successful evaluation of pathfinderTURB results.

Atmospheric boundary-layer classification
Characteristics of mixed layer height, residual layers and cloud base height vary due to synoptic background conditions. In the presence of clouds, the strong gradient near the CBH is often used as a proxy for Z ML (e.g. Davies et al., 2007;Schäfer et al., 2008), but only a few studies explicitly state that CBH is considered representative (Wiegner et al., 2006). With ALC manufacturers using different approaches to automatically detect CBH (Martucci et al., 2010), care is needed in data interpretation. As Schween et al. (2014) discuss, Cumulus (Cu) clouds forming at the top of the ML during daytime are part of the common ABL concept (Stull, 1988), but the relation of Stratus (St) clouds to ABL dynamics is more complex. They suggest Stratus cloud cover periods should be removed from analysis and use a threshold of ≥4 okta (Schween et al., 2014). Apart from slight changes in winter, average seasonal patterns remain similar and comparison to Doppler-derived mixing height worsens slightly. A more sophisticated classification to account for the clouds' impact on Z ML patterns is needed. Pal et al. (2013) classify ABL regimes using cloud cover (cloudy vs. clear-sky) and atmospheric stability (from surface observations) to distinguish between days dominated by surface-driven buoyancy and those with larger-scale effects. Cloud-cover alone may be insufficient to assess the impact of clouds on boundary-layer dynamics (Pal and Haeffelin, 2015), rather it would be useful to distinguish between boundary-layer clouds and those decoupled from the ABL. Poltera et al. (2017) use the cloud-thickness reported by the Lufft CHM15K to distinguish between ABL clouds and those above. Peng et al. (2017) classify by cloud cover and then manually group days by relation to detected Z ML and potential residual layer to determine if the nocturnal detection results could be interpreted as the layer connected to the surface or the residual layer above. Harvey et al. (2013) propose an automatic classification of boundary-layer types based on Doppler lidar observations. To infer cloud types at hourly time-scales they discriminate different turbulence regimes from Doppler wind profile observations and the state of mixing using turbulent sensible heat flux observations from a surface station.

Ceilometer observations
The atmospheric boundary layer over London is characterised using observations from a Vaisala CL31 ceilometer within the LUMO network (http://micromet.reading.ac.uk/). The sensor with a generation 321 engine board, receiver and transmitter has had different firmware versions (Table 1;  The CL31 operates at a wavelength of 905 ±10 nm at 298 K, which makes it sensitive to water vapour (Wiegner and Gasteiger, 2015) in addition to aerosol (Münkel et al., 2007). The augmented attenuation from humidity contributes to an improved SNR in the ABL. The CL31 measurement range is 0-7,700 m from the sensor. Due to its compact single-lens design (Münkel et al., 2007), complete optical overlap of the CL31 is reached at only 70 m above the instrument (Kotthaus et al., 2016) and observations are basically usable up from the first or second range gate (despite the first sample often being very noisy). This gives a clear advantage over other commonly used ALC that usually show great uncertainty in the range below 200-500 m (e.g. Vaisala CL51, Lufft CHM15K). As recommended by Vaisala, resolution is set to 15 s and 10 m given information recorded at higher sampling frequencies or smaller range gates overlaps significantly (Kotthaus et al., 2016). The beam divergence of the CL31 is ±0.4 mrad so that the probed area at 2,000 m is about 2 m 2 . This limited field of view (FOV), can be increased by temporal averaging, e.g. to identify if broken clouds are passing over the sensor.
All data recording and analysis is done in UTC. Time stamps are time ending of an averaging period. Data analysis is done in R (R Core Team, 2017).
In the study period (2011-2016) the sensor was located at two sites (Table 1; Figure 1), with sensor height above ground-level (agl) of 32.1 m (KCL) and 4 m (MR). The great majority of observations were gathered at MR.

Ceilometer data processing
Attenuated backscatter from the CL31 ceilometers is processed according to Kotthaus et al. (2016) to ensure adequate background correction. However, the near-range correction is  (Kotthaus et al., 2016), while preserving detailed features in the atmospheric backscatter. Smoothed attenuated backscatter with an SNR < 0.18 (Kotthaus et al., 2016) is excluded from analysis. Here, the filter is applied with a small range-offset to retain observations with low SNR at the top of the ABL, i.e. data are used for analysis if SNR at the exact range gate g, two range gates below (g−2) or 10 range gates below (g−10) exceeds the quality-control threshold.
The Vaisala CL31 reports CBH for up to three layers at the set resolution (here 15 s, 10 m). The Vaisala CBH algorithm, based on visibility criteria for aviation purposes, is proprietary and provided without details. To derive a CBH with a wider representation of the sky, clouds passing the sensor along the wind direction at cloud level are accounted for by calculating the first percentile of CBH reported in a moving window of 30 min. This percentile is chosen (rather than the minimum) to reduce the impact of single outliers. Minimum CBH for a 15 min block period is the first CBH-bin with at least two counts (i.e. 30 s). Only CBH ≤ 3,000 m agl are considered relevant for the ABL over London. Cloud cover (CC) is the percentage of times with CBH ≤ 3,000 m within the 30 min moving period and then block-averaged to 15 min. Radiosonde data are rare in dense urban settings (Piringer et al., 2007), with no station within or near London. To evaluate Z ML , temperature profiles from Aircraft Meteorological Data Relay (AMDAR) associated with the five London airports (i.e. LHR, LGW, LUT, STN, LCY; Figure 1) are combined. On days with no significant cloud cover, a spatially consistent temperature inversion is assumed to mark the top of the ABL across the extended Greater London area.
AMDAR profiles available from the British Atmospheric Data Centre (BADC; Met Office, 2008) for 2011-2016 are extracted for an area of ±1 • around KCL (Figure 1) up to a height of 4 km. Pressure heights are converted to heights agl as described by Rahn and Mitchell (2016). Required surface pressure observations are available only at LHR (Met Office, 2012), so these are translated to the other airport sites considering orography. Flights are separated by flight number and reporting time. For inclusion in the analysis, data below 1,500 m agl must be present. Each flight is assigned to an airport based on the lowest observation in the profile and to a 15 min time interval based on measurement time closest to 1,000 m agl. For each time interval, the flight with the greatest number of data points is selected. To create a homogeneous dataset, cubic spline functions are fitted through the temperature profiles with a 50 m resolution starting from the first available observation agl.

Characterising the Atmospheric Boundary layer based on ALC Measurements (CABAM)
A novel method to characterise the atmospheric boundary layer solely based on ALC measurements is presented. This includes the tracking of Z ML (section 3.2.1) and classification of the ABL according to cloud cover and cloud type in relation to Z ML (section 3.2.2).

Mixed layer height detection
The algorithm for mixed layer height Z ML detection from ALC backscatter was developed based on the method proposed by Emeis et al. (2008). Smoothed attenuated backscatter profiles (section 3.1.1) are analysed at their sampled resolution (10 m, 15 s), but restricted to a range ≤3,000 m agl (Figure 4a) as the ABL over the study area is located well below this height. If the method was to be applied in a region with significantly more boundary-layer buoyancy, this restriction should be modified. Given LUMO ALC are operated with a small inclination angle (∼3 • ), Z ML detection is performed based on range (i.e. distance from sensor) and final layer heights are converted to m agl before analysis. Daily detection uses 26 hr of data, with 1 hr from each of the two adjacent days. This information at the end (start) of the previous (following) 24 hr reduces discontinuities as the date changes.
Corrected and smoothed attenuated backscatter (section 3.1.1) is used to calculate gradient (G = d × dr −1 ) and second derivative (D = d 2 × dr −2 ) over range windows of 100 m (10 range gates). For the lowest ranges (r < 80 m) this window size is reduced to 20 m. Decrease in attenuated backscatter at range gate g is considered significant if the magnitude of the negative gradient exceeds a threshold TH G , i.e. G| g ≤ TH G , and the second derivative indicates an inflection point, i.e. D| g − 1 < 0 and D| g ≥ 0. Emeis et al. (2008) use a threshold of −0.30×10 −9 m −1 sr −1 below 500 m and −0.60×10 −9 m −1 sr −1 above this height. Here, an average of these two values is used generally (TH G = −0.45×10 −9 sr −1 m −1 ). To minimise false detection due to artefacts induced by smoothing and gradient calculation, this value is doubled below 200 m during daytime (2 × TH G ).
Points of significant gradient G * are converted to layers using the following steps ( Figure 2). Points are connected over 12 iterations (index i) with changing window size in time (∈ {0.25, 2, 10, 20, 20, 30, 30, 30, 45, 60, 60, 90} min) and range (R ∈ {70, 30, 50, 50, 70, 70, 100, 120, 120, 120, 140, 200} m), choosing the closest layer, i.e. with the minimum range difference dr, for connection. If two layers have the same distance to the layer that is currently being traced (n = 2), the connection is established between layers with minimum difference in vertical gradient dG * . After the 2nd, 8th and 12th iterations, layers are removed with insufficient number of data points P over its duration L, or a low detection density K = P/L.
After the connection process (Figure 4b), three quality-control (QC, Figure 2) steps are conducted (Figure 4c) to remove layers in the near range that are considered artificially introduced by instrument-related artefacts (Suppl. S2.1 in File S1), layers above clouds (Suppl. S2.2 in File S1), and those deemed unreliable due to precipitation (Suppl. S2.3 in File S1).
Finally, a number (N) of layers are analysed for attribution of the mixed layer and other significant layers such as the nocturnal residual layer. In the following, subscript "ML" denotes the mixed layer, i.e. the layer with (recent) turbulent connection to the surface. The nocturnal layer is denoted "NT", but not specifically identified in the CABAM final results. NT is used only as a means to distinguish Z ML . Z ML is traced through the course of the day using a series of criteria based on the height of the layer at different times and the associated vertical gradient in attenuated backscatter. The process of layer attribution (Figure 3) addresses different times of the diurnal cycle, defined by sunrise (SR), solar noon (SN), sunset (SS), afternoon (AN = 0.5 × [SN + SS]), and midnight (MN). Solar times are calculated from solar angles for the measurement site using the R insol package (Corripio, 2014). The process of layer attribution is illustrated for an example case (Figure 4c-h).
Detailed rules and decision criteria (Suppl. S1 in File S1) are developed empirically and grouped into seven modules ( Figure 3): 1. Before SR: Initially ML is defined as the lowest layer around SR and traced back in time until MN ( Figure S1, File S1; Figure 4d). 2. After SS: The lowest nocturnal layer before MN is identified as NT and traced back in time till SS ( Figure S2, File S1; Figure 4e). 3. Morning transition: Starting from ML identified in step 1, Z ML is connected to layers above or after during the morning transition ( Figure S3, File S1; Figure 4f). 4. Daytime: It is ensured ML marks the lowest layer present around SN ( Figure S4, File S1). 5. Evening transition: ML is connected to layers below that are close ( Figure S5, File S1; Figure 4g). 6. Sunset: If ML is above NT in the hours before MN, the two layers are swapped ( Figure S6, File S1). 7. Consistency: If ML ends above the start of the mixed layer on the next day (ML +1 ), ML is assigned the residual layer (RL) from SS to avoid discontinuity at midnight. In this case no Z ML is detected between SS and MN.
The result is Z ML plus a number of additional layers that may form residual layers during the night or the top of the ABL during the day when the mixed layer does not extend through the whole ABL. Currently the CABAM code allows nine additional layers to be stored as this was found sufficient for most London cases. This number can be increased if required in other environments.
To ensure the detected mixed layer is connected to the ground, a final check is performed. If strong positive gradients in attenuated backscatter (G > 5 × 10 −9 ) are present within ML (i.e. below Z ML ), the layer is retained for further analysis as an elevated aerosol layer and no Z ML estimate is detected simultaneously.  Overview of steps taken to track the mixed layer (ML) height Z ML through the day. Individual steps address (1) time before sunrise ( Figure S1, File S1), (2) time after sunset ( Figure S2, File S1), (3) morning transition ( Figure S3, File S1), (4) daytime ( Figure S4, File S1), (5) evening transition ( Figure  S5, File S1), (6) sunset ( Figure S6, File S1), and (7)  To reduce the number of layers to be stored, residual layers before sunrise are combined and short layers (<4 hr) are removed. If still more than nine layers are detected in addition to Z ML , layers with the highest detection density are selected. The final output is 15 min block averages (time ending) of Z ML and the additional layer heights, with respective standard deviations and number of samples. A 15 min estimate is considered for final analysis (section 4) if data availability is ≥50%.
In addition to the automatic detection of Z ML , a supervised classification is performed in which erroneous layers (mainly associated with near-range artefacts and ripple effects) that fail automatic quality control are removed manually before layers are connected automatically to Z ML .

Atmospheric boundary-layer classification
To account for effects of clouds on boundary-layer dynamics, ABL characteristics are classified according to cloud cover and cloud type. The CABAM approach uses only ALC observations so other measurements (e.g. sensible heat fluxes, humidity) are available for independent corroboration and analysis in future studies. Classification is conducted based on 24 hr periods from sunset to sunset, i.e. a day centred on SR (day SR ). Using day SR (rather than calendar dates) ensures that the night is treated as a continuous entity. The ABL classification uses CC, CBH (section 3.1.1), Z ML (section 3.2.1) and the rainfall flag derived from attenuated backscatter profiles (Suppl. S2.3 in File S1). Given that both cloud amount and type influence and indicate ABL structure, a distinction is made between convectively driven and more uniform, stable cloud structures. Although two classes do not fully capture the range of cloud types and mechanisms influencing the ABL, it provides a first-order classification. The two classes are hereafter: (a) Cumulus ("Cu") -clouds associated with clear convective activity, and (b) Stratus ("St") -persistent clouds with low vertical variability of CBH. Stratocumulus clouds are not explicitly accounted for and may fall into either category.
Given the focus on morning transition and diurnal evolution of Z ML , the classification does not use observations from the first third of the night (i.e. 0.66 ⋅ H night ; H night : night-length [hr]) and last third of the day period (i.e. 0.66 ⋅ H day ; H day : day-length [hr]). Hence, a cloud forming around sunset will not impact the classification of the day SR period.
Classification has three steps: 1. Individual 15 min periods are classified into the categories "cloudy" (CC > 20%) or clear. Cloudy periods  (Figure 2), (c) as (b) after quality control removed physically unreasonable layer in near range (Suppl. S2.1 in File S1), (d) before sunrise ( Figure S1, File S1): mixed-layer (ML) detected as lowest layer around sunrise and traced back to midnight, (e) after sunset ( Figure S2, File S1): nocturnal layer (NT) detected as lowest layer before midnight and traced back to sunset, (f) morning transition ( Figure S3, File S1): ML connected to other layers before solar noon, (g) evening transition ( Figure S5, File S1): ML connected to NT incorporating layers in-between, and (h) final layer result (ML + additional layers in residual layer) after excluding layers of short duration or low density. SR = sunrise; SN = solar noon; AN = afternoon; and SS = sunset [Colour figure can be viewed at wileyonlinelibrary.com] are subdivided relative to Z ML , i.e. whether the cloud layer coincides with the Z ML or is located above (Z ML < CBH -250 m). 2. Statistics for daytime (day: t > SR) and night-time (night: t ≤ SR) determine if the respective period is mostly clear (cloudy ≤10% of 15 min periods), cloudy (cloudy >50%) or partly cloudy (otherwise), again differentiating between a cloud-topped mixed layer and a detached ABL cloud layer above the Z ML . 3. The entire day SR period is classified as a combination of the night and day indicators (Clear, Cu, St, Z ML < Cu, and Z ML < St). day SR with high amount of missing data (> 6 hr of day SR , or < 1 hr available in either night or day) are excluded. Also, if more than 4 hr during day are considered inappropriate for Z ML detection due to complex precipitation (Suppl. S2.3 in File S1), day SR is grouped into the "rain" category indicating low confidence in Z ML derived for that period.
To classify the predominant cloud type, the longest continuous cloud period during night is examined. If it is persistent (duration ≥5 hr or 75% of night) and its CBH shows little variability (standard deviation after removal of temporal trend * CBH < 60 m), the nocturnal cloud is classified as St, and as Cu otherwise. If this cloud continues into daytime for at least 1.5 hr and ≥ 60% of the daytime clouds have a low CBH (<500 m), the cloud during day is also classified as St. Daytime clouds are also considered St if day is classified as cloudy and the longest continuous cloud with low height variability lasts for ≥4 hr.
Fog is considered to occur when CBH in a time window around sunrise (SR−6 hr to SR+3 hr) is located below 110 m agl for >30 min and rain is detected for less than a third of that low-CBH period. Analogously, the "high fog" class is assigned if CBH < 400 m agl for >30 min during a slightly shorter, later time window (SR−1 hr to SR+5 hr).

Evaluation of mixed layer height against AMDAR
Temperature profiles from AMDAR (section 3.1.3) are used to evaluate Z ML . Clearly, the high temporal coverage from the five airports in the region (Figure 1) is advantageous compared to radiosonde releases. Although data coverage prior to October 2014 is sparse (Met Office, 2008), the height of the first temperature inversion (z ΔT ) is estimated (section 3.1.3) for 35,463 15 min periods (i.e. equivalent to 1 year) with good coverage in all seasons. To focus the comparison on aerosol-dominated ABL conditions, day SR periods with non-cloudy daytime characteristics are selected with at least 1 hr of z ΔT values detected for the time ≥4 hr past sunrise. These are analysed by season (Figure 5a,c,e,g). While z ΔT and Z ML occasionally agree even during the night or the morning transition (MT), a prominent temperature inversion is usually present at some height above the nocturnal Z ML , presumably marking the top of the residual layer. Hence, direct comparison (Figure 5i-l) focuses on periods ≥4 hr after sunrise without clouds below 3,000 m (shading in Figure 5a,c,e,g indicates subset shown in Figure 5b,d,f,h). As many as 1,630 time periods fulfil these criteria with both z ΔT and Z ML detected. While linear regression statistics (Figure 5i-l) suggest limited agreement between the two methods (slope = 0.1-0.87; R 2 = 0.02-0.12), the time series (Figure 5b,d,f,h) reveal a good general match on most days. This is supported by a mean bias error (MBE) between only 63 m (summer) and 302 m (winter).
Temporal variation of z ΔT , linked to the spatial extent of the AMDAR data gathered (Figure 1), explains large parts of the discrepancy. Only occasionally does Z ML consistently differ from a well-defined, spatially consistent inversion height (Figure 5a-h). Such cases may be explained by three different effects: (i) the aerosol layer connected to the surface does not extend to the height of the thermal inversion, i.e. the two indicators refer to different atmospheric layers, (ii) aerosol loading is too low so that the particles captured by the low-power ALC do not sufficiently trace the convective mixing during daytime, or (iii) layer selection by the algorithm is inappropriate. Manual inspection of the cases with the largest discrepancy reveals a distinct aerosol layer located below z ΔT on several occasions (not shown).
The median difference (Δz = z ΔT − Z ML ) is 346 m based on all time periods; however, only 24 m for times >4 hr after sunrise, as the two ABL estimates converge during the day (Figure 6a). CABAM not only detects Z ML but also other layers of significant gradient in attenuated backscatter (section 3.2.1) so that the height z top of the highest detected layer in each measurement period can be compared to z ΔT (Figure 6b). During daytime, z top often agrees with Z ML . However, at night and early morning it frequently represents the top of the residual layer. At night, z ΔT has much better agreement with z top (Figure 6b) compared to Z ML (Figure 6a), and hence seldom marks the top of the mixed layer.

Automatic versus supervised mixed layer height detection
At times the automatic Z ML -detection selects erroneous layers that are instrument-related noise or artefacts (Kotthaus et al., 2016) rather than atmospheric signatures. The QC module designed to exclude artificial layers in the near range <200 m (Suppl. S2.1 in File S1) generally performs well, but it is difficult for the algorithm to determine if a layer is physically reasonable. The ripple effect induced by the instrument transmitter (Kotthaus et al., 2016), or processing procedures internal to the instrument (e.g. height-dependent averaging), may increase the likelihood of significant gradients being detected at certain ranges. These sensor-related artefacts in the attenuated backscatter profiles can cause confusion in the automatic detection and attribution process .
Hence, CABAM Z ML -detection is performed here with an additional supervised QC step (section 3.2.1), that excludes erroneous layers manually before Z ML is automatically tracked through the day. This mostly affects layers induced by near-range artefacts but also some layers ∼670 m were deemed artificial occasionally, based on unrealistically low height variability. Z ML from supervised detection differs from the automatic retrieval for only 1.9% of the 15 min periods. The overall MBE between the two estimates is <5 m (<10 m for days where supervised detection modified results). However, for individual cases the difference can be significant (maximum difference = 2,246 m).
The impact of supervised QC on statistics of minimum Z ML (z min ), maximum Z ML (z max ), and morning transition growth rate is assessed for all 991 day SR periods affected by the supervised detection (Figure 7), by ABL class (section 3.2.2). As seen from the number of day SR periods in each class, the classification results are similar between the two methods. On 11 days artificial layers in the near-range not identified by the automatic QC hindered the successful calculation of morning transition growth rate (Figure 7c), so that slightly more samples are included in the statistics of supervised Z ML .
In general, the supervised QC step has little effect on the summary statistics of all three indicators, with the least influence on z min (Figure 7a). The daytime maximum is slightly increased with supervised QC for classes This suggests that the supervised QC step could be valuable to reduce uncertainty when a short period is being analysed. For long-term applications, however, the automatic retrieval still gives reliable results that can be used to characterise the overall climatology, including seasonal and diurnal variations by ABL class.

Comparison of ABL classification against SYNOP reports
A direct evaluation of the CABAM classification scheme (section 3.2.2) is not possible because SYNOP reports are unavailable for central London. As the augmented surface roughness, lack of moisture and excess of sensible heat associated with the dense urban area likely affect cloud formation and dissipation, spatial differences in cloud cover between the study area and the closest sites providing SYNOP information (Figure 1) are expected.
SYNOP observations from six Met Office sites around London (Figure 1) are used. At each site, times with CBH > 3,000 m are excluded as is done for CABAM. An overall cloud cover estimate for central London is set to the maximum value reported across the Met Office stations. As cloud type is reported at only one site (MNH, west of central London) this is deemed representative of the study area.
As in CABAM, each day SR period is classified from SYNOP reports as a combination of night and day, excluding the first third of night and the last third of day. A period is considered clear if CC > 3 oktas occurs ≤10% of the time. The dominant cloud type is assigned to periods classified as cloudy. For comparison with the SYNOP classes, the "below-cloud" classes of CABAM as well as the classes marked by slight rain are merged with the respective FIGURE 7 Three indicators of the mixed layer height Z ML diurnal cycle, by major ABL classes (see section 3.2.2): (a) nocturnal minimum z min , (b) daily maximum z max , and (c) morning transition growth rate determined from automatic and supervised CABAM detection for 991 day SR periods when supervised QC modified results. Number of day SR periods is given above each sub-plot cloudy class. The "rain" class (complex precipitation does not permit successful detection of Z ML ) is excluded from this comparison.
Very good agreement between CABAM classes and SYNOP reports (Figure 8) is found for the Clear category (112 days, 88% total agreement), clear night followed by Cu day (151 days, 80%), and St (169 days, 58%). However, for the class with highest occurrence, i.e. Cu during both night and day (456 days), a lot of the nights are classified as being clear by the ALC scheme so that the total agreement only reaches 42%. In this case, agreement more than doubles (to 87%) if only daytime periods are considered. For all cloudy classes, agreement is significantly higher if only daytime periods are compared.
Least agreement of only 20% is reached on the 5 days when SYNOPs indicate a clear night followed by a day with St, as the ALC classification mistakes the clouds for Cu. This is likely explained by the "morning bias" of the ALC scheme which is centred around sunrise and a certain duration of the clouds is needed to be classified into St. If the St forms during the day or in the afternoon, the ALC scheme is not able to detect its stratiform nature.
The ALC scheme does not include a stratocumulus (SC) class. In the comparison (Figure 8), the 355 (460) periods classified as SC by SYNOPs during daytime (night-time) are more likely to fall into the Cu category (day: 53%, night: 52%) rather than the St category (day: 33%, night: 20%) of the ALC classification.
Considering the uncertainty introduced by the spatial representation of the SYNOP data to describe the central London area, the CABAM classification scheme for ABL conditions compares generally well to the observed cloud classes. Especially the significant agreement during daytime suggests the ALC-derived classes are sufficiently accurate to enhance analysis of the mixed layer height climatology (Kotthaus and Grimmond, 2018).

SUMMARY AND DISCUSSION
A novel algorithm for Characterising the Atmospheric Boundary layer (ABL) based on Automatic lidar and ceilometer (ALC) Measurements (CABAM) is presented. The tool is capable of automatically tracking the mixed layer (ML) height Z ML , filtering periods affected by complex rain patterns, and further classifies ABL characteristics into nights and days affected by clear sky, convective clouds, stratiform clouds, or ABL clouds that are present but the mixed layer remains below cloud base height (CBH). Given ALC provide vertical profiles of attenuated backscatter and automatic CBH detection, they are very suitable to determine ABL characteristics automatically. CABAM is designed for observations from the Vaisala CL31 ceilometer, an instrument which reaches complete optical overlap by 70 m. Quality-control modules are implemented to reduce uncertainty induced by instrument noise and near-range artefacts. With this careful processing, layers as low as 50 m agl can be detected.
Application of CABAM to six years (2011-2016) of CL31 observations in central London provides high-resolution (15 min, 10 m) results. Retrieved Z ML is successfully evaluated against inversion heights derived from temperature profiles of Aircraft Meteorological Data Relay (AMDAR). For a study area such as London with several planes per hour providing AMDAR data (here flights are extracted every 15 min) there are advantages over the temporal resolution of radiosonde profiles. Day-to-day comparison for clear-sky days has very good agreement in heights between CABAM mixed layer height and AMDAR temperature inversions, with FIGURE 8 Number of 24 hr periods (day SR ) centred at sunrise classified into major ABL-classes: Clear, Cu, and St, based on ALC observations for day (fill colour) and night (striped shading). ALC classes are grouped based on SYNOP categories (which also include a stratocumulus (SC) class). Number of periods in ALC class are normalised by total occurrence of the respective SYNOP class (no.of day SR periods). Agreement (%) is given for the total day SR period and daytime only a mean bias error (MBE) between 63 m in summer and 302 m in winter during daytime. From time series analysis, it is evident that significant scatter of the AMDAR-derived inversion heights is mostly responsible for the discrepancy in overall statistics.
The presence of residual layers poses challenges to Z ML detection as upper boundaries of those layers above ML might be mistaken for Z ML . While CABAM and AMDAR results are very similar in the afternoon when the mixed layer often spans the whole ABL, temperature inversions mark an elevated layer located above the CABAM-derived Z ML during night and the morning transition. The median difference between inversion heights and Z ML is 346 m based on all time periods; however, only 24 m for times >4 hr after sunrise. In addition to the mixed layer, CABAM tracks aerosol layers of the residual layer that might be present above Z ML . During night, the highest of these layers, presumably representing the top of the residual layer, shows good agreement with the AMDAR inversion heights. The additional layers detected by CABAM could be analysed in the future, for example, to evaluate the impact of the residual layer on the morning transition growth rate. The latter is suggested to be increased when residual layers are present (Blay-Carreras et al., 2014). Knowledge on residual layers can also be valuable for the interpretation of pollution concentrations observed within the ABL.
When compared to SYNOP/METAR reports, CABAMderived ABL cloud classes are shown to compare very well during daytime. However, improvements might be possible during the night. Currently, CABAM assigns ABL categories based on 24 hr periods. Results presented suggest it may be beneficial to incorporate observations from the previous day and subsequent night to improve identification of persistent stratiform clouds. Further, application of CABAM to other regions where cloud types are different to southeast England could help to improve the classification scheme. The current algorithm is considered to greatly benefit the analysis of mixed layer height statistics (Kotthaus and Grimmond, 2018), and provides the first ABL classification scheme that distinguishes cloud types solely based on ALC observations.