Sub‐seasonal predictability of the 2017–2018 Southern Hemisphere tropical cyclone season

Multi‐week forecasts of tropical cyclone genesis are evaluated for the 2017–2018 Southern Hemisphere cyclone season. These forecasts were created with the Australian Bureau of Meteorology's new seasonal forecast system, ACCESS‐S. Forecast performance over the season showed positive Brier Skill Scores with good forecast reliability and improved skill over climatology out to week 3. ACCESS‐S provided guidance of the formation of major cyclone events during the season at more than 2 weeks lead time, including Cyclone Gita in the South Pacific which later devastated Tonga, and the landfall of Cyclone Hilda in western Australia.


| INTRODUCTION
The Australian Bureau of Meteorology's new seasonal forecast model (ACCESS-S) became operational in August 2018. This system is based on the UK Met Office Global Seasonal forecast system version 5 (GloSea5; MacLachlan et al., 2015) and differs only in the initialization of the atmosphere and number of ensembles (Hudson et al., 2017). Prior to becoming operational, ACCESS-S was run in testing mode providing forecasts 2 days behind real time from December 2017 until April 2018. The Okubo-Weiss-Zeta parameter (OWZ) tropical cyclone (TC) detection scheme (Tory et al., 2013) was used to compute the location of possible cyclones based on ACCESS-S atmospheric fields.
The OWZ tracker was created to process CMIP5 models for cyclone projections under future climate scenarios. Instead of identifying storms using MSLP and vorticity criteria (which are grid dependent and often not suitable for coarse-resolution coupled models) it searches for conditions conducive to cyclogenesis. These conditions are identified with following atmospheric parameters: solid body rotation at 850 and 500 hPa, relative humidity at 950 and 700 hPa, specific humidity at 950 hPa and low wind shear between 850 and 200 hPa. Points which satisfy certain thresholds for these parameters are identified and trajectories are computed which link each location in time.
Previous work by Camp et al. (2018) showed the ACCESS-S system, combined with the OWZ tracking scheme, had skill in forecasting TC formation throughout the 1990-2012 hindcast period. When using a crossvalidated calibration in overlapping regions of 15 × 20 resolution, skillful forecasts out to week 5 were possible.
The calibration factors computed in that study were applied to real-time ACCESS-S forecasts issued during the 2017-2018 cyclone season. Here we evaluate the skill of the real-time forecasts to predict TC formation during the season. Section 2 provides an overview of the 2017-2018 cyclone season. Section 3 details forecast skill and examines the ability of ACCESS-S to predict TC formation out to week 4. Section 4 provides case studies which examine the forecast predictability of significant TC events during the 2017-2018 cyclone season.

| THE 2017-2018 SOUTHERN HEMISPHERE CYCLONE SEASON
The 2017-2018 Southern Hemisphere cyclone season was classified as average in terms of cyclone numbers. Eight cyclones formed in the south Indian Ocean (between 30 E and 90 E), while nine cyclones formed in the Australian region (90 -160 E) and five in the South Pacific Ocean (160 -240 E). 1 Observed TC data from the U.S. Navy's Joint Typhoon Warning Center (JTWC) operational best tracks, provided through the University Corporation for Atmospheric Research (UCAR, 2018), was used to provide verifying data for position and intensity of TCs every 6 hr. This maintained consistency with Camp et al. (2018), which used JTWC best-track data to generate the calibration factors at each forecast lead time across the Southern Hemisphere.

| FORECAST SKILL
Forecast cyclone tracks were computed using the OWZ tracker on all available ACCESS-S forecasts. Tracks were initially computed weekly (and sometimes twice weekly) from November 22, 2017 to coincide with the release of the European Centre for Medium-Range Weather Forecasts (ECMWF) multi-week tropical cyclone strike probability product (Vitart et al., 2012). In response to interest from forecasters, tracks were computed daily from March 10, 2018 until April 4, 2018.
While the hindcast used in Camp et al. (2018) featured 11 ensemble members per forecast, 33 ensemble members were used for real-time forecasts and tracks using the larger ensemble were available from March 13, 2018.
Reliability plots and Brier skill scores (BSSs; see http:// www.cawcr.gov.au/projects/verification/#BSS) for the period from November 22, 2017 to April 30, 2018 are computed on overlapping regions replicating the method used in Camp et al. (2018). Scores were computed for the raw and calibrated probabilities, the latter using calibration scheme denoted as CAL2-CV in Camp et al. (2018). In this scheme, the forecast probabilities for each hindcast year are scaled by the ratio of the observed climate mean to the forecast mean over the entire hindcast excluding the year of the forecast (leave-one-out-cross-validation). These scaled probabilities are computed in each 15 × 20 region for each forecast lead time. These ratios were then applied to the real time forecasts for each overlapping region.
In addition to raw track probabilities, scores were computed by applying a wind speed threshold to the forecast tracks only. The OWZ tracker outputs 850 hPa wind speed along each storm track. When the OWZ tracker using its default thresholds 2 was applied to the ACCESS-S1 hindcast, too many storms were generated, especially at longer lead times. Applying the wind speed threshold significantly reduced the amount of forecast storms. Subsequent analysis found a threshold of 14 m/s gave the best improvement to BSSs from weeks 2-4. Storms whose 850-hPa wind speed exceeded 14 m/s at least once during their lives were retained for BSS computation. Storms which failed to reach this threshold were discarded.
Note that the calibrated probabilities used the raw forecast tracks as inputs. Attempting calibration using only hindcast tracks which exceeded the 14 m/s threshold gave no improvement in skill. It is presumed this is due to a decrease in sample size (due to fewer storms being generated), but further investigation is warranted.
BSSs for the forecast period from November 22, 2017 to April 30, 2018 are shown in Table 1. As the initial hindcast calibration in Camp et al. (2018) was only performed between November-February, additional scores were computed for the November-February period as the calibration factors may not be applicable at the tail-end of the cyclone season. This shorter period only contained 16 forecast dates (compared to 38 forecasts between November and April). Corresponding reliability plots are shown in Figure 1 for both forecast periods and verification regions.
For the full forecast period (November-April) the use of the wind speed threshold provides the best BSSs, improving reliability and resolution at all forecast lead times. Using the calibrated probabilities produces only a small improvement compared to the raw probabilities (and a slight degradation for week 1 forecasts). For the shorter forecast period (November-February) which corresponds to the period used in the initial hindcast, applying the wind speed threshold provides the best results for the first and second week forecasts. In this instance, the calibrated probabilities provide the best skill in weeks 3 and 4. Once again, the raw and calibrated probabilities in week 1 are very poor. Visual inspection of each forecast showed that the OWZ tracker generated many false alarms inside the first 7 days, but these were often storms with low maximum winds and hence were removed from the analysis when the wind speed threshold was applied.
Reliability plots in Figure 1 show that for the full forecast period (November-April) the use of the wind speed threshold consistently improved the reliability of the forecasts. The use of calibration factors from the initial hindcast improves the resolution at the expense of sharpness compared to the raw tracks, especially for longer lead times. However, the raw tracks using the wind speed threshold still show the best results.
Reliability plots for the shorter forecast period (November-February) suggest that an insufficient number of samples for Days 4-7 (based on empty forecast probability bins) could be a factor in the low BSS for this forecast lead time. At longer lead times the calibrated probabilities provide large improvements in reliability and resolution compared to the raw tracks, and to the raw tracks with wind speed threshold for longer lead times. However, these improvements come with a cost of reduced sharpness (i.e., the ability to predict all ranges of forecast probabilities, particularly high probability events has been reduced). Reliability plots for two forecast periods using three different processing methods (raw tracks, raw tracks with wind speed threshold and calibrated probabilities) for overlapping 15 × 20 boxes in the Southern Hemisphere. Each column represents different lead times (from left to right: [8][9][10][11][12][13][14][15][16][17][18][19][20][21] making landfall in Madagascar on January 5. Figure 2 shows the ACCESS-S forecasts issued on December 12 and 18 leads times of 14-21 Days (week 3) and 8-14 Days (week 2). The valid times of these forecasts range from December 26 to January 2.
The observed tracks of cyclones in the forecast period are shown along with the ACCESS-S raw tracks. Forecast tracks that fail to reach the 14 m/s threshold are plotted with dotted lines. The raw tracks and calibrated probabilities consistently show a strong probability of TC formation in the areas corresponding to the actual formation of Hilda and Ava. Probability anomalies are also plotted. These are computed by subtracting the mean cyclone climatology of 0.20 for weeks 2 and 3 calibrated forecasts (taken from figure 1 (a)) from the real-time forecast probabilities. Note that Ava was declared at 12Z on the January 2 so its position is labeled "INVEST" in Figure 2. The forecast issued on December 18 was particularly emphatic with regard to Hilda, with 10 out of 11 ensemble members forecasting TC landfall over northwest Australia. The forecast also gave guidance of potential TC activity near the dateline. This was related to Depression 04F, which was active from December 20-26 between Noumea and Fiji, but it did not intensify into a cyclone.
In this example, the calibrated probabilities increase the skill of the forecast guidance. For week 3, probabilities are decreased throughout the region, which provides better guidance for the false alarms in the Pacific. Useful guidance still exists though in the Indian Ocean and off northwest Australia where absolute calibrated probabilities remain between 30 and 50%, which is well above the background calibrated forecast climatology value of 20%. Note the November-February climatology of observed storms in week 3 for these overlapping regions is 15% according to Camp et al. (2018). For the second week forecast, applying the calibration decreases probabilities across the region, but increases them between 100 E and 130 E. This decreases guidance in the case of Ava, but gives better guidance for Hilda as well the false alarm in the Pacific.  weakened to a tropical low on February 9. Cyclone Gita formed in the South Pacific on February 9 and made landfall as a Category 4 cyclone in Tonga 2 days later. Gita was the strongest cyclone of the South Pacific season and its track exhibited significant recurvature. Figure 3 shows the ACCESS-S forecasts issued on January 22 and 29 for week 3 and week 2 forecasts covering the period February 6-12, 2018.
This example examines the model's ability to forecast the formation of Gita while predicting behavior of Cebile which was already active in the model's initial conditions. The ACCESS-S raw guidance on 22 January incorrectly forecast continuing cyclone activity in the Coral Sea while also providing good guidance of storm activity between 160 W and 180 W. False alarms were generated close to Madagascar and east of the dateline. Good guidance for Gita was still provided, with some ensembles correctly forecasting changes in track curvature. ACCESS-S gave some guidance for the continued progress of Cebile in the Indian Ocean but suggested it would move closer to Madagascar.
In this instance the calibrated probabilities for week 3 correctly reduced the chance of TC formation near Madagascar and in the Coral Sea; however, they also reduced the probability of TC formation in the South Pacific Ocean, coinciding with the observed location of Cyclone Gita. The calibrated forecasts for week 2 (initialized January 29, 2018) improved the guidance by maintaining the chance of TC formation near Fiji and Tonga at 40-50%, corresponding with formation of Cyclone Gita, while reducing chances in the rest of the region to slightly above climatology. (March 13-19, 2018) The Southern Hemisphere cyclone season became very active in March. Cyclones Eliakim and Linda both formed on March 13. Linda was a short-lived storm and weakened to below cyclone strength within 24 hrs while Eliakim made landfall in Madagascar on March 16 causing widespread flooding and damage. Cyclone Marcus formed north of Darwin on March 16 and moved westward into the Indian Ocean, intensifying into a Category 5 cyclone. Figure 4 shows the ACCESS-S forecasts issued on February 26 and March 3 for weeks 3 and 2, respectively, covering the period March 13-19, 2018.

| Cyclones Eliakim, Linda and Marcus
ACCESS-S forecasts initialized on February 26 correctly predicted high chances of TC formation near Madagascar, the Coral Sea and throughout northern Australia, although false alarms were present around 90 E. The later forecast (initialized on March 5) did not improve forecast guidance. While TC activity near Madagascar and the Coral Sea was still predicted to be higher than climatology, the probabilities reduced compared to the earlier forecast. This trend could be considered correct in the case of Linda (which lasted less than 24 hrs) but not in the case of Eliakim. The chances of TC formation near Darwin remained the same in the later forecast. At the same time, the probability of storm formation at 90 E incorrectly showed a significant increase.
The effect of forecast calibration was mixed in this example. For the week 3 forecast, calibration reduced probabilities of TC formation throughout the region. Only the Coral Sea gave probabilities significantly above climatology. For the second-week forecast, the calibration correctly increased chances of TC formation near Darwin while decreasing chances in the Coral Sea. However, it incorrectly reduced chances near Madagascar. The false alarms at 90 E were not affected by the forecast calibration.

| CONCLUSIONS
The Australian Bureau of Meteorology's new ACCESS-S seasonal forecast system shows useful multi-week forecast skill in predicting TC formation in the Southern Hemisphere when using the OWZ tracking scheme. A wind speed threshold was shown to significantly increase skill of the raw tracks, especially at longer lead times. Results shown here using calibrated probabilities are based on a preliminary hindcast with start dates on November 1, December 1, January 1 and February 1 throughout 1990-2012. The calibrated probabilities gave some improvement to forecasts issued between November and February 2017-2018 but not through the entire forecast period from November-April.
Comparisons with the European Center for Medium Range Weather Forecasting (ECMWF) multi-week cyclone forecasts are shown in Table 2. Scores using their Ensemble Prediction System (EPS) are approximate, based on Vitart et al. (2012, fig. 10). The Southern Hemisphere scores for the EPS were computed using the same method (overlapping 15 × 20 boxes) as used for ACCESS-S (Camp et al., 2018). However, the ECMWF scores are computed from November-April, whereas our scores (using the initial hindcast) are limited to November-February. Table 2 again shows the importance of using the wind speed threshold when applying the OWZ tracker to ACCESS-S. The calibration scheme still maintains skill at longer lead times. The ACCESS-S scores based on the initial hindcast give some indication of its operational performance with the following caveats as the initial hindcast: only spanned November-February, used only 11 ensemble members, and used ERA-Interim initial conditions (operational ACCESS-S forecasts are initialized with the Bureau's global forecast suite ACCESS-G).
The scores for the ECMWF EPS system were computed in 2012 and would not accurately represent the current operational system. While the ACCESS-S scores are lower than the equivalent ECMWF scores, they are still skillful. They are also generated from a completely different atmospheric modeling scheme (the UK Met Office's Unified Model 3 c/f the ECMWF Integrated Forecast System 4 ) and using different tracking schemes (OWZ c/f ECMWF tracking scheme outlined in Vitart et al., 2007). So the ACCESS-S system provides another independent source of forecast guidance.
After the cyclone season ended, the full ACCESS-S hindcast was completed, featuring four start dates for every calendar month throughout 1990-2012. Using the complete hindcast, it may be possible to perform forecast calibration on a sub-seasonal or even monthly basis. This will be subsequently tested on the full 33 member forecasts that were archived almost daily throughout the 2017-2018 cyclone season. The daily variability of each multi-week forecast will also be examined with these archived forecasts.
The dependency of these results on the choice of tracking scheme is still a matter of investigation. The OWZ tracker focuses on identifying regions conducive to cyclogenesis. While this is suitable to long-range cyclone forecasts, it can produce false alarms where moist upper level model fields are not associated with deep convection and MSLP minima. Work to implement the ECMWF (Vitart et al., 2012) and UK Met Office  tropical cyclone trackers on ACCESS-S are ongoing.
Further testing will also include analysis of the Madden-Julian Oscillation (MJO) to see if ACCESS-S can provide skillful multi-week guidance of TC formation in periods when there is no strong MJO signal.