RESEARCH ARTICLE
Online estimation of colored observation-noise parameters within an ensemble Kalman filtering framework
Naila F. Raboudi,
Boujemaa Ait-El-Fquih,
Ibrahim Hoteit,
Naila F. Raboudi
Division of Physical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
Contribution: Conceptualization, Investigation, Methodology, Software, Validation, Visualization, Writing - original draft
Search for more papers by this authorBoujemaa Ait-El-Fquih
Division of Physical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
Contribution: Conceptualization, Investigation, Methodology, Supervision, Writing - review & editing
Search for more papers by this authorCorresponding Author
Ibrahim Hoteit
Division of Physical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
Correspondence
Ibrahim Hoteit, Division of Physical Science and Engineering, KAUST, Thuwal, Jeddah 23955-6900, Saudi Arabia.
Email: [email protected]
Contribution: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing - review & editing
Search for more papers by this authorNaila F. Raboudi,
Boujemaa Ait-El-Fquih,
Ibrahim Hoteit,
Naila F. Raboudi
Division of Physical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
Contribution: Conceptualization, Investigation, Methodology, Software, Validation, Visualization, Writing - original draft
Search for more papers by this authorBoujemaa Ait-El-Fquih
Division of Physical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
Contribution: Conceptualization, Investigation, Methodology, Supervision, Writing - review & editing
Search for more papers by this authorCorresponding Author
Ibrahim Hoteit
Division of Physical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
Correspondence
Ibrahim Hoteit, Division of Physical Science and Engineering, KAUST, Thuwal, Jeddah 23955-6900, Saudi Arabia.
Email: [email protected]
Contribution: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing - review & editing
Search for more papers by this authorFirst published: 18 May 2023
Abstract
This work addresses the problem of data assimilation in large-dimensional systems with colored observation noise of unknown statistics, a scenario that will become more common in the near future with the deployment of denser observational networks of high spatio-temporal coverage. Here, we are interested in the ensemble Kalman filter (EnKF) framework, which has been derived around a white observation noise assumption. Recently, colored observation-noise aware EnKFs in which the noise was modeled as a first-order autoregressive (AR) model were introduced. This work generalizes the above-mentioned filters to learn the statistics of the AR model further online. We follow the state augmentation approach first to estimate the state and the AR model transfer matrix simultaneously, then the variational Bayesian approach to estimate the AR model noise covariance parameters further. We accordingly derive two filtering EnKF-like algorithms, which estimate those statistics together with the system state. We demonstrate the effectiveness of the proposed colored observation-noise aware filtering schemes and compare their performance based on several numerical experiments conducted with the Lorenz-96 model.
REFERENCES
- Ait-El-Fquih, B., El Gharamti, M. and Hoteit, I. (2016) A Bayesian consistent dual ensemble Kalman filter for state-parameter estimation in subsurface hydrology. Hydrology and Earth System Sciences, 20(8), 3289–3307.
- Ait-El-Fquih, B., Giovannelli, J.F., Paul, N., Girard, A. and Hoteit, I. (2020) Parametric Bayesian estimation of point-like pollution sources of groundwater layers. Signal Processing, 168, 107339.
- Ait-El-Fquih, B. and Hoteit, I. (2015) Fast Kalman-like filtering for large-dimensional linear and Gaussian state-space models. IEEE Transactions on Signal Processing, 63(21), 5853–5867.
- Ait-El-Fquih, B. and Hoteit, I. (2022a) Filtering with one-step-ahead smoothing for efficient data assimilation. In: Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications (Vol. IV). Switzerland: Springer, pp. 69–96.
- Ait-El-Fquih, B. and Hoteit, I. (2022b) Parallel- and cyclic-iterative variational Bayes for fast Kalman filtering in large-dimensions. IEEE Transactions on Signal Processing, 70, 5871–5884.
- Anderson, J.L. (2001) An ensemble adjustment Kalman filter for data assimilation. Monthly Weather Review, 129, 2884–2903.
- Anderson, J.L. and Anderson, S.L. (1999) A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Monthly Weather Review, 127, 2741–2758.
- Bennitt, G., Johnson, H., Weston, P., Jones, J. and Pottiaux, E. (2017) An assessment of ground-based gnss zenith total delay observation errors and their correlations using the met office ukv model. Quarterly Journal of the Royal Meteorological Society, 143(707), 2436–2447.
- Bishop, C.H. and Satterfield, E.A. (2013) Hidden error variance theory. Part i: Exposition and analytic model. Monthly Weather Review, 141(5), 1454–1468.
- Bormann, N. and Bauer, P. (2010) Estimates of spatial and interchannel observation-error characteristics for current sounder radiances for numerical weather prediction. I: Methods and application to atovs data. Quarterly Journal of the Royal Meteorological Society, 136(649), 1036–1050.
- Bormann, N., Hernandez-Carrascal, A., Borde, R., Lutz, H.J., Otkin, J.A. and Wanzong, S. (2014) Atmospheric motion vectors from model simulations. Part i: Methods and characterization as single-level estimates of wind. Journal of Applied Meteorology and Climatology, 53(1), 47–64.
- Bryson, A. and Henrikson, L. (1968) Estimation using sampled data containing sequentially correlated noise. Journal of Spacecraft and Rockets, 5(6), 662–665.
- Bryson, A. and Johansen, D. (1965) Linear filtering for time-varying systems using measurements containing colored noise. IEEE Transactions on Automatic Control, 10(1), 4–10.
- Burgers, G., van Leeuwen, P.J. and Evensen, G. (1998) On the analysis scheme in the ensemble Kalman filter. Monthly Weather Review, 126, 1719–1724.
- Chang, G. (2014) On Kalman filter for linear system with colored measurement noise. Journal of Geodesy, 88(12), 1163–1170.
- Chen, G. (1992) Introduction to random signals and applied kalman filtering, 2nd edn. robert grover brown and patrick yc hwang, wiley, new york, 1992. isbn 0-471-52573-1, 512 pp., 62.95. International Journal of Adaptive Control and Signal Processing, 6(5), 516–518.
- Chen, Y. and Zhang, D. (2006) Data assimilation for transient flow in geologic formations via ensemble kalman filter. Advances in Water Resources, 29(8), 1107–1122.
- Cocucci, T.J., Pulido, M., Lucini, M. and Tandeo, P. (2021) Model error covariance estimation in particle and ensemble kalman filters using an online expectation–maximization algorithm. Quarterly Journal of the Royal Meteorological Society, 147(734), 526–543.
- Dee, D.P. (1995) On-line estimation of error covariance parameters for atmospheric data assimilation. Monthly Weather Review, 123(4), 1128–1145.
- Dee, D.P. and Da Silva, A.M. (1999) Maximum-likelihood estimation of forecast and observation error covariance parameters. Part i: Methodology. Monthly Weather Review, 127(8), 1822–1834.
- DelSole, T. and Yang, X. (2010) State and parameter estimation in stochastic dynamical models. Physica D: Nonlinear Phenomena, 239(18), 1781–1788.
- Desbouvries, F., Petetin, Y. and Ait-El-Fquih, B. (2011) Direct, prediction- and smoothing-based Kalman and particle filter algorithms. Signal Process, 91(8), 2064–2077. https://doi.org/10.1016/j.sigpro.2011.03.013.
- Desroziers, G., Berre, L., Chapnik, B. and Poli, P. (2005) Diagnosis of observation, background and analysis-error statistics in observation space. Quarterly Journal of the Royal Meteorological Society: A journal of the Atmospheric Sciences, Applied Meteorology and Physical Oceanography, 131(613), 3385–3396.
- Desroziers, G. and Ivanov, S. (2001) Diagnosis and adaptive tuning of observation-error parameters in a variational assimilation. Quarterly Journal of the Royal Meteorological Society, 127(574), 1433–1452.
- Dreano, D., Tandeo, P., Pulido, M., Ait-El-Fquih, B., Chonavel, T. and Hoteit, I. (2017) Estimating model-error covariances in nonlinear state-space models using kalman smoothing and the expectation–maximization algorithm. Quarterly Journal of the Royal Meteorological Society, 143(705), 1877–1885.
- Edwards, C.A., Moore, A.M., Hoteit, I. and Cornuelle, B.D. (2015) Regional ocean data assimilation. Annual Review of Marine Science, 7, 21–42.
- Evensen, G. (1994) Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. Journal of Geophysical Research: Oceans, 99(C5), 10143–10162.
- Gazit, R. (1997) Digital tracking filters with high order correlated measurement noise. IEEE Transactions on Aerospace and Electronic Systems, 33(1), 171–177.
- Gelb, A. (1974) Applied optimal estimation. Cambridge, London: MIT Press.
- Gharamti, M., Ait-El-Fquih, B. and Hoteit, I. (2015) An iterative ensemble Kalman filter with one-step-ahead smoothing for state-parameters estimation of contaminant transport models. Journal of Hydrology, 527, 442–457.
- Hamill, T.M. (2001) Interpretation of rank histograms for verifying ensemble forecasts. Monthly Weather Review, 129(3), 550–560.
- Hoteit, I., Luo, X., Bocquet, M., Kohl, A. and Ait-El-Fquih, B. (2018) New frontiers in operational oceanography, ch. 17. In: Data assimilation in oceanography: current status and new directions. Tallahassee, Florida: GODAE Ocean View, pp. 465–512.
- Hoteit, I., Pham, D.T. and Blum, J. (2002) A simplified reduced order Kalman filtering and application to altimetric data assimilation in Tropical Pacific. Journal of Marine Systems, 36, 101–127.
-
Hoteit, I., Pham, D.T., Gharamti, M. and Luo, X. (2015) Mitigating observation perturbation sampling errors in the stochastic EnKF. Monthly Weather Review, 143(7), 2918–2936.
- Houtekamer, P.L. and Mitchell, H.L. (1998) Data assimilation using an ensemble Kalman filter technique. Monthly Weather Review, 126, 796–811.
- Jaakkola, T.S. and Jordan, M.I. (2000) Bayesian parameter estimation via variational methods. Statistics and Computing, 10(1), 25–37.
- Jardak, M. and Talagrand, O. (2018) Ensemble variational assimilation as a probabilistic estimator–part 1: The linear and weak non-linear case. Nonlinear Processes in Geophysics, 25(3), 565–587.
- Kalman, R. (1960) A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35–45.
- Karimi, A. and Paul, M.R. (2010) Extensive chaos in the lorenz-96 model. Chaos: An Interdisciplinary Journal of Nonlinear Science, 20(4), 043105.
- Khaki, M., Ait-El-Fquih, B., Hoteit, I., Forootan, E., Awange, J. and Kuhn, M. (2018) Unsupervised ensemble kalman filtering with an uncertain constraint for land hydrological data assimilation. Journal of Hydrology, 564, 175–190.
- Kotsuki, S., Miyoshi, T., Terasaki, K., Lien, G.Y. and Kalnay, E. (2017) Assimilating the global satellite mapping of precipitation data with the nonhydrostatic icosahedral atmospheric model (nicam). Journal of Geophysical Research: Atmospheres, 122(2), 631–650.
- Li, H., Kalnay, E. and Miyoshi, T. (2009) Simultaneous estimation of covariance inflation and observation errors within an ensemble kalman filter. Quarterly Journal of the Royal Meteorological Society: A Journal of the Atmospheric Sciences, Applied Meteorology and Physical Oceanography, 135(639), 523–533.
- Li, J. and Roy, D.P. (2017) A global analysis of sentinel-2a, sentinel-2b and landsat-8 data revisit intervals and implications for terrestrial monitoring. Remote Sensing, 9(9), 902.
- Liu, B., Ait-El-Fquih, B. and Hoteit, I. (2016) Efficient kernel-based ensemble Gaussian mixture filtering. Monthly Weather Review, 144(2), 781–800.
- Lorenz, E.N. and Emanuel, K.A. (1998) Optimal sites for supplementary weather observations: Simulation with a small model. Journal of the Atmospheric Sciences, 55(3), 399–414.
- Miyoshi, T. (2011) The gaussian approach to adaptive covariance inflation and its implementation with the local ensemble transform kalman filter. Monthly Weather Review, 139(5), 1519–1535.
- Miyoshi, T., Kalnay, E. and Li, H. (2013) Estimating and including observation-error correlations in data assimilation. Inverse Problems in Science and Engineering, 21(3), 387–398.
- Moradkhani, H., Sorooshian, S., Gupta, H.V. and Houser, P.R. (2005) Dual state–parameter estimation of hydrological models using ensemble Kalman filter. Advances in Water Resources, 28(2), 135–147.
- Petovello, M.G., O'Keefe, K., Lachapelle, G. and Cannon, M.E. (2009) Consideration of time-correlated errors in a kalman filter applicable to gnss. Journal of Geodesy, 83(1), 51–56.
- Pham, D.T. (2001) Stochastic methods for sequential data assimilation in strongly nonlinear systems. Monthly Weather Review, 129, 1194–1207.
- Pulido, M., Tandeo, P., Bocquet, M., Carrassi, A. and Lucini, M. (2018) Stochastic parameterization identification using ensemble kalman filtering combined with maximum likelihood methods. Tellus A: Dynamic Meteorology and Oceanography, 70(1),
1–17.
- Raboudi, N.F., Ait-El-Fquih, B. and Hoteit, I. (2018) Ensemble kalman filtering with one-step-ahead smoothing. Monthly Weather Review, 146(2), 561–581.
- Raboudi, N.F., Ait-El-Fquih, B., Ombao, H. and Hoteit, I. (2021) Ensemble kalman filtering with coloured observation noise. Quarterly Journal of the Royal Meteorological Society, 147(741), 4408–4424.
- Ruiz, J.J., Pulido, M. and Miyoshi, T. (2013) Estimating model parameters with ensemble-based data assimilation: A review. Journal of the Meteorological Society of Japan. Series II, 91(2), 79–99.
- Sakov, P. and Bertino, L. (2011) Relation between two common localisation methods for the EnKF. Computational Geosciences, 15(2), 225–237.
- Scheffler, G., Ruiz, J. and Pulido, M. (2019) Inference of stochastic parametrizations for model error treatment using nested ensemble kalman filters. Quarterly Journal of the Royal Meteorological Society, 145(722), 2028–2045.
- Simon, D. (2006) Optimal state estimation: Kalman, h infinity, and nonlinear approaches. Hoboken: John Wiley & Sons.
-
Šmídl, V. and Quinn, A. (2006) The variational Bayes method in signal processing. Verlag, Berlin: Springer Science & Business Media.
- Stroud, J.R. and Bengtsson, T. (2007) Sequential state and variance estimation within the ensemble kalman filter. Monthly Weather Review, 135(9), 3194–3208.
- Stroud, J.R., Katzfuss, M. and Wikle, C.K. (2018) A Bayesian adaptive ensemble Kalman filter for sequential state and parameter estimation. Monthly Weather Review, 146(1), 373–386.
- Sun, S. and Deng, Z. (2004) Optimal fusion kalman filter for systems with colored measurement noises. In: Fifth World Congress on Intelligent Control and Automation (IEEE Cat. No. 04EX788), vol. 2. IEEE. pp. 1562–1566.
- Talagrand, O. (1999) Evaluation of probabilistic prediction systems. In: Workshop proceedings" Workshop on predictability", 20-22 October 1997, ECMWF, Reading, UK.
- Tandeo, P., Ailliot, P., Bocquet, M., Carrassi, A., Miyoshi, T., Pulido, M. and Zhen, Y. (2020) A review of innovation-based methods to jointly estimate model and observation error covariance matrices in ensemble data assimilation. Monthly Weather Review, 148(10), 3973–3994.
- Tandeo, P., Pulido, M. and Lott, F. (2015) Offline parameter estimation using enkf and maximum likelihood error covariance estimates: application to a subgrid-scale orography parametrization. Quarterly journal of the royal meteorological society, 141(687), 383–395.
- Tippett, M.K., Anderson, J.L., Bishop, C.H., Hamill, T.M. and Whitaker, J.S. (2003) Ensemble square root filters. Monthly Weather Review, 131, 1485–1490.
- Torres, R., Snoeij, P., Geudtner, D., Bibby, D., Davidson, M., Attema, E., Potin, P., Rommen, B., Floury, N. and Brown, M. (2012) Gmes sentinel-1 mission. Remote Sensing of Environment, 120, 9–24.
- Ueno, G. and Nakamura, N. (2016) Bayesian estimation of the observation-error covariance matrix in ensemble-based filters. Quarterly Journal of the Royal Meteorological Society, 142(698), 2055–2080.
- Wang, K., Li, Y. and Rizos, C. (2012) Practical approaches to kalman filtering with time-correlated measurement errors. IEEE Transactions on Aerospace and Electronic Systems, 48(2), 1669–1681.
- Wen, X.H. and Chen, W.H. (2005) Real-time reservoir model updating using ensemble kalman filter. In: SPE reservoir simulation symposium.
- Weston, P., Bell, W. and Eyre, J. (2014) Accounting for correlated error in the assimilation of high-resolution sounder data. Quarterly Journal of the Royal Meteorological Society, 140(685), 2420–2429.
- Whitaker, J.S. and Hamill, T.M. (2002) Ensemble data assimilation without perturbed observations. Monthly Weather Review, 130, 1913–1924.
