Please wait a minute...
img img
Adv. Search
Advances in Earth Science  2014, Vol. 29 Issue (3): 327-335    DOI: 10.11867/j.issn.1001-8166.2014.03.0327
Orginal Article     
Progress on Verification Methods of Numerical Weather Prediction
Pan Liujie1, Zhang Hongfang2, Wang Jianpeng1
1.Shaanxi Meteorological Observatory, Xi’an 710014, China;
2.Shaanxi Meteorological Service Centre, Xi’an 710014, China
Download:  HTML  PDF (1134KB) 
Export:  BibTeX | EndNote (RIS)      

The numerical weather prediction verification is a key step to improving applied numerical models. In recent years, the concepts used in model verification have been updated constantly and the model verification approaches fit for different prediction products and meeting the needs of different customers have proposed continuously. First, this research gives a brief overview of traditional model verification methods based on contingency table. Second, the spatial verification technology along with high resolution numerical prediction is illustrated. According to different purposes, the spatial verification methods can mostly be classified into two general categories: filtering methods and displacement methods. The filtering methods can be further delineated into neighborhood and scale separation, and the displacement methods can be divided into features-based ones and field deformations. Third, the probability distribution function of ensemble prediction sample members, the similar degrees between probability distribution function of ensemble prediction and the observation distribution function, the probability of an event occurring and such ensemble prediction verification methods are described. Finally, the fields where spatial verification technology and ensemble prediction verification methods can be used are analyzed and some problems concerning model verification and the direction in which these technologies will go are discussed.

Key words:  Ensemble forecasts      Probability forecasts      Spatial forecast verification     
Received:  07 January 2014      Published:  10 March 2014
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
Articles by authors
Zhang Hongfang
Wang Jianpeng
Pan Liujie

Cite this article: 

Pan Liujie, Zhang Hongfang, Wang Jianpeng. Progress on Verification Methods of Numerical Weather Prediction. Advances in Earth Science, 2014, 29(3): 327-335.

URL:     OR

[1] Chen Haishan,Sun Zhaobo. Design of a comprehensive land surface model and its validation Part II: Model validation[J].Chinese Journal of Atmospheric Sciences, 2005, 29(2):272-282.[陈海山,孙照渤.陆面模式CLSM的设计及性能检验II.模式检验[J].大气科学,2005,29(2):272-282.]
[2] Pan Liujie, Zhang Hongfang, Zhu Weijun, et al. Forecast performance verification of the ECMWF model over the Northeast Hemisphere[J].Climatic and Environmental Research, 2013, 18(1):112-123.[潘留杰, 张宏芳, 朱伟军, 等. ECMWF模式对东北半球气象要素场预报能力的检验[J]. 气候与环境研究, 2013, 18 (1) :112-123.]
[3] Wilks D S. Statistical Methods in the Atmospheric Sciences[M]. Oxford,UK:Academic Press, 2005: 234-251.
[4] Gilleland E, David A, Brown G. Intercomparison of spatial forecast verification methods[J]. Weather and Forecasting, 2009, 24(6):1 416-1 429.
[5] Baldwin M E, Kain J S. Sensitivity of several performance measures to displacement error, bias, and event frequency[J].Weather and Forecasting, 2006, 21(5):636-648.
[6] Casati B, Wilson L J, Stephenson D B, et al. Review forecast verification: Current status and future directions[J].Meteorological Applications, 2008, 15(3):3-18.
[7] Zepeda A J, Foufoula G E, Droegemeier K.Space-time rainfall organization and its role in validating quantitative precipitation forecasts[J]. Journal of Geophysical Research, 2000, 105(D8):10 129-10 146.
[8] Tustison B, Foufoula-Georgiou E, Harris D.Scale-recursive estimation for multisensor quantitative precipitation forecast verification: A preliminary assessment[J].Journal of Geophysical Research, 2003, 108(3):11 775-11 784.
[9] Casati B, Ross G, Stephenson D B. A new intensity-scale approach for the verification of spatial precipitation forecasts[J]. Meteorological Applications, 2006, 11:141-154.
[10] Davis C, Brown B, Bullock R. Object-based verification of precipitation forecasts. Part I: Methods and application to mesoscale rain areas[J]. Monthly Weather Review, 2006, 134(10):1 772-1 784.
[11] Davis C A, Brown B G, Bullock R G. Object-based verification of precipitation forecasts. Part II: Application to convective rain systems[J].Monthly Weather Review, 2006, 134(7):1 785-1 795.
[12] Liu Jianguo, Xie Zhenghui, Zhao Linna, et al. BMA probabilistic forecasting for the 24-h TIGGE multi-model ensemble forecasts of surface air temperature[J]. Chinese Journal of Atmospheric Sciences, 2013, 37(1):43-53.[刘建国,谢正辉,赵琳娜,等.基于多模式集合的小时气温概率预报[J]. 大气科学, 2013, 37(1):43-53.]
[13] Zhang Hongfang, Pan Liujie. A comparison analysis of precipitation forecast ability between ECMWF and Japan[J]. Meteorological Monthly,2014,in press.[张宏芳,潘留杰. ECMWF、日本高分辨率模式降水预报能力的对比分析[J].气象,2014,待刊.]
[14] Li Weiping, Liu Xin, Nie Suping, et al. Comparative studies of snow cover parameterization schemes used in climate models[J]. Advances in Earth Science, 2009, 24(5): 512-522.[李伟平,刘新,聂肃平, 等. 气候模式中积雪覆盖率参数化方案的对比研究[J]. 地球科学进展, 2009, 24(5): 512-522.]
[15] Wang Chenghai. Evaluating the progress of the CMIP and its application prospect in China[J]. Advances in Earth Science, 2009, 24(5): 461-468.[王澄海. CMIP研究计划的进展及其在中国地区的检验和应用前景[J]. 地球科学进展, 2009, 24(5): 461-468.]
[16] Kuang Xueyuan, Liu Jian, Wang Hongli, et al. Comparison of simulated and reconstructed precipitation in China during the last millennium[J]. Advances in Earth Science,2009, 24(2): 159-171.[况雪源,刘健,王红丽, 等. 近千年来中国区域降水模拟与重建资料的对比分析[J]. 地球科学进展, 2009, 24(2): 159-171.]
[17] Wang Bin, Xie Xin, Li Lijuan. A review on aspects of climate simulation assessment[J]. Advances in Atmospheric Sciences, 2009, 26(4):736-747.
[18] Liu Yanhua,Zhang Shuwen,Mao Lu, et al. An evaluation of simulated and estimated land surface states with two different models[J]. Advances in Earth Science, 2013,28(8):913-922.[刘彦华,张述文,毛璐, 等. 评估两类模式对陆面状态的模拟和估算[J].地球科学进展,2013,28(8):913-922.]
[19] Brownlee K A. Statistical Theory and Methodology in Science and Engineering[M]. New York: John Wiley & Sons Wiley, 1965:26-30.
[20] Schaefer J T. The critical success index as an indicator of warning skill[J].Weather and Forecasting, 1990, 5(4):570-575.
[21] Mason I. Dependence of the critical success index on sample climate and threshold probability[J]. Australian Meteorological Magazine, 1989, 37:75-81.
[22] Doswell C A, Jones R D, Keller D L. On summary measures of skill in rare event forecasting based on contingency tables[J]. Weather and Forecasting, 1990, 5(4):576-585.
[23] Brier G W. Verification of forecasts expressed in terms of probability[J].Monthly Weather Review, 1950, 78(1):1-3.
[24] Stephenson D B. Use of the “odds ratio” for diagnosing forecast skill[J].Weather and Forecasting, 2000, 15(7):221-232.
[25] Brill K, Mesinger F. Applying a general analytic method for assessing bias sensitivity to bias—Adjusted threat and equitable threat scores[J]. Weather and Forecasting, 2009, 24(6):1 748-1 754.
[26] Ding Jincai. Review of weather prediction verifying techniques[J]. Journal of Nanjin Institute of Meteorology, 1995,18(1):143-150.[丁金才. 天气预报评分方法评述[J]. 南京气象学院学报,1995, 18(1):143-150.]
[27] Jolliffe I T, Stephenson D B. Forecast Verification: A Practitioner’s Guide in Atmospheric Science[M]. Chichester: Wiley and Sons, 2003.
[28] Hoffman R N, Liu Z, Louis J, et al. Distortion representation of forecast errors[J].Monthly Weather Review, 1995, 123(5):2 758-2 770.
[29] Ebert E, McBride J L.Verification of precipitation in weather systems: Determination of systematic errors[J]. Journal of Hydrology, 2000, 239(6):179-202.
[30] Ebert E, McBride J L. Neighborhood verification: A strategy for rewarding close forecasts[J].Weather and Forecasting, 2009, 18(4):632-638.
[31] Baldwin M E, Kain J S. Sensitivity of several performance measures to displacement error, bias, and event frequency[J].Weather and Forecasting, 2006, 21(5):636-648.
[32] Marzbzn C, Sandgathe S. Cluster analysis for object-oriented verification of fields: A variation[J].Monthly Weather Review, 2008, 136(8):1 013-1 025.
[33] Micheas A C, Fox N I, Lack S A. Cell identification and verification of QPF ensembles using shape analysis techniques[J].Journal of Hydrology, 2007, 343(4):105-116.
[34] Lack S, Limpert G L, Fox N I. An object-oriented multiscale verification scheme[J].Monthly Weather Review, 2009, 131(8):1 001-1 005.
[35] Davis C A, Brown B G, Bullock, et al. The method for object-based diagnostic evaluation (MODE) applied to numerical forecasts from the 2005 NSSLSPC spring program[J].Weather and Forecasting, 2009, 24(8):1 252-1 267.
[36] Sock S, Tribbia J, Rakovec J. Object-based analysis and verification of WRF model precipitation in the low- and midlatitude Pacific Ocean[J].Monthly Weather Review, 2010, 138(11):4 561-4 575.
[37] Johnson A, Wang X, Kong F, et al. Object-based evaluation of the impact of horizontal grid spacing on convection—Allowing forecasts[J].Monthly Weather Review, 2013, 141(8):3 413-3 425.
[38] Briggs W M, Levine R A. Wavelets and field forecast verification[J]. Monthly Weather Review, 1997, 125(3):1 329-1 341.
[39] Harris D, Foufoula G E, Droegemeier K, et al. Multiscale statistical properties of a high-resolution precipitation forecast[J]. Journal of Hydrometeorology, 2001, 2: 406-418.
[40] Roberts N M, Lean H W. Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events[J]. Monthly Weather Review, 2008, 136(1):78-97.
[41] Wang N, Lu C. Two-dimensional continuous wavelet analysis and its application to meteorological data[J].Journal of Atmospheric and Oceanic Technology, 2010, 27(2):652-666.
[42] Liu Y, Brown J, Demargne J, et al. A wavelet based approach to assessing timing errors in hydrologic predictions[J]. Journal of Hydrology, 2011, 397: 210-224.
[43] Mittermaier M P. Using an intensity-scale technique to assess the added benefit of high-resolution model precipitation forecasts[J]. Atmospheric Science Letters, 2006, 7: 35-42.
[44] Casati B, Wilson L J. A new spatial-scale decomposition of the brier score: Application to the verification of lightning probability[J]. Monthly Weather Review, 2007, 133(4):81-101.
[45] Casati B. New developments of the intensity-scale technique within the spatial verification methods intercomparison project[J]. Weather and Forecasting, 2010, 138(8):121-128.
[46] Theis S E, Hense A, Damrath U. Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach[J]. Meteorological Applications, 2005, 12: 257-268.
[47] Roberts N M, Lean H W. Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events[J]. Monthly Weather Review, 2007, 136(8):78-97.
[48] Ebert E. Fuzzy verification of high-resolution girded forecasts: A review and proposed framework[J]. Meteorological Applications, 2010, 15: 51-64.
[49] Mittermaier M, Roberts N. Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score[J].Weather and Forecasting, 2010, 25(5):343-354.
[50] Weusthoff T, Ament F, Aroagaus M, et al. Assessing the benefits of convection permitting models by neighborhood verification: Examples from MAP D-PHASE[J].Monthly Weather Review, 2010, 138(8):3 418-3 433.
[51] Gilleland E, David A, Brown B G, et al. Verifying forecasts spatially[J]. American Meteological Society, 2010, 8:1 365-1 373.
[52] Alexander G D, Weinman J A, Karyampudi V M, et al. The effect of assimilating rain ratesv derived from satellites and lightning on forecasts of the 1993 Superstorm[J].Monthly Weather Review, 1999, 127(6):1 433-1 457.
[53] Nehrkorn T, Hoffman R, Grassotti C, et al. Feature calibration and alignment to represent model forecast errors: Empirical regularization[J]. Quarterly Journal of the Royal Meteorological Society, 2003, 129: 195-218.
[54] Keil C, Craig G C.A displacement-based error measure applied in a regional ensemble forecasting system[J].Monthly Weather Review,2007, 135(9):3 248-3 259.
[55] Keil C, Craig G C. A displacement and amplitude score employing an optical flow technique[J].Weather and Forecasting,2009, 24(6):1 297-1 308.
[56] Marzban C, Sandgathe S, Lyons H, et al. Three spatial verification techniques: Cluster analysis, variogram, and optical flow[J].Weather and Forecasting, 2009, 24(11):1 457-1 471.
[57] Marzban C, Sandgathe S. Optical flow for verification[J].Weather and Forecasting, 2010, 25(10):1 479-1 494.
[58] Mesinger F. Bias adjusted precipitation threat scores[J]. Advances in Geosciences, 2008, 16: 137-142.
[59] Lakshmanan V, Kain J. A gaussian mixture model approach to forecast verification[J].Weather and Forecasting, 2010, 25(6):908-920.
[60] Venugopal V, Basu S, Georgiou E F. A new metric for comparing precipitation patterns with an application to ensemble forecasts[J]. Journal of Geophysical Research, 2005, 110: D08111,doi: 10.1029/2004JD005395.
[61] Gilleland E. Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test[J].Monthly Weather Review, 2013, 141(1):340-355.
[62] Du Jun, Chen Jing. The corner stone in facilitating the transition from deterministic to probabilistic forecasts—Ensemble forecasting and its impact on numerical weather prediction[J].Meteorological Monthly, 2010, 36(11):1-11.[杜均,陈静.单一值预报向概率预报转变的基础——谈谈集合预报及其带来的变革[J].气象, 2010, 36(11):1-11.]
[63] Murphy A H. A new vector partition of the probability score[J].Journal of Applied Meteorology, 1973, 12: 595-600.
[64] Stephenson D B, Coelho C, Jolliffe I T. Two extra components in the Brier score decomposition[J].Weather and Forecasting, 2008, 23(4):708-716.
[65] Ferro C. Comparing probabilistic forecasting systems with the Brier score[J].Weather and Forecasting, 2007, 22(3):1 076-1 089.
[66] Ferro C, Richardson D S, Weigel A. On the effect of ensemble size on the discrete and continuous ranked probability scores[J]. Meteorological Applications, 2008, 15: 19-24.
[67] Hamill T M, Colucci S J. Verification of Eta-RSM short range ensemble forecasts[J]. Monthly Weather Review, 1997, 125: 711-724.
[68] Bröcker J, Smith L A. Increasing the reliability of reliability diagrams[J].Weather and Forecasting, 2007, 22(3):651-661.
[69] Wilks D S. The calibration simplex: A generalization of the reliability diagram for three-category probability forecasts[J].Weather and Forecasting, 2013, 28(5):1 210-1 218.
[70] Kharin K, Zwiers F W. Improved seasonal probability forecasts[J]. Journal of Climate, 2007, 16(11):1 684-1 701.
[71] Murphy A H. What is a good forecast? An essay on the nature of goodness in weather forecasting[J].Weather and Forecasting, 1993, 8(4):281-293.
[72] Wilson L J. Comments on “Probabilistic predictions of precipitation using the ECMWF ensemble prediction system”[J].Weather and Forecasting, 2000, 15(3):361-364.
[73] Bowler N E, Pierce C E, Seed A W. STEPS: A probabilistic forecasting scheme which merges an extrapolation nowcast with downscaled NWP[J].Quarterly Journal of the Royal Meteorological Society, 2008, 132(6):2 127-2 155.
[74] Du Jun, Deng Guo. The utility of the transition from deterministic to probabilistic weather forecasts—Verification and application of probabilistic forecasts[J]. Meteorological Monthly, 2010, 36(12):10-18.[杜均,邓国.单一值预报向概率预报转变的价值——谈谈概率预报的检验和应用[J].气象, 2010, 36(12):10-18.]
[75] Wilson L J, Burrows W R, Lanzinger A. A strategy for verification of weather element forecasts from an ensemble prediction system[J]. Monthly Weather Review, 1999, 127(3):956-970.
[76] Bröcker J, Smith L A. Scoring probabilistic forecasts: On the importance of being proper[J].Weather and Forecasting, 2007, 22(2):382-388.
[77] Hersbach H. Decomposition of the continuous rank probability score for ensemble prediction systems[J].Weather and Forecasting, 2000, 15(6):559-570.
[78] Hamill T M. Interpretation of rank histograms for verifying ensemble forecasts[J].Monthly Weather Review, 2001, 129(2):550-560.
[79] Jolliffe I T, Cristina P. Evaluating rank histograms using decompositions of the chi-square test statistic[J].Monthly Weather Review, 2008, 136(6):2 133-2 139.
[80] Dawid A P. Statistical theory: The prequential approach[J]. Journal Royal Statistical Society, 1984, 147:278-292.
[81] Candille G, Talagrand O. Evaluation of probabilistic prediction systems for a scalar variable[J]. Quarterly Journal of the Royal Meteorological Society, 2005, 131: 2 131-2 150.
[82] Wilks D S. The Minimum Spanning Tree (MST) histogram as a verification tool for multidimensional ensemble forecasts[J].Monthly Weather Review,2004, 132(7):1 329-1 340.
[83] Gombos D, Hansen J A. Theory and applications of the minimum spanning tree rank histogram[J].Monthly Weather Review, 2007, 135(4):1 490-1 505.
[84] Roulston M S, Smith L A. Evaluating probabilistic forecasts using information theory[J]. Monthly Weather Review, 2002, 130(6):1 653-1 660.
[85] Gneiting T, Raftery A E. Strictly proper scoring rules prediction and estimation[J].Journal of the American Statistics Association, 2007, 102(3):359-378.
No Suggested Reading articles found!