Advancements in Study on the Application of MODE Verification Method in Weather Forecasting
Received date: 2023-10-08
Revised date: 2024-01-20
Online published: 2024-03-05
Supported by
the Innovation and Development Special Project of China Meteorological Administration(CXFZ2022J023);Review and Summary Special Project of China Meteorological Administration(FPZJ2023-129);Key Areas Project for Social Development in Shaanxi Province(2024SF-YBXM-556)
The objective evaluation of small-scale variables’ forecast performance is vital for the application and development of Numerical Weather Prediction (NWP). Traditional point-to-point verification has significant limitations in the evaluation of high-resolution NWP. The Object-based Diagnostic Evaluation (MODE) method utilizes convolution functions and a given threshold to identify objects in the forecast and observation fields, extract their attributes, and diagnose the performance of the NWP. It has been widely applied in weather forecasting. This paper systematically reviews the academic ideas, technical framework, algorithm flow, and verification indices of the MODE spatial verification method. Subsequently, this paper summarizes the typical applications of MODE verification in precipitation forecasting, weather radar, satellite cloud images, ensemble forecasting, and other elements. It elaborates on the significance of verification results in evaluating the quality of NWP and their role in improving the accuracy of weather forecast results, both subjectively and objectively. Furthermore, it introduces recent updates and developments in MODE verification methods. These include the comprehensive evaluation index MODE Composite Score (MCS), which considers the mismatched attributes of objects, three-dimensional spatiotemporal object tracking using ellipsoids as targets, and the verification method, MODE Time Domain (MTD). Finally, it discusses the MODE verification method's applicability, advantages, and limitations while considering its future development direction and application prospects. The purpose of this study is to provide references for better application and diagnosis of NWP performance using the MODE method.
Key words： Numerical Weather Prediction; Spatial verification; Target object; MODE.
Liujie PAN , Hongfang ZHANG , Jiahuimin LIU , Chunjuan QI , Mian LIANG , Danmeng MA , Peirong LI , Changming DAI , Xingxing GAO . Advancements in Study on the Application of MODE Verification Method in Weather Forecasting[J]. Advances in Earth Science, 2024 , 39(2) : 193 -206 . DOI: 10.11867/j.issn.1001-8166.2024.012
| 1 | GILLELAND E， AHIJEVYCH D， BROWN B G， et al. Intercomparison of spatial forecast verification methods［J］. Weather and Forecasting， 2009， 24（5）： 1 416-1 430. |
| 2 | DAI Kan， BI Baogui， ZHU Yuejian. Investigation of the medium-range forecast errors for the extreme rainfall event in North China during July 19-20， 2016［J］. Chinese Science Bulletin，2018， 63（3）：340-355. |
| 2 | 代刊，毕宝贵，朱跃建. 2016年7月华北极端降水的中期预报误差分析［J］.科学通报， 2018， 63（3）： 340-355. |
| 3 | MA Leiming， LIN Hong， CHU Hai， et al. Research progress of Shanghai operational intelligent forecast technologies on severe convection［J］. Advances in Earth Science， 2023， 38（2）： 111-124. |
| 3 | 马雷鸣， 林红， 储海， 等. 上海强对流智能预报业务新技术研究进展［J］. 地球科学进展， 2023， 38（2）： 111-124. |
| 4 | LIU Donghai， HUANG Jing， LIU Juan， et al. Review and prospect of parameterization schemes of typical mesoscale numerical prediction models［J］. Advances in Earth Science， 2023， 38（4）： 349-362. |
| 4 | 刘东海，黄静，刘娟，等. 典型中尺度数值预报模式参数化方案的综述与展望［J］. 地球科学进展， 2023， 38（4）： 349-362. |
| 5 | CAO Yancha， ZHENG Yongguang， SHENG Jie， et al. Severe convective wind and hail probabilistic forecasting method based on outputs of GRAPES3 km model［J］. Meteorological Monthly， 2021， 47（9）： 1 047-1 061. |
| 5 | 曹艳察， 郑永光， 盛杰， 等. 基于GRAPES_3 km模式输出的风雹概率预报技术研究［J］. 气象， 2021， 47（9）： 1 047-1 061. |
| 6 | MAO Xu， LIU Xinhua， YANG Bo. An optimized probabilistic prediction method for aircraft icing potential based on a convection-allowing model［J］. Chinese Journal of Atmospheric Sciences， 2023， 47（5）： 1 525-1 540. |
| 6 | 毛旭， 刘鑫华， 杨波. 一种优化的基于对流可分辨模式的飞机积冰潜势概率预报方法［J］. 大气科学， 2023， 47（5）： 1 525-1 540. |
| 7 | MASS C F， OVENS D， WESTRICK K， et al. Does increasing horizontal resolution produce more skillful forecasts？［J］. Bulletin of the American Meteorological Society， 2002， 83（3）： 407-430. |
| 8 | SCHAEFER J T. The critical success index as an indicator of warning skill［J］. Weather and Forecasting， 1990， 5（4）： 570-575. |
| 9 | PAN Liujie， ZHANG Hongfang， WANG Jianpeng. Progress on verification methods of numerical weather prediction［J］. Advances in Earth Science， 2014， 29（3）： 327-335. |
| 9 | 潘留杰， 张宏芳， 王建鹏. 数值天气预报检验方法研究进展［J］. 地球科学进展， 2014， 29（3）： 327-335. |
| 10 | BRILL K F， MESINGER F. Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores［J］. Weather and Forecasting， 2009， 24（6）： 1 748-1 754. |
| 11 | EBERT E E. Fuzzy verification of high-resolution gridded forecasts： a review and proposed framework［J］. Meteorological Applications， 2008， 15（1）： 51-64. |
| 12 | AHIJEVYCH D， GILLELAND E， BROWN B G， et al. Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts［J］. Weather and Forecasting， 2009， 24（6）： 1 485-1 497. |
| 13 | GILLELAND E， AHIJEVYCH D A， BROWN B G， et al. Verifying forecasts spatially［J］. Bulletin of the American Meteorological Society， 2010， 91（10）： 1 365-1 376. |
| 14 | EBERT E E， McBRIDE J L. Verification of precipitation in weather systems： determination of systematic errors［J］. Journal of Hydrology， 2000， 239（1/2/3/4）： 179-202. |
| 15 | EBERT E E. Neighborhood verification： a strategy for rewarding close forecasts［J］. Weather and Forecasting， 2009， 24（6）： 1 498-1 510. |
| 16 | PAN Liujie， ZHANG Hongfang， LIU Jing， et al. Comparative analysis of SCMOC and various numerical models for precipitation forecasting［J］. Transactions of Atmospheric Science， 2023， 46（2）： 217-229. |
| 16 | 潘留杰，张宏芳，刘静，等.智能网格SCMOC及多模式降水预报对比［J］.大气科学学报，2023， 46（2）： 217-229. |
| 17 | MARZBAN C， SANDGATHE S， LYONS H， et al. Three spatial verification techniques： cluster analysis， variogram， and optical flow［J］. Weather and Forecasting， 2009， 24（6）： 1 457-1 471. |
| 18 | MICHEAS A C， FOX N I， LACK S A， et al. Cell identification and verification of QPF ensembles using shape analysis techniques［J］. Journal of Hydrology， 2007， 343（3/4）： 105-116. |
| 19 | LACK S A， LIMPERT G L， FOX N I. An object-oriented multiscale verification scheme［J］. Weather and Forecasting， 2010， 25（1）： 79-92. |
| 20 | GILLELAND E. Testing competing precipitation forecasts accurately and efficiently： the spatial prediction comparison test［J］. Monthly Weather Review， 2013， 141（1）： 340-355. |
| 21 | MARZBAN C， SANDGATHE S. Cluster analysis for object-oriented verification of fields： a variation［J］. Monthly Weather Review， 2008， 136（3）： 1 013-1 025. |
| 22 | STEIN J， STOOP F. Neighborhood-based contingency tables including errors compensation［J］. Monthly Weather Review， 2019， 147（1）： 329-344. |
| 23 | SCHWARTZ C S， ROMINE G S， SOBASH R A， et al. NCAR’s experimental real-time convection-allowing ensemble prediction system［J］. Weather and Forecasting， 2015， 30（6）： 1 645-1 654. |
| 24 | PAN Liujie， ZHANG Hongfang， CHEN Xiaoting， et al. Neighborhood-based precipitation forecasting capability analysis of high-resolution models［J］. Journal of Tropical Meteorology， 2015， 31（5）： 632-642. |
| 24 | 潘留杰， 张宏芳， 陈小婷， 等. 基于邻域法的高分辨率模式降水的预报能力分析［J］. 热带气象学报， 2015， 31（5）： 632-642. |
| 25 | PAN Liujie， ZHANG Hongfang， XUE Chunfang， et al. Numerical weather prediction model testing and evaluation systems MET and application［J］. Advances in Meteorological Science and Technology， 2016， 6（4）： 37-43. |
| 25 | 潘留杰， 张宏芳， 薛春芳， 等. 数值模式评估系统MET及其初步应用［J］. 气象科技进展， 2016， 6（4）： 37-43. |
| 26 | PAN Liujie， XUE Chunfang， ZHANG Hongfang， et al. Comparison of three verification methods for high-resolution grid precipitation forecast［J］. Climatic and Environmental Research， 2017， 22（1）： 45-58. |
| 26 | 潘留杰， 薛春芳， 张宏芳， 等. 三种高分辨率格点降水预报检验方法的对比［J］. 气候与环境研究， 2017， 22（1）： 45-58. |
| 27 | KOCHASIC M C， GALLUS W A， SCHAFFER C J. Further evaluation of probabilistic convective precipitation forecasts using the QPF-PoP neighborhood relationship［J］. Weather and Forecasting， 2017， 32（4）： 1 423-1 440. |
| 28 | JOHNSON A， WANG X G， WANG Y M， et al. Neighborhood- and object-based probabilistic verification of the OU MAP ensemble forecasts during 2017 and 2018 hazardous weather testbeds［J］. Weather and Forecasting， 2020， 35（1）： 169-191. |
| 29 | DAVIS C， BROWN B， BULLOCK R. Object-based verification of precipitation forecasts. part I： methodology and application to mesoscale rain areas［J］. Monthly Weather Review， 2006， 134（7）： 1 772-1 784. |
| 30 | DAVIS C A， BROWN B G， BULLOCK R， et al. The Method for Object-based Diagnostic Evaluation （MODE） applied to numerical forecasts from the 2005 NSSL/SPC spring program［J］. Weather and Forecasting， 2009， 24（5）： 1 252-1 267. |
| 31 | DAVIS C， BROWN B， BULLOCK R. Object-based verification of precipitation forecasts. part II： application to convective rain systems［J］. Monthly Weather Review， 2006， 134（7）： 1 785-1 795. |
| 32 | BYTHEWAY J L， KUMMEROW C D. Toward an object-based assessment of high-resolution forecasts of long-lived convective precipitation in the central U.S［J］. Journal of Advances in Modeling Earth Systems， 2015， 7（3）： 1 248-1 264. |
| 33 | FLORA M L， SKINNER P S， POTVIN C K， et al. Object-based verification of short-term， storm-scale probabilistic mesocyclone guidance from an experimental warn-on-forecast system［J］. Weather and Forecasting， 2019， 34（6）： 1 721-1 739. |
| 34 | CHEN Xiao， ZHAO Dong， HE Xiaofeng， et al. Evaluation and analysis of model forecast performance of high wind based on MODE method［J］. Meteorological Monthly， 2018， 44（8）： 1 009-1 019. |
| 34 | 陈笑， 赵东， 何晓凤， 等. 基于MODE对模式预报强风风场的检验分析［J］. 气象， 2018， 44（8）： 1 009-1 019. |
| 35 | MITTERMAIER M， NORTH R， SEMPLE A， et al. Feature-based diagnostic evaluation of global NWP forecasts［J］. Monthly Weather Review， 2016， 144（10）： 3 871-3 893. |
| 36 | GIANNAKAKI P， MARTIUS O. An object-based forecast verification tool for synoptic-scale rossby waveguides［J］. Weather and Forecasting， 2016， 31（3）： 937-946. |
| 37 | YOU Fengchun， WANG Guorong， GUO Rui， et al. The application analysis of MODE method to the rainfall forecast test［J］. Meteorological Monthly， 2011， 37（12）： 1 498-1 503. |
| 37 | 尤凤春， 王国荣， 郭锐， 等. MODE方法在降水预报检验中的应用分析［J］. 气象， 2011， 37（12）： 1 498-1 503. |
| 38 | WANG Guorong， CHEN Min， YOU Fengchun， et al. Method for object-based diagnostic evaluation and its application［J］. Meteorological Science and Technology， 2014， 42（4）： 652-656. |
| 38 | 王国荣， 陈敏， 由凤春， 等. 基于对象诊断的空间检验方法（MODE）［J］. 气象科技， 2014， 42（4）： 652-656. |
| 39 | LIU Jing， REN Chuan， ZHAO Ziqi， et al. Comparative analysis on verification of heavy rainfall forecasts in different regional models［J］. Meteorological Monthly， 2022， 48（10）： 1 292-1 302. |
| 39 | 刘静， 任川， 赵梓淇， 等. 多区域高分辨率模式强降水预报检验分析［J］. 气象， 2022， 48（10）： 1 292-1 302. |
| 40 | YU Biyu， ZHU Kefeng. Application of multiple spatial verification methods to precipitation forecasts from different resolution models［J］. Journal of the Meteorological Sciences， 2022， 42（3）： 341-355. |
| 40 | 俞碧玉， 朱科锋. 多种空间检验方法在不同分辨率模式降水预报评估中的应用［J］. 气象科学， 2022， 42（3）： 341-355. |
| 41 | QU Qiaona， SHENG Chunyan， FAN Sudan， et al. Study on characteristics of rainstorm prediction by ECMWF and CMA-SH9［J］. Journal of the Meteorological Sciences， 2023， 43（2）：196-206. |
| 41 | 曲巧娜，盛春岩，范苏丹，等. 基于ECMWF和华东区域中尺度模式的暴雨预报特征研究［J］. 气象科学，2023，43（2）：196-206. |
| 42 | PAN Liujie， ZHANG Hongfang， LIU Jing， et al. Comparative analysis of SCMOC and various numerical models for pre-cipitation forecasting［J］. Transactions of Atmospheric Sciences， 2023， 46（2）： 217-229. |
| 42 | 潘留杰， 张宏芳， 刘静， 等. 智能网格SCMOC及多模式降水预报对比［J］. 大气科学学报， 2023， 46（2）： 217-229. |
| 43 | CASE J L， KUMAR S V， SRIKISHEN J， et al. Improving numerical weather predictions of summertime precipitation over the southeastern United States through a high-resolution initialization of the surface state［J］. Weather and Forecasting， 2011， 26（6）： 785-807. |
| 44 | WOLFF J K， HARROLD M， FOWLER T， et al. Beyond the basics： evaluating model-based precipitation forecasts using traditional， spatial， and object-based methods［J］. Weather and Forecasting， 2014， 29（6）： 1 451-1 472. |
| 45 | PAN L J， ZHANG H F， LIU J， et al. Comparative analysis of SCMOC and models rainstorm forecasting performance in Qinling Mountains and their surrounding areas［J］. Atmosphere， 2022， 13（5）. DOI：10.3390/atmos13050705 . |
| 46 | ZHANG H F， PAN L J. Diagnostic analysis of multimodel rainstorm forecast for cases based on MODE method［J］. Atmosphere， 2022， 13（7）. DOI：10.3390/atmos13071047 . |
| 47 | DUDA J D， TURNER D D. Large-sample application of radar reflectivity object-based verification to evaluate HRRR warm-season forecasts［J］. Weather and Forecasting， 2021， 36（3）： 805-821. |
| 48 | CONNELLY R， COLLE B A. Validation of snow multibands in the comma head of an extratropical cyclone using a 40-member ensemble［J］. Weather and Forecasting， 2019， 34（5）： 1 343-1 363. |
| 49 | FAIRMAN J G， SCHULTZ D M， KIRSHBAUM D J， et al. Climatology of banded precipitation over the contiguous United States［J］. Monthly Weather Review， 2016， 144（12）： 4 553-4 568. |
| 50 | LAWSON J R， GALLUS W A. Adapting the SAL method to evaluate reflectivity forecasts of summer precipitation in the central United States［J］. Atmospheric Science Letters， 2016， 17（10）： 524-530. |
| 51 | GANETIS S A， COLLE B A， YUTER S E， et al. Environmental conditions associated with observed snowband structures within northeast U.S. winter storms［J］. Monthly Weather Review， 2018， 146（11）： 3 675-3 690. |
| 52 | CINTINEO R， OTKIN J A， XUE M， et al. Evaluating the performance of planetary boundary layer and cloud microphysical parameterization schemes in convection-permitting ensemble forecasts using synthetic GOES-13 satellite observations［J］. Monthly Weather Review， 2014， 142（1）： 163-182. |
| 53 | THOMPSON G， TEWARI M， IKEDA K， et al. Explicitly-coupled cloud physics and radiation parameterizations and subsequent evaluation in WRF high-resolution convective forecasts［J］. Atmospheric Research， 2016， 168： 92-104. |
| 54 | BIKOS D， LINDSEY D T， OTKIN J， et al. Synthetic satellite imagery for real-time high-resolution model evaluation［J］. Weather and Forecasting， 2012， 27（3）： 784-795. |
| 55 | VAN WEVERBERG K， VOGELMANN A M， LIN W， et al. The role of cloud microphysics parameterization in the simulation of mesoscale convective system clouds and precipitation in the tropical Western Pacific［J］. Journal of the Atmospheric Sciences， 2013， 70（4）： 1 104-1 128. |
| 56 | JIN H， PENG M S， JIN Y， et al. An evaluation of the impact of horizontal resolution on tropical cyclone predictions using COAMPS-TC［J］. Weather and Forecasting， 2014， 29（2）： 252-270. |
| 57 | MITTERMAIER M P， BULLOCK R. Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models［J］. Meteorological Applications， 2013， 20（2）： 187-196. |
| 58 | GRIFFIN S M， OTKIN J A， ROZOFF C M， et al. Methods for comparing simulated and observed satellite infrared brightness temperatures and what do they tell us？［J］. Weather and Forecasting， 2017， 32（1）： 5-25. |
| 59 | GRIFFIN S M， OTKIN J A， ROZOFF C M， et al. Seasonal analysis of cloud objects in the High-Resolution Rapid Refresh （HRRR） model using object-based verification［J］. Journal of Applied Meteorology and Climatology， 2017， 56（8）： 2 317-2 334. |
| 60 | NOHARA D， KITOH A， HOSAKA M， et al. Impact of climate change on river discharge projected by multimodel ensemble［J］. Journal of Hydrometeorology， 2006， 7（5）： 1 076-1 089. |
| 61 | GALLUS W A. Application of object-based verification techniques to ensemble precipitation forecasts［J］. Weather and Forecasting， 2010， 25（1）： 144-158. |
| 62 | CLARK A J， GALLUS W A， XUE M， et al. A comparison of precipitation forecast skill between small convection-allowing and large convection-parameterizing ensembles［J］. Weather and Forecasting， 2009， 24（4）： 1 121-1 140. |
| 63 | JI L Y， ZHI X F， SIMMER C， et al. Multimodel ensemble forecasts of precipitation based on an object-based diagnostic evaluation［J］. Monthly Weather Review， 2020， 148（6）： 2 591-2 606. |
| 64 | JOHNSON A， WANG X G. Verification and calibration of neighborhood and object-based probabilistic precipitation forecasts from a multimodel convection-allowing ensemble［J］. Monthly Weather Review， 2012， 140（9）： 3 054-3 077. |
| 65 | GRIFFIN S M， OTKIN J A， THOMPSON G， et al. Assessing the impact of stochastic perturbations in cloud microphysics using GOES-16 infrared brightness temperatures［J］. Monthly Weather Review， 2020， 148（8）： 3 111-3 137. |
| 66 | SKOK G， TRIBBIA J， RAKOVEC J， et al. Object-based analysis of satellite-derived precipitation systems over the low- and midlatitude Pacific Ocean［J］. Monthly Weather Review， 2009， 137（10）： 3 196-3 218. |
| 67 | WHITE R H， BATTISTI D S， SKOK G. Tracking precipitation events in time and space in gridded observational data［J］. Geophysical Research Letters， 2017， 44（16）： 8 637-8 646. |
| 68 | SKOK G， TRIBBIA J， RAKOVEC J. Object-based analysis and verification of WRF model precipitation in the low- and midlatitude Pacific Ocean［J］. Monthly Weather Review， 2010， 138（12）： 4 561-4 575. |
| 69 | SKOK G， BACMEISTER J， TRIBBIA J. Analysis of tropical cyclone precipitation using an object-based algorithm［J］. Journal of Climate， 2013， 26（8）： 2 563-2 579. |
| 70 | CLARK A J， BULLOCK R G， JENSEN T L， et al. Application of object-based time-domain diagnostics for tracking precipitation systems in convection-allowing models［J］. Weather and Forecasting， 2014， 29（3）： 517-542. |
| 71 | ZHANG Hongfang， PAN Liujie， LU Shan， et al. Tracking diagnosis analysis of grid precipitation forecast based on Time-Domain object［J］. Plateau Meteorology， 2021， 40（3）： 559-568. |
| 71 | 张宏芳， 潘留杰， 卢珊， 等. 基于时域对象的网格降水预报的追踪诊断分析［J］. 高原气象， 2021， 40（3）： 559-568. |
| 72 | AYAT H， EVANS J P， SHERWOOD S， et al. Are storm characteristics the same when viewed using merged surface radars or a merged satellite product？［J］. Journal of Hydrometeorology， 2021， 22（1）： 43-62. |
/
| 〈 |
|
〉 |
甘公网安备62010202000687
