基于局地集合变换卡尔曼滤波的全球海洋资料同化系统设计及算法加速

doi:10.11867/j.issn.1001-8166.2019.05.0531

地球科学进展 ›› 2019, Vol. 34 ›› Issue (5): 531 -539. doi: 10.11867/j.issn.1001-8166.2019.05.0531

范峥 ¹(

),李宏 ¹,刘向文 ²,徐芳华 ¹(

)

1. 清华大学地球系统科学系，地球系统数值模拟教育部重点实验室，北京 100084
2. 中国气象局国家气候中心，北京，100081

收稿日期:2019-01-21 修回日期:2019-04-02 出版日期:2019-05-10
通讯作者: 徐芳华 E-mail:fan-z16@mails.tsinghua.edu.cn;fxu@tsinghua.edu.cn
基金资助:
国家重点研发计划项目“基于高分辨率气候系统模式的无缝隙气候预测系统研制与评估”(编号：2016YFA0602100)

Global Ocean Data Assimilation System Design and Algorithm Acceleration Based on Local Ensemble Transform Kalman Filter

Zheng Fan ¹( ),Hong Li ¹,Xiangwen Liu ²,Fanghua Xu ¹( )

1. Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing 100084, China
2. National Climate Center, China Meteorological Administration, Beijing 100081, China

Received:2019-01-21 Revised:2019-04-02 Online:2019-05-10 Published:2019-07-04
Contact: Fanghua Xu E-mail:fan-z16@mails.tsinghua.edu.cn;fxu@tsinghua.edu.cn
About author:Fan Zheng(1994-), male, Bengbu City, Anhui Province, Master student. Research areas include high performance computing in earth science. E-mail: fan-z16@mails.tsinghua.edu.cn
Supported by:
Project supported by the National Key R&D Program of China “Development and evaluation of seamless climate prediction system based on high resolution climate system model”(No. 2016YFA0602100)

全文 ( 6 ) 下载PDF ( 519 )

通过对局地集合变换卡尔曼滤波（LETKF）算法的计算时间复杂度的完整分析，发现计算集合空间分析场误差协方差的逆矩阵这一过程计算量最大，耗时最长。且在并行计算环境下，该步骤CPU计算量分配不均是影响计算效率的直接原因。为解决这一问题，采用“贪心算法”设计了一套新的负载均衡策略，并使用该策略开发了一个基于LETKF和并行海洋模块2（POP2）的高性能并行海洋资料同化系统。将2004年1～2月日平均的最优插值海表温度资料（OISST）和同时期的Argo温盐剖面资料同化进入POP2。结果表明，同化有效降低了温度和盐度的均方根误差。同时，在不改变计算结果的前提下，相比原始同化系统，新系统计算性能提升1倍。在更高分辨率（0.1°×0.1°）下，该系统的计算性能仍然可以提升1倍，说明新设计的负载均衡方案稳定可靠。该方案具有很强的可扩展性和移植性，在业务预报中有广泛的应用前景。

关键词: 海洋资料同化, 局地集合变换卡尔曼滤波, 负载均衡策略, 高性能

An integrated analysis about computational time complexity of the Local Ensemble Transform Kalman Filter (LETKF) was performed. It is found that the calculation step of inverse matrix of the error covariance in ensemble space is the most computationally intensive and time consuming. In a parallel computing environment, the uneven distribution of CPU calculations in this step directly leads to low computational efficiency. To solve this problem, a new load balancing strategy was designed based on the "greedy algorithm". A high-performance parallel ocean data assimilation system based on the LETKF was developed and tested using this strategy. This system was based on the Parallel Ocean Program 2 (POP2) of the Community Earth System Model (CESM). The optimal interpolated sea surface temperature data (OISST) and Argo temperature profile data from January to February, 2004 were assimilated into the POP2. The results show that data assimilation effectively reduces the root mean square error of temperature and salinity. Using the new strategy, the exact same results are obtained but the computation time is reduced by half. At higher resolution (0.1°×0.1°),the computing performance is still doubled, indicating that this load balancing scheme is stable and reliable. In addition, the new method has high scalability and portability with great potential to be applied in operational forecasting.

Key words: Ocean data assimilation, Local Ensemble Transform Kalman filter, Load balancing strategy, High performance

中图分类号:

P717

图1 模式点与观测点分布图

Fig. 1 Distribution of model grid points and observations

图2 CPU计算时间分布

Fig. 2 Distribution of CPU calculation time

表1 算法时间复杂度分析及说明

Table 1 Algorithm time complexity analysis and

时间复杂度	算法步骤及说明
$O k l [g]$	（1）应用观测算子，计算观测空间扰动
$O (k m [g])$	（2）计算背景场扰动
$O k ∑ i = 1 m [g] l i$	（3）选择格点内观测数据，得到局地观测误差协方差
$O k ∑ i = 1 m [g] l i$	（4）计算局地观测中间矩阵
$O k 2 ∑ i = 1 m [g] l i$	（5）计算集合空间分析场误差协方差的逆
$O k 3 m [g]$	（6）计算集合空间分析场误差协方差与背景场—分析场扰动变换矩阵
$O k ∑ i = 1 m [g] l i + k$	（7）计算集合空间卡尔曼增益分析向量
$O k 2 m [g]$	（8）应用卡尔曼增益，得到分析场

表1 算法时间复杂度分析及说明

Table 1 Algorithm time complexity analysis and

时间复杂度	算法步骤及说明
$O k l [g]$	（1）应用观测算子，计算观测空间扰动
$O (k m [g])$	（2）计算背景场扰动
$O k ∑ i = 1 m [g] l i$	（3）选择格点内观测数据，得到局地观测误差协方差
$O k ∑ i = 1 m [g] l i$	（4）计算局地观测中间矩阵
$O k 2 ∑ i = 1 m [g] l i$	（5）计算集合空间分析场误差协方差的逆
$O k 3 m [g]$	（6）计算集合空间分析场误差协方差与背景场—分析场扰动变换矩阵
$O k ∑ i = 1 m [g] l i + k$	（7）计算集合空间卡尔曼增益分析向量
$O k 2 m [g]$	（8）应用卡尔曼增益，得到分析场

图3 2004年1~2月Argo资料空间分布

Fig. 3 Spatial distribution of Argo data in January and February 2004

图4 全球海洋SST的RMSE（℃）分布

Fig. 4 Distribution of RMSE（℃）of global ocean SST

图5 同化前后RMSE垂直分布

Fig. 5 Vertical distribution of RMSE before and after data assimilation

图6 优化前后的CPU计算时间分布

Fig. 6 Distribution of CPU calculation time before and after optimization

1	Shen Zheqi , Tang Youmin , Gao Yanqiu . The theoretical framework of the ensemble-based data assimilation method and its prospect in oceanic data assimilation[J]. Acta Oceanologica Sinica, 2016, 38(3):1-14.
	沈浙奇, 唐佑民, 高艳秋 . 集合资料同化方法的理论框架及其在海洋资料同化的研究展望[J]. 海洋学报, 2016, 38(3):1-14.
2	Wang Shihong , Zhao Yiding , Yin Xunqiang , et al . Current status of global ocean reanalysis datasets [J]. Advances in Earth Science, 2018, 33(8):794-807.
	王世红,赵一丁,尹训强,等 . 全球海洋再分析产品的研究现状[J]. 地球科学进展, 2018, 33(8):794-807.
3	Le Dimet F X , Talagrand O . Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects[J]. Tellus A: Dynamic Meteorology and Oceanography, 1986, 38(2):97-110.
4	Li Hong , Xu Jianping . Development of data assimilation and its application in ocean science[J]. Marine Science Bulletin, 2011, 30(4):463-472.
	李宏, 许建平 . 资料同化技术的发展及其在海洋科学中的应用[J]. 海洋通报, 2011, 30(4):463-472.
5	Kalman R E . A new approach to linear filtering and prediction problems[J]. Journal of Fluids Engineering, 1960，81（1）:35-45.
6	Evensen G . Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics[J]. Journal of Geophysical Research: Oceans, 1994, 99(C5):10 143-10 162.
7	Hunt B R , Kostelich E J , Szunyogh I . Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter[J]. Physica D, 2007, 230(1/2):112-126.
8	Miyoshi T , Aranami K . Applying a four-dimensional local ensemble transform Kalman filter (4D-LETKF) to the JMA nonhydrostatic model (NHM)[J]. Sola, 2006, 2: 128-131.
9	Miyoshi T .Ensemble data assimilation for idealized California current system with ROMS-LETKF[C]//14^th Symposium on Integrated Observing and Assimilation Systems for the Atmosphere, Oceans, and Surface Land . 2010.
10	Xu F H , Oey L Y . State analysis using the Local Ensemble Transform Kalman Filter (LETKF) and the three-layer circulation structure of the Luzon Strait and the South China Sea[J]. Ocean Dynamics, 2014, 64(6):905-923.
11	Mcphaden M J , Busalacchi A J , Cheney R , et al . The Tropical Ocean‐Global Atmosphere observing system: A decade of progress[J]. Journal of Geophysical Research: Oceans, 1998,103(C7). DOI:10.1029/97JC0926 . doi: 10.1029/97JC0926
12	Riser S C , Freeland H J , Roemmich D , et al . Fifteen years of ocean observations with the global Argo array[J]. Nature Climate Change, 2016, 6(2):145-153.
13	Miyoshi T , Yamane S , Enomoto T . Localizing the error covariance by physical distances within a local ensemble transform Kalman filter (LETKF)[J]. Sola, 2007, 3: 89-92.
14	Bonavita M , Torrisi L , Marcucci F . The ensemble Kalman filter in an operational regional NWP system: Preliminary results with real observations[J]. Quarterly Journal of the Royal Meteorological Society, 2008, 134(636): 1 733-1 744.
15	Miyoshi T , Sato Y , Kadowaki T . Ensemble Kalman filter and 4D-Var intercomparison with the Japanese operational global analysis and prediction system[J]. Monthly Weather Review, 2010, 138(7): 2 846-2 866.
16	Wan Xiuquan , Liu Zedong , Shen Biao , et al . Introduction to the Community Earth System Model and application to high performance computing [J]. Advances in Earth Science, 2014, 29(4):482-491.
	万修全,刘泽栋,沈飙,等 . 地球系统模式CESM及其在高性能计算机上的配置应用实例[J]. 地球科学进展, 2014, 29(4):482-491.
17	Reynolds R W , Smith T M , Liu C , et al . Daily high-resolution-blended analyses for sea surface temperature[J]. Journal of Climate, 2007, 20(22):5 473-5 496.
18	Penny S G , Kalnay E , Carton J A , et al . The local ensemble transform Kalman filter and the running-in-place algorithm applied to a global ocean general circulation model[J]. Nonlinear Processes in Geophysics, 2013, 20(6):1 031-1 046.
19	Xu F H , Oey L Y , Miyazawa Y , et al . Hindcasts and forecasts of Loop Current and eddies in the Gulf of Mexico using local ensemble transform Kalman filter and optimum-interpolation assimilation schemes[J]. Ocean Modelling, 2013, 69: 22-38.
20	Penny S G . Data Assimilation of the Global Ocean Using the 4D Local Ensemble Transform Kalman Filter (4D-LETKF) and the Modular Ocean Model (MOM2)[D]. Maryland:University of Maryland, 2011.
21	Hayes B . Computing science: The easiest hard problem[J]. American Scientist, 2002, 90(2):113-117.
22	Korf R E . Multi-way number partitioning[C]//International Joint Conference on Ijcai. DBLP, 2009.
23	Ahmed M S , Nisha T A . Solving K-Set Partition Problem Using Genetic Algorithm[R]. Chicago:East West University, 2014.
24	Karmarker N , Karp R M . The Differencing Method of Set Partitioning[M]. Campbell Hall:Berkeley California University of California at Berkeley, 1983.
25	Cormen T H , Leiserson C E , Rivest R L , et al . Introduction to Algorithms (original book third edition)[J]. Computer Education, 2013(10):237, 161-192.
	Cormen T H , Leiserson C E , Rivest R L ,等 . 算法导论(原书第3版)[J]. 计算机教育, 2013(10):237, 161-192.
26	Large W G , Yeager S G . Diurnal to decadal global forcing for ocean and sea-ice models: The data sets and flux climatologies[J]. National Center for Atmospheric Research, 2004. DOI:10.5065/D6KK98Q6 . doi: 10.5065/D6KK98Q6

[1]	万修全, 刘泽栋, 沈飙, 林霄沛, 吴德星. 地球系统模式CESM及其在高性能计算机上的配置应用实例[J]. 地球科学进展, 2014, 29(4): 482-491.
[2]	刘妙龙,李乔,罗敏. 地理计算——数量地理学的新发展[J]. 地球科学进展, 2000, 15(6): 679-683.

阅读次数

全文

摘要