In this section, we describe the concept of ILS, its physical processes, the input/output (I/O) component, coupling system, regridding table calculator, boundary condition creator, and benchmarking. Then, we explain three experiments to (1) examine computational speed and scalability, (2) show the impact of coupling concerning computational reproducibility, and (3) provide an example of benchmarking.
2.1 Framework and components
2.1.1 Concept of ILS
Figure 1 shows a conceptual diagram of ILS. Multiple terrestrial models are coupled using a general-purpose coupler Jcup (Arakawa et al. 2020). Here, the ILS consists of two land models: a physical land surface model, MATSIRO (Takata et al. 2003; Nitta et al. 2014), and a hydrodynamic model, CaMa-Flood (Yamazaki et al. 2011). An I/O component, used to input and output files, is developed as an independent executable. The ILS employs the Multiple Program Multiple Data (MPMD) approach and executes these models and the I/O component in parallel. Each model can run in its preferred spatial coordinate and resolution. The physical land surface model, MATSIRO, is managed within the ILS because it models essential land processes. However, there are independent groups that have developed other land models. In such a case, the coupling procedure is as follows: (1) port the latest version of the component model, (2) employ the Jcup Application Programming Interface (API) to be coupled with ILS, and (3) prepare regridding tables for spatial interpolation. If there is an updated version of that model, steps (1) and (2) are repeated. This approach allows the latest versions of each land model to be included in the ILS with minimal code modifications. We can use the same procedure for a new different land model, if necessary. As Jcup has been used to couple an atmospheric general circulation model with an oceanic general circulation model (Miyakawa et al. 2017), it is relatively easy to couple the ILS to the atmospheric and oceanic general circulation models. Previously, offline land simulations were conducted by manually modifying the code. However, ILS enables site experiments, global offline experiments, and coupled climate experiments in a single framework without any modification of the code. Our future goal is to contribute to improving climate model simulations by incorporating more sophisticated land models with rapid development and validation cycles. The details of each model of the ILS are described in the following subsections in this section.
To speed up the development and validation cycles, we decided to employ a software management environment. We manage the codes through version-control software and use GitLab (https://about.gitlab.com/) to share information via Wiki, manage issues, and test the code automatically when a merge is requested. Continuous testing is essential for model development. Currently, a 2-day simulation of the global offline experiment and a site experiment are performed, and we are planning to add benchmarking tools as well. The benchmark system is explained in a separate subsection of this section.
2.1.2 Physical processes
Currently, ILS includes two physical models: MATSIRO and CaMa-Flood. As explained above, MATSIRO has been adopted as the MIROC land surface component. It consists of a single-layer canopy and multiple user-defined layers of snow and soil and employs a tile-scheme. MIROC participated in several phases of the Coupled Model Intercomparison Project. As one of the multi-models, many studies have evaluated its simulated land-related variables, such as surface air temperature and/or precipitation over land (Mueller and Seneviratne 2014; Miao et al. 2014), land-atmosphere interactions (Ukkola et al. 2018), evapotranspiration (Lian et al. 2018), surface albedo (Wang et al. 2016), snow cover (Thackeray et al. 2015), and permafrost (Koven et al. 2013). It has also been used in studies assessing water resources and flood impact, which combine offline experiments with observation-based meteorological data as inputs (Yoshimura et al. 2018).
First, we rewrote the MIROC5 version of MATSIRO (Watanabe et al. 2010; hereafter referred to as MATSIRO5) to improve user-friendliness such that new schemes are easier to implement, and thus, the preciseness can be improved. We decoupled MATSIRO from MIROC5, removed all unnecessary parts produced by the decoupling, and separated the rest into MATSIRO’s library and driver. The library includes only calculations of land processes, whereas the driver includes calendar management, parameter settings, reading and writing restart files, and subroutines for coupling with other land models. Users can modify the driver to conduct different types of simulations; ILS can handle any combination of land cover tiles, soil type tiles, and elevation tiles by modifying the driver. Users will choose which tiles to use and how to use them depending on their purposes. The original MATSIRO is a vertical one-dimensional model and has an IJ loop, a two-dimensional loop in space, in the innermost part of the code. We replaced this IJ loop with an expandable universal loop, which can be applied for multiple purposes, not just IJ but also ensemble members or land cover tiles. This simplifies the code, improves readability, and is advantageous for vector processors.
To confirm the reproducibility of the rewritten code, we performed two experiments with the same driving data provided by Plumbing of Land Surface Model (PALS; Best et al. 2015; Abramowitz 2012) using the ILS and MATSIRO5. The experimental settings are described in the experimental section. We examined seven main output variables: soil temperature and moisture in the first layer, snow water equivalent, canopy water, sensible heat flux, latent heat flux, and runoff for 20 sites with a 30-min temporal resolution. The results show bit-identical reproducibility in the single-point floating-point data type in most cases, and the differences are small enough compared to their standard deviations (Fig. 2).
We implemented iterative calculations in the flux calculation to solve the well-known instability of the skin temperature and surface fluxes. The surface heat balance equation can be written as follows:
$$ {\Delta }_g={H}_g+{R}_g+{E}_g+{E}_s+{E}_{sn}-{F}_g-{F}_{sn} $$
$$ {\Delta }_c={H}_c+{R}_c+{E}_i+{E}_{is}+{E}_t $$
where Δg and Δc are the energy divergences at the ground surface and canopy, respectively; Hg and Hc are the respective sensible heat fluxes; Rg and Rc are the respective net radiations; Eg, Es, Esn, Ei, Eis, and Et are the bare ground evaporation flux, bare ground sublimation flux, snow sublimation flux, latent heat flux from canopy interception, sublimation flux from canopy snow interception, and transpiration flux, respectively; and Fg and Fsn are the heat conduction fluxes to the soil and snow surface, respectively. If there is no melting of the ground surface, the above equations are solved by linearizing the skin temperatures Tg and Tc of the bare ground and the canopy, respectively. If the change in Tg or Tc from the previous time step is large, the error of the solution becomes large. In the initial verification of this study, there were some sites where the average daily variation fluctuated. We considered that one of the causes of the calculation instability might be that the skin temperature assumed an unrealistic value owing to an error in the heat balance calculation. Therefore, we implemented the Newton–Raphson method in MATSIRO.
CaMa-Flood (Yamazaki et al. 2011, 2013) is a global hydrodynamic model that uses daily river runoff simulated by land surface models as an input. It is currently one of the most widely used hydrodynamic models, owing to its unique method of representing flood-plain inundation using subgrid topographical parameters. In CaMa-Flood, an adaptive timestep based on the Courant-Friedrichs-Lewy condition is used. Aside from the implementation of modules necessary for the coupling system, the only modifications to the original CaMa-Flood model are related to acquiring daily river runoff data and outputting the simulated results. To input the river runoff data, Jcup is used, and only unit conversion is conducted in the modified model. Outputting the simulated results is conducted by the I/O component, rather than the original modules.
2.1.3 I/O component
In developing the ILS, we decided to develop an I/O component for reading and writing files independently. The I/O component is included in the MPMD program as an independent executable. Models can avoid the relatively time-consuming file input/output, such that the overall execution time can be reduced. The I/O component includes a user interface for temporal interpolations that users can implement freely. The grid interpolation function of the Jcup application programming interface (API) permits the use of input data with different spatial resolutions from that of the experiment. For example, we can use meteorological data with different spatial resolutions for an experiment. It is also possible to combine simulated results for only land grids with missing values for ocean grids before outputting the results. The function allows single-process execution and two options with multiple-process (MP) executions. The first MP option uses the Message Passing Interface (MPI) I/O function. In the second option, the root processor collects data from other processors and outputs them. The I/O component supports the binary and Network Common Data Form (NetCDF) formats. It is designed to be extendable to other output formats such as gtool. For NetCDF, metadata are prepared in JavaScript Object Notation (JSON) format. File names and metadata follow the Assistance for Land Surface Modeling Activities (ALMA) and Climate and Forecast (CF) conventions.
2.1.4 Coupling system
As described above, the ILS is designed to perform simulations by coupling a plurality of components. This section describes the coupling system built into the ILS. The coupling software used in ILS is Jcup (Arakawa et al. 2020). Jcup can couple multiple components combining both parallel and serial implementations. In addition, it has high versatility because there is no restriction on applicable grid coordinates, and users can freely implement interpolation schemes. However, because of its high flexibility, the interface is complicated, making it difficult for beginners to use. Therefore, an API with a more compact interface customized to the ILS was developed. The most distinguishable feature of Jcup is that it requires a correspondence between the sending-side grid point index, receiving-side grid point index, and interpolation coefficients in the interpolation calculation as input information (hereafter, referred to as “regridding table”). Figure 3 shows a typical use of the Jcup API subroutines. Before the time-integration loop, there are three types of subroutines (and moj_put_data), such as the initialization of the coupler, setting of grid information, and setting of the regridding table. In the time-integration loop, moj_set_time is called near the beginning of the loop. Data exchange and interpolation calculations were performed inside this subroutine. Subroutine moj_get_data obtains the data from the sending component, and moj_put_data is a subroutine that transfers the data to be sent to the receiving component to the coupler. These two subroutines can be called anywhere in the time-integration loop.
As described above, Jcup and its API require a regridding table as input information; therefore, the user needs to calculate a regridding table in advance. In previous cases, the calculation tool was developed individually for the models to be coupled. For example, to couple the icosahedral atmospheric model NICAM (Satoh et al. 2014) and the global ocean model COCO (Hasumi 2015), a program that searches for grid points included in each polygon of the opposite model grid and calculates the area of the overlapped region has been developed (Arakawa et al. 2014). For the seismic model Seism3D (Furumura 2005) and the structure model FrontISTR++ (Okuda 2019) coupling, the regridding table could be calculated easily because the grid structure of the seismic model was a simple rectangular grid with a constant grid spacing (Matsumoto et al. 2015).
Unlike these cases, the ILS has the possibility of coupling various models that have their own grid coordinates, and the combination of coupling should be flexible; therefore, a regridding table calculation tool applicable to a wide range of grid coordinates is required. Hence, a general-purpose regridding table calculation tool called Spheroidal Coordinates Regridding Interpolation Table Generator (SPRING) has been developed. The next section details this software.
2.1.5 Regridding table calculator
SPRING (Takeshima 2020) has been developed to couple models with any type of grid coordinate system, such as the latitude–longitude grid, polygonal grid, or grid defined by an aggregation of tiny latitude–longitude cells (Fig. 4). The basin-shaped coordinate system of the CaMa-Flood is of the last type. This tool provides regridding tables, a dataset of overlapping grid indices, and their interpolation weights for these coordinates. In conservative data communication in the ILS, the data value XIJ in the receiving-side grid point IJ is calculated by the following equation:
$$ {X}_{IJ}=\sum \limits_{ij}{c}_{ij}{x}_{ij},\kern0.5em {c}_{ij}=\frac{s_{ij}^{\prime }}{S_{IJ}} $$
where ij is the sending-side grid point index that overlaps with the grid IJ, cij is the interpolation weight, xij is the data value, \( {s}_{ij}^{\prime } \) is the overlapping area between grid IJ and grid ij, and SIJ is the area of the grid IJ. In the default configuration, MATSIRO and CaMa-Flood have basin-shaped coordinates, whose grids consist of tiny latitude–longitude cells, and the forcing data used in this study are expressed on a latitude–longitude grid. Therefore, \( {s}_{ij}^{\prime } \) and SIJ are calculated by the following equation regarding Earth as a perfect sphere with radius a
$$ S=\frac{\pi {a}^2}{180}\left(\sin {\varphi}_1-\sin {\varphi}_2\right)\left({\theta}_2-{\theta}_1\right) $$
where S is the area of a rectangular shape, θ1 and θ2 are the longitudes of the western/eastern boundaries, respectively, and φ1and φ2 are the latitudes of the northern/southern boundaries, respectively. The detailed evaluation of regridding tables will be presented in a separate paper and is beyond the scope of this study.
2.1.6 Boundary condition creator
The ILS includes a tool to create the boundary conditions of MATSIRO. They were manually created for different experiments. However, the information was not sufficiently shared, and sometimes the data included human errors. To avoid these issues, we manage the version of the codes to create boundary conditions. Thus, we can use the common parts repeatedly. Figure 5 shows a conceptual diagram of the boundary creation tool. By combining the original datasets and regridding tables described in the previous section, we can create new boundary condition data for any horizontal resolution, which include land cover type, soil type, soil albedo, standard deviation and gradient of elevation, and leaf area index (LAI). First, the land cover categories in the high-resolution original data are reclassified into categories to be used in MATSIRO. The regridding table is then used to calculate the dominant categories. For soil types, the proportions of sand, silt, and clay are interpolated using a regridding table. The soil type is then determined using the triangle diagram of the soil properties. The soil albedo is simply interpolated. The standard deviation and slope of elevation are calculated from high-resolution topographic data. The LAI is derived by a simple interpolation without the tiling scheme. If a land cover tiling scheme is used, the LAI is first classified by land use data and then averaged. The boundary tool creator allows the implementation of a new method, as the method varies depending on the source data.
2.1.7 Benchmarking
ILS incorporates benchmarking as part of the model development cycle. It is important to validate the model against a variety of terrestrial observations during each model development cycle and to evaluate how new parameterizations or models for preciseness and changes in resolution for detailedness can improve the simulated results. In this paper, we present the results of an experiment using PALS, which includes 20 Fluxnet sites (Fig. 6). The experiment is suitable for benchmarking because the sites cover the locations with various conditions, and its computational cost is low. The experimental settings are shown in the next section.
2.2 Specification of three experiments
First, we conducted global offline land simulations and evaluated the computational performance. The ILS was forced using three-hourly meteorological datasets prepared by the Global Soil Wetness Project Phase 3 (GSWP3; Kim et al. 2019). GSWP3 forcing data is based on downscaling of the Twentieth-Century Reanalysis (Compo et al. 2011), including rainfall, snowfall, temperature, wind speed, specific humidity, shortwave and longwave radiation, and surface pressure. Cloud cover from the same downscaling experiment was also used to partition shortwave radiation into visible and near-infrared, and direct and diffusive components. In this experiment, ILS consists of MATSIRO, CaMa-Flood, and I/O components. The temporal time step of MATSIRO was 1 h. The soil consists of six layers with depths of 5, 20, 75, 100, 200, and 1000 cm, and the snow consists of a maximum of three layers. The number of snow layers changes depending on the amount of snow.
Figure 7 shows the computational flow of the global offline experiment. The I/O component reads the meteorological forcing and boundary data, interpolates them temporally, and sends them to MATSIRO. Then, MATSIRO calculates land processes and sends simulated runoff to the CaMa-Flood routine, where the river processes are calculated. The output data from MATSIRO and CaMa-Flood are sent to the I/O component to be written to files. The experiment was run on a computer cluster with Intel Xeon processors at the University of Tokyo. We changed the number of cores for a component from 1 to 128 while fixing the number of cores (16 cores) for the other two components.
Second, we examined the difference between uncoupled (stand-alone) and coupled CaMa-Flood simulations to check whether coupling, including the regridding tables created by SPRING, works as expected. Note that we did not consider feedback from CaMa-Flood to MATSIRO in the coupled simulation and take the uncoupled simulation as the truth. We conducted a stand-alone river discharge simulation using the original CaMa-Flood with runoff data from MATSIRO in ILS (uncoupled simulation). We compared it with the simulation mentioned above (coupled simulation). This experiment intends to confirm if the coupled simulation is as similar to the uncoupled simulation by CaMa-Flood, in which its original regridding scheme is implemented.
Finally, we conducted site experiments for 20 Fluxnet sites from PALS, as shown in Fig. 6, to (1) validate the rewritten code and (2) show the results of the first benchmark. The sites include tropical, arid, temperate, and cold regions, as assigned by the Koppen climate classification (Peel et al. 2007), and forest and non-forest regions. The dataset includes meteorological forcing data: downward shortwave and longwave radiation, temperature, precipitation, specific humidity, wind speed, and surface pressure. In addition to these data, cloud cover from ERA5 (C3S 2017) was used. PALS also includes surface flux data from the observations and estimated by one-, two-, and three-variable regression equations for validation and benchmarking. The time steps of the input data and simulation were 30 min without any missing data. The simulation periods range from 2 to 10 years, depending on the site. We used the MIROC5 version of the ILS for the verification shown above. Then, we implemented the iteration of the flux calculation to MATSIRO as described above and evaluated the PALS experiment. Further observation stations, validation of global experiments, and increasing the amount of observation data are clearly in our future work.