Increasing the Interoperability of an Earth System Model:
Atmospheric-Ocean Dynamics and Tracer Transports
NCC4-624

Milestone G: Second code improvement completed.
Upgrade OGCM POP to a near global domain (excluding both polar regions) without loss of performance compared to E. Provide scaling curves. Documented source code made publicly available via the Web.

1. Introduction

This CAN addresses major thrusts of the ESS Project: 1) to further our understanding of and ability to predict the dynamic interaction of physical and chemical processes affecting Earth, 2) to incorporate the use of NASA data, and 3) to demonstrate interoperability of codes used in the community of Earth Science. We are following a three-tiered approach.

Tier I consists of integrating the coupled atmosphere-ocean part of the UCLA Earth System Model (ESM) into the ESMF. The coupled atmosphere-ocean part of the UCLA ESM comprises the UCLA atmospheric general circulation model (UCLA AGCM) and the LANL oceanic general circulation model (OGCM) also known as the Parallel Ocean Program (POP). These model components are described in Milestone A.

Tier II addresses the issues of code interoperability by using the ESMF services to couple the AGCM with either POP or the OGCM developed at the Massachusetts Institute of Technology (MIT), and by performing forecasts of El Niño/Southern Oscillation (ENSO).

Tier III focuses on the impact of NASA data and consists of comparing ENSO forecasts made from initial conditions corresponding to the quasi-operational analysis of the time-evolving ocean circulation produced by the consortium for Estimating the Circulation and Climate of the Ocean (ECCO) maintained at the Jet Propulsion Laboratory (JPL). The product is updated approximately once per week, is freely available (http://ecco.jpl.nasa.gov), and is being used for a variety of science applications (e.g., Lee and Fukumori, 2003; Fukumori et al. 2004). The MIT OGCM is a component in ECCO’s data assimilation system, while POP is not.

2. The MIT OGCM

The MIT OGCM domain spans the latitudes 80S to 79N. In the zonal direction the resolution is 1 degree. In the meridional direction, the resolution is 0.3 degrees within 10 degrees of the equator, increasing to 1 degree outside the tropics. There are 46 levels in the vertical, with thicknesses ranging from 10 to 400 m down to a maximum depth of 5815 m, as shown on. The integration time step is 1 hour. In total, there are 360x224x46 grid cells. More details on the MIT OGCM are given in Marshall et al. (1997). The model employs the K-Profile Parameterization (KPP) vertical mixing scheme of Large et al. (1994) and the isopycnal mixing schemes of Redi (1982) and of Gent and McWilliams (1990) with surface tapering as in Large et al. (1997). At the bottom and lateral boundaries no-slip and free-slip conditions are applied, respectively. At the top, a free surface condition is applied.

3. Upgrade of POP code

For a meaningful comparison between forecasts with POP and the MIT OGCM it is necessary that both Ocean Models have the same grid and domains. So far in our NASA CT project, POP's domain has covered selected regions of the World Oceans. Our work in the Second Round targeted the North Atlantic Ocean, which was studied at very high horizontal resolution (1/6 degree in longitude and latitude). Our work on El Niño/Southern Oscillation (ENSO) has targeted the Tropical Pacific from 30S to 50N.

The work performed under Milestone G was to develop a version of POP with exactly the same grid and bathymetry than those used in the MIT OGCM and described in section 2 of this report. All models in this project are integrated into the ESMF in Milestone J. ENSO forecasts are performed and compared in Milestone K.

4. Results

The geometric extension of POP domain is relatively straightforward. Nevertheless, the quality of simulations may be degraded as artificial constraints imposed as the ocean circulations at the model boundaries is prescribed from observational datasets can mask model difficulties. Before the upgrade of the Planetary Boundary Layer (PBL) in Milestone F of this Round we had noticed that including the midlatitudes in the POP domain resulted in a significant local cold bias and an increase in the model’s climate drift.

The results shown in Fig. 1 show that the UCLA AGCM with low resolution (5 longitude by 4 latitude) and upgraded PBL parameterization coupled to the quasi-global version of POP does not have the difficulties detected in previous versions and produces realistic results. The figure shown corresponds to the sea surface temperature (SST) for the month of December 1997 from initial conditions corresponding to June 1, 1997. The performance of the quasi-global POP, therefore, satisfies the requirements for its application to produce ENSO forecasts.

5. Code Performance

The quasi-global POP OGCM is based on version 2.0 of the LANL OGCM. The distribution of sea ice is prescribed from an observed, time-varying climatology. The code performance was evaluated by using two configurations with different number of nodes in the SGI 3000 system at NASA AMES (LOMAX). In the two cases, the number of nodes assigned to the AGCM was large enough to be meaningful and small enough to insure that this model completed each coupling interval before the OGCM. For comparison, we also timed the Tropical Pacific version with a domain between 130E and 70W in longitude, 30S and 50N in latitude. Both models were configured with the same input parameters along with the same physical processes. Table 1 shows the time (seconds) required by the OGCM to complete a 10-day simulation.

 

 

Figure 1. POP (top) and Observation (bottom). Prepared by G. Cazes.

Table 1. Ten-day OGCM Simulation Timing (seconds)

Number of Nodes	Global (G) Number of ocean points: 2,177,448	Pacific Basin (TP) Number of ocean points: 835,876
AGCM	Timing (Seconds)	Timing (Seconds)
31	336.0	165.9
31 for (G) 43 for (TP)	192.3	105.8

OGCM
96

224 for (G)
210 for (TP)

Table 2 shows in more detail the performance of some model components.

Table 2. Ten day OGCM Simulation Timings (seconds): Components

Timer	Tropical Pacific 96 nodes	Tropical Pacific 210 nodes	Global 96 nodes	Global 224 nodes
KPP	26.22	13.33	98.24	41.37
IMPVMIXT	96.75	69.07	218.53	112.22
IMPVMIXU	0.22	0.09	3.26	0.79
BAROCLINIC	90.16	50.90	248.15	118.28
BAROTROPIC	17.58	16.84	24.43	21.49
STEP	162.69	100.72	323.07	178.58
TOTAL	165.90	105.84	336.04	192.33

The interpretation of the results in Tables 1 and 2 is not straightforward since the two configurations (Tropical Pacific and Global) have a different number of land points where some calculations are performed. Nevertheless, we can see that doubling the number of nodes results in about 1.75 speedup for the Global and 1.6 for the Tropical Pacific.

6. Risk assessment

The project plans to practice risk management to anticipate, mitigate, and control risks to the project. All items with high and medium impact risks will generate mitigation plans that will be closely monitored by the P.I and Software Engineer. There will be monthly meetings to review the status of each part of the project and identify critical areas that are behind schedule or have problems that need to be addressed. The meetings will be in person after each milestone is achieved; otherwise they will be by phone conference. The risk assessment and mitigation strategy will be updated by adding newly identified risks, adjusting probability of impact, etc. as the project progresses, and an updated risk report will be produced. This risk report will include risk name, description, level of impact, probability of impact, type of mitigation (avoid, mitigate, or accept). If mitigation action is appropriate, a brief description of the action and current status of the action will be included (see Table 1).

Table 3. Revised List of Risks and Risk Mitigation Strategies

Name
Description of Risk
Level of Impact (H, M, L)
Prob. Of Impact (H, M, L) Type of Mitigation (Avoid, Mitigate, Accept)
Mitigation Actions Upgrade PBL parameterization for AGCM
Completed El Niño simulation Poor performance of AGCM coupled to MIT OGCM
M
L
Mitigate Preliminary results indicate that the coupled model produces a realistic interannual variability. HYPOP Timely availability
M
H Avoid Use POP version 2.0 in a quasi-global domain
El Niño Prediction Poor model simulation of El Niño. If model simulation of ENSO is poor, then predictions of ENSO may be unreliable and science findings unreliable. H L

Mitigate Preliminary results show that the coupled model produces realistic ENSO forecasts, both with POP and MIT OGCM. Availabity of appropriate ESMF interfaces
Available Integration of ESM into ESMF
H
L
Mitigate Strengthen exchange of information between ESMF developers and this project.

Configuration Management

The ESM software is managed using the tool Concurrent Versions System (CVS). CVS provides a repository for code, checkout of code, a history of changes made to the code, and a commit feature to prevent developers from making contradictory changes to the same code. The ESM test scripts and test data are also managed using CVS. Additional documentation on CVS can be obtained from http://www.gnu.org/manual/cvs-1.9/cvs.html.
The ESM’s assigned gatekeeper and backup gatekeeper to accept changes to the baseline version of the software is Joseph A. Spahr (spahr@atmos.ucla.edu). The model codes are available at esm-a.atmos.ucla.edu/~esm. The project has a documentation repository on the public web site (www.atmos.ucla.edu/~mechoso/esm).

Storage, Handling, and Delivery

We employ writable CD-ROMs to produce full snapshots of the repository regular intervals. Backups are kept off-site at a secure location to be determined by the configuration manager. We use the Internet as our primary medium for software distribution.

References

Fukumori, I., T. Lee, B. Cheng, and D. Menemenlis, 2004: The origins, pathway, and destination of Ni˜no-3 water estimated by a simulated passive tracer and its adjoint. J. Phys. Oceanogr., 34(3), 582–604.

Gent, P. R., and J. C. McWilliams, 1990: Isopycnal mixing in ocean circulation models. J. Phys. Oceanogr., 20, 150–155.

Large, W. G., J. C. McWilliams, and S. Doney, 1994: Oceanic vertical mixing: A review and a model with a nonlocal boundary layer parameterization. Rev. Geophysics, 32, 363.

Large, W. G., G. Danabasoglu, S. C. Doney, and J. C. McWilliams, 1997: Sensitivity to surface forcing and boundary layer mixing in a global ocean model: Annual-mean climatology. J. Phys. Oceanogr., 27, 2418–2447.

Lee, T., and I. Fukumori, 2003: Interannual-to-decadal variations of tropical-subtropical exchange in the Pacific Ocean: Boundary versus interior pycnocline transports. J. Climate, 16(24), 4022–4042.

Marshall, J., A. Adcroft, C. Hill, L. Perelman, and C. Heisey, 1997: A finite-volume, incompressible Navier-Stokes model for studies of the ocean on parallel computers. J. Geophys. Res., 102(C3), 5753–5766.

Redi, M. H., 1982: Oceanic isopycnal mixing by coordinate rotation. J. Phys. Oceanogr., 12, 1154–1158.