Increasing the Interoperability of an Earth System Model:
Atmospheric-Ocean Dynamics and Tracer Transports
NCC4-624

Milestone A: Software Engineering Plan Completed
Software Engineer: John R. Baumgardner
Fluid Dynamics Group
Theoretical Division
Mail Stop B216
Los Alamos National Laboratory
Los Alamos, NM 87545

Description of the Application

1. The ESM

1.1 ESM description

      Our current Earth System Model (ESM) comprises four models representing the coupled dynamics, physics and chemistry of the global atmosphere and world oceans: 1) an atmospheric general circulation model (UCLA AGCM), 2) an oceanic general circulation model (Parallel Ocean Model: POP), 3) an atmospheric chemistry and transport model (UCLA ACTM), and 4) a JPL Ocean Chemical Transport Model (OCTM). Data exchanges are carried out through a novel Distributed Data Broker (DDB). This project is based on the AGCM, POP (also in a version with a hybrid vertical coordinate or HYPOP), and the DDB, which are described in the following paragraphs. Another ocean general circulation model developed at MIT will be incorporated as an option.1.2 ESM area of effort
      The ESM will be implemented and optimized on NASA computers, and a documented source code will be made publicly available via the Web. Team members will conduct meetings with ESMF development teams on the design of the interface between the ESM and ESMF. The ESM model components will be integrated into the ESMF framework. The ESMF functionality will be demonstrated by analyzing the El Niño prediction capability of the AGCM coupled to HYPOP and the MIT OGCM, in combination with NASA/JPL ocean data and optimization products.

2. The AGCM

2.1 AGCM description

      In the UCLA AGCM the planetary boundary layer (PBL) processes are parameterized using the mixed-layer approach of Suarez et al. (1983). The formulation of moist processes in the PBL and moisture exchange with the layer above has been recently revised (Li et al., 1999; Li et al., 2002), resulting in an improved simulation of the geographical distribution and optical properties of PBL stratocumulus clouds. Parameterization of cumulus convection, including its interaction with the PBL, follows Arakawa and Schubert (1974) with a prognostic closure following Pan and Randall (1998). The effects of convective downdrafts and vertical momentum and rainwater budgets are included in the cumulus parameterization (Cheng and Arakawa, 1997). The current model version also includes an implementation of the prediction scheme for cloud liquid water and ice due to Köhler (1999). The parameterization of gravity wave drag due to subgrid-scale orography follows Kim (1996). The parameterizations of shortwave and longwave radiative heating are those of Harshvardhan et al. (1989) and Harshvardhan et al. (1987, 1989) respectively. The model has versions with 43⁄4 lat. by 53⁄4 long., 15-layer and 23⁄4 lat. by 2.53⁄4 long., 29-layer. In this project we plan to use the higher resolution version. A more detailed description of the UCLA AGCM can be found in Mechoso et al. (2000) and electronically at http://www.atmos.ucla.edu/esm/agcmdir.
      The AGCM code has been highly optimized for massively parallel computer architectures (Mechoso et al., 1998) including most recently the IBM SP. The parallel version of the code is based on two-dimensional (longitude-latitude) data domain decompositions and uses MPI for message-passing (Wehner et al. 1995, Mechoso et al. 1998). The optimization effort has been made as part of our Round 2 project. A major component of this work was the development of methods for balancing on a Massively Parallel Processor Architecture the computational load of a distributed climate model (Lou and Farrara, 1998). Using the same resolution we plan for the experiments to be performed under this project, the AGCM achieved a performance of approximately 40 Gflops on 512 processors of the CRAY T3E-600 (at NASA Goddard Space Flight Center) with a parallel efficiency of 45%.

2.1 AGCM area of effort

      The parameterization of the planetary boundary layer (PBL) used in the AGCM will be upgraded to a version with multiple layers. This development is considered to be crucial for a better representation of low-level cloudiness in the tropics.

3. The OGCM

3.1 OGCM description

      The Parallel Ocean Program (POP) is a descendent of the Bryan-Cox-Semtner class of ocean models first developed by Kirk Bryan and Michael Cox at the NOAA Geophysical Fluid Dynamics Laboratory in Princeton, NJ, in the late 1960's. POP had its immediate origins in a version of the model developed by Bert Semtner and Bob Chervin at NCAR. Experience with this version led to a number of changes resulting in what is now known as POP. Details of these changes can be found in articles by Smith et al. (1992), Dukowicz et al. (1993), and Dukowicz and Smith (1994). The model has continued to develop to adapt to new machines, incorporate new numerical algorithms and introduce new physical parameterizations.
      The most important algorithmic modification in POP involved the treatment of the barotropic mode. The barotropic streamfunction formulation in the standard Bryan-Cox-Semtner models requires an additional equation to be solved for each continent and island that penetrate the ocean surface. This was computationaly costly even on parallel-vector-processor computers, which had fast memory access. To reduce the number of equations to solve with the barotropic streamfunction formulation, it was common practice to submerge islands, connect them to nearby continents with artificial land bridges, or merge an island chain into a single mass without gaps. On distributed-memory parallel computers, these added equations were even more costly because each required gathering data from an arbitrarily large set of processors to perform a line-integral around each landmass. This computational dilemma was overcome by a new formulation of the barotropic mode based on surface pressure. The boundary condition for the surface pressure at a land-ocean interface point is local, which eliminates the non-local line-integral. Consequently, the surface-pressure formulation permits any number of islands to be included at no additional computational cost, and all channels between islands can be treated as precisely as the resolution of the grid permits. The surface-pressure formulation also allows more realistic, unsmoothed bottom topography to be used with no reduction in time step. This alleviates the difficulty in the barotropic streamfunction formulation that the elliptic problem to be solved is ill-conditioned when bottom topography has large spatial gradients. In addition, the original “rigid-lid” boundary condition was replaced by an implicit free-surface boundary condition that allows the air-sea interface to evolve freely and makes sea-surface height a prognostic variable.
      Another significant feature of POP is that the primitive equations were reformulated and discretized to allow the use of any locally orthogonal horizontal grid. This provides alternatives to the standard latitude-longitude grid with its singularity at the North Pole. This generalization made possible the development of the displaced-pole grid, which moves the singularity arising from convergence of meridians at the North Pole into an adjacent landmass such as North America, Russia or Greenland. Such a displaced pole leaves a smooth, singularity-free grid in the Arctic Ocean. That grid joins smoothly at the equator with a standard Mercator grid in the Southern Hemisphere. The most recent versions of the code also support a tripole grid in which two poles can be placed opposite each other in land masses near the North Pole to give more uniform grid spacing in the Arctic Ocean while maintaining all the advantages of the displaced pole grids.
      POP is written in Fortran90 and can be run on a variety of parallel and serial computer architectures. The most recent version of the code supports current clusters of shared-memory multi-processor nodes through the use of thread-based parallism (OpenMP) between processors on a node and message-passing (MPI or SHMEM) for communication between nodes. The flexibility of mixing thread-based and message-passing programming models gives the user the option of choosing the best combination of styles to suit a given machine.
      In the period 1994-97, POP was used to perform high resolution global ocean simulations, running on the Thinking Machines CM5 computer then located at LANL's Advanced Computing Laboratory. Output from global ocean simulations are available at http://climate.acl.lanl.gov and http://neit.cgd.ucar.edu/oce/bryan/woce-poster.html (see also Maltrud et al., 1998; Smith et al., 2000 and Washington et al., 2000). Recently, computer resources have become available to undertake a 0.1 degree global simulation and this calculation is in process.
      POP and the Los Alamos elastic-viscous-plastic sea-ice dynamics model have been coupled to the NCAR Community Climate Model (CCM) atmospheric and land-surface models, to form the Parallel Climate Model (PCM). This model is being used for climate research and global-warming studies (Washington et al., 2000). POP and the full sea-ice model (CICE) have also been adopted as the ocean and sea ice components of the Community Climate System Model (CCSM) at NCAR. POP and CICE are also being used in coupled model development efforts at Colorado State University.3.1 OGCM area of effort
      The OGCM will be upgraded to include the version with a hybrid vertical coordinate (HYPOP). Also, the MIT OGCM will become an optional component of the ESM.

4. The DDB

4.1 DDB description

      The DDB is a general purpose tool for coupling multiple, possibly heterogeneous, parallel models. It is implemented as a library used by all participating elements, one of which serves as a distinguished process during a startup phase preceding the main computation (Sklower et al., 2002). This “registration broker” process correlates offers to produce quantities with requests to consume them, forwards the list of intersections to each of the producers, and informs each consumer of how many pieces to expect. After the initial phase, the registration broker may participate as a regular member of the computation. A library of data translation routines is included in the DDB to support exchanges of data between models using different computational grids. Having each producer send directly to each consumer conserves bandwidth, reduces memory requirements, and minimizes the delay that would otherwise occur if a centralized element were to reassemble each of the fields and retransmit them. 4.2 DDB description
      A major upgrade of the DDB will be completed. This will include support for MPI and shared memory. The upgrade will also include improvements in performance, diagnostic and error handling, and the user interface.

Risk assessment

      The project plans to practice risk management to anticipate, mitigate, and control risks to the project. All items with high and medium impact risks will generate mitigation plans that will be closely monitored by the P.I and Software Engineer. There will be monthly meetings to review the status of each part of the project and identify critical areas that are behind schedule or have problems that need to be addressed. The meetings will be in person after each milestone is achieved; otherwise they will be by phone conference. The risk assessment and mitigation strategy will be updated by adding newly identified risks, adjusting probability of impact, etc. as the project progresses, and an updated risk report will be produced. This risk report will include risk name, description, level of impact, probability of impact, type of mitigation (avoid, mitigate, or accept). If mitigation action is appropriate, a brief description of the action and current status of the action will be included (see Table 2).

Table 2. Initial list of risks and risk mitigation strategies

Name	Description of Risk	Level of Impact (H, M, L)	Prob. of Impact (H, M, L)	Type of Mitigation (Avoid, Mitigate, Accept)	Mitigation Actions
PBL parameterization for AGCM	Timely availability of upgraded parameterization	M	L	Mitigate	The current version of the AGCM can be used to continue working on the science and technology components of the project, particularly the former. It is possible to increase human effort dedicated to the project component in order to speed it up.
El Nino simulation	Poor performance of AGCM coupled to MIT OGCM	M	H	Mitigate	The UCLA atmospheric model (AGCM) has not yet been coupled to the MIT ocean model (OGCM). The simulated climate is likely to be poor in the first coupled runs. The amount of work required to fix this problem is difficult to predict.
HYPOP	Timely availability of HYPOP	M	L	Mitigate	HYPOP designers at LANL will be made aware of the need for a timelyavailability of their model. Additional resources will be sought to enhance the ocean modeling component.
El Nino prediction	Poor model simulation of El Nino. If model simulation of ENSO is poor, then predictions of ENSO may be unreliable and science findings unreliable	H	M	Mitigate	Model parameterizations will be revised/tuned to improve performance.
ESMF	Timely availability of ESMF interface	M	L	Mitigate	Milestone J depends on availability of ESMF. Work on the science goals of the project can continue with an ESM framework in which components are coupled using the DDB.
Integration of ESM into ESMF	Modifications of ESM for coupling to ESMF	H	H	Mitigate	Strengthen exchange of information between ESMF developers and this project. It is too early for a better definition of this risk.

Configuration Management

      The ESM software will be managed using the tool Concurrent Versions System (CVS). CVS provides a repository for code, checkout of code, a history of changes made to the code, and a commit feature to prevent developers from making contradictory changes to the same code. The ESM test scripts and test data will also be managed using CVS. Additional documentation on CVS can be obtained from http://www.gnu.org/manual/cvs-1.9/cvs.html.
      The ESM’s assigned gatekeeper and backup gatekeeper to accept changes to the baseline version of the software are John D. Farrara and Joseph A. Spahr, respectively. The project will have a documentation repository on the public web site. The released version of each document will be displayed on the site.

Storage, Handling, and Delivery

      Institutional support for automatic, nightly incremental backups is available at both UCLA and LANL. Additionally, since the repository is expected to be no larger than approximately 500 megabytes in size, we plan to employ writable CD-ROMs to produce full snapshots of the repository at no less than weekly intervals. Backups will be kept off-site at a secure location to be determined by the configuration manager. We also plan to use the Internet as our primary medium for software distribution.

References