Atmospheric-Ocean Dynamics and Tracer Transports
NCC4-624

Milestone I

Complete Distributed Data Broker upgrade and development (include support for MPI and shared memory, improve performance, improve diagnostic and error handling, improve user interface) and deliver updated documentation for the Distributed Data Broker.

 1. Introduction

   The Distributed Data Broker (DDB) is a general-purpose tool that is used to facilitate coupling multiple Earth System Components together. A more detailed description can be found at www.atmos.ucla.edu/~mechoso/esm/ddb.html. We next summarize the major characteristics of the DDB developed under Round 2 of this project.

   The specific function of the DDB is to allow for distributed data exchange and synchronization (via blocking reads) between ESM components. The DDB supports a number of powerful features:
1) Transparent data flow data transfers, 2) multi-source to multi-destination, 3) data flow frequency filtering, 4) missing data with partial domain data filling, 5) multiple rectilinear grid specifications with interpolation, 6) automatic weighted, mean-interpolation method that supports both coarse to fine and fine to coarse grid transformation, 7) multiple data type, and 8) application program transparency.

   The DDB is functionally divided into two consecutive processes: Registration and Execution. There are two phases to the registration process, one to provide top-level information on each of the models (components) and variables it will produce, and a second phase to provide the details on each produce and consume request. At the end of the registration process all nodes and components receive the communication matrix for the data flow strategy that will satisfy the requests that were specified. The matrix contents include where, how much, and how often data is going to be sent and received. Also it includes information on whether interpolation is requested and how to flag missing data values.

   Only two calls need to be used after the registration process is completed. The first call transfers data into the DDB and second request data from the DDB. The data transfer request is non-blocking and the data receive request is blocking. The data receive request is satisfied only when all the data pieces that are available to satisfy the requested data domain, including any stencil data required by the grid transformation process, is available at the receiving node. If a portion of the requested data domain is not available (as determined at registration time), or data points are flagged as invalid by the producing component(s) are indicated, the receiving node treats these as missing data.

2. Design Goals

   Due to central position of the DDB in the Earth System Model (ESM), it can affect the efficiency of the entire application. Therefore, efficiency has been a paramount concern in the design and implementation of the data broker.

   Another important design goal was to enable the data broker to perform concurrent node-to-node communications. This distributed approach has many benefits since it does not require gather/scatter operations and reduces each componentís memory requirement for message and internal buffers. The distributed approach and the way in which the DDB handles messages have the advantage of minimizing the potential for a component to hang due to running out of memory. Thus the DDB is significantly faster than the one node gather/scatter methodology and virtually eliminates the need to throttle data flow to protect against message dead locks.

   Another capability of the data broker is the ability to have multiple consumers for a given product sent at different temporal frequencies. For example sea surface temperatures produced by an ocean model can be sent to an atmospheric model every simulated hour, and at the same time sent to a data archive process every simulated day, and or a real-time visual process(s) at a different temporal resolution.

   The DDB was designed to be flexible and configured dynamically at run time.

3. Code Modifications

   The DDBís basic functionality and user's interface was not changed in the latest round of code improvements. An additional user interface was developed to simplify the meta-Registration process. Both interfaces are documented in appendix A. Changes were made to enhance user input argument checking. All user inputs are now checked for consistency and correct range. Error messages are issued when data does not conform to what is expected.

   Enhancements were made to error message reporting and diagnostic output. Error message texts were modified to clarify the exception condition. Additional error checks were added to catch potential error during the registration phase. Code execution flow trace reports, meta-machine configuration, data flow matrix and other diagnostic output were added or enhanced. Some code problems were corrected, portions of the code were cleaned up, and code self-documenting comments were added.

   The largest change to the code was the addition of an interface library, which was developed to translate the internal calls of the DDB to MPI calls. After much analysis of the DDB and its use of PVM, and many false starts in replacing PVM calls by MPI calls, it was determined that the coding style of the DDB had such reliance on the init-pack-send paradigm that is implicit in PVM, that it was highly impracticable to change the code to use MPI's simpler single-call send paradigm. For this reason, it was decided to write a wrapper library for the PVM calls used in the DDB that translated them into MPI calls. A similar procedure had been used previously in the DDB when it was decided to remove the use of Tcl from an old version of the DDB. In that occasion, a wrapper library that pretended to be Tcl but didn't really make any Tcl calls was used. In this case, the PVM wrapper library now available as source code from the Open Channel Foundation (http://www.openchannelfoundation.org/projects/PVM_Wrapper) was developed to remove the DDB's dependence on PVM. The communications performance using the PVM wrapper is similar to the performance of the code using PVM directly, and does not equal the performance that could be obtained were it possible to use MPI directly. However, rewriting the DDB to use MPI directly would likely require a complete re-write of the full DDB code. This is impractical at this time, though it is advised for the future in order to improve the maintainability of the code.

   Initially the DDB was written to run on a distributed memory machine. During this round modification were made to the DDB to allow it to run on shared memory machines. Additionally, C variable typing was changed in the code to allow the DDB work with both 32 or 64 bit addressing modes. A general purpose link list stack operator (DLLIST) is employed in the DDB. In a few places in the DDB, mismatched variable types (intergers (int) and address pointers(int*)) were used in the invocation of this stack operator. Problems occur when int and int* differ in length and how they are stored and retrieved. To make 32 and 64 bit addressing modes to work; mismatched variable types were identified and the code was modified to remove any variable typing discrepancies.We tested the code on was the SGI Origin 2000 and 3000s shared memory machines.

4. Code verification

   The DDB was initially tested using a stand-alone test program to substantiate that it worked in its one node to one node paradigm. The DDB was tested using the UCLA AGCM and the POP OGCM both in one to one, many to one, and many to many node configurations. The many to many node configurations used the built in Linear Interpretation Library (LIL) to do the grid transformation. Test results and functionality were verified to be consistent with the DDB documentation.

5. Timings

   The timing runs utilized the low-resolution Atmosphere and Ocean model in order to minimize the computation needed to do one coupling interval. We also selected a short coupling interval to achieve a large number of coupling events. The result is to maximize the overhead that is incurred by the coupling the two models together and thus exercise the DDB and its capabilities.

   The following table show the results of these 10 simulated day timing runs and the impact the Distributed Data Broker had on the simulations.

Simulation time (seconds per 10 simulated days)
Number of NodesDDB coupling secondsConventional Coupling SecondsDifference Aggregate node seconds
3119419531
64138144384
96122129672
1281141231152


   The coupling interface utilized for the DDB and conventional coupling simulations where exactly the same. The timing difference is therefore the result of using a distributed approach to perform the coupling. As the number of nodes increase the Distributed Data Broker allows for more efficient communications and parallel regridding. The DDB allows for direct communications between model nodes and providing the stencil data needed to do the grid transformation locally. This eliminates the need for gather/scatter operations and doing a global grid transformation. Communications efficiency and parallel efficiency increases significantly with the number of nodes.

6. Summary

   The DDB paradigm and engineering is a good example of how communications could be done in an Earth System Model. The DDBís power is associated with the fact that one component does not need to know the geometry and data flow of any other component. The consumer/producer paradigm that the DDB uses at registration time determines the geometry and data flows dynamically at run time so modules can be added or removed without modifying any code. The only thing that is modified in the driver code is the number of modules and the communicator IDs of the modules. The communicator IDs can be determined dynamically (from the number of nodes assigned to each module), but the number of modules associated with the meta machine must be provided. Using the meta information provided by each of the modules the DDB can dynamically maps each of the modules together. Using a distributed and concurrent message passing paradigm enhances communications efficiency and reduces memory requirements.

   The following is a list of DDB code upgrades performed for this milestone:
  1. Support both PVM and MPI communications.
  2. The DDB checks the input arguments for consistency and valid range.
  3. The DDB configured meta-machine and data flow matrix can be verified at the end of registration.
  4. Modified portions of the code to enhance code documentation, remove errors, zero out created control data structures, and clarified DDB diagnostic output.
  5. Support for 32 and 64 bit addressing modes.
  6. Simplified user interface.
  7. Revised documentation.
Web Pages

   Additional information about data broker works can be found at http://www.atmos.ucla.edu/~mechoso/esm/ddb.html. The source code can be found at http://esma.atmos.ucla.edu/~ esm/ddb.MAR2004.tar.Z.
Appendix A

FORTRAN INTERFACE


   The Distributed Data Broker (DDB) is invoked via function and subroutine calls. There are separate entry points for C and Fortran. This documentation will only address the direct Fortran interface and a simplified Fortran interface. The direct interface includes the native calls into DDB. The simplified interface is a layer above the direct interface. It gathers user input and then call the native interface to setup the DDB. The simplified interface makes a number of assumptions that will be describe later in this Appendix. The simplified interface can also be used as prototype for using the direct DDB calls.

   The direct and simplified interfaces are used to provide metadata to the DDB during the registration phase. The two calls to actually receive and send data are independent of which registration interface used and only depend on the metadata that is sent to the DDB.

   The first part of this appendix first describes the direct interface calls. This is followed by the simplified interface, and the routines not associated with the registration phase.

A. DIRECT INTERFACE

Registration

   There are 2 phases of registration. One provides top-level information about each component and variable that will be exchanged through the data broker, and a second to provide the details of each produce and consume request.

Metaregistration (first phase):

   Each component must designate a host node. Only this node performs the Metaregistration, which informs the the Registration Broker node of the variables it will produce. Only general information about each component is required at this stage (such as component name, gridding, and number of constituent components). The registration broker is a single node that must be identified by it calling the ëMCLIamRegistrationBroker()í routine before the metaregistration starts.

   After the registration broker is identified, function 'MCLStartMetaRegistration()' must be called first; 'MCLMetaRegister()' must be called for each variable to be produced; and finally 'MCLEndMetaRegistration()' is called to terminate the metaregistration procedure.

MCLStartMetaRegistration:

   This function is called by the designated node from each of the components (i.e. ocean model) that will be participating with the DDB (a single node associated with the component). This routine starts the first phase of the Metaregistration process.

Returns 1 on success, 0 on failure.
Arguments:
    tid         == array of task id s for all nodes that
                   are participating in the exchange of data
                   using the data broker for a particular
                   component.
    numTasks    == length of array 'tid'
    modelNam    == Name of the component.
    numLon      == Number of longitude grid points.
    numLat      == Number of latitude grid points.
    numVert     == Number of longitude grid points.
    lonTicks    == Vector of longitude grid point values.
    latTicks    == Vector of latitude grid point values.
    vertTicks   == Vector of vertical grid point values.


    integer function MCLStartMetaRegistration ( 
               tid,         ! type INTEGER(numTasks)
               numTasks,    ! type INTEGER
               modelName,   ! type CHARACTER*128
               numLon,      ! type INTEGER
               numLat,      ! type INTEGER
               numVert,     ! type INTEGER
               lonTicks,    ! type DOUBLE(numLon)
               latTicks,    ! type DOUBLE(numLat)
               vertTicks )  ! type DOUBLE(numVert)
MCLMetaRegister:

Called for each variable to be produced.

Returns 1 on success, 0 on failure.
Arguments:
    VarName     == Name of the variable.
    nameLen     == Character Length of 'VarName'.
    coordType   == Coordinate type.
                   (For future development, use 0 for now)
                       0 == No vertical interperation
    dataType    == Variable type.
                   The value to data type are:
                       0 == SignedCharType
                       1 == SignedShortType
                       2 == SignedIntType
                       3 == SignedLongType
                       4 == UnsignedCharType
                       5 == UnsignedShortType
                       6 == UnsignedIntType
                       7 == UnsignedLongType
                       8 == FloatType
                       9 == DoubleType
    varType     == DDB internal variable storage type:
                   (For future development, use 0 for now)
                       0 == Temporary
                       1 == Persistent. 
    numDims     == Number of dimensions for this variable.
    frequency   == Number of simulated time units between
                   each production of this variable.

    integer function MCLMetaRegister (
               VarName,     ! type CHARACTER*(nameLen)
               nameLen,     ! type INTEGER
               coordType,   ! type INTEGER
               dataType,    ! type INTEGER
               varType,     ! type INTEGER
               numDims,     ! type INTEGER
               frequency )  ! type DOUBLE
MCLEndMetaRegistration:

Terminates the metaregistration process.

Returns 1 on success, 0 on failure.

No arguments.
    integer function MCLEndMetaRegistration ()
Registration (second phase):

   In the registration phase, each task(node) that will produce or consume data will completely details its requirements. Exact grids are supplied in each request. This data registration phase must be initiated by a call to MCLStartRegistration(), followed by calls to MCLRegisterProduce() and/or MCLRegisterConsume() for each variable produced or consumed. The second registration phase is terminated by a call to MCLEndRegistration().

   All of the Linear Interpolation Library (LIL) calls are made in this second phase of registration. As before, this registration phase must be initiated by a call to StartRegistration() and is is terminated by a call to MCLEndRegistration().

   For all ìproducedî variable(s) a call to MCLRegisterProduce() most be made. If the variable being ìconsumedî requires no interpolation then a call to MCLRegisterConsume() is made. Otherwise, if interpolation is required, then a call to LILRequestInterpolatedData() is required.

   Before a call to LILRequestInterpolatedData() is made a call to LILRegisterCoordinates() is required to describe the grid to interpolate to.

MCLStartRegistration:

Initiates the second registration phase.

Returns 1 on success, 0 on failure.

No arguments.
    integer function MCLStartRegistration ()
MCLRegisterProduce:

Called to inform the data broker that the calling node will produce the given data on the given grid.

On success, this routine returns a non-negative integer id used to uniquely identify this data request. Any other value implies failure.
(This value will be used in subsequent calls to 'MCLSendData'.)
Arguments:
    VarName     == Name of the variable.
    nameLen     == Character length of 'VarName'
    coordType   == Coordinate type.
                   (For future development, use 0 for now)
                       0 == No vertical interperation
    dataType    == Variable type.
                   The value to data type are:
                       0 == SignedCharType
                       1 == SignedShortType
                       2 == SignedIntType
                       3 == SignedLongType
                       4 == UnsignedCharType
                       5 == UnsignedShortType
                       6 == UnsignedIntType
                       7 == UnsignedLongType
                       8 == FloatType
                       9 == DoubleType
    varType     == DDB internal variable storage type:
                   (For future development, use 0 for now)
                       0 == Temporary
                       1 == Persistent. 
    numDims     == Number of dimensions for this variable.
    lon0        == Fortran index into the vector LonTicks
                   for the first longitude point which
                   this node will produces data.
    numLon      == Number of Longitude points, starting at
                   lon0 for which this node will produces
                   data.
    lat0        == Fortran index into the vector LatTicks
                   for the first latitude point which
                   this node will produces data.
    numLat      == Number of Latitude points, starting at
                   lat0 for which this node will produces
                   data.
    numVert     == Number of Vertical levels, starting at
                   Vert(1) for which this node will produces
                   data.
    Vert        == Vector of veritical grid points values
                   at which this node produces data.
    frequency   == Number of simulated time units between
                   each production of this variable.

    integer function MCLRegisterProduce (
               VarName,     ! type CHARACTER*(nameLen)
               nameLen,     ! type INTEGER
               coordType,   ! type INTEGER
               dataType,    ! type INTEGER
               varType,     ! type INTEGER
               numDims,     ! type INTEGER
               lon0,        ! type INTEGER
               numLon,      ! type INTEGER
               lat0,        ! type INTEGER
               numLat,      ! type INTEGER
               numVert,     ! type INTEGER
               Vert,        ! type DOUBLE(numVert)
               frequency )  ! type DOUBLE
MCLRegisterConsume:

Called to inform the data broker that the calling node will consume the given data on the given grid.

On success, this routine returns a non-negative integer id used to uniquely identify this data request. Any other value implies failure.
(This value will be used in subsequent calls to 'MCLGetData'.)
Arguments:
    VarName     == Name of the variable.
    nameLen     == Character length of 'VarName'
    coordType   == Coordinate type.
                   (For future development, use 0 for now)
                       0 == No vertical interperation
    dataType    == Variable type.
                   The value to data type are:
                       0 == SignedCharType
                       1 == SignedShortType
                       2 == SignedIntType
                       3 == SignedLongType
                       4 == UnsignedCharType
                       5 == UnsignedShortType
                       6 == UnsignedIntType
                       7 == UnsignedLongType
                       8 == FloatType
                       9 == DoubleType
    varType     == DDB internal variable storage type:
                   (For future development, use 0 for now)
                       0 == Temporary
                       1 == Persistent. 
    numDims     == Number of dimensions for this variable.
    lon0        == Fortran index into the vector LonTicks
                   for the first longitude point which
                   this node will require data.
    numLon      == Number of Longitude points, starting at
                   lon0 for which this node will require
                   data.
    lat0        == Fortran index into the vector LatTicks
                   for the first latitude point which
                   this node will require data.
    numLat      == Number of Latitude points, starting at
                   lat0 for which this node will require
                   data.
    numVert     == Number of Vertical levels, starting at
                   Vert(1) for which this node will
                   require data.
    Vert        == Vector of veritical grid points
                   at which this node require data.
    frequency   == Number of simulated time units between
                   each production of this variable.

    integer function MCLRegisterProduce (
               VarName,     ! type CHARACTER*(nameLen)
               nameLen,     ! type INTEGER
               coordType,   ! type INTEGER
               dataType,    ! type INTEGER
               varType,     ! type INTEGER
               numDims,     ! type INTEGER
               lon0,        ! type INTEGER
               numLon,      ! type INTEGER
               lat0,        ! type INTEGER
               numLat,      ! type INTEGER
               numVert,     ! type INTEGER
               Vert,        ! type DOUBLE(numVert)
               frequency )  ! type DOUBLE
LILRegister Coordinates:

Called to inform the DDB's interpolation library that the node will be requesting data in terms of the supplied coordinate system. A node may make more than one calls to LILRegisterCoordiates to register multiple coordinate systems. This allows for data for a particular variable to be interpolated to the appropriate coordinate system.

On success, this routine returns a non-negative integer id used to uniquely identify this coordinate grid. Any other value implies failure.
(This value will be used in subsequent calls to 'LILRequestInterpolatedData'.)
Arguments:
    VarName     == Name of the coordinate system.
    numLon      == Number of Longitude points in the variable
                   lonTicks.
    numLat      == Number of Latitude points in the variable
                   latTicks.
    numVert     == Number of Vertical levels specified in
                   the variable vertTicks.
    lonTicks    == Longitude vector, of length 'numLon'.
    latTicks    == Latitude vector, of length 'numLat'.
    vertTicks   == Vertical vector, of length 'numVert'.

    integer function LILRegisterCoordinates (
               VarName,     ! type CHARACTER*128
               numLon,      ! type INTEGER
               numLat,      ! type INTEGER
               numVert,     ! type INTEGER
               lonTicks,    ! type DOUBLE(numLon)
               latTicks,    ! type DOUBLE(numLat)
               vertTicks )  ! type DOUBLE(numVert)
Notes:    At this time vertical interpolation is not implemented.

LILRequestInterpolatedData:

Called to inform the data broker that the calling node will consume the given data on the given grid.

On success, this routine returns a non-negative integer id used to uniquely identify this data request. Any other value implies failure.
(This value will be used in subsequent calls to 'MCLGetData'.)
Arguments:
    VarName     == Name of the variable.
    nameLen     == Character length of 'VarName'
    coordToken  == Coordinate ID defined by
                   LILRegisterCoordinates.
    dataType    == Variable type.
                   The value to data type are:
                       0 == SignedCharType
                       1 == SignedShortType
                       2 == SignedIntType
                       3 == SignedLongType
                       4 == UnsignedCharType
                       5 == UnsignedShortType
                       6 == UnsignedIntType
                       7 == UnsignedLongType
                       8 == FloatType
                       9 == DoubleType
    varType     == DDB internal variable storage type:
                   (For future development, use 0 for now)
                       0 == Temporary
                       1 == Persistent. 
    numDims     == Number of dimensions for this variable.
    lon0        == Fortran index into the vector LonTicks
                   for the first longitude point which
                   this node will require data.
    numLon      == Number of Longitude points, starting at
                   lon0 for which this node will require
                   data.
    lat0        == Fortran index into the vector LatTicks
                   for the first latitude point which
                   this node will require data.
    numLat      == Number of Latitude points, starting at
                   lat0 for which this node will require
                   data.
    numVert     == Number of Vertical levels, starting at
                   Vert(1) for which this node will
                   require data.
    vert0       == Fortran index into the vector LatTicks
                   for the first latitude point which
                   this node will require data.
    numVert     == Number of Vertical levels, starting at
                   vert0 for which this node will
                   require data.
    TickModulii == Wrapping modulus value.
    frequency   == Number of simulated time units between
                   each production of this variable.

    integer function MCLRegisterProduce (
               VarName,     ! type CHARACTER*(nameLen)
               nameLen,     ! type INTEGER
               coordToken,  ! type INTEGER
               dataType,    ! type INTEGER
               varType,     ! type INTEGER
               numDims,     ! type INTEGER
               lon0,        ! type INTEGER
               numLon,      ! type INTEGER
               lat0,        ! type INTEGER
               numLat,      ! type INTEGER
               vert0,       ! type INTEGER
               numVert,     ! type INTEGER
               TickModulii, ! type DOUBLE(3)
               frequency )  ! type DOUBLE
Notes:    It is assumed all participants will use the same units for all grid scales. The TickModulus vector is used to indicate which, if any, of the coordinates wrap, and what the modulus value for the wrap. If wrapping is desired in the East-West direction and Ticks are given in degrees, the TickModulus vector would be (360.0D0, 0.0D0, 0.0D0). Accordingly, if wrapping is required in both the latitudinal and longitudinal directions and the units are in radians then (6.328431853D0, 3.1415928D0, 0.0D0) should be used.

   All variables exchanged between two components must have the same values for TickModulus.

LILSetMissingDefault:

Called after issuing a request for interpolated data, a specific sentinel (missing data) value to be supplied for any data that cannot be interpolated or any grid points that are outside produced data domains.

Returns 0 when successful, negative if the supplied ID is not known or is for a field of data type other than Float or Double.
Arguments:
    id          == Variable ID that is returned by
                   LILRequestInterpolatedData.
    value       == Either a float or a double word to be used
                   for the sentinel value.

    integer function LILSetMissingDefault (
               id,          ! type INTEGER
               value )      ! type REAL or DOUBLE
Note:    Value type is specified by the variable meta data associated with id token.

MCLEndRegistration:

Ends the second registration phase.

Returns 1 on success, 0 on failure.

No arguments.
    integer function MCLEndRegistration ()
B. SIMPLIFIED INTERFACE

   The simplified interface has 5 entry points which collect the metadata and one utility routine which provides all the routines that are necessary to interface with DDB except for the explicit data import/export calls (MCLGetData and MCLSendData). The simplified interface assumes that all variables to be transferred are type DOUBLE and that the Linear Interpolation Library will be used.

StartRegistration:

Called to start the Meta-Registration process.
Arguments:
    tid         == array of task id s for all nodes that
                   are participating in the exchange of data
                   using the data broker for a particular
                   component.
    numTasks    == length of array 'tid'
    modelTids   == array of task id s of the designated
                   host for each of the components.
    numModels   == Number of components taking part in
                   this meta machine.
    runName     == Name of this run, eg "Configuration A"
    modelName   == Name of the component.
    startTime   == Start time for the components in the run
    endTime     == End time for the components in the run
                   (estimated if not known)
    wrap        == Wrap modulii vector.

    subroutine StartRegistration (
               RB_tid,      ! type INTEGER
               tid,         ! type INTEGER(numTasks)
               numTasks,    ! type INTEGER
               runName,     ! type CHARACTER*(*)
               startTime,   ! type DOUBLE
               endTime,     ! type DOUBLE
               numModels,   ! type INTEGER
               modelName,   ! type CHARACTER*(*)
               wrap )       ! type DOUBLE(3)
RegVarProduce:

Called to inform the data broker that the calling node will produce the given data on the given grid.
Arguments:
    VarName     == Name of the variable.
    numDims     == Number of dimensions for this variable.
    lon0        == Fortran index into the vector LonTicks
                   for the first longitude point which
                   this node will produces data.
    numLon      == Number of Longitude points, starting at
                   lon0 for which this node will produces
                   data.
    lat0        == Fortran index into the vector LatTicks
                   for the first latitude point which
                   this node will produces data.
    numLat      == Number of Latitude points, starting at
                   lat0 for which this node will produces
                   data.
    vert0       == Fortran index into the vector VertTicks
                   for the first vertical level which
                   this node will produces data.
    numVert     == Number of Vertical levels, starting at
                   vert0 for which this node will produces
                   data.
    frequency   == Number of simulated time units between
                   each production of this variable.

    subroutine RegVarProduce (
               VarName,     ! type CHARACTER*(nameLen)
               numDims,     ! type INTEGER
               lon0         ! type INTEGER
               numLon       ! type INTEGER
               lat0         ! type INTEGER
               numLat       ! type INTEGER
               vert0        ! type INTEGER
               numVert      ! type INTEGER
               frequency )  ! type DOUBLE
RegVarConsume:

Called to inform the data broker that the calling node will consume the given data on the given grid.
Arguments:
    VarName     == Name of the variable.
    numDims     == Number of dimensions for this variable.
    lon0        == Fortran index into the vector LonTicks
                   for the first longitude point which
                   this node will consume data.
    numLon      == Number of Longitude points, starting at
                   lon0 for which this node will consume
                   data.
    lat0        == Fortran index into the vector LatTicks
                   for the first latitude point which
                   this node will consume data.
    numLat      == Number of Latitude points, starting at
                   lat0 for which this node will consume
                   data.
    vert0       == Fortran index into the vector VertTicks
                   for the first vertical level which
                   this node will consume data.
    numVert     == Number of Vertical levels, starting at
                   vert0 for which this node will consume
                   data.
    frequency   == Number of simulated time units between
                   each consumption of this variable.
    missingValue == The sentinel value to indicate missing 
                    data.
    coordName   == The grid coordinate name.

    subroutine RegVarConsume (
               VarName,     ! type CHARACTER*(*)
               numDims,     ! type INTEGER
               lon0,        ! type INTEGER
               numLon,      ! type INTEGER
               lat0,        ! type INTEGER
               numLat,      ! type INTEGER
               vert0,       ! type INTEGER
               numVert,     ! type INTEGER
               frequency,   ! type DOUBLE
               missingValue ! type DOUBLE
               coordName )  ! type CHARACTER*(*)
RegCoord:

Called to inform the DDB's interpolation library that the node will be requesting data in terms of the supplied coordinate system. A node may make more than one calls to RegCoord to register multiple coordinate systems. This allows for data for a particular variable to be interpolated to the appropriate coordinate system. The component's export grid is defined by a special 'coordName' value of "supply". All variables produced will be on this grid.
Arguments:
    coordName   == Name of the coordinate.
    numDims     == Number of dimensions for this variable.
    lonTicks    == Longitude vector, of length 'numLon'.
    numLon      == Number of Longitude points in the variable
                   lonTicks.
    numVert     == Number of Vertical levels specified in
                   the variable vertTicks.
    latTicks    == Latitude vector, of length 'numLat'.
    numLat      == Number of Latitude points in the variable
                   latTicks.
    vertTicks   == Vertical vector, of length 'numVert'.
    numVert     == Number of Vertical levels specified in
                   the variable vertTicks.

    subroutine RegCoord (
               coordName,   ! type CHARACTER*(*)
               lonTicks,    ! type DOUBLE(numLon)
               numLon,      ! type INTEGER
               latTicks,    ! type DOUBLE(numLat)
               numLat,      ! type INTEGER
               vertTicks,   ! type DOUBLE(numVert)
               numVert )    ! type INTEGER
EndRegistration:

Ends the simplified registration processes.

No arguments.
    subroutine EndRegistration ()
C. NON META-REGISTRATION INTERFACES

MCLIAmRegistrationBroker:

Only called by the node designated as the Registration Broker! Need to be the first call made by the direct meta registration process. This routine does not need to be called for the simplified registration process. The registration broker has no function after the registration process is completed ‚ it is not used by the produce/consume calls.

Returns 1 on success, 0 on failure.
Arguments:
    id          == The id of variable to consume.
    runName     == Name of this run, eg "Configuration A"
    startTime   == Start time for the components in the run
    endTime     == End time for the components in the run
                   (estimated if not known)
    modelTids   == Task id s of the designated model nodes.
    numModels   == Number of components taking part in this
                   run.

    integer function MCLIAmRegistrationBroker (
               runName,     ! type CHARACTER*(*)
               startTime,   ! type DOUBLE
               endTime,     ! type DOUBLE
               modelTids,   ! type INTEGER(numModels)
               numModels )  ! type INTEGER
MCLGetData:

Called by a consumer to get data. Note that this is a blocking call and will only return when the data is available.

Returns 1 on success, 0 on failure.
Arguments:
    id            == The id of variable to consume.
    buf           == Buffer/Array containing the data.
    timeStamp     == Time-stamp in time units associated with
                     this data.

    integer function MCLGetData( 
               id,          ! type INTEGER
               buf,         ! type specified during meta
                            ! registration
               timeStamp )  ! type DOUBLE
MCLSendData:

Called by a producer to make data available to consumers.

Returns 1 on success, 0 on failure.
Arguments:
    id          == The id of variable to consume.
    buf         == Buffer/Array containing the data.
    timeStamp   == Time-stamp in time units associated with
                   this data.

    integer function MCLSendData ( 
               id,          ! type INTEGER
               buf,         ! type specified during meta
                            ! registration
               timeStamp )  ! type DOUBLE
MCLSetTracelevel:

Sets the MCL debugging and tracing level, for diagnostic output.

No return value.
Arguments:
    id          == The id of variable to consume.
    level       == Level of tracing.
                   The value to level are:
                      -1 == No Output except for Critical
                            errors
                       0 == Critical messages only
                       1 == Warnings
                       2 == More warnings
                       3 == Flow; function entry and exit
                       4 == Maximum level

    subroutine MCLSetTracelevel ( 
               level )      ! type INTEGER
MCLPoll:

MCLPoll checks for and handles pending messages destined for the data broker. It can be called intermittently to prevent the data broker messages from piling up...

Returns the number of messages processesed.

No arguments.
    integer function MCLPoll ()
MCLTidyOut:

MCLTidyOut redirects standard error (stderr) and standard out (stdout) to a file for each node calling this routine. The output file is named mclt.n where n is the number that is passed to MCLTidyOut. This routine is very useful for debugging in that it separates the messages generated by each node into it's own file.

No return value.
Arguments:
    id          == Is a unique number representing the
                   node (i.e. node number).

    subroutine MCLTidyOut ( 
               id )      ! type INTEGER
var_id:

   A utility function that is called to return the id associated with a variable name. The variable id returned is used for MCLGetData and MCLSendData. This routine is valid only when the simplified interface is used.
Arguments:
    VarName       == Name of the variable.

    integer function var_id (
               VarName )    ! type CHARACTER*(*)