Alexander Shchepetkin Institute of Geophysics and Planetary Physics University of California at Los Angeles 405 Hilgard Ave., Los Angeles CA 90095-1567 e-mail: alex@atmos.ucla.edu alias: old_galaxy

Isopycnic diffusion: Click here to get the tar file for source codes (750 KBytes).

Click here to get the tar file for Gmetas (18 MBytes).

Scientific interests

Nonlinear processes in geophysical fluid dynamics; Vortex dynamics ; Advanced numerical methods for convectively dominated flows; Numerical treatment of contact discontinuities; Computer simulation of turbulent flows; Multigrid methods and their applications in fluid mechanics; Regional and basin scale ocean modeling; Supercomputing and parallel processing;

Ongoing project: Regional Ocean Modeling System

This is a collaborative effort with Hernan Arango and Dale Haidvogel from ocean modeling group of Institute of Marine and Coastal Sciences, Rutgers university.

Re-engineering SCRUM

The goal here is to develop a new model which is capable to provide reliable and more realistic high-resolution simulations for coastal and basin scale configurations and which runs on the newest available family of SGI/CRAY Origin 2000 supercomputers. A descendant/successor of SPEM and then SCRUM family topography following S-coordinate ocean models, the new model, called regional Ocean Modeling System (ROMS) drastically expands the capabilities of the existent SCRUM-3.0 code in terms of numerical/computational performance, grid resolution, as well as update of numerical techniques and parametrizations of unresolved physical processes.

At first, the SCRUM-3.0 code was redesigned in order to comply with the logically shared/physically distributed memory architecture of the Origin 2000 and its synchronization strategy based the write-back memory bus protocol and the global cache coherency mechanism. The new model employs the explicit coarse-grained MP parallelization paradigm, with subdomain partitioning (blocking) in both horizontal dimensions. The explicit MP model is a shared aproach, which implies that the parallel threads are created only once and exist during the whole run time of the program. Each parallel thread is self-aware in sense that it determines by itself which subdomain(s) [tile(s)] it needs to process (rather than, say, waits until the master thread will tell it what to do) and it is also aware of the presence of the other threads. Unlike classical message passing codes, the number of the subdomains is not necessarily equal to the number of CPUs used, but rather chosen in such a way that the storage segment associated with a subdomain fits into the processor cache. Thus, if one needs to run a larger problem on a given number of CPUs, it often turns out that it is more advantageous to increase the number of subdomains, while approximately maintaining their size, rather than keep the number of subdomains, while letting their size to increase. Explicit calls to barrier and lock functions are used to occasionally synchronize the parallel threads. Distribution of the model arrays across the multiple nodes of Cray Origin is done relying on the "first touch" default distribution policy and the affinity between memory placement and workload partitioning is kept consistent throughout the whole duration of the model run.

Going beyond: from parallel version of SCRUM to ROMS

Unlike ocean model parallelization projects of the past, transition from SCRUM to ROMS is not just a rewrite of the code for the purpose of gaining computational speed. Once grid resolution is increased, and the amount of dissipation in the model is decreased accordingly, the flow regime is changing: it becomes turbulent and processes of exchange due to vortex interactions become explicitly resolved. In this situation the typical low second order accuracy finite difference numerical schemes (which were used in ocean modeling for last 30 years) are no longer the methods of choice. Higher-order accuracy (typically third- and fourth-order) schemes, which are better tolerant to the non-smoothness of the fields on the grid scale, are used instead.

It is interesting to note that, this choice is also encouraged by the evolution of the computer architectures, due to fact that the processing power of the machines tends to increase faster than the memory bandwidth. Consequently, more sophisticated methods, which result into more computationally dense programs utilize the resources of the machine in a more balanced way, while the more traditional second-order accurate methods, which were optimal for the computers of the past turned out to be memory bound on the modern machines.

Future Plans

Building embedded coarse-fine grid coupling capability; Data Assimilation capability; Biological/ecosystem model driven/coupled with the existent physical model;

Related links: Patrick Marchesiello , Emanuele Di Lorenzo (see also here ), Art Miller , Bruce Cornuelle, John Moisan, Olaf Haupt .

	W A R N I N G ROMS is under construction and is always going to be this way. Make sure that you read and understand our Disclaimer of Liability You may also want to read this

Atlantic DAMEE project

Thought, ROMS was designed mostly for high-resolution coastal configurations, Atlantic basin-scale simulations have became its first proving ground. Here are some results from our Atlantic "region" runs (yeah, we do have ambition to build a model for the global region!).

Fig. 1 Velocity at surface. Note that despite a relatively coarse resolution of 3/8 degree, we have obtained a reasonably good pattern of the Gulf Stream separation, which detaches off Cape Haterras and meanders. This particular simulation was without any explicit viscosity or diffusivity in horizontal direction, relying only on the hyperdiffusive truncation error of a third-order accurate [in space and time] upstream biased advection schemes, applied for both horizontal momentum and the tracer [temperature and salinity] equations. Click on image to display it in double size

Also note the remarkable nonlinear activity in the equatorial region.

Fig. 2 Velocity at depth of 500 meters. In comparison with Fig. 1, one can no longer see the intense equatorial currents: these are too shallow, and generally limited to the upper 200 meters. One can still see, however, weak traces of the Equatorial undercurrent going to the west. In contrast to that, deep current along Labrador coast is visually intensified. It mets Gulf Stream and literally pushes it away from the coast. Click on image to display it in double size

Also note that the anticyclonic surface circulation in the Gulf of Mexico is no longer seen here; instead the deep cyclonic circulation is more pronounced here.

Note that, compared to Fig. 1, velocity scale has been changed approximately in three times, so that even if some vectors look darker than on Fig. 1, they are not necessarily larger.

Fig. 3 Velocity at depth of 1500 meters. One can no longer see Gulf Stream. Instead note the North Atlantic Countercurrent, heading all the way from the Labrador Peninsula to Brazilian coast, crossing the equator. Click on image to display it in double size

We believe that to a large degree this current topographically controlled. Scrum, as a topography following S-coordinate model has no problem to simulate this current correctly, even at such, relatively coarse resolution.

Also note the presense of topographically trapped circulations, especially in deep regions of Gulf of Mexico and Carribean. These are generally cyclonic. It is possible, that these are explained as topographically rectified flows obtained from varying forcing. We cannot explain everything we see, however.

Velocity scale has been reduced again. Deep currents are generally much less intense.

Fig. 4 Temperature at 50 meters. SUMMER. Click on image to display it in double size

Fig. 5 Temperature at 50 meters. WINTER. Click on image to display it in double size

Fig. 6 Temperature at 500 meters. The same time as on Fig. 5 above. Click on image to display it in double size

This computation was performed in the National Centre for Supercomputing Applications (NCSA) in Urbana-Champaign, Illinois.

Links related to this project : Data Assimilation and Model Evaluation Experiments

US West Coast Modelling Project

US West Coast simulation is the main target for which ROMS is designed. The key guy here is Patrick Marchesiello, who hapened to be the first user of the ROMS since it was created. But since then he became one of the developers. Here I am working mostly on mathematical and computational aspects, while Patrick works on the physical formulation of the problem and the model configuration (although in many cases the border the border between these two groups is not well defined).

Since Patrick is a married man, he does not have time to maintain his own web page. That is why all these pictures ended up here.

Here are few snapshots of sea surface temperature (SST) illustrating seasonal cycle. Model configuration is: curvilinear with grid resolution approximately 10 km in both directions, approximately isotropic; dimensions of the grid are 96 x 256 x 30 points, which results in approximately 250 MBytes of storage; the model is forced by COADS monthly climatological winds as well as surface heat and effective salt fluxes (fresh water evaporation/precipitation); no explicit viscosity and diffusion (except in sponge layers near the radiation open boundaries): for this purpose we rely exclussively on the build-in hyperdiffusive dissipation inherent for the third-order upstream biased advection schemes used for both the momentum and the tracer equations. It was run starting from the Middle of the May Levitus climatology fields for temperature and salinity and zero initial velocities, hence it goes through geostrophic adjustment after startup. It was spun up for two years, and starting from the third year the result were analysed. Day on the plots below means Julian day.

This computation was performed using Cray Origin 200 in Scripps Institution of Oceanography. We thank Peter Niiler for providing this opportunity.

History

May 1995 - present Programmer Analyst in Center for Earth System Research (CESR), Institute of Geophysics and Planetary Physics (I.G.P.P.), University of California at Los Angeles.
Jan 1994 - Apr 1995 -- Post Doctoral Research Associate in Center for Ocean-Atmospheric Prediction Studies (COAPS), Florida State University.
Aug 1992 - Nov 1993 -- Post Doc in the Department of Oceanography, Florida State University.

Education

1992 Ph.D. in Mechanics of Fluid, Gas and Plasma from Moscow Institute of Physics and Technology (MPTI), Dolgoprudny, Moscow region, Rossia.
1988 Degree of engineer-physicist from MPTI.

Publications

A.F.Shchepetkin and J.C.McWilliams, 1998, Quasi-monotone Advection Schemes Based on Explicit Locally Adaptive Dissipation; Monthly Weather Review, 126 , pp. 1541-1580. Click here for the abstract and preprint.

I. Yavneh, A. F. Shchepetkin, J. C. McWilliams and L. P. Graves, 1997, Multigrid Solution of Rotating, Stably Stratified Flows: The Balance Equations and Their Turbulent Dynamics; Journal of Computational Physics, 136, pp. 245-262.

Shchepetkin, A.F., J. J. O'Brien, 1996, A Physically Consistent Formulation of Lateral Friction in Shallow Water Equation Ocean Models, Monthly Weather Review, 124, 1285-1298.
Click here for the abstract and preprint.

Shchepetkin A.F., 1995, Interaction of Turbulent Barotropic Shallow-Water Flow with Topography, 1995 Proceedings Aha Huliko'a Hawaiian Winter Workshop , edit. P. Muller and D. Henderson, Honolulu, HI, pp. 225-237.

Meacham, S.P., K.K. Pankratov, A.F. Shchepetkin and V.V.Zhmur, 1994, The Interaction of Arbitrarily Oriented Ellipsoidal Vortices in a Continuously Stratified Fluid With Background Shear and Strain flows, Dynamics of Atmospheres and Oceans, 21, pp. 167-212.

Zhmur,V.V., Shchepetkin, A.F., 1992, Interaction Between Two Quasigeostrophic Vortices: Tendency to Come Together and Merge, Atmospheric and Oceanic Physics. Russian Academy of Sciences, 28, No 5, pp. 407-416 (English edition).

Zhmur,V.V., Shchepetkin,A.F. 1991, Evolution of an Ellipsoidal Vortex in a Stratified Ocean in the f-plane Approximation, Atmospheric and Oceanic Physics, Russian Academy of Sciences, 27, No 5, pp. 331-346 (English edition).

Postscript file of my CV is available here.