The BREEZE Remote Modeling System (BRMS) for AERMOD operates on a massively parallel computer cluster that combines the processing power of multiple multi-core computers.
Multi-core computers have more than one processing unit, or CPU, on a computer chip.
Each core can be viewed as a separate computer capable of performing independent
calculations. Expert air quality modelers are anxious
to save time and money by utilizing scalable, high- performance computing (HPC)
applications to process the numerically intensive algorithms and data often associated
with dispersion models. The Remote Modeling System utilizeds the BREEZE AERMOD Parallel Fortran application,
which debuted in June 2007 at the
Air and Waste Management Association’s (AWMA)
annual conference.
The Need for Speed
As air quality modeling software and associated preprocessing applications
become more complex to address the physical and chemical processes occurring in
the atmosphere, multi-core computers or clusters are becoming more attractive. Software
designed to utilize the full computing power of these systems allows modelers to
complete model runs in a reasonable amount of time. Currently, serial versions of
the AERMOD air dispersion model available from the U.S. EPA can take anywhere from
a few seconds to complete for a very simple scenario, to over a month or longer
for a complex multi-source, multi-receptor scenario (e.g., SIP or PSD-related scenarios).
Although most modelers now have computers with at minimum dual-core CPU, the dispersion
modeling applications available from EPA do not utilize these capabilities. This
effectively reduces the return on investment for a multi-core CPU. To take advantage
of multi-core computers, the user requires applications that are specifically designed
to run in that environment. True parallel computer applications do this by sending
each core/CPU its own set of instructions simultaneously.
The U.S. EPA serial-version AERMOD dispersion model may be required to make thousands
of computations for each hour of meteorological data processed. Each computation
is completed sequentially, one at a time. Serial processing can be thought of as
if each computation were to line up and file through a single door (where the door
is the CPU). Parallel programs, such as BREEZE AERMOD Parallel, essentially open
more doors (i.e., CPU’s) by sending multiple computations simultaneously to multiple
cores/CPU’s.
The U.S. Department of Defense (DoD) recognized Trinity’s ability to express numerical
algorithms at a higher level of abstraction than those currently employed by publicly
available models and contracted Trinity’s BREEZE software developers to develop
high performance computing parallel versions of applications such as CALPUFF, CALMET,
and SCIPUFF to run on standard PC’s utilizing the Windows or Linux operating systems.
To address DoD’s need for more efficient modeling systems and ensure that these
modeling systems complied with EPA’s Guideline on Air Quality Models (e.g., no changes
to model algorithms), Trinity initiated Project HPCM, the High Performance Computing
Modeling project. Project HPCM’s goal was to identify parallelization techniques
that not only met the needs of the DoD but that could also be used more widely to
produce results identical to those produced by EPA’s public models. A high priority
objective for developing parallel versions of these dispersion models was to generate
unobtrusive code which, when applied to an existing serial program such as AERMOD,
generates a functionally equivalent parallel version (i.e., identical results).
The Challenge
Until recently, parallel programming has been largely restricted to either
purely academic activities or to super-computer systems. Advances in personal computer
technology have made the availability of multi-core/CPU computers mainstream; however,
many authors of serial processing codes have been slow to adapt their applications
to the capabilities of these systems due to the cost and level of effort involved
to properly parallelize code. A significant challenge associated with developing
efficient parallel computer models is the routing of individual computations to
multiple processors without affecting model results. Achieving identical results
to EPA’s public models was the top priority for developers of BREEZE AERMOD Parallel.
BREEZE AERMOD Parallel developers were charged with not only developing a version
of the AERMOD application that optimized the use of multi-cores/CPU’s (e.g., Intel’s
CoreTM Duo processor), but also to extend the processing across multiple, networked
computers, such as in a computing cluster. A cluster refers to a network of independent
computers linked together to form a unified system with the objective of completing
a common task. Clusters are often built using commodity hardware on a private network
with open source operating systems (e.g., Linux), thereby making available powerful
computational resources to those with even the most modest budgets. Simply building
a cluster does not afford a modeler the advantages of a multi-core/CPU system. The
design of the software used on the system is critical. Programs that are poorly
architected or communication-intensive can run even more slowly on clusters or multi-core/CPU
machines than their serial counterparts.
A significant number of environmental regulatory agencies worldwide currently run
numerically intensive applications such as AERMOD and CALPUFF on serial computers
and are faced with roadblocks that prevent them from taking advantage of parallel
architectures. One issue is that users who could benefit most from parallel processing
may lack the knowledge necessary to extract optimal performance from their available
computing resources. Moreover, they tend to have large legacy systems running on
serial machines, and re-writing this legacy code can represent an enormous cost.
The generation of good parallel code is not a simple task. Developing optimized
code requires substantial skill and a significant mind-shift from how serial code
is prepared. BREEZE developers have produced parallel code that can provide substantive
value to dispersion modelers. The following equivalency demonstration is provided
to communicate the equivalency between BREEZE AERMOD Parallel and the EPA’s public
domain AERMOD application.