Going Parallel the New Way

By Ilya Mirman

Scientists and engineers in virtually every field are turning to high-performance computers to simulate and solve some of today’s most important and complex problems. This trend is driven by three key factors. First, the models, algorithms, and phenomena are becoming more complex. Second, Moore’s law has arguably stalled for the typical desktop’s single-core CPU;  advancements in computing power will likely come from putting multiple processors to work simultaneously rather than increasing clock speeds. And third, cost-effective parallel computing hardware has become mainstream, replacing what was once a niche market of multi-million dollar supercomputers. According to market research firm IDC,  the high-performance technical computing (HPC) market has grown by 49 percent over the last two years and will top $8 billion in 2006, with sales in the hundreds of thousands of systems.

The “Software Gap”

Unfortunately, parallel programming is difficult and the “software gap” that exists between new hardware capabilities and actual benefits from programming is wide and growing. There is a dearth of applications available for parallel computers, and custom development of parallel applications today is fundamentally flawed.

In its 2005 report, the President’s I.T. Advisory Committee—which advises the federal government on IT R&D efforts—said, “systems are difficult to program and their achieved performance is a small fraction of theoretical peak … New programming models and languages and high-level, more expressive tools must hide architectural details and parallelism.”

Consider the typical process for developing a custom parallel application (Figure 1, below). The specification is typically some combination of a prototype program written in a desktop-based high-level application (e.g., MATLAB) and “prose” that attempts to capture the particular model, system, or algorithm. A team of parallel programming specialists then writes the application in C, Fortran, and MPI (message passing interface) used for inter-processor communication and synchronization; relatively complex low-level programming.

› › Figure 1:   This image shows a batch workflow of parallel application development.

 

Only after the application is developed for the HPC server can it be executed to allow testing and scaling with the real data. Because each step can take months, scientists and engineers are limited to two or three iterations they can make to the algorithms and models.

This process is slow, inflexible, remarkably error-prone, and expensive. First, the specialists—of whom there is a serious shortage—must correctly interpret the end user’s specification. The end user must then interpret the test results to understand whether problems encountered are due to the original specification or the parallel programming. In many cases, the correct algorithm, approach, or key to the problem may not be known up front, and may typically be discovered only by running the code on the HPC with the actual input data. And it is highly likely that a change in the user’s specification—due to errors, changed algorithms, changed application requirements, or changed hardware—will require a major rewrite by the specialists.

Furthermore, because batch runs can take hours, days, or even weeks, the iteration and refinement of the parallel application is severely limited. When the goal is reducing time-to-solution, interactivity is critical because the engineer’s and scientist’s time is precious. While interactivity is built into desktop science and engineering tools,  it has not been available in HPC, which remains firmly in the batch world.

For all these reasons, it is not uncommon to spend 75 percent of time-to-solution programming the models for use on HPC servers, rather than developing and refining them up front, or using them during production to make decisions and discoveries.

The Interactive Workflow

New tools are now becoming available that bridge the previously distinct computing environments of desktop workstations and parallel servers. Both environments offer much, but the growing chasm between them must be bridged in order to solve the growing array of computational challenges facing scientists and engineers.

Today, the desktop is the focal point of computing in science and engineering, particularly during the early stages of new product or system modeling, simulation, and optimization. The interactivity offered by these tools aid the iterative process of research and discovery. Millions of engineers and scientists have at their disposal a rich set of interactive high-level applications that enable development of custom applications.

‹ ‹ Figure 2:   An illustration of interactive development of parallel applications.

 

These very high-level languages (VHLLs)—e.g., MATLAB,  Mathematica, IDL, Excel, Maple, and Python—have largely supplanted Fortran and C as the scientific and engineering tools of choice. These desktop tools make it easy to manipulate high-level objects (e.g.,  matrices), hiding many of the underlying low-level programming complexities from users. They also support rapid iteration and refinement by enabling an interactive development and execution environment.

In the past decade substantial research has gone into ways to extend these familiar desktop tools to parallel computers, with more than 30 projects focused on MATLAB alone. The approaches have varied significantly in features, performance, and ease of use. Though most have not made it out of academia, several commercial options have emerged.

So, what are the considerations in choosing a parallel programming solution? First, it must enable you to write applications in your favorite VHLL. Second, it must automatically parallelize the code,  effectively hiding the challenges of parallel programming. Third, it must be able to run the custom application interactively on HPC servers. And fourth, it must be extensible, enabling the reuse of existing serial and parallel applications and libraries.

Such an approach (Figure 2, above) lets you prototype and scale in a tightly coupled process, in real time, with fine-grained control of both algorithms and data, transparently harnessing the HPC’s computing resources. You can write just enough of the application to start testing with real data, as you incrementally refine the application. In other words, with an interactive workflow, the time to “first calculation” (and in a sense “partial satisfaction”) can be within minutes, rather than the months or years required to first program the parallel application. 

› ›  Figure 3:   Star-P is among the first commercial platforms to bridge desktop MATLAB and HPC.

 

Interactive usage would allow you to observe the application run, stop it, change parameters, and potentially decide to further refine the application. Because learning happens only through feedback, incremental refinement and continuous testing are key. The interactive workflow could potentially save years for a project while dramatically increasing the feedback cycles during development and testing of the custom parallel application, resulting in higher quality models, algorithms, decisions,  and products.

An Early Adopter

Bryan Wong is an early adopter of mainstream parallel computing. As a researcher at Massachusetts Institute of Technology’s Department of Chemistry, he is performing computationally intensive modeling of molecules that play a central role in the dynamics of weather and pollution. As the mathematical model’s size and complexity outgrew his desktop computer, Wong had two options: learn the details of MPI-based parallel programming and spend months writing the C++ code, or extend his MATLAB code to a parallel computer using the Star-P interactive parallel computing platform from Interactive Supercomputing. Wong chose the latter (Figure 3, above), using MATLAB on his PC, in conjunction with Star-P, to run advanced simulations on a 16-processor SGI Altix high-performance server. As a result, Wong worked in his familiar environment, leveraged his MATLAB code, and carried out calculations (Figure 4, below) that were impossible to do with serial MATLAB alone.

“Our molecular models involved computations with 20,000 x 20,000 matrices, so we quickly reached the limits of desktop memory and computing power,” explained Wong. “With Star-P, we could work interactively within our MATLAB environment, extending our computations to larger and more accurate molecular models without the need for C++ and MPI.”

‹ ‹ Figure 4:   MATLAB-based Molecular Simulation.

 

The emergence of new custom programming tools for scientists and engineers is an enormously promising development. Hiding parallel programming difficulties while enabling a tightly coupled interactive environment will be critical to expanding access to—and unlocking the potential of—parallel computers. After decades of being viewed as an expensive niche and the domain of specialists, mainstream parallel computing is finally here, and HPC servers can now be used just as interactively as desktop PCs.

Ilya Mirman is a vice president at Interactive Supercomputing (ISC). Prior to joining ISC, he was a VP  at SolidWorks, helping establish it as a standard in 3D mechanical design software. Mirman has a BS in mechanical engineering from the University of Massachusetts, an MS in mechanical engineering from Stanford, and an MBA from MIT’s Sloan School. Send your feedback about this article through e-mail by clicking here. Please reference “Going Parallel, June 2006” in your message.

Company Information

Star-P
Interactive Supercomputing—ISC
Waltham, MA

MATLAB
The MathWorks
Natick, MA

Silicon Graphics, Inc.—SGI
Mountain View, CA