We have applied TAU's profiling capabilities to evaluate the performance of the Uintah computational framework (UCF) [2]. The case study simulates a capped metal cylinder filled with high-energy material and suspended above a pool of hydrocarbon fuel burning with an open flame. Energy from the flame is transported through the metal cylinder to the high-energy material, causing it to undergo complex chemical changes. Solid deformations, deterioration and cracking occur in both the cylinder and the high-energy material as pressure within the cylinder builds, eventually leading to rupture and detonation. To implement this simulation, a CFD component that simulates hydrocarbon combustion and reactant transport is coupled with a component that uses the material point method (MPM) to simulate the mechanics of solid deformation and energy transport within the cylinder.
The TAU profiling strategy for Uintah is to observe the performance of the framework at the level of patches, the unit of domain partitioning. Thus, we instrument UCF with dynamic phases where the phase name contains the patch identifier. Figure 1 shows the performance data obtained from the simulation as displayed by ParaProf. In this profile, we can distinguish the time taken by a computation ``task' (such as MPM::InterpolateParticlesToGrid) when it was executed within a given patch. It shows the callgraph of this simulation where the width of each node is mapped to the inclusive time spent in that routine. The color of the box shows the exclusive time spent in that routine (blue represents a low cost, red a high cost). As seen in this ParaProf figure, we can partition the time spent in different routines based on the calling phase. The full paper will show other parallel profile displays that highlight features of TAU's phase based profiling.
Besides the Uintah computational framework, TAU has been applied successfully to several frameworks that are used for computational fluid dynamics simulations. These include VTF from Caltech, MFIX from NETL, ESMF coupled flow application from UCAR, NASA and other institutions, SAMRAI from LLNL, Miranda from LLNL, GrACE from Rutgers University, SAGE from SAIC, and Flash2 from University of Chicago. Though the nature of instrumentation and the performance data presented from TAU differs in each framework, our full paper will show examples of how TAU is applied in these frameworks. Our work in performance evaluation of adaptive scientific computations can be broadly applied to other CFD codes. Thus, CFD frameworks can benefit from the integration of portable performance profiling and tracing support using TAU.
