TAU Tracing in POOMA

Tracing generates a log of events that took place in each node for an application. These event traces can be viewed by tools such as VAMPIR .

Settings

Configure POOMA with TRACE option.
% cd $POOMA_ROOT % conf O2K64 MPI TRACE % cd $POOMA_ROOT/lib/$POOMA_ARCH % make

Tracing can be used with or without Profiling. TAU Tracing package comes with some utilities for trace processing. Install these as follows:

% cd $POOMA_ROOT/src/Profile/utils % make
Add to your .cshrc file
set path=($path $POOMA_ROOT/src/Profile/utils)

It is important to make the utilities with the same set of options as the library (-32 or -64) as it deals with binary files.

Generating and Processing Traces

Event traces could be large and so using the scratch partition is recommended for speed and size constraints. Set the environment variable TRACEDIR to specify the directory in which the trace files should be generated.
% setenv TRACEDIR /scratch/dixiebutte/sameer/sindex % cd $POOMA_ROOT/test/sindex % make sindex % mpirun -np 4 sindex --commlib mpi

Selective Tracing enables the user to limit the size of the traces to manageable sizes and focus attention on relevant parts of the application.

% mpirun -np 4 sindex --commlib mpi --profile sparse+field+io+utils

This generates trace files named tau.<nodeid>.trc and event definition files events.<nodeid>.edf .

To merge the traces to a single binary trace file named say sindex.trc, use
% tau_merge tau.*.trc sindex.trc

This generates a single merged event definition file named tau.edf and the binary trace sindex.trc as specified on the command line. To convert this single trace to an ascii text trace in VAMPIR trace format named say sindex.pv , use

% tau_convert -pv -compact sindex.trc tau.edf sindex.pv

Note: tau_convert could be used to analyse unmerged single traces as well. It can be used to see the raw dump of the trace using the following command :

To see the trace of node 2, use
% tau_convert -dump tau.0002.trc events.2.edf

It can also convert the trace to other file formats such as SDDF and ALOG.

Using VAMPIR

On ASCI BlueMountain Origin 2000s, add to your .cshrc file the following lines:

# For VAMPIR (PARvis) Trace browser setenv PAL_ROOT /usr/local/pooma/vampir setenv PAL_LICENSEFILE /usr/local/pooma/vampir/etc/license.dat set path=($path /usr/local/pooma/vampir/bin)

To view the trace file cd to the trace directory and invoke vampir. A 24 bit display is recommended to view the function groups in distinct colors.