The CT macro returns the runtime type information string of a variable. This is useful in constructing the type parameter of the TAU_PROFILE macro. For templates, the type information can be constructed using the type of the return and the type of each of the arguments (parameters) of the template. The example in the following macro will clarify this.

This macro assigns the string constructed in type_string to the variable. The + operator and the CT macro can be used to construct the type string of an object. This is useful in identifying templates uniquely, as shown below.

When PLayout is instantiated with "UniformCartesian<3U, double>", this generates the unique template name :

The following example illustrates the usage of the CT macro to extract the name of the class associated with the given object using CT(*this);

When PLayout is instantiated with "UniformCartesian<3U, double>", this generates the unique template name:

With TAU_PROFILE_TIMER, a group of one or more statements is profiled. This macro has a timer variable as its first argument, and then strings for name and type, as described earlier. It associates the timer to the profile group specified in the last parameter.

The macro TAU_PROFILE_START starts the timer associated with the set of statements that are to be profiled.

The macro TAU_PROFILE_STOP stops the timer.

TAU_PROFILE_START(sendtimer);

TAU_PROFILE_STOP(sendtimer);

TAU_PROFILE_STMT declares a variable that is used only during profiling or for execution of a statement that takes place only when the instrumentation is active. When instrumentation is inactive (i.e., when profiling and tracing are turned off as described in Chapter 2), all macros are defined as null.

TAU_PROFILE_INIT parses the command-line arguments for the names of profile groups that are to be selectively enabled for instrumentation. By default, i.e., if this macro is not used, functions belonging to all profile groups are enabled.

The TAU_PROFILE_SET_NODE macro sets the node identifier of the executing task for profiling and tracing. TAU uses the HPC++ model of computation, where the tasks are identified using node, context and thread. The profile data files generated will accordingly be named
profile.<node>.<context>.<thread>.

TAU_PROFILE_SET_CONTEXT sets the context parameter of the executing task for profiling and tracing purposes. This is similar to setting the node parameter with TAU_PROFILE_SET_NODE.

To register a thread with the profiling system, invoke the TAU_PROFILE_REGISTER_THREAD macro in the run method of the thread prior to executing any other TAU macro. This sets up thread identifiers that are later used by the instrumentation system.

TAU_PROFILE_EXIT should be called prior to an error exit from the program so that any profiles or event traces can be dumped to disk before quitting.

TAU can profile user-defined events using TAU_REGISTER_EVENT. The meaning of the event is determined by the user.

TAU_EVENT associates a value with some user-defined event. When the event is triggered and this macro is executed, TAU maintains statistics, such as maximum, minimum values, standard deviation, number of samples, etc. for tracking this event.

TAU_REPORT_THREAD_STATISTICS prints the aggregate, as well as per thread user event statistics. Typically, this should be called just before the main thread exits.

TAU_TRACE_RECVMSG traces a receive operation where tag represents the type of the message received from the source process.

TAU allows the user to map performance data of entities from one layer to another in a multi-layered software. Mapping is used in profiling (and tracing) both synchronous and asynchronous models of computation. For mapping, the following macros are used. First locate and identify the higher-level statement using the TAU_MAPPING macro. Then, associate a function identifier with it using the TAU_MAPPING_OBJECT. Associate the high level statement to a FunctionInfo object that will be visible to lower level code, using TAU_MAPPING_LINK, and then profile entire blocks using TAU_MAPPING_PROFILE. Independent sets of statements can be profiled using TAU_MAPPING_PROFILE_TIMER and TAU_MAPPING_PROFILE_START and TAU_MAPPING_PROFILE_STOP macros using the FunctionInfo object.

TAU_MAPPING is used to encapsulate the C++ statement that we want to map to some other layer. The other layer can execute synchronously or asynchronously with respect to this statement. The key corresponds to a number that the lower layer will use to refer to this statement. For example,

Array <2> A(N, N), B(N, N), C(N,N), D(N, N);
//Original statement:
A = B + C + D;
//Instrumented statement:
TAU_MAPPING(A = B + C + D; , TAU_USER);

To create storage for an identifier associated with a higher level statement that is mapped using TAU_MAPPING, we use the TAU_MAPPING_OBJECT macro. For example, in the TAU_MAPPING example, the array expressions are created into objects of a class ExpressionKernel, and each statement is an object that is an instance of this class. To embed the identity of the statement we store the mapping object in a data field in this class. This is shown below:

TAU_MAPPING_OBJECT(TauMapFI)

TAU_MAPPING_LINK creates a link between the object defined in TAU_MAPPING_OBJECT (that identifies a statement) and the actual higher-level statement that is mapped with TAU_MAPPING. The Key argument represents a profile group to which the statement belongs, as specified in the TAU_MAPPING macro argument. For the example of array statements, this link should be created in the constructor of the class that represents the expression. TAU_MAPPING_LINK should be executed before any measurement takes place. It assigns the identifier of the statement to the object to which FuncIdVar refers. For e.g.:

TAU_MAPPING_LINK(TauMapFI, TAU_USER)

The TAU_MAPPING_PROFILE macro measures the time and attributes it to the statement mapped in TAU_MAPPING macro. It takes as its argument the identifier of the higher level statement that is stored using TAU_MAPPING_OBJECT and linked to the statement using TAU_MAPPING_LINK macros. TAU_MAPPING_PROFILE measures the time spent in the entire block in which it is invoked. For example, if the time spent in the run method of the class does work that must be associated with the higher-level array expression, then, we can instrument it as follows:

TAU_MAPPING_PROFILE(TauMapFI)

TAU_MAPPING_PROFILE_TIMER enables timing of individual statements, instead of complete blocks. It will attribute the time to a higher-level statement. The second argument is the identifier of the statement that is obtained after TAU_MAPPING_OBJECT and TAU_MAPPING_LINK have executed. The timer argument in this macro is any variable that is used subsequently to start and stop the timer.

TAU_MAPPING_PROFILE_START starts the timer that is created using TAU_MAPPING_PROFILE_TIMER. This will measure the elapsed time in groups of statements, instead of the entire block. A corresponding stop statement stops the timer as described next.

TAU_MAPPING_PROFILE_STOP stops the timer associated with the mapped lower-level statements. This is used in conjunction with TAU_MAPPING_PROFILE_TIMER and TAU_MAPPING_PROFILE_START macros. For example:

TAU_MAPPING_PROFILE_TIMER(timer, TauMapFI);

TAU_MAPPING_PROFILE_START(timer);

TAU_MAPPING_PROFILE_STOP(timer);

The API for instrumenting C source code is similar to the C++ API. The difference is that the TAU_PROFILE() macro is not available for identifying an entire block of code or function. Instead, routine transitions are explicitly specified using TAU_PROFILE_TIMER() macro with TAU_PROFILE_START() and TAU_PROFILE_STOP() macros to indicate the entry and exit from a routine. Note that, TAU_TYPE_STRING() and CT() macros are not applicable for C. It is important to declare the TAU_PROFILE_TIMER() macro after all the variables have been declared in the function and before the execution of the first C statement.

call TAU_PROFILE_INIT()

To profile a block of Fortran code, such as a function, subroutine, loop..., the user must first declare a profiler, which is an integer array of two elements (pointer) with the save attribute, and pass it as the first parameter to the TAU_PROFILE_TIMER subroutine. The second parameter must contain the name of the routine, which is enclosed in a single quote. TAU_PROFILE_TIMER declares the profiler that must be used to profile a block of code. The profiler is used to profile the statements using TAU_PROFILE_START and TAU_PROFILE_STOP as explained later. For example:

TAU_PROFILE_START starts the timer for profiling a set of statements. The timer (or the profiler) must be declared using TAU_PROFILE_TIMER routine, prior to using TAU_PROFILE_START.

TAU_PROFILE_STOP stops the timer used to profile a set of statements. It is used in conjunction with TAU_PROFILE_TIMER and TAU_PROFILE_START subroutines.

c--------------------------------------------------
integer profiler(2)

call TAU_PROFILE_TIMER(profiler, "setbv")

call TAU_PROFILE_START(profiler)

call TAU_PROFILE_STOP(profiler)

The TAU_PROFILE_SET_NODE macro sets the node identifier of the executing task for profiling and tracing. TAU uses the HPC++ model of computation, where the tasks are identified using node, context and thread. The profile data files generated will accordingly be named
profile.<node>.<context>.<thread>.

TAU_PROFILE_SET_CONTEXT sets the context parameter of the executing task for profiling and tracing purposes. This is similar to setting the node parameter with TAU_PROFILE_SET_NODE.

To register a thread with the profiling system, invoke the TAU_PROFILE_REGISTER_THREAD routine in the run method of the thread prior to executing any other TAU routine. This sets up thread identifiers that are later used by the instrumentation system.

TAU_PROFILE_EXIT should be called prior to an error exit from the program so that any profiles or event traces can be dumped to disk before quitting.

TAU can profile user-defined events using TAU_REGISTER_EVENT. The meaning of the event is determined by the user. The first argument to TAU_REGISTER_EVENT is the pointer to an integer array. This array is declared with a save attribute as shown below.

TAU_EVENT associates a value with some user-defined event. When the event is triggered and this macro is executed, TAU maintains statistics, such as maximum, minimum values, standard deviation, number of samples, etc. for tracking this event.

TAU_REPORT_THREAD_STATISTICS prints the aggregate, as well as per thread user event statistics. Typically, this should be called just before the main thread exits.

TAU_TRACE_SENDMSG traces an inter-process message communication when a tagged message is sent to a destination process.

TAU_TRACE_RECVMSG traces a receive operation where tag represents the type of the message received from the source process.

In C++, a single macro viz., TAU_PROFILE, is sufficient to profile a block of statements. In C, and Fortran the user must use statement level timers to achieve this, using TAU_PROFILE_TIMER, TAU_PROFILE_START and TAU_PROFILE_STOP. Instrumentation of C++ source code can be done manually or by using tau_instrumentor, a tool that can automatically insert TAU annotations in the source code. We are working on a Fortran 90 instrumentor, which should be available in the near future.

After instrumentation and compilation are completed, the profiled application is run to generate the profile data files. These files can be stored in a directory specified by the environment variable PROFILEDIR as explained in Chapter 2. By default, all instrumented routines and statements are measured. To selectively measure groups of routines and statements, we can use the command-line parameter --profile to specify the statements to be profiled. For example:

The above profiles routines belonging to TAU_IO, TAU_FIELD and TAU_USER2 profile groups. For a detailed list of groups, please refer to
URL: http://www.acl.lanl.gov/tau (Docs->Tutorial->Selective Profiling section)

Performance counters exist on modern microprocessors. These count hardware performance events such as cache misses, floating point operations, etc. while the program executes on the processor. Performance Counter Library (PCL, URL:http://www.fz-juelich.de/zam/PCL/) package provides a uniform interface to access these performance counters. TAU uses PCL to access these hardware performance counters. To use PCL, download and install PCL. Then, configure TAU using the -pcl=<dir> configuration command-line option to specify the location of PCL. Build TAU and applications as you normally would (as described in Chapters 2 and 3). While running the application, set the environment variable PCL_EVENT to specify which hardware performance counter TAU should use while profiling the application. For example to measure the floating point operations in routines,

pprof sorts and displays profile data generated by TAU. To view the profile, merely execute pprof in the directory where profile files are located (or set the PROFILEDIR environment variable).

In the project management window, select a project by typing in a project file name with a .pmf extension (e.g., matrix.pmf), and clicking (the first mouse button) on the "Create" button.

NOTE: If the project window does not appear, it is probably due to problems with the security of the X-display. You may not use xhost + while using racy. You may need to explicitly turn off xhost by invoking xhost - and using Xauthentication. The cookies are generated and stored in the ~/.Xauthority file. If your server is not configured to generate these cookies, please contact your system administrator for configuring the X display to make it secure. Users may also want to use ssh, rather than telnet/rlogin to login to a remote node where racy is invoked. If this is done, the cookies need not be copied to the remote node explicitly by the user. For more information please refer to :

URL: http://www.acl.lanl.gov/tau/docs/faq.html#security

If user-defined events are profiled in addition to routines, then an event window appears with the function profile window. Navigation of user event profiles is identical to the navigation of function profiles as described above. The following figure demonstrates the use of user-defined event profiles from the PaRP project (URL: http://www.csi.uoregon.edu/parp)

TAU must be configured with the -TRACE option to generate event traces. This can be used in conjunction with -PROFILE to generate both profiles and traces. The traces are stored in a directory specified by the environment variable TRACEDIR, or the current directory, by default. For example:

tautrace.<node>.<context>.<thread>.trc and events.<node>.edf

Using the utility tau_merge, these traces are then merged as shown below:

To convert merged or per-thread traces to another trace format, the utility tau_convert is used as shown below:

tau_convert can also be used to convert traces to the Vampir trace format (URL: http://www.pallas.de). For single-threaded applications (such as the MPI application above), the -pv option is used to generate Vampir traces as follows:

When multiple threads are used on a node (as with -pthread or -tulipthread options during configure), the -vampir option is used to convert the traces to the vampir trace format, as shown below:

NOTE: To ensure that inter-process communication events are recorded in the traces, in addition to the routine transitions, it is necessary to insert TAU_TRACE_SENDMSG and TAU_TRACE_RECVMSG macro calls in the source code during instrumentation. This is not needed when the TAU MPI Wrapper library is used.

Vampir is used to display scheduling of work packets or iterates in SMARTS, the Shared Memory Asynchronous Runtime System (URL: http://www.acl.lanl.gov/smarts) :

http://www.acl.lanl.gov/tau

Sameer Shende <sameer@cs.uoregon.edu>