Source-Level Instrumentation

For other programming languages that the TAU performance system supports (C, C++, Fortran), standard routine entry/exit instrumentation is supplemented by the ability to specify ``user-defined'' performance events. These events can be associated with any code points the user desires. TAU provides an API to define the events, and to start and stop event profiling around code sections, including individual statements. However, in the first version of our Java JVMPI-based instrumentation, we were only able to see Java method invocation events. The definition and profiling of user events at the Java source level was not possible.

To accomplish this in TAU's current implementation for Java, we developed a source-level API in the form of a TAU Java package for creating user-level event timers. This API is consistent with similar capabilities TAU provides for other languages. The user can define events timers of a TAU.Profile class and then annotate the source code at desired places to start and stop the timers. Below is a example code segment demonstrating the API's use:

 import TAU.*;

 // Create timer
 static TAU.Profile t= new TAU.Profile("Tau Timer", 
   "test", "TAU_DEFAULT", TAU.Profile.TAU_DEFAULT);

 t.Start();
 // Code segment here
 t.Stop();

The TAU Java package provides the API, but utilizes JNI to interface with the TAU profiling library. This library is implemented as a dynamic shared object that is loaded by the JVM or the TAU Java package. It is within the TAU profiling library that the performance measurements are made. However, TAU captures performance data with respect to nodes and threads of execution. What makes Java source-level instrumentation interesting is that node identification and JVM thread information is not accessible at the Java language level. Where does TAU get this information?

To maintain a common performance data repository in which performance data from multiple ``streams'' comes together and presents a consistent picture, we need instrumentation at various levels to co-operate. As shown in Figure 1, the TAU profiling library uses JNI to interface with the JVMPI layer to determine which JVM thread of execution is associated with Java method events and with MPI events. In the same manner, TAU uses this mechanism to determine thread information for user-defined events at the source level. To determine node information, TAU queries the MPI library to find out its process rank.

Thus, TAU instrumentation occurs at the Java source level, at the MPI wrapper library level, and at the virtual machine level. These different layers together form a consistent view of the execution model and thus must synchronize effectively to maintain the multi-threaded performance data in a consistent state.

 [bigger image]

Table 1: TAU overhead for the parallel Java application Life