Hi, my name is Sameer Shende from the University of
Oregon and I'll describe the TAU Performance System.
The TAU performance system is a profiling and tracking
toolkit that can show you how much time is being spent
in your application code regions such as routines and
outer loops, statements, or openmp loops. Instead of
time, you can see that the total number of instructions
executed. You could use LIKWID or PAPI to measure
floating-point instructions level 1 and 2 Data
cache misses and other counters. You can see the
time taken in operating system routines for thread
scheduling and measure how much time is really being
wasted. It can show you the memory usage of your code
or the I/O or the contribution of each phase of your
application or even how it scales.
TAU has three main components instrumentation
or adding hooks to your code, measurement to create
profiles or traces, and Analysis where you can visualize
the data that is generated. It supports different languages
including Fortran, C, C++, Java, Python, UPC. And a number of different
run times. It can interface with a measurement library
such a Score-P to do generate OTF2 traces and cubex profiles.
It can also connect to a TAUdb database
and supports the 3D profile browser in ParaProf and
data analysis toolkit, PerfExplorer.
instrument different runtimes including OpenCL, CUDA, OpenACC
It supports ROCm is supports
the Kokkos profiling API and also Python and you
can mix and match these run times. So you could measure
the time spent in MPI with OpenMP or CUDA or Kokkos
with OpenMP and so on it does this with two different
ways. You can use direct instrumentation via
probes with a start-and-stop call to a timer or
indirectly using sampling. With sampling, you do
not need to modify the code. You can just get a periodic
interrupt whether it is from a hardware performance
counters or the wall clock timer and then it can correlate
the samples to the source locations. And create
different types of profiles besides flat profiles
or callpath profiles that show you the time spent along an edge of a call graph, or a call site profile
which shows how much time was spent
at a given location or even a phase profile. Now we
do this using a tool called tau_exec. We can launch the
application. So instead of launching the MPI application
using mpirun a.out, we use mpirun tau_exec a.out and it then can intercept the runtime
system calls for the MPI or openmp or other libraries and
introduce other options such as event-based sampling.
So you do this by launching the code using tau_exec tool
and you can use the tau_exec -ebs flag to launch it
with event based sampling. It works like this if an uninstrumented
application Is launched with mpirun you can just
say mpirun tau_exec and then give some options
such as -ompt to instrument the OpenMP tools interface or -ebs for event-based sampling.
A number of different options exist for gpus.
When you configure TAU with different packages
it creates a shared directory with tags in its name
And these tags are also present in the TAU Makefile
includes the configuration parameters in a stub
Makefile. So if you configure TAU with PDT and PAPI,
it will create this configuration and without
PAPI, it will create this and you can launch tau_exec with
the -T option to specify those tags. You can
also launch tau_exec with the options for run
times such as -rocm, -cupti, -opencl, -openacc, -io profiling
and so on. There are other runtime environment variables to generate traces, call path
profiles, memory footprint profiles. You can track
a number of system quantities like this and track
memory and debugging options. You can also instrument
the source code using the program database toolkit (PDT) where you can
parse the source code and then add hooks to this in
a copy of the instrumented source code typically when
you install TAU, you can point to the PDT package and
turn on other options such as turning on openmp tools interface
or specifying the names of compilers and
using -bfd=download. BFD uses the information
in your program to go to correlate the addresses to the
source locations and you can set the TAU_MAKEFILE
environment variable and change the compilers
to use the TAU compiler scripts or instead just use
an un-instrumented binary with tau_exec. To install
TAU on your laptop, please follow these directions.
So when you configure TAU you can typically load a TAU module on a system
and set the TAU Makefile and
then replace the compilers, create the instrumented
executable and launch it and then use the pprof and paraprof tools. Here are examples of the tags that I
I mentioned earlier that you can
choose a makefile like this and also set compile
time options using the TAU_OPTIONS environment
variable. Once you generate the performance data,
you can analyze it using paraprof and paraprof can
connect to the TAUdb database. There are other tools such as
perfexplorer that we will also demonstrate.
I would like to mention that this work is being done
at the University of Oregon in Eugene. And I would
like to thank our sponsors from the Department of
Energy, Department of Defense, National Science
Foundation. NASA, CEA France, and our partners at various institutions.
This work was supported by the Exascale Computing
Project. We will now look at the Hands-On exercises.
cd tutorial. make clean; make suite; cd to the
bin directory? So I can say export
OMP_NUM_THREADS=2. And normally I would run
this as mpirun -np 4 ./bt-mz.W.4. You can see that it's taking quite
a bit of time. Now, I should check the settings on the
VirtualBox image. I see but it has 2 processors and
6GB RAM. If I want to profile this application. I
would use tau_exec to launch this code. I
see that it's taking about 81.47 seconds. Now if I run it with tau_exec -ebs
then it will just intercept the MPI calls using the wrapper interposition
library. Now, at this stage, it should generate all the profile files. Instrumentation
and you can see that it took 83.32 seconds.
with instrumentation and you can see MPI rank 0 through 3 and 2
threads of execution. Now you can say paraprof --pack bt_original.ppk. Now I can launch pprof and see the text based output. I can see that poll_no_cancel
is taking up a lot of time and it is called from MPI_Waitall.
I can launch paraprof and I can see all the threads.
If I click on the right I can say show Thread Statistics
table. It says 84 seconds were spent in the
application. 84.18 on thread 0 here it shows that
MPI_Waitall took 54.35 seconds. All these MPI calls add
up to roughly 55 seconds and
the rest of the application takes 28.79seconds.