Static SHMEM API Wrapping Example
=================================

Compiles all program objects as normal (i.e. no instrumentation) then uses
tau_cc.sh to wrap SHMEM API calls at link time.


Profiling
---------

    % export TAU_MAKEFILE=$PWD/../../../x86_64/lib/Makefile.tau-shmem-pdt
    % make profile


Tracing
-------

    To view the trace you'll need to forward your X11 display or copy the
    tau.slog2 file to a local machine where TAU is already installed.

    % export TAU_MAKEFILE=$PWD/../../../x86_64/lib/Makefile.tau-shmem-pdt
    % make trace
    % jumpshot tau.slog2


Example Output
--------------

oshcc -Wall -c rotput.c -o rotput.o
tau_cc.sh -optLinkOnly  rotput.o -o rotput 
Debug: Moving these libraries to the end of the link line:


Debug: Linking with TAU Options
Executing> /usr/local/packages/openshmem-1.2/bin/oshcc rotput.o -o rotput @/home/users/jlinford/workspace/tau2/x86_64/lib/wrappers/shmem/link_options.tau -L/home/users/jlinford/workspace/tau2/x86_64/lib -ltau-shmem-pdt -L/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib -L/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib64 -Wl,-rpath,/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib -Wl,-rpath,/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib64 -lbfd -liberty -lz -Wl,--export-dynamic -lrt -L/home/users/jlinford/workspace/tau2/x86_64/libunwind-1.1-oshcc/lib -lunwind -Wl,-rpath,/home/users/jlinford/workspace/tau2/x86_64/libunwind-1.1-oshcc/lib -ldl -lm -L/storage/packages/gcc/4.8.5/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.8.5/ -lstdc++ -lgcc_s -L/home/users/jlinford/workspace/tau2/x86_64/lib -lTauShmem-shmem-pdt -L/usr/local/packages/openshmem-1.2/lib -L/usr/local/packages/openshmem-1.2/lib -lopenshmem -L/home/users/jlinford/workspace/tau2/x86_64/lib/static-shmem-pdt


Debug:
Executing> /usr/local/packages/openshmem-1.2/bin/oshcc rotput.o -o rotput -I/home/users/jlinford/workspace/tau2/include -I/usr/local/packages/openshmem-1.2/include -I/usr/local/packages/openshmem-1.2/include/mpp @/home/users/jlinford/workspace/tau2/x86_64/lib/wrappers/shmem/link_options.tau -L/home/users/jlinford/workspace/tau2/x86_64/lib -ltau-shmem-pdt -L/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib -L/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib64 -Wl,-rpath,/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib -Wl,-rpath,/home/users/jlinford/workspace/tau2/x86_64/binutils-2.23.2/lib64 -lbfd -liberty -lz -Wl,--export-dynamic -lrt -L/home/users/jlinford/workspace/tau2/x86_64/libunwind-1.1-oshcc/lib -lunwind -Wl,-rpath,/home/users/jlinford/workspace/tau2/x86_64/libunwind-1.1-oshcc/lib -ldl -lm -L/storage/packages/gcc/4.8.5/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.8.5/ -lstdc++ -lgcc_s -L/home/users/jlinford/workspace/tau2/x86_64/lib -lTauShmem-shmem-pdt -L/usr/local/packages/openshmem-1.2/lib -L/usr/local/packages/openshmem-1.2/lib -lopenshmem -L/home/users/jlinford/workspace/tau2/x86_64/lib/static-shmem-pdt

oshrun -np 4 ./rotput
   2: got    2: CORRECT
   0: got    0: CORRECT
   3: got    3: CORRECT
   1: got    1: CORRECT
pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0         0.09           10           1          10      10041 .TAU application
 94.1            9            9           1           0       9444 void shmem_init(void) C
  3.2        0.324        0.324           1           0        324 void shmem_finalize(void) C
  1.4        0.135        0.143           5           5         29 void shmem_barrier_all(void) C
  0.2        0.017        0.023           1           1         23 void shmem_free(void *) C
  0.2        0.022        0.022           1           0         22 void shmem_int_put(int *, const int *, size_t, int) C
  0.1        0.008        0.008           5           0          2 void shmem_quiet(void) C
  0.0        0.001        0.001           1           0          1 int shmem_my_pe(void) C
  0.0            0            0           1           0          0 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

NODE 1;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.151           10           1          10      10042 .TAU application
 94.5            9            9           1           0       9488 void shmem_init(void) C
  3.2        0.326        0.326           1           0        326 void shmem_finalize(void) C
  0.3        0.025        0.035           5           5          7 void shmem_barrier_all(void) C
  0.3        0.031        0.031           1           0         31 void shmem_int_put(int *, const int *, size_t, int) C
  0.1         0.01        0.013           1           1         13 void shmem_free(void *) C
  0.1         0.01         0.01           5           0          2 void shmem_quiet(void) C
  0.0        0.001        0.001           1           0          1 int shmem_my_pe(void) C
  0.0            0            0           1           0          0 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

NODE 2;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.103            9           1          10       9916 .TAU application
 87.7            8            8           1           0       8694 void shmem_init(void) C
  8.9        0.872        0.881           5           5        176 void shmem_barrier_all(void) C
  2.1        0.208        0.208           1           0        208 void shmem_finalize(void) C
  0.2        0.022        0.022           1           0         22 void shmem_int_put(int *, const int *, size_t, int) C
  0.1        0.007        0.013           1           1         13 void shmem_free(void *) C
  0.1        0.009        0.009           5           0          2 void shmem_quiet(void) C
  0.0        0.001        0.001           1           0          1 int shmem_n_pes(void) C
  0.0            0            0           1           0          0 int shmem_my_pe(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

NODE 3;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.154           10           1          10      10042 .TAU application
 94.5            9            9           1           0       9489 void shmem_init(void) C
  3.2        0.326        0.326           1           0        326 void shmem_finalize(void) C
  0.3        0.021        0.032           5           5          6 void shmem_barrier_all(void) C
  0.3         0.03         0.03           1           0         30 void shmem_int_put(int *, const int *, size_t, int) C
  0.1         0.01        0.013           1           1         13 void shmem_free(void *) C
  0.1        0.011        0.011           5           0          2 void shmem_quiet(void) C
  0.0        0.001        0.001           1           0          1 int shmem_my_pe(void) C
  0.0            0            0           1           0          0 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

FUNCTION SUMMARY (total):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.498           40           4          40      10010 .TAU application
 92.7           37           37           4           0       9279 void shmem_init(void) C
  3.0            1            1           4           0        296 void shmem_finalize(void) C
  2.7            1            1          20          20         55 void shmem_barrier_all(void) C
  0.3        0.105        0.105           4           0         26 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.044        0.062           4           4         16 void shmem_free(void *) C
  0.1        0.038        0.038          20           0          2 void shmem_quiet(void) C
  0.0        0.003        0.003           4           0          1 int shmem_my_pe(void) C
  0.0        0.001        0.001           4           0          0 int shmem_n_pes(void) C

FUNCTION SUMMARY (mean):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.124           10           1          10      10010 .TAU application
 92.7            9            9           1           0       9279 void shmem_init(void) C
  3.0        0.296        0.296           1           0        296 void shmem_finalize(void) C
  2.7        0.263        0.273           5           5         55 void shmem_barrier_all(void) C
  0.3       0.0262       0.0262           1           0         26 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.011       0.0155           1           1         16 void shmem_free(void *) C
  0.1       0.0095       0.0095           5           0          2 void shmem_quiet(void) C
  0.0      0.00075      0.00075           1           0          1 int shmem_my_pe(void) C
  0.0      0.00025      0.00025           1           0          0 int shmem_n_pes(void) C
