Dynamic SHMEM API Wrapping Example
==================================

Compiles and links the program as normal.  Uses tau_exec to dynamically
wrap the OpenSHMEM API at program load time.


Profiling
---------

    Note that you do *not* need to set TAU_MAKEFILE since the TAU compiler
    wrapper scripts are never used to build the binary.

    % make profile

Tracing
-------

    To view the trace you'll need to forward your X11 display or copy the
    tau.slog2 file to a local machine where TAU is already installed.

    % make trace
    % jumpshot tau.slog2


Example Output
--------------

oshrun -np 4 tau_exec -T serial,shmem,pdt -shmem ./rotput
   2: got    2: CORRECT
   3: got    3: CORRECT
   0: got    0: CORRECT
   1: got    1: CORRECT
pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0         0.15           13           1          10      13203 .TAU application
 96.5           12           12           1           0      12740 void shmem_init(void) C
  1.4        0.179         0.19           5           5         38 void shmem_barrier_all(void) C
  0.6        0.079        0.079           1           0         79 void shmem_finalize(void) C
  0.2         0.03         0.03           1           0         30 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.011        0.026           1           1         26 void shmem_free(void *) C
  0.1        0.011        0.011           5           0          2 void shmem_quiet(void) C
  0.0        0.002        0.002           1           0          2 int shmem_my_pe(void) C
  0.0        0.001        0.001           1           0          1 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

NODE 1;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.205           13           1          10      13161 .TAU application
 97.4           12           12           1           0      12813 void shmem_init(void) C
  0.4        0.039        0.052           5           5         10 void shmem_barrier_all(void) C
  0.4        0.049        0.049           1           0         49 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.015        0.029           1           1         29 void shmem_free(void *) C
  0.2        0.023        0.023           1           0         23 void shmem_finalize(void) C
  0.1        0.013        0.013           5           0          3 void shmem_quiet(void) C
  0.0        0.003        0.003           1           0          3 int shmem_my_pe(void) C
  0.0        0.001        0.001           1           0          1 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

NODE 2;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.139           13           1          10      13145 .TAU application
 96.4           12           12           1           0      12669 void shmem_init(void) C
  2.1        0.267        0.276           5           5         55 void shmem_barrier_all(void) C
  0.2        0.028        0.028           1           0         28 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.009        0.027           1           1         27 void shmem_free(void *) C
  0.2        0.021        0.021           1           0         21 void shmem_finalize(void) C
  0.1        0.009        0.009           5           0          2 void shmem_quiet(void) C
  0.0        0.002        0.002           1           0          2 int shmem_my_pe(void) C
  0.0        0.001        0.001           1           0          1 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

NODE 3;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.184           13           1          10      13149 .TAU application
 96.1           12           12           1           0      12633 void shmem_init(void) C
  2.0        0.254        0.264           5           5         53 void shmem_barrier_all(void) C
  0.2         0.03         0.03           1           0         30 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.011        0.026           1           1         26 void shmem_free(void *) C
  0.2        0.022        0.022           1           0         22 void shmem_finalize(void) C
  0.1         0.01         0.01           5           0          2 void shmem_quiet(void) C
  0.0        0.004        0.004           1           0          4 int shmem_my_pe(void) C
  0.0        0.001        0.001           1           0          1 int shmem_n_pes(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         1          4          4          4          0  Message size sent to all nodes
---------------------------------------------------------------------------------------

FUNCTION SUMMARY (total):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.678           52           4          40      13164 .TAU application
 96.6           50           50           4           0      12714 void shmem_init(void) C
  1.5        0.739        0.782          20          20         39 void shmem_barrier_all(void) C
  0.3        0.145        0.145           4           0         36 void shmem_finalize(void) C
  0.3        0.137        0.137           4           0         34 void shmem_int_put(int *, const int *, size_t, int) C
  0.2        0.046        0.108           4           4         27 void shmem_free(void *) C
  0.1        0.043        0.043          20           0          2 void shmem_quiet(void) C
  0.0        0.011        0.011           4           0          3 int shmem_my_pe(void) C
  0.0        0.004        0.004           4           0          1 int shmem_n_pes(void) C

FUNCTION SUMMARY (mean):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0         0.17           13           1          10      13164 .TAU application
 96.6           12           12           1           0      12714 void shmem_init(void) C
  1.5        0.185        0.196           5           5         39 void shmem_barrier_all(void) C
  0.3       0.0362       0.0362           1           0         36 void shmem_finalize(void) C
  0.3       0.0343       0.0343           1           0         34 void shmem_int_put(int *, const int *, size_t, int) C
  0.2       0.0115        0.027           1           1         27 void shmem_free(void *) C
  0.1       0.0107       0.0107           5           0          2 void shmem_quiet(void) C
  0.0      0.00275      0.00275           1           0          3 int shmem_my_pe(void) C
  0.0        0.001        0.001           1           0          1 int shmem_n_pes(void) C


