To demonstrate the use of tau_wrap to create a wrapper library for HDF5, we use:

% make clean
% make

/home/demotau/DEMO/hdf5_tau_exec% make
Building the wrapper ...: Step 1.: Copy the header interfaces to a wrapper directory
mkdir -p hdfwrapper; cp -r /usr/local/ACTS/hdf5-1.6.10/include/* hdfwrapper

Building the wrapper ...: Step 2.: Copy the optional selective instrumentation file to the wrapper directory
cp select.tau hdfwrapper

Building the wrapper ...: Step 3.: Parse the header file containing the external package interfaces
cd hdfwrapper; /usr/local/packages/pdt/i386_linux/bin/cparse hdf5.h;

Building the wrapper ...: Step 4.: Invoke tau_wrap and specify the PDB file, the header file and the wrapper library sources.
Building the wrapper ...:        : Optionally specify the selective instrumentation file and a group (hdf5).
Building the wrapper ...:        : Specify the runtime library to be pre-loaded using the -r <libname> option.
cd hdfwrapper; tau_wrap hdf5.h.pdb hdf5.h -o hdf5.inst.c -f select.tau -g hdf5 -r libhdf5.so

Building the wrapper ...: Step 5 : Build the wrapper library using the source generated by the tau_wrap tool.
cd hdfwrapper/wrapper; gmake  TAU_MAKEFILE=/usr/local/packages/tau/i386_linux/lib/Makefile.tau-pdt AR='ar             '
gmake[1]: Entering directory `/home/demotau/DEMO/hdf5_tau_exec/hdfwrapper/wrapper'
gcc    -DPROFILING_ON                        -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS                      -DTAU_LINUX_TIMERS                                 -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE                    -DTAU_BFD     -DHAVE_TR1_HASH_MAP     -fPIC  -I/usr/local/packages/tau-2.19.2/include  -I..  -c hdf5.inst.c -o hdf5_wrap.o
gcc    -shared                   -o libhdf5_wrap.so hdf5_wrap.o  -L/usr/local/packages/tau-2.19.2/i386_linux/lib -lTAUsh-pdt              -lbfd       -Wl,--export-dynamic    -ldl
gmake[1]: Leaving directory `/home/demotau/DEMO/hdf5_tau_exec/hdfwrapper/wrapper'

Step 6: Building the uninstrumented application...
gcc -I/usr/local/ACTS/hdf5-1.6.10/include checkhdf.c -o checkhdf -L/usr/local/ACTS/hdf5-1.6.10/lib -lhdf5 -lm
Done!


pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call
---------------------------------------------------------------------------------------
100.0        0.526            2           1          21       2391 .TAU application
 52.5            1            1           1           1       1256 hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C
 13.9        0.333        0.333           1           0        333 herr_t H5Fclose(hid_t) C
  4.9        0.117        0.117           1           0        117 hid_t H5Dcreate(hid_t, const char *, hid_t, hid_t, hid_t) C
  2.4        0.058        0.058           1           0         58 herr_t H5Dwrite(hid_t, hid_t, hid_t, hid_t, hid_t, const void *) C
  2.1        0.051        0.051           1           0         51 herr_t H5Dclose(hid_t) C
  0.8        0.018        0.018           3           1          6 hid_t H5FD_sec2_init(void) C
  0.5        0.012        0.012           1           0         12 hid_t H5Screate_simple(int, const hsize_t *, const hsize_t *) C
  0.4         0.01         0.01           1           0         10 herr_t H5check_version(unsigned int, unsigned int, unsigned int) C
  0.2        0.005        0.005           1           0          5 herr_t H5Tset_order(hid_t, H5T_order_t) C
  0.2        0.004        0.004           1           0          4 hid_t H5Tcopy(hid_t) C
  0.1        0.003        0.003           1           0          3 void H5FD_core_term(void) C
  0.1        0.003        0.003           1           0          3 void H5FD_family_term(void) C
  0.1        0.003        0.003           1           0          3 void H5FD_multi_term(void) C
  0.1        0.002        0.002           1           0          2 herr_t H5Sclose(hid_t) C
  0.1        0.002        0.002           1           0          2 void H5FD_log_term(void) C
  0.0        0.001        0.001           1           0          1 herr_t H5Eget_auto(H5E_auto_t *, void **) C
  0.0        0.001        0.001           1           0          1 herr_t H5Tclose(hid_t) C
  0.0        0.001        0.001           2           0          0 herr_t H5open(void) C
  0.0        0.001        0.001           1           0          1 void H5FD_sec2_term(void) C
  0.0            0            0           1           0          0 void H5FD_stdio_term(void) C
/home/demotau/DEMO/hdf5_tau_exec%

The profile has routines from HDF5.

[sameer@zorak hdf5]$ make
mkdir -p hdfwrapper; cp -r /usr/local/packages/hdf5-1.6.5/include/* hdfwrapper
cp select.tau hdfwrapper
cd hdfwrapper; /usr/local/packages/pdtoolkit-3.16/x86_64/bin/cparse hdf5.h;
cd hdfwrapper; tau_wrap hdf5.h.pdb hdf5.h -o hdf5.inst.c -f select.tau -g hdf5
cd hdfwrapper/wrapper; gmake  TAU_MAKEFILE=/home/users/sameer/tau2/x86_64/lib/Makefile.tau-pdt AR='ar             '
gmake[1]: Entering directory `/home/sameer/tau2/examples/iowrappers/hdf5/hdfwrapper/wrapper'
gcc    -DPROFILING_ON                        -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS                      -DTAU_LINUX_TIMERS                                 -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE                    -DTAU_BFD     -DHAVE_TR1_HASH_MAP     -I/home/users/sameer/tau2/include  -I..  -c hdf5.inst.c -o hdf5_wrap.o
ar              rcv  libhdf5_wrap.a hdf5_wrap.o
a - hdf5_wrap.o
gmake[1]: Leaving directory `/home/sameer/tau2/examples/iowrappers/hdf5/hdfwrapper/wrapper'
tau_cc.sh -tau_options='-optCompile=-I`pwd`/hdfwrapper/wrapper -optLinking="-L`pwd`/hdfwrapper/wrapper/ -lhdf5_wrap" '  -I/usr/local/packages/hdf5-1.6.5/include checkhdf.c -o checkhdf -L/usr/local/packages/hdf5-1.6.5/lib -lhdf5 -lm


[sameer@zorak hdf5]$ ls
checkhdf  checkhdf.c  checkhdf.o  hdfwrapper  Makefile  README  select.tau
[sameer@zorak hdf5]$ ls hdfwrapper/
H5ACpublic.h  H5FDgass.h      H5FDstdio.h   H5MMpublic.h  H5Zpublic.h
H5api_adpt.h  H5FDlog.h       H5FDstream.h  H5Opublic.h   hdf5.h
H5Apublic.h   H5FDmpi.h       H5Fpublic.h   H5Ppublic.h   hdf5.h.pdb
H5Bpublic.h   H5FDmpio.h      H5Gpublic.h   H5pubconf.h   select.tau
H5Cpublic.h   H5FDmpiposix.h  H5HGpublic.h  H5public.h    wrapper
H5Dpublic.h   H5FDmulti.h     H5HLpublic.h  H5Rpublic.h
H5Epublic.h   H5FDpublic.h    H5IM.h        H5Spublic.h
H5FDcore.h    H5FDsec2.h      H5Ipublic.h   H5TA.h
H5FDfamily.h  H5FDsrb.h       H5LT.h        H5Tpublic.h
[sameer@zorak hdf5]$ ls hdfwrapper/wrapper/
hdf5.h  hdf5.inst.c  hdf5_wrap.o  libhdf5_wrap.a  Makefile
[sameer@zorak hdf5]$ ./checkhdf
Initializing sigaction handlers
[sameer@zorak hdf5]$ ls
checkhdf    checkhdf.o  Makefile       README  select.tau
checkhdf.c  hdfwrapper  profile.0.0.0  SDS.h5
[sameer@zorak hdf5]$ pprof

Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        0.073            1           1          13       1548 int main(void) C 
 69.3            1            1           1           0       1072 hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C
  9.9        0.154        0.154           1           0        154 herr_t H5Fclose(hid_t) C
  6.2        0.096        0.096           1           0         96 hid_t H5Dcreate(hid_t, const char *, hid_t, hid_t, hid_t) C
  4.1        0.063        0.063           1           0         63 herr_t H5Dclose(hid_t) C
  2.9        0.045        0.045           1           0         45 herr_t H5Dwrite(hid_t, hid_t, hid_t, hid_t, hid_t, const void *) C
  1.5        0.023        0.023           1           0         23 herr_t H5check_version(unsigned int, unsigned int, unsigned int) C
  0.7        0.011        0.011           1           0         11 hid_t H5Screate_simple(int, const hsize_t *, const hsize_t *) C
  0.2        0.003        0.003           1           0          3 herr_t H5Tset_order(hid_t, H5T_order_t) C
  0.1        0.002        0.002           1           0          2 herr_t H5Sclose(hid_t) C
  0.1        0.002        0.002           1           0          2 herr_t H5Tclose(hid_t) C
  0.1        0.002        0.002           2           0          1 herr_t H5open(void) C
  0.1        0.002        0.002           1           0          2 hid_t H5Tcopy(hid_t) C



Now, if we just compile the checkhdf without any instrumentation, we can use tau_exec -io to see the POSIX I/O contribution:

% gcc checkhdf.c -I<hdf5dir>/include -L<hdf5dir>/lib -lhdf5 -ldl -lm
% ./checkhdf
or
% tau_exec
Usage: tau_exec [options] [--] <exe> <exe options>

Options:
        -v       verbose mode
        -qsub    Use qsub mode (BG/P only, see below)
        -io      track I/O
        -memory  track memory
        -T <DISABLE,ICPC,MPI,OPENMPI12,OPENMPI12_SILC0.1,PAPI,PDT,PROFILE,PTHREAD,PYTHON,SCOREP,SERIAL> : specify TAU option
        -loadlib=<file.so>   : specify additional load library
        -XrunTAUsh-<options> : specify TAU library directly

Notes:
        Defaults if unspecified: -T MPI
        MPI is assumed unless SERIAL is specified

Example:
    mpirun -np 2 tau_exec -io ./ring

qsub mode (IBM BG/P only):
    Original:
      qsub -n 1 --mode smp -t 10 ./a.out
    With TAU:
      tau_exec -qsub -io -memory -- qsub -n 1 --mode smp -t 10 ./a.out

[sameer@zorak hdf5_tau_exec]$ tau_exec -T serial -io ./checkhdf
Initializing sigaction handlers
[sameer@zorak hdf5_tau_exec]$ pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call
---------------------------------------------------------------------------------------
100.0            1            1           1           8       1453 .TAU application
  4.5        0.066        0.066           2           0         33 write()
  3.4         0.05         0.05           2           0         25 open64()
  0.3        0.005        0.005           2           0          2 close()
  0.1        0.002        0.002           2           0          1 lseek64()
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         2       1576        120        848        728  Bytes Written
         2       1576        120        848        728  Bytes Written : .TAU application => write()
         2       1576        120        848        728  Bytes Written <file="SDS.h5">
         2       1576        120        848        728  Bytes Written <file="SDS.h5"> : .TAU application => write()
         2        788          6        397        391  Write Bandwidth (MB/s)
         2        788          6        397        391  Write Bandwidth (MB/s) : .TAU application => write()
         2        788          6        397        391  Write Bandwidth (MB/s) <file="SDS.h5">
         2        788          6        397        391  Write Bandwidth (MB/s) <file="SDS.h5"> : .TAU application => write()
---------------------------------------------------------------------------------------


If we wish to see the time spent in the HDF5 library without either instrumenting checkhdf binary or the HDF5 library, we can automate the generation of an LD preloadable library using tau_wrap and then load the generated wrapper library in the address space of the uninstrumented checkhdf using tau_exec as shown below:



[sameer@zorak hdf5_tau_exec]$ make clean
/bin/rm -rf checkhdf.o checkhdf profile.* *.trc *.edf MULT* hdfwrapper SDS.h5
[sameer@zorak hdf5_tau_exec]$ make
mkdir -p hdfwrapper; cp -r /usr/local/packages/hdf5-1.6.5/include/* hdfwrapper
cp select.tau hdfwrapper
cd hdfwrapper; /usr/local/packages/pdtoolkit-3.16/x86_64/bin/cparse hdf5.h;
cd hdfwrapper; tau_wrap hdf5.h.pdb hdf5.h -o hdf5.inst.c -f select.tau -g hdf5 -r libhdf5.so
cd hdfwrapper/wrapper; gmake  TAU_MAKEFILE=/home/users/sameer/tau2/x86_64/lib/Makefile.tau-pdt AR='ar             '
gmake[1]: Entering directory `/home/sameer/tau2/examples/iowrappers/hdf5_tau_exec/hdfwrapper/wrapper'
gcc    -DPROFILING_ON                        -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS                      -DTAU_LINUX_TIMERS                                 -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE                    -DTAU_BFD   -DHAVE_GNU_DEMANGLE   -DHAVE_TR1_HASH_MAP     -fPIC  -I/home/users/sameer/tau2/include  -I..  -c hdf5.inst.c -o hdf5_wrap.o
gcc    -shared                   -o libhdf5_wrap.so hdf5_wrap.o  -L/home/users/sameer/tau2/x86_64/lib -lTAUsh-pdt             -L/home/users/sameer/tau2/x86_64/binutils-2.20/lib -L/home/users/sameer/tau2/x86_64/binutils-2.20/lib64 -lbfd -liberty -lz       -Wl,--export-dynamic    -ldl -ldl
gmake[1]: Leaving directory `/home/sameer/tau2/examples/iowrappers/hdf5_tau_exec/hdfwrapper/wrapper'
gcc -I/usr/local/packages/hdf5-1.6.5/include checkhdf.c -o checkhdf -L/usr/local/packages/hdf5-1.6.5/lib -lhdf5 -lm -ldl
[sameer@zorak hdf5_tau_exec]$ ldd ./checkhdf
        libhdf5.so.0 => /usr/local/packages/hdf5-1.6.5/lib/libhdf5.so.0 (0x0000002a95558000)
        libm.so.6 => /lib64/tls/libm.so.6 (0x0000003b72700000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003b72900000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x0000003b72400000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x0000003b72b00000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003b72200000)
[sameer@zorak hdf5_tau_exec]$ nm -A checkhdf | grep -i tau
[sameer@zorak hdf5_tau_exec]$ echo "No instrumentation in checkhdf"
No instrumentation in checkhdf
[sameer@zorak hdf5_tau_exec]$

[sameer@zorak hdf5_tau_exec]$ rm profile.0.0.0
[sameer@zorak hdf5_tau_exec]$ tau_exec -T serial -io -loadlib=/home/sameer/tau2/examples/iowrappers/hdf5_tau_exec/hdfwrapper/wrapper/libhdf5_wrap.so ./checkhdf
Initializing sigaction handlers
[sameer@zorak hdf5_tau_exec]$ ls
checkhdf    hdfwrapper  profile.0.0.0  SDS.h5
checkhdf.c  Makefile    README         select.tau
[sameer@zorak hdf5_tau_exec]$ pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call
---------------------------------------------------------------------------------------
100.0        0.153            1           1          13       1650 .TAU application
 66.7            1            1           1           4       1100 hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C
  9.0        0.132        0.149           1           3        149 herr_t H5Fclose(hid_t) C
  5.5         0.09         0.09           1           0         90 hid_t H5Dcreate(hid_t, const char *, hid_t, hid_t, hid_t) C
  4.6        0.021        0.076           1           2         76 herr_t H5Dclose(hid_t) C
  4.2        0.069        0.069           2           0         34 write()
  3.1        0.051        0.051           1           0         51 herr_t H5Dwrite(hid_t, hid_t, hid_t, hid_t, hid_t, const void *) C
  2.5        0.041        0.041           2           0         20 open64()
  0.8        0.013        0.013           1           0         13 hid_t H5Screate_simple(int, const hsize_t *, const hsize_t *) C
  0.7        0.011        0.011           1           0         11 herr_t H5check_version(unsigned int, unsigned int, unsigned int) C
  0.4        0.007        0.007           2           1          4 hid_t H5FD_sec2_init(void) C
  0.2        0.004        0.004           2           0          2 close()
  0.1        0.002        0.002           1           0          2 herr_t H5Sclose(hid_t) C
  0.1        0.002        0.002           1           0          2 herr_t H5Tset_order(hid_t, H5T_order_t) C
  0.1        0.002        0.002           1           0          2 hid_t H5Tcopy(hid_t) C
  0.1        0.002        0.002           2           0          1 lseek64()
  0.1        0.001        0.001           1           0          1 herr_t H5Tclose(hid_t) C
  0.0            0            0           2           0          0 herr_t H5open(void) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         2       1576        120        848        728  Bytes Written
         1        120        120        120          0  Bytes Written : herr_t H5Dclose(hid_t) C => write()
         1       1576       1576       1576          0  Bytes Written : herr_t H5Fclose(hid_t) C => write()
         2       1576        120        848        728  Bytes Written <file="SDS.h5">
         1        120        120        120          0  Bytes Written <file="SDS.h5"> : herr_t H5Dclose(hid_t) C => write()
         1       1576       1576       1576          0  Bytes Written <file="SDS.h5"> : herr_t H5Fclose(hid_t) C => write()
         2      525.3          6      265.7      259.7  Write Bandwidth (MB/s)
         1          6          6          6          0  Write Bandwidth (MB/s) : herr_t H5Dclose(hid_t) C => write()
         1      525.3      525.3      525.3          0  Write Bandwidth (MB/s) : herr_t H5Fclose(hid_t) C => write()
         2      525.3          6      265.7      259.7  Write Bandwidth (MB/s) <file="SDS.h5">
         1          6          6          6          0  Write Bandwidth (MB/s) <file="SDS.h5"> : herr_t H5Dclose(hid_t) C => write()
         1      525.3      525.3      525.3          0  Write Bandwidth (MB/s) <file="SDS.h5"> : herr_t H5Fclose(hid_t) C => write()
---------------------------------------------------------------------------------------
[sameer@zorak hdf5_tau_exec]$

And now all the HDF5 library calls are instrumented as well as the underlying POSIX I/O calls called by the library. Note: main is not instrumented as checkhdf does not have any instrumentation. Instead we see the time spent in ".TAU Application" that shows the time spent from the beginning of the application to the end. 

To see the POSIX I/O calls called by the HDF5 wrapper library, we turn on callpath profiling:
[sameer@zorak hdf5_tau_exec]$ setenv TAU_CALLPATH 1
[sameer@zorak hdf5_tau_exec]$ setenv TAU_CALLPATH_DEPTH 10

[sameer@zorak hdf5_tau_exec]$ tau_exec -T serial -io -loadlib=/home/sameer/tau2/examples/iowrappers/hdf5_tau_exec/hdfwrapper/wrapper/libhdf5_wrap.so ./checkhdf
Initializing sigaction handlers
[sameer@zorak hdf5_tau_exec]$ pprof
Reading Profile files in profile.*

NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call
---------------------------------------------------------------------------------------
100.0        0.159            1           1          13       1829 .TAU application
 62.4            1            1           1           3       1142 .TAU application => hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C
 62.4            1            1           1           3       1142 hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C
 10.8        0.157        0.197           1           3        197 .TAU application => herr_t H5Fclose(hid_t) C
 10.8        0.157        0.197           1           3        197 herr_t H5Fclose(hid_t) C
  5.3        0.028        0.097           1           2         97 .TAU application => herr_t H5Dclose(hid_t) C
  5.3        0.028        0.097           1           2         97 herr_t H5Dclose(hid_t) C
  5.1        0.094        0.094           1           0         94 .TAU application => hid_t H5Dcreate(hid_t, const char *, hid_t, hid_t, hid_t) C
  5.1        0.094        0.094           1           0         94 hid_t H5Dcreate(hid_t, const char *, hid_t, hid_t, hid_t) C
  4.9         0.09         0.09           2           0         45 write()
  3.5        0.064        0.064           1           0         64 .TAU application => herr_t H5Dclose(hid_t) C => write()
  3.3         0.06         0.06           1           0         60 .TAU application => herr_t H5Dwrite(hid_t, hid_t, hid_t, hid_t, hid_t, const void *) C
  3.3         0.06         0.06           1           0         60 herr_t H5Dwrite(hid_t, hid_t, hid_t, hid_t, hid_t, const void *) C
  2.1        0.038        0.038           1           0         38 .TAU application => herr_t H5check_version(unsigned int, unsigned int, unsigned int) C
  2.1        0.038        0.038           2           0         19 .TAU application => hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C => open64()
  2.1        0.038        0.038           1           0         38 herr_t H5check_version(unsigned int, unsigned int, unsigned int) C
  2.1        0.038        0.038           2           0         19 open64()
  1.4        0.026        0.026           1           0         26 .TAU application => herr_t H5Fclose(hid_t) C => write()
  1.1        0.012        0.021           1           1         21 .TAU application => hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C => hid_t H5FD_sec2_init(void) C
  1.1        0.021        0.021           2           1         10 hid_t H5FD_sec2_init(void) C
  0.9        0.017        0.017           1           0         17 .TAU application => hid_t H5Screate_simple(int, const hsize_t *, const hsize_t *) C
  0.9        0.017        0.017           1           0         17 hid_t H5Screate_simple(int, const hsize_t *, const hsize_t *) C
  0.6        0.011        0.011           2           0          6 lseek64()
  0.5        0.009        0.009           1           0          9 .TAU application => hid_t H5Fcreate(const char *, unsigned int, hid_t, hid_t) C => hid_t H5FD_sec2_init(void) C => hid_t H5FD_sec2_init(void) C
  0.4        0.008        0.008           1           0          8 .TAU application => herr_t H5Fclose(hid_t) C => close()
  0.4        0.008        0.008           1           0          8 close()
  0.3        0.006        0.006           1           0          6 .TAU application => herr_t H5Fclose(hid_t) C => lseek64()
  0.3        0.006        0.006           1           0          6 .TAU application => herr_t H5Tset_order(hid_t, H5T_order_t) C
  0.3        0.006        0.006           1           0          6 herr_t H5Tset_order(hid_t, H5T_order_t) C
  0.3        0.005        0.005           1           0          5 .TAU application => herr_t H5Dclose(hid_t) C => lseek64()
  0.3        0.005        0.005           1           0          5 .TAU application => herr_t H5Sclose(hid_t) C
  0.3        0.005        0.005           2           0          2 .TAU application => herr_t H5open(void) C
  0.3        0.005        0.005           1           0          5 .TAU application => hid_t H5Tcopy(hid_t) C
  0.3        0.005        0.005           1           0          5 herr_t H5Sclose(hid_t) C
  0.3        0.005        0.005           2           0          2 herr_t H5open(void) C
  0.3        0.005        0.005           1           0          5 hid_t H5Tcopy(hid_t) C
  0.2        0.004        0.004           1           0          4 .TAU application => herr_t H5Tclose(hid_t) C
  0.2        0.004        0.004           1           0          4 herr_t H5Tclose(hid_t) C
---------------------------------------------------------------------------------------

USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples   MaxValue   MinValue  MeanValue  Std. Dev.  Event Name
---------------------------------------------------------------------------------------
         2       1576        120        848        728  Bytes Written
         1        120        120        120          0  Bytes Written : .TAU application => herr_t H5Dclose(hid_t) C => write()
         1       1576       1576       1576          0  Bytes Written : .TAU application => herr_t H5Fclose(hid_t) C => write()
         2       1576        120        848        728  Bytes Written <file="SDS.h5">
         1        120        120        120          0  Bytes Written <file="SDS.h5"> : .TAU application => herr_t H5Dclose(hid_t) C => write()
         1       1576       1576       1576          0  Bytes Written <file="SDS.h5"> : .TAU application => herr_t H5Fclose(hid_t) C => write()
         2      525.3      4.615        265      260.4  Write Bandwidth (MB/s)
         1      4.615      4.615      4.615          0  Write Bandwidth (MB/s) : .TAU application => herr_t H5Dclose(hid_t) C => write()
         1      525.3      525.3      525.3          0  Write Bandwidth (MB/s) : .TAU application => herr_t H5Fclose(hid_t) C => write()
         2      525.3      4.615        265      260.4  Write Bandwidth (MB/s) <file="SDS.h5">
         1      4.615      4.615      4.615          0  Write Bandwidth (MB/s) <file="SDS.h5"> : .TAU application => herr_t H5Dclose(hid_t) C => write()
         1      525.3      525.3      525.3          0  Write Bandwidth (MB/s) <file="SDS.h5"> : .TAU application => herr_t H5Fclose(hid_t) C => write()
---------------------------------------------------------------------------------------

