A.) Install KTAU kernel module and userspace tools

1.) First check to see if the kernel headers are available on your system.

rpm -qa | grep kernel-devel

2.) If the kernel development package is not installed, go a head and install it.
sudo yum install kernel-devel

NOTE: If you wish to try KTAU in a VM, you may download the HPCLinux 32 bit VM from:
http://tau.uoregon.edu/livedvd/latest.ova
and use VirtualBox [http://www.virtualbox.org] and use "File -> Import
Appliance -> latest.ova (Choose)". Then start the VM. When it prompts for the
livetau or root password enter it. Please contact tau-bugs@cs.uoregon.edu for
this password. 

3.) Download the ktau source from the Performance Research Lab website.
wget http://tau.uoregon.edu/ktau.tgz

4.) Untar the source code.
tar xzvf ktau.tar.gz

5.) You should have a directory called "ktau-3.0" that contains the KTAU module source code and userspace source code, more detailed instructions on how to Install and use ktau are the INSTALL and USAGE files respectively. This document will walk through a simple example using matmult.

6.) Change into the ktau directory.
cd ktau-3.0

7.) Build the kernel modules
make

8.) Insert the kernel module into your kernel. This requires root privlidges.
sudo insmod core/ktau_base.ko

9.) Change into the directory containg the source code for the userspace utlities.
cd user-src/src

10.) Make and install the userspace utlities
make 
make install

11.) Add the userspace utlities to your path:
echo -e "\nexport PATH=$PATH:$HOME/ktau-3.0/user-src/bin/\n" >> .bashrc


B.) Setup TAU for integration with ktau.

1.) Download TAU (v2.21.3b1 or higher) from the UO Performance Research Lab website.
wget http://www.cs.uoregon.edu/Research/tau/tau_releases/tau-2.21.3b1.tar.gz

2.) Untar the TAU source fromthe tarball.
tar xzvf tau-2.21.3b1.tar.gz

3.) Change directories into the TAU directory.
cd tau-2.21.3b1

4.) Configure tau however you normally would like to use TAU, but make sure to add the -ktau-ng flag in.
./configure -ktau-ng 

5.) Add tau to your path.
echo -e "\nexport PATH=\$PATH:$HOME/tau-2.21.3b1/x86_64/bin\n" >> ~/.bashrc

6.) Set your environment variables to use the new tau makefile
echo -e "\nexport TAU_MAKEFILE=/home/users/wsvoorhees/tau-2.21.3b1/x86_64/lib/Makefile.tau-ktau-ng\n" >> ~/.bashrc

7.) Incorperate the new environmental variables you set into your current environment.
source ~/.bashrc

8.) Compile the matmult code using your new TAU installation using the
Makefile.tau with the -ktau-ng in its name.
cd ~/tau-2.21.3b1/examples/matmult
make

C.) Run Ktau/tau instrumented matmult and produce a merged profile.

1.) cd ~/ktau-3.0/user-src/bin

2.) You can modify the config.ini file to change what instrumentation is enabled.

3.) ./timeKtau -f config.ini `which mpirun` -np 4 /home/users/voorhees/tau-2.21.3b1/examples/matmult

4.) This will create two directories one which contains the profiles generated by ktau, and one which contains the profiles generated by tau.

5.) merge the two directories (Kprofile... and Tprofile...) using the include ktau_profile_merge.py script
ktau_profile_merge.py Kprofile* Tprofile* merged_directory

6.) Switch into that directory and view the results with paraprof
cd merged_directory
paraprof

Click on a node to view the merged profile showing contributions from the user
space and the kernel space.