Using TAU with charm++

(tested with 5.4 release 1)

Note that when TAU is built with charm++, only charm++ programs can use the resulting library. Additionally, no TAU calls can be made before the ConverseInit call.

When using TAU_COMPILER with charm++, the following change must be made to charm/src/conv-core/converse.h :

static __inline__ unsigned long long int rdtsc(void)
{
        unsigned long long int x;
#ifdef CMK_IA64
	__asm__ __volatile__("mov %0=ar.itc" : "=r"(x) :: "memory");
#else
        __asm__ volatile (".byte 0x0f, 0x31" : "=A" (x));
#endif
        return x;
}

Must be changed to :

static __inline__ unsigned long long int rdtsc(void)
{
        unsigned long long int x;
#ifndef TAU_CHARM
#ifdef CMK_IA64
	__asm__ __volatile__("mov %0=ar.itc" : "=r"(x) :: "memory");
#else
        __asm__ volatile (".byte 0x0f, 0x31" : "=A" (x));
#endif
#endif
        return x;
}

This is due to the EDG front-end parse not parsing the __asm__ directive properly.

Using TAU with NAMD

For NAMD version 2.6 and above see: Building TAU with NAMD

this website should have the most up to date information on how to build NAMD with TAU.

For NAMD version 2.5 use these instructions:

Example: To build TAU for NAMD 2.5 (with automatic instrumentation) :

Download and install PDT

  wget http://www.cs.uoregon.edu/research/paracomp/proj/pdtoolkit/Download/pdt_latest.tar.gz
  tar xzf pdt_latest.tar.gz
  cd pdtoolkit-3.3.1
  ./configure ; make ; make install

Download and install TAU

  wget http://www.cs.uoregon.edu/research/paracomp/proj/tau/tauprofile/dist/tau_latest.tar.gz
  tar xzf tau_latest.tar.gz
  cd tau-2.14.3
  ./configure -pdt=<path to pdt> -charm=<path to charm>

Note that TAU can be configure to use various options (-TRACE, -PROFILECALLPATH, etc) with -PROFILE as the default.

Apply the TAU NAMD 2.5 patch
```
cd NAMD_2.5_Source
patch < path/to/tau/examples/charm/namd-2.5.patch
```
Alternatively see the manual patching instructions below.

Make the following changes to arch/Linux-i686-g++.arch (or your .arch file):

Add "include /include/Makefile"

Change

"CXX = ..."

"CXX = $(TAU_COMPILER) -optTauSelectFile=<path to select.tau> ..."

Change

"CC = ..."

"CC = $(TAU_COMPILER) -optTauSelectFile=<path to select.tau> ..."

For example :

include /home/amorris/tau2/include/Makefile

NAMD_ARCH = Linux-i686
CHARMARCH = net-linux

CXX = $(TAU_COMPILER) -optTauSelectFile=/home/amorris/select.tau g++ -DSOCKLEN_T=socklen_t -DNO_STRSTREAM_H
CXXOPTS = -O3 -march=pentiumpro -ffast-math -static
CC = $(TAU_COMPILER) -optTauSelectFile=/home/amorris/select.tau gcc
COPTS = -O3 -march=pentiumpro -ffast-math -static

We recommend that you set the environment variables PROFILEDIR and TRACEDIR
```
export PROFILEDIR=/tmp
export TRACEDIR=/tmp
```
After execution, you should file profile.X.X.X files in your $PROFILEDIR, view these wwith pprof or paraprof

Manual patching instructions

Make the above change to converse.h

Create the selective instrumentation file. We have had success using the following select.tau file :

BEGIN_EXCLUDE_LIST
void Sequencer::thread#
void Controller::thread#
int NAMD_read_int#
void PDBDATA::scan#
Real PDBAtom::zcoor#
Real PDBAtom::ycoor#
Real PDBAtom::xcoor#
Bool Molocule::is_hydrogen#
END_EXCLUDE_LIST

BEGIN_FILE_EXCLUDE_LIST
*mainfunc.C
*memusage.C
*BackEnd.C
END_FILE_EXCLUDE_LIST

Make the following changes to Makefile :

Add $(TAU_COMPILER) to the link phase, as shown

namd2:	$(INCDIR) $(DSTDIR) $(OBJS) $(LIBS)
	$(MAKEBUILDINFO)
	$(TAU_COMPILER) $(CHARMC) -verbose -ld++-option \
	"$(COPTI)$(CHARMINC) $(COPTI)$(INCDIR) $(COPTI)$(SRCDIR) $(CXXOPTS)" \
	-module NeighborLB -module commlib -language charm++ \
	$(BUILDINFO).o \
	$(OBJS) \
	$(DPMTALIB) \
	$(DPMELIB) \
	$(TCLLIB) \
	$(FFTLIB) \
	$(PLUGINLIB) \
	-lm -o namd2

Make the following changes to BackEnd.C

Add the following to the top of BackEnd.C:

#include <TAU.h>
extern "C" void Tau_create_top_level_timer_if_necessary(void);

Change BackEnd::init as follows:

void BackEnd::init(int argc, char **argv) {
  ConverseInit(argc, argv, slave_init, 1, 1);  // calls slave_init on others
  TAU_PROFILE_SET_NODE(CmiMyPe());
  Tau_create_top_level_timer_if_necessary();
  TAU_PROFILE("BackEnd::init", "", TAU_DEFAULT);
  cpuTime_start = CmiCpuTimer();
  wallTime_start = CmiWallTimer();
  if ( CmiMyPe() ) {
    slave_init(argc, argv);  // for procs that call main
    TAU_PROFILE_EXIT("Calling ConverseExit()");
    ConverseExit();  // should never return
  }
  all_init(argc, argv);

Make the following changes to mainfunc.C

Add the following to the top of mainfun.C:
```
#include <TAU.h>
```

Change main as follows:

int main(int argc, char **argv) {
  BackEnd::init(argc,argv);
  TAU_PROFILE("int main(int, char**) C", " ", TAU_DEFAULT);
  ScriptTcl *script = new ScriptTcl;
  Node::Object()->setScript(script);

  ...


  TAU_PROFILE_EXIT("main exiting");
  BackEnd::exit();
  return 0;
}

Change Controller.C as follows:

Add the following to the top of Controller.C:
```
#include <TAU.h>
```

Make the following change to ::threadRun

void Controller::threadRun(Controller* arg)
{
  TAU_REGISTER_THREAD();
  arg->algorithm();
}

Make the following change to ::terminate

void Controller::terminate(void) {
  TAU_PROFILE_EXIT("Controller::terminate");
  BackEnd::awaken();
  CthFree(thread);
  CthSuspend();
}

Change Sequencer.C as follows:

Add the following to the top of Sequencer.C:
```
#include <TAU.h>
```

Make the following change to ::threadRun

void Sequencer::threadRun(Sequencer* arg)
{
  TAU_REGISTER_THREAD();
  arg->algorithm();
}

Make the following change to ::terminate

void Sequencer::terminate(void) {
  TAU_PROFILE_EXIT("Sequencer::terminate");
  CthFree(thread);
  CthSuspend();
}