Table of Contents
In this exercise, you will use the TAU performance observation tools to automatically generate a proxy component that monitors all of the method invocations on a port and allowing you to track their performance information. While this approach won't provide all of the performance details of what is going on inside each component, it gives you a very simple way to begin analyzing the performance of a CCA-based application in order to identify which components might have performance issues.
We will be using C++ classic components for this example. While the performance component and interface exist for babel components, the automatic proxy generator is only designed for classic components at the present time.
A simple classic CCA application is provided in $PERF/tutorials/classic/original
Make sure you can compile at run it before moving on. Use make and ./demo to test it. You should get output similar to this:
(19159) CmdLineClientMain.cxx: MPI_Init not called in ccafe-single mode. (19159) CmdLineClientMain.cxx: Try running with ccafe-single --ccafe-mpi yes , or (19159) CmdLineClientMain.cxx: try setenv CCAFE_USE_MPI 1 to force MPI_Init. (19159) my rank: -1, my pid: 19159 my rank: -1, my pid: 19159 my rank: -1, my pid: 19159 my rank: -1, my pid: 19159Type: One Processor Interactive CCAFFEINE configured with babel. cca> CmdContextCCAMPI::initRC: Found rc. cca> cca> cca># There are allegedly 3 classes in the component path cca> cca># factory added {./libDriver_CCA.so}:{create_Driver} as Driver Loaded Driver NOW GLOBAL . cca># factory added {./libMidpointIntegrator_CCA.so}:{create_MidpointIntegrator} as MidpointIntegrator Loaded MidpointIntegrator NOW GLOBAL . cca># factory added {./libPiFunction_CCA.so}:{create_PiFunction} as PiFunction Loaded PiFunction NOW GLOBAL . cca> cca>pi_func of type PiFunction successfully instantiated cca>mid_integrator of type MidpointIntegrator successfully instantiated cca>driver of type Driver successfully instantiated cca> cca>mid_integrator))))FunctionPort---->FunctionPort((((pi_func connection made successfully cca>driver))))IntegratorPort---->IntegratorPort((((mid_integrator connection made successfully cca> cca>integral(0, 1, 1000) = 3.14159 integral(0, 10, 10000) = 5.88451 integral(0, 100, 100000) = 6.24319 ##specific go command successful cca> cca> bye! exit
We will now generate a proxy for the Integrator port.
The first step in generating proxy components for classic C++ ports is to analyze the source code of a component that provides the port using the PDT (Program Database Toolkit).
![]() | Note |
---|---|
PDT is installed on thor.cs.indiana.edu at /san/shared/cca/tutorial/share/pdtoolkit-3.2 |
We use cxxparse from PDT to generate a .pdb file:
>cxxparse MidpointIntegrator_CCA.cc \ -I $CCA/include/cca-spec-classic-0.5.6 \ -I.
This gives us MidpointIntegrator_CCA.pdb
Now we use the proxy generator:
>tau_pg Usage: tau_pg [Arguments] Generate instrumented proxies for CCA components from a pdb file Example: tau_pg -c integrators::ccaports::Integrator -t integrators.ccaports.Integrator -n IntegratorProxy -p IntegratorPort -d ParallelIntegrator_CCA.pdb -o Proxy.cc -h ports/Integrator_CCA.h -f select.dat Required Arguments -c <component name> Full name of the component (e.g. integrators::ccaports::Integrator) -t <type name> Type of component (e.g. integrators.ccaports.Integrator) -p <port name> Name of port to generate proxy for (e.g. IntegratorPort) -d <pdbfile name> Name of pdb file created from cxxparse (pdtoolkit) (e.g. ParallelIntegrator_CCA.pdb) -h <header file> Header file for this port (e.g. ports/Integrator_CCA.h) Optional Arguments -n <proxy name> Name of the proxy component (default: base of component name + Proxy) -o <output filename> Name of output file (default: proxy.cc) -f <selective instrumentation file> Pre-generated Selective instrumentation file -m Generate MasterMind component proxy -v Verbose output -? Output help
To generate a proxy component for the integrator port, we use the following:
>$PERF/tau_pg -c integrators::ccaports::Integrator -t integrators.ccaports.Integrator \ -n IntegratorMeasurementProxy -p IntegratorPort -d MidpointIntegrator_CCA.pdb \ -o IntegratorProxy.cc -h ports/Integrator_CCA.h
This generates a file called IntegratorProxy.cc which is a classic C++ component that both uses and provides an integrator component.
Compile it:
>g++ -shared -o libIntegratorProxy.so IntegratorProxy.cc \ -I $CCA/include/cca-spec-classic-0.5.6/ \ -I $PERF/classic/
In order to use the proxy we simply need to make our CCA application aware of the new component, then place it between the Driver and MidpointIntegrator.
Add the appropriate lines to Integrator.cca:
!location= libDriver_CCA.so create_Driver Driver libMidpointIntegrator_CCA.so create_MidpointIntegrator MidpointIntegrator libPiFunction_CCA.so create_PiFunction PiFunction libIntegratorProxy.so createIntegratorProxy IntegratorProxy
And make the necessary changes to the rc file to interpose the IntegratorProxy between the Driver and the MidpointIntegrator:
#!ccaffeine bootstrap file. # ------- don't change anything ABOVE this line.------------- path append . path append $PERF/classic repository get-global Driver repository get-global MidpointIntegrator repository get-global PiFunction repository get-global TauMeasurement repository get-global IntegratorProxy create PiFunction pi_func create MidpointIntegrator mid_integrator create Driver driver create TauMeasurement tau create IntegratorProxy proxy connect mid_integrator FunctionPort pi_func FunctionPort connect driver IntegratorPort proxy IntegratorPortProvide connect proxy IntegratorPortUse mid_integrator IntegratorPort connect proxy MeasurementPort tau MeasurementPort go driver Go quit
And now run it with ./demo, you should now have a file called "profile.0.0.0" containing the profile data. View this with pprof and you should get output similar to this:
NODE 0;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 5 5 3 0 1826 integrate double (double, double, int)
In this section, we will cover the Classic C++ CCA component interface to the Classic TAU Performance component
The Classic C++ port interface is given in TauMeasurement_CCA.h:
#ifndef TauMeasurement_CCA_H #define TauMeasurement_CCA_H #include "Measurement_CCA.h" #include <TAU.h> namespace performance { /** * This class implements the performance Timer interface */ class TauTimer: public virtual performance::Timer { public: TauTimer (); TauTimer (string name); TauTimer (string name, string type); TauTimer (string name, string type, string group); ~TauTimer (); virtual void start(void); virtual void stop(void); virtual void setName(string name); virtual string getName(void); virtual void setType(string name); virtual string getType(void); virtual void setGroupName(string name); virtual string getGroupName(void); virtual void setGroupId(unsigned long group); virtual unsigned long getGroupId(void); private: TAU_MAPPING_OBJECT(tautimer) }; /** * This class implements the Timer Query interface */ class TauQuery: public virtual performance::Query { public: TauQuery (); ~TauQuery (); /** * getTimerNames. Returns lists of Timers. */ virtual void getTimerNames(const char **& timerList, int& numFuncs); /** * getCounterNames. Returns lists of counters. */ virtual void getCounterNames(const char **& counterList, int& numCounters); /** * getTimerData. Returns lists of metrics. */ virtual void getTimerData(const char **& inTimerList, int numTimers, double **& counterExclusive, double **& counterInclusive, int*& numCalls, int*& numChildCalls, const char **& counterNames, int& numCounters); /** * dumpProfileData. Writes the entire profile to disk in a dump file. * It maintains a consistent state and represents the instantaneous * profile data had the application terminated at the instance this call * is invoked. */ virtual void dumpProfileData(void); /** * dumpProfileDataPrefix. Writes the entire profile to disk in a dump * file prefixed by 'prefix'. It maintains a consistent state and * represents the instantaneous profile data had the application * terminated at the instance this call is invoked. */ virtual void dumpProfileDataPrefix(const char *prefix); /** * dumpProfileDataIncremental. Writes the entire profile to disk in a * dump file whose name contains the current timestamp. * It maintains a consistent state and represents the instantaneous * profile data had the application terminated at the instance this call * is invoked. This call allows us to build a set of timestamped profile * files. */ virtual void dumpProfileDataIncremental(void); /** * dumpTimerNames. Writes the list of timer names to a dump file on the * disk. */ virtual void dumpTimerNames(void); /** * dumpTimerData. Writes the profile of the given set of timers to the * disk. This allows the user to select the set of routines to dump and * periodically write the performance data of a subset of timers to disk * for monitoring purposes. */ virtual void dumpTimerData(const char **& inTimerList, int numTimers); /** * dumpTimerDataIncremental. Writes the profile of the given set of * timers to the disk. The dump file name contains the current timestamp * when the data was dumped. This allows the user to select the set of * routines to dump and periodically write the performance data of a * subset of timers to the disk and maintain a timestamped set of values * for post-mortem analysis of how the performance data varied for a * given set of routimes with time. */ virtual void dumpTimerDataIncremental(const char **& inTimerList, int numTimers); }; /** * This class implements the performance user defined evevnt interface */ class TauEvent: public virtual performance::Event { public: /** * Constructor */ TauEvent (); /** * Constructor with name */ TauEvent (string name); /** * Destructor */ ~TauEvent (); /** * Trigger the event and give it a value. (e.g., size of a message) */ virtual void trigger(double data); private: TauUserEvent *event; }; /** * This class implements the TAU runtime instrumentation control feature */ class TauControl: public virtual performance::Control { public: TauControl (); ~TauControl (); /** * Control instrumentation. Enable group Id. */ virtual void enableGroupId(unsigned long id); /** * Control instrumentation. Disable group Id. */ virtual void disableGroupId(unsigned long id); /** * Control instrumentation. Enable group name. */ virtual void enableGroupName(string name); /** * Control instrumentation. Disable group name. */ virtual void disableGroupName(string name); /** * Control instrumentation. Enable all groups. */ virtual void enableAllGroups(void); /** * Control instrumentation. Disable all groups. */ virtual void disableAllGroups(void); }; namespace ccaimpl { /** * This class implements the CCA Measurement interface */ class TauMeasurement: public virtual performance::ccaports::Measurement, public virtual classic::gov::cca::Component { public: TauMeasurement (); ~TauMeasurement (); /* Timing measurements based on start/stop */ virtual performance::Timer* createTimer(void); virtual performance::Timer* createTimer(string name); virtual performance::Timer* createTimer(string name, string type); virtual performance::Timer* createTimer(string name, string type, string group); /* Query performance data */ virtual performance::Query* createQuery(void); /* User defined events for tracking application events */ virtual performance::Event* createEvent(void); virtual performance::Event* createEvent(string name); /** * Create a Control interface for selectively enabling and disabling * the instrumentation based on groups */ virtual performance::Control* createControl(void); virtual void setServices (classic::gov::cca::Services * services); private: classic::gov::cca::Services * frameworkServices; }; } } extern "C" { /** Construct and return a TauMeasurement instance as a pointer to a CCA component. */ classic::gov::cca::Component * create_TauMeasurement (); #define TAUMEASUREMENT_ALIAS "TauMeasurement" }; #endif // TauMeasurement_CCA_H
The Classic C++ interface has a one to one correspondence with the Babel interface described below. Users are encouraged to read the explanation of the interface described in the next section.
Example: The following is an excerpt from MonteCarloIntegrator_CCA.cc showing an example of using the Classic C++ interface:
double MonteCarloIntegrator::integrate (double lowBound, double upBound, int count) { classic::gov::cca::Port * port; double sum = 0.0; // Get RandomGenerator port port = frameworkServices->getPort ("RandomGeneratorPort"); if (port) random_m = dynamic_cast < randomgen::ccaports::RandomGenerator * >(port); if (random_m == 0) { cerr << "Connected to something other than a RandomGenerator port" << endl; return -1; } // Get Function port port = frameworkServices->getPort ("FunctionPort"); if (port) function_m = dynamic_cast < functions::ccaports::Function * >(port); if (function_m == 0) { cerr << "Connected to something other than a Function port" << endl; return -1; } // Get Measurement port port = frameworkServices->getPort ("MeasurementPort"); if (port) measurement_m = dynamic_cast < performance::ccaports::Measurement * >(port); if (measurement_m == 0) { cerr << "Connected to something other than a Measurement port" << endl; return -1; } static performance::Timer* top = measurement_m->createTimer( string("Top Level timer")); top->start(); static performance::Timer* t = measurement_m->createTimer( string("IntegrateTimer")); cout <<"Before start!"<<endl; t->start(); for (int i = 0; i < count; i++) { double x = random_m->getRandomNumber (); sum = sum + function_m->evaluate (x); } /* You can change the timer name... string s = "NEW MonteCarloIntegrator's Integrate()"; t->setName(s); */ for (int i =0; i < 100000; i++) foo(2); /* Example of a user defined event */ /* static performance::Event* evt = measurement_m->createEvent(string("Memory")); evt->trigger(1024); evt->trigger(2048); */ /* Example of controlling instrumentation based on groups */ /* static performance::Control * ctrl = measurement_m->createControl(); ctrl->disableGroupName(string("Monte")); foo(23); */ t->stop(); static performance::Query* q = measurement_m->createQuery(); const char **funcList; int numFuncs; q->getTimerNames(funcList, numFuncs); for(int i = 0; i < numFuncs; i++) { cout <<"The function names are: "<<funcList[i]<<endl; } /* Now see the list of counters */ const char **counterList; int numCounters; q->getCounterNames(counterList, numCounters); for(int i = 0; i < numCounters; i++) { cout <<"The Counter names are: "<<counterList[i]<<endl; } /* Now get the metrics */ double **counterExclusive; double **counterInclusive; int *numCalls; int *numChildCalls; const char **counterNames; int numberOfCounters; q->getTimerData(funcList, numFuncs, counterExclusive, counterInclusive, numCalls, numChildCalls, counterNames, numberOfCounters); q->dumpTimerData(funcList, numFuncs); for(int i = 0; i < numFuncs; i++) { cout << "Timer: "<<funcList[i]<< " NumCalls: "<<numCalls[i] << " NumChildCalls: "<<numChildCalls[i]<<endl; for (int cnt = 0; cnt < numberOfCounters; cnt++) { cout <<"Counter: "<<counterNames[cnt] <<" Excl: "<<counterExclusive[i][cnt] <<" Incl: "<<counterInclusive[i][cnt]<<endl; } } /* CHECK */ // STOP Top level timer top->stop(); // Release ports frameworkServices->releasePort ("RandomGeneratorPort"); frameworkServices->releasePort ("FunctionPort"); frameworkServices->releasePort ("MeasurementPort"); return (upBound - lowBound) * sum / count; }
The TAU performance component provides a generic interface to performance tools for component based software. Following is the interface given as a SIDL file.
package performance version 1.0 { interface Timer { /* Start/stop the Timer */ void start(); void stop(); /* Set/get the Timer name */ void setName(in string name); string getName(); /* Set/get Timer type information (e.g., signature of the routine) */ void setType(in string name); string getType(); /* Set/get the group name associated with the Timer */ void setGroupName(in string name); string getGroupName(); /* Set/get the group id associated with the Timer */ void setGroupId(in long group); long getGroupId(); } /* Query interface to obtain timing information */ interface Query { /* Get the list of Timer and Counter names */ array<string> getTimerNames(); array<string> getCounterNames(); /* Get the timer data */ void getTimerData(in array<string> timerList, out array<double, 2> counterExclusive, out array<double, 2> counterInclusive, out array<int> numCalls, out array<int> numChildCalls, out array<string> counterNames, out int numCounters); /* Writes instantaneous profile to disk in a dump file. */ void dumpProfileData(); /* Writes instantaneous profile to disk in a dump file with a specified prefix. */ void dumpProfileDataPrefix(in string prefix); /* Writes the instantaneous profile to disk in a dump file whose name * contains the current timestamp. */ void dumpProfileDataIncremental(); /* Writes the list of timer names to a dump file on the disk */ void dumpTimerNames(); /* Writes the profile of the given set of timers to the disk. */ void dumpTimerData(in array<string> timerList); /* Writes the profile of the given set of timers to the disk. The dump * file name contains the current timestamp when the data was dumped. */ void dumpTimerDataIncremental(in array<string> timerList); } /* User defined event profiles for application specific events */ interface Event { /* Set the name of the event */ void setName(in string name); /* Trigger the event */ void trigger(in double data); } /* Interface for runtime instrumentation control based on groups */ interface Control { /* Enable/disable group id */ void enableGroupId(in long id); void disableGroupId(in long id); /* Enable/disable group name */ void enableGroupName(in string name); void disableGroupName(in string name); /* Enable/disable all groups */ void enableAllGroups(); void disableAllGroups(); } /* Interface to create performance component instances */ interface Measurement extends gov.cca.Port { /* Create a Timer */ Timer createTimer(); Timer createTimerWithName(in string name); Timer createTimerWithNameType(in string name, in string type); Timer createTimerWithNameTypeGroup(in string name, in string type, in string group); /* Create a Query interface */ Query createQuery(); /* Create a User Defined Event interface */ Event createEvent(); Event createEventWithName(in string name); /* Create a Control interface for selectively enabling and disabling * the instrumentation based on groups */ Control createControl(); } /* TAU */ /* Implementation of performance component Timer interface*/ class TauTimer implements-all Timer { } /* Implementation of performance component Event interface*/ class TauEvent implements-all Event { } /* Implementation of performance component Query interface*/ class TauQuery implements-all Query { } /* Implementation of performance component Control interface*/ class TauControl implements-all Control { } /* Implementation of performance component Measurement interface*/ class TauMeasurement implements-all Measurement, gov.cca.Component { } }
Given this interface, a number of performance tools (TAU, gprof, etc) could implement a performance component.
The performance component is composed of 5 interfaces:
Timer - Provides access to timers
void start() : start the timer
void stop() : stop the timer
void setName(in string name) : set the name of this timer
string getName() : get the name of this timer
void setGroupName(in string name) : set the group for this timer
string getGroupName() : get the name of the group for this timer
void setGroupId(in long group) : set the ID of the group for this timer
long getGroupId() : get the ID of the group for this timer
Query - Provides access to runtime performance data. Note that due to the use of shared libraries, this interface also provides access to any performance data obtained through the TAU API by the TAU runtime system.
array<string> getTimerNames() : retrieves a one dimensional array of strings containing each of the timer names that the performance component is tracking.
array<string> getCounterNames() : retrieves a one dimensional array of strings containing the names of each of the counters that the performance component is tracking. These are dependent on how the TAU libraries were compiled (with hardware performance counters, etc). See the TAU documentation for details on using hardware performance counters and multiple counters.
void getTimerData(in array<string> timerList, out array<double, 2> counterExclusive, out array<double, 2> counterInclusive, out array<int> numCalls, out array<int> numChildCalls, out array<string> counterNames, out int numCounters) : Returns all timer data available.
void dumpProfileData() : Dumps profile data to disk. Data is output in file prefixed with 'dump' rather than 'profile' (dump.0.0.0 for example)
void dumpProfileDataPrefix(in string prefix) : Dumps profile data to disk. Data is output in file prefixed with the specified prefix
void dumpProfileDataIncremental() : Dumps profile data to disk in files that contain the current timestamp.
void dumpTimerNames() : Dumps the list of timers being tracked to a file on the disk named "dump_functionnames_n,c,t.X.X.X" Where the X's take the place of the node, context and thread.
void dumpTimerData(in array<string> timerList) : Dumps profile data for the given set of timers to disk.
void dumpTimerDataIncremental(in array<string> timerList) : Dumps profile data for the given set of timers to disk. The dump filename will contain the current timestamp.
Event - Interface for application specific events. See the TAU documentation for more information about user events.
void setName(in string name) : Set the name for this event
void trigger(in double data) : Trigger the event with the given data value
Control - Interface for runtime instrumentation control based on groups.
void enableGroupId(in long id) : Enable timers belonging to the given group id
void disableGroupId(in long id) : Disable timers belonging to the given group id
void enableGroupName(in string name) : Enable timers belonging to the given group name
void disableGroupName(in string name) : Disable timers belonging to the given group name
void enableAllGroups() : Enable all timers
void disableAllGroups() : Disable all timers
Measurement - Interface for creating objects that implement the Timer, Query, Event, and Control interfaces
Timer createTimer() : Creates a timer
Timer createTimerWithName(in string name) : Creates a timer with a given name
Timer createTimerWithNameType(in string name, in string type) : Creates a timer with a given name and type
Timer createTimerWithNameTypeGroup(in string name, in string type) : Creates a timer with a given name and type belonging to a particular group
Query createQuery() : Creates an object implementing the Query interface
Event createEvent() : Creates a user event object
Event createEventWithName(in string name) : Creates a user event object with a specified name
Control createControl() : Creates an object implementing the Control interface
Example: Following is an excerpt from integrators_MonteCarloIntegrator_Impl.cc which uses the TAU performance component.
double integrators::MonteCarloIntegrator_impl::integrate ( /*in*/ double lowBound, /*in*/ double upBound, /*in*/ int32_t count ) throw () { // DO-NOT-DELETE splicer.begin(integrators.MonteCarloIntegrator.integrate) // insert implementation here gov::cca::Port port; double sum = 0.0; functions::Function function_m; static performance::Measurement measurement_m; randomgen::RandomGenerator random_m; // Get RandomGenerator port port = frameworkServices.getPort ("RandomGeneratorPort"); if (port._not_nil ()) random_m = port; if (random_m._is_nil ()) { cerr << "Connected to something other than a RandomGenerator port" << endl; return -1; } // Get Function port port = frameworkServices.getPort ("FunctionPort"); if (port._not_nil ()) function_m = port; if (function_m._is_nil ()) { cerr << "Connected to something other than a Function port" << endl; return -1; } // Get Measurement port port = frameworkServices.getPort ("MeasurementPort"); if (port._not_nil ()) measurement_m = port; if (measurement_m._is_nil ()) { cerr << "Connected to something other than a Measurement port" << endl; return -1; } static performance::Timer top = measurement_m.createTimerWithNameTypeGroup("Top Level", "Type", "TAU_GROUP_CCA"); static performance::Timer line = measurement_m.createTimerWithNameTypeGroup("loop", "", "TAU_GROUP_CCA"); static performance::Query query = measurement_m.createQuery(); static performance::Timer third = measurement_m.createTimerWithNameTypeGroup("Third", "", "TAU_GROUP_CCA"); top.start(); for (int i = 0; i < count; i++) { double x = random_m.getRandomNumber (); line.start(); x = lowBound + (upBound - lowBound) * x; sum = sum + function_m.evaluate (x); line.stop(); } ::sidl::array< ::std::string> nm = query.getTimerNames(); ::sidl::array< ::std::string> ct = query.getCounterNames(); cout <<"Name of counter 0 : "<<ct.get(0)<<endl; for (int j = nm.lower(0); j < nm.upper(0)+1; j++) { cout <<"Name of Timer "<< j << " = " <<nm.get(j)<<endl; } ::sidl::array< double> ex, in; ::sidl::array< int> numCalls, numChildCalls; ::sidl::array< ::std::string> counterNames; int numCounters; query.getTimerData(nm, ex, in, numCalls, numChildCalls, counterNames, numCounters); cout <<"Num of Counters = "<< numCounters <<endl; cout <<"Calls for loop func: "<<numCalls.get(1)<<endl; cout <<"Counter 0 name = "<<counterNames.get(0) <<endl; //query.dumpTimerDataIncremental(nm); top.stop(); query.dumpTimerNames(); // Release ports frameworkServices.releasePort ("RandomGeneratorPort"); frameworkServices.releasePort ("FunctionPort"); frameworkServices.releasePort ("MeasurementPort"); return (upBound - lowBound) * sum / count; // DO-NOT-DELETE splicer.end(integrators.MonteCarloIntegrator.integrate) }
Users are encouraged to visit and read the documentation for TAU available at http://www.cs.uoregon.edu/research/paracomp/tau/tautools