TAUdb
Introduction
TAUdb (TAU Database), formerly known as PerfDMF (Performance Data Management Framework) is a an API/Toolkit that sits atop a DBMS to manage and analyze performance data. The API is available in its native Java form as well as C.
Prerequisites
-
A supported Database Management System (DBMS). TAUdb currently supports PostgreSQL, MySQL, Oracle, H2, and Derby. For use with the C API, only PostgreSQL is supported (SQLite support is currently being evaluated). Because they are Java only, H2 and Derby can NO be accessed with the C API.
-
Java 1.5.
-
If the C API is desired, a working C compiler is required, along with the following libraries: libpq (PostgreSQL libraries), libxml2, libz, libuuid. These libraries are all commonly installed by default on *NIX systems.
Installation
The TAUdb utilities and applications are installed as part of the standard TAU release. Shell scripts are installed in the TAU bin directory to configure and run the utilities. It is assumed that the user has installed TAU and run TAU’s configure and 'make install'.
-
(Optionally) Create a database. This step will depend on the user’s chosen DBMS.
-
H2: Because it is an embedded, file-based DBMS, H2 does not require creating the database before configuring TAUdb. TAUdb takes advantage of the "auto-server" capabilities in H2, so multiple clients can connect to the same database at the same time. Users should use the H2 DBMS if they expect to maintain a small to moderate local repository of performance data, and want the convenience of connecting to the database from multiple clients.
-
Derby: Because it is an embedded, file-based DBMS, Derby does not require creating the database before configuring TAUdb. Be advised that the Derby DBMS does not allow multiple clients to connect to the same database. For that reason, we suggest users use the H2 DBMS if a file-based database is desired. Derby support is maintained for backwards compatability.
-
PostgreSQL:
-
$ createdb -O taudb taudb Or, from
Or, from psql
psql=# create database taudb with owner = taudb;
-
MySQL: From the MySQL prompt
mysql> create database taudb;
-
Oracle: It is recommended that you create a tablespace for taudb:
create tablespace taudb datafile '/path/to/somewhere' size 500m reuse; Then, create a user that has this tablespace as default:
Then, create a user that has this tablespace as default:
create user amorris identified by db; grant create session to amorris; grant create table to amorris; grant create sequence to amorris; grant create trigger to amorris; alter user amorris quota unlimited on taudb; alter user amorris default tablespace taudb; TAUdb is set up to use the Oracle Thin Java driver. You will have to obtain this jar file for your DBMS. In our case, it was ojdbc14.jar.
TAUdb is set up to use the Oracle Thin Java driver. You will have to obtain this jar file for your DBMS. In our case, it was ojdbc14.jar.
-
Configure a TAUdb connection. To configure TAUdb, run the
taudb_configure
program from the TAU bin directory.The configuration program will prompt the user for several values. The default values will work for most users. When configuration is complete, it will connect to the database and test the configuration. If the configuration is valid and the schema is not already found in the database (as will be the case on initial configuration), the schema will be uploaded. Be sure to specify the correct version of the schema for your DBMS.
An example session for configuring a database is below. The user is creating an H2 database, with default settings including no username and no password (recommended for file-based databases when security is not an issue).
$ taudb_configure Configuration file NOT found... a new configuration file will be created. Welcome to the configuration program for PerfDMF. This program will prompt you for some information necessary to ensure the desired behavior for the PerfDMF tools. You will now be prompted for new values, if desired. The current or default values for each prompt are shown in parenthesis. To accept the current/default value, just press Enter/Return. Please enter the name of this configuration. ():documentation_example Please enter the database vendor (oracle, postgresql, mysql, db2, derby or h2). (h2): Please enter the JDBC jar file. (/Users/khuck/src/tau2/apple/lib/h2.jar): Please enter the JDBC Driver name. (org.h2.Driver): Please enter the path to the database directory. (/Users/khuck/.ParaProf/documentation_example): Please enter the database username. (): Store the database password in CLEAR TEXT in your configuration file? (y/n):y Please enter the database password: Please enter the PerfDMF schema file. (/Users/khuck/src/tau2/etc/taudb.sql): Writing configuration file: /Users/khuck/.ParaProf/perfdmf.cfg.documentation_example Now testing your database connection. Database created, command: jdbc:h2:/Users/khuck/.ParaProf/documentation_example/perfdmf;AUTO_SERVER=TRUE;create=true Uploading Schema: /Users/khuck/src/tau2/etc/taudb.sql Found /Users/khuck/src/tau2/etc/taudb.sql ... Loading Successfully uploaded schema Database connection successful. Configuration complete.
Using TAUdb
The easiest way to interact with TAUdb is to use ParaProf which provides a GUI interface to all of the database information. In addition, the following commandline utilities are provided.
perfdmf_createapp (deprecated - only supported for older PerfDMF databases)
This utility creates applications with a given name
$ perfdmf_createapp -n "New Application" Created Application, ID: 24
perfdmf_createexp (deprecated - only supported for older PerfDMF databases)
This utility creates experiments with a given name, under a specified application
$ perfdmf_createexp -a 24 -n "New Experiment" Created Experiment, ID: 38
taudb_loadtrial
This utility uploads a trial to the database with a given name, under a specified experiment
$ taudb_loadtrial -h Usage: perfdmf_loadtrial -a <appName> -x <expName> -n <name> [options] <files> Required Arguments: -n, --name <text> Specify the name of the trial -a, --applicationname <string> Specify associated application name for this trial -x, --experimentname <string> Specify associated experiment name for this trial ...or... -n, --name <text> Specify the name of the trial -e, --experimentid <number> Specify associated experiment ID for this trial Optional Arguments: -c, --config <name> Specify the name of the configuration to use -g, --configFile <file> Specify the configuration file to use (overrides -c) -f, --filetype <filetype> Specify type of performance data, options are: profiles (default), pprof, dynaprof, mpip, gprof, psrun, hpm, packed, cube, hpc, ompp, snap, perixml, gptl, paraver, ipm, google -t, --trialid <number> Specify trial ID -i, --fixnames Use the fixnames option for gprof -z, --usenull Include NULL values as 0 for mean calculation -r, --reduce <percentage> Aggregate all timers less than percentage as "other" -m, --metadata <filename> XML metadata for the trial Notes: For the TAU profiles type, you can specify either a specific set of profile files on the commandline, or you can specify a directory (by default the current directory). The specified directory will be searched for profile.*.*.* files, or, in the case of multiple counters, directories named MULTI_* containing profile data. Examples: perfdmf_loadtrial -e 12 -n "Batch 001" This will load profile.* (or multiple counters directories MULTI_*) into experiment 12 and give the trial the name "Batch 001" perfdmf_loadtrial -e 12 -n "HPM data 01" -f hpm perfhpm* This will load perfhpm* files of type HPMToolkit into experiment 12 and give the trial the name "HPM data 01" perfdmf_loadtrial -a "NPB2.3" -x "parametric" -n "64" par64.ppk This will load packed profile par64.ppk into the experiment named "parametric" under the application named "NPB2.3" and give the trial the name "64". The application and experiment will be created if not found.
TAUdb supports a large number of parallel profile formats:
TAU Profiles (profiles) - Output from the TAU measurement library, these files generally take the form of profile.X.X.X
, one for each node/context/thread combination. When multiple counters are used, each metric is located in a directory prefixed with "MULTI". To launch ParaProf with all the metrics, simply launch it from the root of the MULTI directories.
ParaProf Packed Format (ppk) - Export format supported by PerfDMF/ParaProf. Typically .ppk.
TAU Merged Profiles (snap) - Merged and snapshot profile format supported by TAU. Typically tauprofile.xml.
TAU pprof (pprof) - Dump Output from TAU’s pprof -d
. Provided for backward compatibility only.
DynaProf (dynaprof) - Output From DynaProf’s wallclock and papi probes.
mpiP (mpip) - Output from mpiP.
gprof (gprof) - Output from gprof, see also the --fixnames option.
PerfSuite (psrun) - Output from PerfSuite psrun files.
HPM Toolkit (hpm) - Output from IBM’s HPM Toolkit.
Cube (cube) - Output from Kojak Expert tool for use with Cube.
Cube3 (cube3) - Output from Kojak Expert tool for use with Cube3 and Cube4.
HPCToolkit (hpc) - XML data from hpcquick. Typically, the user runs hpcrun, then hpcquick on the resulting binary file.
OpenMP Profiler (ompp) - CSV format from the ompP OpenMP Profiler (http://www.ompp-tool.com). The user must use OMPP_OUTFORMAT=CVS.
PERI XML (perixml) - Output from the PERI data exchange format.
General Purpose Timing Library (gptl) - Output from the General Purpose Timing Library.
Paraver (paraver) - 2D output from the Paraver trace analysis tool from BSC.
IPM (ipm) - Integrated Performance Monitoring format, from NERSC.
Google (google) - Google Profiles.
TAUdb Views
In order to provide flexible data management, the application / experiment / trial hierarchy was removed in the conversion from PerfDMF to TAUdb. In addition, trial metadata was promoted from an XML blob in PerfDMF to queryable tables. Users can now organize their data in arbitrary hierarchies using Views and SubViews. Creating and using Views is outlined in the ParaProf User Manual, in Chapter 2.
Database Schema
The database schema in TAUdb is designed to flexibly and efficiently store multidimensional parallel performance data. There are 5 dimensions to the actual timer measurements, and 4 dimensions to the counter measurements
Timer dimensions
-
Process and thread of execution
-
Timer source code location (i.e. foo())
-
Metric of interest (i.e. FP_OPS, TIME)
-
Phase of execution (i.e. iteration number, timestamp)
-
Dynamic timer context (i.e. parameter values)
Counter dimensions
-
Process and thread of execution
-
Timer source code location (i.e. foo())
-
Phase of execution (i.e. iteration number, timestamp)
-
Dynamic timer context (i.e. parameter values)
SQL for TAUdb
Below is the SQL schema definition for TAUdb.
/****************************/ /* CREATE THE STATIC TABLES */ /****************************/ CREATE TABLE schema_version ( version INT NOT NULL, description VARCHAR NOT NULL ); /* IF THE SCHEMA IS MODIFIED, INCREMENT THIS VALUE */ /* 0 = PERFDMF (ORIGINAL) */ /* 1 = TAUDB (APRIL, 2012) */ /*VALUES (1, 'TAUdb redesign from Spring, 2012');*/ INSERT INTO schema_version (version, description) VALUES (2, 'Changes after Nov. 9, 2012 release'); /* These are our supported parsers. */ CREATE TABLE data_source ( id INT UNIQUE NOT NULL, name VARCHAR NOT NULL, description VARCHAR ); INSERT INTO data_source (name,id,description) VALUES ('ppk',0,'TAU Packed profiles (TAU)'); INSERT INTO data_source (name,id,description) VALUES ('TAU profiles',1,'TAU profiles (TAU)'); INSERT INTO data_source (name,id,description) VALUES ('DynaProf',2,'PAPI DynaProf profiles (UTK)'); INSERT INTO data_source (name,id,description) VALUES ('mpiP',3,'mpiP: Lightweight, Scalable MPI Profiling (Vetter, Chambreau)'); INSERT INTO data_source (name,id,description) VALUES ('HPM',4,'HPM Toolkit profiles (IBM)'); INSERT INTO data_source (name,id,description) VALUES ('gprof',5,'gprof profiles (GNU)'); INSERT INTO data_source (name,id,description) VALUES ('psrun',6,'PerfSuite psrun profiles (NCSA)'); INSERT INTO data_source (name,id,description) VALUES ('pprof',7,'TAU pprof.dat output (TAU)'); INSERT INTO data_source (name,id,description) VALUES ('Cube',8,'Cube data (FZJ)'); INSERT INTO data_source (name,id,description) VALUES ('HPCToolkit',9,'HPC Toolkit profiles (Rice Univ.)'); INSERT INTO data_source (name,id,description) VALUES ('SNAP',10,'TAU Snapshot profiles (TAU)'); INSERT INTO data_source (name,id,description) VALUES ('OMPP',11,'OpenMP Profiler profiles (Fuerlinger)'); INSERT INTO data_source (name,id,description) VALUES ('PERIXML',12,'Data Exchange Format (PERI)'); INSERT INTO data_source (name,id,description) VALUES ('GPTL',13,'General Purpose Timing Library (ORNL)'); INSERT INTO data_source (name,id,description) VALUES ('Paraver',14,'Paraver profiles (BSC)'); INSERT INTO data_source (name,id,description) VALUES ('IPM',15,'Integrated Performance Monitoring (NERSC)'); INSERT INTO data_source (name,id,description) VALUES ('Google',16,'Google profiles (Google)'); INSERT INTO data_source (name,id,description) VALUES ('Cube3',17,'Cube 3D profiles (FZJ)'); INSERT INTO data_source (name,id,description) VALUES ('Gyro',100,'Self-timing profiles from Gyro application'); INSERT INTO data_source (name,id,description) VALUES ('GAMESS',101,'Self-timing profiles from GAMESS application'); INSERT INTO data_source (name,id,description) VALUES ('Other',999,'Other profiles'); /* threads make it convenient to identify timer values. Special values for thread_index: -1 mean (nulls ignored) -2 total -3 stddev (nulls ignored) -4 min -5 max -6 mean (nulls are 0 value) -7 stddev (nulls are 0 value) */ CREATE TABLE derived_thread_type ( id INT NOT NULL, name VARCHAR NOT NULL, description VARCHAR NOT NULL ); INSERT INTO derived_thread_type (id, name, description) VALUES (-1, 'MEAN', 'MEAN (nulls ignored)'); INSERT INTO derived_thread_type (id, name, description) VALUES (-2, 'TOTAL', 'TOTAL'); INSERT INTO derived_thread_type (id, name, description) VALUES (-3, 'STDDEV', 'STDDEV (nulls ignored)'); INSERT INTO derived_thread_type (id, name, description) VALUES (-4, 'MIN', 'MIN'); INSERT INTO derived_thread_type (id, name, description) VALUES (-5, 'MAX', 'MAX'); INSERT INTO derived_thread_type (id, name, description) VALUES (-6, 'MEAN', 'MEAN (nulls are 0 value)'); INSERT INTO derived_thread_type (id, name, description) VALUES (-7, 'STDDEV', 'STDDEV (nulls are 0 value)'); /**************************/ /* CREATE THE TRIAL TABLE */ /**************************/ /* trials are the top level table */ CREATE TABLE trial ( id SERIAL NOT NULL PRIMARY KEY, name VARCHAR, /* where did this data come from? */ data_source INT, /* number of processes */ node_count INT, /* legacy values - these are actually "max" values - i.e. not all nodes have * this many threads */ contexts_per_node INT, /* how many threads per node? */ threads_per_context INT, /* total number of threads */ total_threads INT, /* reference to the data source table. */ FOREIGN KEY(data_source) REFERENCES data_source(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /******************************/ /* CREATE THE DATA DIMENSIONS */ /******************************/ /* threads are the "location" dimension */ CREATE TABLE thread ( id SERIAL NOT NULL PRIMARY KEY, /* trial this thread belongs to */ trial INT NOT NULL, /* process rank, really */ node_rank INT NOT NULL, /* legacy value */ context_rank INT NOT NULL, /* thread rank relative to the process */ thread_rank INT NOT NULL, /* thread index from 0 to N-1 */ thread_index INT NOT NULL, FOREIGN KEY(trial) REFERENCES trial(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* metrics are things like num_calls, num_subroutines, TIME, PAPI counters, and derived metrics. */ CREATE TABLE metric ( id SERIAL NOT NULL PRIMARY KEY, /* trial this value belongs to */ trial INT NOT NULL, /* name of the metric */ name VARCHAR NOT NULL, /* if this metric is derived by one of the tools */ derived BOOLEAN NOT NULL DEFAULT FALSE, FOREIGN KEY(trial) REFERENCES trial(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* timers are timers, capturing some interval value. For callpath or phase profiles, the parent refers to the calling function or phase. */ CREATE TABLE timer ( id SERIAL NOT NULL PRIMARY KEY, /* trial this value belongs to */ trial INT NOT NULL, /* name of the timer */ name VARCHAR NOT NULL, /* short name of the timer - without source or parameter info */ short_name VARCHAR NOT NULL, /* filename */ source_file VARCHAR, /* line number of the start of the block of code */ line_number INT, /* line number of the end of the block of code */ line_number_end INT, /* column number of the start of the block of code */ column_number INT, /* column number of the end of the block of code */ column_number_end INT, FOREIGN KEY(trial) REFERENCES trial(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* timer index on the trial and name columns */ CREATE INDEX timer_trial_index on timer (trial, name); /***********************************/ /* CREATE THE TIMER RELATED TABLES */ /***********************************/ /* timer groups are the groups such as TAU_DEFAULT, MPI, OPENMP, TAU_PHASE, TAU_CALLPATH, TAU_PARAM, etc. This mapping table allows for NxN mappings between timers and groups */ CREATE TABLE timer_group ( timer INT, group_name VARCHAR NOT NULL, FOREIGN KEY(timer) REFERENCES timer(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* index for faster queries into groups */ CREATE INDEX timer_group_index on timer_group (timer, group_name); /* timer parameters are parameter based profile values. * an example is foo (x,y) where x=4 and y=10. In that example, * timer would be the index of the timer with the * name 'foo (x,y) <x>=<4> <y>=<10>'. This table would have two * entries, one for the x value and one for the y value. */ CREATE TABLE timer_parameter ( timer INT, parameter_name VARCHAR NOT NULL, parameter_value VARCHAR NOT NULL, FOREIGN KEY(timer) REFERENCES timer(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* timer callpath have the information about the call graph in a trial. * If the profile is "flat", these will all have no parents. Otherwise, * the parent points to a node in the callgraph, the calling timer * (function). */ CREATE TABLE timer_callpath ( id SERIAL NOT NULL PRIMARY KEY, /* what timer is this? */ timer INT NOT NULL, /* what is the parent timer? */ parent INT, FOREIGN KEY(timer) REFERENCES timer(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(parent) REFERENCES timer_callpath(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* By definition, profiles have no time data. However, there are a few * examples where time ranges make sense, such as tracking call stacks * or associating metadata to a particular phase. The time_range table * is used to give other measurements a time context. The iteration * start and end can be used to indicate which loop iterations or * calls to a function are relevant for this time range. */ CREATE TABLE time_range ( id SERIAL NOT NULL PRIMARY KEY, /* starting iteration */ iteration_start INT NOT NULL, /* ending iteration. */ iteration_end INT, /* starting timestamp */ time_start BIGINT NOT NULL, /* ending timestamp. */ time_end BIGINT ); /* timer_call_data records have the dynamic information for when a node * in the callgraph is visited by a thread. If you are tracking dynamic * callstacks, you would use the time_range field. If you are storing * snapshot data, you would use the time_range field. */ CREATE TABLE timer_call_data ( id SERIAL NOT NULL PRIMARY KEY, /* what callgraph node is this? */ timer_callpath INT NOT NULL, /* what thread is this? */ thread INT NOT NULL, /* how many times this timer was called */ calls INT, /* how many subroutines this timer called */ subroutines INT, /* what is the time_range? this is for supporting snapshots */ time_range INT, FOREIGN KEY(timer_callpath) REFERENCES timer_callpath(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(thread) REFERENCES thread(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(time_range) REFERENCES time_range(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* timer values have the timer of one timer on one thread for one metric, at one location on the callgraph. */ CREATE TABLE timer_value ( /* what node in the callgraph and thread is this? */ timer_call_data INT NOT NULL, /* what metric is this? */ metric INT NOT NULL, /* The inclusive value for this timer */ inclusive_value DOUBLE PRECISION, /* The exclusive value for this timer */ exclusive_value DOUBLE PRECISION, /* The inclusive percent for this timer */ inclusive_percent DOUBLE PRECISION, /* The exclusive percent for this timer */ exclusive_percent DOUBLE PRECISION, /* The variance for this timer */ sum_exclusive_squared DOUBLE PRECISION, FOREIGN KEY(timer_call_data) REFERENCES timer_call_data(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(metric) REFERENCES metric(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* one metric, one thread, one timer */ CREATE INDEX timer_value_index on timer_value (timer_call_data, metric); /*************************************/ /* CREATE THE COUNTER RELATED TABLES */ /*************************************/ /* counters measure some counted value. */ CREATE TABLE counter ( id SERIAL NOT NULL PRIMARY KEY, trial INT NOT NULL, name VARCHAR NOT NULL, FOREIGN KEY(trial) REFERENCES trial(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* counter index on the trial and name columns */ CREATE INDEX counter_trial_index on counter (trial, name); CREATE TABLE counter_value ( /* what counter is this? */ counter INT NOT NULL, /* where in the callgraph? */ timer_callpath INT, /* what thread is this? */ thread INT NOT NULL, /* The total number of samples */ sample_count INT, /* The maximum value seen */ maximum_value DOUBLE PRECISION, /* The minimum value seen */ minimum_value DOUBLE PRECISION, /* The mean value seen */ mean_value DOUBLE PRECISION, /* The variance for this counter */ standard_deviation DOUBLE PRECISION, FOREIGN KEY(counter) REFERENCES counter(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(timer_callpath) REFERENCES timer_callpath(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(thread) REFERENCES thread(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* one thread, one counter */ CREATE INDEX counter_value_index on counter_value (counter, thread); /**************************************/ /* CREATE THE METADATA RELATED TABLES */ /**************************************/ /* primary metadata is metadata that is not nested, does not contain unique data for each thread. */ CREATE TABLE primary_metadata ( trial INT NOT NULL, name VARCHAR NOT NULL, value VARCHAR, FOREIGN KEY(trial) REFERENCES trial(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* create an index for faster queries against the primary_metadata table */ CREATE INDEX primary_metadata_index on primary_metadata (trial, name); /* secondary metadata is metadata that could be nested, could contain unique data for each thread, and could be an array. */ CREATE TABLE secondary_metadata ( id VARCHAR NOT NULL PRIMARY KEY, /* trial this value belongs to */ trial INT NOT NULL, /* this metadata value could be associated with a thread */ thread INT, /* this metadata value could be associated with a timer that happened */ timer_callpath INT, /* which call to the context timer was this? */ time_range INT, /* this metadata value could be a nested structure */ parent VARCHAR, /* the name of the metadata field */ name VARCHAR NOT NULL, /* the value of the metadata field */ value VARCHAR, /* this metadata value could be an array - so tokenize it */ is_array BOOLEAN DEFAULT FALSE, FOREIGN KEY(trial) REFERENCES trial(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(thread) REFERENCES thread(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(timer_callpath) REFERENCES timer_callpath(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(parent) REFERENCES secondary_metadata(id) ON DELETE NO ACTION ON UPDATE NO ACTION, FOREIGN KEY(time_range) REFERENCES time_range(id) ON DELETE NO ACTION ON UPDATE NO ACTION ); /* create an index for faster queries against the secondary_metadata table */ CREATE INDEX secondary_metadata_index on secondary_metadata (trial, name, thread, parent); /**************************************/ /* CREATE THE METADATA RELATED TABLES */ /**************************************/ /* this is the view table, which organizes and filters trials */ create table taudb_view ( id SERIAL NOT NULL PRIMARY KEY, /* views can be nested */ parent INTEGER NULL, /* name of the view */ name VARCHAR NOT NULL, /* view conjoin type for parameters */ conjoin VARCHAR NOT NULL, FOREIGN KEY (parent) REFERENCES taudb_view(id) ON DELETE CASCADE ON UPDATE CASCADE ); create table taudb_view_parameter ( /* the view ID */ taudb_view INTEGER NOT NULL, /* the table name for the where clause */ table_name VARCHAR NOT NULL, /* the column name for the where clause. If the table_name is one of the metadata tables, this is the value of the "name" column */ column_name VARCHAR NOT NULL, /* the operator for the where clause */ operator VARCHAR NOT NULL, /* the value for the where clause */ value VARCHAR NOT NULL, FOREIGN KEY (taudb_view) REFERENCES taudb_view(id) ON DELETE CASCADE ON UPDATE CASCADE ); /* simple view of all trials */ INSERT INTO taudb_view (parent, name, conjoin) VALUES (NULL, 'All Trials', 'and'); /* must have a parameter or else the sub views for this view do not work correctly*/ INSERT INTO taudb_view_parameter (taudb_view, table_name, column_name, operator, value) VALUES (1, 'trial', 'total_threads', '>', '-1'); /* the application and experiment columns are not used in the latest schema, but keeping them makes the code in PerfExplorer simpler. */ create table analysis_settings ( id SERIAL NOT NULL PRIMARY KEY, taudb_view INTEGER NULL, application INTEGER NULL, experiment INTEGER NULL, trial INTEGER NULL, metric INTEGER NULL, method VARCHAR(255) NOT NULL, dimension_reduction VARCHAR(255) NOT NULL, normalization VARCHAR(255) NOT NULL, FOREIGN KEY (taudb_view) REFERENCES taudb_view(id) ON DELETE CASCADE ON UPDATE CASCADE, FOREIGN KEY (trial) REFERENCES trial(id) ON DELETE CASCADE ON UPDATE CASCADE, FOREIGN KEY (metric) REFERENCES metric(id) ON DELETE CASCADE ON UPDATE CASCADE ); create table analysis_result ( id SERIAL NOT NULL PRIMARY KEY, analysis_settings INTEGER NOT NULL, description VARCHAR(255) NOT NULL, thumbnail_size INTEGER NULL, image_size INTEGER NULL, thumbnail BYTEA NULL, image BYTEA NULL, result_type INTEGER NOT NULL ); /* Performance indexes! */ create index trial_name_index on trial(name); create index timer_name_index on timer(name); CREATE INDEX timer_callpath_parent on timer_callpath(parent); CREATE INDEX thread_trial on thread(trial); CREATE INDEX timer_call_data_timer_callpath on timer_call_data(timer_callpath); CREATE INDEX counter_name_index on counter(name); CREATE INDEX timer_call_data_thread on timer_call_data(thread); /* SHORT TERM FIX! These views make sure that charts (mostly) work... for now. */ DROP VIEW IF EXISTS interval_location_profile; DROP VIEW IF EXISTS interval_mean_summary; DROP VIEW IF EXISTS interval_total_summary; DROP VIEW IF EXISTS interval_event_value; DROP VIEW IF EXISTS interval_event; DROP VIEW IF EXISTS atomic_location_profile; DROP VIEW IF EXISTS atomic_mean_summary; DROP VIEW IF EXISTS atomic_total_summary; DROP VIEW IF EXISTS atomic_event_value; DROP VIEW IF EXISTS atomic_event; CREATE OR REPLACE VIEW interval_event (id, trial, name, group_name, source_file, line_number, line_number_end) AS SELECT tcp.id, t.trial, t.name, tg.group_name, t.source_file, t.line_number, t.line_number_end FROM timer_callpath tcp INNER JOIN timer t ON tcp.timer = t.id INNER JOIN timer_group tg ON tg.timer = t.id; CREATE OR REPLACE VIEW interval_event_value (interval_event, node, context, thread, metric, inclusive_percentage, inclusive, exclusive_percentage, exclusive, call, subroutines, inclusive_per_call, sum_exclusive_squared) AS SELECT tcd.timer_callpath, t.node_rank, t.context_rank, t.thread_rank, tv.metric, tv.inclusive_percent, tv.inclusive_value, tv.exclusive_percent, tv.exclusive_value, tcd.calls, tcd.subroutines, tv.inclusive_value / tcd.calls, tv.sum_exclusive_squared FROM timer_value tv INNER JOIN timer_call_data tcd on tv.timer_call_data = tcd.id INNER JOIN thread t on tcd.thread = t.id; CREATE OR REPLACE VIEW interval_location_profile AS SELECT * from interval_event_value WHERE thread >= 0; CREATE OR REPLACE VIEW interval_total_summary AS SELECT * from interval_event_value WHERE thread = -2; CREATE OR REPLACE VIEW interval_mean_summary AS SELECT * from interval_event_value WHERE thread = -1; CREATE OR REPLACE VIEW atomic_event (id, trial, name, group_name, source_file, line_number) AS SELECT c.id, c.trial, c.name, NULL, NULL, NULL FROM counter c; CREATE OR REPLACE VIEW atomic_event_value (atomic_event, node, context, thread, sample_count, maximum_value, minimum_value, mean_value, standard_deviation) AS SELECT cv.counter, t.node_rank, t.context_rank, t.thread_rank, cv.sample_count, cv.maximum_value, cv.minimum_value, cv.mean_value, cv.standard_deviation FROM counter_value cv INNER JOIN thread t ON cv.thread = t.id; CREATE OR REPLACE VIEW atomic_location_profile AS SELECT * FROM atomic_event_value WHERE thread >= 0; CREATE OR REPLACE VIEW atomic_total_summary AS SELECT * FROM atomic_event_value WHERE thread = -2; CREATE OR REPLACE VIEW atomic_mean_summary AS SELECT * FROM atomic_event_value WHERE thread >= -1;
TAUdb C API
TAUdb C API Overview
The C API for TAUdb is currently under development, but there is a beta version of the API available. The API provides the following capabilities:
-
Loading trials from the database
-
Inserting trials into the database
-
Parsing TAU profile files
TAUdb C Structures
The C structures are roughly organized as a tree, with a trial object at the root.
-
taudb_trial: A top-level structure which contains the collections of all the performance data dimensions.
-
taudb_primary_metadata: Name/value pairs which describe the properties of the trial.
-
taudb_secondary_metadata: Name/value pairs which describe the properties of the trial. Unlike primary_metadata values, secondary_metadata objects can have complex value types. They are also associated with a measurement context - a thread of execution, a timer, a timestamp, an iteration, etc.
-
taudb_thread: A structure which represents a thread of execution in the parallel measurement.
-
taudb_time_range: A structure which holds a time-range value of beginning and ending iteration numbers or timestamps.
-
taudb_metric: A structure which represents a unit of measurement, such as TIME, FP_OPS, L1_DCM, etc.
-
taudb_timer: A structure which represents a region of code. For example, a phase, a function, a loop, a basic block, or even a line of code.
-
taudb_timer_parameter: A structure which represents parameter values, when parameter based profiling is used.
-
taudb_timer_group: A structure which represents a semantic grouping of timers, such as "I/O", "MPI", "OpenMP", etc.
-
taudb_timer_callpath: A structure which represents a node in the dynamic callpath tree. Timer_callpaths with a null parent are either top level timers, or a timers in a flat profile.
-
taudb_timer_call_data: A structure which represents a tuple between a thread of execution and a node on the timer callpath tree.
-
taudb_timer_value: A structure which represents a tuple between a timer_call_data object and a metric. The timer_value contains the measurement of one metric for one timer on one thread of execution.
-
taudb_counter: A structure which represents a counter in the profile. For example, the number of bytes transferred on an MPI_Send() timer.
-
taudb_counter_value: A structure which represents a counter measurement on one thread of execution.
Below are the object definitions, from the TAUdb C header file.
#ifndef TAUDB_STRUCTS_H #define TAUDB_STRUCTS_H 1 #include "time.h" #include "uthash.h" #include "taudb_structs.h" #if defined __TAUDB_POSTGRESQL__ #include "libpq-fe.h" #elif defined __TAUDB_SQLITE__ #include "sqlite3.h" #endif #ifndef boolean #define TRUE 1 #define FALSE 0 typedef int boolean; #endif typedef struct taudb_prepared_statement { char* name; UT_hash_handle hh; /* hash index for hashing by name */ } TAUDB_PREPARED_STATEMENT; /* forward declarations to ease objects that need to know about * each other and have doubly-linked relationships */ struct taudb_timer_call_data; struct taudb_timer_value; struct taudb_timer_callpath; struct taudb_timer_group; struct taudb_timer_parameter; struct taudb_timer; struct taudb_counter_value; struct taudb_counter; struct taudb_primary_metadata; struct taudb_secondary_metadata; struct taudb_time_range; struct taudb_thread; struct taudb_metric; struct taudb_trial; struct perfdmf_experiment; struct perfdmf_application; typedef struct taudb_configuration { char* jdbc_db_type; /* to identify DBMS vendor. * postgresql, mysql, h2, derby, etc. */ char* db_hostname; /* server host name */ char* db_portnum; /* server port number */ char* db_dbname; /* the database name at the server */ char* db_schemaprefix; /* the schema prefix. This is appended to * all table names for some DBMSs */ char* db_username; /* the database username */ char* db_password; /* the database password for username */ char* db_schemafile; /* full or relative path to the schema file, * used for configuration, not used in C API */ } TAUDB_CONFIGURATION; typedef enum taudb_database_schema_version { TAUDB_2005_SCHEMA, TAUDB_2012_SCHEMA } TAUDB_SCHEMA_VERSION; typedef struct taudb_data_source { int id; char* name; char*description; UT_hash_handle hh1; /* hash index for hashing by id */ UT_hash_handle hh2; /* hash index for hashing by name */ } TAUDB_DATA_SOURCE; typedef struct taudb_connection { TAUDB_CONFIGURATION *configuration; #if defined __TAUDB_POSTGRESQL__ PGconn *connection; PGresult *res; TAUDB_PREPARED_STATEMENT *statements; #elif defined __TAUDB_SQLITE__ sqlite3 *connection; sqlite3_stmt *ppStmt; int rc; #endif TAUDB_SCHEMA_VERSION schema_version; boolean inTransaction; boolean inPortal; TAUDB_DATA_SOURCE* data_sources_by_id; TAUDB_DATA_SOURCE* data_sources_by_name; } TAUDB_CONNECTION; /* these are the derived thread indexes. */ #define TAUDB_MEAN_WITHOUT_NULLS -1 #define TAUDB_TOTAL -2 #define TAUDB_STDDEV_WITHOUT_NULLS -3 #define TAUDB_MIN -4 #define TAUDB_MAX -5 #define TAUDB_MEAN_WITH_NULLS -6 #define TAUDB_STDDEV_WITH_NULLS -7 /* trials are the top level structure */ typedef struct taudb_trial { /* actual data from the database */ int id; char* name; struct taudb_data_source* data_source; int node_count; /* i.e. number of processes. */ int contexts_per_node; /* rarely used, usually 1. */ int threads_per_context; /* max number of threads per process * (can be less on individual processes) */ int total_threads; /* total number of threads */ /* arrays of data for this trial */ struct taudb_metric* metrics_by_id; struct taudb_metric* metrics_by_name; struct taudb_thread* threads; struct taudb_time_range* time_ranges; struct taudb_timer* timers_by_id; struct taudb_timer* timers_by_name; struct taudb_timer_group* timer_groups; struct taudb_timer_callpath* timer_callpaths_by_id; struct taudb_timer_callpath* timer_callpaths_by_name; struct taudb_timer_call_data* timer_call_data_by_id; struct taudb_timer_call_data* timer_call_data_by_key; struct taudb_counter* counters_by_id; struct taudb_counter* counters_by_name; struct taudb_counter_value* counter_values; struct taudb_primary_metadata* primary_metadata; struct taudb_secondary_metadata* secondary_metadata; struct taudb_secondary_metadata* secondary_metadata_by_key; } TAUDB_TRIAL; /*********************************************/ /* data dimensions */ /*********************************************/ /* thread represents one physical & logical * location for a measurement. */ typedef struct taudb_thread { int id; /* database id, also key to hash */ struct taudb_trial* trial; int node_rank; /* which process does this thread belong to? */ int context_rank; /* which context? USUALLY 0 */ int thread_rank; /* what is this thread's rank in the process */ int index; /* what is this threads OVERALL index? * ranges from 0 to trial.thread_count-1 */ struct taudb_secondary_metadata* secondary_metadata; UT_hash_handle hh; } TAUDB_THREAD; /* metrics are things like TIME, PAPI counters, and derived metrics. */ typedef struct taudb_metric { int id; /* database value, also key to hash */ char* name; /* key to hash hh2 */ boolean derived; /* was this metric measured, or created by a * post-processing tool? */ UT_hash_handle hh1; /* hash index for hashing by id */ UT_hash_handle hh2; /* hash index for hashing by name */ } TAUDB_METRIC; /* Time ranges are ways to delimit the profile data within time ranges. They are also useful for secondary metadata which is associated with a specific call to a function. */ typedef struct taudb_time_range { int id; /* database value, also key to hash */ int iteration_start; int iteration_end; uint64_t time_start; uint64_t time_end; /* was this metric measured, * or created by a post-processing tool? */ UT_hash_handle hh; } TAUDB_TIME_RANGE; /* timers are interval timers, capturing some interval value. * For callpath or phase profiles, the parent refers to the calling * function or phase. Timers can also be sample locations, or * phases (dynamic or static), or sample aggregations (intermediate) */ typedef struct taudb_timer { int id; /* database value, also key to hash */ struct taudb_trial* trial; /* pointer back to trial - NOTE: Necessary? */ char* name; /* the full timer name, can have file, line, etc. */ char* short_name; /* just the function name, for example */ char* source_file; /* what source file does this function live in? */ int line_number; /* what line does the timer start on? */ int line_number_end; /* what line does the timer end on? */ int column_number; /* what column number does the timer start on? */ int column_number_end; /* what column number does the timer end on? */ struct taudb_timer_group* groups; /* hash of groups, * using group hash handle hh2 */ struct taudb_timer_parameter* parameters; /* array of parameters */ UT_hash_handle trial_hash_by_id; /* hash key for id lookup */ UT_hash_handle trial_hash_by_name; /* hash key for name lookup * in temporary hash */ UT_hash_handle group_hash_by_name; /* hash key for name lookup * in timer group */ } TAUDB_TIMER; /*********************************************/ /* timer related structures */ /*********************************************/ /* timer groups are the groups such as tau_default, mpi, openmp, tau_phase, tau_callpath, tau_param, etc. this mapping table allows for nxn mappings between timers and groups */ typedef struct taudb_timer_group { char* name; struct taudb_timer* timers; /* hash of timers, * using timer hash handle hh3 */ UT_hash_handle trial_hash_by_name; // hash handle for trial UT_hash_handle timer_hash_by_name; // hash handle for timers } TAUDB_TIMER_GROUP; /* timer parameters are parameter based profile values. an example is foo (x,y) where x=4 and y=10. in that example, timer would be the index of the timer with the name 'foo (x,y) <x>=<4> <y>=<10>'. this table would have two entries, one for the x value and one for the y value. The parameter can also be a phase / iteration index. */ typedef struct taudb_timer_parameter { char* name; char* value; UT_hash_handle hh; } TAUDB_TIMER_PARAMETER; /* callpath objects contain the merged dynamic callgraph tree seen * during execution */ typedef struct taudb_timer_callpath { int id; /* link back to database, and hash key */ struct taudb_timer* timer; /* which timer is this? */ struct taudb_timer_callpath *parent; /* callgraph parent */ char* name; /* a string which has the aggregated callpath. */ UT_hash_handle hh1; /* hash key for hash by id */ UT_hash_handle hh2; /* hash key for name (a => b => c...) lookup */ } TAUDB_TIMER_CALLPATH; /* timer_call_data objects are observations of a node of the callgraph for one of the threads. */ typedef struct taudb_call_data_key { struct taudb_timer_callpath *timer_callpath; /* link back to database */ struct taudb_thread *thread; /* link back to database, roundabout way */ char* timestamp; /* timestamp in case we are in a snapshot or something */ } TAUDB_TIMER_CALL_DATA_KEY; typedef struct taudb_timer_call_data { int id; /* link back to database */ TAUDB_TIMER_CALL_DATA_KEY key; /* hash table key */ int calls; /* number of times this timer was seen */ int subroutines; /* number of timers this timer calls */ struct taudb_timer_value* timer_values; UT_hash_handle hh1; UT_hash_handle hh2; } TAUDB_TIMER_CALL_DATA; /* finally, timer_values are specific measurements during one of the observations of the node of the callgraph on a thread. */ typedef struct taudb_timer_value { struct taudb_metric* metric; /* which metric is this? */ double inclusive; /* the inclusive value of this metric */ double exclusive; /* the exclusive value of this metric */ double inclusive_percentage; /* the inclusive percentage of * total time of the application */ double exclusive_percentage; /* the exclusive percentage of * total time of the application */ double sum_exclusive_squared; /* how much variance did we see * every time we measured this timer? */ char *key; /* hash table key - metric name */ UT_hash_handle hh; } TAUDB_TIMER_VALUE; /*********************************************/ /* counter related structures */ /*********************************************/ /* counters measure some counted value. An example would be MPI message size * for an MPI_Send. */ typedef struct taudb_counter { int id; /* database reference */ struct taudb_trial* trial; char* name; UT_hash_handle hh1; /* hash key for hashing by id */ UT_hash_handle hh2; /* hash key for hashing by name */ } TAUDB_COUNTER; /* counters are atomic counters, not just interval timers */ typedef struct taudb_counter_value_key { struct taudb_counter* counter; /* the counter we are measuring */ struct taudb_thread* thread; /* where this measurement is */ struct taudb_timer_callpath* context; /* the calling context (can be null) */ char* timestamp; /* timestamp in case we are in a snapshot or something */ } TAUDB_COUNTER_VALUE_KEY; typedef struct taudb_counter_value { TAUDB_COUNTER_VALUE_KEY key; int sample_count; /* how many times did we see take this count? */ double maximum_value; /* what was the max value we saw? */ double minimum_value; /* what was the min value we saw? */ double mean_value; /* what was the average value we saw? */ double standard_deviation; /* how much variance was there? */ UT_hash_handle hh1; /* hash key for hashing by key */ } TAUDB_COUNTER_VALUE; /*********************************************/ /* metadata related structures */ /*********************************************/ /* primary metadata is metadata that is not nested, does not contain unique data for each thread. */ typedef struct taudb_primary_metadata { char* name; char* value; UT_hash_handle hh; /* uses the name as the key */ } TAUDB_PRIMARY_METADATA; /* primary metadata is metadata that could be nested, could contain unique data for each thread, and could be an array. */ typedef struct taudb_secondary_metadata_key { struct taudb_timer_callpath *timer_callpath; /* link back to database */ struct taudb_thread *thread; /* link back to database, roundabout way */ struct taudb_secondary_metadata* parent; /* self-referencing */ struct taudb_time_range* time_range; char* name; } TAUDB_SECONDARY_METADATA_KEY; typedef struct taudb_secondary_metadata { char* id; /* link back to database */ TAUDB_SECONDARY_METADATA_KEY key; int num_values; /* can have arrays of data */ char** value; int child_count; struct taudb_secondary_metadata* children; /* self-referencing */ UT_hash_handle hh; /* uses the id as a compound key */ UT_hash_handle hh2; /* uses the key as a compound key */ } TAUDB_SECONDARY_METADATA; /* these are for supporting the older schema */ typedef struct perfdmf_experiment { int id; char* name; struct taudb_primary_metadata* primary_metadata; } PERFDMF_EXPERIMENT; typedef struct perfdmf_application { int id; char* name; struct taudb_primary_metadata* primary_metadata; } PERFDMF_APPLICATION; #endif /* TAUDB_STRUCTS_H */
TAUdb C API
#ifndef TAUDB_API_H #define TAUDB_API_H 1 #include "taudb_structs.h" /* when a "get" function is called, this global has the number of top-level objects that are returned. */ extern int taudb_numItems; /* the database version */ extern enum taudb_database_schema_version taudb_version; /* to connect to the database */ extern TAUDB_CONNECTION* taudb_connect_config(char* config_name); extern TAUDB_CONNECTION* taudb_connect_config_file(char* config_file_name); /* test the connection status */ extern int taudb_check_connection(TAUDB_CONNECTION* connection); /* disconnect from the database */ extern int taudb_disconnect(TAUDB_CONNECTION* connection); /************************************************/ /* query functions */ /************************************************/ /* functions to support the old database schema - avoid these if you can */ extern PERFDMF_APPLICATION* perfdmf_query_applications(TAUDB_CONNECTION* connection); extern PERFDMF_EXPERIMENT* perfdmf_query_experiments(TAUDB_CONNECTION* connection, PERFDMF_APPLICATION* application); extern PERFDMF_APPLICATION* perfdmf_query_application(TAUDB_CONNECTION* connection, char* name); extern PERFDMF_EXPERIMENT* perfdmf_query_experiment(TAUDB_CONNECTION* connection, PERFDMF_APPLICATION* application, char* name); extern TAUDB_TRIAL* perfdmf_query_trials(TAUDB_CONNECTION* connection, PERFDMF_EXPERIMENT* experiment); /* get the data sources */ extern TAUDB_DATA_SOURCE* taudb_query_data_sources(TAUDB_CONNECTION* connection); extern TAUDB_DATA_SOURCE* taudb_get_data_source_by_id(TAUDB_DATA_SOURCE* data_sources, const int id); extern TAUDB_DATA_SOURCE* taudb_get_data_source_by_name(TAUDB_DATA_SOURCE* data_sources, const char* name); /* using the properties set in the filter, find a set of trials */ extern TAUDB_TRIAL* taudb_query_trials(TAUDB_CONNECTION* connection, boolean complete, TAUDB_TRIAL* filter); extern TAUDB_PRIMARY_METADATA* taudb_query_primary_metadata(TAUDB_CONNECTION* connection, TAUDB_TRIAL* filter); extern TAUDB_PRIMARY_METADATA* taudb_get_primary_metadata_by_name(TAUDB_PRIMARY_METADATA* primary_metadata, const char* name); extern TAUDB_SECONDARY_METADATA* taudb_query_secondary_metadata(TAUDB_CONNECTION* connection, TAUDB_TRIAL* filter); /* get the threads for a trial */ extern TAUDB_THREAD* taudb_query_threads(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_THREAD* taudb_query_derived_threads(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_THREAD* taudb_get_thread(TAUDB_THREAD* threads, int thread_index); extern int taudb_get_total_threads(TAUDB_THREAD* threads); /* get the metrics for a trial */ extern TAUDB_METRIC* taudb_query_metrics(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_METRIC* taudb_get_metric_by_name(TAUDB_METRIC* metrics, const char* name); extern TAUDB_METRIC* taudb_get_metric_by_id(TAUDB_METRIC* metrics, const int id); /* get the time_ranges for a trial */ extern TAUDB_TIME_RANGE* taudb_query_time_range(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_TIME_RANGE* taudb_get_time_range(TAUDB_TIME_RANGE* time_ranges, const int id); /* get the timers for a trial */ extern TAUDB_TIMER* taudb_query_timers(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_TIMER* taudb_get_timer_by_id(TAUDB_TIMER* timers, int id); extern TAUDB_TIMER* taudb_get_trial_timer_by_name(TAUDB_TIMER* timers, const char* id); extern TAUDB_TIMER* taudb_get_trial_timer_by_name(TAUDB_TIMER* timers, const char* id); extern TAUDB_TIMER_GROUP* taudb_query_timer_groups(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern void taudb_parse_timer_group_names(TAUDB_TRIAL* trial, TAUDB_TIMER* timer, char* group_names); extern TAUDB_TIMER_GROUP* taudb_get_timer_group_from_trial_by_name(TAUDB_TIMER_GROUP* timers, const char* name); extern TAUDB_TIMER_GROUP* taudb_get_timer_group_from_timer_by_name(TAUDB_TIMER_GROUP* timers, const char* name); extern TAUDB_TIMER_CALLPATH* taudb_query_timer_callpaths(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, TAUDB_TIMER* timer); extern TAUDB_TIMER_CALLPATH* taudb_get_timer_callpath_by_id(TAUDB_TIMER_CALLPATH* timers, int id); extern TAUDB_TIMER_CALLPATH* taudb_get_timer_callpath_by_name(TAUDB_TIMER_CALLPATH* timers, const char* id); extern TAUDB_TIMER_CALLPATH* taudb_query_all_timer_callpaths(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern char* taudb_get_callpath_string(TAUDB_TIMER_CALLPATH* timer_callpath); /* get the counters for a trial */ extern TAUDB_COUNTER* taudb_query_counters(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_COUNTER* taudb_get_counter_by_id(TAUDB_COUNTER* counters, int id); extern TAUDB_COUNTER* taudb_get_counter_by_name(TAUDB_COUNTER* counters, const char* id); extern TAUDB_COUNTER_VALUE* taudb_query_counter_values(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); TAUDB_COUNTER_VALUE* taudb_get_counter_value(TAUDB_COUNTER_VALUE* counter_values, TAUDB_COUNTER* counter, TAUDB_THREAD* thread, TAUDB_TIMER_CALLPATH* context, char* timestamp); /* get the timer call data for a trial */ extern TAUDB_TIMER_CALL_DATA* taudb_query_timer_call_data(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, TAUDB_TIMER_CALLPATH* timer_callpath, TAUDB_THREAD* thread); extern TAUDB_TIMER_CALL_DATA* taudb_query_all_timer_call_data(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_TIMER_CALL_DATA* taudb_query_timer_call_data_stats(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, TAUDB_TIMER_CALLPATH* timer_callpath, TAUDB_THREAD* thread); extern TAUDB_TIMER_CALL_DATA* taudb_query_all_timer_call_data_stats(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_TIMER_CALL_DATA* taudb_get_timer_call_data_by_id(TAUDB_TIMER_CALL_DATA* timer_call_data, int id); extern TAUDB_TIMER_CALL_DATA* taudb_get_timer_call_data_by_key(TAUDB_TIMER_CALL_DATA* timer_call_data, TAUDB_TIMER_CALLPATH* callpath, TAUDB_THREAD* thread, char* timestamp); /* get the timer values for a trial */ extern TAUDB_TIMER_VALUE* taudb_query_timer_values(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, TAUDB_TIMER_CALLPATH* timer_callpath, TAUDB_THREAD* thread, TAUDB_METRIC* metric); extern TAUDB_TIMER_VALUE* taudb_query_timer_stats(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, TAUDB_TIMER_CALLPATH* timer_callpath, TAUDB_THREAD* thread, TAUDB_METRIC* metric); extern TAUDB_TIMER_VALUE* taudb_query_all_timer_values(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_TIMER_VALUE* taudb_query_all_timer_stats(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); extern TAUDB_TIMER_VALUE* taudb_get_timer_value(TAUDB_TIMER_CALL_DATA* timer_call_data, TAUDB_METRIC* metric); /* find main */ extern TAUDB_TIMER* taudb_query_main_timer(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial); /* save everything */ extern void taudb_save_trial(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update, boolean cascade); extern void taudb_save_threads(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_metrics(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_timers(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_time_ranges(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_timer_groups(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_timer_parameters(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_timer_callpaths(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_timer_call_data(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_timer_values(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_counters(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_counter_values(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_primary_metadata(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); extern void taudb_save_secondary_metadata(TAUDB_CONNECTION* connection, TAUDB_TRIAL* trial, boolean update); /************************************************/ /* memory functions */ /************************************************/ extern char* taudb_strdup(const char* in_string); extern TAUDB_TRIAL* taudb_create_trials(int count); extern TAUDB_METRIC* taudb_create_metrics(int count); extern TAUDB_TIME_RANGE* taudb_create_time_ranges(int count); extern TAUDB_THREAD* taudb_create_threads(int count); extern TAUDB_SECONDARY_METADATA* taudb_create_secondary_metadata(int count); extern TAUDB_PRIMARY_METADATA* taudb_create_primary_metadata(int count); extern TAUDB_PRIMARY_METADATA* taudb_resize_primary_metadata(int count, TAUDB_PRIMARY_METADATA* old_primary_metadata); extern TAUDB_COUNTER* taudb_create_counters(int count); extern TAUDB_COUNTER_VALUE* taudb_create_counter_values(int count); extern TAUDB_TIMER* taudb_create_timers(int count); extern TAUDB_TIMER_PARAMETER* taudb_create_timer_parameters(int count); extern TAUDB_TIMER_GROUP* taudb_create_timer_groups(int count); extern TAUDB_TIMER_GROUP* taudb_resize_timer_groups(int count, TAUDB_TIMER_GROUP* old_groups); extern TAUDB_TIMER_CALLPATH* taudb_create_timer_callpaths(int count); extern TAUDB_TIMER_CALL_DATA* taudb_create_timer_call_data(int count); extern TAUDB_TIMER_VALUE* taudb_create_timer_values(int count); extern void taudb_delete_trials(TAUDB_TRIAL* trials, int count); /************************************************/ /* Adding objects to the hierarchy */ /************************************************/ extern void taudb_add_metric_to_trial(TAUDB_TRIAL* trial, TAUDB_METRIC* metric); extern void taudb_add_time_range_to_trial(TAUDB_TRIAL* trial, TAUDB_TIME_RANGE* time_range); extern void taudb_add_thread_to_trial(TAUDB_TRIAL* trial, TAUDB_THREAD* thread); extern void taudb_add_secondary_metadata_to_trial(TAUDB_TRIAL* trial, TAUDB_SECONDARY_METADATA* secondary_metadata); extern void taudb_add_secondary_metadata_to_secondary_metadata (TAUDB_SECONDARY_METADATA* parent, TAUDB_SECONDARY_METADATA* child); extern void taudb_add_primary_metadata_to_trial(TAUDB_TRIAL* trial, TAUDB_PRIMARY_METADATA* primary_metadata); extern void taudb_add_counter_to_trial(TAUDB_TRIAL* trial, TAUDB_COUNTER* counter); extern void taudb_add_counter_value_to_trial(TAUDB_TRIAL* trial, TAUDB_COUNTER_VALUE* counter_value); extern void taudb_add_timer_to_trial(TAUDB_TRIAL* trial, TAUDB_TIMER* timer); extern void taudb_add_timer_parameter_to_trial(TAUDB_TRIAL* trial, TAUDB_TIMER_PARAMETER* timer_parameter); extern void taudb_add_timer_group_to_trial(TAUDB_TRIAL* trial, TAUDB_TIMER_GROUP* timer_group); extern void taudb_add_timer_to_timer_group(TAUDB_TIMER_GROUP* timer_group, TAUDB_TIMER* timer); extern void taudb_add_timer_callpath_to_trial(TAUDB_TRIAL* trial, TAUDB_TIMER_CALLPATH* timer_callpath); extern void taudb_add_timer_call_data_to_trial(TAUDB_TRIAL* trial, TAUDB_TIMER_CALL_DATA* timer_call_data); extern void taudb_add_timer_value_to_timer_call_data (TAUDB_TIMER_CALL_DATA* timer_call_data, TAUDB_TIMER_VALUE* timer_value); /* Profile parsers */ extern TAUDB_TRIAL* taudb_parse_tau_profiles(const char* directory_name); /* Analysis routines */ extern void taudb_compute_statistics(TAUDB_TRIAL* trial); /* iterators */ extern TAUDB_DATA_SOURCE* taudb_next_data_source_by_name_from_connection (TAUDB_DATA_SOURCE* current); extern TAUDB_DATA_SOURCE* taudb_next_data_source_by_id_from_connection (TAUDB_DATA_SOURCE* current); extern TAUDB_THREAD* taudb_next_thread_by_index_from_trial(TAUDB_THREAD* current); extern TAUDB_METRIC* taudb_next_metric_by_name_from_trial(TAUDB_METRIC* current); extern TAUDB_METRIC* taudb_next_metric_by_id_from_trial(TAUDB_METRIC* current); extern TAUDB_TIME_RANGE* taudb_next_time_range_by_id_from_trial(TAUDB_TIME_RANGE* current); extern TAUDB_TIMER* taudb_next_timer_by_name_from_trial(TAUDB_TIMER* current); extern TAUDB_TIMER* taudb_next_timer_by_id_from_trial(TAUDB_TIMER* current); extern TAUDB_TIMER* taudb_next_timer_by_name_from_group(TAUDB_TIMER* current); extern TAUDB_TIMER_GROUP* taudb_next_timer_group_by_name_from_trial (TAUDB_TIMER_GROUP* current); extern TAUDB_TIMER_GROUP* taudb_next_timer_group_by_name_from_timer (TAUDB_TIMER_GROUP* current); extern TAUDB_TIMER_PARAMETER* taudb_next_timer_parameter_by_name_from_timer (TAUDB_TIMER_PARAMETER* current); extern TAUDB_TIMER_CALLPATH* taudb_next_timer_callpath_by_name_from_trial (TAUDB_TIMER_CALLPATH* current); extern TAUDB_TIMER_CALLPATH* taudb_next_timer_callpath_by_id_from_trial (TAUDB_TIMER_CALLPATH* current); extern TAUDB_TIMER_CALL_DATA* taudb_next_timer_call_data_by_key_from_trial (TAUDB_TIMER_CALL_DATA* current); extern TAUDB_TIMER_CALL_DATA* taudb_next_timer_call_data_by_id_from_trial (TAUDB_TIMER_CALL_DATA* current); extern TAUDB_TIMER_VALUE* taudb_next_timer_value_by_metric_from_timer_call_data (TAUDB_TIMER_VALUE* current); extern TAUDB_COUNTER* taudb_next_counter_by_name_from_trial(TAUDB_COUNTER* current); extern TAUDB_COUNTER* taudb_next_counter_by_id_from_trial(TAUDB_COUNTER* current); extern TAUDB_COUNTER_VALUE* taudb_next_counter_value_by_key_from_trial(TAUDB_COUNTER_VALUE* current); extern TAUDB_PRIMARY_METADATA* taudb_next_primary_metadata_by_name_from_trial (TAUDB_PRIMARY_METADATA* current); extern TAUDB_SECONDARY_METADATA* taudb_next_secondary_metadata_by_key_from_trial (TAUDB_SECONDARY_METADATA* current); extern TAUDB_SECONDARY_METADATA* taudb_next_secondary_metadata_by_id_from_trial (TAUDB_SECONDARY_METADATA* current); #endif /* TAUDB_API_H */
TAUdb C API Examples
Creating a trial and inserting into the database
#include "taudb_api.h" #include <stdio.h> #include <string.h> #include <sys/types.h> #include <dirent.h> #include "dump_functions.h" int main (int argc, char** argv) { TAUDB_CONNECTION* connection = NULL; if (argc >= 2) { connection = taudb_connect_config(argv[1]); } else { fprintf(stderr, "Please specify a TAUdb config file.\n"); exit(1); } printf("Checking connection...\n"); taudb_check_connection(connection); // create a trial TAUDB_TRIAL* trial = taudb_create_trials(1); trial->name = taudb_strdup("TEST TRIAL"); // set the data source to "other" trial->data_source = taudb_get_data_source_by_id( taudb_query_data_sources(connection), 999); // create some metadata TAUDB_PRIMARY_METADATA* pm = taudb_create_primary_metadata(1); pm->name = taudb_strdup("Application"); pm->value = taudb_strdup("Test Application"); taudb_add_primary_metadata_to_trial(trial, pm); pm = taudb_create_primary_metadata(1); pm->name = taudb_strdup("Start Time"); pm->value = taudb_strdup("2012-11-07 12:30:00"); taudb_add_primary_metadata_to_trial(trial, pm); // alternatively, you can allocate the primary metadata in blocks pm = taudb_create_primary_metadata(10); pm[0].name = taudb_strdup("ClientID"); pm[0].value = taudb_strdup("joe_user"); taudb_add_primary_metadata_to_trial(trial, &(pm[0])); pm[1].name = taudb_strdup("hostname"); pm[1].value = taudb_strdup("hopper04"); taudb_add_primary_metadata_to_trial(trial, &(pm[1])); pm[2].name = taudb_strdup("Operating System"); pm[2].value = taudb_strdup("Linux"); taudb_add_primary_metadata_to_trial(trial, &(pm[2])); pm[3].name = taudb_strdup("Release"); pm[3].value = taudb_strdup("2.6.32.36-0.5-default"); taudb_add_primary_metadata_to_trial(trial, &(pm[3])); pm[4].name = taudb_strdup("Machine"); pm[4].value = taudb_strdup("Hopper.nersc.gov"); taudb_add_primary_metadata_to_trial(trial, &(pm[4])); pm[5].name = taudb_strdup("CPU Cache Size"); pm[5].value = taudb_strdup("512 KB"); taudb_add_primary_metadata_to_trial(trial, &(pm[5])); pm[6].name = taudb_strdup("CPU Clock Frequency"); pm[6].value = taudb_strdup("800.000 MHz"); taudb_add_primary_metadata_to_trial(trial, &(pm[6])); pm[7].name = taudb_strdup("CPU Model"); pm[7].value = taudb_strdup("Quad-Core AMD Opteron(tm) Processor 8378"); taudb_add_primary_metadata_to_trial(trial, &(pm[7])); // create a metric TAUDB_METRIC* metric = taudb_create_metrics(1); metric->name = taudb_strdup("TIME"); taudb_add_metric_to_trial(trial, metric); // create a thread TAUDB_THREAD* thread = taudb_create_threads(1); thread->node_rank = 1; thread->context_rank = 1; thread->thread_rank = 1; thread->index = 1; taudb_add_thread_to_trial(trial, thread); // create a timer, timer_callpath, timer_call_data, timer_value TAUDB_TIMER_GROUP* timer_group = taudb_create_timer_groups(1); TAUDB_TIMER* timer = taudb_create_timers(1); TAUDB_TIMER_CALLPATH* timer_callpath = taudb_create_timer_callpaths(1); TAUDB_TIMER_CALL_DATA* timer_call_data = taudb_create_timer_call_data(1); TAUDB_TIMER_VALUE* timer_value = taudb_create_timer_values(1); timer->name = taudb_strdup( "int main(int, char **) [{kernel.c} {134,1}-{207,1}]"); timer->short_name = taudb_strdup("main"); timer->source_file = taudb_strdup("kernel.c"); timer->line_number = 134; timer->column_number = 1; timer->line_number_end = 207; timer->column_number_end = 1; taudb_add_timer_to_trial(trial, timer); timer_group->name = taudb_strdup("TAU_DEFAULT"); taudb_add_timer_group_to_trial(trial, timer_group); taudb_add_timer_to_timer_group(timer_group, timer); timer_callpath->timer = timer; timer_callpath->parent = NULL; taudb_add_timer_callpath_to_trial(trial, timer_callpath); timer_call_data->key.timer_callpath = timer_callpath; timer_call_data->key.thread = thread; timer_call_data->calls = 1; timer_call_data->subroutines = 0; taudb_add_timer_call_data_to_trial(trial, timer_call_data); timer_value->metric = metric; // 5 seconds, or 5 million microseconds timer_value->inclusive = 5000000; timer_value->exclusive = 5000000; timer_value->inclusive_percentage = 100.0; timer_value->exclusive_percentage = 100.0; timer_value->sum_exclusive_squared = 0.0; taudb_add_timer_value_to_timer_call_data(timer_call_data, timer_value); // compute stats printf("Computing Stats...\n"); taudb_compute_statistics(trial); // save the trial! printf("Testing inserts...\n"); boolean update = FALSE; boolean cascade = TRUE; taudb_save_trial(connection, trial, update, cascade); printf("Disconnecting...\n"); taudb_disconnect(connection); printf("Done.\n"); return 0; }
Querying a trial from the database
#include "taudb_api.h" #include <stdio.h> #include <string.h> void dump_metadata(TAUDB_PRIMARY_METADATA *metadata) { printf("%d metadata fields:\n", HASH_COUNT(metadata)); TAUDB_PRIMARY_METADATA * current; for(current = metadata; current != NULL; current = taudb_next_primary_metadata_by_name_from_trial(current)) { printf(" %s = %s\n", current->name, current->value); } } void dump_secondary_metadata(TAUDB_SECONDARY_METADATA *metadata) { printf("%d secondary metadata fields:\n", HASH_COUNT(metadata)); TAUDB_SECONDARY_METADATA * current; for(current = metadata; current != NULL; current = taudb_next_secondary_metadata_by_key_from_trial(current)) { printf(" %s = %s\n", current->key.name, current->value[0]); } } void dump_trial(TAUDB_CONNECTION* connection, TAUDB_TRIAL* filter, boolean haveTrial) { TAUDB_TRIAL* trial; if (haveTrial) { trial = filter; } else { trial = taudb_query_trials(connection, FALSE, filter); } TAUDB_TIMER* timer = taudb_query_main_timer(connection, trial); printf("Trial name: '%s', id: %d, main: '%s'\n\n", trial->name, trial->id, timer->name); } int main (int argc, char** argv) { printf("Connecting...\n"); TAUDB_CONNECTION* connection = NULL; if (argc >= 2) { connection = taudb_connect_config(argv[1]); } else { fprintf(stderr, "Please specify a TAUdb config file.\n"); exit(1); } printf("Checking connection...\n"); taudb_check_connection(connection); printf("Testing queries...\n"); int t; // test the "find trials" method to populate the trial TAUDB_TRIAL* filter = taudb_create_trials(1); filter->id = atoi(argv[2]); TAUDB_TRIAL* trials = taudb_query_trials(connection, TRUE, filter); int numTrials = taudb_numItems; for (t = 0 ; t < numTrials ; t = t+1) { printf(" Trial name: '%s', id: %d\n", trials[t].name, trials[t].id); dump_metadata(trials[t].primary_metadata); dump_secondary_metadata(trials[t].secondary_metadata); dump_trial(connection, &(trials[t]), TRUE); } printf("Disconnecting...\n"); taudb_disconnect(connection); printf("Done.\n"); return 0; }