Copyright 1998 IEEE. Published in the Proceedings of HPDC-7'98, 28-31 July 1998 at Chicago, Illinois. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: + Intl. 732-562-3966.
The Distributed Array Query and Visualization (DAQV) project aims to develop systems and tools that facilitate interacting with distributed programs and data structures. Arrays distributed across the processes of a parallel or distributed application are made available to external clients via well-defined interfaces and protocols. Our design considers the broad issues of language targets, models of interaction, and abstractions for data access, while our implementation attempts to provide a general framework that can be adapted to a range of application scenarios. The paper describes the second generation of DAQV work and places it in the context of the more general distributed array access problem. Current applications and future work are also described.Keywords: parallel tools, distributed arrays, visualization, computational steering, model coupling, runtime data access, runtime program interaction, Fortran 90
If you cite this paper, please use the following reference:
Steven T. Hackstadt, Christopher W. Harrop, and Allen D. Malony, A Framework for Interacting with Distributed Programs and Data, Proceedings of the Seventh IEEE International Symposium on High Performance Distributed Computing (HPDC7), Chicago, IL, July 28-31, 1998, pp. 206-214. Also available as University of Oregon, Department of Computer and Information Science, Technical Report CIS-TR-98-02, June 1998. |
DAQV provides one solution to the general problem of providing high-level access to distributed arrays for the purpose of visualization and analysis. It does this by "exposing" the distributed data structures of a parallel (or distributed) program to external tools via interfaces that obviate the need to know about data decompositions, symbol tables, or the number of processes involved or where they are located. The goal is to provide access at a meaningful and portable level -- a level at which the user is able to interpret program data and at which external tools need only know simple, logical structures.
The seminal work on DAQV, by authors Hackstadt and Malony [8,9] and under the auspices of the Parallel Tools (Ptools) Consortium [23], resulted in a publicly available DAQV reference implementation. Ptools' user-oriented process was instrumental in identifying the needs to be addressed by DAQV and refining the scope of the original project. Though the Ptools portion of the project was completed in June 1997, we continue to follow the Ptools model by working closely with scientists to determine the functionality and operation of our tools.
Our current project, referred to as DAQV-II, is funded by Los Alamos National Laboratory (LANL) [20] and is being carried out in conjunction with the Computational Science Institute (CSI) at the University of Oregon [2]. The programming and execution requirements of this user community have driven the evolution and realization of a new operational model for DAQV. For example, a recent project in which our earlier work played a central role addressed the development of a domain-specific environment (DSE) for seismic tomography [3]. That environment is being reimplemented with DAQV-II as a result of the project's evolving needs.
In this context, DAQV-II continues to be the source of interesting research issues and challenging implementation problems. Many of these will be revealed in this paper by describing three aspects of DAQV-II: how it differs from the original reference implementation, how it operates, and how it is used to address the needs of scientists. These topics are covered in sections on design, implementation, and application, respectively. First, though, we briefly describe work related to DAQV-II.
Because the distribution of parallel data is an important factor in
the performance behavior of a program, viewing data and performance
information in relation to the distribution assists the user in tuning
endeavors. For example, the GDDT tool [19] provides a static depiction
of how parallel array data gets allocated on processors under
different distribution methods; it also supports an external interface
by which runtime information can be collected. In the DAVis tool [16],
distribution visualization is combined with dynamic data visualization
to understand the effects of decomposition on algorithm
operation. Similarly, Kimelman et al. show how a variety of runtime
information can be correlated to data distribution to better visualize
the execution of HPF programs [17]. And in the IVD tool [15] a data
distribution specification provided by the user is used to reconstruct
a distributed array that has been saved in partitioned form. DAQV-II
could easily provide distribution and runtime information to these
tools from a running application. In this way, DAQV-II is unique
because it combines distributed program interaction and data access at
a level low enough that computational steering, distribution
visualization, and other distributed data performance tools could be
made to use it.
Our original implementation of DAQV (referred to hereafter as DAQV-I)
targeted High Performance Fortran (HPF) [13]. We relied heavily on the
HPF language, compiler, and runtime support in implementing key
components (e.g., data access, interprocess communication) of the
DAQV-I system. This "language-level" implementation afforded us almost
guaranteed portability across different HPF computing environments and
minimized our worry about accessing and handling distributed data. On
the other hand, DAQV-I functionality was limited to applications
written in High Performance Fortran. In our work with CSI scientists,
as well as in our other collaborations, there was considerable
reluctance in embracing HPF because of its performance and restricted
(i.e., data-parallel) programming model. In contrast, there was
significant interest in gaining DAQV functionality for Fortran 77 and
Fortran 90 applications that use a message passing library such as
PVM [4] or MPI [22].
Thus, in DAQV-II, Fortran 90 and MPI became our primary implementation
targets. However, we also wanted to avoid being explicitly tied to a
given language. Thus, DAQV-II does not actually rely upon Fortran 90
(or MPI) for any key functionality; in contrast to the first version,
this makes it easy to retarget to other languages such as Fortran 77,
HPF, C, and C++. The primary disadvantage of this generalization is
that DAQV-II must now address more fully the general problem of
reassembling distributed arrays; we can no longer rely on a HPF-like
runtime environment for this support. This presented a tremendous
technical challenge that was cleverly avoided in the original work. In
DAQV-II, we currently support HPF-like BLOCK and CYCLIC data
distributions over multiple dimensions.
DAQV-I imposed a very strict client/server model of interaction on the
tools and applications involved. Tools (e.g., for visualization,
debugging, etc.) were clients of the HPF application, which
essentially became a distributed data server when linked with the DAQV
server library. The model also distinguished between data clients and
control clients; only a single control client could connect to a
DAQV-I server, while any number of data clients were allowed. This
model was simple for users to understand, relatively easy to
implement, and avoided synchronization issues among clients. However,
the model was too restrictive for addressing emerging requirements
like computational steering and simulation coupling [18], which
require bidirectional data flow and peer-based interaction.
In response, DAQV-II implements a more general model of interaction
and places less emphasis on client/server relationships. The
high-level view of DAQV-II is one of tools and applications that
interoperate with each other by exchanging data and sharing control of
each other's execution. Every component of the environment, whether it
is a tool or DAQV-enabled application, is viewed equally. Thus, there
is no required distinction between data and control clients; every
component is allowed to access the data of other components and to
register data (distributed or not) for access by other components. Of
course, this does not preclude a strict client/server relationship
like that in DAQV-I, if that is all that is required. However, DAQV-II
affords many additional system topologies, including peer
relationships in which multiple DAQV-enabled applications can read and
write each other's data. Such functionality is critical for model
coupling, the sharing of control and the exchange of data between
multiple simulations at runtime. Execution control is required so that
the coupled simulations can be synchronized when necessary; data
access is required so that, for example, the intermediate output of
one model can be used as the input to some other one.
The original motivation for DAQV-I stemmed from a desire for simple,
high-level access to distributed data. Users wanted to be able to "get
the data" using something as simple as a PRINT statement [8]. They
also wanted to use existing visualization tools to display the
data. This resulted in the development of a push model for data access
in which the user inserted calls to a DAQV_PUSH() procedure in their
code (much like PRINT statements). When the program was executed, data
would be delivered automatically to the appropriate display tool. This
very simple model allowed users to do one-time visual data dumps or to
create animations of data over time. However, to support a more
interactive and flexible approach to array visualization, we also
implemented a pull model which allowed rudimentary control over
program execution and selection of the arrays to be visualized. This
control and selection was allowed at explicit yield points compiled
into the code and was accessed at runtime through an external
interface. For our research in the CSI, however, we learned a valuable
lesson from DAQV-I: a push model for data access is unnecessary, and
even undesirable, in interactive domain-specific environments.
Push data transfers are fixed because they are compiled into the
application. Thus, a push statement in the body of a loop results in a
data transfer on every iteration. But for simulations with long,
computationally intense loops, a scientist's need to check data is the
exception, not the norm. Thus, despite the original Ptools feedback,
our scientists desired a low-overhead, interactive means of accessing
program data. Having two different modes of access is unnecessarily
complicated, especially when external clients can be written to mimic
push-like behavior.
As a result, we reformulated and generalized the abstraction for data
access in DAQV-II to include both the reading and writing of data
(e.g., for computational steering). We have termed the new
abstractions simply probe (for reading data) and mutate (for changing
data). We have preserved the notion of applications yielding to
DAQV-II to allow data to be repeatedly probed or mutated. However, it
is the responsibility of the user to make sure that probe and mutate
instrumentation is placed in semantically and scientifically
meaningful locations in the code. Finally, DAQV-II is able to restrict
access only to in-scope arrays, a feature not supported in DAQV-I.
In summary, the lessons learned from the DAQV-I reference
implementation, combined with a shifting focus in our own research
toward domain-specific environments, computational steering, and model
coupling, have resulted in fundamental changes in our design for
DAQV-II. To that end, our research into the design of a system for
distributed array interaction has focused on the important issues of
language targets, models of interaction, and data access
abstractions.
Figure 1: DAQV-II consists of a library for external client tools, a
library for application processes with distributed data, several
threads, and a master process.
2 Related work
DAQV is a confluence of several research ideas. Not surprisingly,
certain aspects of its design, functionality, and implementation
appear in other types of tools. For example, systems like Falcon [5],
CUMULVS [7], VASE [14,26], and SCIRun [24] typically provide complete,
closed environments for computational steering. At the core of these
systems is functionality similar to that provided by DAQV-II, but
DAQV-II is unique in that it implements "computational steering
middleware," which allows it to support steering functionality without
imposing constraints on the system and environment provided to the end
user. In this way, DAQV-II also supports a robust, extensible
framework for developing runtime interaction and computational
steering systems. This style of software development is consistent
with our in domain-specific environments [3] for computational
science; DAQV-II represents functionality that can be easily
integrated with and adapted to domain-specific software. As we
describe later, we have coupled DAQV-II with MATLAB to provide a
system for runtime simulation-tool interaction with seismic tomography
applications [12]. DAQV-II is also unique in that it allows
interactions with distributed data; of the steering systems listed
above, only CUMULVS directly supports distributed data. With respect
to performance and efficiency, domain-specific runtime visualization
systems like pV3 [10] are often able to make assumptions and
optimizations that obviate the need to directly address the more
general distributed data problem. 3 Design
In considering the general problem of interacting with distributed
arrays on parallel and distributed systems, our design of DAQV-II was
guided by three important considerations: language targets, models of
interaction, and abstractions for data access. Decisions made in these
areas play a significant role in determining the usability,
flexibility, and implementation difficulty of the resulting
system. 3.1 Language targets
The choice of language targets for a distributed array interaction
system has two important implications. First, the amount of compiler
and runtime support offered by a given language for handling
distributed data directly impacts the technical difficulty of
implementing distributed array access for that language. The second
issue is whether providing distributed array access capabilities for a
given language will actually be useful to scientists. 3.2 Models of interaction
Interacting with distributed arrays and parallel programs can follow
different models of interaction. In our work, we have considered both
client/server and peer-based models. We have found that client/server
interactions tend to be easier to understand and implement, while
peer-based models are more general and allow a wider range of
interaction. 3.3 Abstractions for data access
The development of effective abstractions for accessing distributed
data can have a direct impact on the usability and applicability of a
system. Our efforts toward the creation of abstractions for data
access have focused on capturing different "perspectives" of
distributed data access. In our experimentation, we have portrayed
DAQV functionality from two different perspectives: (1) who (or what)
decides to access data, and (2) how the data is to be accessed. 4 Implementation
The implementation of DAQV-II can be divided into two parts, the
client and application libraries. The client library is used by all
components (i.e., tools and applications) in a DAQV system. It
contains the interfaces and protocols for attaching to and detaching
from other tools and/or applications, learning about what data is
available, requesting and altering that data, and controlling program
execution. The second part, the application library, is used by a
parallel or distributed program that wishes to make distributed data
available via DAQV-II. The application library contains code that
supports communication among the application processes, answers data
requests, and reassembles distributed data; it is the core of DAQV-II
and will be the focus of our attention in this section. Figure 1
reveals the libraries, threads, and processes that make up the DAQV
architecture.
4.1 Application structure
Our primary application targets consist of single-program
multiple-data (SPMD) processes written in Fortran 77 or Fortran 90
that may communicate with each other through some message-passing
service such as PVM or MPI. We refer to the individual SPMD processes
as the application processes. Each application process contains local
data that is part of a logical, global array that has been distributed
among the application processes. However, the concept of data
distribution is likely to be foreign or irrelevant to both the
language (e.g., Fortran 90) and the communication system (e.g.,
MPI). The distribution may manifest itself in certain aspects of the
application's algorithms, but we have no means of determining that
from the source code. Hence, for our purposes, the "data distribution"
exists solely in the programmer's higher-level understanding of the
task being solved; therefore, the programmer is our only source for
obtaining that information. We are currently developing tools that
support collecting this information from the user.
Figure 2: A high-level view of DAQV-II events and protocols for attaching to and detaching from applications, accessing data, and controlling execution. |
For our example, let us assume the existence of an executing, DAQV-enabled, parallel program that has registered a distributed array, A, with DAQV-II. Let us also assume the existence of a DAQV-enabled client tool capable of sending and receiving the events shown in Figure 2. Further, we assume that this tool has already attached to the application's master process and has received the necessary registration information. The process we are about to describe begins when the user makes a request for some piece of the array A.
The tool uses a procedure provided by the DAQV client library to send a request for the desired array section to the master process. This procedure causes a Nexus remote service request (RSR) to be made to the master process, which responds by adding the request to a queue of pending events. Execution control is returned to the client tool as soon as the master process receives the request; the tool may now periodically query the client library to see if the request has been answered.
Meanwhile, on the application side, the parallel program eventually reaches a call to the procedural interface routine DAQV_PROBE(), which has the array symbol A as one of its arguments. Upon invocation, DAQV_PROBE() transfers execution control to the slave threads. The threads send the argument list of the DAQV_PROBE() call to the master process, which confirms whether this is a valid probe point.
If the call to DAQV_PROBE() is valid, the master searches its event queue for pending probe requests from external tools. Each item on the queue must be cross-referenced with the argument list from the call to DAQV_PROBE() so that only requests for arrays that are actually in scope are answered.
The master forwards each valid request in its event queue to the slave threads. Using distribution information provided by the user when the array was registered, the slave threads determine what part of the array, if any, resides locally.2 The threads each allocate a buffer and fill it with the appropriate values read from the address space of their associated application process. The slave threads respond to the master process by sending the filled buffers. The master process coordinates receipt of the data from the slaves, assembles the data into a single buffer, and sends it to the external tool that originally requested it.
Similar types of interactions between the master process and slave threads occur when application processes register distributed arrays and when data is mutated. As described above, data access (i.e., probe and mutate) is complicated by having to interpret array distributions and map global data requests from tools into localized requests for the application processes. These routines can be a bottleneck if not implemented with care; we continue to optimize and refine these parts of our implementation.
Figure 3: A generic prototype client for DAQV-II that can launch other
clients for visualizing, analyzing, or mutating program data.
(Note: clicking on each window will produce a full-size version.)
Now that DAQV-II supports Fortran 90 and improved support for client
interaction, we are reimplementing the environment in a new version
called TierraLab [11,12]. Built as an extension to MATLAB [21],
TierraLab offers powerful, interactive, runtime data analysis and
visualization capabilities not previously available to the
scientists. The environment is already beginning to be used by
geophysicists at the University of Oregon. A TierraLab session with
two seismic visualizations created in MATLAB is shown in Figure 4.
Figure 4: The TierraLab environment, developed for use by
seismologists, integrates DAQV-II functionality into MATLAB's powerful
data analysis and display capabilities.
The design and implementation of DAQV-II has been driven by the need
to interact with distributed data in parallel scientific
applications. While we have identified certain priority targets for
our work (i.e., SPMD Fortran 90 codes with implicitly distributed
data), the DAQV-II model meets a significantly broader range of
requirements. We feel that the functionality of DAQV-II is compelling
enough that a more general class of distributed computing problems can
also be addressed. In particular, there are problems whose solutions
traditionally have not been viewed as involving "distributed data." We
are currently exploring two examples of this possible role for
DAQV-II.
Consider a system like the Network Weather Service [27] which monitors
the performance and utilization of a collection of distributed,
heterogeneous machines. Each node maintains an array of metrics that
are periodically reported (or collected) by a client responsible for
displaying the results. This is fundamentally a distributed data
problem. As an experiment, we constructed a simple distributed load
monitoring tool written in C and using DAQV-II. As shown in Figure 5,
a DAQV-enabled load daemon is executed on each of eight machines in a
distributed environment and communicates with a DAQV master process
executed somewhere else in the system. Each daemon tracks the
1-minute, 5-minute, and 15-minute load averages in a 3-element
array. This array is globally viewed as a two dimensional array
distributed across eight machines. A client tool may make requests for
the load information across all or some of the machines through the
master process via a logical reference to all or part of the global
array. These requests, in turn, are converted into requests for the
associated data from the appropriate load daemons. In response to its
request, the client receives a single, global array of load data, and
data is only communicated when it is requested. In this case, we are
essentially imposing the notion of a BLOCK data distribution on the
distributed load data. The advantage of using DAQV-II is in the simple
interface it supports and the ease with which client tools can be
developed. (As a comment on DAQV-II's high-level utility, we built
this prototype in less than four hours.) In addition, the portability
of DAQV-II is limited only by the portability of the Nexus
communications library.
Figure 5: A distributed load monitor implemented in C and using
DAQV-II.
(Note: clicking on the DAQV Load Client window will produce a full-size version.)
We are also applying this technique to other performance metrics and
runtime application performance data. For example, we are currently
integrating DAQV-II with TAU, Tuning and Analysis Utilities for
C++ [1]. Our initial work [25] is focusing on providing client access
to TAU's performance callstack, an efficient performance view of a
program's execution profile at runtime. This profile shows where time
is being spent in a program's code for each thread of execution. The
client needs to display this data with respect to individual threads
as well as the entire application. The DAQV-II framework will provide
access to this data and support the interaction between the display
client and the executing program.
2. Although not discussed here, the determination of where distributed
data resides in non-trivial. DAQV-II allows access to any regular
sub-section of a multidimensional array distributed in a
BLOCK or CYCLIC manner in each dimension.
5 Applications
Our current prototype clients for DAQV-II are shown in Figure 3. These
clients demonstrate DAQV functionality and are intended to be used as
models for other development efforts. As we have mentioned earlier,
our interest in supporting the activities of computational scientists
is largely responsible for shaping the capabilities of DAQV-II. We
plan to use DAQV-II in a number of ways that pertain to these
activities. DAQV-I was the cornerstone of a complete program
interaction, steering, and visualization environment for a seismic
tomography application [3]. However, the performance penalty incurred
by using HPF (which DAQV-I required) was too severe, and, as a result,
the environment was not used by the scientists.
6 Conclusion
In summary, DAQV-II is an example of a distributed array interaction
framework. In our work, we have experimented with different language
targets, models of interaction, and data access abstractions. Our
current design and implementation reflects the lessons we have learned
through this process as well as our focus on addressing the evolving
requirements of high-performance computational science
applications. The development of DAQV-II is an ongoing research
challenge. Support for data mutation, streaming and parallel data
transfers, and optimized distribution mapping algorithms top our list
for future work. Through the construction of higher-level
domain-specific environments that are based on DAQV-II, we hope to
establish its generality and efficacy for providing a range of
interactive capabilities with parallel and distributed programs and
data. Additional information and documentation on DAQV-II can be found
on our web site: <http://www.cs.uoregon.edu/research/paraducks/>.
Acknowledgments
The authors would like to thank Chad Busche (Dept. of Computer and
Information Science, Univ. of Oregon) for his substantial work on the
implementation of DAQV-II and Laura Magde (Dept. of Geological
Sciences, Univ. of Oregon) for providing the visualizations in Figure
4. References
Footnotes
1. "DAQV" is pronounced "dave"; the "Q" is silent.
Last modified: Tue Dec 7 09:48:33 PST 1999
Steven Hackstadt /
hacks@cs.uoregon.edu