Technology for empirical performance evaluation of parallel programs is driven by the increasing complexity of high perfonnance computing environments and programming methodologies. This complexity-arising from the use of high-level parallel languages, domain-specific numerical frameworks, heterogeneous execution models and platforms, multi-level software optimization strategies, and multiple compilation models widens the semantic gap between a programmer's understanding of his/her code and it's runtime behavior. To keep pace, perfonnance tools must provide for the effective instrumentation of complex software and the correlation of runtime performance data with user-level semantics.

To address these issues, this dissertation contributes:

• a strategy for utilizing multi-level instrumentation to improve the coverage of perfor­mance measurement in complex, layered software;
• techniques for mapping low-level performance data to higher levels of abstraction in order to reduce the semantic gap between user's abstractions and runtime time behav­ior; and
• the concept of instrumentation-aware compilation that extends traditional compilers to preserve the semantics of fine-grained performance instrumentation despite aggressive program restructuring.In each case, the dissertation provides prototype implementations and case studies of the needed tools and frameworks.

This dissertation research aims to influence the way performance observation tools and compilers for high performance computers are designed and implemented.