To evaluate the efficacy of overhead compensation, we must implement the methods described above in a real parallel profiling system and demonstrate their ability to improve application profiling results. To this end, we have implemented the overhead compensation techniques in the TAU parallel performance system [21]. TAU uses measured profiling to generate both flat profiles and callpath profiles. Inclusive and exclusive overhead compensation is implemented in TAU for both profiling modes.
How will we know if our overhead compensation works successfully? The standard measure for intrusion error is usually given as a percentage slowdown in total execution time. Thus, one can test the ability of a compensation-enabled profiling tool to accurately recover total execution time from profile measurements with varying levels of instrumentation. As the instrumentation increases, so likely will the intrusion and the overhead compensation techniques will be more stressed. However, it is important to understand that accurate performance profiling will also depend on the precision of measurement, in particular, the ability to observe small performance phenomena. Overhead compensation can improve measurement accuracy, but it cannot remove measurement uncertainty for small events.
For any level of instrumentation, it is reasonable to expect that less instrumentation leads to more accurate profiling results than more instrumentation. Thus, if the total execution time is accurate, we might assume the rest of the profile statistics are also. However, there are two issues to keep in mind. First, performance variability due to environmental factors can arise even in un-instrumented applications. Second, the success of overhead compensation on profile statistics is difficult to assess given that the ``real'' profile values are not known. The best we can do then is to compare profiling results at one level of instrumentation with results from using less instrumentation, under the assumption that the execution is relatively stable and the results from less instrumented runs are more reliable and accurate.