The combination of scalable performance observation and online operation sets a high standard for effective use of present day performance tools. Many performance systems are not built for scale and work primarily offline. Our experience is one of extending the existing TAU performance system to address problems of scale through improved measurement selectivity, new statistical clustering functions, parallel analysis, and three-dimensional visualization. In addition, online support in TAU is now possible for both profiling and tracing using a Push model of data access. We have demonstrated these capabilities for applications over 500 processes. However, it is by no means correct to consider our TAU experience as evidence for a general purpose solution. As with TAU, it is reasonable to expect that other traditional offline tools could be brought online under the right system conditions. In the wrong circumstances, the approaches may be ineffective either because they process larger volumes of data or require more analysis power. Any solution to scalable, online performance observation will necessarily be application and system dependent, and will require an integrated analysis of engineering tradeoffs that include concerns for intrusion and quality of performance data. Our goal is to continue to advance the TAU performance system for scalability and online support to better understand where and how these tradeoffs arise and apply.