In current practice, there is a sharp divide between unit, integration, and system testing on the one hand, and feedback from deployed software on the other. While developers have access to a variety of software monitoring tools in the development environment, monitoring in the deployed environment is typically limited to error and sanity checks, and the channel from users back to developers is a puny dribble of trouble reports.

This asymmetry was reasonable at one time, but with ubiquitous networking it is no longer necessary. The internet provides a new opportunity for extending monitoring capabilities from the development environment into the fielded environment.

Initially we are focusing on beta testing, and term the effort "beta carotene." Beta carotene is about richer monitoring and feedback of deployed software, validating and refining the models and assumptions used in quality assurance activities based on actual usage.

Beta carotene will provide monitoring of deployed software with adjustable level and focus to address varying performance requirements, with user control to address concerns of security and confidentiality.

Residual Test Coverage Monitoring

An important first step toward the goals of beta carotene is very low-cost monitoring of the "residue" of coverage testing, i.e., checking for test obligations which were not fulfilled in the development environment but correspond to execution behaviors that occur in actual use. Our work to date has focused on monitoring of dynamic test thoroughness, as indicated by structural test coverage criteria, in deployed software. Residual coverage monitoring should indicate whether any test obligations which were not fulfilled in the development environment correspond to execution behaviors that occur in actual use. This could occur, for example, if an execution path that has been presumed infeasible by developers is in fact feasible. [New in 2001 - Gretel tool for residual test coverage monitoring]

The primary question addressed by the current work is whether such monitoring can have sufficiently low impact on execution characteristics, particularly efficiency and responsiveness, to be acceptable to users. A proof-of-concept prototytpe system has been constructed for residual testing of Java programs, and the results are very encouraging: After the first few executions cover the high-frequency program paths, residual monitoring of the as-yet-unexecuted portions has almost no impact on execution performance. [More]

Contact: Michal Young