In current practice, there is a sharp divide between unit, integration, and system testing on the one hand, and feedback from deployed software on the other. While developers have access to a variety of software monitoring tools in the development environment, monitoring in the deployed environment is typically limited to error and sanity checks, and the channel from users back to developers is a puny dribble of trouble reports.
This asymmetry was reasonable at one time, but with ubiquitous networking it is no longer necessary. The internet provides a new opportunity for extending monitoring capabilities from the development environment into the fielded environment.
Initially we are focusing on beta testing, and term the effort "beta
carotene." Beta carotene is about richer monitoring and feedback of
deployed software, validating and refining the models
and assumptions used in quality assurance activities based on actual
usage.
Beta carotene will provide monitoring of deployed software with adjustable
level and focus to address varying performance requirements, with user control
to address concerns of security and confidentiality.
An important first step toward the goals of beta carotene is very low-cost
monitoring of the "residue" of coverage testing, i.e., checking for
test obligations which were not fulfilled in the development environment but
correspond to execution behaviors that occur in actual use. Our work to date
has focused on monitoring of dynamic test thoroughness, as indicated by structural
test coverage criteria, in deployed software. Residual coverage monitoring should
indicate whether any test obligations which were not fulfilled in the development
environment correspond to execution behaviors that occur in actual use. This
could occur, for example, if an execution path that has been presumed infeasible
by developers is in fact feasible. [New
in 2001 - Gretel tool for residual test coverage monitoring]
The primary question addressed by the current work is whether such monitoring
can have sufficiently low impact on execution characteristics, particularly
efficiency and responsiveness, to be acceptable to users. A proof-of-concept
prototytpe system has been constructed for residual testing of Java programs,
and the results are very encouraging: After the first few executions cover
the high-frequency program paths, residual monitoring of the as-yet-unexecuted
portions has almost no impact on execution performance.
[More]
Contact: Michal Young