Test Oracles

Luciano Baresi and Michal Young
Technical Report CIS-TR-01-02
August 2001

Abstract

All software testing methods depend on the availability of an oracle, that is, some method for checking whether the system under test has behaved correctly on a particular execution. This report surveys proposed approaches to the oracle problem that are general in the sense that they require neither pre-computed input/output pairs nor a previous version of the system under test. Oracles that are general in this sense can be applied to a wide range of software applications.

Oracles are either based on a program specification, or are a (perhaps very incomplete) specification of intended behavior. Some oracles, particularly those expressed in assertion languages embedded in program text, describe acceptable behaviors entirely at an implementation level. Others are derived from or associated with external, usually more abstract specifications or models. In either case, a key problem for test oracles is bridging the gap between abstract specifications and efficiently checkable properties of concrete execution sequences. Common problems that have been addressed in different ways include evaluation of predicates involving quantification over large or infinite sets and predicates relating values at different points in execution history. The survey is not encyclopedic, but discusses representative examples of the main approaches and tactics for solving common problems.

Download: as pdf (304k, 55pages)

Note: This survey paper has been submitted to ACM Computing Surveys. We will appreciate your corrections and suggestions for improving the paper. There will presumably be a revised version of the paper regardless of whether it is accepted or rejected by the journal. Please contact us by email: michal@cs.uoregon.edu, baresi@elet.polimi.it.

If you wish to cite the paper as a UO CIS technical report and are a LaTeX user, , the following bibtex entry might be useful (with \usepackage{url} in the LaTeX file).

@TechReport{Baresi:Oracles,
    author = {Baresi, Luciano and Young, Michal},
    title = {Test Oracles},
    institution = {University of Oregon,
    Dept. of Computer and Information Science},
    year = 2001,
    type = {Technical Report},
     number = {CIS-TR-01-02},
    address = {Eugene, Oregon, U.S.A.},
    month = {August},
    note = {\url{http://www.cs.uoregon.edu/~michal/pubs/oracles.html}}
}