An Overview of ADEPT:
A Declarative, Eval-based Program Tester
Phil Pfeiffer
July, 2003
Abstract: ADEPT is an open-source unit testing tool for the Python programming language. ADEPT's declarative approach to test definition allows test cases to be coded based on method type (i.e., error, null, get, set, get+set, get property error, set property error, get property, set property), and then interpreted, using ADEPT's eval()-based test driver. Supporting classes automatically index and label tests, and log results for later recall. The ADEPT distribution includes a user manual; a series of illustrative examples, with supporting code; and a 782-test-case unit test suite for ADEPT, written in ADEPT.
ADEPT is an open-source unit testing tool for the Python programming language. ADEPT addresses the same basic concerns as unittest, a de facto standard for unit testing (cf. [VonRossum02b], §5.3). ADEPT and unittest differ, however, in their formalisms for test case definition. In unittest, test cases are class methods in a class that inherits from a distinguished class. In ADEPT, test cases are tuples that describe a test's intent, define its core action, and characterize that action’s expected consequences.
A short example should serve to illustrate ADEPT’s perceived strength: the succinctness of the resulting test cases. Consider the following unittest code for testing a Widget class’s constructor and resize() methods:
# ---- unittest example, from Python 2.2 Library Reference ----
# ---- [section 5.3.1, "Organizing test code"] ----
class WidgetTestCase(unittest.TestCase):
def setUp(self):
self.widget = Widget("The widget")
def tearDown(self):
self.widget.dispose()
self.widget = None
def testDefaultSize(self):
self.failUnless(self.widget.size() == (50,50), 'incorrect default size')
def testResize(self):
self.widget.resize(100,150)
self.failUnless(self.widget.size() == (100,150), 'wrong size after resize')
def suite():
suite = unittest.TestSuite()
suite.addTest(WidgetTestCase("testDefaultSize"))
suite.addTest(WidgetTestCase("testResize"))
return suite
The same code, minus the finalizers, can be expressed in four lines of ADEPT:
getTest = ("check default size", eqValue((50,50)), Widget("The Widget").size)
testWidget = Widget("The widget")
setTest =\ ("check resize", eqValue((100,150)), "testWidget.size()",
testWidget.resize, (100,150))
testSuite = ( (“get actions”, (getTest,)), (“set actions”, (setTest,)) )
The getTest tuple typifies what ADEPT calls a get actions test: a check for an action's return value. This tuple directs ADEPT to record success iff eqValue((50,50))(Widget("The widget").size()) evaluates to 1 (true). To intepret this example, it helps to know that
· eqValue is a functor: a class that defines a __call__() method, and whose instances act like functions. Here, the expression eqValue((50,50))(x) returns 1 iff x evaluates to (50,50).
· The null argument list for Widget.size() is supplied by default.
The setTest tuple typifies what ADEPT calls a set actions test: a check for an action's effects. This tuple directs ADEPT to execute testWidget.resize(100,150), then record success iff the expression eqValue((100,150))(testWidget.size()) evaluates to 1.
The getTest and setTest tuples can be evaluated by tagging them with “get actions” and “set actions” test type strings, embedding them in testSuite, and passing them, along with an ADEPT test logging object, to ADEPT's doTestSuite() test engine.
>>> widgetLogger = TestLogger() #
labels tests and stores labels and results
>>> doTestSuite(suite, widgetLogger) # assumption: all tests succeed
(no output)
>>>
When doTestSuite() completes, widgetLogger can be queried for the names and statuses of the just-completed tests:
>>>
widgetLogger.failureCount()
0
>>> widgetLogger.successCount()
2
>>> widgetLogger.failures()
()
>>> widgetLogger.successes()
((2,
1), (1, 1))
>>> map(widgetLogger.getBanner, widgetLogger.successes)
('1.1. get actions: check default size',
'2.1. set actions: check resize')
ADEPT’s default indexing scheme assigns major numbers to test sets—labeled sets of test cases like (“get actions”, (getTest,))—and minor test numbers to individual cases. Index assignments, like doTestSuite() status messages, can be varied using a combination of class properties and keywords.
The following is a brief overview of ADEPT’s key features (§2.1) and known limitations (§2.2).
ADEPT supports nine built-in test types:
· get actions tests: tests of values returned by callable objects—e.g., free functions, methods, functors;
· set actions tests: tests of effects produced by callable objects;
· get+set actions tests: tests of values returned and effects produced by callable objects;
· error actions tests: tests of exceptions generated by applications of callable objects;
· null actions tests: tests of callable actions that should return None;
· get prop actions tests: tests of read operations on properties, a type of Python virtual class attribute;
· set prop actions tests: tests of write operations on properties;
· get prop error actions tests: tests of property read operations that should throw exceptions;
· set prop error actions tests: tests of property write operations that should throw exceptions.
ADEPT also features extended support for __repr__() (representation) method testing. In Python, class methods named __repr__(), by convention, should return
… the "official" string representation of an object. If at
all possible, this should look like a valid Python expression that could be used to recreate an
object with the same value (given an appropriate environment). (§3.3.1,
[vonRossum02a])
An ADEPT functor, evalsTo(), verifies that an object obj’s repr string evaluates to obj’s original value. evalsTo() actually tests if `obj` == eval(attemptToMakeParseable(`obj`))—a slightly more forgiving form of the more straightforward `obj` == eval(`obj`) test that compensates for deficiencies in built-in reprs for functions and types. These objects’ reprs, which are generated as syntactically invalid strings like <function f at 0xnnnnn> and <type ‘int’>, are recast by attemptToMakeParseable() as f and int, respectively, before completing the check.
A second test driver, doTestStack(), supports the execution of test stacks: nested, labeled lists of test suites and stacks like
threeTierTestSuite =\
( “Program test suite”,
( "tests of
function foo",
(
("error actions", (errorActionTest1, errorActionTest2)),
("null
actions", (nullActionTest1, nullActionTest2)),
),
),
( "tests
of class Bar",
(
("error actions", (errorActionTest3, errorActionTest4)),
("get actions", (getActionTest1,
getActionTest2, getActionTest3)),
("set actions", (setActionTest1,
setActionTest2)),
),
)
)
Executing
doTestStack(threeTierTestSuite, logger) initiates an
eleven-case-long test run in which test cases errorActionTest1 through setActionTest2 are assigned indices of 1.1.1,
1.1.2, 1.2.1, 1.2.2, 1.3.1, 1.3.2, 1.4.1, 1.4.2, 1.4.3, 1.5.1, and 1.5.2,
respectively.
ADEPT supports the use of nested lists of predicates to check outcomes. These nested lists are interpreted as arbitrarily nested conjuncts of disjuncts, in a way similar to the interpretation of AND/OR game decision trees. The following four test tuples illustrate the use of multiple levels of nesting to impose increasingly fussier constraints on the content of a class’s __str__() method:
siblingsClassTest1
=\
(
"Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",
isInstanceOf(str), # check if output is of type string
Siblings('Ron','Percy',mom='Molly').__str__,
)
siblingsClassTest2
=\
(
"Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",
( isInstanceOf(str), # check if output is of type string
containsRE(‘Ron’), # and the result string contains ‘Ron’
containsRE(‘Percy’), # and the result string contains ‘Percy’
containsRE(‘Molly’), # and the result string contains ‘Molly’
)
Siblings('Ron','Percy',mom='Molly').__str__,
)
siblingsClassTest3
=\
(
"Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",
( isInstanceOf(str), # check if output is of type string, and
( containsRE(‘Ron’), # the result string either contains ‘Ron’ ...
containsRE(‘Ronald’), # or ‘Ronald’
)
containsRE(‘Percy’), # and the result string contains ‘Percy’
containsRE(‘Molly’), # and the result string contains ‘Molly’
)
Siblings('Ron','Percy',mom='Molly').__str__,
)
siblingsClassTest4
=\
(
"Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",
( isInstanceOf(str), # check if output is of type string, and either
(\
(
containsRE(‘Ron’), # the result string contains ‘Ron’ ...
lacksRE(‘Ronald’), # but not ‘Ronald’
), # or
( containsRE(‘Ronald’), # the result string contains ‘Ronald’ ...
lacksRE(‘Ron’), # but not ‘Ron’
)
),
containsRE(‘Percy’), # and the result string contains ‘Percy’
containsRE(‘Molly’), # and the result string contains ‘Molly’
)
Siblings('Ron','Percy',mom='Molly').__str__,
)
Using a combination of doTestSuite()/doTestStack() keywords, keywords, TestLogger keywords, and TestLogger properties, an ADEPT client code can control
· The number of fields in test index tuples, and the indices’ values;
· The types of output messages displayed by doTestSuite() during test evaluation (test indices, test banners, malformed test error messages, test failed messages, and test succeeded messages); and
· The destination file, or pseudo-file, for test suite messages.
TestLogger.getBanner() supports an optional, mask argument for pruning fields from output banners. For example, if logger.getBanner((2,1)) == 'set actions: check resize', then
logger.getBanner((1,1),(1,1)) == 'set actions: check resize'
logger.getBanner((1,1),(1,0)) == 'set actions: '
logger.getBanner((1,1),(0,1)) == 'check resize'
logger.getBanner((1,1),(0,0))
== ' '
ADEPT’s set of built-in tests can be extended by adding an entry to an internal dictionary of test types, and associating an element of this dictionary with a function that implements the specified type of test. Similarly, a built-in test can be modified by changing the function that implements that test, and possibly modifying the appropriate entry in the test types dictionary. Such modifications can be used to address some of the limitations discussed below.
ADEPT, like Python’s eval() primitive, does not directly support tests on assignment statements, like a = (1,2,3). However, updates to components of mutable objects like a[1:2] = [1,2,3] can be tested by testing the underlying class method that does the update—in this example, a.__setslice__().
Before a statement like print x can be tested, it must first be rewritten as print >>outputStream, x. The name outputStream must then be bound to a Python class with a write() method that stores output in a way that the test code can later recover.
ADEPT does not currently support the use of predicates to test an exception’s arglist component.
A Python property is a kind of virtual class attribute that represents a shorthand for up to four functions: a get function that is evaluated when the property is read; a set function that is evaluated when the property is written; a del function that is evaluated when the property is deleted; and a docstring function. Currently, ADEPT does not support the testing of property delete and docstring methods.
ADEPT does not currently support the explicit inclusion of initializer and finalizer actions in test tuples.
ADEPT was not
developed as a tool for automating test case generation. While the author regards ADEPT as a
potentially useful back-end for automated test case generation, the sort of
semantic analysis needed to generate semantically interesting test cases is
well beyond the scope of this work.
The ADEPT user manual [Pfeiffer03] discusses ADEPT’s features and limitations in far more detail. Sections 2 through 6 of the manual cover the nine built-in test types; the doTestSuite() test engine; advanced features (multiple predicates, doTestStack(), mask tuples); the ADEPT distribution; and ADEPT limitations, respectively. Where appropriate, the manual discusses tradeoffs among competing strategies for defining test cases. The discussion of ADEPT’s limitations includes suggestions for incorporating missing features into the utility, if desired: most were omitted because the author did not need them to test his codes.
The manual illustrates ADEPT’s operation using a series of related examples involving a class that maintains a set of siblings (e.g., ‘Bill’, ‘Charlie’, ‘Percy’, ‘Fred’, ‘George’, ‘Ron’, and ‘Ginny’) and a mom and dad property (e.g., ‘Molly’ and ‘Arthur’). These examples are included in a demonstrator program, adeptExamples.py, that accompanies this suite.
The ADEPT distribution a final tool: a 782-test-case-long test suite for ADEPT itself, which proved instrumental in ADEPT’s development. This suite can be used as another model for coding in ADEPT.
References
[Pfeiffer03] Pfeiffer,
Phil, ADEPT: A Declarative, Eval-based Program Tester: User’s Manual,
July 2003 [available at http://csciwww.etsu.edu/phil/samples/adept.txt]
[vonRossum02a] von Rossum, Guido and Drake, Fred, Python Language Reference Manual, Release 2.2.2, 14 October 2002 [available from www.python.doc]
[vonRossum02b] von Rossum, Guido and Drake, Fred, Python Library Reference, Release 2.2.2, 14 October 2002 [available from www.python.doc]