An Overview of ADEPT: A Declarative, Eval-based Program Tester

Phil Pfeiffer

July, 2003

Abstract:  ADEPT is an open-source unit testing tool for the Python programming language.  ADEPT's declarative approach to test definition allows test cases to be coded based on method type (i.e., error, null, get, set, get+set, get property error, set property error, get property, set property), and then interpreted, using ADEPT's eval()-based test driver.  Supporting classes automatically index and label tests, and log results for later recall.  The ADEPT distribution includes a user manual; a series of illustrative examples, with  supporting code; and a 782-test-case unit test suite for ADEPT, written in ADEPT.

1.         Introduction:  About ADEPT

ADEPT is an open-source unit testing tool for the Python programming language.  ADEPT addresses the same basic concerns as unittest, a de facto standard for unit testing (cf. [VonRossum02b], §5.3).  ADEPT and unittest differ, however, in their formalisms for test case definition.  In unittest, test cases are class methods in a class that inherits from a distinguished class.  In ADEPT, test cases are tuples that describe a test's intent, define its core action, and characterize that action’s expected consequences.

A short example should serve to illustrate ADEPT’s perceived strength: the succinctness of the resulting test cases.  Consider the following unittest code for testing a Widget class’s constructor and resize() methods:

    # ---- unittest example, from Python 2.2 Library Reference ----

    # ---- [section 5.3.1, "Organizing test code"] ----


    class WidgetTestCase(unittest.TestCase):

        def setUp(self):

           self.widget = Widget("The widget")


        def tearDown(self):


           self.widget = None


        def testDefaultSize(self):

           self.failUnless(self.widget.size() == (50,50), 'incorrect default size')


        def testResize(self):


           self.failUnless(self.widget.size() == (100,150), 'wrong size after resize')


    def suite():

        suite = unittest.TestSuite()



        return suite

The same code, minus the finalizers, can be expressed in four lines of ADEPT:

    getTest =  ("check default size", eqValue((50,50)), Widget("The Widget").size)

    testWidget = Widget("The widget")

    setTest =\ ("check resize", eqValue((100,150)), "testWidget.size()",

                 testWidget.resize, (100,150))

    testSuite = ( (“get actions”, (getTest,)), (“set actions”, (setTest,)) )

The getTest tuple typifies what ADEPT calls a get actions test: a check for an action's return value.  This tuple directs ADEPT to record success iff eqValue((50,50))(Widget("The widget").size()) evaluates to 1 (true).  To intepret this example, it helps to know that

·         eqValue is a functor: a class that defines a __call__() method, and whose instances act like functions. Here, the expression eqValue((50,50))(x) returns 1 iff x evaluates to (50,50). 

·         The null argument list for Widget.size() is supplied by default. 

The setTest tuple typifies what ADEPT calls a set actions test: a check for an action's effects.  This tuple directs ADEPT to execute testWidget.resize(100,150), then record success iff the expression     eqValue((100,150))(testWidget.size()) evaluates to 1.

The getTest and setTest tuples can be evaluated by tagging them with “get actions” and “set actions” test type strings, embedding them in testSuite, and passing them, along with an ADEPT test logging object, to ADEPT's doTestSuite() test engine.

    >>> widgetLogger = TestLogger()       # labels tests and stores labels and results

    >>> doTestSuite(suite, widgetLogger)  # assumption: all tests succeed (no output)


When doTestSuite() completes, widgetLogger  can be queried for the names and statuses of the just-completed tests:

    >>> widgetLogger.failureCount()


    >>> widgetLogger.successCount()


    >>> widgetLogger.failures()


    >>> widgetLogger.successes()

    ((2, 1), (1, 1))

    >>> map(widgetLogger.getBanner, widgetLogger.successes)

    ('1.1. get actions: check default size',

     '2.1. set actions: check resize')

ADEPT’s default indexing scheme assigns major numbers to test sets—labeled sets of test cases like (“get actions”, (getTest,))—and minor test numbers to individual cases.  Index assignments, like doTestSuite() status messages, can be varied using a combination of class properties and keywords.

2.         Overview of ADEPT Characteristics

The following is a brief overview of ADEPT’s key features (§2.1) and known limitations (§2.2).

2.1       ADEPT Features

2.1.1    Method and Property Testing

ADEPT supports nine built-in test types:

·         get actions tests:   tests of values returned by callable objects—e.g., free functions, methods, functors;

·         set actions tests:   tests of effects produced by callable objects;

·         get+set actions tests:  tests of values returned and effects produced by callable objects;

·         error actions tests:   tests of exceptions generated by applications of callable objects;

·         null actions tests:  tests of callable actions that should return None;

·         get prop actions tests:  tests of read operations on properties, a type of Python virtual class attribute;

·         set prop actions tests:  tests of write operations on properties;

·         get prop error actions tests:  tests of property read operations that should throw exceptions;

·         set prop error actions tests:  tests of property write operations that should throw exceptions.

ADEPT also features extended support for __repr__() (representation) method testing. In Python, class methods named __repr__(), by convention, should return

… the "official" string representation of an object. If at all possible, this should look like a valid Python  expression that could be used to recreate an object with the same value (given an appropriate environment).  (§3.3.1, [vonRossum02a])

An ADEPT functor, evalsTo(), verifies that an object obj’s repr string evaluates to obj’s original value.  evalsTo() actually tests if  `obj` == eval(attemptToMakeParseable(`obj`))—a slightly more forgiving form of the more straightforward `obj` == eval(`obj`) test that compensates for deficiencies in built-in  reprs for functions and types.  These objects’ reprs, which are generated as syntactically invalid strings like <function f at 0xnnnnn> and <type ‘int’>, are recast by attemptToMakeParseable() as f and int, respectively, before completing the check.

2.1.2    Test Stack Execution

A second test driver, doTestStack(), supports the execution of test stacks: nested, labeled lists of test suites and stacks like

     threeTierTestSuite =\

         ( “Program test suite”,

       ( "tests of function foo",

         ( ("error actions", (errorActionTest1, errorActionTest2)),

           ("null actions", (nullActionTest1, nullActionTest2)),



       ( "tests of class Bar",

          ( ("error actions", (errorActionTest3, errorActionTest4)),

            ("get actions", (getActionTest1, getActionTest2, getActionTest3)),

            ("set actions", (setActionTest1, setActionTest2)),




Executing doTestStack(threeTierTestSuite, logger) initiates an eleven-case-long test run in which test cases errorActionTest1 through setActionTest2 are assigned indices of 1.1.1, 1.1.2, 1.2.1, 1.2.2, 1.3.1, 1.3.2, 1.4.1, 1.4.2, 1.4.3, 1.5.1, and 1.5.2, respectively.

2.1.3    Use of Nested Predicates for Validating Actions

ADEPT supports the use of nested lists of predicates to check outcomes.  These nested lists are interpreted as arbitrarily nested conjuncts of disjuncts, in a way similar to the interpretation of AND/OR game decision trees.  The following four test tuples illustrate the use of multiple levels of nesting to impose increasingly fussier constraints on the content of a class’s __str__() method:

   siblingsClassTest1 =\

      ( "Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",

         isInstanceOf(str),           # check if output is of type string




   siblingsClassTest2 =\

      ( "Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",

         ( isInstanceOf(str),         # check if output is of type string

           containsRE(‘Ron’),         # and the result string contains ‘Ron’

           containsRE(‘Percy’),       # and the result string contains ‘Percy’

           containsRE(‘Molly’),       # and the result string contains ‘Molly’





   siblingsClassTest3 =\

      ( "Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",

         ( isInstanceOf(str),        # check if output is of type string, and

           ( containsRE(‘Ron’),       # the result string either contains ‘Ron’ ...

             containsRE(‘Ronald’),    # or ‘Ronald’


           containsRE(‘Percy’),       # and the result string contains ‘Percy’

           containsRE(‘Molly’),       # and the result string contains ‘Molly’





   siblingsClassTest4 =\

      ( "Siblings('Ron','Percy',mom='Molly').__str__() name Ron, Percy, and Molly",

         ( isInstanceOf(str),         # check if output is of type string, and either


             ( containsRE(‘Ron’),     # the result string contains ‘Ron’ ...

               lacksRE(‘Ronald’),     # but not ‘Ronald’

             ),                       # or

             ( containsRE(‘Ronald’),  # the result string contains ‘Ronald’ ...

               lacksRE(‘Ron’),        # but not ‘Ron’



           containsRE(‘Percy’),       # and the result string contains ‘Percy’

           containsRE(‘Molly’),       # and the result string contains ‘Molly’




2.1.4    Indexing, Status Messages, and Output Redirection

Using a combination of doTestSuite()/doTestStack() keywords,  keywords, TestLogger keywords, and TestLogger properties, an ADEPT client code can control

·         The number of fields in test index tuples, and the indices’ values;

·         The types of output messages displayed by doTestSuite() during test evaluation  (test indices, test banners, malformed test error messages, test failed messages, and test succeeded messages); and

·         The destination file, or pseudo-file, for test suite messages.

2.1.5    Output Masking

TestLogger.getBanner() supports an optional, mask argument for pruning fields from output banners.  For example, if logger.getBanner((2,1)) == 'set actions: check resize', then

logger.getBanner((1,1),(1,1)) == 'set actions: check resize'

logger.getBanner((1,1),(1,0)) == 'set actions: '

logger.getBanner((1,1),(0,1)) == 'check resize'

logger.getBanner((1,1),(0,0)) == ' '

2.1.6    Extensibility

ADEPT’s set of built-in tests can be extended by adding an entry to an internal dictionary of test types, and associating an element of this dictionary with a function that implements the specified type of test.  Similarly,  a built-in test can be modified by changing the function that implements that test, and possibly modifying the appropriate entry in the test types dictionary.  Such modifications can be used to address some of the limitations discussed below.

2.2       ADEPT Limitations

2.2.1    Assignment Statement Testing

ADEPT, like Python’s eval() primitive, does not directly support tests on assignment statements, like a = (1,2,3).  However, updates to components of mutable objects like a[1:2] = [1,2,3] can be tested by testing the underlying class method that does the update—in this example, a.__setslice__().

2.2.2    Print Statement Testing (Requires Pseudo-Files)

Before a statement like print x can be tested, it must first be rewritten as print >>outputStream, x.  The name outputStream must then be bound to a Python class with a write() method that stores output in a way that the test code can later recover.

2.2.3    Exception Arglist Testing

ADEPT does not currently support the use of predicates to test an exception’s arglist component.

2.2.4    Property Delete and Docstring Testing

A Python property is a kind of virtual class attribute that represents a shorthand for up to four functions: a get function that is evaluated when the property is read; a set function that is evaluated when the property is written; a del function that is evaluated when the property is deleted; and a docstring function.  Currently, ADEPT does not support the testing of property delete and docstring methods.

2.2.5    Initializers and Finalizers

ADEPT does not currently support the explicit inclusion of initializer and finalizer actions in test tuples.

2.2.5    Test Case Generation

ADEPT was not developed as a tool for automating test case generation.  While the author regards ADEPT as a potentially useful back-end for automated test case generation, the sort of semantic analysis needed to generate semantically interesting test cases is well beyond the scope of this work.

3          For Further Information

The ADEPT user manual [Pfeiffer03] discusses ADEPT’s features and limitations in far more detail.  Sections 2 through 6 of the manual cover the nine built-in test types; the doTestSuite() test engine; advanced features (multiple predicates, doTestStack(), mask tuples); the ADEPT distribution; and ADEPT limitations, respectively. Where appropriate, the manual discusses tradeoffs among competing strategies for defining test cases.  The discussion of ADEPT’s limitations includes suggestions for incorporating missing features into the utility, if desired: most were omitted because the author did not need them to test his codes.

The manual illustrates ADEPT’s operation using a series of related examples involving a class that maintains  a set of siblings (e.g., ‘Bill’, ‘Charlie’, ‘Percy’, ‘Fred’, ‘George’, ‘Ron’, and ‘Ginny’) and a mom and dad property (e.g., ‘Molly’ and ‘Arthur’).  These examples are included in a demonstrator program,, that accompanies this suite.

The ADEPT distribution a final tool: a 782-test-case-long test suite for ADEPT itself, which proved instrumental in ADEPT’s development.  This suite can be used as another model for coding in ADEPT.


[Pfeiffer03]                Pfeiffer, Phil, ADEPT: A Declarative, Eval-based Program Tester: User’s Manual, July 2003  [available at]

[vonRossum02a]        von Rossum, Guido and Drake, Fred, Python Language Reference Manual, Release 2.2.2, 14 October 2002  [available from www.python.doc]

[vonRossum02b]        von Rossum, Guido and Drake, Fred, Python Library Reference, Release 2.2.2, 14 October 2002  [available from www.python.doc]