====================================================================== ADEPT: A Declarative, Eval-based Program Tester Author: Phil Pfeiffer Date: July, 2003 Version: 0.9 ====================================================================== --------------------------------------------------------------------- ADEPT testing tool for Python 2.2 Oak Ridge National Laboratory, Oak Ridge, TN, East Tennessee State University, Johnson City, TN Author: Phil Pfeiffer (C) 2003 All Rights Reserved NOTICE Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted provided that the above copyright notice appear in all copies and that both the copyright notice and this permission notice appear in supporting documentation. Neither the Oak Ridge National Laboratory, nor East Tennessee State University, nor the Authors make any representations about the suitability of this software for any purpose. This software is provided "as is" without express or implied warranty. The development of the ADEPT testing tool was funded by the U.S. Department of Energy and the East Tennessee State University. ---------------------------------------------------------------------- Abstract: ADEPT is an open-source unit testing tool for the Python programming language. ADEPT's declarative approach to test definition allows test cases to be coded based on method type (i.e., error, null, get, set, get+set, get property error, set property error, get property, set property), and then interpreted, using ADEPT's eval()-based test driver. Supporting classes automatically index and label tests, and log results for later recall. The ADEPT distribution includes a user manual; a series of illustrative examples, with supporting code; and a 782-test-case unit test suite for ADEPT, written in ADEPT. 1. Introduction 1.1 ADEPT vs. unittest--a Short Comparison ADEPT is an application for unit-testing Python programs. ADEPT addresses the same basic concerns as unittest, a de facto standard for unit testing [cf. Python Library manual, Section 5.3], but differs from unittest in its declarative approach to supporting test creation. unittest supports a statement-list-oriented approach to test creation. unittest users define tests by subclassing special test classes, and overloading the bodies of special methods that initialize, run, and finalize tests: # ---- unittest example, from Python 2.2 Library Reference ---- # ---- [section 5.3.1, "Organizing test code"] ---- class WidgetTestCase(unittest.TestCase): def setUp(self): self.widget = Widget("The widget") def tearDown(self): self.widget.dispose() self.widget = None def testDefaultSize(self): self.failUnless(self.widget.size() == (50,50), 'incorrect default size') def testResize(self): self.widget.resize(100,150) self.failUnless(self.widget.size() == (100,150), 'wrong size after resize') unittest test classes are then installed in container classes that support automated test execution: # ---- unittest example, continued ---- def suite(): suite = unittest.TestSuite() suite.addTest(WidgetTestCase("testDefaultSize")) suite.addTest(WidgetTestCase("testResize")) return suite ADEPT's declarative approach to test creation is made possible by Python's eval() primitive. ADEPT users create test cases as tuples that describe a test's intent, define its core action, and characterize that action's expected consequences: # ---- unittest widget examples, redone in ADEPT ---- getTest =\ ("check Widget default size", # description for tests log eqValue((50,50)), # test predicate for action's result: # eqValue((50,50)) is a callable obj. # that returns true (1) if applied to # (50,50), and false (0) otherwise Widget("The Widget").size, # method that does the action # *args list for action is missing; # method gets () as *args # **kwds dict for action is missing; # method gets {} as **kwds ) sampleWidget = Widget("The widget") # "witness" object for set test setTest =\ ("test resize to (100,150)", # description for tests log eqValue((100,150)), # test predicate for action's effect: # eqValue((100,150)) is a callable obj. # that returns true (1) if applied to # (100,150), and false (0) otherwise "sampleWidget.size()", # expr that yields the action's effect sampleWidget.resize, # method that does the action (100, 150), # *args list for action # **kwds dict for action is missing; # method gets {} as **kwds ) The getTest and setTest tuples shown here typify what ADEPT calls "get actions" and "set actions" tests: checks for an action's returned results and effects, respectively. The getTest tuple, when evaluated as a "get actions" test, directs ADEPT to evaluate eqValue((50,50))(Widget("The widget").size()) and record success iff this expression evaluates to 1 (true). The setTest tuple, when evaluated as a "set actions" test, directs ADEPT to first execute sampleWidget.resize(100,150), then evaluate eqValue((100,150))(sampleWidget.size()) and record success iff this expression evaluates to 1 (true). Using ADEPT, the two tuples are evaluated by tagging them with a test type string; embedding them in a list; and passing this list, along with an ADEPT test logging object, to ADEPT's test engine, doTestSuite(): # the logger labels tests, and stores labels and test results # >>> suite = [("get actions",(getTest,)), ("set actions",(setTest,))] >>> widgetLogger = TestLogger() >>> doTestSuite(suite, widgetLogger) # # assumption here: all tests succeeded # >>> doTestSuite(), by default, generates output messages only when tests fail. doTestSuite() keyword parameters, described in Section 3, change the function's policies for labeling tests and generating output messages. When doTestSuite() completes, the logger object can be queried for the names and statuses of the just-completed tests--or of any past tests that used the logger object: >>> widgetLogger.failureCount() 0 >>> widgetLogger.successCount() 2 >>> widgetLogger.failures() () # # for sort order, assign output to a list, and sort before printing # >>> widgetLogger.successes() ((2, 1), (1, 1)) >>> map(widgetLogger.getBanner, widgetLogger.successes) ('2.1. set actions: check Widget.resize(100,150)', '1.1. get actions: check Widget default size') 1.2 What Follows The balance of this report discusses ADEPT and its use in more detail: -. Appendix A defines a class, Siblings class, that the ensuing narrative uses to illustrate ADEPT's operation. Sibling class objects consist of a "mom" property, a "dad" property, and the following ten methods: __init__(*args, **kwds): populates a Siblings class object with a list of names. also supports the optional initialization of a Siblings object's "mom" and "dad" properties, using its **kwds argument. __str__(), prints a message that documents the object's current contents __repr__(), returns a string of the form "Siblings(name1, name2, ...)"--a string that, when evaluated, equals the value of the original object. count(), returns the number of siblings in the current instance. Note: "mom" and "dad" are not counted as siblings. contains(name), returns true (i.e., 1) if the list of siblings contains "name" add(name), adds "name" to the list of siblings drop(name), removes "name" from the list of siblings replace(oldName, newName), substitutes "newName" for "oldName" in the siblings list addAndNotify(name), adds "name" to the list of siblings, and returns true iff "name" was not yet in the list __eq__(other), returns true if "other" is an instance of Siblings class that has the same "mom" property, "dad" property, and the same set of siblings as the current instance. Appendix A also defines an auxiliary method that the Siblings class uses to validate name parameters. This method, validateName, -. returns None if its argument is a string that consists of a capital letter, followed by an arbitrary sequence of letters and hyphens--and -. throws an exception if the argument is either a non-string (TypeError), or a string with invalid content (ValueError). -. Section 2 discusses ADEPT's built-in test types, which include five tests of executable methods (error, null, get, set, get+set), and four tests of Python **properties** (get error, set error, get, set). The section covers each test's purpose and operation; documents each test's format; and discusses representative strategies for test usage. -. Section 3 discusses the use of doTestSuite() to vary how doTestSuite() manages tests. The widget example in Section 1.1 shows doTestSuite()'s default mode of operation, which -. suppresses status messages for successful tests; and -. numbers test instances, using a two-level indexing scheme that assigns major and minor numbers to -. distinct executions of test sets (major numbers), and -. distinct tests (minor numbers); and Section 3 describes the following mechanisms for altering doTestSuite()'s default operation: -. doTestSuite()'s "show" keywords, which specify the types of status messages that are output; -. doTestSuite()'s "ostream" keyword, which specifies the object to which output is directed; and -. the TestLogger class's index property and increment methods, which support the management of test indexing. Examples include test runs that use different kinds of indexing, as well as strategies for using iterators to drive tests. -. Section 4 discusses features not covered in Sections 2 and 3, including -. ADEPT's mechanism for using multiple predicates--rather than the one predicate shown in the examples above--to check a test's outcome; -. doTestStack(), an extension of doTestSuite() that executes multi-tiered test suites; and -. the use of "mask tuples" to limit banner output. -. Section 5 reviews the programs in the main ADEPT package: -. adept.py, the suite proper; -. adeptTestSuite.py, a 782-case test suite for ADEPT, in ADEPT; and -. adeptExamples.py, a test suite that runs the examples in this manual. -. Section 6 discusses ADEPT's design, including -. gaps in ADEPT's feature set; -. procedures for adding new types of tests; and -. inherent limitations in ADEPT's testing algorithm. 2. ADEPT Test Types 2.1 ADEPT Test Categories ADEPT currently supports nine types of tests. Five of these test what will be referred to here as **methods**--codes associated with objects created by "def" keywords, and other executable objects. Test types include -. "error actions" test: executes a method, and checks for a specified exception. -. "null actions" test: executes a method, then verifies that the method returns 'None'. -. "get actions" test: executes a method, then uses a caller-supplied predicate to validate the method's return value. -. "set actions" test: executes a method, then uses a caller-supplied expression to retrieve the effect, and a caller-supplied predicate to validate it. -. "get+set actions" test: executes a method, then uses predicates to validate the method's result *and* effect. The remaining four ADEPT tests test the operation of class **properties**--a sort of virtual data attribute introduced in Python 2.2. Test types include -. "get prop error actions" test: reads a property's value, and checks for a specified exception. -. "set prop error actions" test: updates a property, and checks for a specified exception. -. "get prop actions" test: reads a property's value, then uses a caller-supplied predicate to validate the result. -. "set prop actions" test: updates a property, then uses a caller-supplied predicate to retrieve the update's effect, and a caller-supplied predicate to validate it. A final section covers the use of "get actions" tests to test Python's __repr__() (representation) methods: a kind of method that, by convention, should return a string that evaluates to a copy to the original object. 2.2 Method Testing Test Types 2.2.1 Introduction: "Method" as a Shorthand for "Callable Object" The tests described below can be applied to any callable Python object, including free functions; class methods; classes (i.e., class __init__ methods); lambda expressions; and **functors**--instances of classes that support __call__ methods. 2.2.2 Error Actions Error action tests apply a method to an argument list, and succeed if this application yields a tuple-specified type of exception: # sample tests for the validateName/Siblings codes # errorActionTest1 =\ ("validateName should reject non-string (3)'", TypeError, # expected exception validateName, # method that does the action (3,), # *args list for action # method gets {} as **kwds ) errorActionTest2 =\ ("validateName should reject 'R0n'", ValueError, # expected exception validateName, # method that does the action ('R0n',), # *args list for action # method gets {} as **kwds ) errorActionTest3 =\ ("construcing Siblings object with dad='Ar2r' should fail", ValueError, # expected exception Siblings, # method that does the action # (i.e., Siblings.__init__()) (), # *args list for action {'dad':'Ar2r'}, # keyword dict to pass to method ) errorActionTest4 =\ ("adding 'Ch@rl1e' to Siblings('Ron','Bill') should fail", ValueError, # expected exception Siblings('Ron','Bill').add, # method that does the action ('Ch@rl1e',), # *args list for action # method gets {} as **kwds ) errorActionTestSet =\ ("error actions", (errorActionTest1, errorActionTest2, errorActionTest3, errorActionTest4)) Each error action test tuple consists of between three and five items: -. A string that documents the test's purpose. ADEPT folds this string into a descriptive banner, which can be recovered by supplying its index to a test's logger object: e.g., >>> logger = TestLogger() >>> doTestSuite(logger, "Sibling tests", errorActionTestSet) >>> logger.getBanner((1,1,2)) '1.1.2. Sibling tests: error actions: validateName should reject 'R0n'' This string has no other effect on a test's execution or outcome. -. The type of exception to anticipate. An error test succeeds iff the action specified by the tuple's remaining arguments raises either this exception, or a subclass of this exception. For example, -. specifying an expected exception of "KeyError" directs ADEPT to record success when the action raises a KeyError exception; -. specifying an expected exception of "LookupError" directs ADEPT to record success when the action raises KeyError or IndexError; and -. specifying an expected exception of "Exception" directs ADEPT to record success when the action raises any Exception -. An instance of the method (i.e., callable object) to test. -. An optional argument list to pass to the specified method instance. This parameter, if omitted, defaults to the null list, (). -. An optional keyword dict to pass to the specified method instance. This parameter, if omitted, defaults to the null dict, {}. In the examples shown here, -. The errorActionTest1 tuple, when evaluated as an "error actions" test, causes ADEPT to execute validateName(3), and record success, since the action raises "IndexError". -. The errorActionTest2 tuple, when evaluated as an "error actions" test, causes ADEPT to execute validateName('R0n'), and record success, since the action raises "ValueError". -. The errorActionTest3 tuple, when evaluated as an "error actions" test, causes ADEPT to execute Siblings(dad='Ar2r'), and record success, since the action raises "ValueError". -. The errorActionTest4 tuple, when evaluated as an "error actions" test, causes ADEPT to execute Siblings('Ron','Bill').add('Ch@rlie'), and report success, since the action raises "ValueError". 2.2.3 Null Actions Null action tests apply a method to an argument list, and succeed if this action returns 'None': # more sample tests for the validateName/Siblings codes # nullActionTest1 =\ ("validateName should accept 'Virginia'", validateName, # method that does the action ('Virginia',), # *args list for action # method gets {} as **kwds ) nullActionTest2 =\ ("validateName should accept 'Art'", validateName, # method that does the action ('Art',), # *args list for action # method gets {} as **kwds ) nullActionTestSet = ("null actions", (nullActionTest1, nullActionTest2)) Each null action test tuple consists of between two and four items: -. A string that describes the test. -. An instance of the method to test. -. An optional argument list to pass to the instance (default: ()). -. An optional keyword dict to pass to the instance (default: {}). In the examples shown here, -. The nullActionTest1 tuple, when evaluated as a "null actions" test, causes ADEPT to record success, since validateName('Virginia') returns None. -. The nullActionTest2 tuple, when evaluated as a "null actions" test, causes ADEPT to record success, since validateName('Art') returns None. 2.2.4 Get Actions Get action tests apply a method to an argument list, and check whether the value obtained is accepted by a user-supplied predicate: # more sample tests for the validateName/Siblings codes # getActionTest1 =\ ("Siblings('Fred','Bill','Percy').contains('Percy') should yield 1", eqValue(1), # test predicate for action's result Siblings('Fred','Bill','Percy').contains, # method that does the action ('Percy',), # *args list for action # method gets {} as its **kwds arg ) getActionTest2 =\ ("Siblings('Fred','Bill','Percy').contains('Ron') should yield 0", eqValue(0), # test predicate for action's result Siblings('Fred','Bill','Percy').contains, # method that does the action ('Ron',), # *args list for action # method gets {} as its **kwds arg ) getActionTest3 =\ ("Siblings('Fred','Bill',mom='Molly').contains('Molly') should yield 0", eqValue(0), # test predicate for action's result Siblings('Fred','Bill',mom='Molly').contains, # method that does the action ('Molly',), # *args list for action # method gets {} as its **kwds arg ) getActionTest4 =\ ("Check Siblings('Fred','Bill','Ron') eq Siblings('Fred','Bill','Ron')", eqValue(Siblings('Fred','Bill','Ron')), # test predicate for action's result Siblings, # method that does the action ('Fred','Bill','Ron'), # *args list for action # method gets {} as its **kwds arg ) getActionTest5 =\ ("Check Siblings('Fred','Bill','Ron') eq Siblings('Ron','Fred','Bill')", eqValue(1), # test predicate for action's result Siblings('Fred','Bill','Ron').__eq__, # method that does the action (Siblings('Ron','Fred','Bill'), ) # *args list for action # method gets {} as its **kwds arg ) getActionTest6 =\ ("`Siblings('Fred','Bill','Ron')` should equal repr for permutation", eqValue(Siblings('Fred','Bill','Ron').__repr__()), # test predicate for action's result Siblings('Fred','Bill','Ron').__repr__, # method that does the action # method gets () as its arglist # method gets {} as its **kwds arg ) getActionTest7 =\ ("`Siblings('Fred','Bill','Ron')` should equal repr for permutation", eqRepr(Siblings('Fred','Bill','Ron')), # test predicate for action's result Siblings, # method that does the action ('Ron','Fred','Bill'), # *args list for action # method gets {} as its **kwds arg ) getActionTest8 =\ ("Siblings('Ginny','Percy',mom='Molly').__str__() should yield a string", isInstanceOf(str), # test predicate for action's result Siblings('Ginny','Percy',mom='Molly').__str__, # method that does the action # method gets () as its arglist # method gets {} as its **kwds arg ) getActionTestSet =\ ("get actions", (getActionTest1, getActionTest2, getActionTest3, getActionTest4, getActionTest5, getActionTest6, getActionTest7, getActionTest8)) Each get action test tuple consists of between three and five items: -. A string that describes the test. -. A predicate that accepts one argument--ADEPT will pass it the value returned by the action--and that returns true iff the value satisfies the predicate. -. An instance of the method to test. -. An optional argument list to pass to the instance (default: ()). -. An optional keyword dict to pass to the instance (default: {}). Every predicate shown above is an instance of a **functor**--a class that supports a __call__() method. Each functor supports two methods: -. an __init__() method, which accepts a value against which to test. -. a __call__() method, which accepts a value to test. An instance of a class with a __call__() method will behave like a def-defined method when invoked with an argument list. For example, the expression eqValue(x)(y), where class eqValue is defined as class eqValue(object): def __init__(self, expectedValue): self.__expectedValue = expectedValue def __call__(self, actualValue): return actualValue == self.__expectedValue returns 1 (true) when x == y, and false otherwise. Tests 1 through 6 above use eqValue() to evaluate test results, in the following ways: -. The getActionTest1 tuple, when evaluated as a "get actions" test, causes ADEPT to record success, since -. Siblings('Fred','Bill','Percy').contains('Percy') returns 1, and -. eqValue(1)(1) returns 1. -. The getActionTest2 tuple, when evaluated as a "get actions" test, causes ADEPT to record success, since -. Siblings('Fred','Bill','Percy').contains('Ron') returns 0, and -. eqValue(0)(0) returns 1. -. The getActionTest3 tuple, when evaluated as a "get actions" test, causes ADEPT to record success, since -. Siblings('Fred','Bill',mom='Molly').contains('Molly') returns 0 (since 'Molly' is not a sibling), and -. eqValue(0)(0) returns 1. -. The getActionTest4 tuple, when evaluated as a "get actions" test, causes ADEPT to record success, since -. Siblings('Fred','Bill','Ron').__eq__(Siblings('Fred','Bill','Ron')) returns 1 (because __eq__ treats the lists of siblings as sets when comparing lists of siblings), and -. eqValue(1)(1) returns 1. -. The getActionTest5 tuple, when evaluated as a "get actions" test, causes ADEPT to record success, since -. Siblings('Fred','Bill','Ron').__eq__(Siblings('Ron','Fred','Bill')) returns 1 (because __eq__ treats the lists of siblings as sets when comparing lists of siblings), and -. eqValue(1)(1) returns 1. -. The getActionTest6 tuple, when evaluated as a "get actions" test, causes ADEPT to record success, since -. Siblings.__repr__() sorts sibling names before generating output, thereby ensuring that the repr string for any permutation of 'Fred', 'Ron', and 'Bill' is "Siblings('Bill','Fred','Ron')", and -. eqValue(\ "Siblings(''Bill','Fred','Ron')")("Siblings(''Bill','Fred','Ron')") returns 1. Test getActionTest4 illustrates the importance of defining __eq__ in the desired way. If the definition of __eq__() is removed from Siblings, test 4 will use Python's built-in equality operator to do the test. This operator uses a hash-based identity test rather than content to establish equality. For example, given the assignments a = b = Siblings('Fred','Bill','Ron') c = Siblings('Fred','Bill','Ron') Python will treat a and b as equal; a and c as unequal, and b and c as unequal. This definition would also cause test 4 to fail. Sample test 5 could also have been written as getActionTest5b =\ ("Check Siblings('Fred','Bill','Ron') eq Siblings('Ron','Fred','Bill')", eqValue(Siblings('Fred','Bill','Ron')), # test predicate for action's result Siblings, # method that does the action ('Ron','Fred','Bill'), # *args list for action # method gets {} as its **kwds arg ) The one reason, if any, for favoring test 5 over test 5b is that test 5 makes that test's focus on __eq__() more apparent. Test getActionTest7 is an alternative to getActionTest6 that shows the use of __repr__() to check equality. According to the Python Language reference, Obj.__repr__(), where Obj is an arbitrary object, should, if possible, encode Obj's contents in a way that can be used to reconstruct Obj's value--presumably using eval(). If a class's repr strings are indeed equivalent to the original objects, *and* repr strings are also **normalized**--that is, if a == b => `a` == `b` then the following eqRepr functor can be also be used to test for equality: class eqRepr(object): def __init__(self, expectedValue): self.__expectedValue = expectedValue def __call__(self, actualValue): return `actualValue` == `self.__expectedValue` The final test, getActionTest8, illustrates the use of an approximate check where precise testing would be cumbersome to do. The functor, isInstanceOf, class isInstanceOf(object): def __init__(self, expectedType): self.__expectedType = expectedType def __call__(self, actualValue): return isinstance(actualValue, self.__expectedType) verifies the type of Siblings.__str__()'s return value, rather than the value itself. Python's guidelines for __str__() state only that __str__() should return a string-based characterization of an instance's contents: they do not prescribe what that characterization should contain. Functors included in the ADEPT test suite include -. eqBasename: check if two pathname string's basenames are equal -. eqValue: use __eq__ to check object equality -. eqRepr: check repr string equality (i.e., to check object equality) -. eqContents: check if two sequence objects have equal contents -. containsRE: check if a string contains a given RE -. lacksRE: check if a string lacks a given RE -. evalsTo: check if a string evaluates to a value (discussed below) -. isInstanceOf: check if object is an instance of a type Other user-defined functors may, of course, be used to test for other conditions: e.g., class inBounds(object): def __init__(self, lowBound, highBound): self.low, self.high = lowBound, highBound def __call__(self, v): return (v >= self.low) and (v <= self.high) stillInSchool = Siblings('Fred','George','Ron','Ginny',dad='Arthur',mom='Molly) # should succeed, since Siblings.count doesn't count mom or dad # getActionTest10 =\ ("there are still somewhere between 3 and 5 kids in school, right?", inBounds(3, 5), stillInSchool.count, ) ADEPT also supports the use of two or more predicates to check a test's outcome. The use of multiple predicates is discussed at length in Section 4, where this feature, together with the containsRE and lacksRE functors, are used to develop a stronger version of getActionTest8. 2.2.5 Set Actions Set action tests apply a method to an argument list; evaluate a user-supplied string to retrieve the application's effect; and verify this effect, using a user-supplied predicate: setTestObj_1 = Siblings('Percy','George','Bill') setTestObj_234 = Siblings('Percy','George','Bill') setActionTest1 =\ ("drop Bill from Siblings('Percy','George','Bill')", eqValue(0), # test for action's effect "setTestObj_1.contains('Bill')", # expr that yields action's effect setTestObj_1.drop, # method that does the action ('Bill',), # *args list for action # method gets {} as **kwds ) setActionTest2 =\ ("drop Bill from Siblings('Percy','George','Bill')", eqValue(Siblings('Percy','George')), # test for action's effect "setTestObj_234", # expr that yields action's effect setTestObj_234.drop, # method that does the action ('Bill',), # *args list for action # method gets {} as **kwds ) setActionTest3 =\ ("replace Percy with Fred in Siblings('Percy','George') (cont. prev.)", eqValue(Siblings('Fred','George')), # test for action's effect "setTestObj_234", # expr that yields action's effect setTestObj_234.replace, # method that does the action ('Percy', 'Fred',), # *args list for action # method gets {} as **kwds ) setActionTest4 =\ ("add Ron to Siblings('Fred','George') (continues previous test)", eqValue(Siblings('Fred','George','Ron')), # test for action's effect "setTestObj_234", # expr that yields action's effect setTestObj_234.add, # method that does the action ('Ron',), # *args list for action # method gets {} as **kwds ) setActionTestSet =\ ("set actions", (setActionTest1, setActionTest2, setActionTest3, setActionTest4)) Each set action test tuple consists of between four and six items: -. A string that describes the test. -. A predicate that accepts one argument--ADEPT will pass it the effect produced by the action--and that returns true iff the value satisfies the predicate. -. A string that, when evaluated, returns the effect produced by applying the method to its arguments. -. An instance of the method to test. -. An optional argument list to pass to the instance (default: ()). -. An optional keyword dict to pass to the instance (default: {}). In the examples shown here, -. The setActionTest1 tuple, when evaluated as a "set actions" test, causes ADEPT to -. alter the value of setTestObj_1 from Siblings('Percy','George','Bill') to Siblings('Percy','George'), then -. record success, since evaluating eqValue(0)(setTestObj_1.contains('Bill')) returns 1 -. The setActionTest2 tuple, when evaluated as a "set actions" test, causes ADEPT to -. alter setTestObj_234 from Siblings('Percy','George','Bill') to Siblings('Percy','George'), then -. record success, since eqValue(Siblings('Percy','George'))(setTestObj_234) returns 1 -. The setActionTest3 tuple, when evaluated as a "set actions" test, causes ADEPT to -. alter the value of setTestObj_234 from Siblings('Percy','George') to Siblings('Fred','George'), then -. record success, since eqValue(Siblings('Fred','George'))(setTestObj_234) returns 1 -. The setActionTest4 tuple, when evaluated as a "set actions" test, causes ADEPT to -. alter the value of setTestObj_234 from Siblings('Fred','George') to Siblings('Fred','George','Ron'), then -. record success, since eqValue(Siblings('Fred','George','Ron'))(setTestObj_234) returns 1 The variables setTestObj_1 and setTestObj_234 are examples of what are referred to here as **witness objects**. Witness objects, which function as mechanisms for capturing the effect of a set operation, are something of a necessary evil in the testing of imperative languages. Methods that are designed to change objects in lasting ways need to be tested by first applying those methods to sample objects, then verifying their effects. ADEPT simply leaves the creation, management, and naming of witness objects up to the tester, as a way of keeping the suite simple. The series of examples above shows two strategies for managing witness objects: -. Use a different witness object for each test, as a way of isolating each test from all others. This strategy is illustrated by setActionTest1. -. Use a common witness object to drive a series of related tests. This strategy is illustrated by sample set action tests 2, 3, and 4. As a rule, the author recommends that tests be run in isolation. Designs that force testers to cascade tests to achieve the effect of unit testing should probably be rethought, where redesign is possible. As a part of set action testing, the user must also specify expressions for recovering an action's effect. ADEPT's mechanisms for specifying and testing effects are expressive enough to allow the user to apply arbitrarily complex predicates to arbitrary collections of objects. The author, however, recommends structuring a typical test as a simple check of a single object. There are two reasons for this recommendation: -. In theory, a careful check of a set action's operation ought to assess that action's impact on *every* mutable object in its environment. (Consider, for example, errors that arise from updates through inadvertant aliases.) Such checks, however, would be difficult to write, and expensive to do in Python, which imposes few restrictions on what actions can do. Limiting checks to those objects that an action is designed to update represents a compromise between what should be checked, and what can be checked without too much work. -. If there is a need to test a method's effect on two or more objects, there is probably a more urgent need to rethink the method. A classic principle of software design, which originated in work by David Parnas and Larry Constantine in the 1970's, holds that methods that do one well-defined action on one well-defined object are easier to understand, develop, and maintain than methods that do sets of more loosely related tasks. With the possible exception of transactions--sets of actions that must be done as atomic actions to maintain a system's integrity--methods that do multiple things to multiple objects should probably be broken into a set of functionally cohesive actions before even testing them. This redesign should improve the program's overall quality, *and* make the resulting actions easier to test. Even when a test is framed as a simple assertion about a single action on single object, the tester may face choices regarding how much of an object to examine to confirm an action's effect. This concern arises in connection with tests of complex objects, like the ones shown here. -. The designer, on the one hand, may choose to test an action's effect by checking the value of the entire updated object. This "holistic" approach to testing is illustrated by sample tests 2 through 4 above, and elsewhere throughout the document. -. The designer may also choose to test only those parts of an object that an action is designed to effect. Tests that illustrate this more "focused" approach include Section 1's second test, which uses widget.size() to test widget.resize(), and setActionTest1, which uses Siblings.contains() to test Siblings.drop(). Focusing on only that part of an object that was supposed to change risks missing unintended effects: e.g., accidental changes to a widget's color or line width, or unintended erasures of Percy or George along with Bill. Focused testing, however, may be the only practical means of testing a particular class's instances, particularly in situations involving complex classes and third-party software. 2.2.6 Get+Set Actions Get+set action tests combine a set action with check for a desired result *and* a desired effect, as shown below: getsetTestObj_12 = Siblings('Ron') getSetActionTest1 =\ ("add Ginny to Siblings('Ron')", eqValue(1), # test for action's result eqValue(Siblings('Ginny','Ron')), # test for action's effect "getsetTestObj__12", # expr that yields action's effect getsetTestObj_12.addAndNotify, # method that does the action ('Ginny',), # *args list for action # method gets {} as **kwds ) getSetActionTest2 =\ ("add Ron to Siblings('Ron','Ginny')", eqValue(0), # test for action's result eqValue(Siblings('Ginny','Ron')), # test for action's effect "getsetTestObj__12", # expr that yields action's effect getsetTestObj_12.addAndNotify, # method that does the action ('Ron',), # *args list for action # method gets {} as **kwds ) getSetActionTestSet =\ ("get+set actions", (getSetActionTest1, getSetActionTest2)) Each get+set action test tuple consists of between five and seven items: -. A string that describes the test. -. A predicate that accepts one argument--ADEPT will pass it the result returned by the action--and that returns true iff the value satisfies the predicate. -. A predicate that accepts one argument--ADEPT will pass it the effect produced by the action--and that returns true iff the value satisfies the predicate. -. A string that, when evaluated, returns the effect produced by applying the method to its arguments. -. An instance of the method to test. -. An optional argument list to pass to the instance (default: ()). -. An optional keyword dict to pass to the instance (default: {}). Tests of get/set actions combine the functionality of the get and set tests discussed in the previous section. The two examples shown here test Siblings.addAndNotify(), a routine that -. adds its argument to a Siblings class instance, and -. returns 1 iff the name was not yet present. The first tuple, when executed as a "get+set action" test, should return 1, because it has to add 'Ginny' to the instance's set of siblings. The second tuple, when executed as a "get+set action" test, should return 0, since 'Ron' is already in getSetCaseCase_12. The routine used to illustrate get+set testing, Siblings.addAndNotify(), exhibits what Constantine refers to as "sequential cohesion": a packaging of two or more distinct actions that do not, in and of themselves, constitute a complete, well defined action. Get+set methods, which typically have words like "and" in their description, tend to exhibit lower levels of cohesion than the pure get and pure set methods from which they are cobbled together. Get+set testing, however, needs to be supported in tools like ADEPT, because what users "should" design and what users do design are different things. And the need for get+set testing was identified in the context of a design problem that could not be designed away: i.e., the creation of a dict-like class that supports a version of Python's sequentially cohesive dict method, setdefault(): a.setdefault(k[, x]) a[k] if k in a, else x (also setting it) [source: Python Library Reference, 2.2.7, "Mapping Types"] 2.3 Property-Testing Tests 2.3.1 Concerning Properties and Property Tests Properties, which were added to Python in v2.2, are sets of functions that constitute what might be termed a *virtual* data attribute. The Siblings class's 'mom' and 'dad' properties, for example, support the use of Siblings.getMom(), Siblings.setMom(), Siblings.getDad(), and Siblings.setDad() to update their host instances--but do so in a way that is transparent to the client code: >>> sample = Siblings() >>> sample.mom = 'Molly' >>> sample.dad = 'Arthur' >>> sample.mom 'Molly' >>> sample.dad 'Arthur' Python's eval() method does not support assignment statements, a limitation that complicates eval()-based testing. Operations on properties, however, can be tested by recasting them as calls to the actual (class) methods that Python uses to implement properties. Python, for example, recasts sample.mom = "Molly" as Siblings.mom.fset(sample,'Molly')--and then, in turn, as a call to Siblings.setMom(). ADEPT's property tests hide the details of property implementation by allowing the tester to write tuples that name the class, property, and instance being tested; ADEPT then creates and executes the required function calls transparently, on the tester's behalf. ADEPT supports tests that check how properties are read and written. A Python property may also be associated with two other kinds of actions, a delete action and a docstring action: # start of an extended version of the Siblings class that supports delete # and docstring operations on "dad" # class Siblings(object): """ keeps unordered list of siblings, and their mom and dad """ def getDad(self): if self.__dad is None: raise AttributeError, "dad not specified" return self.__dad def setDad(self, dadName): validateName(dadName) self.__dad = dadName def delDad(self): validateName(dadName) del self.__dad def docForDad(self): return "siblings' father: value is " + `self.__dad` dad = property(getDad, setDad, delDad, docForDad) ADEPT currently fails to support tests of delete and docstring actions, because of the author's desire to limit the complexity of the baseline suite. These tests, however, can be added to ADEPT, if needed; see the design notes section for a brief overview of how to add tests to ADEPT. 2.3.2 Get Property Error Action Tests Get property error action tests attempt to read a specified property, and succeed if the attempted read generates the anticipated exception: # defined as witness variables because of the properties problem: # forces properties to evaluate in tester env, rather than test suite env # getPropErrTestObj_1 = Siblings('Fred',mom='Molly') getPropErrTestObj_2 = Siblings('Ron',dad='Arthur') getPropErrTest1 =\ ("get Dad from partially initialized Siblings object (mom, no dad)", AttributeError, # expected exception "Siblings", # name of property's class (*quoted*) "dad", # name of property to read (*quoted*) "getPropErrTestObj_1", # instance of class to test (*quoted*) ) getPropErrTest2 =\ ("get Mom from partially initialized Siblings object (dad, no mom)", AttributeError, # expected exception "Siblings", # name of property's class (*quoted*) "mom", # name of property to read (*quoted*) "getPropErrTestObj_2", # instance of class to test (*quoted*) ) getPropErrTestSet =\ ("get prop error actions", (getPropErrTest1, getPropErrTest2)) Get property error action tests take five arguments: -. A string that describes the test. -. The type of exception to anticipate. -. The name of the class to test, as a string. -. The name of the property to test, as a string. -. The name of the class instance to test, as a string. The getPropErrTest1 tuple, when evaluated as a "set prop error actions" test, causes ADEPT to record success, since -. Siblings('Fred',mom='Molly') sets self.__dad to None, and -. Siblings.dad.fget(x,'dad') invokes Siblings.getDad(), which -. raises AttributeError when x.__dad--here, Siblings('Fred',mom='Molly').__dad--is None. Similarly, the sampleSetPropErrTest2 tuple, when evaluated as a "set prop error actions" test, yields success, since -. Siblings('Ron',dad='Arthur') sets self.__mom to None, and -. Siblings.dad.fget(x,'mom') invokes Siblings.getMom(), which -. raises AttributeError when x.__mom--here, Siblings('Fred',dad='Arthur').__mom--is None. 2.3.3 Set Property Error Action Tests Set property error action tests attempt to update a specified property, and succeed if the attempted write generates the anticipated exception: setPropErrTest1 =\ ("set Siblings() to invalid value (3)", TypeError, # expected exception "Siblings", # name of property's class (*quoted*) "dad", # name of property to read (*quoted*) "Siblings()", # instance of class to test (*quoted*) "3", # new property value (*quoted expr*) ) setPropErrTest2 =\ ("set Siblings() to invalid string ('Mrs. Molly W.')", ValueError, # expected exception "Siblings", # name of property's class (*quoted*) "mom", # name of property to read (*quoted*) "Siblings()", # instance of class to test (*quoted*) "'Mrs. Molly W.'", # new property value (*quoted expr*) ) setPropErrTestSet =\ ("set prop error actions", (setPropErrTest1, setPropErrTest2)) Set property error action tests take six arguments: -. A string that describes the test. -. The type of exception to anticipate. -. The name of the class to test, as a string. -. The name of the property to test, as a string. -. The name of the class instance to test, as a string. -. A value to assign to the specified property. The setPropErrTest1 tuple, when evaluated as a "set prop error actions" test, causes ADEPT to record success, since -. Siblings('Ron',dad='Arthur') sets self.__mom to None, -. Siblings.dad.fset(x,'mom') invokes Siblings.setDad(3), which -. raises TypeError when its argument--here, 3--is not a string. Similarly, the setPropErrTest2 tuple, when evaluated as a "set prop error actions" test, yields success, since -. Siblings('Ron',dad='Arthur') sets self.__mom to None, and -. Siblings.dad.fset(x,'mom') invokes Siblings.setDad(3), which -. raises ValueError when its argument--here, 'Ms. Molly W.'--is not a one- word name. 2.3.4 Get Property Action Tests Get property action tests read a class property, and check whether the value obtained is accepted by a user-supplied predicate: # These constructors are defined as witness variables because they # reference class properties--and, hence, cannot be passed to doTestSuite() # by name, since doTestSuite() is currently incapable of parsing these # strings, and working around the class's internal references to its # own properties # getPropTestObj_1 = Siblings('Harry',dad='James') getPropTestObj_2 = Siblings('Harry',mom='Lily') getPropTest1 =\ ("get dad from Siblings('Harry',dad='James')", eqValue('James'), # predicate for testing result "Siblings", # name of property's class (*quoted*) "dad", # name of property to read (*quoted*) "getPropTestObj_1", # instance of class to test (*quoted*) ) getPropTest2 =\ ("get mom from Siblings('Harry',mom='Lily')", eqValue('Lily'), # predicate for testing result "Siblings", # name of property's class (*quoted*) "mom", # name of property to read (*quoted*) "getPropTestObj_2", # instance of class to test (*quoted*) ) getPropTestSet = ("get prop actions", (getPropTest1, getPropTest2)) Get property action tests take five arguments: -. A string that describes the test. -. A predicate to use for validating the value returned by get. -. The name of the class to test, as a string. -. The name of the property to test, as a string. -. The name of the class instance to test, as a string. The getPropTest1 tuple, when evaluated as a "get prop actions" test, causes ADEPT to record success, since -. Siblings.dad.fget(Siblings('Harry',dad='James'),'dad') invokes Siblings.getDad(), which -. returns self.__dad: i.e., 'James' Similarly, the getPropTest2 tuple, when evaluated as a "get prop actions" test, yields success, since -. Siblings.mom.fget(Siblings('Harry',mom='Lily'),'mom') invokes Siblings.getMom(), which -. returns self.__mom: i.e., 'Lily' 2.3.5 Set Property Action Tests Set property action tests update a property, and check whether the expected effect is accepted by a user-supplied predicate: setPropTestObj = Siblings('Harry') setPropTest1 =\ ("set dad for Siblings('Harry') to James", eqValue(Siblings('Harry',dad='James')), # predicate for testing result "setPropTestObj", # expr that yields action's effect "Siblings", # name of property's class (*quoted*) "dad", # name of property to read (*quoted*) "setPropTestObj", # instance of class to test (*quoted*) "'James'", # new property value (*quoted expr*) ) setPropTest2 =\ ("set mom for Siblings('Harry') to Lily", eqValue(Siblings('Harry',mom='Lily',dad='James')), # predicate for testing result "setPropTestObj", # expr that yields the action's effect "Siblings", # name of property's class (*quoted*) "mom", # name of property to read (*quoted*) "setPropTestObj", # instance of class to test (*quoted*) "'Lily'", # new property value (*quoted expr*) ) setPropTestSet = ("set prop actions", (setPropTest1, setPropTest2)) Set property action tests take seven arguments: -. A string that describes the test. -. A predicate to use for testing the set property action's effect. -. A string that, when evaluated, returns the effect of the update. -. The name of the class to test, as a string. -. The name of the property to test, as a string. -. The name of the class instance to test, as a string. -. A value to assign to the specified property. The setPropTest1 tuple, when evaluated as a "set prop actions" test, causes ADEPT to record success, since -. Siblings.dad.fset(setPropTestObj,'dad','James') changes setPropTestObj to Siblings('Harry',dad='James'), which is -. equal to the first argument of the eqValue() test predicate Similarly, the setPropTest2 tuple, when evaluated as a "set prop actions" test, yields success, since -. Siblings.mom.fset(setPropTestObj,'mom','Lily') changes setPropTestObj to Siblings('Harry',dad='James',mom='Lily'), which is -. equal to the first argument of the eqValue() test predicate 2.4 Using "get actions" Tests to Test Repr Strings 2.4.1 Concerning Repr Methods The software design community has no single name for a method of a class C that "dumps" the contents of a class instance--i.e., that returns a comprehensive image of all data internal to the current instance of C. These functions, which will be referred to here as "dumpers", are useful tools for examining, altering, and recreating an object's contents--e.g., as a part of a debugging session, a backup and restore procedure, or a procedure that transmits the object to a remote system. The standard Python term for a "dumper" function is a __repr__() method: __repr__(self) Called by the repr() built-in function and by string conversions (reverse quotes) to compute the "official" string representation of an object. If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). [source: Python Language Reference Manual, Section 3.3.1] The Python guidelines for writing repr methods fail to specify how to recreate an object from its repr string. Python's library, however, provides a clear choice for object recreation: eval(), a function that converts a syntactically valid string into a Python object, relative to a user-specified environment. A simple strategy for converting an instance Obj of a class C to a semantically sound repr string uses C's constructor to derive a string that populates a new instance of C with Obj's contents. The idea is to generate a string of the form "C(...)" that, when evaluated, returns a new instance of C that has the same internal attributes as Obj. This approach to designing repr methods is exemplified by Siblings, whose __repr__() method exports the totality of an instance's contents, and whose __init__() method is designed as a complement to Siblings.__repr__(). This constructor-centric repr strategy suffers from three drawbacks: *. The strategy blurs the distinction between public and private data by using -. __repr__() to dump private data, along with public data, and -. __init__() to load private data items during class construction. Python, however, already blurs this distinction by treating "private" as a convention instead of an language-enforced assertion. *. The strategy complicates the design of user-friendly __init__ methods, by forcing a class's designer to find ways of -. allowing __repr__() methods, in effect, to supply __init__ methods with what would otherwise be private data, while -. allowing "ordinary" codes to avoid specifying these values. This second problem can be addressed by using a constructor's **kwds argument initialize "incidental" or "private" attributes, like "mom" and "dad" (cf. Siblings.__init__() in Appendix A). Alternatively, the final parameters in a constructor's argument list can be designed to treat "incidental" and "private" values as optional arguments, and to associate them with reasonable defaults. *. The "constructor-centric repr" strategy cannot be applied directly to subclasses of immutable classes like int and str, since immutability fixes the signatures of all base class methods, including __init__(). One admittedly messy workaround for the "frozen constructor" problem uses the equivalent of a C++ friend function to fully initialize class instances: def makeInstanceOfOnlyChild(name, dad, mom): validateName(dad) validateName(mom) instance = OnlyChild(name) instance.mom, instance.dad = mom, dad return instance class OnlyChild(str): def getVal(self): return self.__hash__() val = property(getVal) def getDad(self): if self.__dad is None: raise AttributeError, "dad unspecified" return self.__dad def setDad(self, dadName): validateName(dadName) self.__dad = dadName dad = property(getDad, setDad) def getMom(self): if self.__mom is None: raise AttributeError, "mom unspecified" return self.__mom def setMom(self, momName): validateName(momName) self.__mom = momName mom = property(getMom, setMom) def __init__(self, name): # since str is immutable, __init__'s parameter list cannot # accept values for mom and dad # self.__dad = self.__mom = None super(OnlyChildName, self).__init__(name) def __repr__(self): # __repr__ string references "makeInstanceOfOnlyChild" because # __init__() can't be made to accept values for mom and dad # return "makeInstanceOfOnlyChild(" +\ self.__repr__() + "," + `self.mom` + "," + `self.dad` + ")" richKid = makeInstanceOfOnlyChild('Draco', 'Lucius', 'Narcissa') 2.4.2 ADEPT Repr Testing The goal of repr testing to to determine whether an object "o" can be recreated from its __repr__ string--or, in Python notation, whether o == eval(`o`) If this test fails, then o.__repr__() is not a comprehensive "dumper" for o's class, in the sense recommended by Python's language reference. Repr strings can be checked using "get actions" tests, as follows: reprTest1 =\ ("check that Siblings() is invertible", evalsTo(Siblings()), # predicate for action's result Siblings, # method that does the action # method gets () as arglist # method gets {} as **kwds arg ) reprTest2 =\ ("check that Siblings('Parma','Parvati') is invertible", evalsTo(Siblings('Parma','Parvati')), # predicate for action's result Siblings, # method that does the action ('Parma', 'Parvati', ) # method gets () as arglist # method gets {} as **kwds arg ) reprTest3 =\ ("check that Siblings(mom='Lily',dad='James') is invertible", evalsTo(Siblings(mom='Lily',dad='James')), # predicate for action's result Siblings, # method that does the action (), # method gets () as arglist {'mom':'Lily','dad':'James'}, # **kwds arg ) reprTest4 =\ ("check that Siblings('Harry',mom='Lily',dad='James') is invertible", evalsTo(Siblings('Harry',mom='Lily',dad='James')), # predicate for action's result Siblings, # method that does the action ('Harry',), # method arglist {'mom':'Lily','dad':'James'}, # method **kwds arg ) reprTestSet = ("get actions", (reprTest1, reprTest2, reprTest3, reprTest4)) Repr testing is complicated by the failure of some Python objects to return syntactically valid repr strings. These objects include -. any user-defined class that either -. fails to adhere to Python's repr convention, or -. returns a string of the form "<...>"--the recommended alternative to syntactically valid expressions for "un-repr-izable" objects. -. lambda expressions, which return strings of the form " at ~someaddress~>" -. files, which return strings of the form "" -. function definitions, which return strings of the form " at ~someaddress~>" -. types, which return strings of the form "" One strategy for generating invertable repr strings for objects with unrepr-izable component values is -. to pass and store these values by name, rather than by value, and then -. to use the 'by-name' version for repr strings, and the "by value" for other purposes: e.g., class SomeClass(object): def __init__(self, SomeClass, ...): self.__problemObjectName = problemObjectName self.__problemObjectValue = eval(problemObjectName) ... def someFunction(self): ... f(self.__problemObjectValue, ...) ... def __repr__(self): return "SomeClass(" + self.__problemObjectName + "," + .. It is, however, possible to avoid call-by-name for stored named functions and types, since these objects' names can be recovered from their repr strings by first parsing the repr string, then extracting the string's second argument, and replacing the original with the second argument, minus any surrounding quotes. ADEPT's evalsTo() functor maps a string like o = "tuple( , , '')" to o_prime = "tuple(f, int, '')" before checking if o == eval(o_prime). Bracketed expressions that fail to correspond to a self-contained token--e.g., bracketed expressions that appear in strings--left intact. Unfortunately, no form of repr testing will correctly handle all repr strings, so long as any Python object--starting with lambda objects--generates a repr string that fails to adequately represent the original object. 3. Using doTestSuite() to Run Sets of Tests: Annotated Examples ADEPT's test engine, doTestSuite(), requires one parameter: -. a test suite: a list of lists that pairs -. strings that type tests (e.g., "get actions", "set actions") with -. test tuples of the specified type If doTestSuite() is called directly, instead of from doNestedTestSuite() (cf. Section 4), -. an instance of TestLogger, a class that doTestSuite() uses to -. generate an ID and a banner for each test, and -. store each test's banner and result by ID. doTestSuite() supports the use of five keyword parameters that control the types of messages displayed during test suite execution: -. showTestIDs (default==0). Passing "showTestIDs=1" to doTestSuite() causes doTestSuite() to display each test's identifying index--e.g., "1.1.", "1.2", etc.--as that test executes. This keyword is superseded by showTestBanners, below. -. showTestBanners (default==0). Passing "showTestIDs=1" to doTestSuite() causes doTestSuite() to display each test's identifying banner--e.g., "1.1. get test: get some stuff"-- as that test executes. This keyword, if enabled, supersedes the showTestIDs keyword, above. -. showTestFailures (default==1). Passing "showTestFailures=0" to doTestSuite() causes doTestSuite() to suppress announcements of test cases that execute, but that fail to yield the expected results, effects, or exceptions. -. showMalformedTests (default==1). Passing "showMalformedTests=0" to doTestSuite() causes doTestSuite() to suppress announcements of test cases that cannot be executed, due to problems with test tuple format or content. -. showTestSuccesses (default==0). Passing "showTestSuccesses=1" to doTestSuite() causes doTestSuite() to announce test cases that succeed. A sixth keyword parameter, "ostream", allows a StatusMessager's output to be redirected to any object that supports a method named "write". The use of user-specified output streams to capture and test print statement output is discussed in the section on ADEPT's limitations, below. The rest of this section covers doTestSuite()'s default mode of operation in more detail, and illustrates the use of TestLogger class's index property and increment() methods to support user-managed test index generation. The examples given here assume a testing environment with the following logical logical objects: *. the ADEPT test suite *. the Siblings class (cf. Appendix A) *. the validateName function (cf. Appendix A) *. the following objects from Section 2: -. The 32 tuples at the head of the various sections that define the sample tests: error action tuples: errorActionTest1 - errorActionTest4 null action tuples: nullActionTest1, nullActionTest2 get action tuples: getActionTest1 - getActionTest8 set action tuples: setActionTest1 - setActionTest4 get+set action tuples: getSetActionTest1, getSetActionTest2 get prop error action tuples: getPropErrTest1, getPropErrTest2 set prop error action tuples: setPropErrTest1, setPropErrTest2 get prop action tuples: getPropTest1, getPropTest2 set prop action tuples: setPropTest1, setPropTest2 repr test tuples: reprTest1 - reprTest4 -. The eight witness objects for the "set actions", "get+set actions", and property tests: setTestObj_1, setTestObj_234, getsetTestObj_12, getPropErrTestObj_1, getPropErrTestObj_2, getPropTestObj_1, getPropTestObj_2, setPropTestObj -. The ten lists that define the various sample test sets: errorActionTestSet, nullActionTestSet, getActionTestSet, setActionTestSet, getSetActionTestSet, getPropErrTestSet, setPropErrTestSet, getPropTestSet, setPropTestSet, reprTestSet *. the following test-set and test-suite objects: testSetList =\ [ errorActionTestSet, nullActionTestSet, getActionTestSet, setActionTestSet, getSetActionTestSet, getPropErrTestSet, setPropErrTestSet, getPropTestSet, setPropTestSet, reprTestSet ] # really, a two-level suite, but each test type is paired with exactly # one test # oneTierTestSuite =\ reduce(\ lambda singletonTestSetList, singletonTestSet: \ singletonTestSetList + singletonTestSet, map(lambda (testLabel, testCaseList): \ [(testLabel,(testCase,)) for testCase in testCaseList], testSetList), []) twoTierTestSuite = testSetList[:] # a suite that splits the two-tier suite into two subsuites: # -. a first that's specific to validateName # -. a second that's specific to the Siblings class # threeTierTestSuite =\ [ [ "validateName tests", ( ("error actions", (errorActionTest1, errorActionTest2)), nullActionTestSet, ), ], [ "Siblings class tests", [ ("error actions", (errorActionTest3, errorActionTest4)) ] +\ testSetList[2:] ], ] 3.1 Default Index Generation for doTestSuite() doTestSuite(), in its default mode of operation, generates sequences of two-level test indices, updating the major number at the start of every new test set, and the minor for every new test. *. [Example 3.1.1] Executing the following code logger = TestLogger() doTestSuite(oneTierTestSuite, logger) assigns the following indices to the 32 tests in oneTierTestSuite: error action test set: 1.1 - 4.1 null action test set: 5.1 - 7.1 ... repr test set: 29.1 - 32.1 *. [Example 3.1.2] Executing the following code logger = TestLogger() doTestSuite(twoTierTestSuite, logger) assigns the following indices to the 32 tests in twoTierTestSuite: error action test set: 1.4 - 1.4 null action test set: 2.1 - 2.2 get action test set: 3.1 - 3.8 ... repr test set: 10.1 - 10.4 3.2 Using TestLogger.index and TestLogger.increment() to Change Indexing TestLogger's index property and increment() methods help to control the indices generated by doTestSuite(). The need for tester-controlled indexing arises in situations where -. a test run is structured as a series of calls to doTestSuite(), and -. the test suite designer wants to assign test numbers in a way that reflects the hierarchical structure of the overall suite. 3.2.1 Example Involving Three-Tier Test Suite Executing the following code logger = TestLogger() for (suiteName, suite) in threeTierTestSuite: doTestSuite(suite, logger) logger.increment(1) assigns the following indices to the 32 tests in threeTierTestSuite: validateName tests: error action test set: 1.1 - 1.2 null action test set: 2.1 - 2.2 Siblings class tests: error action test set: 3.1 - 3.2 get action test set: 4.1 - 4.2 ... repr test set: 11.1 - 11.4 If the call to logger.increment() is omitted from the "for" loop, the null action and error action tests are both assigned 2.1 and 2.2, obliterating the record of the null action tests. 3.2.2 Example Involving One-Tier Test Suite The following pattern of indices-- error action test set: 1.1.1 - 1.1.4 null action test set: 1.2.1 - 1.2.2 error action test set: 1.3.1 - 1.3.2 ... repr test set: 1.10.1 - 1.10.4 --can be assigned to the tests in the one-tier suite using TestLogger.index and TestLogger.incrementIndex(), as shown below: logger = TestLogger() # # adjust logger to support extended indexing: # -. specify 3-component index (default: 2) # -. set initial index to 1.1.1. # -. associate the string "Siblings tests:" with all banners # of the form 1.*.* # logger.index = TestIndex(1,0,0) logger.setBannerSegment(1, "Siblings tests: ") # # track changes to test type--and change major test number # whenever the test type changes # previousTestType = '' for test in oneTierTestSuite: if test[0] != previousTestType: logger.increment(2) previousTestType = test[0] doTestSuite((test,), logger) Comments: *. The initial assignment to logger.index sets up the use of a three- level index, instead of the default, two-level index. *. The call to logger.setBannerSegment() associates the prefix "Siblings tests: " with every banner whose index is 1.x.x. -. "Siblings tests: " is treated as a prefix because the string is associated with the index's first field--thanks to the use of '1' as setBannerSegment's leading argument. -. "Siblings tests: " is associated with banners of the form 1.x.x because the current test index's leading component is '1'--thanks to the assignment to logger.index *. doTestSuite() is called once per test case, rather than once for the entire suite, to prevent it from incrementing the major test number for every new test set in oneTierTestSuite. Instead, the loop increments major test numbers itself, whenever the test type changes. 3.2.3 Example Involving Two-Tier Test Suite The following pattern of indices-- error action test set: 1.1.1 - 1.1.4 null action test set: 1.2.1 - 1.2.2 error action test set: 1.3.1 - 1.3.2 ... repr test set: 1.10.1 - 1.10.4 --can be assigned to the tests in the two-tier suite using TestLogger.index, as shown below: logger = TestLogger() logger.index = TestIndex(1,1,1) logger.setBannerSegment(1, "Siblings tests: ") doTestSuite(twoTierTestSuite, logger) Here again, the initial assignment to logger.index sets up the use of a three-level index. The test suite's structure, when combined with doTestSuite()'s default behavior, yields the desired indices. 3.2.4 Second Example Involving Three-Tier Test Suite The following pattern of indices-- validateName tests: error action test set: 1.1.1 - 1.1.2 null action test set: 1.2.1 - 1.2.2 Siblings class tests: error action test set: 2.1.1 - 2.1.2 get action test set: 2.2.1 - 2.2.8 ... repr test set: 2.9.1 - 2.9.4 --can be assigned to the tests in the three-tier suite using TestLogger.index and TestLogger.incrementIndex(), as shown below: logger = TestLogger() logger.index = TestIndex(1,1,1) for (suiteName, suite) in threeTierTestSuite: logger.setBannerSegment(1, suiteName) doTestSuite(suite, logger) logger.increment(1) Once again, the initial assignment to logger.index sets up the use of a three-level index. The call to logger.incrementIndex() steps index's initial field--field 1--which doTestSuite() does not update. 3.3 Examples with Iterators In the examples shown above, it *should* be possible to replace the references to test suites with references to iterators, which return elements of those suites when accessed by "for" statements: # Example 3.3.1: **** don't try this in Python 2.2 **** # logger = TestLogger() nextPass = TestCaseIteratorGenerator().__iter__() try: while 1: # # WARNING: thanks to an apparent error in Python 2.2, # the statement in nextPass.next() that raises # StopIteration on the 33rd call to nextPass.next() # does **not** halt the operation of doTestSuite()'s # main 'for' loop. # doTestSuite(nextPass, logger) except: pass As Example 3.3.1 notes, this code--which would generate 1.1 .. 32.1 as indices, in any case--DOES NOT WORK in Python 2.2. The following does: # Example 3.3.2 - case-at-a-time iterator, smarter index generation # logger = TestLogger() nextPass = TestCaseIteratorGenerator().__iter__() logger.index = TestIndex(1,0,0) logger.setBannerSegment(1, "Siblings tests: ") previousTestType = '' for test in nextPass: if test[0] != previousTestType: logger.incrementIndex(2) previousTestType = test[0] doTestSuite(test, logger) The nextPass object (see Appendix B) simply uses a self-contained index to step through a predefined object--oneTierTestSuite. For the sake of this discussion, however, imagine a version of this iterator that somehow generated the 32 test cases dynamically, one (testType, (testCase,)) pair at a time, without resorting to a predefined list. This iterator would reduce the test procedure's peak memory demand by eliminating the need to keep all tests in memory at all times; each of the two-element "test" objects shown here could be garbage-collected immediately after doTestSuite() executes. Iterators for doTestSuite() could, in theory, take two other forms: *. An object that returns the same sequence of test **sets** as successive iterations over twoTierTestSuite: i.e., -. the first call to nextPass returns ("error actions", (errorActionTest1, errorActionTest2, errorActionTest3, errorActionTest4,)) -. the second call to nextPass returns ("null actions", (nullActionTest1, nullActionTest2)) -. the third call to nextPass returns ("get actions", (getActionTest1, getActionTest2, getActionTest3, getActionTest4, getActionTest5, getActionTest6, getActionTest7, getActionTest8)) and so forth. This iterator-definition strategy allows the use of simpler loop control logic, but at a cost of a greater peak demand on program memory. *. An iterator that returns iterators embedded in lists: i.e., -. ("error actions", nextErrorCase) on the first call -. ("null actions", nextNullCase) on the second call -. ("error actions", nextGetCase) on the third call and so forth, where -. nextErrorCase is an iterator that dynamically generates errorActionTest1, errorActionTest2, errorActionTest3, and errorActionTest4 on successive calls; -. nextNullCase is an iterator that dynamically generates getActionTest1, getActionTest2, getActionTest3, getActionTest4, getActionTest5, getActionTest6, getActionTest7, and getActionTest8 on successive calls; and so forth. This iterator-definition strategy also allows the use of simpler loop control logic, but at a cost of more coding. 4. Additional Features 4.1 Using Nested Predicates to Fine-Tune Tests Any argument in any ADEPT test tuple that corresponds to a validation predicate may be structured instead as a sequence of predicates. This sequence of predicates is treated as a list of conditions that a result, or an effect, must satisfy for a test to succeed: 1st-level-pred has the form (pred1, pred2, pred3, ... predn) => pred <=> pred1 and pred2 and pred3 ... and predn If any of the elements in this (first-level) list of predicates, in turn, is structured as a sequence, then that (second-level) sequence of predicates is treated as a set of alternative requirements, any *one* of which the result, or effect, must satisfy: 2nd-level-pred, predk, has the form (predk1, predk2, predk3, ... predkm) => predk <=> predk1 or predk2 or predk3 or ... predkm If any of these second-level predicates--here, predk1 ... predkm--is, in turn, structured as a sequence, then this (third-level) sequence is treated as another list of predicates, *all* of which must be satisfied for that sequence to evaluate to true. In general, -. any sequence of predicates that occurs at an odd-numbered level in a hierarchy of test predicates--top, third, fifth, etc.--is treated as a conjunction of constraints; -. any sequence of predicates that occurs at an even-numbered level in a hierarcy of test predicates--second, fourth, sixth, etc.--is treated as a disjunction of constraints. This strategy for predicate interpretation, in short, mimics the classic conjunctive-normal-form-like strategy for building AND/OR decisions tree--data structures used to model choices in zero-sum, turn-based games like chess. The following three examples are provided to help clarify ADEPT's strategy for predicate tree intepretation: # Example 8a: redo of get action test 8 with one-level predicate list # # stricter check of output of Siblings.__str__()-- # try to validate content as well as result type # test8aTestStr =\ "Siblings('Ginny','Percy',mom='Molly').__str__() should yield a string" +\ "that names Ginny, Percy, and Molly" # conjunction of four conditions # test8aPredicateList =\ (isInstanceOf(str), containsRE('Ginny'), containsRE('Percy'), containsRE('Molly'), ) test8aMethodInstance = Siblings('Ginny','Percy',mom='Molly').__str__ getActionTest8a = (test8aTestStr, test8aPredicateList, test8aMethodInstance) # Example 8b: redo of get action test 8a with two-level predicate list # # slightly more forgiving than example 8a-- # will accept "Ginny" or "Virginia", \ # in addition to requiring "Percy" and "Molly" # # conjunction of four conditions, # the second of which is a disjunction # test8bPredicateList =\ (isInstanceOf(str), ( containsRE('Ginny'), containsRE('Virginia') ), containsRE('Percy'), containsRE('Molly'), ) getActionTest8b = (test8aTestStr, test8bPredicateList, test8aMethodInstance) # Example 8c: redo of get action test 8b with three-level predicate list # # tightens example 2: # will accept "Ginny" or "Virginia", but not both # # conjunction of four conditions, # the second of which is a disjunction of two conjunctions # test8cPredicateList =\ (isInstanceOf(str), (\ (\ containsRE('Ginny'), lacksRE('Virginia'), ), (\ containsRE('Virginia'), lacksRE('Ginny'), ), ), containsRE('Percy'), containsRE('Molly'), ) getActionTest8c = (test8aTestStr, test8cPredicateList, test8aMethodInstance) 4.2 doTestStack() Section 3 discusses strategies for coordinating test indexes across multiple calls to doTestSuite(). ADEPT provides another mechanism for coordinating the execution of multiple test cases: doTestStack(). A test stack is defined recursively as -. a test suite - that is, a test set, with a test suite label (i.e., "error actions", "null actions", "get actions", etc.); or -. a list of test stacks, with a test stack label (i.e., any label other than a known ADEPT test type) doTestStack(), when invoked on a test stack, does an in-order walk of that stack, -. associating test stack labels with indices, which are -. incremented and extended with new components as the walk progresses. For example, the code fragment # threeTierTestSuite as defined in Section 3 # doTestStack( ['three-level suite', threeTierTestSuite ] ) would, in its default mode of operation, assign the following indices and labels to the suite's tests: "1.1.1.1 validateName tests: error actions: validateName should reject non-string (3)" "1.1.1.2 validateName tests: error actions: validateName should reject 'R0n'" "1.1.2.1 validateName tests: null actions: validateName should accept 'Virginia'" "1.1.2.2 validateName tests: null actions: validateName should accept 'Art'" "1.2.1.1 Siblings class tests: error actions: construcing Siblings object with dad='Ar2r' should fail" ... doTestStack(), like doTestSuite(), can be used to continue earlier test runs. The function accepts -. an object of type TestLogger, which can be a TestLogger object from a previous test (optional parameter), and -. a prefix that is prepended to all indexes generated by the test (set by a keyword parameter, indexPrefix) 4.3 Test Masks The TestLogger class's getBanners() function accepts an optional mask tuple that controls those segments (fields) of a test banner that the calling code receives. A zero element in a mask tuple suppresses the inclusion of the corresponding banner segment. The tuple, if omitted, defaults to all 1's. Examples: assume that a TestLogger object, logger, associates the index 2.2.1 with a three-element banner of the form "2.1.1 Siblings class tests: error actions: dad=`Ar2r` is invalid" Then -. logger.getBanner((2,1,1),(0,0,0)) returns "" -. logger.getBanner((2,1,1),(0,0,1)) returns "dad=`Ar2r` is invalid" -. logger.getBanner((2,1,1),(0,1,0)) returns "error actions: " -. logger.getBanner((2,1,1),(0,1,1)) returns "error actions: dad=`Ar2r` is invalid" -. logger.getBanner((2,1,1),(1,0,0)) returns "Siblings class tests: " -. logger.getBanner((2,1,1),(1,0,1)) returns "Siblings class tests: dad=`Ar2r` is invalid" -. logger.getBanner((2,1,1),(1,1,0)) returns "Siblings class tests: error actions: " -. logger.getBanner((2,1,1),(1,1,1)) and logger.getBanner((2,1,1)) both return "Siblings class tests: error actions: dad=`Ar2r` is invalid" 5. The ADEPT Distribution ADEPT is currently distributed with three executable (.py) files: -. adept.py, which implements ADEPT proper, and which should imported into ADEPT-based test suites, using a statement like from adept import * -. adeptTestSuite.py, a test suite for ADEPT written in ADEPT; and -. adeptExamples.py, a test program that contains many of the examples in this document. All executables have been tested under Python 2.2. 6. ADEPT Design Notes Section 6, which focuses on the design of ADEPT's feature set, covers gaps in ADEPT's feature set (Section 6.1); the addition of new tests to ADEPT's test set (Section 6.2); and inherent limitations in ADEPT's design related to repr testing and cross-platform portability (Section 6.3). 6.1 Gaps in ADEPT's Feature Set 6.1.1 Assignment Statement Testing ADEPT currently supports no tests that check the operation of a statement like a = (1, 2, 3) ADEPT uses Python's built-in eval() primitive to test set actions. eval(), unfortunately, is an expression-evaluating function that raises a syntax error when asked to evaluate an expression containing an assignment operator. In order to add an assignment-checking test, logic would have to be created that emulates Python's logic for implementing assignment. This logic, in effect, would have to -. determine the scope in which the name on the statement's left-hand side should be defined; -. fetch a reference to the dictionary that corresponds to this scope; and -. (re)bind -. the key in this dictionary that corresponds to the name--here, for example, 'a'--to -. the value that is referenced--and perhaps created--by the evaluation of the statement's right-hand side--here, (1, 2, 3). Updates to components of mutable objects, however, can be tested like any other set action, provided that one can determine the name of the class method that implements the update. For example, effect of the following two statements-- a[1] = (1, 2, 3) a[:2] = (1, 2, 3) can be tested by using "set action" tests that name a.__setitem__ and a.__setslice__, respectively. 6.1.2 "Bare-Bones" Print Statement Testing Using ADEPT, the operation of a statement like if __flag__: print advisoryMessage can be tested by first writing the statement with an explicit output argument, like so-- if __flag__: print >>outputStream, advisoryMessage and then using logic like the following-- .... # specify output destination based on program's mode of operation # if __testEnabled__: outputStream = OutputStream() else: outputStream = sys.stdout --to control whether output is forwarded directly to the user, or sent to an intermediate "pseudo-file" object for capture and analysis. A Python pseudo-file object is any object with a write() statement, like the one below, which is used by adeptTestSuite.py to test the operation of ADEPT's StatusMessager class: class OutputStream(object): def __init__(self): self.__contents = '' def write(self, v): self.__contents += v def get(self): return self.__contents def clear(self): self.__contents = '' The strategy's main problem, of course, is that the code being tested must use the "chevron" (>>) form of the print statement--or be rewritten to use it. A somewhat more subtle problem involves the strategy's impact on repr method design. Pseudo-file objects, like instances of OutputStream, do not repr-ize properly--a concern discussed at length under repr stream testing, above. 6.1.3 Exception Arglist Testing ADEPT does not currently provide a mechanism for checking the contents of an exception object's argument list. Exception argument arg checking could be supported by extending the interpretation of the "error action" tuple's second, exception argument, so that -. a singleton argument would be interpreted as it is now, and -. a pair would be interpreted as a two-element value, -. whose first element represented an exception (as now), and -. whose second element represented an arglist-checking predicate. This feature, which was omitted to simplify the baseline suite, might be revisited in the future. Arglist checking would be useful in situations where -. a tester wants to ensure that a program's exception messages are sufficiently informative by checking for major keywords and keyphrases, or -. a code uses an exception's arglist to return parameters to a recovery routine, and the recovery routine, in turn, attempts to use these parameters to continue a computation. 6.1.4 Delete Property and Get Property Docstring Testing ADEPT currently fails to provide a mechanism for testing the operation of property delete and property docstring methods. The decision not to support these features, which was made, in part, to simplify the baseline suite, might be revisited in the future. In the meantime, "del prop error action", "del prop action", "doc prop error action", and "doc prop action" tests could be added to the suite by treating -. a "del prop error action" test as a set-prop-error-action-like test; -. a "del prop action" test as a set-prop-actions-like test whose "expected effect" string checked for the disappearance of an object; -. a "doc prop error action" test as a get-prop-error-action-like test; and -. a "doc prop action" test as a kind of get actions test whose result test checks whether the returned value is a string. See Section 6.2 for more on how to add tests to the suite. 6.1.5 Initializers and Finalizers ADEPT, unlike unittest, fails to support explicit initializer and finalizer components for test case. The decision not to support explicit initializers and finalizers was a design decision that balanced one kind of simplicity against another. Specifically, adding initializers and finalizers would *. allow code for defining and managing witness variables to be packaged into "set action", "get+set action", and "set prop action" tuples, *. at the expense of -. complicating the tuples themselves, and -. creating a need to emulate the effect of assignment statements within the routines that handle these actions. This is another decision that might be revisited. In the meantime, see Section 6.2 for more on how to update the suite. 6.2 Adding a New Tests to ADEPT: Working Notes The data structure testAttrs, which characterizes the nature of each test, is one of two starting points for adding a new test. testAttrs is structured as a two-level dictionary: *. the first level currently contains nine entries--one for each of the built-in test type tags. *. the second level currently enumerates each test type's characteristic features, including -. a string that describes a test type's purpose (used to create banners); -. the name of a method that applies the test; -. counts of the test's minimum and maximum number of arguments; and, -. for all optional arguments, a default value for that argument. A block of code following testAttr's definition verifies that every entry in testAttrs conforms various requirements that govern that entry's content. A new type of scalar, property, or method test can be added to ADEPT by *. creating a new test method for the required type of test, using one of the existing test methods as a starting point: i.e., -. testInvalidMethodCall -. testNullMethodCall -. testGetMethodCall -. testSetMethodCall -. testGetSetMethodCall -. testInvalidGetPropCall -. testInvalidSetPropCall -. testGetPropCall -. testSetPropCall *. defining a new tag for the type of test; and *. creating a new entry for this tag in the testAttrs dictionary that characterizes the test's operation. 6.3 Inherent Limitations 6.3.1 Repr Tests ADEPT's repr test algorithm, as noted earlier, is subject to the limitations of ADEPT object repr strings. Constructs like lambda expressions and pseudo- files cannot be repr-tested, at least not for classes that treat such objects as pass-by-value parameters, rather than pass-by-name parameters. The accuracy of repr testing is also damaged by the presence of user-defined objects whose repr strings or __eq__ methods characterize objects in incomplete or misleading ways. 6.3.2 Cross-Language Portability ADEPT's major asset--its use of eval() to compact test cases--may also defeat any attempt to port ADEPT to a language that fails to support eval(). This includes, for example, strongly typed languages, which, as a rule, don't support eval(), because of the havok this primitive wreaks with compile-time type checking. 7. Acknowledgments Thanks are extended to Dr. Stephen L. Scott of ORNL, for his trust, support, and encouragement, all of which were crucial to the completion of this work. Thanks are also extended to ORNL's Brian Luethke, Thomas Naughton, Rebecca Fahey, and Tom Barron for their making time to discuss ADEPT, and for their expressions of encouragement. Finally, a special word of thanks is extended to Tom Barron, for the idea that evolved into AND/OR predicate testing. Appendix A. Sibling Class Example Sections 2, 3, and 4 use the following code to illustrate ADEPT's operation: # ------------------------------------------------------------------- # Siblings class, with supporting method for validating data # --------------------------------------------------------- # Siblings class: # track siblings and their parents # validateName: # check that names are strings with an initial capital letter, # followed by letters and (possibly) hyphens # ------------------------------------------------------------------- def validateName(name): """ throws exception if name str fails to match prescribed pattern """ if not isinstance(name, str): raise TypeError if len(re.compile("[A-Z][a-zA-Z\-]*").match(name).group()) != len(name): raise ValueError class Siblings(object): """ keeps unordered list of siblings, and possibly their mom and dad """ # ### class properties ### # def getDad(self): if self.__dad is None: raise AttributeError, "dad not specified" return self.__dad def setDad(self, dadName): validateName(dadName) self.__dad = dadName dad = property(getDad, setDad) def getMom(self): if self.__mom is None: raise AttributeError, "mom not specified" return self.__mom def setMom(self, momName): validateName(momName) self.__mom = momName mom = property(getMom, setMom) # ### class infrastructure methods ### # def __init__(self, *args, **kwds): # # ---- first, initialize class to the empty object ---- # (a best practice: allows __repr__ to execute successfully # executing __init__ generates an error) # self.__dad = self.__mom = None self.__siblings = [] # # ---- then, validate arguments, and init class ---- # if kwds.get('mom', None) is not None: validateName(kwds['mom']) if kwds.get('dad', None) is not None: validateName(kwds['dad']) for name in args: validateName(name) self.__siblings.extend(list(args)) self.__dad = kwds.get('dad', None) self.__mom = kwds.get('mom', None) def __str__(self): def dadStr(): if self.__dad is None: return "not specified" else: return `self.__dad` def momStr(): if self.__mom is None: return "not specified" else: return `self.__mom` return "" def __repr__(self): if len(self.__siblings) > 0: siblings = self.__siblings siblings.sort() siblingsString =\ string.join(map(lambda x: '"' + x + '"', siblings), ',') + "," else: siblingsString = "" return \ "Siblings(" + siblingsString +\ "dad=" + `self.__dad` + "," +\ "mom=" + `self.__mom` + ")" def __eq__(self, other): if not isinstance(other, Siblings): return 0 selfSiblings, otherSiblings = self.__siblings, other.__siblings selfSiblings.sort() otherSiblings.sort() return (selfSiblings == otherSiblings) and \ (self.__mom == other.__mom) and (self.__dad == other.__dad) def __ne__(self, other): return 1 - self.__eq__(other) # ### class-specific methods ### # def count(self): return len(self.__siblings) def contains(self, name): return name in self.__siblings def add(self, name): validateName(name) if name not in self.__siblings: self.__siblings.append(name) def drop(self, name): self.__siblings.remove(name) def replace(self, oldName, newName): validateName(newName) self.__siblings.remove(oldName) self.__siblings.add(newName) def addAndNotify(self, name): """ returns 1 iff sibling was not missing before add """ result = name not in self.__siblings self.add(name) return result Appendix B. TestCaseIteratorGenerator Example class TestCaseIteratorGenerator(object): class TestCaseIterator(object): def __init__(self): self.nextTestCase = -1 # # iterator protocol, part 1: # define a function named __iter__ that returns the iterator itself # def __iter__(self): return self # # iterator protocol, part 2: # define a function named next() that returns the next element in the # iteration, or StopIteration if the iteration is complete # def next(self): self.nextTestCase += 1 if self.nextTestCase < len(oneTierTestSuite): return oneTierTestSuite[self.nextTestCase] else: raise StopIteration def __init__(self): pass # # iterator protocol, part 3: # the iterator generator must support a function, __iter__(), # that returns a new iterator # def __iter__(self): return TestCaseIteratorGenerator.TestCaseIterator()