Generative Tests
Both nose and py.test allow you to write generative tests. These are basically generators that yield the actual test functions and their arguments. The test runners will then test all the yielded functions with their arguments.
However I'm left wondering why this is better then just iterating in the test function and doing many asserts? Let me demonstrate with the py.test example:
def test_generative():
for x in (42,17,49):
yield check, x
def check(arg):
assert arg % 7 == 0 # second generated tests fails!
Why is that code better then:
def test_something():
for x in (42,17,49):
assert x % 7 == 0
I know that it could possibly tell you slightly more, but in general I keep testing until all of my tests succeed so I don't really mind if one test includes more then one assert statement. I'll just keep fixing things till the test passes.
What is the motivation for generative tests?
As an aside (and the inspiration for this post thanks to Holger Krekel talking about py.test), PyCon UK was great. I was considering a post titled "Resolver Systems Ltd is Evil since they provided the Sunday morning hangovers, but thought that was a bit too sensationalist.
11 comments:
Robert Kern said...
Knowing the variety of ways in which your code fails (and where it doesn’t fail) helps you find the bug.
René Dudfield said...
Keeping the tests separate is good for a number of reasons...
-* Isolate crashes.
-* Parallelism, dividing tests up between many workers.
-* Measuring each test individually.
Michael Foord said...
There was much about py.test that was inspiring - but I had exactly the same reaction to you about the generator tests.
Test modularity is great, but why not call 'check' directly?
I can't see any tangible benefit from making a test into a generator.
Mike Pirnat said...
Generative tests are clearly beneficial to those of us who're now struggling with dot addiction. ;-)
Ορέστης said...
My guess is that when using the generative test, you can wrap the call to 'check' with something like:
try:
check(*args)
except:
print 'error for', *args
only once in your test framework, whereas the other way round you would have to add the reporting in every check.
Not a very big argument, but still a nice pattern if you do it a lot.
Michael Foord said...
I don't see any benefit over the following pattern instead:
def check(x):
# do assert
# with error handling
def test_generative():
for x in (42,17,49):
check(x)
Ορέστης said...
Your check method includes both the assertion and the error checking. My check method contains only the assertion. That leads to cleaner tests. Let's stop spamming the comments of this blog. :)
Unknown said...
Heh, that's not spamming!
But when do you want to do error handling for your assert statments in unittests? I don't think I've ever done something like that. Usually the test runner shows enough info and even if it doesn't it tend to add print statments before the asserts and rely on py.test or nose to catch the output.
Unknown said...
In your example the generative test case will test x when it is 42, 17, and 49. In the other case only 42 and 17 will be tested as 17 will fail the assertion. Often it can be helpful to see all the failing test cases as it can provide better insights as to what really needs to be fix. Sometimes the first case that fails might not be a good indicated of the issue at hand. This happens all to frequent when you're not testing the normal or "happy" paths.
Michael Foord said...
Ok - that's actually a valid use case. :-)
Every generated test is a separate test case and even if one fails they will all be executed.
Dave Kirby said...
One advantage is if the test framework creates unique names for each test (nose does, I don't know about py.test) then you can run a single test by naming it on the command line.
With the non-generative test you would have to edit the test to comment out all the cases you were not interested in.
New comments are not allowed.