Recently I was writing a Python-language tool, and some of my doctests (text fixtures, within module comments) were failing. When I tried to import the StringIO module in my test, I got a quite annoying message, “Got nothing”, and the test didn’t work as I wanted. I asked StackOverflow. User wim there gave me a crucial insight, but didn’t explain the underlying cause of my problem. I read the doctest code, and came up with an explanation that satisfied me. I am posting it here, as an aid to others. The gist of the insight: What looks like a multi-line doctest fixture is in fact a succession of single-line doctest “Examples”, some which return no useful result but which set up state for later Examples. Each single-line Example should each have a >>> prefix, not a ... prefix. But, there are Examples that require the ... prefix. The difference lies in Python’s definition of an Interactive Statement.

The Question

I posted a question much like this to StackOverflow:

Why is importing a module breaking my doctest (Python 2.7)?

I tried to use a StringIO instance in a doctest in my class, in a Python 2.7 program. Instead of getting any output from the test, I get a response, “Got nothing”.

This simplified test case demonstrates the error:

#!/usr/bin/env python2.7
# encoding: utf-8

class Dummy(object):
    '''Dummy: demonstrates a doctest problem
    >>> from StringIO import StringIO
    ... s = StringIO()
    ... print("s is created")
    s is created
    '''

if __name__ == "__main__":
    import doctest
    doctest.testmod()

Expected behaviour: test passes.

Observed behaviour: test fails, with output like this:

% ./src/doctest_fail.py
**********************************************************************
File "./src/doctest_fail.py", line 7, in __main__.Dummy
Failed example:
    from StringIO import StringIO
    s = StringIO()
    print("s is created")
Expected:
    s is created
Got nothing
**********************************************************************
1 items had failures:
    1 of 1 in __main__.Dummy
***Test Failed*** 1 failures.

Why is this doctest failing? What change to I need to make in order to be able to use StringIO-like functionality (a literal string with a file interface) in my doctests?

(I had originally suspected the StringIO module of being part of the problem. My original question title was, “Why is use of StringIO breaking my doctest (Python 2.7)”. When I realised that suspicion was incorrect, I edited the question on StackOverflow.)

The Answer

StackOverflow expert wim was quick with the crucial insight: “It’s the continuation line syntax (...) that is confusing doctest parser.” Wim then rewrote my example so that it functioned correctly. Excellent!  Thank you, wim.

The Explanation

I wasn’t satisfied, however. I know from  didn’t explain the underlying cause of my problem. I read the doctest code, and came up with an explanation that satisfied me. Below is an improved version of the answer I posted to StackOverflow at the time.

The example fails, because it uses the PS2 syntax (...) instead of PS1 syntax (>>>) in front of separate simple statements.

Change ... to >>>:


#!/usr/bin/env python2.7
# encoding: utf-8

class Dummy(object):
    '''Dummy: demonstrates a doctest problem
    >>> from StringIO import StringIO
    >>> s = StringIO()
    >>> print("s is created")
    s is created
    '''

if __name__ == "__main__":
    import doctest
    doctest.testmod()

Now the corrected example, renamed doctest_pass.py, runs with no errors. It produces no output, meaning that all tests pass:

% ./src/doctest_pass.py

Why is the >>> syntax correct? The Python Library Reference for doctest, 25.2.3.2. How are Docstring Examples Recognized? should be the place to find the answer, but it isn’t terribly clear about this syntax.

Doctest scans through a docstring, looking for “Examples”. Where it sees the PS1 string >>>, it takes everything from there to the end of the line as an Example. It also appends any following lines which begin with the PS2 string ... to the Example (See: _EXAMPLE_RE in class doctest.DocTestParser, lines 584-595). It takes the subsequent lines, until the next blank line or line starting with the PS1 string, as the Wanted Output.

Doctest compiles each Example as a Python “interactive statement”, using the compile() built-in function in an exec statement (See: doctest.DocTestRunner.__run(), lines 1314-1315).

An “interactive statement” is a statement list ending with a newline, or a Compound Statement. A compound statement, e.g. an if or try statement, “in general, […spans] multiple lines, although in simple incarnations a whole compound statement may be contained in one line.” Here is a multi-line compound statement:

if 1 > 0:
    print("As expected")
else:
    print("Should not happen")

A statement list is one or more simple statements on a single line, separated by semicolons.


from StringIO import StringIO
s = StringIO(); print("s is created")

So, the question’s doctest failed because it contained one Example with three simple statements, and no semicolon separators. Changing the PS2 strings to PS1 strings succeeds, because it turns the docstring into a sequence of three Examples, each with one simple statement. Although these three lines work together to set up one test of one piece of functionality, they are not a single test fixture. They are three tests, two of which set up state but do not really test the main functionality.

By the way, you can see the number of Examples which doctest recognises by using the -v flag. Note that it says, “3 tests in __main__.Dummy“. One might think of the three lines as one test unit, but doctest sees three Examples. The first two Examples have no expected output. When the Example executes and generates no output, that counts as a “pass”.


% ./src/doctest_pass.py -v
Trying:
    from StringIO import StringIO
Expecting nothing
ok
Trying:
    s = StringIO()
Expecting nothing
ok
Trying:
    print("s is created")
Expecting:
    s is created
ok
1 items had no tests:
    __main__
1 items passed all tests:
    3 tests in __main__.Dummy
3 tests in 2 items.
3 passed and 0 failed.
Test passed.

Within a single docstring, the Examples are executed in sequence. State changes from each Example are preserved for the following Examples in the same docstring. Thus the import statement defines a module name, the s = assignment statement uses that module name and defines a variable name, and so on. The doctest documentation, 25.2.3.3. What’s the Execution Context?, obliquely discloses this when it says, “examples can freely use … names defined earlier in the docstring being run.”

The preceding sentence in that section, “each time doctest finds a docstring to test, it uses a shallow copy of M’s globals, so that … one test in M can’t leave behind crumbs that accidentally allow another test to work”, is a bit misleading. It is true that one test in M can’t affect a test in a different docstring. However, within a single docstring, an earlier test will certainly leave behind crumbs, which might well affect later tests.

But there is an example doctest, in the Python Library Reference for doctest, 25.2.3.2. How are Docstring Examples Recognized?, which uses ... syntax. Why doesn’t it use >>> syntax? Because that example consists of an if statement, which is a compound statement on multiple lines. As such, its second and subsequent lines are marked with the PS2 strings.  It’s unfortunate that this is the only example of a multi-line fixture in the documentation, because it can be misleading about when to use PS1 instead of PS2 strings.

1 Comment »