Labour, symbols, and free: the gate to digital-based music-making

Posted by Jim DeLaHunt on 05 Apr 2017 | Tagged as: Keyboard Philharmonic, culture

(Background: I was asked recently for a writing sample, and I took the opportunity to restate, more concisely, what I’m trying to do with Keyboard Philharmonic.)

Musicians performing classical music and opera, and teachers and students of this music, are on the cusp of a transformation from printed music scores to digital scores. This will be as significant as the shift of text communication from printed books and magazines, to web articles, blogs, emails, and tweets.

I believe a particular model is the right next step. I call it, “Labour, symbols, and free”. It is a policy package for a music score transcription effort. It fills a gap in the present situation, and opens a gate to move forward. I will also describe the strategic context. Continue Reading »

Open Data Day 2017, Team Meta, and a Prize!

Posted by Jim DeLaHunt on 31 Mar 2017 | Tagged as: Vancouver, government, meetings and conferences, web technology

Open Data Day was celebrated here on Sunday, 4. March 2017. The Open Data Society of B.C. sponsored a buzzing, successful hackathon, with participants from several communities in the Lower Mainland, Vancouver Island, and even Washington State.

I plunged back into my continuing project for Vancouver Open Data Day, to make a language census of Vancouver’s Open Data Catalogue. You can check out our Team Meta #VODay hackathon report as submitted via github. I’ve summarised it below.  I was very pleased to be awarded the City of Vancouver Focus Challenge Prize for the work we accomplished that day. Continue Reading »

Python multi-line doctests, and “Got nothing” message

Posted by Jim DeLaHunt on 31 Jan 2017 | Tagged as: Python, robobait, software engineering

Recently I was writing a Python-language tool, and some of my doctests (text fixtures, within module comments) were failing. When I tried to import the StringIO module in my test, I got a quite annoying message, “Got nothing”, and the test didn’t work as I wanted. I asked StackOverflow. User wim there gave me a crucial insight, but didn’t explain the underlying cause of my problem. I read the doctest code, and came up with an explanation that satisfied me. I am posting it here, as an aid to others. The gist of the insight: What looks like a multi-line doctest fixture is in fact a succession of single-line doctest “Examples”, some which return no useful result but which set up state for later Examples. Each single-line Example should each have a >>> prefix, not a prefix. But, there are Examples that require the prefix. The difference lies in Python’s definition of an Interactive Statement.

The Question

I posted a question much like this to StackOverflow:

Why is importing a module breaking my doctest (Python 2.7)?

I tried to use a StringIO instance in a doctest in my class, in a Python 2.7 program. Instead of getting any output from the test, I get a response, “Got nothing”.

This simplified test case demonstrates the error:

#!/usr/bin/env python2.7
# encoding: utf-8

class Dummy(object):
    ”’Dummy: demonstrates a doctest problem
    >>> from StringIO import StringIO
    … s = StringIO()
    … print(”s is created”)
    s is created
    ”’

if __name__ == “__main__”:
    import doctest
    doctest.testmod()

Expected behaviour: test passes.

Observed behaviour: test fails, with output like this:

% ./src/doctest_fail.py
**********************************************************************
File "./src/doctest_fail.py", line 7, in __main__.Dummy
Failed example:
    from StringIO import StringIO
    s = StringIO()
    print(”s is created”)
Expected:
    s is created
Got nothing
**********************************************************************
1 items had failures:
    1 of 1 in __main__.Dummy
***Test Failed*** 1 failures.

Why is this doctest failing? What change to I need to make in order to be able to use StringIO-like functionality (a literal string with a file interface) in my doctests?

(I had originally suspected the StringIO module of being part of the problem. My original question title was, “Why is use of StringIO breaking my doctest (Python 2.7)”. When I realised that suspicion was incorrect, I edited the question on StackOverflow.)

The Answer

StackOverflow expert wim was quick with the crucial insight: “It’s the continuation line syntax () that is confusing doctest parser.” Wim then rewrote my example so that it functioned correctly. Excellent!  Thank you, wim.

The Explanation

I wasn’t satisfied, however. I know from  didn’t explain the underlying cause of my problem. I read the doctest code, and came up with an explanation that satisfied me. Below is an improved version of the answer I posted to StackOverflow at the time.

The example fails, because it uses the PS2 syntax (...) instead of PS1 syntax (>>>) in front of separate simple statements.

Change ... to >>>:


#!/usr/bin/env python2.7
# encoding: utf-8

class Dummy(object):
    ”’Dummy: demonstrates a doctest problem
    >>> from StringIO import StringIO
    >>> s = StringIO()
    >>> print(”s is created”)
    s is created
    ”’

if __name__ == “__main__”:
    import doctest
    doctest.testmod()

Now the corrected example, renamed doctest_pass.py, runs with no errors. It produces no output, meaning that all tests pass:

% ./src/doctest_pass.py

Why is the >>> syntax correct? The Python Library Reference for doctest, 25.2.3.2. How are Docstring Examples Recognized? should be the place to find the answer, but it isn’t terribly clear about this syntax.

Doctest scans through a docstring, looking for “Examples”. Where it sees the PS1 string >>>, it takes everything from there to the end of the line as an Example. It also appends any following lines which begin with the PS2 string ... to the Example (See: _EXAMPLE_RE in class doctest.DocTestParser, lines 584-595). It takes the subsequent lines, until the next blank line or line starting with the PS1 string, as the Wanted Output.

Doctest compiles each Example as a Python “interactive statement”, using the compile() built-in function in an exec statement (See: doctest.DocTestRunner.__run(), lines 1314-1315).

An “interactive statement” is a statement list ending with a newline, or a Compound Statement. A compound statement, e.g. an if or try statement, “in general, […spans] multiple lines, although in simple incarnations a whole compound statement may be contained in one line.” Here is a multi-line compound statement:

if 1 > 0:
    print(”As expected”)
else:
    print(”Should not happen”)

A statement list is one or more simple statements on a single line, separated by semicolons.


from StringIO import StringIO
s = StringIO(); print("s is created")

So, the question’s doctest failed because it contained one Example with three simple statements, and no semicolon separators. Changing the PS2 strings to PS1 strings succeeds, because it turns the docstring into a sequence of three Examples, each with one simple statement. Although these three lines work together to set up one test of one piece of functionality, they are not a single test fixture. They are three tests, two of which set up state but do not really test the main functionality.

By the way, you can see the number of Examples which doctest recognises by using the -v flag. Note that it says, “3 tests in __main__.Dummy“. One might think of the three lines as one test unit, but doctest sees three Examples. The first two Examples have no expected output. When the Example executes and generates no output, that counts as a “pass”.


% ./src/doctest_pass.py -v
Trying:
    from StringIO import StringIO
Expecting nothing
ok
Trying:
    s = StringIO()
Expecting nothing
ok
Trying:
    print(”s is created”)
Expecting:
    s is created
ok
1 items had no tests:
    __main__
1 items passed all tests:
    3 tests in __main__.Dummy
3 tests in 2 items.
3 passed and 0 failed.
Test passed.

Within a single docstring, the Examples are executed in sequence. State changes from each Example are preserved for the following Examples in the same docstring. Thus the import statement defines a module name, the s = assignment statement uses that module name and defines a variable name, and so on. The doctest documentation, 25.2.3.3. What’s the Execution Context?, obliquely discloses this when it says, “examples can freely use … names defined earlier in the docstring being run.”

The preceding sentence in that section, “each time doctest finds a docstring to test, it uses a shallow copy of M’s globals, so that … one test in M can’t leave behind crumbs that accidentally allow another test to work”, is a bit misleading. It is true that one test in M can’t affect a test in a different docstring. However, within a single docstring, an earlier test will certainly leave behind crumbs, which might well affect later tests.

But there is an example doctest, in the Python Library Reference for doctest, 25.2.3.2. How are Docstring Examples Recognized?, which uses ... syntax. Why doesn’t it use >>> syntax? Because that example consists of an if statement, which is a compound statement on multiple lines. As such, its second and subsequent lines are marked with the PS2 strings.  It’s unfortunate that this is the only example of a multi-line fixture in the documentation, because it can be misleading about when to use PS1 instead of PS2 strings.

Submission to the Committee on Electoral Reform (ERRE)

Posted by Jim DeLaHunt on 08 Oct 2016 | Tagged as: Canada, Democratic Reform, government, politics

I just submitted a brief to Canada’s Special Parliamentary Committee on Electoral Reform, or ERRE. I expect it will show up on their docket in due course, but you can read it here first. There are many briefs, some very good, but this one is mine. Continue Reading »

Celebration of the Life of Elizabeth Allwyn (1927-2016)

Posted by Jim DeLaHunt on 26 Sep 2016 | Tagged as: personal

Elizabeth B. Allwyn 1927-2016Our mother, Elizabeth Boise Allwyn, passed away on September 18 from illness. The family invites her friends and neighbours to a celebration of her life.

September 28, 2016 (Weds) 2:00-2:45pm

Dining Room, the Willows

3115 Squalicum Parkway, Bellingham [map]

As our mother wished, this will be a simple affair. Family members and friends will speak on different aspects of Elizabeth and what she brought to the world. There will be an open space for participants to share their own feelings and memories.

A short reception will follow.

The family asks that, in lieu of flowers, donations be made to Lydia Place (lydiaplace.org) and to the National Multiple Sclerosis Society, Greater Illinois Chapter (nationalmssociety.org/Chapters/ILD).

Directions:  The Willows is near St Joseph’s hospital and the Sunset Drive exit of I-5. By bus, Whatcom Transit Authority routes 3 and 4 stop right in front of the building. By car, take I-5 to exit 255 (Sunset Drive). Head west on Sunset Drive; after 0.3 mi, right (north) onto Ellis Street; after 0.2 mi, first right onto Squalicum Parkway (St Joseph’s Hospital is straight ahead); after 0.1 mi, at a sign for The Willows, left onto Levin Lane, then forward until you enter the Willows parking lot. There is limited parking in the Willows. You might find it easier to park in the clinic parking lot to the left just before the Willows driveway. Continue Reading »

Town Hall on Electoral Reform, Dr Hedy Fry MP in Vancouver Centre, 15 Aug 2016

Posted by Jim DeLaHunt on 31 Aug 2016 | Tagged as: Canada, Vancouver, government, meetings and conferences, politics

I favour electoral reform. I am a newly-minted Canadian, who deeply hopes my first vote for Parliament will not be conducted under the current, archaic, unfair First-Past-the-Post system. So, when my MP convened a Town Hall meeting on electoral reform, I made a point of attending. Here are some notes on the event. I hope they are helpful documentation for other democratic reform advocates.

Continue Reading »

Resetting GnuCash’s import transaction matching

Posted by Jim DeLaHunt on 29 Jul 2016 | Tagged as: robobait

I use the GnuCash free software accounting software. Like many accounting tools, it can import bank or credit card transactions, and has a way to learn the correct mapping from transaction data to my own account structure. And, sometimes the tool gets the mapping wrong, and needs to be reset.  Here is how I was able to perform this reset. I post it here in the hopes it will help others. Fasten your seatbelt, it involves some awfully technical command-line tools, including XSLT processing. Continue Reading »

Requirements for comparing digital scores

Posted by Jim DeLaHunt on 30 Jun 2016 | Tagged as: Keyboard Philharmonic, culture, meetings and conferences, music

Back in May, as part of the Music Encoding 2016 conference in Montreal, we had a discussion about comparing digital scores. Just as you can compare text files, and get a concise statement of differences, we brainstormed about requirements for comparing music scores at the notation level. This blog post is a record of that discussion.

Continue Reading »

Encoding Music at Music Encoding Conference 2016

Posted by Jim DeLaHunt on 31 May 2016 | Tagged as: Canada, Keyboard Philharmonic, meetings and conferences, music

A couple of weeks ago the Music Encoding Conference 2016 was held at McGill University, Montréal, Canada. I attended on behalf of the Keyboard Philharmonic project. I was like a kid in a candy store: so many people with so much experience in representing music notation digitally, so many interesting talks, so much friendliness. I also had the temerity to hold, despite my first-time status, a workshop on the first day of the conference: “Encoding Music at Music Encoding”, where we would follow the Keyboard Philharmonic process to encode a short score. The goal was to release it to the public domain by the end of the conference. Here is how we did.

Continue Reading »

Recruiting people, and structuring their work

Posted by Jim DeLaHunt on 30 Apr 2016 | Tagged as: Keyboard Philharmonic, culture, music

The Keyboard Philharmonic overview mentions that one focus of the project is to be “a vehicle for recruiting people and structuring their work for useful results”. There are reasons why this focus is important.  Continue Reading »

Next »