robobait

Archived Posts from this Category

LiClipse (for Mac) includes its own copy of the JRE

Posted by Jim DeLaHunt on 10 Dec 2017 | Tagged as: robobait, software engineering

LiClipse is the developers tool I use for writing Python code. Based on the Eclipse IDE, it accepts numerous plugins to support other programming languages like Java and C, and related tools, such as the Git version control system. Eclipse is mostly Java language code, and it runs on a JRE (Java Runtime Environment). Last month, I wanted to contribute code to a git repository which I accessed via HTTPS. That worked more smoothly if I could put an SSL certificate into the JRE, and I’ll skip the details of why for now.

So I looked up the Java Home of the JRE installed on my Mac OS X laptop (short answer: it’s the path output by running /usr/libexec/java_home). I installed the SSL certificate there. It did not work. That was a sign that LiClipse did not use that JRE. Did it perhaps include its own JRE?  After some investigation, I found out the answer: yes!

Here’s the explanation. I hope this helps others. Continue Reading »

When I run “ffmpeg” in the background, how do I prevent “suspended (tty output)”?

Posted by Jim DeLaHunt on 04 Nov 2017 | Tagged as: robobait, software engineering

I recently had a problem, “When I run ffmpeg in the background, how do I prevent suspended (tty output)?”. I solved it. Here is my solution, in the hopes that it will help others seeing the same problem.

I have a sh script which calls ffmpeg on several files. When I try to run this script in the background, redirecting output to a file, the job starts but then immediately suspends:

% bin/mp3convert.sh path/a/b &> ~/tmp/log.txt &
[1] 93352
% [1]  + suspended (tty output)  bin/mp3convert.sh path/a/b &>

Continue Reading »

CD storage and the 25cm x 15cm box

Posted by Jim DeLaHunt on 30 Sep 2017 | Tagged as: culture, music, robobait

I guess I’m exactly the right age to have this problem: hundreds of liner notes from CD albums, stripped from their jewel cases and also CD boxed-sets and many different CD gatefold cases. If I were any younger, I’d be subscribing to some music streaming service, or downloading pirated albums as MP3 files. If I were any older, I’d be building elaborate shelf units to store the hundreds of intact jewel cases, and keeping a multi-disc CD player running to play the music.  But here I am, old enough to buy CDs as a way to pay the artists for their work, but young enough to want to rip the CDs into music files, upload them to a file server, and play them via computer or from my smartphone.

All of this leaves me with a problem: having put the ripped CDs onto nice compact 100-disc spindles, where do I put the liner notes and booklets so that I have access to them if I want ?  I can throw out the regular or the 2-disc jewel cases, because those are generic. But the artwork is indispensable.  On the other hand, I don’t need to keep it out on a open shelf to browse. It’s fine for me to put it away, and retrieve it only when I need it.  And now I am happy to report a solution: a box, of just the right size to hold liner notes or CD box-sets or gatefold cases efficiently, easily available from shipping materials suppliers, and very affordably priced. I post this in the hopes of helping some else who is trying to solve the same problem. Continue Reading »

Python multi-line doctests, and “Got nothing” message

Posted by Jim DeLaHunt on 31 Jan 2017 | Tagged as: Python, robobait, software engineering

Recently I was writing a Python-language tool, and some of my doctests (text fixtures, within module comments) were failing. When I tried to import the StringIO module in my test, I got a quite annoying message, “Got nothing”, and the test didn’t work as I wanted. I asked StackOverflow. User wim there gave me a crucial insight, but didn’t explain the underlying cause of my problem. I read the doctest code, and came up with an explanation that satisfied me. I am posting it here, as an aid to others. The gist of the insight: What looks like a multi-line doctest fixture is in fact a succession of single-line doctest “Examples”, some which return no useful result but which set up state for later Examples. Each single-line Example should each have a >>> prefix, not a prefix. But, there are Examples that require the prefix. The difference lies in Python’s definition of an Interactive Statement.

The Question

I posted a question much like this to StackOverflow:

Why is importing a module breaking my doctest (Python 2.7)?

I tried to use a StringIO instance in a doctest in my class, in a Python 2.7 program. Instead of getting any output from the test, I get a response, “Got nothing”.

This simplified test case demonstrates the error:

#!/usr/bin/env python2.7
# encoding: utf-8

class Dummy(object):
    ”’Dummy: demonstrates a doctest problem
    >>> from StringIO import StringIO
    … s = StringIO()
    … print(”s is created”)
    s is created
    ”’

if __name__ == “__main__”:
    import doctest
    doctest.testmod()

Expected behaviour: test passes.

Observed behaviour: test fails, with output like this:

% ./src/doctest_fail.py
**********************************************************************
File "./src/doctest_fail.py", line 7, in __main__.Dummy
Failed example:
    from StringIO import StringIO
    s = StringIO()
    print(”s is created”)
Expected:
    s is created
Got nothing
**********************************************************************
1 items had failures:
    1 of 1 in __main__.Dummy
***Test Failed*** 1 failures.

Why is this doctest failing? What change to I need to make in order to be able to use StringIO-like functionality (a literal string with a file interface) in my doctests?

(I had originally suspected the StringIO module of being part of the problem. My original question title was, “Why is use of StringIO breaking my doctest (Python 2.7)”. When I realised that suspicion was incorrect, I edited the question on StackOverflow.)

The Answer

StackOverflow expert wim was quick with the crucial insight: “It’s the continuation line syntax () that is confusing doctest parser.” Wim then rewrote my example so that it functioned correctly. Excellent!  Thank you, wim.

The Explanation

I wasn’t satisfied, however. I know from  didn’t explain the underlying cause of my problem. I read the doctest code, and came up with an explanation that satisfied me. Below is an improved version of the answer I posted to StackOverflow at the time.

The example fails, because it uses the PS2 syntax (...) instead of PS1 syntax (>>>) in front of separate simple statements.

Change ... to >>>:


#!/usr/bin/env python2.7
# encoding: utf-8

class Dummy(object):
    ”’Dummy: demonstrates a doctest problem
    >>> from StringIO import StringIO
    >>> s = StringIO()
    >>> print(”s is created”)
    s is created
    ”’

if __name__ == “__main__”:
    import doctest
    doctest.testmod()

Now the corrected example, renamed doctest_pass.py, runs with no errors. It produces no output, meaning that all tests pass:

% ./src/doctest_pass.py

Why is the >>> syntax correct? The Python Library Reference for doctest, 25.2.3.2. How are Docstring Examples Recognized? should be the place to find the answer, but it isn’t terribly clear about this syntax.

Doctest scans through a docstring, looking for “Examples”. Where it sees the PS1 string >>>, it takes everything from there to the end of the line as an Example. It also appends any following lines which begin with the PS2 string ... to the Example (See: _EXAMPLE_RE in class doctest.DocTestParser, lines 584-595). It takes the subsequent lines, until the next blank line or line starting with the PS1 string, as the Wanted Output.

Doctest compiles each Example as a Python “interactive statement”, using the compile() built-in function in an exec statement (See: doctest.DocTestRunner.__run(), lines 1314-1315).

An “interactive statement” is a statement list ending with a newline, or a Compound Statement. A compound statement, e.g. an if or try statement, “in general, […spans] multiple lines, although in simple incarnations a whole compound statement may be contained in one line.” Here is a multi-line compound statement:

if 1 > 0:
    print(”As expected”)
else:
    print(”Should not happen”)

A statement list is one or more simple statements on a single line, separated by semicolons.


from StringIO import StringIO
s = StringIO(); print("s is created")

So, the question’s doctest failed because it contained one Example with three simple statements, and no semicolon separators. Changing the PS2 strings to PS1 strings succeeds, because it turns the docstring into a sequence of three Examples, each with one simple statement. Although these three lines work together to set up one test of one piece of functionality, they are not a single test fixture. They are three tests, two of which set up state but do not really test the main functionality.

By the way, you can see the number of Examples which doctest recognises by using the -v flag. Note that it says, “3 tests in __main__.Dummy“. One might think of the three lines as one test unit, but doctest sees three Examples. The first two Examples have no expected output. When the Example executes and generates no output, that counts as a “pass”.


% ./src/doctest_pass.py -v
Trying:
    from StringIO import StringIO
Expecting nothing
ok
Trying:
    s = StringIO()
Expecting nothing
ok
Trying:
    print(”s is created”)
Expecting:
    s is created
ok
1 items had no tests:
    __main__
1 items passed all tests:
    3 tests in __main__.Dummy
3 tests in 2 items.
3 passed and 0 failed.
Test passed.

Within a single docstring, the Examples are executed in sequence. State changes from each Example are preserved for the following Examples in the same docstring. Thus the import statement defines a module name, the s = assignment statement uses that module name and defines a variable name, and so on. The doctest documentation, 25.2.3.3. What’s the Execution Context?, obliquely discloses this when it says, “examples can freely use … names defined earlier in the docstring being run.”

The preceding sentence in that section, “each time doctest finds a docstring to test, it uses a shallow copy of M’s globals, so that … one test in M can’t leave behind crumbs that accidentally allow another test to work”, is a bit misleading. It is true that one test in M can’t affect a test in a different docstring. However, within a single docstring, an earlier test will certainly leave behind crumbs, which might well affect later tests.

But there is an example doctest, in the Python Library Reference for doctest, 25.2.3.2. How are Docstring Examples Recognized?, which uses ... syntax. Why doesn’t it use >>> syntax? Because that example consists of an if statement, which is a compound statement on multiple lines. As such, its second and subsequent lines are marked with the PS2 strings.  It’s unfortunate that this is the only example of a multi-line fixture in the documentation, because it can be misleading about when to use PS1 instead of PS2 strings.

Resetting GnuCash’s import transaction matching

Posted by Jim DeLaHunt on 29 Jul 2016 | Tagged as: robobait

I use the GnuCash free software accounting software. Like many accounting tools, it can import bank or credit card transactions, and has a way to learn the correct mapping from transaction data to my own account structure. And, sometimes the tool gets the mapping wrong, and needs to be reset.  Here is how I was able to perform this reset. I post it here in the hopes it will help others. Fasten your seatbelt, it involves some awfully technical command-line tools, including XSLT processing. Continue Reading »

Resolving error “Error: Unable to find ‘python.swg’”, when installing pycdio on Mac OS X

Posted by Jim DeLaHunt on 25 Jan 2015 | Tagged as: Python, robobait

I just resolved a problem installing a Python module pycdio on my Mac OS X 10.10.1 “Yosemite” operating system. The error message was obscure: “Error: Unable to find ‘python.swg’”, and “Error: Unable to find ‘typemaps.i’”. The solution involved something non-obvious about how Mac Ports handles swig. Here’s my notes, in hopes of helping others seeing this error.

Continue Reading »

How to extract URLs with Apache OpenOffice, from formatted text and HTML tables

Posted by Jim DeLaHunt on 31 Mar 2014 | Tagged as: robobait, software engineering

I use and value a good spreadsheet application the way chefs use and value good knives. I have countless occasions to do ad-hoc data processing and conversion, and I tend to turn to spreadsheets even more often I turn to a good text editor. I know a lot of ways to get the job done with spreadsheets. But recently I learned a new trick. I’m delighted to share it with you here.

The situation: you have an HTML document, with a list of linked text. Imagine a list of projects, each with a link to a project URL (the names aren’t meaningful):

The task is to convert this list of formatted links into a table, with the project name in column A, and the URL in column B.  The trick is to use an OpenOffice macro, which exposes the URL (and other facets of formatted text) as OpenOffice functions. Continue Reading »

Top Posts: StackOverflow “How do I get SQLAlchemy to correctly insert a unicode ellipsis into a mySQL table?”

Posted by Jim DeLaHunt on 31 Jul 2013 | Tagged as: Unicode, robobait, software engineering

I post on various forums around the net, and a few of my posts there get some very gratifying kudos. I’ve been a diligent contributor to StackOverflow, the Q-and-A site for software developers. I’m in the top 15% of contributors overall, and one of the top 25 answerers of Unicode-related questions. Here’s my second best-voted answer in StackOverflow so far.

The question, How do I get SQLAlchemy to correctly insert a unicode ellipsis into a mySQL table?,  was asked by user kvedananda in February 2012. In abbreviated form, it was:

Continue Reading »

Canada Post and USPS rate cards, 2013 rates

Posted by Jim DeLaHunt on 30 Apr 2013 | Tagged as: Canada, USA, robobait

Canada Post and the US Postal Service raised their postage rates again in January 2013. I was busy then, but I’ve grabbed a moment and updated my handy Canada Post and USPS postage rate quick reference card. The Canada Post rate increases were effective January 14, 2013, and the USPS increases were effective January 27.

My Canada Post and USPS Postage Rates project page,  http://jdlh.com/en/pr/postage_card.html, has links to download the latest charts as I update them.  The spreadsheet source file for the charts is also there. Both are licensed CC-BY-SA, so please feel free to re-use and modify them (as long as you attribute my work and share your product as freely).

Heads up: Canada Post has already received approval for first-class mail rate increases in 2014. The 2013 increases of both agencies came almost exactly one year after their 2012 increases, so I won’t be surprised if this becomes an annual event. The good news is that both Canada Post and USPS offer “perpetual” or “forever” stamps, which are worth first-class basic domestic postage, whatever the price may increase to.

Enjoy!

The Social Justice Tithe

Posted by Jim DeLaHunt on 31 Dec 2012 | Tagged as: culture, personal, robobait

With the year coming to an end, it is the season of making donations to organisations doing good in the world. In both Canada and the USA, this is motivated by a tax deadline; donations to certain charities by December 31 can be tax deductions for that year.  It’s an opportunity to lay out here a concept that I helped draft a decade ago: the “Social Justice Tithe”.

The Social Justice Tithe means giving at least 10% of your income to some combination of charities, religious groups, and political groups that enact your values.

Continue Reading »

Next Page »