Jim DeLaHunt, world-ready

This post has been a long time in the making. A year ago, I started work on my Twanguages code. This was code to analyse a corpus of Twitter messages, and try to discern patterns about language use, geography, and character encoding.  I decided to use the Django web framework and the Python language for the Twanguages analysis code.  I know Python, but I was learning Django for the first time.

Django is really, really marvellous.  When I tried this expression, and got the Python array of records I was expecting,

q2 = TwUser.objects.annotate(ntweets=Count('twstatus')).filter(ntweets__gt=1)

I wrote in my log, “I think I just fell in love. Power and concision in a tool, awesome.”

But Django gave me fits.  It has its share of quirks to trap the unwary novice. Eventually I began writing notes about “Django gotchas” in my log.  Some of them are Django being difficult, or inadequate. Some are me being a clueless novice, and Django not rescuing me from my folly. But all of them were obstacles.  I share them in the hopes of helping another Django novice.

Here are my Django gotchas.  They are ranked from the most distressing to most benign. They apply to Django 1.1, the current version at the time. (As of August 2010, the current version is 1.2.1.) A couple of gotchas were addressed by Django 1.2, so I moved them down to a section of their own. The rest presumably still apply to Django 1.2, but I haven’t gone back to check.

1. API fails unhelpfully. I wrote a simple query expression like:

   S2 = models.TwStatus.objects.get( key )

   I got a lot of weird errors, e.g. “ValueError: too many values to unpack” (where key is string) and “TypeError: ‘long’ object is not iterable” (where key is long). I had made a mistake, of course; the call to get() should have a keyword argument of “id__exact” or the like, not a positional argument. The correct spelling is this:

   S2 = models.TwStatus.objects.get( id__exact=key )

   The gotcha is that Django’s .get() isn’t written defensively. It isn’t very robust to programmer errors. Instead of checking parameters and giving clear error messages, it lets bad parameters through, only to have them fail obscurely deep in the framework. If defensive programming of the Django API would slow it down too much in production, I’d love to have a debug mode I could invoke during development. Continue Reading »

   No Comments »

There is a lot of international, multilingual, and multicultural activity in Vancouver. Also, there’s a thriving tech scene. But there’s no place for the people in the intersection of those two circles — those interested in and working on the internationalisation, localisation, and multilingual aspects of technology projects — to get together and share ideas. I think there ought to be.

And I’ll even propose a name: IMLIG1604, the I18n L10n M11l I6t G3p (Internationalisation, Localisation, and Multilingual Interest Group) for North America’s 604 area code. If you can decipher the title, you’re in the club!

Continue Reading »

No Comments »

What “twanguage” do you “tweet”?  Twitter, the buzzing conversation of brief web and SMS messsages, exploded into wide use in 2009. But just how wide?  To how many countries has it spread?  And into which languages?  I’m aiming to find out.

I’ve started a project named “Twanguages”, a language census of a sample of Twitter’s global traffic. I’m curious: which are the top languages? Are #hashtags localised? How does language correlate with location?  And which Unicode character is the most rarely used?

I’ll be  presenting our results at the 33rd Internationalization and Unicode Conference (IUC33), held in San Jose, California, on October 14-16, 2009. I have a place cleared for a Twanguages project page, and I’ll post interim results there as they become available (right now it’s only a placeholder). Stay tuned!

Continue Reading »

No Comments »

This was fun!  On Tuesday night (9. Sept 2008), there was a tribute to the 20th anniversary of Unicode at the 32nd Internationalization and Unicode Conference.  I wrote this in a creative fury on Monday afternoon. The anniversary celebration was at an evening reception. It was very funny and enjoyable. Several other people contributed amusing tributes. My song appeared to be well-received. I hope you enjoy it.

Unicode Doggerel

(Sung to the tune of  “I am the Very Model of a Modern Major General”, by Gilbert and Sullivan.)

I am the very model of a modern text encoding scheme,
a million scalars, astral planes, and UTFs like six-&-teen,
and UAX and UTR, collation, bidi, properties,
I am the very model of a modern text encoding scheme.

Continue Reading »

No Comments »

Does anybody know of a simple script-detection algorithm (or heuristic) for font switching?

This came up with one of my clients. Suppose you have a guest book on your web site, and seven visitors left you the following inspiring messages:

1. すべての人間は、生まれながらにして自由であり、かつ、尊厳と権利とについて平等である。
2. 人人生而自由,在尊严和权利上一律平等。
3. Semua orang dilahirkan merdeka dan mempunyai martabat dan hak-hak yang sama.
4. 人人生而自由，在尊嚴和權利上一律平等。
5. Alle Menschen sind frei und gleich an Würde und Rechten geboren.
6. ‘Ολοι οι άνθρωποι γεννιούνται ελεύθεροι και ίσοι στην αξιοπρέπεια και τα δικαιώματα.
7. 모든 인간은 태어날 때부터 자유로우며 그 존엄과 권리에 있어 동등하다.

(It looks like your visitors all read the Universal Declaration of Human Rights courtesy of the UDHR in Unicode project).

Now suppose you are so touched that you want to lay out all seven messages in a PDF file, and print it out as a booklet.  You have a beautiful layout template, and various complementary fonts: Latin script, Japanese, Korean, simplified Chinese, Traditional Chinese, and Greek script.

Which font to you apply to each message?  More importantly, is there a simple heuristic by which software can make the choice? (More after the jump.)

Continue Reading »

1 Comment »

Oh right, I forgot to mention: I’ve been accepted to present to the 32nd Internationalization & Unicode Conference this September! I’m presenting on a topic which I’ve been working on lately: multilingual websites. The title is: Web 2.0 goes to Babel: Multilingual websites and user-supplied content.

Continue Reading »

No Comments »