Jim DeLaHunt, world-ready

I post on various forums around the net, and a few of my posts there get some very gratifying kudos. I’ve been a diligent contributor to StackOverflow, the Q-and-A site for software developers. I’m in the top 15% of contributors overall, and one of the top 25 answerers of Unicode-related questions. Here’s my top-voted answer in StackOverflow so far.

The question, Django headache with simple non-ascii string,  was asked by user Ezequiel in January 2010. In abbreviated form, it was:

I just created the following model:

class Categoria(models.Model):
    nombre=models.CharField(max_length=30)
    padre=models.ForeignKey('self', blank=True, null=True)

    def __unicode__(self):
        return self.nombre

Then registered to the admin interface and syncdb’d

Everything ok if I just add plain ASCII chars. But if I add a “Categoria” named “á” (to say something) I get:

Environment:

Request Method: GET
Request URL: http://192.168.2.103:8000/administracion/locales/categoria/Django
Version: 1.1.1
Python Version: 2.6.4
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.admin',
'cruzandoelsuquiaDJ.locales']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')

Template error:
In template /usr/lib/pymodules/python2.6/django/contrib/admin/templates/admin/change_list.html, error at line 78
Caught an exception while rendering: ('ascii', '\xc3\xa1', 0, 1, 'ordinal not in range(128)')
68 :         {% endif %}[...some detail elided --JDLH...]
Traceback:
File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py" in get_response
92.                 response = callback(request, *callback_args, **callback_kwargs)

[...some detail elided --JDLH...]

File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779.                 bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
71.             result = node.render(context)
File "/usr/lib/pymodules/python2.6/django/template/loader_tags.py" in render
24.         result = self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779.                 bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
81.             raise wrapped
Exception Type: TemplateSyntaxError at /administracion/locales/categoria/
Exception Value: Caught an exception while rendering: ('ascii', '\xc3\xa1', 0, 1, 'ordinal not in range(128)')

My django version is 1.1 and my database is 5.1.37-1ubuntu5 with utf8 charset and the table is using a utf8_bin collation.
This problem seems too basic to be true, and I’m a django newbie so I’m sorry in advance if I’m missing something very simple 🙂

This is my answer to StackOverflow. 

Django generally has very good Unicode support (see the Django 1.1 “Unicode data” documentation  for details). In my code I find that, if I’m having a problem with  simple Unicode features, the problem usually is that I’m not  understanding Django’s details well, not that Django has a bug in its  Unicode support.
The “Unicode Data” page tells us that “All of Django’s database  backends … automatically convert strings retrieved from the database  into Python Unicode strings. You don’t even need to tell Django what  encoding your database uses: that is handled transparently.” So your  simple return self.nombre should return a Python Unicode string.
However, the Django 1.1 “Databases” page has an important note about how the MySQL backend handles the utf8_bin collation:
…if you really want case-sensitive   comparisons on a particular column or   table, you would change the column or   table to use the utf8_bin collation.   The main thing to be aware of in this   case is that if you are using MySQLdb   1.2.2, the database backend in Django will then return bytestrings (instead   of unicode strings) for any character   fields it returns receive from the   database. This is a strong variation   from Django’s normal practice of   always returning unicode strings. It   is up to you, the developer, to handle   the fact that you will receive   bytestrings if you configure your   table(s) to use utf8_bin collation.   Django itself should work smoothly   with such columns, but if your code   must be prepared to call   django.utils.encoding.smart_unicode()   at times if it really wants to work   with consistent data …
So, in your original example, the column “nombre” used utf8_bin collation. This meant that self.nombre  was returning a Python byte string. When you put it in an expression  that required a Python Unicode string, Python performed its default  conversion. This is the equivalent of self.nombre.decode('ascii'). And of course, .decode('ascii') fails when it encounters any byte above 0x7F, such as the UTF-8 bytes which encode “á”.
You discovered the two ways to solve this problem. The first is to convert the Python byte string returned by self.nombre into a Python Unicode string explicitly.  I’ll bet the following simpler code would have worked:
return self.nombre.decode('utf8') 
The second approach is to change the MySQL collation for column  “nombre”, which causes Django’s MySQL backend to return Python Unicode  strings instead of the unusual byte strings.  Then your original  expression gives a Python Unicode string:
return self.nombre 
Hope this helps.

 
				

					No Comments »