I post on various forums around the net, and a few of my posts there get some very gratifying kudos. I’ve been a diligent contributor to StackOverflow, the Q-and-A site for software developers. I’m in the top 15% of contributors overall, and one of the top 25 answerers of Unicode-related questions. Here’s my top-voted answer in StackOverflow so far.

The question, Django headache with simple non-ascii string,  was asked by user Ezequiel in January 2010. In abbreviated form, it was:


I just created the following model:

class Categoria(models.Model):
    nombre=models.CharField(max_length=30)
    padre=models.ForeignKey('self', blank=True, null=True)
    def __unicode__(self):
        return self.nombre

Then registered to the admin interface and syncdb’d

Everything ok if I just add plain ASCII chars. But if I add a “Categoria” named “á” (to say something) I get:

Environment:

Request Method: GET
Request URL: http://192.168.2.103:8000/administracion/locales/categoria/Django
Version: 1.1.1
Python Version: 2.6.4
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.admin',
'cruzandoelsuquiaDJ.locales']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')

Template error:
In template /usr/lib/pymodules/python2.6/django/contrib/admin/templates/admin/change_list.html, error at line 78
Caught an exception while rendering: ('ascii', '\xc3\xa1', 0, 1, 'ordinal not in range(128)')
68 :         {% endif %}[...some detail elided --JDLH...]
Traceback:
File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py" in get_response
92.                 response = callback(request, *callback_args, **callback_kwargs)

[...some detail elided --JDLH...]

File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779.                 bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
71.             result = node.render(context)
File "/usr/lib/pymodules/python2.6/django/template/loader_tags.py" in render
24.         result = self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779.                 bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
81.             raise wrapped
Exception Type: TemplateSyntaxError at /administracion/locales/categoria/
Exception Value: Caught an exception while rendering: ('ascii', '\xc3\xa1', 0, 1, 'ordinal not in range(128)')

My django version is 1.1 and my database is 5.1.37-1ubuntu5 with utf8 charset and the table is using a utf8_bin collation.

This problem seems too basic to be true, and I’m a django newbie so I’m sorry in advance if I’m missing something very simple :)


This is my answer to StackOverflow.

Django generally has very good Unicode support (see the Django 1.1 “Unicode data” documentation for details). In my code I find that, if I’m having a problem with simple Unicode features, the problem usually is that I’m not understanding Django’s details well, not that Django has a bug in its Unicode support.

The “Unicode Data” page tells us that “All of Django’s database backends … automatically convert strings retrieved from the database into Python Unicode strings. You don’t even need to tell Django what encoding your database uses: that is handled transparently.” So your simple return self.nombre should return a Python Unicode string.

However, the Django 1.1 “Databases” page has an important note about how the MySQL backend handles the utf8_bin collation:

…if you really want case-sensitive comparisons on a particular column or table, you would change the column or table to use the utf8_bin collation. The main thing to be aware of in this case is that if you are using MySQLdb 1.2.2, the database backend in Django will then return bytestrings (instead of unicode strings) for any character fields it returns receive from the database. This is a strong variation from Django’s normal practice of always returning unicode strings. It is up to you, the developer, to handle the fact that you will receive bytestrings if you configure your table(s) to use utf8_bin collation. Django itself should work smoothly with such columns, but if your code must be prepared to call django.utils.encoding.smart_unicode() at times if it really wants to work with consistent data …

So, in your original example, the column “nombre” used utf8_bin collation. This meant that self.nombre was returning a Python byte string. When you put it in an expression that required a Python Unicode string, Python performed its default conversion. This is the equivalent of self.nombre.decode('ascii'). And of course, .decode('ascii') fails when it encounters any byte above 0×7F, such as the UTF-8 bytes which encode “á”.

You discovered the two ways to solve this problem. The first is to convert the Python byte string returned by self.nombre into a Python Unicode string explicitly. I’ll bet the following simpler code would have worked:

return self.nombre.decode('utf8') 

The second approach is to change the MySQL collation for column “nombre”, which causes Django’s MySQL backend to return Python Unicode strings instead of the unusual byte strings. Then your original expression gives a Python Unicode string:

return self.nombre 

Hope this helps.


profile for Jim DeLaHunt on Stack Exchange, a network of free, community-driven Q&A sites