Merged Unicode branch into trunk (r4952:5608). This should be fully

backwards compatible for all practical purposes.

Fixed #2391, #2489, #2996, #3322, #3344, #3370, #3406, #3432, #3454, #3492, #3582, #3690, #3878, #3891, #3937, #4039, #4141, #4227, #4286, #4291, #4300, #4452, #4702


git-svn-id: http://code.djangoproject.com/svn/django/trunk@5609 bcc190cf-cafb-0310-a4f2-bffc1f526a37

11 files changed, 583 insertions, 72 deletions

diff --git a/docs/contributing.txt b/docs/contributing.txt
index 2813793fa5..e58b06881a 100644
--- a/docs/contributing.txt
+++ b/docs/contributing.txt
@@ -368,6 +368,7 @@ Model style
         * All database fields
         * ``class Meta``
         * ``class Admin``
+        * ``def __unicode__()``
         * ``def __str__()``
         * ``def save()``
         * ``def get_absolute_url()``
diff --git a/docs/db-api.txt b/docs/db-api.txt
index a4b920fb33..ef3d811189 100644
--- a/docs/db-api.txt
+++ b/docs/db-api.txt
@@ -15,14 +15,14 @@ a weblog application::
         name = models.CharField(maxlength=100)
         tagline = models.TextField()
 
-        def __str__(self):
+        def __unicode__(self):
             return self.name
 
     class Author(models.Model):
         name = models.CharField(maxlength=50)
         email = models.URLField()
 
-        def __str__(self):
+        def __unicode__(self):
             return self.name
 
     class Entry(models.Model):
@@ -32,7 +32,7 @@ a weblog application::
         pub_date = models.DateTimeField()
         authors = models.ManyToManyField(Author)
 
-        def __str__(self):
+        def __unicode__(self):
             return self.headline
 
 Creating objects
diff --git a/docs/forms.txt b/docs/forms.txt
index f6cb55a3f6..18d3d3fcbe 100644
--- a/docs/forms.txt
+++ b/docs/forms.txt
@@ -47,7 +47,7 @@ this document, we'll be working with the following model, a "place" object::
         class Admin:
             pass
 
-        def __str__(self):
+        def __unicode__(self):
             return self.name
 
 Defining the above class is enough to create an admin interface to a ``Place``,
diff --git a/docs/i18n.txt b/docs/i18n.txt
index 27abadacc9..9fe02c0342 100644
--- a/docs/i18n.txt
+++ b/docs/i18n.txt
@@ -68,23 +68,41 @@ In Python code
 Standard translation
 ~~~~~~~~~~~~~~~~~~~~
 
-Specify a translation string by using the function ``_()``. (Yes, the name of
-the function is the "underscore" character.) This function is available
-globally in any Python module; you don't have to import it.
+Specify a translation string by using the function ``ugettext()``. It's
+convention to import this as a shorter alias, ``_``, to save typing.
+
+.. note::
+    Python's standard library ``gettext`` module installs ``_()`` into the
+    global namespace, as an alias for ``gettext()``. In Django, we have chosen
+    not to follow this practice, for a couple of reasons:
+
+      1. For international character set (Unicode) support, ``ugettext()`` is
+         more useful than ``gettext()``. Sometimes, you should be using
+         ``ugettext_lazy()`` as the default translation method for a particular
+         file. Without ``_()`` in the global namespace, the developer has to
+         think about which is the most appropriate translation function.
+
+      2. The underscore character (``_``) is used to represent "the previous
+         result" in Python's interactive shell and doctest tests. Installing a
+         global ``_()`` function causes interference. Explicitly importing
+         ``ugettext()`` as ``_()`` avoids this problem.
 
 In this example, the text ``"Welcome to my site."`` is marked as a translation
 string::
 
+    from django.utils.translation import ugettext as _
+
     def my_view(request):
         output = _("Welcome to my site.")
         return HttpResponse(output)
 
-The function ``django.utils.translation.gettext()`` is identical to ``_()``.
-This example is identical to the previous one::
+Obviously, you could code this without using the alias. This example is
+identical to the previous one::
+
+    from django.utils.translation import ugettext
 
-    from django.utils.translation import gettext
     def my_view(request):
-        output = gettext("Welcome to my site.")
+        output = ugettext("Welcome to my site.")
         return HttpResponse(output)
 
 Translation works on computed values. This example is identical to the previous
@@ -107,7 +125,7 @@ examples, is that Django's translation-string-detecting utility,
 ``make-messages.py``, won't be able to find these strings. More on
 ``make-messages`` later.)
 
-The strings you pass to ``_()`` or ``gettext()`` can take placeholders,
+The strings you pass to ``_()`` or ``ugettext()`` can take placeholders,
 specified with Python's standard named-string interpolation syntax. Example::
 
     def my_view(request, n):
@@ -120,14 +138,14 @@ while a Spanish translation may be ``"Me llamo Adrian."`` -- with the
 placeholder (the name) placed after the translated text instead of before it.
 
 For this reason, you should use named-string interpolation (e.g., ``%(name)s``)
-instead of positional interpolation (e.g., ``%s`` or ``%d``). If you used
-positional interpolation, translations wouldn't be able to reorder placeholder
-text.
+instead of positional interpolation (e.g., ``%s`` or ``%d``) whenever you
+have more than a single parameter. If you used positional interpolation,
+translations wouldn't be able to reorder placeholder text.
 
 Marking strings as no-op
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-Use the function ``django.utils.translation.gettext_noop()`` to mark a string
+Use the function ``django.utils.translation.ugettext_noop()`` to mark a string
 as a translation string without translating it. The string is later translated
 from a variable.
 
@@ -139,35 +157,35 @@ as when the string is presented to the user.
 Lazy translation
 ~~~~~~~~~~~~~~~~
 
-Use the function ``django.utils.translation.gettext_lazy()`` to translate
+Use the function ``django.utils.translation.ugettext_lazy()`` to translate
 strings lazily -- when the value is accessed rather than when the
-``gettext_lazy()`` function is called.
+``ugettext_lazy()`` function is called.
 
 For example, to translate a model's ``help_text``, do the following::
 
-    from django.utils.translation import gettext_lazy
+    from django.utils.translation import ugettext_lazy
 
     class MyThing(models.Model):
-        name = models.CharField(help_text=gettext_lazy('This is the help text'))
+        name = models.CharField(help_text=ugettext_lazy('This is the help text'))
 
-In this example, ``gettext_lazy()`` stores a lazy reference to the string --
+In this example, ``ugettext_lazy()`` stores a lazy reference to the string --
 not the actual translation. The translation itself will be done when the string
 is used in a string context, such as template rendering on the Django admin site.
 
-If you don't like the verbose name ``gettext_lazy``, you can just alias it as
+If you don't like the verbose name ``ugettext_lazy``, you can just alias it as
 ``_`` (underscore), like so::
 
-    from django.utils.translation import gettext_lazy as _
+    from django.utils.translation import ugettext_lazy as _
 
     class MyThing(models.Model):
         name = models.CharField(help_text=_('This is the help text'))
 
-Always use lazy translations in `Django models`_. And it's a good idea to add
+Always use lazy translations in `Django models`_. It's a good idea to add
 translations for the field names and table names, too. This means writing
 explicit ``verbose_name`` and ``verbose_name_plural`` options in the ``Meta``
 class, though::
 
-    from django.utils.translation import gettext_lazy as _
+    from django.utils.translation import ugettext_lazy as _
 
     class MyThing(models.Model):
         name = models.CharField(_('name'), help_text=_('This is the help text'))
@@ -180,24 +198,24 @@ class, though::
 Pluralization
 ~~~~~~~~~~~~~
 
-Use the function ``django.utils.translation.ngettext()`` to specify pluralized
+Use the function ``django.utils.translation.ungettext()`` to specify pluralized
 messages. Example::
 
-    from django.utils.translation import ngettext
+    from django.utils.translation import ungettext
     def hello_world(request, count):
-        page = ngettext('there is %(count)d object', 'there are %(count)d objects', count) % {
+        page = ungettext('there is %(count)d object', 'there are %(count)d objects', count) % {
             'count': count,
         }
         return HttpResponse(page)
 
-``ngettext`` takes three arguments: the singular translation string, the plural
+``ungettext`` takes three arguments: the singular translation string, the plural
 translation string and the number of objects (which is passed to the
 translation languages as the ``count`` variable).
 
 In template code
 ----------------
 
-Using translations in `Django templates`_ uses two template tags and a slightly
+Translations in `Django templates`_ uses two template tags and a slightly
 different syntax than in Python code. To give your template access to these
 tags, put ``{% load i18n %}`` toward the top of your template.
 
@@ -243,7 +261,7 @@ To pluralize, specify both the singular and plural forms with the
     {% endblocktrans %}
 
 Internally, all block and inline translations use the appropriate
-``gettext`` / ``ngettext`` call.
+``ugettext`` / ``ungettext`` call.
 
 Each ``RequestContext`` has access to two translation-specific variables:
 
@@ -276,6 +294,71 @@ string, so they don't need to be aware of translations.
 
 .. _Django templates: ../templates_python/
 
+Working with lazy translation objects
+=====================================
+
+Using ``ugettext_lazy()`` and ``ungettext_lazy()`` to mark strings in models
+and utility functions is a common operation. When you're working with these
+objects elsewhere in your code, you should ensure that you don't accidentally
+convert them to strings, because they should be converted as late as possible
+(so that the correct locale is in effect). This necessitates the use of a
+couple of helper functions.
+
+Joining strings: string_concat()
+--------------------------------
+
+Standard Python string joins (``''.join([...])``) will not work on lists
+containing lazy translation objects. Instead, you can use
+``django.utils.translation.string_concat()``, which creates a lazy object that
+concatenates its contents *and* converts them to strings only when the result
+is included in a string. For example::
+
+    from django.utils.translation import string_concat
+    ...
+    name = ugettext_lazy(u'John Lennon')
+    instrument = ugettext_lazy(u'guitar')
+    result = string_concat([name, ': ', instrument])
+
+In this case, the lazy translations in ``result`` will only be converted to
+strings when ``result`` itself is used in a string (usually at template
+rendering time).
+
+The allow_lazy() decorator
+--------------------------
+
+Django offers many utility functions (particularly in ``django.utils``) that
+take a string as their first argument and do something to that string. These
+functions are used by template filters as well as directly in other code.
+
+If you write your own similar functions and deal with translations, you'll 
+face the problem of what to do when the first argument is a lazy translation
+object. You don't want to convert it to a string immediately, because you might
+be using this function outside of a view (and hence the current thread's locale
+setting will not be correct).
+
+For cases like this, use the  ``django.utils.functional.allow_lazy()``
+decorator. It modifies the function so that *if* it's called with a lazy
+translation as the first argument, the function evaluation is delayed until it
+needs to be converted to a string.
+
+For example::
+
+    from django.utils.functional import allow_lazy
+
+    def fancy_utility_function(s, ...):
+        # Do some conversion on string 's'
+        ...
+    fancy_utility_function = allow_lazy(fancy_utility_function, unicode)
+
+The ``allow_lazy()`` decorator takes, in addition to the function to decorate,
+a number of extra arguments (``*args``) specifying the type(s) that the
+original function can return. Usually, it's enough to include ``unicode`` here
+and ensure that your function returns only Unicode strings.
+
+Using this decorator means you can write your function and assume that the
+input is a proper string, then add support for lazy translation objects at the
+end.
+
 How to create language files
 ============================
 
@@ -487,26 +570,26 @@ Notes:
 
     * If you define a custom ``LANGUAGES`` setting, as explained in the
       previous bullet, it's OK to mark the languages as translation strings
-      -- but use a "dummy" ``gettext()`` function, not the one in
+      -- but use a "dummy" ``ugettext()`` function, not the one in
       ``django.utils.translation``. You should *never* import
       ``django.utils.translation`` from within your settings file, because that
       module in itself depends on the settings, and that would cause a circular
       import.
 
-      The solution is to use a "dummy" ``gettext()`` function. Here's a sample
+      The solution is to use a "dummy" ``ugettext()`` function. Here's a sample
       settings file::
 
-          gettext = lambda s: s
+          ugettext = lambda s: s
 
           LANGUAGES = (
-              ('de', gettext('German')),
-              ('en', gettext('English')),
+              ('de', ugettext('German')),
+              ('en', ugettext('English')),
           )
 
       With this arrangement, ``make-messages.py`` will still find and mark
       these strings for translation, but the translation won't happen at
       runtime -- so you'll have to remember to wrap the languages in the *real*
-      ``gettext()`` in any code that uses ``LANGUAGES`` at runtime.
+      ``ugettext()`` in any code that uses ``LANGUAGES`` at runtime.
 
     * The ``LocaleMiddleware`` can only select languages for which there is a
       Django-provided base translation. If you want to provide translations
@@ -712,23 +795,23 @@ interface to access it::
 
     document.write(gettext('this is to be translated'));
 
-There even is a ``ngettext`` interface and a string interpolation function::
+There even is a ``ungettext`` interface and a string interpolation function::
 
     d = {
         count: 10
     };
-    s = interpolate(ngettext('this is %(count)s object', 'this are %(count)s objects', d.count), d);
+    s = interpolate(ungettext('this is %(count)s object', 'this are %(count)s objects', d.count), d);
 
 The ``interpolate`` function supports both positional interpolation and named
 interpolation. So the above could have been written as::
 
-    s = interpolate(ngettext('this is %s object', 'this are %s objects', 11), [11]);
+    s = interpolate(ungettext('this is %s object', 'this are %s objects', 11), [11]);
 
 The interpolation syntax is borrowed from Python. You shouldn't go over the top
 with string interpolation, though: this is still JavaScript, so the code will
 have to do repeated regular-expression substitutions. This isn't as fast as
 string interpolation  in Python, so keep it to those cases where you really
-need it (for example, in conjunction with ``ngettext`` to produce proper
+need it (for example, in conjunction with ``ungettext`` to produce proper
 pluralizations).
 
 Creating JavaScript translation catalogs
@@ -750,16 +833,13 @@ Specialities of Django translation
 If you know ``gettext``, you might note these specialities in the way Django
 does translation:
 
-    * The string domain is ``django`` or ``djangojs``. The string domain is used to
-      differentiate between different programs that store their data in a
-      common message-file library (usually ``/usr/share/locale/``). The ``django``
-      domain is used for python and template translation strings and is loaded into
-      the global translation catalogs. The ``djangojs`` domain is only used for
-      JavaScript translation catalogs to make sure that those are as small as
-      possible.
-    * Django only uses ``gettext`` and ``gettext_noop``. That's because Django
-      always uses ``DEFAULT_CHARSET`` strings internally. There isn't much use
-      in using ``ugettext``, because you'll always need to produce utf-8
-      anyway.
+    * The string domain is ``django`` or ``djangojs``. The string domain is
+      used to differentiate between different programs that store their data
+      in a common message-file library (usually ``/usr/share/locale/``). The
+      ``django`` domain is used for python and template translation strings
+      and is loaded into the global translation catalogs. The ``djangojs``
+      domain is only used for JavaScript translation catalogs to make sure
+      that those are as small as possible.
     * Django doesn't use ``xgettext`` alone. It uses Python wrappers around
       ``xgettext`` and ``msgfmt``. That's mostly for convenience.
+
diff --git a/docs/model-api.txt b/docs/model-api.txt
index 22ff7445b5..4b0bc0d238 100644
--- a/docs/model-api.txt
+++ b/docs/model-api.txt
@@ -1367,10 +1367,11 @@ A few special cases to note about ``list_display``:
               born_in_fifties.boolean = True
 
 
-    * The ``__str__()`` method is just as valid in ``list_display`` as any
-      other model method, so it's perfectly OK to do this::
+    * The ``__str__()`` and ``__unicode__()`` methods are just as valid in
+      ``list_display`` as any other model method, so it's perfectly OK to do
+      this::
 
-          list_display = ('__str__', 'some_other_field')
+          list_display = ('__unicode__', 'some_other_field')
 
     * Usually, elements of ``list_display`` that aren't actual database fields
       can't be used in sorting (because Django does all the sorting at the
@@ -1776,11 +1777,13 @@ A few object methods have special meaning:
 -----------
 
 ``__str__()`` is a Python "magic method" that defines what should be returned
-if you call ``str()`` on the object. Django uses ``str(obj)`` in a number of
-places, most notably as the value displayed to render an object in the Django
-admin site and as the value inserted into a template when it displays an
-object. Thus, you should always return a nice, human-readable string for the
-object's ``__str__``. Although this isn't required, it's strongly encouraged.
+if you call ``str()`` on the object. Django uses ``str(obj)`` (or the related
+function, ``unicode(obj)`` -- see below) in a number of places, most notably
+as the value displayed to render an object in the Django admin site and as the
+value inserted into a template when it displays an object. Thus, you should
+always return a nice, human-readable string for the object's ``__str__``.
+Although this isn't required, it's strongly encouraged (see the description of
+``__unicode__``, below, before putting ``_str__`` methods everywhere).
 
 For example::
 
@@ -1789,7 +1792,32 @@ For example::
         last_name = models.CharField(maxlength=50)
 
         def __str__(self):
-            return '%s %s' % (self.first_name, self.last_name)
+            # Note use of django.utils.encoding.smart_str() here because
+            # first_name and last_name will be unicode strings.
+            return smart_str('%s %s' % (self.first_name, self.last_name))
+
+``__unicode__``
+---------------
+
+The ``__unicode__()`` method is called whenever you call ``unicode()`` on an
+object. Since Django's database backends will return Unicode strings in your
+model's attributes, you would normally want to write a ``__unicode__()``
+method for your model. The example in the previous section could be written
+more simply as::
+
+    class Person(models.Model):
+        first_name = models.CharField(maxlength=50)
+        last_name = models.CharField(maxlength=50)
+
+        def __unicode__(self):
+            return u'%s %s' % (self.first_name, self.last_name)
+
+If you define a ``__unicode__()`` method on your model and not a ``__str__()``
+method, Django will automatically provide you with a ``__str__()`` that calls
+``__unicode()__`` and then converts the result correctly to a UTF-8 encoded
+string object. This is recommended development practice: define only
+``__unicode__()`` and let Django take care of the conversion to string objects
+when required.
 
 ``get_absolute_url``
 --------------------
@@ -1823,7 +1851,9 @@ But this template code is good::
     characters (required by the URI spec, `RFC 2396`_) that have been
     URL-encoded, if necessary. Code and templates using ``get_absolute_url()``
     should be able to use the result directly without needing to do any
-    further processing.
+    further processing. You may wish to use the
+    ``django.utils.encoding.iri_to_uri()`` function to help with this if you
+    are using unicode strings a lot.
 
 .. _RFC 2396: http://www.ietf.org/rfc/rfc2396.txt
 
diff --git a/docs/newforms.txt b/docs/newforms.txt
index 41db04a7dd..c2e08c63b9 100644
--- a/docs/newforms.txt
+++ b/docs/newforms.txt
@@ -1456,7 +1456,7 @@ Consider this set of models::
         title = models.CharField(maxlength=3, choices=TITLE_CHOICES)
         birth_date = models.DateField(blank=True, null=True)
 
-        def __str__(self):
+        def __unicode__(self):
             return self.name
 
     class Book(models.Model):
diff --git a/docs/overview.txt b/docs/overview.txt
index 7b3559663a..041ad152c7 100644
--- a/docs/overview.txt
+++ b/docs/overview.txt
@@ -27,7 +27,7 @@ quick example::
     class Reporter(models.Model):
         full_name = models.CharField(maxlength=70)
 
-        def __str__(self):
+        def __unicode__(self):
             return self.full_name
 
     class Article(models.Model):
@@ -36,7 +36,7 @@ quick example::
         article = models.TextField()
         reporter = models.ForeignKey(Reporter)
 
-        def __str__(self):
+        def __unicode__(self):
             return self.headline
 
 Install it
diff --git a/docs/settings.txt b/docs/settings.txt
index 897cdc8099..9c9602d9ec 100644
--- a/docs/settings.txt
+++ b/docs/settings.txt
@@ -442,6 +442,16 @@ Default: ``False``
 
 Whether to use a TLS (secure) connection when talking to the SMTP server.
 
+FILE_CHARSET
+------------
+
+**New in Django development version**
+
+Default: ``'utf-8'``
+
+The character encoding used to decode any files read from disk. This includes
+template files and initial SQL data files.
+
 FIXTURE_DIRS
 -------------
 
diff --git a/docs/templates.txt b/docs/templates.txt
index c32b1af1dd..f5ed1fe74b 100644
--- a/docs/templates.txt
+++ b/docs/templates.txt
@@ -1040,6 +1040,16 @@ right-most digit, 2 is the second-right-most digit, etc. Returns the original
 value for invalid input (if input or argument is not an integer, or if argument
 is less than 1). Otherwise, output is always an integer.
 
+iriencode
+~~~~~~~~~
+
+Converts an IRI (Internationalized Resource Identifier) to a string that is
+suitable for including in a URL. This is necessary if you're trying to use
+strings containing non-ASCII characters in a URL.
+
+It's safe to use this filter on a string that has already gone through the
+``urlencode`` filter.
+
 join
 ~~~~
 
diff --git a/docs/tutorial01.txt b/docs/tutorial01.txt
index fdac9c554e..aea3af6922 100644
--- a/docs/tutorial01.txt
+++ b/docs/tutorial01.txt
@@ -474,22 +474,39 @@ Once you're in the shell, explore the database API::
 
 Wait a minute. ``<Poll: Poll object>`` is, utterly, an unhelpful
 representation of this object. Let's fix that by editing the polls model (in
-the ``polls/models.py`` file) and adding a ``__str__()`` method to both
+the ``polls/models.py`` file) and adding a ``__unicode__()`` method to both
 ``Poll`` and ``Choice``::
 
     class Poll(models.Model):
         # ...
-        def __str__(self):
+        def __unicode__(self):
             return self.question
 
     class Choice(models.Model):
         # ...
-        def __str__(self):
+        def __unicode__(self):
             return self.choice
 
-It's important to add ``__str__()`` methods to your models, not only for your
-own sanity when dealing with the interactive prompt, but also because objects'
-representations are used throughout Django's automatically-generated admin.
+It's important to add ``__unicode__()`` methods to your models, not only for
+your own sanity when dealing with the interactive prompt, but also because
+objects' representations are used throughout Django's automatically-generated
+admin.
+
+.. admonition:: Why ``__unicode__()`` and not ``__str__()``?
+
+	If you're familiar with Python, you might be in the habit of adding
+	``__str__()`` methods to your classes, not ``__unicode__()`` methods.
+    We use ``__unicode__()`` here because Django models deal with Unicode by
+    default. All data stored in your database is converted to Unicode when it's
+    returned.
+
+	Django models have a default ``__str__()`` method that calls ``__unicode__()``
+	and converts the result to a UTF-8 bytestring. This means that ``unicode(p)``
+	will return a Unicode string, and ``str(p)`` will return a normal string,
+	with characters encoded as UTF-8.
+
+	If all of this is jibberish to you, just remember to add ``__unicode__()``
+	methods to your models. With any luck, things should Just Work for you.
 
 Note these are normal Python methods. Let's add a custom method, just for
 demonstration::
@@ -509,7 +526,7 @@ Let's jump back into the Python interactive shell by running
 
     >>> from mysite.polls.models import Poll, Choice
 
-    # Make sure our __str__() addition worked.
+    # Make sure our __unicode__() addition worked.
     >>> Poll.objects.all()
     [<Poll: What's up?>]
 
diff --git a/docs/unicode.txt b/docs/unicode.txt
new file mode 100644
index 0000000000..2313c28c98
--- /dev/null
+++ b/docs/unicode.txt
@@ -0,0 +1,363 @@
+======================
+Unicode data in Django
+======================
+
+**New in Django development version**
+
+Django natively supports Unicode data everywhere. Providing your database can
+somehow store the data, you can safely pass around Unicode strings to
+templates, models and the database.
+
+This document tells you what you need to know if you're writing applications
+that use data or templates that are encoded in something other than ASCII.
+
+Creating the database
+=====================
+
+Make sure your database is configured to be able to store arbitrary string
+data. Normally, this means giving it an encoding of UTF-8 or UTF-16. If you use
+a more restrictive encoding -- for example, latin1 (iso8859-1) -- you won't be
+able to store certain characters in the database, and information will be lost.
+
+ * MySQL users, refer to the `MySQL manual`_ (section 10.3.2 for MySQL 5.1) for
+   details on how to set or alter the database character set encoding.
+
+ * PostgreSQL users, refer to the `PostgreSQL manual`_ (section 21.2.2 in
+   PostgreSQL 8) for details on creating databases with the correct encoding.
+
+ * SQLite users, there is nothing you need to do. SQLite always uses UTF-8
+   for internal encoding.
+
+.. _MySQL manual: http://www.mysql.org/doc/refman/5.1/en/charset-database.html
+.. _PostgreSQL manual: http://www.postgresql.org/docs/8.2/static/multibyte.html#AEN24104
+
+All of Django's database backends automatically convert Unicode strings into
+the appropriate encoding for talking to the database. They also automatically
+convert strings retrieved from the database into Python Unicode strings. You
+don't even need to tell Django what encoding your database uses: that is
+handled transparently.
+
+For more, see the section "The database API" below.
+
+General string handling
+=======================
+
+Whenever you use strings with Django -- e.g., in database lookups, template
+rendering or anywhere else -- you have two choices for encoding those strings.
+You can use Unicode strings, or you can use normal strings (sometimes called
+"bytestrings") that are encoded using UTF-8.
+
+.. warning::
+    A bytestring does not carry any information with it about its encoding.
+    For that reason, we have to make an assumption, and Django assumes that all
+    bytestrings are in UTF-8.
+
+    If you pass a string to Django that has been encoded in some other format,
+    things will go wrong in interesting ways. Usually, Django will raise a
+    ``UnicodeDecodeError`` at some point.
+
+If your code only uses ASCII data, it's safe to use your normal strings,
+passing them around at will, because ASCII is a subset of UTF-8.
+
+Don't be fooled into thinking that if your ``DEFAULT_CHARSET`` setting is set
+to something other than ``'utf-8'`` you can use that other encoding in your
+bytestrings! ``DEFAULT_CHARSET`` only applies to the strings generated as
+the result of template rendering (and e-mail). Django will always assume UTF-8
+encoding for internal bytestrings. The reason for this is that the
+``DEFAULT_CHARSET`` setting is not actually under your control (if you are the
+application developer). It's under the control of the person installing and
+using your application -- and if that person chooses a different setting, your
+code must still continue to work. Ergo, it cannot rely on that setting.
+
+In most cases when Django is dealing with strings, it will convert them to
+Unicode strings before doing anything else. So, as a general rule, if you pass
+in a bytestring, be prepared to receive a Unicode string back in the result.
+
+Translated strings
+------------------
+
+Aside from Unicode strings and bytestrings, there's a third type of string-like
+object you may encounter when using Django. The framework's
+internationalization features introduce the concept of a "lazy translation" --
+a string that has been marked as translated but whose actual translation result
+isn't determined until the object is used in a string. This feature is useful
+in cases where the translation locale is unknown until the string is used, even
+though the string might have originally been created when the code was first
+imported.
+
+Normally, you won't have to worry about lazy translations. Just be aware that
+if you examine an object and it claims to be a
+``django.utils.functional.__proxy__`` object, it is a lazy translation.
+Calling ``unicode()`` with the lazy translation as the argument will generate a
+Unicode string in the current locale.
+
+For more details about lazy translation objects, refer to the
+internationalization_ documentation.
+
+.. _internationalization: ../i18n/#lazy-translation
+
+Useful utility functions
+------------------------
+
+Because some string operations come up again and again, Django ships with a few
+useful functions that should make working with Unicode and bytestring objects
+a bit easier.
+
+Conversion functions
+~~~~~~~~~~~~~~~~~~~~
+
+The ``django.utils.encoding`` module contains a few functions that are handy
+for converting back and forth between Unicode and bytestrings.
+
+    * ``smart_unicode(s, encoding='utf-8', errors='strict')`` converts its
+      input to a Unicode string. The ``encoding`` parameter specifies the input
+      encoding. (For example, Django uses this internally when processing form
+      input data, which might not be UTF-8 encoded.) The ``errors`` parameter
+      takes any of the values that are accepted by Python's ``unicode()``
+      function for its error handling.
+
+      If you pass ``smart_unicode()`` an object that has a ``__unicode__``
+      method, it will use that method to do the conversion.
+
+    * ``force_unicode(s, encoding='utf-8', errors='strict')`` is identical to
+      ``smart_unicode()`` in almost all cases. The difference is when the
+      first argument is a `lazy translation`_ instance. While
+      ``smart_unicode()`` preserves lazy translations, ``force_unicode()``
+      forces those objects to a Unicode string (causing the translation to
+      occur). Normally, you'll want to use ``smart_unicode()``. However,
+      ``force_unicode()`` is useful in template tags and filters that
+      absolutely *must* have a string to work with, not just something that can
+      be converted to a string.
+
+    * ``smart_str(s, encoding='utf-8', strings_only=False, errors='strict')``
+      is essentially the opposite of ``smart_unicode()``. It forces the first
+      argument to a bytestring. The ``strings_only`` parameter, if set to True,
+      will result in Python integers, booleans and ``None`` not being
+      converted to a string (they keep their original types). This is slightly
+      different semantics from Python's builtin ``str()`` function, but the
+      difference is needed in a few places within Django's internals.
+
+Normally, you'll only need to use ``smart_unicode()``. Call it as early as
+possible on any input data that might be either Unicode or a bytestring, and
+from then on, you can treat the result as always being Unicode.
+
+URI and IRI handling
+~~~~~~~~~~~~~~~~~~~~
+
+Web frameworks have to deal with URLs (which are a type of URI_). One
+requirement of URLs is that they are encoded using only ASCII characters.
+However, in an international environment, you might need to construct a
+URL from an IRI_ -- very loosely speaking, a URI that can contain Unicode
+characters. Quoting and converting an IRI to URI can be a little tricky, so
+Django provides some assistance.
+
+    * The function ``django.utils.encoding.iri_to_uri()`` implements the
+      conversion from IRI to URI as required by the specification (`RFC
+      3987`_).
+
+    * The functions ``django.utils.http.urlquote()`` and
+      ``django.utils.http.urlquote_plus()`` are versions of Python's standard
+      ``urllib.quote()`` and ``urllib.quote_plus()`` that work with non-ASCII
+      characters. (The data is converted to UTF-8 prior to encoding.)
+
+These two groups of functions have slightly different purposes, and it's
+important to keep them straight. Normally, you would use ``urlquote()`` on the
+individual portions of the IRI or URI path so that any reserved characters
+such as '&' or '%' are correctly encoded. Then, you apply ``iri_to_uri()`` to
+the full IRI and it converts any non-ASCII characters to the correct encoded
+values.
+
+.. note::
+    Technically, it isn't correct to say that ``iri_to_uri()`` implements the
+    full algorithm in the IRI specification. It doesn't (yet) perform the
+    international domain name encoding portion of the algorithm.
+
+The ``iri_to_uri()`` function will not change ASCII characters that are
+otherwise permitted in a URL. So, for example, the character '%' is not
+further encoded when passed to ``iri_to_uri()``. This means you can pass a
+full URL to this function and it will not mess up the query string or anything
+like that.
+
+An example might clarify things here::
+
+    >>> urlquote(u'Paris & Orléans')
+    u'Paris%20%26%20Orl%C3%A9ans'
+    >>> iri_to_uri(u'/favorites/François/%s' % urlquote(u'Paris & Orléans'))
+    '/favorites/Fran%C3%A7ois/Paris%20%26%20Orl%C3%A9ans'
+
+If you look carefully, you can see that the portion that was generated by
+``urlquote()`` in the second example was not double-quoted when passed to
+``iri_to_uri()``. This is a very important and useful feature. It means that
+you can construct your IRI without worrying about whether it contains
+non-ASCII characters and then, right at the end, call ``iri_to_uri()`` on the
+result.
+
+The ``iri_to_uri()`` function is also idempotent, which means the following is
+always true::
+
+    iri_to_uri(iri_to_uri(some_string)) = iri_to_uri(some_string)
+
+So you can safely call it multiple times on the same IRI without risking
+double-quoting problems.
+
+.. _URI: http://www.ietf.org/rfc/rfc2396.txt
+.. _IRI: http://www.ietf.org/rfc/rfc3987.txt
+.. _RFC 3987: IRI_
+
+Models
+======
+
+Because all strings are returned from the database as Unicode strings, model
+fields that are character based (CharField, TextField, URLField, etc) will
+contain Unicode values when Django retrieves data from the database. This
+is *always* the case, even if the data could fit into an ASCII bytestring.
+
+You can pass in bytestrings when creating a model or populating a field, and
+Django will convert it to Unicode when it needs to.
+
+Choosing between ``__str__()`` and ``__unicode__()``
+----------------------------------------------------
+
+One consequence of using Unicode by default is that you have to take some care
+when printing data from the model.
+
+In particular, rather than giving your model a ``__str__()`` method, we
+recommended you implement a ``__unicode__()`` method. In the ``__unicode__()``
+method, you can quite safely return the values of all your fields without
+having to worry about whether they fit into a bytestring or not. (The way
+Python works, the result of ``__str__()`` is *always* a bytestring, even if you
+accidentally try to return a Unicode object).
+
+You can still create a ``__str__()`` method on your models if you want, of
+course, but you shouldn't need to do this unless you have a good reason.
+Django's ``Model`` base class automatically provides a ``__str__()``
+implementation that calls ``__unicode__()`` and encodes the result into UTF-8.
+This means you'll normally only need to implement a ``__unicode__()`` method
+and let Django handle the coercion to a bytestring when required.
+
+Taking care in ``get_absolute_url()``
+-------------------------------------
+
+URLs can only contain ASCII characters. If you're constructing a URL from
+pieces of data that might be non-ASCII, be careful to encode the results in a
+way that is suitable for a URL. The ``django.db.models.permalink()`` decorator
+handles this for you automatically.
+
+If you're constructing a URL manually (i.e., *not* using the ``permalink()``
+decorator), you'll need to take care of the encoding yourself. In this case,
+use the ``iri_to_uri()`` and ``urlquote()`` functions that were documented
+above_. For example::
+
+    from django.utils.encoding import iri_to_uri
+    from django.utils.http import urlquote
+
+    def get_absolute_url(self):
+        url = u'/person/%s/?x=0&y=0' % urlquote(self.location)
+        return iri_to_uri(url)
+
+This function returns a correctly encoded URL even if ``self.location`` is
+something like "Jack visited Paris & Orléans". (In fact, the ``iri_to_uri()``
+call isn't strictly necessary in the above example, because all the
+non-ASCII characters would have been removed in quoting in the first line.)
+
+.. _above: uri_and_iri_
+
+The database API
+================
+
+You can pass either Unicode strings or UTF-8 bytestrings as arguments to
+``filter()`` methods and the like in the database API. The following two
+querysets are identical::
+
+    qs = People.objects.filter(name__contains=u'Å')
+    qs = People.objects.filter(name__contains='\xc3\85') # UTF-8 encoding of Å
+
+Templates
+=========
+
+You can use either Unicode or bytestrings when creating templates manually::
+
+	from django.template import Template
+	t1 = Template('This is a bytestring template.')
+	t2 = Template(u'This is a Unicode template.')
+
+But the common case is to read templates from the filesystem, and this creates
+a slight complication: not all filesystems store their data encoded as UTF-8.
+If your template files are not stored with a UTF-8 encoding, set the ``FILE_CHARSET``
+setting to the encoding of the files on disk. When Django reads in a template
+file, it will convert the data from this encoding to Unicode. (``FILE_CHARSET``
+is set to ``'utf-8'`` by default.)
+
+The ``DEFAULT_CHARSET`` setting controls the encoding of rendered templates.
+This is set to UTF-8 by default.
+
+Template tags and filters
+-------------------------
+
+A couple of tips to remember when writing your own template tags and filters:
+
+    * Always return Unicode strings from a template tag's ``render()`` method
+      and from template filters.
+
+    * Use ``force_unicode()`` in preference to ``smart_unicode()`` in these
+      places. Tag rendering and filter calls occur as the template is being
+      rendered, so there is no advantage to postponing the conversion of lazy
+      translation objects into strings. It's easier to work solely with Unicode
+      strings at that point.
+
+E-mail
+======
+
+Django's e-mail framework (in ``django.core.mail``) supports Unicode
+transparently. You can use Unicode data in the message bodies and any headers.
+However, you're still obligated to respect the requirements of the e-mail
+specifications, so, for example, e-mail addresses should use only ASCII
+characters.
+
+The following code example demonstrates that everything except e-mail addresses
+can be non-ASCII::
+
+    from django.core.mail import EmailMessage
+
+    subject = u'My visit to Sør-Trøndelag'
+    sender = u'Arnbjörg Ráðormsdóttir <arnbjorg@example.com>'
+    recipients = ['Fred <fred@example.com']
+    body = u'...'
+    EmailMessage(subject, body, sender, recipients).send()
+
+Form submission
+===============
+
+HTML form submission is a tricky area. There's no guarantee that the
+submission will include encoding information, which means the framework might
+have to guess at the encoding of submitted data.
+
+Django adopts a "lazy" approach to decoding form data. The data in an
+``HttpRequest`` object is only decoded when you access it. In fact, most of
+the data is not decoded at all. Only the ``HttpRequest.GET`` and
+``HttpRequest.POST`` data structures have any decoding applied to them. Those
+two fields will return their members as Unicode data. All other attributes and
+methods of ``HttpRequest`` return data exactly as it was submitted by the
+client.
+
+By default, the ``DEFAULT_CHARSET`` setting is used as the assumed encoding
+for form data. If you need to change this for a particular form, you can set
+the ``encoding`` attribute on the ``GET`` and ``POST`` data structures. For
+convenience, changing the ``encoding`` property on an ``HttpRequest`` instance
+does this for you. For example::
+
+    def some_view(request):
+        # We know that the data must be encoded as KOI8-R (for some reason).
+        request.encoding = 'koi8-r'
+        ...
+
+You can even change the encoding after having accessed ``request.GET`` or
+``request.POST``, and all subsequent accesses will use the new encoding.
+
+Most developers won't need to worry about changing form encoding, but this is
+a useful feature for applications that talk to legacy systems whose encoding
+you cannot control.
+
+Django does not decode the data of file uploads, because that data is normally
+treated as collections of bytes, rather than strings. Any automatic decoding
+there would alter the meaning of the stream of bytes.