queryset-refactor: Merged from trunk up to [6689].

git-svn-id: http://code.djangoproject.com/svn/django/branches/queryset-refactor@6690 bcc190cf-cafb-0310-a4f2-bffc1f526a37

8 files changed, 945 insertions, 137 deletions

diff --git a/docs/custom_model_fields.txt b/docs/custom_model_fields.txt
new file mode 100644
index 0000000000..74eb10aa82
--- /dev/null
+++ b/docs/custom_model_fields.txt
@@ -0,0 +1,567 @@
+===================
+Custom Model Fields
+===================
+
+**New in Django development version**
+
+Introduction
+============
+
+The `model reference`_ documentation explains how to use Django's standard
+field classes. For many purposes, those classes are all you'll need. Sometimes,
+though, the Django version won't meet your precise requirements, or you'll want
+to use a field that is entirely different from those shipped with Django.
+
+Django's built-in field types don't cover every possible database column type --
+only the common types, such as ``VARCHAR`` and ``INTEGER``. For more obscure
+column types, such as geographic polygons or even user-created types such as
+`PostgreSQL custom types`_, you can define your own Django ``Field`` subclasses.
+
+Alternatively, you may have a complex Python object that can somehow be
+serialized to fit into a standard database column type. This is another case
+where a ``Field`` subclass will help you use your object with your models.
+
+Our example object
+------------------
+
+Creating custom fields requires a bit of attention to detail. To make things
+easier to follow, we'll use a consistent example throughout this document.
+Suppose you have a Python object representing the deal of cards in a hand of
+Bridge_. It doesn't matter if you don't know how to play Bridge. You only need
+to know that 52 cards are dealt out equally to four players, who are
+traditionally called *north*, *east*, *south* and *west*. Our class looks
+something like this::
+
+    class Hand(object):
+        def __init__(self, north, east, south, west):
+            # Input parameters are lists of cards ('Ah', '9s', etc)
+            self.north = north
+            self.east = east
+            self.south = south
+            self.west = west
+
+        # ... (other possibly useful methods omitted) ...
+
+This is just an ordinary Python class, nothing Django-specific about it. We
+would like to be able to things like this in our models (we assume the
+``hand`` attribute on the model is an instance of ``Hand``)::
+
+
+    example = MyModel.objects.get(pk=1)
+    print example.hand.north
+
+    new_hand = Hand(north, east, south, west)
+    example.hand = new_hand
+    example.save()
+
+We assign to and retrieve from the ``hand`` attribute in our model just like
+any other Python class. The trick is to tell Django how to handle saving and
+loading such an object.
+
+In order to use the ``Hand`` class in our models, we **do not** have to change
+this class at all. This is ideal, because it means you can easily write
+model support for existing classes where you cannot change the source code.
+
+.. note::
+    You might only be wanting to take advantage of custom database column
+    types and deal with the data as standard Python types in your models;
+    strings, or floats, for example. This case is similar to our ``Hand``
+    example and we'll note any differences as we go along.
+
+.. _model reference: ../model_api/
+.. _PostgreSQL custom types: http://www.postgresql.org/docs/8.2/interactive/sql-createtype.html
+.. _Bridge: http://en.wikipedia.org/wiki/Contract_bridge
+
+Background Theory
+=================
+
+Database storage
+----------------
+
+The simplest way to think of a model field is that it provides a way to take a
+normal Python object -- string, boolean, ``datetime``, or something more
+complex like ``Hand`` -- and convert it to and from a format that is useful
+when dealing with the database (and serialization, but, as we'll see later,
+that falls out fairly naturally once you have the database side under control).
+
+Fields in a model must somehow be converted to fit into an existing database
+column type. Different databases provide different sets of valid column types,
+but the rule is still the same: those are the only types you have to work
+with.  Anything you want to store in the database must fit into one of
+those types.
+
+Normally, you're either writing a Django field to match a particular database
+column type, or there's a fairly straightforward way to convert your data to,
+say, a string.
+
+For our ``Hand`` example, we could convert the card data to a string of 104
+characters by concatenating all the cards together in a pre-determined order.
+Say, all the *north* cards first, then the *east*, *south* and *west* cards, in
+that order. So ``Hand`` objects can be saved to text or character columns in
+the database.
+
+What does a field class do?
+---------------------------
+
+All of Django's fields (and when we say *fields* in this document, we always
+mean model fields and not `form fields`_) are subclasses of
+``django.db.models.Field``. Most of the information that Django records about a
+field is common to all fields -- name, help text, validator lists, uniqueness
+and so forth. Storing all that information is handled by ``Field``. We'll get
+into the precise details of what ``Field`` can do later on; for now, suffice it
+to say that everything descends from ``Field`` and then customises key pieces
+of the class behaviour.
+
+.. _form fields: ../newforms/#fields
+
+It's important to realise that a Django field class is not what is stored in
+your model attributes. The model attributes contain normal Python objects. The
+field classes you define in a model are actually stored in the ``Meta`` class
+when the model class is created (the precise details of how this is done are
+unimportant here). This is because the field classes aren't necessary when
+you're just creating and modifying attributes. Instead, they provide the
+machinery for converting between the attribute value and what is stored in the
+database or sent to the serializer.
+
+Keep this in mind when creating your own custom fields. The Django ``Field``
+subclass you write provides the machinery for converting between your Python
+instances and the database/serializer values in various ways (there are
+differences between storing a value and using a value for lookups, for
+example). If this sounds a bit tricky, don't worry. It will hopefully become
+clearer in the examples below. Just remember that you will often end up
+creating two classes when you want a custom field. The first class is the
+Python object that your users will manipulate. They will assign it to the model
+attribute, they will read from it for displaying purposes, things like that.
+This is the ``Hand`` class in our example. The second class is the ``Field``
+subclass. This is the class that knows how to convert your first class back and
+forth between its permanent storage form and the Python form.
+
+Writing a ``Field`` subclass
+=============================
+
+When you are planning your ``Field`` subclass, first give some thought to
+which existing field your new field is most similar to. Can you subclass an
+existing Django field and save yourself some work? If not, you should subclass the ``Field`` class, from which everything is descended.
+
+Initialising your new field is a matter of separating out any arguments that
+are specific to your case from the common arguments and passing the latter to
+the ``__init__()`` method of ``Field`` (or your parent class).
+
+In our example, the Django field we create is going to be called
+``HandField``. It's not a bad idea to use a similar naming scheme to Django's
+fields so that our new class is identifiable and yet clearly related to the
+``Hand`` class it is wrapping. It doesn't behave like any existing field, so
+we'll subclass directly from ``Field``::
+
+    from django.db import models
+
+    class HandField(models.Field):
+        def __init__(self, *args, **kwargs):
+            kwargs['max_length'] = 104
+            super(HandField, self).__init__(*args, **kwargs)
+
+Our ``HandField`` will accept most of the standard field options (see the list
+below), but we ensure it has a fixed length, since it only needs to hold 52
+card values plus their suits; 104 characters in total.
+
+.. note::
+    Many of Django's model fields accept options that they don't do anything
+    with. For example, you can pass both ``editable`` and ``auto_now`` to a
+    ``DateField`` and it will simply ignore the ``editable`` parameter
+    (``auto_now`` being set implies ``editable=False``). No error is raised in
+    this case.
+
+    This behaviour simplifies the field classes, because they don't need to
+    check for options that aren't necessary. They just pass all the options to
+    the parent class and then don't use them later on. It is up to you whether
+    you want your fields to be more strict about the options they select, or
+    to use the simpler, more permissive behaviour of the current fields.
+
+The ``Field.__init__()`` method takes the following parameters, in this
+order:
+
+    - ``verbose_name``
+    - ``name``
+    - ``primary_key``
+    - ``max_length``
+    - ``unique``
+    - ``blank``
+    - ``null``
+    - ``db_index``
+    - ``core``
+    - ``rel``: Used for related fields (like ``ForeignKey``). For advanced use
+      only.
+    - ``default``
+    - ``editable``
+    - ``serialize``: If ``False``, the field will not be serialized when the
+      model is passed to Django's serializers_. Defaults to ``True``.
+    - ``prepopulate_from``
+    - ``unique_for_date``
+    - ``unique_for_month``
+    - ``unique_for_year``
+    - ``validator_list``
+    - ``choices``
+    - ``radio_admin``
+    - ``help_text``
+    - ``db_column``
+    - ``db_tablespace``: Currently only used with the Oracle backend and only
+      for index creation. You can usually ignore this option.
+
+All of the options without an explanation in the above list have the same
+meaning they do for normal Django fields. See the `model documentation`_ for
+examples and details.
+
+.. _serializers: ../serialization/
+.. _model documentation: ../model-api/
+
+The ``SubfieldBase`` metaclass
+------------------------------
+
+As we indicated in the introduction_, field subclasses are often needed for
+two reasons. Either to take advantage of a custom database column type, or to
+handle complex Python types. A combination of the two is obviously also
+possible. If you are only working with custom database column types and your
+model fields appear in Python as standard Python types direct from the
+database backend, you don't need to worry about this section.
+
+If you are handling custom Python types, such as our ``Hand`` class, we need
+to make sure that when Django initialises an instance of our model and assigns
+a database value to our custom field attribute we convert that value into the
+appropriate Python object. The details of how this happens internally are a
+little complex. For the field writer, though, things are fairly simple. Make
+sure your field subclass uses ``django.db.models.SubfieldBase`` as its
+metaclass. This ensures that the ``to_python()`` method, documented below_,
+will always be called when the attribute is initialised.
+
+Our ``HandField`` class now looks like this::
+
+    class HandField(models.Field):
+        __metaclass__ = models.SubfieldBase
+
+        def __init__(self, *args, **kwargs):
+            # ...
+
+.. _below: #to-python-self-value
+
+Useful methods
+--------------
+
+Once you've created your ``Field`` subclass and setup up the
+``__metaclass__``, if necessary, there are a few standard methods you need to
+consider overriding. Which of these you need to implement will depend on you
+particular field behaviour. The list below is in approximately decreasing
+order of importance, so start from the top.
+
+``db_type(self)``
+~~~~~~~~~~~~~~~~~
+
+Returns the database column data type for the ``Field``, taking into account
+the current ``DATABASE_ENGINE`` setting.
+
+Say you've created a PostgreSQL custom type called ``mytype``. You can use this
+field with Django by subclassing ``Field`` and implementing the ``db_type()``
+method, like so::
+
+    from django.db import models
+
+    class MytypeField(models.Field):
+        def db_type(self):
+            return 'mytype'
+
+Once you have ``MytypeField``, you can use it in any model, just like any other
+``Field`` type::
+
+    class Person(models.Model):
+        name = models.CharField(max_length=80)
+        gender = models.CharField(max_length=1)
+        something_else = MytypeField()
+
+If you aim to build a database-agnostic application, you should account for
+differences in database column types. For example, the date/time column type
+in PostgreSQL is called ``timestamp``, while the same column in MySQL is called
+``datetime``. The simplest way to handle this in a ``db_type()`` method is to
+import the Django settings module and check the ``DATABASE_ENGINE`` setting.
+For example::
+
+    class MyDateField(models.Field):
+        def db_type(self):
+            from django.conf import settings
+            if settings.DATABASE_ENGINE == 'mysql':
+                return 'datetime'
+            else:
+                return 'timestamp'
+
+The ``db_type()`` method is only called by Django when the framework constructs
+the ``CREATE TABLE`` statements for your application -- that is, when you first
+create your tables. It's not called at any other time, so it can afford to
+execute slightly complex code, such as the ``DATABASE_ENGINE`` check in the
+above example.
+
+Some database column types accept parameters, such as ``CHAR(25)``, where the
+parameter ``25`` represents the maximum column length. In cases like these,
+it's more flexible if the parameter is specified in the model rather than being
+hard-coded in the ``db_type()`` method. For example, it wouldn't make much
+sense to have a ``CharMaxlength25Field``, shown here::
+
+    # This is a silly example of hard-coded parameters.
+    class CharMaxlength25Field(models.Field):
+        def db_type(self):
+            return 'char(25)'
+
+    # In the model:
+    class MyModel(models.Model):
+        # ...
+        my_field = CharMaxlength25Field()
+
+The better way of doing this would be to make the parameter specifiable at run
+time -- i.e., when the class is instantiated. To do that, just implement
+``__init__()``, like so::
+
+    # This is a much more flexible example.
+    class BetterCharField(models.Field):
+        def __init__(self, max_length, *args, **kwargs):
+            self.max_length = max_length
+            super(BetterCharField, self).__init__(*args, **kwargs)
+
+        def db_type(self):
+            return 'char(%s)' % self.max_length
+
+    # In the model:
+    class MyModel(models.Model):
+        # ...
+        my_field = BetterCharField(25)
+
+Finally, if your column requires truly complex SQL setup, return ``None`` from
+``db_type()``. This will cause Django's SQL creation code to skip over this
+field. You are then responsible for creating the column in the right table in
+some other way, of course, but this gives you a way to tell Django to get out
+of the way.
+
+
+``to_python(self, value)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Converts between all the ways your field can receive its initial value and the
+Python object you want to end up with. The default version just returns
+``value``, so is useful is the database backend returns the data already in
+the correct form (a Python string, for example).
+
+Normally, you will need to override this method. As a general rule, be
+prepared to accept an instance of the right type (e.g. ``Hand`` in our ongoing
+example), a string (from a deserializer, for example), and whatever the
+database wrapper returns for the column type you are using.
+
+In our ``HandField`` class, we are storing the data in a character field in
+the database, so we need to be able to process strings and ``Hand`` instances
+in ``to_python()``::
+
+    class HandField(models.Field):
+        # ...
+
+        def to_python(self, value):
+            if isinstance(value, Hand):
+                return value
+
+            # The string case
+            p1 = re.compile('.{26}')
+            p2 = re.compile('..')
+            args = [p2.findall(x) for x in p1.findall(value)]
+            return Hand(*args)
+
+Notice that we always return a ``Hand`` instance from this method. That is the
+Python object we want to store in the model's attribute.
+
+``get_db_prep_save(self, value)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is the reverse of ``to_python()`` when working with the database backends
+(as opposed to serialization). The ``value`` parameter is the current value of
+the model's attribute (a field has no reference to its containing model, so it
+cannot retrieve the value itself) and the method should return data in a
+format that can be used as a parameter in a query for the database backend.
+
+For example::
+
+    class HandField(models.Field):
+        # ...
+
+        def get_db_prep_save(self, value):
+            return ''.join([''.join(l) for l in (self.north,
+                    self.east, self.south, self.west)])
+
+
+``pre_save(self, model_instance, add)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This method is called just prior to ``get_db_prep_save()`` and should return
+the value of the appropriate attribute from ``model_instance`` for this field.
+The attribute name is in ``self.attname`` (this is set up by ``Field``). If
+the model is being saved to the database for the first time, the ``add``
+parameter will be ``True``, otherwise it will be ``False``.
+
+Often you won't need to override this method. However, at times it can be very
+useful. For example, the Django ``DateTimeField`` uses this method to set the
+attribute to the correct value before returning it in the cases when
+``auto_now`` or ``auto_now_add`` are set on the field.
+
+If you do override this method, you must return the value of the attribute at
+the end. You should also update the model's attribute if you make any changes
+to the value so that code holding references to the model will always see the
+correct value.
+
+``get_db_prep_lookup(self, lookup_type, value)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Prepares the ``value`` for passing to the database when used in a lookup (a
+``WHERE`` constraint in SQL). The ``lookup_type`` will be one of the valid
+Django filter lookups: ``exact``, ``iexact``, ``contains``, ``icontains``,
+``gt``, ``gte``, ``lt``, ``lte``, ``in``, ``startswith``, ``istartswith``,
+``endswith``, ``iendswith``, ``range``, ``year``, ``month``, ``day``,
+``isnull``, ``search``, ``regex``, and ``iregex``.
+
+Your method must be prepared to handle all of these ``lookup_type`` values and
+should raise either a ``ValueError`` if the ``value`` is of the wrong sort (a
+list when you were expecting an object, for example) or a ``TypeError`` if
+your field does not support that type of lookup. For many fields, you can get
+by with handling the lookup types that need special handling for your field
+and pass the rest of the ``get_db_prep_lookup()`` method of the parent class.
+
+If you needed to implement ``get_db_prep_save()``, you will usually need to
+implement ``get_db_prep_lookup()``. The usual reason is because of the
+``range``  and ``in`` lookups. In these case, you will passed a list of
+objects (presumably of the right type) and will need to convert them to a list
+of things of the right type for passing to the database. Sometimes you can
+reuse ``get_db_prep_save()``, or at least factor out some common pieces from
+both methods into a help function.
+
+For example::
+
+    class HandField(models.Field):
+        # ...
+
+        def get_db_prep_lookup(self, lookup_type, value):
+            # We only handle 'exact' and 'in'. All others are errors.
+            if lookup_type == 'exact':
+                return self.get_db_prep_save(value)
+            elif lookup_type == 'in':
+                return [self.get_db_prep_save(v) for v in value]
+            else:
+                raise TypeError('Lookup type %r not supported.' % lookup_type)
+
+
+``formfield(self, form_class=forms.CharField, **kwargs)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the default form field to use when this field is displayed
+in a model. This method is called by the `helper functions`_
+``form_for_model()`` and ``form_for_instance()``.
+
+All of the ``kwargs`` dictionary is passed directly to the form field's
+``__init__()`` method. Normally, all you need to do is set up a good default
+for the ``form_class`` argument and then delegate further handling to the
+parent class. This might require you to write a custom form field (and even a
+form widget). See the `forms documentation`_ for information about this. Also
+have a look at ``django.contrib.localflavor`` for some examples of custom
+widgets.
+
+Continuing our ongoing example, we can write the ``formfield()`` method as::
+
+    class HandField(models.Field):
+        # ...
+
+        def formfield(self, **kwargs):
+            # This is a fairly standard way to set up some defaults
+            # whilst letting the caller override them.
+            defaults = {'form_class': MyFormField}
+            defaults.update(kwargs)
+            return super(HandField, self).formfield(**defaults)
+
+This assumes we have some ``MyFormField`` field class (which has its own
+default widget) imported. This document doesn't cover the details of writing
+custom form fields.
+
+.. _helper functions: ../newforms/#generating-forms-for-models
+.. _forms documentation: ../newforms/
+
+``get_internal_type(self)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Returns a string giving the name of the ``Field`` subclass we are emulating at
+the database level. This is used to determine the type of database column for
+simple cases.
+
+If you have created a ``db_type()`` method, you do not need to worry about
+``get_internal_type()`` -- it won't be used much. Sometimes, though, your
+database storage is similar in type to some other field, so you can use that
+other field's logic to create the right column.
+
+For example::
+
+    class HandField(models.Field):
+        # ...
+
+        def get_internal_type(self):
+            return 'CharField'
+
+No matter which database backend we are using, this will mean that ``syncdb``
+and other SQL commands create the right column type for storing a string.
+
+If ``get_internal_type()`` returns a string that is not known to Django for
+the database backend you are using -- that is, it doesn't appear in
+``django.db.backends.<db_name>.creation.DATA_TYPES`` -- the string will still
+be used by the serializer, but the default ``db_type()`` method will return
+``None``. See the documentation of ``db_type()`` above_ for reasons why this
+might be useful. Putting a descriptive string in as the type of the field for
+the serializer is a useful idea if you are ever going to be using the
+serializer output in some other place, outside of Django.
+
+.. _above: #db-type-self
+
+``flatten_data(self, follow, obj=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. admonition:: Subject to change
+
+    Although implementing this method is necessary to allow field
+    serialization, the API might change in the future.
+
+Returns a dictionary, mapping the field's attribute name to a flattened string
+version of the data. This method has some internal uses that aren't of
+interest to use here (mostly having to do with manipulators). For our
+purposes, it is sufficient to return a one item dictionary that maps the
+attribute name to a string.
+
+This method is used by the serializers to convert the field into a string for
+output. You can ignore the input parameters for serialization purposes,
+although calling ``Field._get_val_from_obj(obj)`` is the best way to get the
+value to serialize.
+
+For example, since our ``HandField`` uses strings for its data storage anyway,
+we can reuse some existing conversion code::
+
+    class HandField(models.Field):
+        # ...
+
+        def flatten_data(self, follow, obj=None):
+            value = self._get_val_from_obj(obj)
+            return {self.attname: self.get_db_prep_save(value)}
+
+Some general advice
+--------------------
+
+Writing a custom field can be a tricky process sometimes, particularly if you
+are doing complex conversions between your Python types and your database and
+serialization formats. A couple of tips to make things go more smoothly:
+
+ 1. Look at the existing Django fields (in
+ ``django/db/models/fields/__init__.py``) for inspiration. Try to find a field
+ that is already close to what you want and extend it a little bit, in
+ preference to creating an entirely new field from scratch.
+
+ 2. Put a ``__str__()`` or ``__unicode__()`` method on the class you are
+ wrapping up as a field. There are a lot of places where the default behaviour
+ of the field code is to call ``force_unicode()`` on the value (in our
+ examples in this document, ``value`` would be a ``Hand`` instance, not a
+ ``HandField``). So if your ``__unicode__()`` method automatically converts to
+ the string form of your Python object, you can save yourself a lot of work.
+
diff --git a/docs/flatpages.txt b/docs/flatpages.txt
index d082090689..7c27fe8793 100644
--- a/docs/flatpages.txt
+++ b/docs/flatpages.txt
@@ -113,3 +113,10 @@ Here's a sample ``flatpages/default.html`` template::
     {{ flatpage.content }}
     </body>
     </html>
+
+Since you're already entering raw HTML into the admin page for a flatpage,
+both ``flatpage.title`` and ``flatpage.content`` are marked as **not**
+requiring `automatic HTML escaping`_  in the template.
+
+.. _automatic HTML escaping: ../templates/#automatic-html-escaping
+
diff --git a/docs/middleware.txt b/docs/middleware.txt
index 30a4899a3e..41b1a96b89 100644
--- a/docs/middleware.txt
+++ b/docs/middleware.txt
@@ -104,8 +104,7 @@ Handles conditional GET operations. If the response has a ``ETag`` or
 ``Last-Modified`` header, and the request has ``If-None-Match`` or
 ``If-Modified-Since``, the response is replaced by an HttpNotModified.
 
-Also removes the content from any response to a HEAD request and sets the
-``Date`` and ``Content-Length`` response-headers.
+Also sets the ``Date`` and ``Content-Length`` response-headers.
 
 django.middleware.http.SetRemoteAddrFromForwardedFor
 ----------------------------------------------------
diff --git a/docs/model-api.txt b/docs/model-api.txt
index b49963d8f5..ca84c84d09 100644
--- a/docs/model-api.txt
+++ b/docs/model-api.txt
@@ -1013,111 +1013,12 @@ See the `One-to-one relationship model example`_ for a full example.
 Custom field types
 ------------------
 
-**New in Django development version**
-
-Django's built-in field types don't cover every possible database column type --
-only the common types, such as ``VARCHAR`` and ``INTEGER``. For more obscure
-column types, such as geographic polygons or even user-created types such as
-`PostgreSQL custom types`_, you can define your own Django ``Field`` subclasses.
-
-.. _PostgreSQL custom types: http://www.postgresql.org/docs/8.2/interactive/sql-createtype.html
-
-.. admonition:: Experimental territory
-
-    This is an area of Django that traditionally has not been documented, but
-    we're starting to include bits of documentation, one feature at a time.
-    Please forgive the sparseness of this section.
-
-    If you like living on the edge and are comfortable with the risk of
-    unstable, undocumented APIs, see the code for the core ``Field`` class
-    in ``django/db/models/fields/__init__.py`` -- but if/when the innards
-    change, don't say we didn't warn you.
-
-To create a custom field type, simply subclass ``django.db.models.Field``.
-Here is an incomplete list of the methods you should implement:
-
-``db_type()``
-~~~~~~~~~~~~~
-
-Returns the database column data type for the ``Field``, taking into account
-the current ``DATABASE_ENGINE`` setting.
-
-Say you've created a PostgreSQL custom type called ``mytype``. You can use this
-field with Django by subclassing ``Field`` and implementing the ``db_type()``
-method, like so::
-
-    from django.db import models
-
-    class MytypeField(models.Field):
-        def db_type(self):
-            return 'mytype'
-
-Once you have ``MytypeField``, you can use it in any model, just like any other
-``Field`` type::
-
-    class Person(models.Model):
-        name = models.CharField(max_length=80)
-        gender = models.CharField(max_length=1)
-        something_else = MytypeField()
-
-If you aim to build a database-agnostic application, you should account for
-differences in database column types. For example, the date/time column type
-in PostgreSQL is called ``timestamp``, while the same column in MySQL is called
-``datetime``. The simplest way to handle this in a ``db_type()`` method is to
-import the Django settings module and check the ``DATABASE_ENGINE`` setting.
-For example::
-
-    class MyDateField(models.Field):
-        def db_type(self):
-            from django.conf import settings
-            if settings.DATABASE_ENGINE == 'mysql':
-                return 'datetime'
-            else:
-                return 'timestamp'
-
-The ``db_type()`` method is only called by Django when the framework constructs
-the ``CREATE TABLE`` statements for your application -- that is, when you first
-create your tables. It's not called at any other time, so it can afford to
-execute slightly complex code, such as the ``DATABASE_ENGINE`` check in the
-above example.
-
-Some database column types accept parameters, such as ``CHAR(25)``, where the
-parameter ``25`` represents the maximum column length. In cases like these,
-it's more flexible if the parameter is specified in the model rather than being
-hard-coded in the ``db_type()`` method. For example, it wouldn't make much
-sense to have a ``CharMaxlength25Field``, shown here::
-
-    # This is a silly example of hard-coded parameters.
-    class CharMaxlength25Field(models.Field):
-        def db_type(self):
-            return 'char(25)'
-
-    # In the model:
-    class MyModel(models.Model):
-        # ...
-        my_field = CharMaxlength25Field()
-
-The better way of doing this would be to make the parameter specifiable at run
-time -- i.e., when the class is instantiated. To do that, just implement
-``__init__()``, like so::
-
-    # This is a much more flexible example.
-    class BetterCharField(models.Field):
-        def __init__(self, max_length, *args, **kwargs):
-            self.max_length = max_length
-            super(BetterCharField, self).__init__(*args, **kwargs)
-
-        def db_type(self):
-            return 'char(%s)' % self.max_length
-
-    # In the model:
-    class MyModel(models.Model):
-        # ...
-        my_field = BetterCharField(25)
+If one of the existing model fields cannot be used to fit your purposes, or if
+you wish to take advantage of some less common database column types, you can
+create your own field class. Full coverage of creating your own fields is
+provided in the `Custom Model Fields`_ documentation.
 
-Note that if you implement ``__init__()`` on a ``Field`` subclass, it's
-important to call ``Field.__init__()`` -- i.e., the parent class'
-``__init__()`` method.
+.. _Custom Model Fields: ../custom_model_fields/
 
 Meta options
 ============
diff --git a/docs/serialization.txt b/docs/serialization.txt
index fa9b4edd51..cf8b196931 100644
--- a/docs/serialization.txt
+++ b/docs/serialization.txt
@@ -47,14 +47,14 @@ This is useful if you want to serialize data directly to a file-like object
 Subset of fields
 ~~~~~~~~~~~~~~~~
 
-If you only want a subset of fields to be serialized, you can 
+If you only want a subset of fields to be serialized, you can
 specify a ``fields`` argument to the serializer::
 
     from django.core import serializers
     data = serializers.serialize('xml', SomeModel.objects.all(), fields=('name','size'))
 
 In this example, only the ``name`` and ``size`` attributes of each model will
-be serialized. 
+be serialized.
 
 .. note::
 
@@ -111,9 +111,9 @@ Django "ships" with a few included serializers:
     ``python``  Translates to and from "simple" Python objects (lists, dicts,
                 strings, etc.).  Not really all that useful on its own, but
                 used as a base for other serializers.
-    
-    ``yaml``    Serializes to YAML (Yet Another Markup Lanuage). This 
-                serializer is only available if PyYAML_ is installed. 
+
+    ``yaml``    Serializes to YAML (Yet Another Markup Lanuage). This
+                serializer is only available if PyYAML_ is installed.
     ==========  ==============================================================
 
 .. _json: http://json.org/
@@ -135,6 +135,23 @@ For example::
     json_serializer = serializers.get_serializer("json")()
     json_serializer.serialize(queryset, ensure_ascii=False, stream=response)
 
+Django ships with a copy of simplejson_ in the source. Be aware, that if
+you're using that for serializing directly that not all Django output can be
+passed unmodified to simplejson. In particular, `lazy translation objects`_
+need a `special encoder`_ written for them. Something like this will work::
+
+    from django.utils.functional import Promise
+    from django.utils.encoding import force_unicode
+
+    class LazyEncoder(simplejson.JSONEncoder):
+        def default(self, obj):
+            if isinstance(obj, Promise):
+                return force_unicode(obj)
+            return obj
+
+.. _lazy translation objects: ../i18n/#lazy-translation
+.. _special encoder: http://svn.red-bean.com/bob/simplejson/tags/simplejson-1.7/docs/index.html
+
 Writing custom serializers
 ``````````````````````````
 
diff --git a/docs/sitemaps.txt b/docs/sitemaps.txt
index 1d4fba2626..eb749dda2f 100644
--- a/docs/sitemaps.txt
+++ b/docs/sitemaps.txt
@@ -47,7 +47,7 @@ Initialization
 ==============
 
 To activate sitemap generation on your Django site, add this line to your
-URLconf_:
+URLconf_::
 
     (r'^sitemap.xml$', 'django.contrib.sitemaps.views.sitemap', {'sitemaps': sitemaps})
 
diff --git a/docs/templates.txt b/docs/templates.txt
index 5d5f657747..07258bb46a 100644
--- a/docs/templates.txt
+++ b/docs/templates.txt
@@ -280,7 +280,9 @@ Here are some tips for working with inheritance:
     * If you need to get the content of the block from the parent template,
       the ``{{ block.super }}`` variable will do the trick. This is useful
       if you want to add to the contents of a parent block instead of
-      completely overriding it.
+      completely overriding it. Data inserted using ``{{ block.super }}`` will
+      not be automatically escaped (see the `next section`_), since it was
+      already escaped, if necessary, in the parent template.
 
     * For extra readability, you can optionally give a *name* to your
       ``{% endblock %}`` tag. For example::
@@ -299,6 +301,132 @@ it also defines the content that fills the hole in the *parent*. If there were
 two similarly-named ``{% block %}`` tags in a template, that template's parent
 wouldn't know which one of the blocks' content to use.
 
+.. _next section: #automatic-html-escaping
+
+Automatic HTML escaping
+=======================
+
+**New in Django development version**
+
+A very real problem when creating HTML (and other) output using templates and
+variable substitution is the possibility of accidently inserting some variable
+value that affects the resulting HTML. For example, a template fragment such as
+::
+
+    Hello, {{ name }}.
+
+seems like a harmless way to display the user's name. However, if you are
+displaying data that the user entered directly and they had entered their name as ::
+
+    <script>alert('hello')</script>
+
+this would always display a Javascript alert box when the page was loaded.
+Similarly, if you were displaying some data generated by another process and it
+contained a '<' symbol, you couldn't just dump this straight into your HTML,
+because it would be treated as the start of an element. The effects of these
+sorts of problems can vary from merely annoying to allowing exploits via `Cross
+Site Scripting`_ (XSS) attacks.
+
+.. _Cross Site Scripting: http://en.wikipedia.org/wiki/Cross-site_scripting
+
+In order to provide some protection against these problems, Django
+provides automatic (but controllable) HTML escaping for data coming from
+tempate variables. Inside this tag, any data that comes from template
+variables is examined to see if it contains one of the five HTML characters
+(<, >, ', " and &) that often need escaping and those characters are converted
+to their respective HTML entities. It causes no harm if a character is
+converted to an entity when it doesn't need to be, so all five characters are
+always converted.
+
+Since some variables will contain data that is *intended* to be rendered
+as HTML, template tag and filter writers can mark their output strings as
+requiring no further escaping. For example, the ``unordered_list`` filter is
+designed to return raw HTML and we want the template processor to simply
+display the results as returned, without applying any escaping. That is taken
+care of by the filter. The template author need do nothing special in that
+case.
+
+By default, automatic HTML escaping is always applied. However, sometimes you
+will not want this to occur (for example, if you're using the templating
+system to create an email). To control automatic escaping inside your template,
+wrap the affected content in the ``autoescape`` tag, like so::
+
+    {% autoescape off %}
+        Hello {{ name }}
+    {% endautoescape %}
+
+The auto-escaping tag passes its effect onto templates that extend the
+current one as well as templates included via the ``include`` tag, just like
+all block tags.
+
+The ``autoescape`` tag takes either ``on`` or ``off`` as its argument. At times, you might want to force auto-escaping when it would otherwise be disabled. For example::
+
+    Auto-escaping is on by default. Hello {{ name }}
+
+    {% autoescape off %}
+        This will not be auto-escaped: {{ data }}.
+
+        Nor this: {{ other_data }}
+        {% autoescape on %}
+            Auto-escaping applies again, {{ name }}
+        {% endautoescape %}
+    {% endautoescape %}
+
+For individual variables, the ``safe`` filter can also be used to indicate
+that the contents should not be automatically escaped::
+
+    This will be escaped: {{ data }}
+    This will not be escaped: {{ data|safe }}
+
+Think of *safe* as shorthand for *safe from further escaping* or *can be
+safely interpreted as HTML*. In this example, if ``data`` contains ``'<a>'``,
+the output will be::
+
+    This will be escaped: &lt;a&gt;
+    This will not be escaped: <a>
+
+Generally, you won't need to worry about auto-escaping very much. View
+developers and custom filter authors need to think about when their data
+shouldn't be escaped and mark it appropriately. They are in a better position
+to know when that should happen than the template author, so it is their
+responsibility. By default, all output is escaped unless the template
+processor is explicitly told otherwise.
+
+You should also note that if you are trying to write a template that might be
+used in situations where automatic escaping is enabled or disabled and you
+don't know which (such as when your template is included in other templates),
+you can safely write as if you were in an ``{% autoescape off %}`` situation.
+Scatter ``escape`` filters around for any variables that need escaping. When
+auto-escaping is on, these extra filters won't change the output -- any
+variables that use the ``escape`` filter do not have further automatic
+escaping applied to them.
+
+String literals and automatic escaping
+--------------------------------------
+
+Sometimes you will pass a string literal as an argument to a filter. For
+example::
+
+    {{ data|default:"This is a string literal." }}
+
+All string literals are inserted **without** any automatic escaping into the
+template, if they are used (it's as if they were all passed through the
+``safe`` filter). The reasoning behind this is that the template author is in
+control of what goes into the string literal, so they can make sure the text
+is correctly escaped when the template is written.
+
+This means you would write ::
+
+    {{ data|default:"3 &gt; 2" }}
+
+...rather than ::
+
+    {{ data|default:"3 > 2" }}  <-- Bad! Don't do this.
+
+This doesn't affect what happens to data coming from the variable itself.
+The variable's contents are still automatically escaped, if necessary, since
+they're beyond the control of the template author.
+
 Using the built-in reference
 ============================
 
@@ -374,6 +502,24 @@ available, and what they do.
 Built-in tag reference
 ----------------------
 
+autoescape
+~~~~~~~~~~
+
+**New in Django development version**
+
+Control the current auto-escaping behaviour. This tag takes either ``on`` or
+``off`` as an argument and that determines whether auto-escaping is in effect
+inside the block.
+
+When auto-escaping is in effect, all variable content has HTML escaping applied
+to it before placing the result into the output (but after any filters have
+been applied). This is equivalent to manually applying the ``escape`` filter
+attached to each variable.
+
+The only exceptions are variables that are already marked as 'safe' from
+escaping, either by the code that populated the variable, or because it has
+the ``safe`` or ``escape`` filters applied.
+
 block
 ~~~~~
 
@@ -452,7 +598,7 @@ just like in variable syntax.
 
 Sample usage::
 
-    {% filter escape|lower %}
+    {% filter force_escape|lower %}
         This text will be HTML-escaped, and will appear in all lowercase.
     {% endfilter %}
 
@@ -740,7 +886,7 @@ Available format strings:
                       if they're zero and the special-case
                       strings 'midnight' and 'noon' if
                       appropriate. Proprietary extension.
-    r                 RFC 822 formatted date.                   ``'Thu, 21 Dec 2000 16:01:07 +0200'``
+    r                 RFC 2822 formatted date.                  ``'Thu, 21 Dec 2000 16:01:07 +0200'``
     s                 Seconds, 2 digits with leading zeros.     ``'00'`` to ``'59'``
     S                 English ordinal suffix for day of the     ``'st'``, ``'nd'``, ``'rd'`` or ``'th'``
                       month, 2 characters.
@@ -1076,6 +1222,10 @@ Returns true if the value is divisible by the argument.
 escape
 ~~~~~~
 
+**New in Django development version:** The behaviour of this filter has
+changed slightly in the development version (the affects are only applied
+once, after all other filters).
+
 Escapes a string's HTML. Specifically, it makes these replacements:
 
     * ``"&"`` to ``"&amp;"``
@@ -1084,6 +1234,16 @@ Escapes a string's HTML. Specifically, it makes these replacements:
     * ``'"'`` (double quote) to ``'&quot;'``
     * ``"'"`` (single quote) to ``'&#39;'``
 
+The escaping is only applied when the string is output, so it does not matter
+where in a chained sequence of filters you put ``escape``: it will always be
+applied as though it were the last filter. If you want escaping to be applied
+immediately, use the ``force_escape`` filter.
+
+Applying ``escape`` to a variable that would normally have auto-escaping
+applied to the result will only result in one round of escaping being done. So
+it is safe to use this function even in auto-escaping environments. If you want
+multiple escaping passes to be applied, use the ``force_escape`` filter.
+
 filesizeformat
 ~~~~~~~~~~~~~~
 
@@ -1106,25 +1266,50 @@ floatformat
 When used without an argument, rounds a floating-point number to one decimal
 place -- but only if there's a decimal part to be displayed. For example:
 
-    * ``36.123`` gets converted to ``36.1``
-    * ``36.15`` gets converted to ``36.2``
-    * ``36`` gets converted to ``36``
+========  =======================  ======
+value     Template                 Output
+========  =======================  ======
+34.23234  {{ value|floatformat }}  34.2
+34.00000  {{ value|floatformat }}  34
+34.26000  {{ value|floatformat }}  34.3
+========  =======================  ======
+
+If used with a numeric integer argument, ``floatformat`` rounds a number to
+that many decimal places.  For example:
 
-If used with a numeric integer argument, ``floatformat`` rounds a number to that
-many decimal places.  For example:
+========  =========================  ======
+value     Template                   Output
+========  =========================  ======
+34.23234  {{ value|floatformat:3 }}  34.232
+34.00000  {{ value|floatformat:3 }}  34.000
+34.26000  {{ value|floatformat:3 }}  34.260
+========  =========================  ======
 
-    * ``36.1234`` with floatformat:3 gets converted to ``36.123``
-    * ``36`` with floatformat:4 gets converted to ``36.0000``
+If the argument passed to ``floatformat`` is negative, it will round a number
+to that many decimal places -- but only if there's a decimal part to be
+displayed. For example:
 
-If the argument passed to ``floatformat`` is negative, it will round a number to
-that many decimal places -- but only if there's a decimal part to be displayed.
-For example:
+========  ============================  ======
+value     Template                      Output
+========  ============================  ======
+34.23234  {{ value|floatformat:"-3" }}  34.232
+34.00000  {{ value|floatformat:"-3" }}  34
+34.26000  {{ value|floatformat:"-3" }}  34.260
+========  ============================  ======
 
-    * ``36.1234`` with floatformat:-3 gets converted to ``36.123``
-    * ``36`` with floatformat:-4 gets converted to ``36``
+Using ``floatformat`` with no argument is equivalent to using ``floatformat``
+with an argument of ``-1``.
 
-Using ``floatformat`` with no argument is equivalent to using ``floatformat`` with
-an argument of ``-1``.
+force_escape
+~~~~~~~~~~~~
+
+**New in Django development version**
+
+Applies HTML escaping to a string (see the ``escape`` filter for details).
+This filter is applied *immediately* and returns a new, escaped string. This
+is useful in the rare cases where you need multiple escaping or want to apply
+other filters to the escaped results. Normally, you want to use the ``escape``
+filter.
 
 get_digit
 ~~~~~~~~~
@@ -1250,6 +1435,12 @@ Right-aligns the value in a field of a given width.
 
 **Argument:** field size
 
+safe
+~~~~
+
+Marks a string as not requiring further HTML escaping prior to output. When
+autoescaping is off, this filter has no effect.
+
 slice
 ~~~~~
 
diff --git a/docs/templates_python.txt b/docs/templates_python.txt
index bd105888ce..e4658f6461 100644
--- a/docs/templates_python.txt
+++ b/docs/templates_python.txt
@@ -219,13 +219,13 @@ be replaced with the name of the invalid variable.
 
     While ``TEMPLATE_STRING_IF_INVALID`` can be a useful debugging tool,
     it is a bad idea to turn it on as a 'development default'.
-    
+
     Many templates, including those in the Admin site, rely upon the
     silence of the template system when a non-existent variable is
     encountered. If you assign a value other than ``''`` to
     ``TEMPLATE_STRING_IF_INVALID``, you will experience rendering
     problems with these templates and sites.
-    
+
     Generally, ``TEMPLATE_STRING_IF_INVALID`` should only be enabled
     in order to debug a specific template problem, then cleared
     once debugging is complete.
@@ -722,6 +722,95 @@ decorator instead::
 If you leave off the ``name`` argument, as in the second example above, Django
 will use the function's name as the filter name.
 
+Filters and auto-escaping
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**New in Django development version**
+
+When you are writing a custom filter, you need to give some thought to how
+this filter will interact with Django's auto-escaping behaviour.  Firstly, you
+should realise that there are three types of strings that can be passed around
+inside the template code:
+
+ * raw strings are the native Python ``str`` or ``unicode`` types. On
+   output, they are escaped if auto-escaping is in effect and presented
+   unchanged, otherwise.
+
+ * "safe" strings are strings that are safe from further escaping at output
+   time. Any necessary escaping has already been done. They are commonly used
+   for output that contains raw HTML that is intended to be intrepreted on the
+   client side.
+
+   Internally, these strings are of type ``SafeString`` or ``SafeUnicode``,
+   although they share a common base class in ``SafeData``, so you can test
+   for them using code like::
+
+    if isinstance(value, SafeData):
+        # Do something with the "safe" string.
+
+ * strings which are marked as "needing escaping" are *always* escaped on
+   output, regardless of whether they are in an ``autoescape`` block or not.
+   These strings are only escaped once, however, even if auto-escaping
+   applies. This type of string is internally represented by the types
+   ``EscapeString`` and ``EscapeUnicode``. You will not normally need to worry
+   about these; they exist for the implementation of the ``escape`` filter.
+
+Inside your filter, you will need to think about three areas in order to be
+auto-escaping compliant:
+
+ 1. If your filter returns a string that is ready for direct output (it should
+    be considered a "safe" string), you should call
+    ``django.utils.safestring.mark_safe()`` on the result prior to returning.
+    This will turn the result into the appropriate ``SafeData`` type. This is
+    often the case when you are returning raw HTML, for example.
+
+ 2. If your filter is given a "safe" string, is it guaranteed to return a
+    "safe" string? If so, set the ``is_safe`` attribute on the function to be
+    ``True``. For example, a filter that replaced a word consisting only of
+    digits with the number spelt out in words is going to be
+    safe-string-preserving, since it cannot introduce any of the five dangerous
+    characters: <, >, ", ' or &. We can write::
+
+        @register.filter
+        def convert_to_words(value):
+            # ... implementation here ...
+            return result
+
+        convert_to_words.is_safe = True
+
+    Note that this filter does not return a universally safe result (it does
+    not return ``mark_safe(result)``) because if it is handed a raw string such
+    as '<a>', this will need further escaping in an auto-escape environment.
+    The ``is_safe`` attribute only talks about the the result when a safe
+    string is passed into the filter.
+
+ 3. Will your filter behave differently depending upon whether auto-escaping
+    is currently in effect or not? This is normally a concern when you are
+    returning mixed content (HTML elements mixed with user-supplied content).
+    For example, the ``ordered_list`` filter that ships with Django needs to
+    know whether to escape its content or not. It will always return a safe
+    string. Since it returns raw HTML, we cannot apply escaping to the
+    result -- it needs to be done in-situ.
+
+    For these cases, the filter function needs to be told what the current
+    auto-escaping setting is. Set the ``needs_autoescape`` attribute on the
+    filter to ``True`` and have your function take an extra argument called
+    ``autoescape`` with a default value of ``None``. When the filter is called,
+    the ``autoescape`` keyword argument will be ``True`` if auto-escaping is in
+    effect. For example, the ``unordered_list`` filter is written as::
+
+        def unordered_list(value, autoescape=None):
+            # ... lots of code here ...
+
+            return mark_safe(...)
+
+        unordered_list.is_safe = True
+        unordered_list.needs_autoescape = True
+
+By default, both the ``is_safe`` and ``needs_autoescape`` attributes are
+``False``. You do not need to specify them if ``False`` is an acceptable
+value.
+
 Writing custom template tags
 ----------------------------
 
@@ -840,6 +929,43 @@ Ultimately, this decoupling of compilation and rendering results in an
 efficient template system, because a template can render multiple context
 without having to be parsed multiple times.
 
+Auto-escaping considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The output from template tags is not automatically run through the
+auto-escaping filters. However, there are still a couple of things you should
+keep in mind when writing a template tag:
+
+If the ``render()`` function of your template stores the result in a context
+variable (rather than returning the result in a string), it should take care
+to call ``mark_safe()`` if appropriate. When the variable is ultimately
+rendered, it will be affected by the auto-escape setting in effect at the
+time, so content that should be safe from further escaping needs to be marked
+as such.
+
+Also, if your template tag creates a new context for performing some
+sub-rendering, you should be careful to set the auto-escape attribute to the
+current context's value. The ``__init__`` method for the ``Context`` class
+takes a parameter called ``autoescape`` that you can use for this purpose. For
+example::
+
+    def render(self, context):
+        # ...
+        new_context = Context({'var': obj}, autoescape=context.autoescape)
+        # ... Do something with new_context ...
+
+This is not a very common situation, but it is sometimes useful, particularly
+if you are rendering a template yourself. For example::
+
+    def render(self, context):
+        t = template.load_template('small_fragment.html')
+        return t.render(Context({'var': obj}, autoescape=context.autoescape))
+
+If we had neglected to pass in the current ``context.autoescape`` value to our
+new ``Context`` in this example, the results would have *always* been
+automatically escaped, which may not be the desired behaviour if the template
+tag is used inside a ``{% autoescape off %}`` block.
+
 Registering the tag
 ~~~~~~~~~~~~~~~~~~~
 
@@ -917,7 +1043,7 @@ current context, available in the ``render`` method::
         def __init__(self, date_to_be_formatted, format_string):
             self.date_to_be_formatted = date_to_be_formatted
             self.format_string = format_string
-        
+
         def render(self, context):
             try:
                 actual_date = resolve_variable(self.date_to_be_formatted, context)
@@ -934,26 +1060,26 @@ format it accordingly.
     ``template.resolve_variable()`` is still available, but has been deprecated
     in favor of a new ``template.Variable`` class. Using this class will usually
     be more efficient than calling ``template.resolve_variable``
-    
+
     To use the ``Variable`` class, simply instantiate it with the name of the
     variable to be resolved, and then call ``variable.resolve(context)``. So,
     in the development version, the above example would be more correctly
     written as:
-    
+
     .. parsed-literal::
-    
+
         class FormatTimeNode(template.Node):
             def __init__(self, date_to_be_formatted, format_string):
                 self.date_to_be_formatted = **Variable(date_to_be_formatted)**
                 self.format_string = format_string
-        
+
             def render(self, context):
                 try:
                     actual_date = **self.date_to_be_formatted.resolve(context)**
                     return actual_date.strftime(self.format_string)
                 except template.VariableDoesNotExist:
                     return ''
-    
+
     Changes are highlighted in bold.
 
 Variable resolution will throw a ``VariableDoesNotExist`` exception if it cannot