Fixed #18757, #14462, #21565 -- Reworked database-python type conversions

Complete rework of translating data values from database

Deprecation of SubfieldBase, removal of resolve_columns and
convert_values in favour of a more general converter based approach and
public API Field.from_db_value(). Now works seamlessly with aggregation,
.values() and raw queries.

Thanks to akaariai in particular for extensive advice and inspiration,
also to shaib, manfre and timograham for their reviews.

4 files changed, 101 insertions, 119 deletions

diff --git a/docs/howto/custom-model-fields.txt b/docs/howto/custom-model-fields.txt
index f24746c2f0..568831523c 100644
--- a/docs/howto/custom-model-fields.txt
+++ b/docs/howto/custom-model-fields.txt
@@ -317,77 +317,6 @@ and reconstructing the field::
     new_instance = MyField(*args, **kwargs)
     self.assertEqual(my_field_instance.some_attribute, new_instance.some_attribute)
 
-
-The ``SubfieldBase`` metaclass
-------------------------------
-
-.. class:: django.db.models.SubfieldBase
-
-As we indicated in the introduction_, field subclasses are often needed for
-two reasons: either to take advantage of a custom database column type, or to
-handle complex Python types. Obviously, a combination of the two is also
-possible. If you're only working with custom database column types and your
-model fields appear in Python as standard Python types direct from the
-database backend, you don't need to worry about this section.
-
-If you're handling custom Python types, such as our ``Hand`` class, we need to
-make sure that when Django initializes an instance of our model and assigns a
-database value to our custom field attribute, we convert that value into the
-appropriate Python object. The details of how this happens internally are a
-little complex, but the code you need to write in your ``Field`` class is
-simple: make sure your field subclass uses a special metaclass:
-
-For example, on Python 2::
-
-    class HandField(models.Field):
-
-        description = "A hand of cards (bridge style)"
-
-        __metaclass__ = models.SubfieldBase
-
-        def __init__(self, *args, **kwargs):
-            ...
-
-On Python 3, in lieu of setting the ``__metaclass__`` attribute, add
-``metaclass`` to the class definition::
-
-    class HandField(models.Field, metaclass=models.SubfieldBase):
-        ...
-
-If you want your code to work on Python 2 & 3, you can use
-:func:`six.with_metaclass`::
-
-    from django.utils.six import with_metaclass
-
-    class HandField(with_metaclass(models.SubfieldBase, models.Field)):
-        ...
-
-This ensures that the :meth:`.to_python` method will always be called when the
-attribute is initialized.
-
-``ModelForm``\s and custom fields
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If you use :class:`~django.db.models.SubfieldBase`, :meth:`.to_python` will be
-called every time an instance of the field is assigned a value (in addition to
-its usual call when retrieving the value from the database). This means that
-whenever a value may be assigned to the field, you need to ensure that it will
-be of the correct datatype, or that you handle any exceptions.
-
-This is especially important if you use :doc:`ModelForms
-</topics/forms/modelforms>`. When saving a ModelForm, Django will use
-form values to instantiate model instances. However, if the cleaned
-form data can't be used as valid input to the field, the normal form
-validation process will break.
-
-Therefore, you must ensure that the form field used to represent your
-custom field performs whatever input validation and data cleaning is
-necessary to convert user-provided form input into a
-``to_python()``-compatible model field value. This may require writing a
-custom form field, and/or implementing the :meth:`.formfield` method on
-your field to return a form field class whose ``to_python()`` returns the
-correct datatype.
-
 Documenting your custom field
 -----------------------------
 
@@ -500,59 +429,79 @@ over this field. You are then responsible for creating the column in the right
 table in some other way, of course, but this gives you a way to tell Django to
 get out of the way.
 
-.. _converting-database-values-to-python-objects:
+.. _converting-values-to-python-objects:
 
-Converting database values to Python objects
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Converting values to Python objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionchanged:: 1.8
+
+    Historically, Django provided a metaclass called ``SubfieldBase`` which
+    always called :meth:`~Field.to_python` on assignment. This did not play
+    nicely with custom database transformations, aggregation, or values
+    queries, so it has been replaced with :meth:`~Field.from_db_value`.
 
 If your custom :class:`~Field` class deals with data structures that are more
-complex than strings, dates, integers or floats, then you'll need to override
-:meth:`~Field.to_python`. As a general rule, the method should deal gracefully
-with any of the following arguments:
+complex than strings, dates, integers, or floats, then you may need to override
+:meth:`~Field.from_db_value` and :meth:`~Field.to_python`.
+
+If present for the field subclass, ``from_db_value()`` will be called in all
+circumstances when the data is loaded from the database, including in
+aggregates and :meth:`~django.db.models.query.QuerySet.values` calls.
+
+``to_python()`` is called by deserialization and during the
+:meth:`~django.db.models.Model.clean` method used from forms.
+
+As a general rule, ``to_python()`` should deal gracefully with any of the
+following arguments:
 
 * An instance of the correct type (e.g., ``Hand`` in our ongoing example).
 
-* A string (e.g., from a deserializer).
+* A string
 
-* Whatever the database returns for the column type you're using.
+* ``None`` (if the field allows ``null=True``)
 
 In our ``HandField`` class, we're storing the data as a VARCHAR field in the
-database, so we need to be able to process strings and ``Hand`` instances in
-:meth:`.to_python`::
+database, so we need to be able to process strings and ``None`` in the
+``from_db_value()``. In ``to_python()``, we need to also handle ``Hand``
+instances::
 
     import re
 
+    from django.core.exceptions import ValidationError
+    from django.db import models
+
+    def parse_hand(hand_string):
+        """Takes a string of cards and splits into a full hand."""
+        p1 = re.compile('.{26}')
+        p2 = re.compile('..')
+        args = [p2.findall(x) for x in p1.findall(hand_string)]
+        if len(args) != 4:
+            raise ValidationError("Invalid input for a Hand instance")
+        return Hand(*args)
+
     class HandField(models.Field):
         # ...
 
+        def from_db_value(self, value, connection):
+            if value is None:
+                return value
+            return parse_hand(value)
+
         def to_python(self, value):
             if isinstance(value, Hand):
                 return value
 
-            # The string case.
-            p1 = re.compile('.{26}')
-            p2 = re.compile('..')
-            args = [p2.findall(x) for x in p1.findall(value)]
-            if len(args) != 4:
-                raise ValidationError("Invalid input for a Hand instance")
-            return Hand(*args)
-
-Notice that we always return a ``Hand`` instance from this method. That's the
-Python object type we want to store in the model's attribute. If anything is
-going wrong during value conversion, you should raise a
-:exc:`~django.core.exceptions.ValidationError` exception.
+            if value is None:
+                return value
 
-**Remember:** If your custom field needs the :meth:`~Field.to_python` method to be
-called when it is created, you should be using `The SubfieldBase metaclass`_
-mentioned earlier. Otherwise :meth:`~Field.to_python` won't be called
-automatically.
+            return parse_hand(value)
 
-.. warning::
+Notice that we always return a ``Hand`` instance from these methods. That's the
+Python object type we want to store in the model's attribute.
 
-    If your custom field allows ``null=True``, any field method that takes
-    ``value`` as an argument, like :meth:`~Field.to_python` and
-    :meth:`~Field.get_prep_value`, should handle the case when ``value`` is
-    ``None``.
+For ``to_python()``, if anything goes wrong during value conversion, you should
+raise a :exc:`~django.core.exceptions.ValidationError` exception.
 
 .. _converting-python-objects-to-query-values:
 
diff --git a/docs/internals/deprecation.txt b/docs/internals/deprecation.txt
index 10ef6e5850..6085117b93 100644
--- a/docs/internals/deprecation.txt
+++ b/docs/internals/deprecation.txt
@@ -57,6 +57,8 @@ about each item can often be found in the release notes of two versions prior.
 * The ``is_admin_site`` argument to
   ``django.contrib.auth.views.password_reset()`` will be removed.
 
+* ``django.db.models.field.subclassing.SubfieldBase`` will be removed.
+
 .. _deprecation-removed-in-1.9:
 
 1.9
diff --git a/docs/ref/models/fields.txt b/docs/ref/models/fields.txt
index e06a359ec8..d7f15d6643 100644
--- a/docs/ref/models/fields.txt
+++ b/docs/ref/models/fields.txt
@@ -1532,7 +1532,7 @@ Field API reference
     ``Field`` is an abstract class that represents a database table column.
     Django uses fields to create the database table (:meth:`db_type`), to map
     Python types to database (:meth:`get_prep_value`) and vice-versa
-    (:meth:`to_python`), and to apply :doc:`/ref/models/lookups`
+    (:meth:`from_db_value`), and to apply :doc:`/ref/models/lookups`
     (:meth:`get_prep_lookup`).
 
     A field is thus a fundamental piece in different Django APIs, notably,
@@ -1609,17 +1609,26 @@ Field API reference
 
         See :ref:`converting-query-values-to-database-values` for usage.
 
-    When loading data, :meth:`to_python` is used:
+    When loading data, :meth:`from_db_value` is used:
 
-    .. method:: to_python(value)
+    .. method:: from_db_value(value, connection)
+
+        .. versionadded:: 1.8
+
+        Converts a value as returned by the database to a Python object. It is
+        the reverse of :meth:`get_prep_value`.
 
-        Converts a value as returned by the database (or a serializer) to a
-        Python object. It is the reverse of :meth:`get_prep_value`.
+        This method is not used for most built-in fields as the database
+        backend already returns the correct Python type, or the backend itself
+        does the conversion.
 
-        The default implementation returns ``value``, which is the common case
-        when the database backend already returns the correct Python type.
+        See :ref:`converting-values-to-python-objects` for usage.
 
-        See :ref:`converting-database-values-to-python-objects` for usage.
+        .. note::
+        
+            For performance reasons, ``from_db_value`` is not implemented as a
+            no-op on fields which do not require it (all Django fields).
+            Consequently you may not call ``super`` in your definition.
 
     When saving, :meth:`pre_save` and :meth:`get_db_prep_save` are used:
 
@@ -1644,15 +1653,6 @@ Field API reference
 
         See :ref:`preprocessing-values-before-saving` for usage.
 
-    Besides saving to the database, the field also needs to know how to
-    serialize its value (inverse of :meth:`to_python`):
-
-    .. method:: value_to_string(obj)
-
-        Converts ``obj`` to a string. Used to serialize the value of the field.
-
-        See :ref:`converting-model-field-to-serialization` for usage.
-
     When a lookup is used on a field, the value may need to be "prepared".
     Django exposes two methods for this:
 
@@ -1682,6 +1682,26 @@ Field API reference
         ``prepared`` describes whether the value has already been prepared with
         :meth:`get_prep_lookup`.
 
+    Fields often receive their values as a different type, either from
+    serialization or from forms.
+    
+    .. method:: to_python(value)
+
+        Converts the value into the correct Python object. It acts as the
+        reverse of :meth:`value_to_string`, and is also called in
+        :meth:`~django.db.models.Model.clean`.
+
+        See :ref:`converting-values-to-python-objects` for usage.
+
+    Besides saving to the database, the field also needs to know how to
+    serialize its value:
+
+    .. method:: value_to_string(obj)
+
+        Converts ``obj`` to a string. Used to serialize the value of the field.
+
+        See :ref:`converting-model-field-to-serialization` for usage.
+
     When using :class:`model forms <django.forms.ModelForm>`, the ``Field``
     needs to know which form field it should be represented by:
 
diff --git a/docs/releases/1.8.txt b/docs/releases/1.8.txt
index a95024420c..34530885cc 100644
--- a/docs/releases/1.8.txt
+++ b/docs/releases/1.8.txt
@@ -736,3 +736,14 @@ also been deprecated.
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 It's a legacy option that should no longer be necessary.
+
+``SubfieldBase``
+~~~~~~~~~~~~~~~~
+
+``django.db.models.fields.subclassing.SubfieldBase`` has been deprecated and
+will be removed in Django 2.0. Historically, it was used to handle fields where
+type conversion was needed when loading from the database, but it was not used
+in ``.values()`` calls or in aggregates. It has been replaced with
+:meth:`~django.db.models.Field.from_db_value`. Note that the new approach does
+not call the :meth:`~django.db.models.Fields.to_python`` method on assignment
+as was the case with ``SubfieldBase``.