Fixed #12663 -- Formalized the Model._meta API for retrieving fields.

Thanks to Russell Keith-Magee for mentoring this Google Summer of
Code 2014 project and everyone else who helped with the patch!

3 files changed, 380 insertions, 0 deletions

diff --git a/docs/ref/models/fields.txt b/docs/ref/models/fields.txt
index 35997801fb..bc214fae00 100644
--- a/docs/ref/models/fields.txt
+++ b/docs/ref/models/fields.txt
@@ -1790,3 +1790,95 @@ Field API reference
 
         This method must be added to fields prior to 1.7 to migrate its data
         using :doc:`/topics/migrations`.
+
+.. _model-field-attributes:
+
+=========================
+Field attribute reference
+=========================
+
+.. versionadded:: 1.8
+
+Every ``Field`` instance contains several attributes that allow
+introspecting its behavior. Use these attributes instead of ``isinstance``
+checks when you need to write code that depends on a field's functionality.
+These attributes can be used together with the :ref:`Model._meta API
+<model-meta-field-api>` to narrow down a search for specific field types.
+Custom model fields should implement these flags.
+
+Attributes for fields
+=====================
+
+.. attribute:: Field.auto_created
+
+     Boolean flag that indicates if the field was automatically created, such
+     as the ``OneToOneField`` used by model inheritance.
+
+.. attribute:: Field.concrete
+
+    Boolean flag that indicates if the field has a database column associated
+    with it.
+
+.. attribute:: Field.hidden
+
+    Boolean flag that indicates if a field is used to back another non-hidden
+    field's functionality (e.g. the ``content_type`` and ``object_id`` fields
+    that make up a ``GenericForeignKey``). The ``hidden`` flag is used to
+    distinguish what constitutes the public subset of fields on the model from
+    all the fields on the model.
+
+    .. note::
+
+        :meth:`Options.get_fields()
+        <django.db.models.options.Options.get_fields()>`
+        excludes hidden fields by default. Pass in ``include_hidden=True`` to
+        return hidden fields in the results.
+
+.. attribute:: Field.is_relation
+
+    Boolean flag that indicates if a field contains references to one or
+    more other models for its functionality (e.g. ``ForeignKey``,
+    ``ManyToManyField``, ``OneToOneField``, etc.).
+
+.. attribute:: Field.model
+
+    Returns the model on which the field is defined. If a field is defined on
+    a superclass of a model, ``model`` will refer to the superclass, not the
+    class of the instance.
+
+Attributes for fields with relations
+====================================
+
+These attributes are used to query for the cardinality and other details of a
+relation. These attribute are present on all fields; however, they will only
+have meaningful values if the field is a relation type
+(:attr:`Field.is_relation=True <Field.is_relation>`).
+
+.. attribute:: Field.one_to_many
+
+    Boolean flag that is ``True`` if the field has a one-to-many relation, such
+    as a ``ForeignKey``; ``False`` otherwise.
+
+.. attribute:: Field.one_to_one
+
+    Boolean flag that is ``True`` if the field has a one-to-one relation, such
+    as a ``OneToOneField``; ``False`` otherwise.
+
+.. attribute:: Field.many_to_many
+
+    Boolean flag that is ``True`` if the field has a many-to-many relation;
+    ``False`` otherwise. The only field included with Django where this is
+    ``True`` is ``ManyToManyField``.
+
+.. attribute:: Field.many_to_one
+
+    Boolean flag that is ``True`` if the field has a many-to-one relation, such
+    as a ``GenericRelation`` or the reverse of a ``ForeignKey``; ``False``
+    otherwise.
+
+.. attribute:: Field.related_model
+
+    Points to the model the field relates to. For example, ``Author`` in
+    ``ForeignKey(Author)``. If a field has a generic relation (such as a
+    ``GenericForeignKey`` or a ``GenericRelation``) then ``related_model``
+    will be ``None``.
diff --git a/docs/ref/models/index.txt b/docs/ref/models/index.txt
index d1d3680dd8..284743e7d0 100644
--- a/docs/ref/models/index.txt
+++ b/docs/ref/models/index.txt
@@ -8,6 +8,7 @@ Model API reference. For introductory material, see :doc:`/topics/db/models`.
    :maxdepth: 1
 
    fields
+   meta
    relations
    class
    options
diff --git a/docs/ref/models/meta.txt b/docs/ref/models/meta.txt
new file mode 100644
index 0000000000..fdad7b2ad0
--- /dev/null
+++ b/docs/ref/models/meta.txt
@@ -0,0 +1,287 @@
+===================
+Model ``_meta`` API
+===================
+
+.. module:: django.db.models.options
+   :synopsis: Model meta-class layer
+
+.. class:: Options
+
+The model ``_meta`` API is at the core of the Django ORM. It enables other
+parts of the system such as lookups, queries, forms, and the admin to
+understand the capabilities of each model. The API is accessible through
+the ``_meta`` attribute of each model class, which is an instance of an
+``django.db.models.options.Options`` object.
+
+Methods that it provides can be used to:
+
+* Retrieve all field instances of a model
+* Retrieve a single field instance of a model by name
+
+.. versionchanged:: 1.8
+
+    The Model ``_meta`` API has always existed as a Django internal, but
+    wasn't formally documented and supported. As part of the effort to
+    make this API public, some of the already existing API entry points
+    have changed slightly. A :ref:`migration guide <migrating-old-meta-api>`
+    has been provided to assist in converting your code to use the new,
+    official API.
+
+.. _model-meta-field-api:
+
+Field access API
+================
+
+Retrieving a single field instance of a model by name
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. method:: Options.get_field(field_name)
+
+    Returns the field instance given a name of a field.
+
+    ``field_name`` can be the name of a field on the model, a field
+    on an abstract or inherited model, or a field defined on another
+    model that points to the model. In the latter case, the ``field_name``
+    will be the ``related_name`` defined by the user or the name automatically
+    generated by Django itself.
+
+    :attr:`Hidden fields <django.db.models.Field.hidden>` cannot be retrieved
+    by name.
+
+    If a field with the given name is not found a
+    :class:`~django.core.exceptions.FieldDoesNotExist` exception will be
+    raised.
+
+    .. code-block:: python
+
+        >>> from django.contrib.auth.models import User
+
+        # A field on the model
+        >>> User._meta.get_field('username')
+        <django.db.models.fields.CharField: username>
+
+        # A field from another model that has a relation with the current model
+        >>> User._meta.get_field('logentry')
+        <ManyToOneRel: admin.logentry>
+
+        # A non existent field
+        >>> User._meta.get_field('does_not_exist')
+        Traceback (most recent call last):
+            ...
+        FieldDoesNotExist: User has no field named 'does_not_exist'
+
+    .. deprecated:: 1.8
+
+        :meth:`Options.get_field()` previously accepted a ``many_to_many``
+        parameter which could be set to ``False`` to avoid searching
+        ``ManyToManyField``\s. The old behavior has been preserved for
+        backwards compatibility; however, the parameter and this behavior
+        has been deprecated.
+
+        If you wish to filter out ``ManyToManyField``\s, you can inspect the
+        :attr:`Field.many_to_many <django.db.models.Field.many_to_many>`
+        attribute after calling ``get_field()``.
+
+Retrieving all field instances of a model
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. method:: Options.get_fields(include_parents=True, include_hidden=False)
+
+    .. versionadded:: 1.8
+
+    Returns a tuple of fields associated with a model. ``get_fields()`` accepts
+    two parameters that can be used to control which fields are returned:
+
+    ``include_parents``
+        ``True`` by default. Recursively includes fields defined on parent
+        classes. If set to ``False``, ``get_fields()`` will only search for
+        fields declared directly on the current model. Fields from models that
+        directly inherit from abstract models or proxy classes are considered
+        to be local, not on the parent.
+
+    ``include_hidden``
+        ``False`` by default. If set to ``True``, ``get_fields()`` will include
+        fields that are used to back other field's functionality. This will
+        also include any fields that have a ``related_name`` (such
+        as :class:`~django.db.models.ManyToManyField`, or
+        :class:`~django.db.models.ForeignKey`) that start with a "+".
+
+    .. code-block:: python
+
+        >>> from django.contrib.auth.models import User
+        >>> User._meta.get_fields()
+        (<ManyToOneRel: admin.logentry>,
+         <django.db.models.fields.AutoField: id>,
+         <django.db.models.fields.CharField: password>,
+         <django.db.models.fields.DateTimeField: last_login>,
+         <django.db.models.fields.BooleanField: is_superuser>,
+         <django.db.models.fields.CharField: username>,
+         <django.db.models.fields.CharField: first_name>,
+         <django.db.models.fields.CharField: last_name>,
+         <django.db.models.fields.EmailField: email>,
+         <django.db.models.fields.BooleanField: is_staff>,
+         <django.db.models.fields.BooleanField: is_active>,
+         <django.db.models.fields.DateTimeField: date_joined>,
+         <django.db.models.fields.related.ManyToManyField: groups>,
+         <django.db.models.fields.related.ManyToManyField: user_permissions>)
+
+        # Also include hidden fields.
+        >>> User._meta.get_fields(include_hidden=True)
+        (<ManyToOneRel: auth.user_groups>,
+         <ManyToOneRel: auth.user_user_permissions>,
+         <ManyToOneRel: admin.logentry>,
+         <django.db.models.fields.AutoField: id>,
+         <django.db.models.fields.CharField: password>,
+         <django.db.models.fields.DateTimeField: last_login>,
+         <django.db.models.fields.BooleanField: is_superuser>,
+         <django.db.models.fields.CharField: username>,
+         <django.db.models.fields.CharField: first_name>,
+         <django.db.models.fields.CharField: last_name>,
+         <django.db.models.fields.EmailField: email>,
+         <django.db.models.fields.BooleanField: is_staff>,
+         <django.db.models.fields.BooleanField: is_active>,
+         <django.db.models.fields.DateTimeField: date_joined>,
+         <django.db.models.fields.related.ManyToManyField: groups>,
+         <django.db.models.fields.related.ManyToManyField: user_permissions>)
+
+.. _migrating-old-meta-api:
+
+Migrating from the old API
+==========================
+
+As part of the formalization of the ``Model._meta`` API (from the
+:class:`django.db.models.options.Options` class), a number of methods and
+properties have been deprecated and will be removed in Django 2.0.
+
+These old APIs can be replicated by either:
+
+* invoking :meth:`Options.get_field()
+  <django.db.models.options.Options.get_field()>`, or;
+
+* invoking :meth:`Options.get_fields()
+  <django.db.models.options.Options.get_fields()>` to retrieve a list of all
+  fields, and then filtering this list using the :ref:`field attributes
+  <model-field-attributes>` that describe (or retrieve, in the case of
+  ``_with_model`` variants) the properties of the desired fields.
+
+Although it's possible to make strictly equivalent replacements of the old
+methods, that might not be the best approach. Taking the time to refactor any
+field loops to make better use of the new API - and possibly include fields
+that were previously excluded - will almost certainly result in better code.
+
+Assuming you have a model named ``MyModel``, the following substitutions
+can be made to convert your code to the new API:
+
+* ``MyModel._meta.get_field(name)``::
+
+      f = MyModel._meta.get_field(name)
+
+  then check if:
+
+  - ``f.auto_created == False``, because the new ``get_field()``
+    API will find "reverse" relations), and:
+
+  - ``f.is_relation and f.related_model is None``, because the new
+    ``get_field()`` API will find
+    :class:`~django.contrib.contenttypes.fields.GenericForeignKey` relations;
+
+* ``MyModel._meta.get_field_by_name(name)``:
+
+  ``get_field_by_name()`` returned four values:
+  ``(field, model, direct,  m2m)``:
+
+  - ``field`` can be found by ``MyModel._meta.get_field(name)``
+
+  - ``model`` can be found through the
+    :attr:`~django.db.models.Field.model` attribute on the field.
+
+  - ``direct`` can be found by: ``not field.auto_created or field.concrete``
+
+    The :attr:`~django.db.models.Field.auto_created` check excludes
+    all "forward" and "reverse" relations that are created by Django, but
+    this also includes ``AutoField`` and ``OneToOneField`` on proxy models.
+    We avoid filtering out these attributes using the
+    :attr:`concrete <django.db.models.Field.concrete>` attribute.
+
+  - ``m2m`` can be found through the
+    :attr:`~django.db.models.Field.many_to_many` attribute on the field.
+
+* ``MyModel._meta.get_fields_with_model()``::
+
+      [
+          (f, f.model if f.model != MyModel else None)
+          for f in MyModel._meta.get_fields()
+          if not f.is_relation
+              or f.one_to_one
+              or (f.one_to_many and f.related_model)
+      ]
+
+* ``MyModel._meta.get_concrete_fields_with_model()``::
+
+      [
+          (f, f.model if f.model != MyModel else None)
+          for f in MyModel._meta.get_fields()
+          if f.concrete and (
+              not f.is_relation
+              or f.one_to_one
+              or (f.one_to_many and f.related_model)
+          )
+      ]
+
+* ``MyModel._meta.get_m2m_with_model()``::
+
+      [
+          (f, f.model if f.model != MyModel else None)
+          for f in MyModel._meta.get_fields()
+          if f.many_to_many and not f.auto_created
+      ]
+
+* ``MyModel._meta.get_all_related_objects()``::
+
+      [
+          f for f in MyModel._meta.get_fields()
+          if f.many_to_one and f.auto_created
+      ]
+
+* ``MyModel._meta.get_all_related_objects_with_model()``::
+
+      [
+          (f, f.model if f.model != MyModel else None)
+          for f in MyModel._meta.get_fields()
+          if f.many_to_one and f.auto_created
+      ]
+
+* ``MyModel._meta.get_all_related_many_to_many_objects()``::
+
+      [
+          f for f in MyModel._meta.get_fields(include_hidden=True)
+          if f.many_to_many and f.auto_created
+      ]
+
+* ``MyModel._meta.get_all_related_m2m_objects_with_model()``::
+
+      [
+          (f, f.model if f.model != MyModel else None)
+          for f in MyModel._meta.get_fields(include_hidden=True)
+          if f.many_to_many and f.auto_created
+      ]
+
+* ``MyModel._meta.get_all_field_names()``::
+
+      from itertools import chain
+      list(set(chain.from_iterable(
+          (field.name, field.attname) if hasattr(field, 'attname') else (field.name,)
+          for field in MyModel._meta.get_fields()
+          # For complete backwards compatibility, you may want to exclude
+          # GenericForeignKey from the results.
+          if not (field.one_to_many and field.related_model is None)
+      )))
+
+  This provides a 100% backwards compatible replacement, ensuring that both
+  field names and attribute names ``ForeignKey``\s are included, but fields
+  associated with ``GenericForeignKey``\s are not. A simpler version would be::
+
+      [f.name for f in MyModel._meta.get_fields()]
+
+  While this isn't 100% backwards compatible, it may be sufficient in many
+  situations.