Fixed #16187 -- refactored ORM lookup system

Allowed users to specify which lookups or transforms ("nested lookus")
are available for fields. The implementation is now class based.

Squashed commit of the following:

commit fa7a7195f1952a9c8dea7f6e89ee13f81757eda7
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 18 10:53:24 2014 +0200

    Added lookup registration API docs

commit eb1c8ce164325e0d8641f14202e12486c70efdb6
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Tue Jan 14 18:59:36 2014 +0200

    Release notes and other minor docs changes

commit 11501c29c9352d17f22f3a0f59d3b805913dedcc
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Jan 12 20:53:03 2014 +0200

    Forgot to add custom_lookups tests in prev commit

commit 83173b960ea7eb2b24d573f326be59948df33536
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Jan 12 19:59:12 2014 +0200

    Renamed Extract -> Transform

commit 3b18d9f3a1bcdd93280f79654eba0efa209377bd
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Jan 12 19:51:53 2014 +0200

    Removed suggestion of temporary lookup registration from docs

commit 21d0c7631c161fc0c67911480be5d3f13f1afa68
Merge: 2509006 f2dc442
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Jan 12 09:38:23 2014 -0800

    Merge pull request #2 from mjtamlyn/lookups_3

    Reworked custom lookups docs.

commit f2dc4429a1da04c858364972eea57a35a868dab4
Author: Marc Tamlyn <marc.tamlyn@gmail.com>
Date:   Sun Jan 12 13:15:05 2014 +0000

    Reworked custom lookups docs.

    Mostly just formatting and rewording, but also replaced the example
    using ``YearExtract`` to  use an example which is unlikely to ever be
    possible directly in the ORM.

commit 250900650628d1f11beadb22814abd666029fb81
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Jan 12 13:19:13 2014 +0200

    Removed unused import

commit 4fba5dfaa022653ffa72497258ffd8f8b7476f92
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 22:34:41 2014 +0200

    Added docs to index

commit 6d53963f375c77a1f287833b19b976d23f36c30b
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 22:10:24 2014 +0200

    Dead code removal

commit f9cc0390078e21f1ea5a7bc1f15b09f8f6b0904d
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 19:00:43 2014 +0200

    A new try for docs

commit 33aa18a6e3c831930bda0028222a26f9c1d96e66
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 14:57:12 2014 +0200

    Renamed get_cols to get_group_by_cols

commit c7d5f8661b7d364962bed2e6f81161c1b4f1bcc3
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 14:45:53 2014 +0200

    Altered query string customization for backends vendors

    The new way is trying to call first method 'as_' + connection.vendor.
    If that doesn't exist, then call as_sql().

    Also altered how lookup registration is done. There is now
    RegisterLookupMixin class that is used by Field, Extract and
    sql.Aggregate. This allows one to register lookups for extracts and
    aggregates in the same way lookup registration is done for fields.

commit 90e7004ec14e15503f828cc9bde2a7dab593814d
Merge: 66649ff f7c2c0a
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 13:21:01 2014 +0200

    Merge branch 'master' into lookups_3

commit 66649ff891c7c73c7eecf6038c9a6802611b5d8a
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Jan 11 13:16:01 2014 +0200

    Some rewording in docs

commit 31b8faa62714b4b6b6057a9f5cc106c4dd73caab
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Dec 29 15:52:29 2013 +0200

    Cleanup based on review comments

commit 1016159f34674c0df871ed891cde72be8340bb5d
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Dec 28 18:37:04 2013 +0200

    Proof-of-concept fix for #16731

    Implemented only for SQLite and PostgreSQL, and only for startswith
    and istartswith lookups.

commit 193cd097ca8f2cc6a911e57b8e3fb726f96ee6a6
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Dec 28 17:57:58 2013 +0200

    Fixed #11722 -- iexact=F() produced invalid SQL

commit 08ed3c3b49e100ed9019831e770c25c8f61b70f9
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Dec 21 23:59:52 2013 +0200

    Made Lookup and Extract available from django.db.models

commit b99c8d83c972786c6fcd0e84c9e5cb08c1368300
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Dec 21 23:06:29 2013 +0200

    Fixed review notes by Loic

commit 049eebc0703c151127f4f0265beceea7b8b39e72
Merge: ed8fab7 b80a835
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Dec 21 22:53:10 2013 +0200

    Merge branch 'master' into lookups_3

    Conflicts:
    	django/db/models/fields/__init__.py
    	django/db/models/sql/compiler.py
    	django/db/models/sql/query.py
    	tests/null_queries/tests.py

commit ed8fab7fe8867ff3eb801c3697a426478387bb2f
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Dec 21 22:47:23 2013 +0200

    Made Extracts aware of full lookup path

commit 27a57b7aed91b2f346abc4a77da838bffa17c727
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Dec 1 21:10:11 2013 +0200

    Removed debugger import

commit 074e0f5aca0572e368c11e6d2c73c9026e7d63d7
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Dec 1 21:02:16 2013 +0200

    GIS lookup support added

commit 760e28e72bae475b442b026650969b0d182dbe53
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Dec 1 20:04:31 2013 +0200

    Removed usage of Constraint, used Lookup instead

commit eac47766844b90e7d3269e7a8c012eee34ec0093
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Dec 1 02:22:30 2013 +0200

    Minor cleanup of Lookup API

commit 2adf50428d59a783078b0da3d5d035106640c899
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sun Dec 1 02:14:19 2013 +0200

    Added documentation, polished implementation

commit 32c04357a87e3727a34f8c5e6ec0114d1fbbb303
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Nov 30 23:10:15 2013 +0200

    Avoid OrderedDict creation on lookup aggregate check

commit 7c8b3a32cc17b4dbca160921d48125f1631e0df4
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Nov 30 23:04:34 2013 +0200

    Implemented nested lookups

    But there is no support of using lookups outside filtering yet.

commit 4d219d4cdef21d9c14e5d6b9299d583d1975fcba
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Wed Nov 27 22:07:30 2013 +0200

    Initial implementation of custom lookups

9 files changed, 57 insertions, 30 deletions

diff --git a/django/contrib/contenttypes/generic.py b/django/contrib/contenttypes/generic.py
index a92f0f0dc8..7d130bec24 100644
--- a/django/contrib/contenttypes/generic.py
+++ b/django/contrib/contenttypes/generic.py
@@ -12,7 +12,7 @@ from django.db import models, router, transaction, DEFAULT_DB_ALIAS
 from django.db.models import signals
 from django.db.models.fields.related import ForeignObject, ForeignObjectRel
 from django.db.models.related import PathInfo
-from django.db.models.sql.where import Constraint
+from django.db.models.sql.datastructures import Col
 from django.forms import ModelForm, ALL_FIELDS
 from django.forms.models import (BaseModelFormSet, modelformset_factory,
     modelform_defines_fields)
@@ -236,7 +236,8 @@ class GenericRelation(ForeignObject):
         field = self.rel.to._meta.get_field_by_name(self.content_type_field_name)[0]
         contenttype_pk = self.get_content_type().pk
         cond = where_class()
-        cond.add((Constraint(remote_alias, field.column, field), 'exact', contenttype_pk), 'AND')
+        lookup = field.get_lookup('exact')(Col(remote_alias, field, field), contenttype_pk)
+        cond.add(lookup, 'AND')
         return cond
 
     def bulk_related_objects(self, objs, using=DEFAULT_DB_ALIAS):
diff --git a/django/contrib/gis/db/backends/mysql/operations.py b/django/contrib/gis/db/backends/mysql/operations.py
index 81aa1e2fb6..989b5a1292 100644
--- a/django/contrib/gis/db/backends/mysql/operations.py
+++ b/django/contrib/gis/db/backends/mysql/operations.py
@@ -49,9 +49,7 @@ class MySQLOperations(DatabaseOperations, BaseSpatialOperations):
         return placeholder
 
     def spatial_lookup_sql(self, lvalue, lookup_type, value, field, qn):
-        alias, col, db_type = lvalue
-
-        geo_col = '%s.%s' % (qn(alias), qn(col))
+        geo_col, db_type = lvalue
 
         lookup_info = self.geometry_functions.get(lookup_type, False)
         if lookup_info:
diff --git a/django/contrib/gis/db/backends/oracle/operations.py b/django/contrib/gis/db/backends/oracle/operations.py
index 8212938f01..b3aa191ccc 100644
--- a/django/contrib/gis/db/backends/oracle/operations.py
+++ b/django/contrib/gis/db/backends/oracle/operations.py
@@ -231,10 +231,7 @@ class OracleOperations(DatabaseOperations, BaseSpatialOperations):
 
     def spatial_lookup_sql(self, lvalue, lookup_type, value, field, qn):
         "Returns the SQL WHERE clause for use in Oracle spatial SQL construction."
-        alias, col, db_type = lvalue
-
-        # Getting the quoted table name as `geo_col`.
-        geo_col = '%s.%s' % (qn(alias), qn(col))
+        geo_col, db_type = lvalue
 
         # See if a Oracle Geometry function matches the lookup type next
         lookup_info = self.geometry_functions.get(lookup_type, False)
diff --git a/django/contrib/gis/db/backends/postgis/operations.py b/django/contrib/gis/db/backends/postgis/operations.py
index 40ca09f0fa..e8da6a52a6 100644
--- a/django/contrib/gis/db/backends/postgis/operations.py
+++ b/django/contrib/gis/db/backends/postgis/operations.py
@@ -478,10 +478,7 @@ class PostGISOperations(DatabaseOperations, BaseSpatialOperations):
         (alias, col, db_type), the lookup type string, lookup value, and
         the geometry field.
         """
-        alias, col, db_type = lvalue
-
-        # Getting the quoted geometry column.
-        geo_col = '%s.%s' % (qn(alias), qn(col))
+        geo_col, db_type = lvalue
 
         if lookup_type in self.geometry_operators:
             if field.geography and not lookup_type in self.geography_operators:
diff --git a/django/contrib/gis/db/backends/spatialite/operations.py b/django/contrib/gis/db/backends/spatialite/operations.py
index 5149b541a2..ad44b470b3 100644
--- a/django/contrib/gis/db/backends/spatialite/operations.py
+++ b/django/contrib/gis/db/backends/spatialite/operations.py
@@ -324,10 +324,7 @@ class SpatiaLiteOperations(DatabaseOperations, BaseSpatialOperations):
         [a tuple of (alias, column, db_type)], lookup type, lookup
         value, the model field, and the quoting function.
         """
-        alias, col, db_type = lvalue
-
-        # Getting the quoted field as `geo_col`.
-        geo_col = '%s.%s' % (qn(alias), qn(col))
+        geo_col, db_type = lvalue
 
         if lookup_type in self.geometry_functions:
             # See if a SpatiaLite geometry function matches the lookup type.
diff --git a/django/contrib/gis/db/models/constants.py b/django/contrib/gis/db/models/constants.py
new file mode 100644
index 0000000000..4ece41415c
--- /dev/null
+++ b/django/contrib/gis/db/models/constants.py
@@ -0,0 +1,15 @@
+from django.db.models.sql.constants import QUERY_TERMS
+
+GIS_LOOKUPS = {
+    'bbcontains', 'bboverlaps', 'contained', 'contains',
+    'contains_properly', 'coveredby', 'covers', 'crosses', 'disjoint',
+    'distance_gt', 'distance_gte', 'distance_lt', 'distance_lte',
+    'dwithin', 'equals', 'exact',
+    'intersects', 'overlaps', 'relate', 'same_as', 'touches', 'within',
+    'left', 'right', 'overlaps_left', 'overlaps_right',
+    'overlaps_above', 'overlaps_below',
+    'strictly_above', 'strictly_below'
+}
+ALL_TERMS = GIS_LOOKUPS | QUERY_TERMS
+
+__all__ = ['ALL_TERMS', 'GIS_LOOKUPS']
diff --git a/django/contrib/gis/db/models/fields.py b/django/contrib/gis/db/models/fields.py
index 734f805c8d..2daf08147b 100644
--- a/django/contrib/gis/db/models/fields.py
+++ b/django/contrib/gis/db/models/fields.py
@@ -2,6 +2,8 @@ from django.db.models.fields import Field
 from django.db.models.sql.expressions import SQLEvaluator
 from django.utils.translation import ugettext_lazy as _
 from django.contrib.gis import forms
+from django.contrib.gis.db.models.constants import GIS_LOOKUPS
+from django.contrib.gis.db.models.lookups import GISLookup
 from django.contrib.gis.db.models.proxy import GeometryProxy
 from django.contrib.gis.geometry.backend import Geometry, GeometryException
 from django.utils import six
@@ -284,6 +286,10 @@ class GeometryField(Field):
         """
         return connection.ops.get_geom_placeholder(self, value)
 
+for lookup_name in GIS_LOOKUPS:
+    lookup = type(lookup_name, (GISLookup,), {'lookup_name': lookup_name})
+    GeometryField.register_lookup(lookup)
+
 
 # The OpenGIS Geometry Type Fields
 class PointField(GeometryField):
diff --git a/django/contrib/gis/db/models/lookups.py b/django/contrib/gis/db/models/lookups.py
new file mode 100644
index 0000000000..cad6c69000
--- /dev/null
+++ b/django/contrib/gis/db/models/lookups.py
@@ -0,0 +1,28 @@
+from django.db.models.lookups import Lookup
+from django.db.models.sql.expressions import SQLEvaluator
+
+
+class GISLookup(Lookup):
+    def as_sql(self, qn, connection):
+        from django.contrib.gis.db.models.sql import GeoWhereNode
+        # We use the same approach as was used by GeoWhereNode. It would
+        # be a good idea to upgrade GIS to use similar code that is used
+        # for other lookups.
+        if isinstance(self.rhs, SQLEvaluator):
+            # Make sure the F Expression destination field exists, and
+            # set an `srid` attribute with the same as that of the
+            # destination.
+            geo_fld = GeoWhereNode._check_geo_field(self.rhs.opts, self.rhs.expression.name)
+            if not geo_fld:
+                raise ValueError('No geographic field found in expression.')
+            self.rhs.srid = geo_fld.srid
+        db_type = self.lhs.output_type.db_type(connection=connection)
+        params = self.lhs.output_type.get_db_prep_lookup(
+            self.lookup_name, self.rhs, connection=connection)
+        lhs_sql, lhs_params = self.process_lhs(qn, connection)
+        # lhs_params not currently supported.
+        assert not lhs_params
+        data = (lhs_sql, db_type)
+        spatial_sql, spatial_params = connection.ops.spatial_lookup_sql(
+            data, self.lookup_name, self.rhs, self.lhs.output_type, qn)
+        return spatial_sql, spatial_params + params
diff --git a/django/contrib/gis/db/models/sql/query.py b/django/contrib/gis/db/models/sql/query.py
index 3923bf460e..f3fa1f6322 100644
--- a/django/contrib/gis/db/models/sql/query.py
+++ b/django/contrib/gis/db/models/sql/query.py
@@ -1,6 +1,7 @@
 from django.db import connections
 from django.db.models.query import sql
 
+from django.contrib.gis.db.models.constants import ALL_TERMS
 from django.contrib.gis.db.models.fields import GeometryField
 from django.contrib.gis.db.models.sql import aggregates as gis_aggregates
 from django.contrib.gis.db.models.sql.conversion import AreaField, DistanceField, GeomField
@@ -9,19 +10,6 @@ from django.contrib.gis.geometry.backend import Geometry
 from django.contrib.gis.measure import Area, Distance
 
 
-ALL_TERMS = set([
-    'bbcontains', 'bboverlaps', 'contained', 'contains',
-    'contains_properly', 'coveredby', 'covers', 'crosses', 'disjoint',
-    'distance_gt', 'distance_gte', 'distance_lt', 'distance_lte',
-    'dwithin', 'equals', 'exact',
-    'intersects', 'overlaps', 'relate', 'same_as', 'touches', 'within',
-    'left', 'right', 'overlaps_left', 'overlaps_right',
-    'overlaps_above', 'overlaps_below',
-    'strictly_above', 'strictly_below'
-])
-ALL_TERMS.update(sql.constants.QUERY_TERMS)
-
-
 class GeoQuery(sql.Query):
     """
     A single spatial SQL query.