diff options
| author | Jorge C. Leitão <jorgecarleitao@gmail.com> | 2014-06-11 17:09:10 +0200 |
|---|---|---|
| committer | Tim Graham <timograham@gmail.com> | 2014-07-08 20:05:49 -0400 |
| commit | 8780849da06f866720284d2c61d0aae5890e92ae (patch) | |
| tree | bd06a66b22d673b27db99c3fe7a5ff5eda6e948f /docs/howto | |
| parent | 503e59c9b0cfd93a47dc1921452cece49055451b (diff) | |
Fixed #22812 -- Refactored lookup API documentation.
Thanks Anssi and Tim for reviews.
Diffstat (limited to 'docs/howto')
| -rw-r--r-- | docs/howto/custom-lookups.txt | 268 | ||||
| -rw-r--r-- | docs/howto/custom-model-fields.txt | 2 | ||||
| -rw-r--r-- | docs/howto/index.txt | 1 |
3 files changed, 270 insertions, 1 deletions
diff --git a/docs/howto/custom-lookups.txt b/docs/howto/custom-lookups.txt new file mode 100644 index 0000000000..af7c18e38c --- /dev/null +++ b/docs/howto/custom-lookups.txt @@ -0,0 +1,268 @@ +============== +Custom Lookups +============== + +.. versionadded:: 1.7 + +.. currentmodule:: django.db.models + +Django offers a wide variety of :ref:`built-in lookups <field-lookups>` for +filtering (for example, ``exact`` and ``icontains``). This documentation +explains how to write custom lookups and how to alter the working of existing +lookups. For the API references of lookups, see the :doc:`/ref/models/lookups`. + +A simple lookup example +~~~~~~~~~~~~~~~~~~~~~~~ + +Let's start with a simple custom lookup. We will write a custom lookup ``ne`` +which works opposite to ``exact``. ``Author.objects.filter(name__ne='Jack')`` +will translate to the SQL:: + + "author"."name" <> 'Jack' + +This SQL is backend independent, so we don't need to worry about different +databases. + +There are two steps to making this work. Firstly we need to implement the +lookup, then we need to tell Django about it. The implementation is quite +straightforward:: + + from django.db.models import Lookup + + class NotEqual(Lookup): + lookup_name = 'ne' + + def as_sql(self, qn, connection): + lhs, lhs_params = self.process_lhs(qn, connection) + rhs, rhs_params = self.process_rhs(qn, connection) + params = lhs_params + rhs_params + return '%s <> %s' % (lhs, rhs), params + +To register the ``NotEqual`` lookup we will just need to call +``register_lookup`` on the field class we want the lookup to be available. In +this case, the lookup makes sense on all ``Field`` subclasses, so we register +it with ``Field`` directly:: + + from django.db.models.fields import Field + Field.register_lookup(NotEqual) + +We can now use ``foo__ne`` for any field ``foo``. You will need to ensure that +this registration happens before you try to create any querysets using it. You +could place the implementation in a ``models.py`` file, or register the lookup +in the ``ready()`` method of an ``AppConfig``. + +Taking a closer look at the implementation, the first required attribute is +``lookup_name``. This allows the ORM to understand how to interpret ``name__ne`` +and use ``NotEqual`` to generate the SQL. By convention, these names are always +lowercase strings containing only letters, but the only hard requirement is +that it must not contain the string ``__``. + +We then need to define the ``as_sql`` method. This takes a ``SQLCompiler`` +object, called ``qn``, and the active database connection. ``SQLCompiler`` +objects are not documented, but the only thing we need to know about them is +that they have a ``compile()`` method which returns a tuple containing a SQL +string, and the parameters to be interpolated into that string. In most cases, +you don't need to use it directly and can pass it on to ``process_lhs()`` and +``process_rhs()``. + +A ``Lookup`` works against two values, ``lhs`` and ``rhs``, standing for +left-hand side and right-hand side. The left-hand side is usually a field +reference, but it can be anything implementing the :ref:`query expression API +<query-expression>`. The right-hand is the value given by the user. In the +example ``Author.objects.filter(name__ne='Jack')``, the left-hand side is a +reference to the ``name`` field of the ``Author`` model, and ``'Jack'`` is the +right-hand side. + +We call ``process_lhs`` and ``process_rhs`` to convert them into the values we +need for SQL using the ``qn`` object described before. These methods return +tuples containing some SQL and the parameters to be interpolated into that SQL, +just as we need to return from our ``as_sql`` method. In the above example, +``process_lhs`` returns ``('"author"."name"', [])`` and ``process_rhs`` returns +``('"%s"', ['Jack'])``. In this example there were no parameters for the left +hand side, but this would depend on the object we have, so we still need to +include them in the parameters we return. + +Finally we combine the parts into a SQL expression with ``<>``, and supply all +the parameters for the query. We then return a tuple containing the generated +SQL string and the parameters. + +A simple transformer example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The custom lookup above is great, but in some cases you may want to be able to +chain lookups together. For example, let's suppose we are building an +application where we want to make use of the ``abs()`` operator. +We have an ``Experiment`` model which records a start value, end value, and the +change (start - end). We would like to find all experiments where the change +was equal to a certain amount (``Experiment.objects.filter(change__abs=27)``), +or where it did not exceed a certain amount +(``Experiment.objects.filter(change__abs__lt=27)``). + +.. note:: + This example is somewhat contrived, but it nicely demonstrates the range of + functionality which is possible in a database backend independent manner, + and without duplicating functionality already in Django. + +We will start by writing a ``AbsoluteValue`` transformer. This will use the SQL +function ``ABS()`` to transform the value before comparison:: + + from django.db.models import Transform + + class AbsoluteValue(Transform): + lookup_name = 'abs' + + def as_sql(self, qn, connection): + lhs, params = qn.compile(self.lhs) + return "ABS(%s)" % lhs, params + +Next, lets register it for ``IntegerField``:: + + from django.db.models import IntegerField + IntegerField.register_lookup(AbsoluteValue) + +We can now run the queries we had before. +``Experiment.objects.filter(change__abs=27)`` will generate the following SQL:: + + SELECT ... WHERE ABS("experiments"."change") = 27 + +By using ``Transform`` instead of ``Lookup`` it means we are able to chain +further lookups afterwards. So +``Experiment.objects.filter(change__abs__lt=27)`` will generate the following +SQL:: + + SELECT ... WHERE ABS("experiments"."change") < 27 + +Subclasses of ``Transform`` usually only operate on the left-hand side of the +expression. Further lookups will work on the transformed value. Note that in +this case where there is no other lookup specified, Django interprets +``change__abs=27`` as ``change__abs__exact=27``. + +When looking for which lookups are allowable after the ``Transform`` has been +applied, Django uses the ``output_field`` attribute. We didn't need to specify +this here as it didn't change, but supposing we were applying ``AbsoluteValue`` +to some field which represents a more complex type (for example a point +relative to an origin, or a complex number) then we may have wanted to specify +``output_field = FloatField``, which will ensure that further lookups like +``abs__lte`` behave as they would for a ``FloatField``. + +Writing an efficient abs__lt lookup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When using the above written ``abs`` lookup, the SQL produced will not use +indexes efficiently in some cases. In particular, when we use +``change__abs__lt=27``, this is equivalent to ``change__gt=-27`` AND +``change__lt=27``. (For the ``lte`` case we could use the SQL ``BETWEEN``). + +So we would like ``Experiment.objects.filter(change__abs__lt=27)`` to generate +the following SQL:: + + SELECT .. WHERE "experiments"."change" < 27 AND "experiments"."change" > -27 + +The implementation is:: + + from django.db.models import Lookup + + class AbsoluteValueLessThan(Lookup): + lookup_name = 'lt' + + def as_sql(self, qn, connection): + lhs, lhs_params = qn.compile(self.lhs.lhs) + rhs, rhs_params = self.process_rhs(qn, connection) + params = lhs_params + rhs_params + lhs_params + rhs_params + return '%s < %s AND %s > -%s' % (lhs, rhs, lhs, rhs), params + + AbsoluteValue.register_lookup(AbsoluteValueLessThan) + +There are a couple of notable things going on. First, ``AbsoluteValueLessThan`` +isn't calling ``process_lhs()``. Instead it skips the transformation of the +``lhs`` done by ``AbsoluteValue`` and uses the original ``lhs``. That is, we +want to get ``27`` not ``ABS(27)``. Referring directly to ``self.lhs.lhs`` is +safe as ``AbsoluteValueLessThan`` can be accessed only from the +``AbsoluteValue`` lookup, that is the ``lhs`` is always an instance of +``AbsoluteValue``. + +Notice also that as both sides are used multiple times in the query the params +need to contain ``lhs_params`` and ``rhs_params`` multiple times. + +The final query does the inversion (``27`` to ``-27``) directly in the +database. The reason for doing this is that if the self.rhs is something else +than a plain integer value (for example an ``F()`` reference) we can't do the +transformations in Python. + +.. note:: + In fact, most lookups with ``__abs`` could be implemented as range queries + like this, and on most database backends it is likely to be more sensible to + do so as you can make use of the indexes. However with PostgreSQL you may + want to add an index on ``abs(change)`` which would allow these queries to + be very efficient. + +Writing alternative implementations for existing lookups +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sometimes different database vendors require different SQL for the same +operation. For this example we will rewrite a custom implementation for +MySQL for the NotEqual operator. Instead of ``<>`` we will be using ``!=`` +operator. (Note that in reality almost all databases support both, including +all the official databases supported by Django). + +We can change the behavior on a specific backend by creating a subclass of +``NotEqual`` with a ``as_mysql`` method:: + + class MySQLNotEqual(NotEqual): + def as_mysql(self, qn, connection): + lhs, lhs_params = self.process_lhs(qn, connection) + rhs, rhs_params = self.process_rhs(qn, connection) + params = lhs_params + rhs_params + return '%s != %s' % (lhs, rhs), params + Field.register_lookup(MySQLNotExact) + +We can then register it with ``Field``. It takes the place of the original +``NotEqual`` class as it has the same ``lookup_name``. + +When compiling a query, Django first looks for ``as_%s % connection.vendor`` +methods, and then falls back to ``as_sql``. The vendor names for the in-built +backends are ``sqlite``, ``postgresql``, ``oracle`` and ``mysql``. + +How Django determines the lookups and transforms which are used +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In some cases you may which to dynamically change which ``Transform`` or +``Lookup`` is returned based on the name passed in, rather than fixing it. As +an example, you could have a field which stores coordinates or an arbitrary +dimension, and wish to allow a syntax like ``.filter(coords__x7=4)`` to return +the objects where the 7th coordinate has value 4. In order to do this, you +would override ``get_lookup`` with something like:: + + class CoordinatesField(Field): + def get_lookup(self, lookup_name): + if lookup_name.startswith('x'): + try: + dimension = int(lookup_name[1:]) + except ValueError: + pass + finally: + return get_coordinate_lookup(dimension) + return super(CoordinatesField, self).get_lookup(lookup_name) + +You would then define ``get_coordinate_lookup`` appropriately to return a +``Lookup`` subclass which handles the relevant value of ``dimension``. + +There is a similarly named method called ``get_transform()``. ``get_lookup()`` +should always return a ``Lookup`` subclass, and ``get_transform()`` a +``Transform`` subclass. It is important to remember that ``Transform`` +objects can be further filtered on, and ``Lookup`` objects cannot. + +When filtering, if there is only one lookup name remaining to be resolved, we +will look for a ``Lookup``. If there are multiple names, it will look for a +``Transform``. In the situation where there is only one name and a ``Lookup`` +is not found, we look for a ``Transform`` and then the ``exact`` lookup on that +``Transform``. All call sequences always end with a ``Lookup``. To clarify: + +- ``.filter(myfield__mylookup)`` will call ``myfield.get_lookup('mylookup')``. +- ``.filter(myfield__mytransform__mylookup)`` will call + ``myfield.get_transform('mytransform')``, and then + ``mytransform.get_lookup('mylookup')``. +- ``.filter(myfield__mytransform)`` will first call + ``myfield.get_lookup('mytransform')``, which will fail, so it will fall back + to calling ``myfield.get_transform('mytransform')`` and then + ``mytransform.get_lookup('exact')``. diff --git a/docs/howto/custom-model-fields.txt b/docs/howto/custom-model-fields.txt index 4d06634856..65278d13f0 100644 --- a/docs/howto/custom-model-fields.txt +++ b/docs/howto/custom-model-fields.txt @@ -679,7 +679,7 @@ Django filter lookups: ``exact``, ``iexact``, ``contains``, ``icontains``, .. versionadded:: 1.7 - If you are using :doc:`Custom lookups </ref/models/custom-lookups>` the + If you are using :doc:`Custom lookups </howto/custom-lookups>` the ``lookup_type`` can be any ``lookup_name`` used by the project's custom lookups. diff --git a/docs/howto/index.txt b/docs/howto/index.txt index 80e20923ed..091afe69bc 100644 --- a/docs/howto/index.txt +++ b/docs/howto/index.txt @@ -12,6 +12,7 @@ you quickly accomplish common tasks. auth-remote-user custom-management-commands custom-model-fields + custom-lookups custom-template-tags custom-file-storage deployment/index |
