Fixed #33646 -- Added async-compatible interface to QuerySet.

Thanks Simon Charette for reviews.

Co-authored-by: Carlton Gibson <carlton.gibson@noumenal.es>
Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>

4 files changed, 334 insertions, 21 deletions

diff --git a/docs/ref/models/querysets.txt b/docs/ref/models/querysets.txt
index 94614aad72..a9da1dcf7e 100644
--- a/docs/ref/models/querysets.txt
+++ b/docs/ref/models/querysets.txt
@@ -34,6 +34,19 @@ You can evaluate a ``QuerySet`` in the following ways:
   Note: Don't use this if all you want to do is determine if at least one
   result exists. It's more efficient to use :meth:`~QuerySet.exists`.
 
+* **Asynchronous iteration.**. A ``QuerySet`` can also be iterated over using
+  ``async for``::
+
+      async for e in Entry.objects.all():
+        results.append(e)
+
+  Both synchronous and asynchronous iterators of QuerySets share the same
+  underlying cache.
+
+  .. versionchanged:: 4.1
+
+    Support for asynchronous iteration was added.
+
 * **Slicing.** As explained in :ref:`limiting-querysets`, a ``QuerySet`` can
   be sliced, using Python's array-slicing syntax. Slicing an unevaluated
   ``QuerySet`` usually returns another unevaluated ``QuerySet``, but Django
@@ -176,6 +189,12 @@ Django provides a range of ``QuerySet`` refinement methods that modify either
 the types of results returned by the ``QuerySet`` or the way its SQL query is
 executed.
 
+.. note::
+
+    These methods do not run database queries, therefore they are **safe to**
+    **run in asynchronous code**, and do not have separate asynchronous
+    versions.
+
 ``filter()``
 ~~~~~~~~~~~~
 
@@ -1581,6 +1600,13 @@ A queryset that has deferred fields will still return model instances. Each
 deferred field will be retrieved from the database if you access that field
 (one at a time, not all the deferred fields at once).
 
+.. note::
+
+    Deferred fields will not lazy-load like this from asynchronous code.
+    Instead, you will get a ``SynchronousOnlyOperation`` exception. If you are
+    writing asynchronous code, you should not try to access any fields that you
+    ``defer()``.
+
 You can make multiple calls to ``defer()``. Each call adds new fields to the
 deferred set::
 
@@ -1703,6 +1729,11 @@ options.
 Using :meth:`only` and omitting a field requested using :meth:`select_related`
 is an error as well.
 
+As with ``defer()``, you cannot access the non-loaded fields from asynchronous
+code and expect them to load. Instead, you will get a
+``SynchronousOnlyOperation`` exception. Ensure that all fields you might access
+are in your ``only()`` call.
+
 .. note::
 
     When calling :meth:`~django.db.models.Model.save()` for instances with
@@ -1946,10 +1977,25 @@ something *other than* a ``QuerySet``.
 These methods do not use a cache (see :ref:`caching-and-querysets`). Rather,
 they query the database each time they're called.
 
+Because these methods evaluate the QuerySet, they are blocking calls, and so
+their main (synchronous) versions cannot be called from asynchronous code. For
+this reason, each has a corresponding asynchronous version with an ``a`` prefix
+- for example, rather than ``get(…)`` you can ``await aget(…)``.
+
+There is usually no difference in behavior apart from their asynchronous
+nature, but any differences are noted below next to each method.
+
+.. versionchanged:: 4.1
+
+    The asynchronous versions of each method, prefixed with ``a`` was added.
+
 ``get()``
 ~~~~~~~~~
 
 .. method:: get(*args, **kwargs)
+.. method:: aget(*args, **kwargs)
+
+*Asynchronous version*: ``aget()``
 
 Returns the object matching the given lookup parameters, which should be in
 the format described in `Field lookups`_. You should use lookups that are
@@ -1989,10 +2035,17 @@ can use :exc:`django.core.exceptions.ObjectDoesNotExist`  to handle
     except ObjectDoesNotExist:
         print("Either the blog or entry doesn't exist.")
 
+.. versionchanged:: 4.1
+
+    ``aget()`` method was added.
+
 ``create()``
 ~~~~~~~~~~~~
 
 .. method:: create(**kwargs)
+.. method:: acreate(*args, **kwargs)
+
+*Asynchronous version*: ``acreate()``
 
 A convenience method for creating an object and saving it all in one step.  Thus::
 
@@ -2013,10 +2066,17 @@ database, a call to ``create()`` will fail with an
 :exc:`~django.db.IntegrityError` since primary keys must be unique. Be
 prepared to handle the exception if you are using manual primary keys.
 
+.. versionchanged:: 4.1
+
+    ``acreate()`` method was added.
+
 ``get_or_create()``
 ~~~~~~~~~~~~~~~~~~~
 
 .. method:: get_or_create(defaults=None, **kwargs)
+.. method:: aget_or_create(defaults=None, **kwargs)
+
+*Asynchronous version*: ``aget_or_create()``
 
 A convenience method for looking up an object with the given ``kwargs`` (may be
 empty if your model has defaults for all fields), creating one if necessary.
@@ -2138,10 +2198,17 @@ whenever a request to a page has a side effect on your data. For more, see
   chapter because it isn't related to that book, but it can't create it either
   because ``title`` field should be unique.
 
+.. versionchanged:: 4.1
+
+    ``aget_or_create()`` method was added.
+
 ``update_or_create()``
 ~~~~~~~~~~~~~~~~~~~~~~
 
 .. method:: update_or_create(defaults=None, **kwargs)
+.. method:: aupdate_or_create(defaults=None, **kwargs)
+
+*Asynchronous version*: ``aupdate_or_create()``
 
 A convenience method for updating an object with the given ``kwargs``, creating
 a new one if necessary. The ``defaults`` is a dictionary of (field, value)
@@ -2188,10 +2255,17 @@ Like :meth:`get_or_create` and :meth:`create`, if you're using manually
 specified primary keys and an object needs to be created but the key already
 exists in the database, an :exc:`~django.db.IntegrityError` is raised.
 
+.. versionchanged:: 4.1
+
+    ``aupdate_or_create()`` method was added.
+
 ``bulk_create()``
 ~~~~~~~~~~~~~~~~~
 
 .. method:: bulk_create(objs, batch_size=None, ignore_conflicts=False, update_conflicts=False, update_fields=None, unique_fields=None)
+.. method:: abulk_create(objs, batch_size=None, ignore_conflicts=False, update_conflicts=False, update_fields=None, unique_fields=None)
+
+*Asynchronous version*: ``abulk_create()``
 
 This method inserts the provided list of objects into the database in an
 efficient manner (generally only 1 query, no matter how many objects there
@@ -2267,10 +2341,15 @@ support it).
     parameters were added to support updating fields when a row insertion fails
     on conflict.
 
+    ``abulk_create()`` method was added.
+
 ``bulk_update()``
 ~~~~~~~~~~~~~~~~~
 
 .. method:: bulk_update(objs, fields, batch_size=None)
+.. method:: abulk_update(objs, fields, batch_size=None)
+
+*Asynchronous version*: ``abulk_update()``
 
 This method efficiently updates the given fields on the provided model
 instances, generally with one query, and returns the number of objects
@@ -2313,10 +2392,17 @@ The ``batch_size`` parameter controls how many objects are saved in a single
 query. The default is to update all objects in one batch, except for SQLite
 and Oracle which have restrictions on the number of variables used in a query.
 
+.. versionchanged:: 4.1
+
+    ``abulk_update()`` method was added.
+
 ``count()``
 ~~~~~~~~~~~
 
 .. method:: count()
+.. method:: acount()
+
+*Asynchronous version*: ``acount()``
 
 Returns an integer representing the number of objects in the database matching
 the ``QuerySet``.
@@ -2342,10 +2428,17 @@ database query like ``count()`` would.
 If the queryset has already been fully retrieved, ``count()`` will use that
 length rather than perform an extra database query.
 
+.. versionchanged:: 4.1
+
+    ``acount()`` method was added.
+
 ``in_bulk()``
 ~~~~~~~~~~~~~
 
 .. method:: in_bulk(id_list=None, *, field_name='pk')
+.. method:: ain_bulk(id_list=None, *, field_name='pk')
+
+*Asynchronous version*: ``ain_bulk()``
 
 Takes a list of field values (``id_list``) and the ``field_name`` for those
 values, and returns a dictionary mapping each value to an instance of the
@@ -2374,19 +2467,29 @@ Example::
 
 If you pass ``in_bulk()`` an empty list, you'll get an empty dictionary.
 
+.. versionchanged:: 4.1
+
+    ``ain_bulk()`` method was added.
+
 ``iterator()``
 ~~~~~~~~~~~~~~
 
 .. method:: iterator(chunk_size=None)
+.. method:: aiterator(chunk_size=None)
+
+*Asynchronous version*: ``aiterator()``
 
 Evaluates the ``QuerySet`` (by performing the query) and returns an iterator
-(see :pep:`234`) over the results. A ``QuerySet`` typically caches its results
-internally so that repeated evaluations do not result in additional queries. In
-contrast, ``iterator()`` will read results directly, without doing any caching
-at the ``QuerySet`` level (internally, the default iterator calls ``iterator()``
-and caches the return value). For a ``QuerySet`` which returns a large number of
-objects that you only need to access once, this can result in better
-performance and a significant reduction in memory.
+(see :pep:`234`) over the results, or an asynchronous iterator (see :pep:`492`)
+if you call its asynchronous version ``aiterator``.
+
+A ``QuerySet`` typically caches its results internally so that repeated
+evaluations do not result in additional queries. In contrast, ``iterator()``
+will read results directly, without doing any caching at the ``QuerySet`` level
+(internally, the default iterator calls ``iterator()`` and caches the return
+value). For a ``QuerySet`` which returns a large number of objects that you
+only need to access once, this can result in better performance and a
+significant reduction in memory.
 
 Note that using ``iterator()`` on a ``QuerySet`` which has already been
 evaluated will force it to evaluate again, repeating the query.
@@ -2395,6 +2498,11 @@ evaluated will force it to evaluate again, repeating the query.
 long as ``chunk_size`` is given. Larger values will necessitate fewer queries
 to accomplish the prefetching at the cost of greater memory usage.
 
+.. note::
+
+    ``aiterator()`` is *not* compatible with previous calls to
+    ``prefetch_related()``.
+
 On some databases (e.g. Oracle, `SQLite
 <https://www.sqlite.org/limits.html#max_variable_number>`_), the maximum number
 of terms in an SQL ``IN`` clause might be limited. Hence values below this
@@ -2411,7 +2519,9 @@ once or streamed from the database using server-side cursors.
 
 .. versionchanged:: 4.1
 
-    Support for prefetching related objects was added.
+    Support for prefetching related objects was added to ``iterator()``.
+
+    ``aiterator()`` method was added.
 
 .. deprecated:: 4.1
 
@@ -2471,6 +2581,9 @@ value for ``chunk_size`` will result in Django using an implicit default of
 ~~~~~~~~~~~~
 
 .. method:: latest(*fields)
+.. method:: alatest(*fields)
+
+*Asynchronous version*: ``alatest()``
 
 Returns the latest object in the table based on the given field(s).
 
@@ -2512,18 +2625,32 @@ readability.
 
         Entry.objects.filter(pub_date__isnull=False).latest('pub_date')
 
+.. versionchanged:: 4.1
+
+    ``alatest()`` method was added.
+
 ``earliest()``
 ~~~~~~~~~~~~~~
 
 .. method:: earliest(*fields)
+.. method:: aearliest(*fields)
+
+*Asynchronous version*: ``aearliest()``
 
 Works otherwise like :meth:`~django.db.models.query.QuerySet.latest` except
 the direction is changed.
 
+.. versionchanged:: 4.1
+
+    ``aearliest()`` method was added.
+
 ``first()``
 ~~~~~~~~~~~
 
 .. method:: first()
+.. method:: afirst()
+
+*Asynchronous version*: ``afirst()``
 
 Returns the first object matched by the queryset, or ``None`` if there
 is no matching object. If the ``QuerySet`` has no ordering defined, then the
@@ -2542,17 +2669,31 @@ equivalent to the above example::
     except IndexError:
         p = None
 
+.. versionchanged:: 4.1
+
+    ``afirst()`` method was added.
+
 ``last()``
 ~~~~~~~~~~
 
 .. method:: last()
+.. method:: alast()
+
+*Asynchronous version*: ``alast()``
 
 Works like  :meth:`first()`, but returns the last object in the queryset.
 
+.. versionchanged:: 4.1
+
+    ``alast()`` method was added.
+
 ``aggregate()``
 ~~~~~~~~~~~~~~~
 
 .. method:: aggregate(*args, **kwargs)
+.. method:: aaggregate(*args, **kwargs)
+
+*Asynchronous version*: ``aaggregate()``
 
 Returns a dictionary of aggregate values (averages, sums, etc.) calculated over
 the ``QuerySet``. Each argument to ``aggregate()`` specifies a value that will
@@ -2585,10 +2726,17 @@ control the name of the aggregation value that is returned::
 For an in-depth discussion of aggregation, see :doc:`the topic guide on
 Aggregation </topics/db/aggregation>`.
 
+.. versionchanged:: 4.1
+
+    ``aaggregate()`` method was added.
+
 ``exists()``
 ~~~~~~~~~~~~
 
 .. method:: exists()
+.. method:: aexists()
+
+*Asynchronous version*: ``aexists()``
 
 Returns ``True`` if the :class:`.QuerySet` contains any results, and ``False``
 if not. This tries to perform the query in the simplest and fastest way
@@ -2618,10 +2766,17 @@ more overall work (one query for the existence check plus an extra one to later
 retrieve the results) than using ``bool(some_queryset)``, which retrieves the
 results and then checks if any were returned.
 
+.. versionchanged:: 4.1
+
+    ``aexists()`` method was added.
+
 ``contains()``
 ~~~~~~~~~~~~~~
 
 .. method:: contains(obj)
+.. method:: acontains(obj)
+
+*Asynchronous version*: ``acontains()``
 
 .. versionadded:: 4.0
 
@@ -2647,10 +2802,17 @@ know that it will be at some point, then using ``some_queryset.contains(obj)``
 will make an additional database query, generally resulting in slower overall
 performance.
 
+.. versionchanged:: 4.1
+
+    ``acontains()`` method was added.
+
 ``update()``
 ~~~~~~~~~~~~
 
 .. method:: update(**kwargs)
+.. method:: aupdate(**kwargs)
+
+*Asynchronous version*: ``aupdate()``
 
 Performs an SQL update query for the specified fields, and returns
 the number of rows matched (which may not be equal to the number of rows
@@ -2721,6 +2883,10 @@ update a bunch of records for a model that has a custom
         e.comments_on = False
         e.save()
 
+.. versionchanged:: 4.1
+
+    ``aupdate()`` method was added.
+
 Ordered queryset
 ^^^^^^^^^^^^^^^^
 
@@ -2739,6 +2905,9 @@ unique field in the order that is specified without conflicts. For example::
 ~~~~~~~~~~~~
 
 .. method:: delete()
+.. method:: adelete()
+
+*Asynchronous version*: ``adelete()``
 
 Performs an SQL delete query on all rows in the :class:`.QuerySet` and
 returns the number of objects deleted and a dictionary with the number of
@@ -2789,6 +2958,10 @@ ForeignKeys which are set to :attr:`~django.db.models.ForeignKey.on_delete`
 Note that the queries generated in object deletion is an implementation
 detail subject to change.
 
+.. versionchanged:: 4.1
+
+    ``adelete()`` method was added.
+
 ``as_manager()``
 ~~~~~~~~~~~~~~~~
 
@@ -2798,10 +2971,16 @@ Class method that returns an instance of :class:`~django.db.models.Manager`
 with a copy of the ``QuerySet``’s methods. See
 :ref:`create-manager-with-queryset-methods` for more details.
 
+Note that unlike the other entries in this section, this does not have an
+asynchronous variant as it does not execute a query.
+
 ``explain()``
 ~~~~~~~~~~~~~
 
 .. method:: explain(format=None, **options)
+.. method:: aexplain(format=None, **options)
+
+*Asynchronous version*: ``aexplain()``
 
 Returns a string of the ``QuerySet``’s execution plan, which details how the
 database would execute the query, including any indexes or joins that would be
@@ -2841,6 +3020,10 @@ adverse effects on your database. For example, the ``ANALYZE`` flag supported
 by MariaDB, MySQL 8.0.18+, and PostgreSQL could result in changes to data if
 there are triggers or if a function is called, even for a ``SELECT`` query.
 
+.. versionchanged:: 4.1
+
+    ``aexplain()`` method was added.
+
 .. _field-lookups:
 
 ``Field`` lookups
diff --git a/docs/releases/4.1.txt b/docs/releases/4.1.txt
index 0de527447b..b564a325a9 100644
--- a/docs/releases/4.1.txt
+++ b/docs/releases/4.1.txt
@@ -43,6 +43,28 @@ View subclasses may now define async HTTP method handlers::
 
 See :ref:`async-class-based-views` for more details.
 
+Asynchronous ORM interface
+--------------------------
+
+``QuerySet`` now provides an asynchronous interface for all data access
+operations. These are named as-per the existing synchronous operations but with
+an ``a`` prefix, for example ``acreate()``, ``aget()``, and so on.
+
+The new interface allows you to write asynchronous code without needing to wrap
+ORM operations in ``sync_to_async()``::
+
+    async for author in Author.objects.filter(name__startswith="A"):
+        book = await author.books.afirst()
+
+Note that, at this stage, the underlying database operations remain
+synchronous, with contributions ongoing to push asynchronous support down into
+the SQL compiler, and integrate asynchronous database drivers. The new
+asynchronous queryset interface currently encapsulates the necessary
+``sync_to_async()`` operations for you, and will allow your code to take
+advantage of developments in the ORM's asynchronous support as it evolves.
+
+See :ref:`async-queries` for details and limitations.
+
 .. _csrf-cookie-masked-usage:
 
 ``CSRF_COOKIE_MASKED`` setting
diff --git a/docs/topics/async.txt b/docs/topics/async.txt
index 101bbeeabb..9e2785c351 100644
--- a/docs/topics/async.txt
+++ b/docs/topics/async.txt
@@ -61,27 +61,39 @@ In both ASGI and WSGI mode, you can still safely use asynchronous support to
 run code concurrently rather than serially. This is especially handy when
 dealing with external APIs or data stores.
 
-If you want to call a part of Django that is still synchronous, like the ORM,
-you will need to wrap it in a :func:`sync_to_async` call. For example::
+If you want to call a part of Django that is still synchronous, you will need
+to wrap it in a :func:`sync_to_async` call. For example::
 
     from asgiref.sync import sync_to_async
 
-    results = await sync_to_async(Blog.objects.get, thread_sensitive=True)(pk=123)
+    results = await sync_to_async(sync_function, thread_sensitive=True)(pk=123)
 
-You may find it easier to move any ORM code into its own function and call that
-entire function using :func:`sync_to_async`. For example::
+If you accidentally try to call a part of Django that is synchronous-only
+from an async view, you will trigger Django's
+:ref:`asynchronous safety protection <async-safety>` to protect your data from
+corruption.
 
-    from asgiref.sync import sync_to_async
+Queries & the ORM
+-----------------
 
-    def _get_blog(pk):
-        return Blog.objects.select_related('author').get(pk=pk)
+.. versionadded:: 4.1
 
-    get_blog = sync_to_async(_get_blog, thread_sensitive=True)
+With some exceptions, Django can run ORM queries asynchronously as well::
 
-If you accidentally try to call a part of Django that is still synchronous-only
-from an async view, you will trigger Django's
-:ref:`asynchronous safety protection <async-safety>` to protect your data from
-corruption.
+    async for author in Author.objects.filter(name__startswith="A"):
+        book = await author.books.afirst()
+
+Detailed notes can be found in :ref:`async-queries`, but in short:
+
+* All ``QuerySet`` methods that cause a SQL query to occur have an
+  ``a``-prefixed asynchronous variant.
+
+* ``async for`` is supported on all QuerySets (including the output of
+  ``values()`` and ``values_list()``.)
+
+Transactions do not yet work in async mode. If you have a piece of code that
+needs transactions behavior, we recommend you write that piece as a single
+synchronous function and call it using :func:`sync_to_async`.
 
 Performance
 -----------
diff --git a/docs/topics/db/queries.txt b/docs/topics/db/queries.txt
index 8118532513..af26462367 100644
--- a/docs/topics/db/queries.txt
+++ b/docs/topics/db/queries.txt
@@ -849,6 +849,102 @@ being evaluated and therefore populate the cache::
     Simply printing the queryset will not populate the cache. This is because
     the call to ``__repr__()`` only returns a slice of the entire queryset.
 
+.. _async-queries:
+
+Asynchronous queries
+====================
+
+.. versionadded:: 4.1
+
+If you are writing asynchronous views or code, you cannot use the ORM for
+queries in quite the way we have described above, as you cannot call *blocking*
+synchronous code from asynchronous code - it will block up the event loop
+(or, more likely, Django will notice and raise a ``SynchronousOnlyOperation``
+to stop that from happening).
+
+Fortunately, you can do many queries using Django's asynchronous query APIs.
+Every method that might block - such as ``get()`` or ``delete()`` - has an
+asynchronous variant (``aget()`` or ``adelete()``), and when you iterate over
+results, you can use asynchronous iteration (``async for``) instead.
+
+Query iteration
+---------------
+
+.. versionadded:: 4.1
+
+The default way of iterating over a query - with ``for`` - will result in a
+blocking database query behind the scenes as Django loads the results at
+iteration time. To fix this, you can swap to ``async for``::
+
+    async for entry in Authors.objects.filter(name__startswith="A"):
+        ...
+
+Be aware that you also can't do other things that might iterate over the
+queryset, such as wrapping ``list()`` around it to force its evaluation (you
+can use ``async for`` in a comprehension, if you want it).
+
+Because ``QuerySet`` methods like ``filter()`` and ``exclude()`` do not
+actually run the query - they set up the queryset to run when it's iterated
+over - you can use those freely in asynchronous code. For a guide to which
+methods can keep being used like this, and which have asynchronous versions,
+read the next section.
+
+``QuerySet`` and manager methods
+--------------------------------
+
+.. versionadded:: 4.1
+
+Some methods on managers and querysets - like ``get()`` and ``first()`` - force
+execution of the queryset and are blocking. Some, like ``filter()`` and
+``exclude()``, don't force execution and so are safe to run from asynchronous
+code. But how are you supposed to tell the difference?
+
+While you could poke around and see if there is an ``a``-prefixed version of
+the method (for example, we have ``aget()`` but not ``afilter()``), there is a
+more logical way - look up what kind of method it is in the
+:doc:`QuerySet reference </ref/models/querysets>`.
+
+In there, you'll find the methods on QuerySets grouped into two sections:
+
+* *Methods that return new querysets*: These are the non-blocking ones,
+  and don't have asynchronous versions. You're free to use these in any
+  situation, though read the notes on ``defer()`` and ``only()`` before you use
+  them.
+
+* *Methods that do not return querysets*: These are the blocking ones, and
+  have asynchronous versions - the asynchronous name for each is noted in its
+  documentation, though our standard pattern is to add an ``a`` prefix.
+
+Using this distinction, you can work out when you need to use asynchronous
+versions, and when you don't. For example, here's a valid asynchronous query::
+
+    user = await User.objects.filter(username=my_input).afirst()
+
+``filter()`` returns a queryset, and so it's fine to keep chaining it inside an
+asynchronous environment, whereas ``first()`` evaluates and returns a model
+instance - thus, we change to ``afirst()``, and use ``await`` at the front of
+the whole expression in order to call it in an asynchronous-friendly way.
+
+.. note::
+
+    If you forget to put the ``await`` part in, you may see errors like
+    *"coroutine object has no attribute x"* or *"<coroutine …>"* strings in
+    place of your model instances. If you ever see these, you are missing an
+    ``await`` somewhere to turn that coroutine into a real value.
+
+Transactions
+------------
+
+.. versionadded:: 4.1
+
+Transactions are **not** currently supported with asynchronous queries and
+updates. You will find that trying to use one raises
+``SynchronousOnlyOperation``.
+
+If you wish to use a transaction, we suggest you write your ORM code inside a
+separate, synchronous function and then call that using sync_to_async - see
+:doc:/topics/async` for more.
+
 .. _querying-jsonfield:
 
 Querying ``JSONField``