Fixed #28586 -- Added model field fetch modes.

May your database queries be much reduced with minimal effort.

co-authored-by: Andreas Pelme <andreas@pelme.se>
co-authored-by: Simon Charette <charette.s@gmail.com>
co-authored-by: Jacob Walls <jacobtylerwalls@gmail.com>

9 files changed, 275 insertions, 42 deletions

diff --git a/docs/ref/exceptions.txt b/docs/ref/exceptions.txt
index bbd959e95d..93c6ec4203 100644
--- a/docs/ref/exceptions.txt
+++ b/docs/ref/exceptions.txt
@@ -165,6 +165,16 @@ Django core exception classes are defined in ``django.core.exceptions``.
     - A field name is invalid
     - A query contains invalid order_by arguments
 
+``FieldFetchBlocked``
+---------------------
+
+.. versionadded:: 6.1
+
+.. exception:: FieldFetchBlocked
+
+    Raised when a field would be fetched on-demand and the
+    :attr:`~django.db.models.RAISE` fetch mode is active.
+
 ``ValidationError``
 -------------------
 
diff --git a/docs/ref/models/instances.txt b/docs/ref/models/instances.txt
index c8cf5957ba..2ce8dc4a36 100644
--- a/docs/ref/models/instances.txt
+++ b/docs/ref/models/instances.txt
@@ -180,10 +180,10 @@ update, you could write a test similar to this::
         obj.refresh_from_db()
         self.assertEqual(obj.val, 2)
 
-Note that when deferred fields are accessed, the loading of the deferred
-field's value happens through this method. Thus it is possible to customize
-the way deferred loading happens. The example below shows how one can reload
-all of the instance's fields when a deferred field is reloaded::
+When a deferred field is loaded on-demand for a single model instance, the
+loading happens through this method. Thus it is possible to customize the way
+this loading happens. The example below shows how one can reload all of the
+instance's fields when a deferred field is loaded on-demand::
 
     class ExampleModel(models.Model):
         def refresh_from_db(self, using=None, fields=None, **kwargs):
diff --git a/docs/ref/models/querysets.txt b/docs/ref/models/querysets.txt
index f290970d2c..3840a2f97e 100644
--- a/docs/ref/models/querysets.txt
+++ b/docs/ref/models/querysets.txt
@@ -1022,15 +1022,38 @@ Uses SQL's ``EXCEPT`` operator to keep only elements present in the
 
 See :meth:`union` for some restrictions.
 
+``fetch_mode()``
+~~~~~~~~~~~~~~~~
+
+.. versionadded:: 6.1
+
+.. method:: fetch_mode(mode)
+
+Returns a ``QuerySet`` that sets the given fetch mode for all model instances
+created by this ``QuerySet``. The fetch mode controls on-demand loading of
+fields when they are accessed, such as for foreign keys and deferred fields.
+For example, to use the :attr:`~django.db.models.FETCH_PEERS` mode to
+batch-load all related objects on first access:
+
+.. code-block:: python
+
+    from django.db import models
+
+    books = Book.objects.fetch_mode(models.FETCH_PEERS)
+
+See more in the :doc:`fetch mode topic guide </topics/db/fetch-modes>`.
+
 ``select_related()``
 ~~~~~~~~~~~~~~~~~~~~
 
 .. method:: select_related(*fields)
 
-Returns a ``QuerySet`` that will "follow" foreign-key relationships, selecting
-additional related-object data when it executes its query. This is a
-performance booster which results in a single more complex query but means
-later use of foreign-key relationships won't require database queries.
+Returns a ``QuerySet`` that will join in the named foreign-key relationships,
+selecting additional related objects when it executes its query. This method
+can be a performance booster, fetching data ahead of time rather than
+triggering on-demand loading through the model instances'
+:doc:`fetch mode </topics/db/fetch-modes>`, at the cost of a more complex
+initial query.
 
 The following examples illustrate the difference between plain lookups and
 ``select_related()`` lookups. Here's standard lookup::
@@ -1050,20 +1073,8 @@ And here's ``select_related`` lookup::
     # in the previous query.
     b = e.blog
 
-You can use ``select_related()`` with any queryset of objects::
-
-    from django.utils import timezone
-
-    # Find all the blogs with entries scheduled to be published in the future.
-    blogs = set()
-
-    for e in Entry.objects.filter(pub_date__gt=timezone.now()).select_related("blog"):
-        # Without select_related(), this would make a database query for each
-        # loop iteration in order to fetch the related blog for each entry.
-        blogs.add(e.blog)
-
-The order of ``filter()`` and ``select_related()`` chaining isn't important.
-These querysets are equivalent::
+You can use ``select_related()`` with any queryset. The order of chaining with
+other methods isn't important. For example, these querysets are equivalent::
 
     Entry.objects.filter(pub_date__gt=timezone.now()).select_related("blog")
     Entry.objects.select_related("blog").filter(pub_date__gt=timezone.now())
@@ -1141,12 +1152,15 @@ that is that ``select_related('foo', 'bar')`` is equivalent to
 
 .. method:: prefetch_related(*lookups)
 
-Returns a ``QuerySet`` that will automatically retrieve, in a single batch,
-related objects for each of the specified lookups.
+Returns a ``QuerySet`` that will automatically retrieve the given lookups, each
+in one extra batch query. Prefetching is a way to optimize database access
+when you know you'll be accessing related objects later, so you can avoid
+triggering the on-demand loading behavior of the model instances'
+:doc:`fetch mode </topics/db/fetch-modes>`.
 
-This has a similar purpose to ``select_related``, in that both are designed to
-stop the deluge of database queries that is caused by accessing related
-objects, but the strategy is quite different.
+This method has a similar purpose to :meth:`select_related`, in that both are
+designed to eagerly fetch related objects. However, they work in different
+ways.
 
 ``select_related`` works by creating an SQL join and including the fields of
 the related object in the ``SELECT`` statement. For this reason,
diff --git a/docs/releases/6.1.txt b/docs/releases/6.1.txt
index 5e852785d9..80470dbcd6 100644
--- a/docs/releases/6.1.txt
+++ b/docs/releases/6.1.txt
@@ -26,6 +26,51 @@ only officially support, the latest release of each series.
 What's new in Django 6.1
 ========================
 
+Model field fetch modes
+-----------------------
+
+The on-demand fetching behavior of model fields is now configurable with
+:doc:`fetch modes </topics/db/fetch-modes>`. These modes allow you to control
+how Django fetches data from the database when an unfetched field is accessed.
+
+Django provides three fetch modes:
+
+1. ``FETCH_ONE``, the default, fetches the missing field for the current
+   instance only. This mode represents Django's existing behavior.
+
+2. ``FETCH_PEERS`` fetches a missing field for all instances that came from
+   the same :class:`~django.db.models.query.QuerySet`.
+
+   This mode works like an on-demand ``prefetch_related()``. It can reduce most
+   cases of the "N+1 queries problem" to two queries without any work to
+   maintain a list of fields to prefetch.
+
+3. ``RAISE`` raises a :exc:`~django.core.exceptions.FieldFetchBlocked`
+   exception.
+
+   This mode can prevent unintentional queries in performance-critical
+   sections of code.
+
+Use the new method :meth:`.QuerySet.fetch_mode` to set the fetch mode for model
+instances fetched by the ``QuerySet``:
+
+.. code-block:: python
+
+    from django.db import models
+
+    books = Book.objects.fetch_mode(models.FETCH_PEERS)
+    for book in books:
+        print(book.author.name)
+
+Despite the loop accessing the ``author`` foreign key on each instance, the
+``FETCH_PEERS`` fetch mode will make the above example perform only two
+queries:
+
+1. Fetch all books.
+2. Fetch associated authors.
+
+See :doc:`fetch modes </topics/db/fetch-modes>` for more details.
+
 Minor features
 --------------
 
diff --git a/docs/spelling_wordlist b/docs/spelling_wordlist
index 2898f85d5b..b35c94fc10 100644
--- a/docs/spelling_wordlist
+++ b/docs/spelling_wordlist
@@ -535,6 +535,7 @@ unencrypted
 unescape
 unescaped
 unevaluated
+unfetched
 unglamorous
 ungrouped
 unhandled
diff --git a/docs/topics/db/fetch-modes.txt b/docs/topics/db/fetch-modes.txt
new file mode 100644
index 0000000000..e76bb28a59
--- /dev/null
+++ b/docs/topics/db/fetch-modes.txt
@@ -0,0 +1,138 @@
+===========
+Fetch modes
+===========
+
+.. versionadded:: 6.1
+
+.. module:: django.db.models.fetch_modes
+
+.. currentmodule:: django.db.models
+
+When accessing model fields that were not loaded as part of the original query,
+Django will fetch that field's data from the database. You can customize the
+behavior of this fetching with a **fetch mode**, making it more efficient or
+even blocking it.
+
+Use :meth:`.QuerySet.fetch_mode` to set the fetch mode for model
+instances fetched by a ``QuerySet``:
+
+.. code-block:: python
+
+    from django.db import models
+
+    books = Book.objects.fetch_mode(models.FETCH_PEERS)
+
+Fetch modes apply to:
+
+* :class:`~django.db.models.ForeignKey` fields
+* :class:`~django.db.models.OneToOneField` fields and their reverse accessors
+* Fields deferred with :meth:`.QuerySet.defer` or :meth:`.QuerySet.only`
+* :ref:`generic-relations`
+
+Available modes
+===============
+
+.. admonition:: Referencing fetch modes
+
+    Fetch modes are defined in ``django.db.models.fetch_modes``, but for
+    convenience they're imported into :mod:`django.db.models`. The standard
+    convention is to use ``from django.db import models`` and refer to the
+    fetch modes as ``models.<mode>``.
+
+Django provides three fetch modes. We'll explain them below using these models:
+
+.. code-block:: python
+
+    from django.db import models
+
+
+    class Author(models.Model): ...
+
+
+    class Book(models.Model):
+        author = models.ForeignKey(Author, on_delete=models.CASCADE)
+        ...
+
+…and this loop:
+
+.. code-block:: python
+
+    for book in books:
+        print(book.author.name)
+
+…where ``books`` is a ``QuerySet`` of ``Book`` instances using some fetch mode.
+
+.. attribute:: FETCH_ONE
+
+Fetches the missing field for the current instance only. This is the default
+mode.
+
+Using ``FETCH_ONE`` for the above example would use:
+
+* 1 query to fetch ``books``
+* N queries, where N is the number of books, to fetch the missing ``author``
+  field
+
+…for a total of 1+N queries. This query pattern is known as the "N+1 queries
+problem" because it often leads to performance issues when N is large.
+
+.. attribute:: FETCH_PEERS
+
+Fetches the missing field for the current instance and its "peers"—instances
+that came from the same initial ``QuerySet``. The behavior of this mode is
+based on the assumption that if you need a field for one instance, you probably
+need it for all instances in the same batch, since you'll likely process them
+all identically.
+
+Using ``FETCH_PEERS`` for the above example would use:
+
+* 1 query to fetch ``books``
+* 1 query to fetch all missing ``author`` fields for the batch of books
+
+…for a total of 2 queries. The batch query makes this mode a lot more efficient
+than ``FETCH_ONE`` and is similar to an on-demand call to
+:meth:`.QuerySet.prefetch_related` or
+:func:`~django.db.models.prefetch_related_objects`. Using ``FETCH_PEERS`` can
+reduce most cases of the "N+1 queries problem" to two queries without
+much effort.
+
+The "peer" instances are tracked in a list of weak references, to avoid
+memory leaks where some peer instances are discarded.
+
+.. attribute:: RAISE
+
+Raises a :exc:`~django.core.exceptions.FieldFetchBlocked` exception.
+
+Using ``RAISE`` for the above example would raise an exception at the access of
+``book.author`` access, like:
+
+.. code-block:: python
+
+    FieldFetchBlocked("Fetching of Primary.value blocked.")
+
+This mode can prevent unintentional queries in performance-critical
+sections of code.
+
+.. _fetch-modes-custom-manager:
+
+Make a fetch mode the default for a model class
+===============================================
+
+Set the default fetch mode for a model class with a
+:ref:`custom manager <custom-managers>` that overrides ``get_queryset()``:
+
+.. code-block:: python
+
+    from django.db import models
+
+
+    class BookManager(models.Manager):
+        def get_queryset(self):
+            return super().get_queryset().fetch_mode(models.FETCH_PEERS)
+
+
+    class Book(models.Model):
+        title = models.TextField()
+        author = models.ForeignKey("Author", on_delete=models.CASCADE)
+
+        objects = BookManager()
diff --git a/docs/topics/db/index.txt b/docs/topics/db/index.txt
index 67a71fd820..6caf9f15e9 100644
--- a/docs/topics/db/index.txt
+++ b/docs/topics/db/index.txt
@@ -13,6 +13,7 @@ Generally, each model maps to a single database table.
 
    models
    queries
+   fetch-modes
    aggregation
    search
    managers
diff --git a/docs/topics/db/optimization.txt b/docs/topics/db/optimization.txt
index bb70efa362..3be0bd2cb5 100644
--- a/docs/topics/db/optimization.txt
+++ b/docs/topics/db/optimization.txt
@@ -196,28 +196,46 @@ thousands of records are returned. The penalty will be compounded if the
 database lives on a separate server, where network overhead and latency also
 play a factor.
 
-Retrieve everything at once if you know you will need it
-========================================================
+Retrieve related objects efficiently
+====================================
+
+Generally, accessing the database multiple times to retrieve different parts
+of a single "set" of data is less efficient than retrieving it all in one
+query. This is particularly important if you have a query that is executed in a
+loop, and could therefore end up doing many database queries, when only one
+is needed. Below are some techniques to combine queries for efficiency.
+
+Use the ``FETCH_PEERS`` fetch mode
+----------------------------------
 
-Hitting the database multiple times for different parts of a single 'set' of
-data that you will need all parts of is, in general, less efficient than
-retrieving it all in one query. This is particularly important if you have a
-query that is executed in a loop, and could therefore end up doing many
-database queries, when only one was needed. So:
+Use the :attr:`~django.db.models.FETCH_PEERS` fetch mode to make on-demand
+field access more efficient with bulk-fetching. Enable all it for all usage of
+your models :ref:`with a custom manager <fetch-modes-custom-manager>`.
+
+Using this fetch mode is easier than declaring fields to fetch with
+:meth:`~django.db.models.query.QuerySet.select_related` or
+:meth:`~django.db.models.query.QuerySet.prefetch_related`, especially when it's
+hard to predict which fields will be accessed.
 
 Use ``QuerySet.select_related()`` and ``prefetch_related()``
 ------------------------------------------------------------
 
-Understand :meth:`~django.db.models.query.QuerySet.select_related` and
-:meth:`~django.db.models.query.QuerySet.prefetch_related` thoroughly, and use
-them:
+When the :attr:`~django.db.models.FETCH_PEERS` fetch mode is not appropriate or
+efficient enough, use :meth:`~django.db.models.query.QuerySet.select_related`
+and :meth:`~django.db.models.query.QuerySet.prefetch_related`. Understand their
+documentation thoroughly and apply them where needed.
+
+It may be useful to apply these methods in :doc:`managers and default managers
+</topics/db/managers>`. Be aware when your manager is and is not used;
+sometimes this is tricky so don't make assumptions.
 
-* in :doc:`managers and default managers </topics/db/managers>` where
-  appropriate. Be aware when your manager is and is not used; sometimes this is
-  tricky so don't make assumptions.
+Use ``prefetch_related_objects()``
+----------------------------------
 
-* in view code or other layers, possibly making use of
-  :func:`~django.db.models.prefetch_related_objects` where needed.
+Where :attr:`~django.db.models.query.QuerySet.prefetch_related` would be useful
+after the queryset has been evaluated, use
+:func:`~django.db.models.prefetch_related_objects` to execute an extra
+prefetch.
 
 Don't retrieve things you don't need
 ====================================
diff --git a/docs/topics/db/queries.txt b/docs/topics/db/queries.txt
index d24505b039..ed1d3ea9ed 100644
--- a/docs/topics/db/queries.txt
+++ b/docs/topics/db/queries.txt
@@ -1702,6 +1702,12 @@ the link from the related model to the model that defines the relationship.
 For example, a ``Blog`` object ``b`` has a manager that returns all related
 ``Entry`` objects in the ``entry_set`` attribute: ``b.entry_set.all()``.
 
+These accessors may be prefetched by the ``QuerySet`` methods
+:meth:`~django.db.models.query.QuerySet.select_related` or
+:meth:`~django.db.models.query.QuerySet.prefetch_related`. If not prefetched,
+access will trigger an on-demand fetch through the model's
+:doc:`fetch mode </topics/db/fetch-modes>`.
+
 All examples in this section use the sample ``Blog``, ``Author`` and ``Entry``
 models defined at the top of this page.