Fixed #35859 -- Added background Tasks framework interface.

This work implements what was defined in DEP 14
(https://github.com/django/deps/blob/main/accepted/0014-background-workers.rst).

Thanks to Raphael Gaschignard, Eric Holscher, Ran Benita, Sarah Boyce,
Jacob Walls, and Natalia Bidart for the reviews.

5 files changed, 586 insertions, 0 deletions

diff --git a/docs/ref/checks.txt b/docs/ref/checks.txt
index e1ea5bc753..138db8708e 100644
--- a/docs/ref/checks.txt
+++ b/docs/ref/checks.txt
@@ -597,6 +597,14 @@ Signals
   a lazy reference to the sender ``<app label>.<model>``, but app
   ``<app label>`` isn't installed or doesn't provide model ``<model>``.
 
+Tasks
+-----
+
+* **tasks.E001**: ``ENQUEUE_ON_COMMIT`` cannot be used when no databases are
+  configured.
+* **tasks.E002**: ``ENQUEUE_ON_COMMIT`` cannot be used on a database which
+  doesn't support transactions.
+
 Templates
 ---------
 
diff --git a/docs/ref/index.txt b/docs/ref/index.txt
index 3741b82aad..af32b131cb 100644
--- a/docs/ref/index.txt
+++ b/docs/ref/index.txt
@@ -26,6 +26,7 @@ API Reference
    schema-editor
    settings
    signals
+   tasks
    templates/index
    template-response
    unicode
diff --git a/docs/ref/settings.txt b/docs/ref/settings.txt
index c16547e72a..54957a726a 100644
--- a/docs/ref/settings.txt
+++ b/docs/ref/settings.txt
@@ -2766,6 +2766,82 @@ backend definition in :setting:`STORAGES`.
     Defining this setting overrides the default value and is *not* merged with
     it.
 
+.. setting:: TASKS
+
+``TASKS``
+---------
+
+.. versionadded:: 6.0
+
+Default::
+
+    {
+        "default": {
+            "BACKEND": "django.tasks.backends.immediate.ImmediateBackend",
+        }
+    }
+
+A dictionary containing the settings for all Task backends to be used with
+Django. It is a nested dictionary whose contents maps backend aliases to a
+dictionary containing the options for each backend.
+
+The :setting:`TASKS` setting must configure a ``default`` backend; any number
+of additional backends may also be specified. Depending on which backend is
+used, other options may be required. The following options are available as
+standard.
+
+.. setting:: TASKS-BACKEND
+
+``BACKEND``
+~~~~~~~~~~~
+
+Default: ``''`` (Empty string)
+
+The Tasks backend to use. The built-in backends are:
+
+* ``'django.tasks.backends.dummy.DummyBackend'``
+* ``'django.tasks.backends.immediate.ImmediateBackend'``
+
+You can use a backend that doesn't ship with Django by setting
+:setting:`BACKEND <TASKS-BACKEND>` to a fully-qualified path of a backend
+class (i.e. ``mypackage.backends.whatever.WhateverBackend``).
+
+.. setting:: TASKS-ENQUEUE_ON_COMMIT
+
+``ENQUEUE_ON_COMMIT``
+~~~~~~~~~~~~~~~~~~~~~
+
+Default: ``True``
+
+Whether to enqueue a Task only after the current transaction, if any, commits
+successfully, instead of enqueueing immediately.
+
+This can also be configured on a per-Task basis.
+
+See :ref:`Task transactions <task-transactions>` for more information.
+
+.. setting:: TASKS-QUEUES
+
+``QUEUES``
+~~~~~~~~~~
+
+Default: ``["default"]``
+
+Specify the queue names supported by the backend. This can be used to ensure
+Tasks aren't enqueued to queues which do not exist.
+
+To disable queue name validation, set to an empty list (``[]``).
+
+.. setting:: TASKS-OPTIONS
+
+``OPTIONS``
+~~~~~~~~~~~
+
+Default: ``{}``
+
+Extra parameters to pass to the Task backend. Available parameters vary
+depending on the Task backend.
+
 .. setting:: TEMPLATES
 
 ``TEMPLATES``
diff --git a/docs/ref/signals.txt b/docs/ref/signals.txt
index 6dc5122f96..82b92e12c2 100644
--- a/docs/ref/signals.txt
+++ b/docs/ref/signals.txt
@@ -703,3 +703,60 @@ Arguments sent with this signal:
     The database connection that was opened. This can be used in a
     multiple-database configuration to differentiate connection signals
     from different databases.
+
+Tasks signals
+=============
+
+.. versionadded:: 6.0
+
+Signals sent by the :doc:`tasks </ref/tasks>` framework.
+
+``task_enqueued``
+-----------------
+
+.. data:: django.tasks.signals.task_enqueued
+   :module:
+
+Sent once a Task has been enqueued. If
+:attr:`django.tasks.Task.enqueue_on_commit` is set, the signal is only sent
+once the transaction commits successfully.
+
+Arguments sent with this signal:
+
+``sender``
+    The backend class which the Task was enqueued on to.
+
+``task_result``
+    The enqueued :class:`TaskResult <django.tasks.TaskResult>`.
+
+``task_started``
+----------------
+
+.. data:: django.tasks.signals.task_started
+   :module:
+
+Sent when a Task has started executing.
+
+Arguments sent with this signal:
+
+``sender``
+    The backend class which the Task was enqueued on to.
+
+``task_result``
+    The started :class:`TaskResult <django.tasks.TaskResult>`.
+
+``task_finished``
+-----------------
+
+.. data:: django.tasks.signals.task_finished
+   :module:
+
+Sent once a Task has finished executing, successfully or otherwise.
+
+Arguments sent with this signal:
+
+``sender``
+    The backend class which the Task was enqueued on to.
+
+``task_result``
+    The finished :class:`TaskResult <django.tasks.TaskResult>`.
diff --git a/docs/ref/tasks.txt b/docs/ref/tasks.txt
new file mode 100644
index 0000000000..3134243d40
--- /dev/null
+++ b/docs/ref/tasks.txt
@@ -0,0 +1,444 @@
+=====
+Tasks
+=====
+
+.. versionadded:: 6.0
+
+.. module:: django.tasks
+   :synopsis: Django's built-in background Task system.
+
+Task definition
+===============
+
+The ``task`` decorator
+----------------------
+
+.. function:: task(*, priority=0, queue_name="default", backend="default", enqueue_on_commit=None, takes_context=False)
+
+    The ``@task`` decorator defines a :class:`Task` instance. This has the
+    following optional arguments:
+
+    * ``priority``: Sets the :attr:`~Task.priority` of the ``Task``. Defaults
+      to 0.
+    * ``queue_name``: Sets the :attr:`~Task.queue_name` of the ``Task``.
+      Defaults to ``"default"``.
+    * ``backend``: Sets the :attr:`~Task.backend` of the ``Task``. Defaults to
+      ``"default"``.
+    * ``enqueue_on_commit``: Sets :attr:`~Task.enqueue_on_commit` for the
+      ``Task``. Defaults to ``None``.
+    * ``takes_context``: Controls whether the ``Task`` function accepts a
+      :class:`TaskContext`. Defaults to ``False``. See :ref:`Task context
+      <task-context>` for details.
+
+    If the defined ``Task`` is not valid according to the backend,
+    :exc:`~django.tasks.exceptions.InvalidTask` is raised.
+
+    See :ref:`defining tasks <defining-tasks>` for usage examples.
+
+``Task``
+--------
+
+.. class:: Task
+
+    Represents a Task to be run in the background. Tasks should be defined
+    using the :func:`task` decorator.
+
+    Attributes of ``Task`` cannot be modified. See :ref:`modifying Tasks
+    <modifying-tasks>` for details.
+
+    .. attribute:: Task.priority
+
+        The priority of the ``Task``. Priorities must be between -100 and 100,
+        where larger numbers are higher priority, and will be run sooner.
+
+        The backend must have :attr:`.supports_priority` set to ``True`` to use
+        this feature.
+
+    .. attribute:: Task.backend
+
+        The alias of the backend the ``Task`` should be enqueued to. This must
+        match a backend defined in :setting:`BACKEND <TASKS-BACKEND>`.
+
+    .. attribute:: Task.queue_name
+
+        The name of the queue the ``Task`` will be enqueued on to. Defaults to
+        ``"default"``. This must match a queue defined in
+        :setting:`QUEUES <TASKS-QUEUES>`, unless
+        :setting:`QUEUES <TASKS-QUEUES>` is set to ``[]``.
+
+    .. attribute:: Task.run_after
+
+        The earliest time the ``Task`` will be executed. This can be a
+        :class:`timedelta <datetime.timedelta>`, which is used relative to the
+        current time, a timezone-aware :class:`datetime <datetime.datetime>`,
+        or ``None`` if not constrained. Defaults to ``None``.
+
+        The backend must have :attr:`.supports_defer` set to ``True`` to use
+        this feature. Otherwise,
+        :exc:`~django.tasks.exceptions.InvalidTask` is raised.
+
+    .. attribute:: Task.enqueue_on_commit
+
+        Whether the ``Task`` should be enqueued when the transaction commits
+        successfully, or immediately. Defaults to :setting:`ENQUEUE_ON_COMMIT
+        <TASKS-ENQUEUE_ON_COMMIT>` for the backend.
+
+        See :ref:`Task transactions <task-transactions>` for more information.
+
+    .. attribute:: Task.name
+
+        The name of the function decorated with :func:`task`. This name is not
+        necessarily unique.
+
+    .. method:: Task.using(*, priority=None, backend=None, queue_name=None, run_after=None)
+
+        Creates a new ``Task`` with modified defaults. The existing ``Task`` is
+        left unchanged.
+
+        ``using`` allows modifying the following attributes:
+
+        * :attr:`priority <Task.priority>`
+        * :attr:`backend <Task.backend>`
+        * :attr:`queue_name <Task.queue_name>`
+        * :attr:`run_after <Task.run_after>`
+
+        See :ref:`modifying Tasks <modifying-tasks>` for usage examples.
+
+    .. method:: Task.enqueue(*args, **kwargs)
+
+        Enqueues the ``Task`` to the ``Task`` backend for later execution.
+
+        Arguments are passed to the ``Task``'s function after a round-trip
+        through a :func:`json.dumps`/:func:`json.loads` cycle. Hence, all
+        arguments must be JSON-serializable and preserve their type after the
+        round-trip.
+
+        If the ``Task`` is not valid according to the backend,
+        :exc:`~django.tasks.exceptions.InvalidTask` is raised.
+
+        See :ref:`enqueueing Tasks <enqueueing-tasks>` for usage examples.
+
+    .. method:: Task.aenqueue(*args, **kwargs)
+
+        The ``async`` variant of :meth:`enqueue <Task.enqueue>`.
+
+    .. method:: Task.get_result(result_id)
+
+        Retrieves a result by its id.
+
+        If the result does not exist, :exc:`TaskResultDoesNotExist
+        <django.tasks.exceptions.TaskResultDoesNotExist>` is raised. If the
+        result is not the same type as the current Task,
+        :exc:`TaskResultMismatch <django.tasks.exceptions.TaskResultMismatch>`
+        is raised. If the backend does not support ``get_result()``,
+        :exc:`NotImplementedError` is raised.
+
+    .. method:: Task.aget_result(*args, **kwargs)
+
+        The ``async`` variant of :meth:`get_result <Task.get_result>`.
+
+Task context
+============
+
+.. class:: TaskContext
+
+    Contains context for the running :class:`Task`. Context only passed to a
+    ``Task`` if it was defined with ``takes_context=True``.
+
+    Attributes of ``TaskContext`` cannot be modified.
+
+    .. attribute:: TaskContext.task_result
+
+        The :class:`TaskResult` currently being run.
+
+    .. attribute:: TaskContext.attempt
+
+        The number of the current execution attempts for this Task, starting at
+        1.
+
+Task results
+============
+
+.. class:: TaskResultStatus
+
+    An Enum representing the status of a :class:`TaskResult`.
+
+    .. attribute:: TaskResultStatus.READY
+
+        The :class:`Task` has just been enqueued, or is ready to be executed
+        again.
+
+    .. attribute:: TaskResultStatus.RUNNING
+
+        The :class:`Task` is currently being executed.
+
+    .. attribute:: TaskResultStatus.FAILED
+
+        The :class:`Task` raised an exception during execution, or was unable
+        to start.
+
+    .. attribute:: TaskResultStatus.SUCCESSFUL
+
+        The :class:`Task` has finished executing successfully.
+
+.. class:: TaskResult
+
+    The ``TaskResult`` stores the information about a specific execution of a
+    :class:`Task`.
+
+    Attributes of ``TaskResult`` cannot be modified.
+
+    .. attribute:: TaskResult.task
+
+        The :class:`Task` the result was enqueued for.
+
+    .. attribute:: TaskResult.id
+
+        A unique identifier for the result, which can be passed to
+        :meth:`Task.get_result`.
+
+        The format of the id will depend on the backend being used. Task result
+        ids are always strings less than 64 characters.
+
+        See :ref:`Task results <task-results>` for more details.
+
+    .. attribute:: TaskResult.status
+
+        The :class:`status <TaskResultStatus>` of the result.
+
+    .. attribute:: TaskResult.enqueued_at
+
+        The time when the ``Task`` was enqueued.
+
+        If :attr:`Task.enqueue_on_commit` was set, this is the time the
+        transaction committed.
+
+    .. attribute:: TaskResult.started_at
+
+        The time when the ``Task`` began execution, on its first attempt.
+
+    .. attribute:: TaskResult.last_attempted_at
+
+        The time when the most recent ``Task`` run began execution.
+
+    .. attribute:: TaskResult.finished_at
+
+        The time when the ``Task`` finished execution, whether it failed or
+        succeeded.
+
+    .. attribute:: TaskResult.backend
+
+        The backend the result is from.
+
+    .. attribute:: TaskResult.errors
+
+        A list of :class:`TaskError` instances for the errors raised as part of
+        each execution of the Task.
+
+    .. attribute:: TaskResult.return_value
+
+        The return value from the ``Task`` function.
+
+        If the ``Task`` did not finish successfully, :exc:`ValueError` is
+        raised.
+
+        See :ref:`return values <task-return-values>` for usage examples.
+
+    .. method:: TaskResult.refresh
+
+        Refresh the result's attributes from the queue store.
+
+    .. method:: TaskResult.arefresh
+
+        The ``async`` variant of :meth:`TaskResult.refresh`.
+
+    .. attribute:: TaskResult.is_finished
+
+        Whether the ``Task`` has finished (successfully or not).
+
+    .. attribute:: TaskResult.attempts
+
+        The number of times the Task has been run.
+
+        If the task is currently running, it does not count as an attempt.
+
+    .. attribute:: TaskResult.worker_ids
+
+        The ids of the workers which have executed the Task.
+
+
+Task errors
+-----------
+
+.. class:: TaskError
+
+    Contains information about the error raised during the execution of a
+    ``Task``.
+
+    .. attribute:: TaskError.traceback
+
+        The traceback (as a string) from the raised exception when the ``Task``
+        failed.
+
+    .. attribute:: TaskError.exception_class
+
+        The exception class raised when executing the ``Task``.
+
+Backends
+========
+
+Base backend
+------------
+
+.. module:: django.tasks.backends.base
+
+.. class:: BaseTaskBackend
+
+    ``BaseTaskBackend`` is the parent class for all Task backends.
+
+    .. attribute:: BaseTaskBackend.options
+
+        A dictionary of extra parameters for the Task backend. These are
+        provided using the :setting:`OPTIONS <TASKS-OPTIONS>` setting.
+
+    .. method:: BaseTaskBackend.enqueue(task, args, kwargs)
+
+        Task backends which subclass ``BaseTaskBackend`` should implement this
+        method as a minimum.
+
+        When implemented, ``enqueue()`` enqueues the ``task``, a :class:`.Task`
+        instance, for later execution. ``args`` are the positional arguments
+        and ``kwargs`` are the keyword arguments to be passed to the ``task``.
+        Returns a :class:`~django.tasks.TaskResult`.
+
+    .. method:: BaseTaskBackend.aenqueue(task, args, kwargs)
+
+        The ``async`` variant of :meth:`BaseTaskBackend.enqueue`.
+
+    .. method:: BaseTaskBackend.get_result(result_id)
+
+        Retrieve a result by its id. If the result does not exist,
+        :exc:`TaskResultDoesNotExist
+        <django.tasks.exceptions.TaskResultDoesNotExist>` is raised.
+
+        If the backend does not support ``get_result()``,
+        :exc:`NotImplementedError` is raised.
+
+    .. method:: BaseTaskBackend.aget_result(result_id)
+
+        The ``async`` variant of :meth:`BaseTaskBackend.get_result`.
+
+    .. method:: BaseTaskBackend.validate_task(task)
+
+        Validates whether the provided ``Task`` is able to be enqueued using
+        the backend. If the Task is not valid,
+        :exc:`InvalidTask <django.tasks.exceptions.InvalidTask>`
+        is raised.
+
+Feature flags
+~~~~~~~~~~~~~
+
+Some backends may not support all features Django provides. It's possible to
+identify the supported functionality of a backend, and potentially change
+behavior accordingly.
+
+.. attribute:: BaseTaskBackend.supports_defer
+
+    Whether the backend supports enqueueing Tasks to be executed after a
+    specific time using the :attr:`~django.tasks.Task.run_after` attribute.
+
+.. attribute:: BaseTaskBackend.supports_async_task
+
+    Whether the backend supports enqueueing async functions (coroutines).
+
+.. attribute:: BaseTaskBackend.supports_get_result
+
+    Whether the backend supports retrieving ``Task`` results from another
+    thread after they have been enqueued.
+
+.. attribute:: BaseTaskBackend.supports_priority
+
+    Whether the backend supports executing Tasks as ordered by their
+    :attr:`~django.tasks.Task.priority`.
+
+The below table notes which of the :ref:`built-in backends
+<task-available-backends>` support which features:
+
+============================ ======================= ===========================
+Feature                      :class:`.DummyBackend`  :class:`.ImmediateBackend`
+============================ ======================= ===========================
+:attr:`.supports_defer`      Yes                     No
+:attr:`.supports_async_task` Yes                     Yes
+:attr:`.supports_get_result` No                      No [#fnimmediateresult]_
+:attr:`.supports_priority`   Yes [#fndummypriority]_ Yes [#fnimmediatepriority]_
+============================ ======================= ===========================
+
+.. _task-available-backends:
+
+Available backends
+------------------
+
+Immediate backend
+~~~~~~~~~~~~~~~~~
+
+.. module:: django.tasks.backends.immediate
+
+.. class:: ImmediateBackend
+
+    The :ref:`immediate backend <immediate-task-backend>` executes Tasks
+    immediately, rather than in the background.
+
+Dummy backend
+~~~~~~~~~~~~~
+
+.. module:: django.tasks.backends.dummy
+
+.. class:: DummyBackend
+
+    The :ref:`dummy backend <dummy-task-backend>` does not execute enqueued
+    Tasks. Instead, it stores task results for later inspection.
+
+    .. attribute:: DummyBackend.results
+
+        A list of results for the enqueued Tasks, in the order they were
+        enqueued.
+
+    .. method:: DummyBackend.clear
+
+        Clears the list of stored results.
+
+Exceptions
+==========
+
+.. module:: django.tasks.exceptions
+
+.. exception:: InvalidTask
+
+    Raised when the :class:`.Task` attempting to be enqueued
+    is invalid.
+
+.. exception:: InvalidTaskBackend
+
+    Raised when the requested :class:`.BaseTaskBackend` is invalid.
+
+.. exception:: TaskResultDoesNotExist
+
+    Raised by :meth:`~django.tasks.backends.base.BaseTaskBackend.get_result`
+    when the provided ``result_id`` does not exist.
+
+.. exception:: TaskResultMismatch
+
+    Raised by :meth:`~django.tasks.Task.get_result` when the provided
+    ``result_id`` is for a different Task than the current Task.
+
+.. rubric:: Footnotes
+.. [#fnimmediateresult] The :class:`.ImmediateBackend` doesn't officially
+    support ``get_result()``, despite implementing the API, since the result
+    cannot be retrieved from a different thread.
+.. [#fndummypriority] The :class:`.DummyBackend` has ``supports_priority=True``
+    so that it can be used as a drop-in replacement in tests. Since this
+    backend never executes Tasks, the ``priority`` value has no effect.
+.. [#fnimmediatepriority] The :class:`.ImmediateBackend` has
+    ``supports_priority=True`` so that it can be used as a drop-in replacement
+    in tests. Because Tasks run as soon as they are scheduled, the ``priority``
+    value has no effect.