Refs #32880 -- Created a new logging how-to document.

Moved how-to material from topic document into a new document, and
added new material. Introduced minor improvements to logging
reference document.

4 files changed, 364 insertions, 103 deletions

diff --git a/docs/howto/index.txt b/docs/howto/index.txt
index ffe4c5519e..75304f6827 100644
--- a/docs/howto/index.txt
+++ b/docs/howto/index.txt
@@ -22,6 +22,7 @@ you quickly accomplish common tasks.
    error-reporting
    initial-data
    legacy-databases
+   logging
    outputting-csv
    outputting-pdf
    overriding-templates
diff --git a/docs/howto/logging.txt b/docs/howto/logging.txt
new file mode 100644
index 0000000000..f2bd0b8485
--- /dev/null
+++ b/docs/howto/logging.txt
@@ -0,0 +1,343 @@
+.. _logging-how-to:
+
+================================
+How to configure and use logging
+================================
+
+.. seealso::
+
+    * :ref:`Django logging reference <logging-ref>`
+    * :ref:`Django logging overview <logging-explanation>`
+
+Django provides a working :ref:`default logging configuration
+<default-logging-configuration>` that is readily extended.
+
+Make a basic logging call
+=========================
+
+To send a log message from within your code, you place a logging call into it.
+
+.. admonition:: Don't be tempted to use logging calls in ``settings.py``.
+
+    The way that Django logging is configured as part of the ``setup()``
+    function means that logging calls placed in ``settings.py`` may not work as
+    expected, because *logging will not be set up at that point*. To explore
+    logging, use a view function as suggested in the example below.
+
+First, import the Python logging library, and then obtain a logger instance
+with :py:func:`logging.getLogger`. Provide the ``getLogger()`` method with a
+name to identify it and the records it emits. A good option is to use
+``__name__`` (see :ref:`naming-loggers` below for more on this) which will
+provide the name of the current Python module as a dotted path::
+
+    import logging
+
+    logger = logging.getLogger(__name__)
+
+It's a good convention to perform this declaration at module level.
+
+And then in a function, for example in a view, send a record to the logger::
+
+    def some_view(request):
+        ...
+        if some_risky_state:
+            logger.warning('Platform is running at risk')
+
+When this code is executed, a :py:class:`~logging.LogRecord` containing that
+message will be sent to the logger. If you're using Django's default logging
+configuration, the message will appear in the console.
+
+The ``WARNING`` level used in the example above is one of several
+:ref:`logging severity levels <topic-logging-parts-loggers>`: ``DEBUG``,
+``INFO``, ``WARNING``, ``ERROR``, ``CRITICAL``. So, another example might be::
+
+    logger.critical('Payment system is not responding')
+
+.. important::
+
+    Records with a level lower than ``WARNING`` will not appear in the console
+    by default. Changing this behavior requires additional configuration.
+
+Customize logging configuration
+===============================
+
+Although Django's logging configuration works out of the box, you can control
+exactly how your logs are sent to various destinations - to log files, external
+services, email and so on - with some additional configuration.
+
+You can configure:
+
+* logger mappings, to determine which records are sent to which handlers
+* handlers, to determine what they do with the records they receive
+* filters, to provide additional control over the transfer of records, and
+  even modify records in-place
+* formatters, to convert :class:`~logging.LogRecord` objects to a string or
+  other form for consumption by human beings or another system
+
+There are various ways of configuring logging. In Django, the
+:setting:`LOGGING` setting is most commonly used. The setting uses the
+:ref:`dictConfig format <logging-config-dictschema>`, and extends the
+:ref:`default logging configuration <default-logging-definition>`.
+
+See :ref:`configuring-logging` for an explanation of how your custom settings
+are merged with Django's defaults.
+
+See the :mod:`Python logging documentation <python:logging.config>` for
+details of other ways of configuring logging. For the sake of simplicity, this
+documentation will only consider configuration via the ``LOGGING`` setting.
+
+.. _basic-logger-configuration:
+
+Basic logging configuration
+---------------------------
+
+When configuring logging, it makes sense to
+
+Create a ``LOGGING`` dictionary
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In your ``settings.py``::
+
+    LOGGING = {
+        'version': 1,                       # the dictConfig format version
+        'disable_existing_loggers': False,  # retain the default loggers
+    }
+
+It nearly always makes sense to retain and extend the default logging
+configuration by setting ``disable_existing_loggers`` to ``False``.
+
+Configure a handler
+~~~~~~~~~~~~~~~~~~~
+
+This example configures a single handler named ``file``, that uses Python's
+:class:`~logging.FileHandler` to save logs of level ``DEBUG`` and higher to the
+file ``general.log`` (at the project root):
+
+.. code-block:: python
+    :emphasize-lines: 3-8
+
+    LOGGING = {
+        [...]
+        'handlers': {
+            'file': {
+                'class': 'logging.FileHandler',
+                'filename': 'general.log',
+            },
+        },
+    }
+
+Different handler classes take different configuration options. For more
+information on available handler classes, see the
+:class:`~django.utils.log.AdminEmailHandler` provided by Django and the various
+:py:mod:`handler classes <logging.handlers>` provided by Python.
+
+Logging levels can also be set on the handlers (by default, they accept log
+messages of all levels). Using the example above, adding:
+
+.. code-block:: python
+    :emphasize-lines: 4
+
+    {
+        'class': 'logging.FileHandler',
+        'filename': 'general.log',
+        'level': 'DEBUG',
+    }
+
+would define a handler configuration that only accepts records of level
+``DEBUG`` and higher.
+
+Configure a logger mapping
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To send records to this handler, configure a logger mapping to use it for
+example:
+
+.. code-block:: python
+    :emphasize-lines: 3-8
+
+    LOGGING = {
+        [...]
+        'loggers': {
+            '': {
+                'level': 'DEBUG',
+                'handlers': ['file'],
+            },
+        },
+    }
+
+The mapping's name determines which log records it will process. This
+configuration (``''``) is *unnamed*. That means that it will process records
+from *all* loggers (see :ref:`naming-loggers` below on how to use the mapping
+name to determine the loggers for which it will process records).
+
+It will forward messages of levels ``DEBUG`` and higher to the handler named
+``file``.
+
+Note that a logger can forward messages to multiple handlers, so the relation
+between loggers and handlers is many-to-many.
+
+If you execute::
+
+    logger.debug('Attempting to connect to API')
+
+in your code, you will find that message in the file ``general.log`` in the
+root of the project.
+
+Configure a formatter
+~~~~~~~~~~~~~~~~~~~~~
+
+By default, the final log output contains the message part of each :class:`log
+record <logging.LogRecord>`. Use a formatter if you want to include additional
+data. First name and define your formatters - this example defines
+formatters named ``verbose`` and ``simple``:
+
+.. code-block:: python
+    :emphasize-lines: 3-12
+
+    LOGGING = {
+        [...]
+        'formatters': {
+            'verbose': {
+                'format': '{name} {levelname} {asctime} {module} {process:d} {thread:d} {message}',
+                'style': '{',
+            },
+            'simple': {
+                'format': '{levelname} {message}',
+                'style': '{',
+            },
+        },
+    }
+
+The ``style`` keyword allows you to specify ``{`` for :meth:`str.format` or
+``$`` for :class:`string.Template` formatting; the default is ``$``.
+
+See :ref:`logrecord-attributes` for the :class:`~logging.LogRecord` attributes
+you can include.
+
+To apply a formatter to a handler, add a ``formatter`` entry to the handler's
+dictionary referring to the formatter by name, for example:
+
+.. code-block:: python
+    :emphasize-lines: 5
+
+    'handlers': {
+        'file': {
+            'class': 'logging.FileHandler',
+            'filename': 'general.log',
+            'formatter': 'verbose',
+        },
+    },
+
+.. _naming-loggers:
+
+Use logger namespacing
+~~~~~~~~~~~~~~~~~~~~~~
+
+The unnamed logging configuration ``''`` captures logs from any Python
+application. A named logging configuration will capture logs only from loggers
+with matching names.
+
+The namespace of a logger instance is defined using
+:py:func:`~logging.getLogger`. For example in ``views.py`` of ``my_app``::
+
+    logger = logging.getLogger(__name__)
+
+will create a logger in the ``my_app.views`` namespace. ``__name__`` allows you
+to organize log messages according to their provenance within your project's
+applications automatically. It also ensures that you will not experience name
+collisions.
+
+A logger mapping named ``my_app.views`` will capture records from this logger:
+
+.. code-block:: python
+    :emphasize-lines: 4
+
+    LOGGING = {
+        [...]
+        'loggers': {
+            'my_app.views': {
+                ...
+            },
+        },
+    }
+
+A logger mapping named ``my_app`` will be more permissive, capturing records
+from loggers anywhere within the ``my_app`` namespace (including
+``my_app.views``, ``my_app.utils``, and so on):
+
+.. code-block:: python
+    :emphasize-lines: 4
+
+    LOGGING = {
+        [...]
+        'loggers': {
+            'my_app': {
+                ...
+            },
+        },
+    }
+
+You can also define logger namespacing explicitly::
+
+    logger = logging.getLogger('project.payment')
+
+and set up logger mappings accordingly.
+
+.. _naming-loggers-hierarchy:
+
+Using logger hierarchies and propagation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Logger naming is *hierarchical*. ``my_app`` is the parent of ``my_app.views``,
+which is the parent of ``my_app.views.private``. Unless specified otherwise,
+logger mappings will propagate the records they process to their parents - a
+record from a logger in the ``my_app.views.private`` namespace will be handled
+by a mapping for both ``my_app`` and ``my_app.views``.
+
+To manage this behavior, set the propagation key on the mappings you define::
+
+    LOGGING = {
+        [...]
+        'loggers': {
+            'my_app': {
+                [...]
+            },
+            'my_app.views': {
+                [...]
+            },
+            'my_app.views.private': {
+                [...]
+                'propagate': False,
+            },
+        },
+    }
+
+``propagate`` defaults to ``True``. In this example, the logs from
+``my_app.views.private`` will not be handled by the parent, but logs from
+``my_app.views`` will.
+
+Configure responsive logging
+----------------------------
+
+Logging is most useful when it contains as much information as possible, but
+not information that you don't need - and how much you need depends upon what
+you're doing. When you're debugging, you need a level of information that would
+be excessive and unhelpful if you had to deal with it in production.
+
+You can configure logging to provide you with the level of detail you need,
+when you need it. Rather than manually change configuration to achieve this, a
+better way is to apply configuration automatically according to the
+environment.
+
+For example, you could set an environment variable ``DJANGO_LOG_LEVEL``
+appropriately in your development and staging environments, and make use of it
+in a logger mapping thus::
+
+    'level': os.getenv('DJANGO_LOG_LEVEL', 'WARNING')
+
+\- so that unless the environment specifies a lower log level, this
+configuration will only forward records of severity ``WARNING`` and above to
+its handler.
+
+Other options in the configuration (such as the ``level`` or ``formatter``
+option of handlers) can be similarly managed.
diff --git a/docs/ref/logging.txt b/docs/ref/logging.txt
index 8c682dfceb..7fdb7791f5 100644
--- a/docs/ref/logging.txt
+++ b/docs/ref/logging.txt
@@ -1,9 +1,14 @@
-.. _logging_ref:
+.. _logging-ref:
 
 =======
 Logging
 =======
 
+.. seealso::
+
+    * :ref:`logging-how-to`
+    * :ref:`Django logging overview <logging-explanation>`
+
 .. module:: django.utils.log
    :synopsis: Logging tools for Django applications
 
@@ -46,11 +51,17 @@ parents, up to the root ``django`` logger. The ``console`` and ``mail_admins``
 handlers are attached to the root logger to provide the behavior described
 above.
 
+Python's own defaults send records of level ``WARNING`` and higher
+to the console.
+
+.. _default-logging-definition:
+
 Default logging definition
 --------------------------
 
-The default configuration is available as ``django.utils.log.DEFAULT_LOGGING``
-and defined in :source:`django/utils/log.py`::
+Django's default logging configuration inherits Python's defaults. It's
+available as ``django.utils.log.DEFAULT_LOGGING`` and defined in
+:source:`django/utils/log.py`::
 
     {
         'version': 1,
@@ -246,8 +257,8 @@ as explained in :ref:`django-db-logger`.
 Handlers
 --------
 
-Django provides one log handler in addition to those provided by the
-Python logging module.
+Django provides one log handler in addition to :mod:`those provided by the
+Python logging module <python:logging.handlers>`.
 
 .. class:: AdminEmailHandler(include_html=False, email_backend=None, reporter_class=None)
 
diff --git a/docs/topics/logging.txt b/docs/topics/logging.txt
index b9a47ff324..1e06ce356a 100644
--- a/docs/topics/logging.txt
+++ b/docs/topics/logging.txt
@@ -1,10 +1,13 @@
+.. _logging-explanation:
+
 =======
 Logging
 =======
 
 .. seealso::
 
-    :ref:`Django logging reference <logging_ref>`.
+    * :ref:`logging-how-to`
+    * :ref:`Django logging reference <logging-ref>`
 
 Python programmers will often use ``print()`` in their code as a quick and
 convenient debugging tool. Using the logging framework is only a little more
@@ -163,10 +166,6 @@ and has access to the information, and so on.
 Configuring logging
 ===================
 
-It isn't enough to just put logging calls into your code. You also need to
-configure the loggers, handlers, filters, and formatters to ensure you can use
-the logging output.
-
 Python's logging library provides several techniques to configure
 logging, ranging from a programmatic interface to configuration files.
 By default, Django uses the :ref:`dictConfig format
@@ -457,96 +456,3 @@ Note that the default configuration process only calls
 configuring the logging in your settings file will load your logging config
 immediately. As such, your logging config must appear *after* any settings on
 which it depends.
-
-.. _logging-how-to:
-
-How to use logging
-==================
-
-Django provides a :ref:`default logging configuration
-<default-logging-configuration>`, so you don't need to provide any additional
-configuration in order to start using logging (it's the default configuration
-that for example generates the messages that appear in the console when using
-the :djadmin:`runserver`).
-
-Make a basic logging call
--------------------------
-
-To send a log message from within your code, you place a logging call into it.
-
-.. admonition:: Don't be tempted to use logging calls in ``settings.py``
-
-    The way that Django logging is configured as part of the ``setup()``
-    function means that logging calls placed in ``settings.py`` may not work as
-    expected, because *logging will not be set up at that point*. To explore
-    logging, use a view function as suggested in the example below.
-
-First, import the Python logging library, and then obtain a logger instance
-with :py:func:`logging.getLogger`. The ``getLogger()`` method must be provided
-with a name. A good option is to use ``__name__``, which will provide the name
-of the current Python module (see :ref:`naming-loggers` for use of explicit
-naming)::
-
-    import logging
-
-    logger = logging.getLogger(__name__)
-
-And then in a function, for example in a view, send a message to the logger::
-
-    def some_view(request):
-        ...
-        if some_risky_state:
-            logger.warning('Platform is running at risk')
-
-When this code is executed, that message will be sent to the logger (and if
-you're using Django's default logging configuration, it will appear in the
-console).
-
-The ``WARNING`` level used in the example above is one of several
-:ref:`logging severity levels <topic-logging-parts-loggers>`: ``DEBUG``,
-``INFO``, ``WARNING``, ``ERROR``, ``CRITICAL``. So, another example might be::
-
-    logger.critical('Payment system is not responding')
-
-The default logging configuration, which Django inherits from the Python
-logging module, prints all messages of level ``WARNING`` and higher to the
-console. Django's own defaults will *not* pass ``INFO`` or lower severity
-messages from applications other than Django itself to the console - that will
-need to be configured explicitly.
-
-.. _naming-loggers:
-
-Name logger instances
----------------------
-
-Every logger instance has a name. By convention, the logger name is usually
-``__name__``, the name of the Python module in which
-:func:`logging.getLogger()` is called. This allows you to filter and handle
-logging calls on a per-module basis. However, if you have some other way of
-organizing your logging messages, you can provide any dot-separated name to
-identify your logger::
-
-    # Get an instance of a specific named logger
-    logger = logging.getLogger('project.interesting.stuff')
-
-.. _naming-loggers-hierarchy:
-
-Logger hierarchy
-~~~~~~~~~~~~~~~~
-
-The dotted paths of logger names define a hierarchy. The
-``project.interesting`` logger is considered to be a parent of the
-``project.interesting.stuff`` logger; the ``project`` logger is a parent of the
-``project.interesting`` logger. (Note that this hierarchy does not need to
-reflect the actual Python module hierarchy.)
-
-Why is the hierarchy important? Well, because loggers can be set to
-*propagate* their logging calls to their parents. In this way, you can
-define a single set of handlers at the root of a logger tree, and
-capture all logging calls in the subtree of loggers. A logger defined
-in the ``project`` namespace will catch all logging messages issued on
-the ``project.interesting`` and ``project.interesting.stuff`` loggers.
-
-This propagation can be controlled on a per-logger basis. If
-you don't want a particular logger to propagate to its parents, you
-can turn off this behavior.