Fixed #15667 -- Added template-based widget rendering.

Thanks Carl Meyer and Tim Graham for contributing to the patch.

8 files changed, 299 insertions, 60 deletions

diff --git a/docs/internals/deprecation.txt b/docs/internals/deprecation.txt
index c1a1c41cd6..1510ca659b 100644
--- a/docs/internals/deprecation.txt
+++ b/docs/internals/deprecation.txt
@@ -51,6 +51,9 @@ details on these changes.
 * Support for regular expression groups with ``iLmsu#`` in ``url()`` will be
   removed.
 
+* Support for ``Widget.render()`` methods without the ``renderer`` argument
+  will be removed.
+
 .. _deprecation-removed-in-2.0:
 
 2.0
diff --git a/docs/ref/forms/api.txt b/docs/ref/forms/api.txt
index 194c390484..c74cfaed48 100644
--- a/docs/ref/forms/api.txt
+++ b/docs/ref/forms/api.txt
@@ -720,6 +720,29 @@ When set to ``True`` (the default), required form fields will have the
 ``use_required_attribute=False`` to avoid incorrect browser validation when
 adding and deleting forms from a formset.
 
+Configuring the rendering of a form's widgets
+---------------------------------------------
+
+.. attribute:: Form.default_renderer
+
+.. versionadded:: 1.11
+
+Specifies the :doc:`renderer <renderers>` to use for the form. Defaults to
+``None`` which means to use the default renderer specified by the
+:setting:`FORM_RENDERER` setting.
+
+You can set this as a class attribute when declaring your form or use the
+``renderer`` argument to ``Form.__init__()``. For example::
+
+    from django import forms
+
+    class MyForm(forms.Form):
+        default_renderer = MyRenderer()
+
+or::
+
+    form = MyForm(renderer=MyRenderer())
+
 Notes on field ordering
 -----------------------
 
diff --git a/docs/ref/forms/index.txt b/docs/ref/forms/index.txt
index 618c705b92..241f979954 100644
--- a/docs/ref/forms/index.txt
+++ b/docs/ref/forms/index.txt
@@ -12,5 +12,6 @@ Detailed form API reference. For introductory material, see the
    fields
    models
    formsets
+   renderers
    widgets
    validation
diff --git a/docs/ref/forms/renderers.txt b/docs/ref/forms/renderers.txt
new file mode 100644
index 0000000000..c94b5ef226
--- /dev/null
+++ b/docs/ref/forms/renderers.txt
@@ -0,0 +1,131 @@
+======================
+The form rendering API
+======================
+
+.. module:: django.forms.renderers
+   :synopsis: Built-in form renderers.
+
+.. versionadded:: 1.11
+
+    In older versions, widgets are rendered using Python. All APIs described
+    in this document are new.
+
+Django's form widgets are rendered using Django's :doc:`template engines
+system </topics/templates>`.
+
+The form rendering process can be customized at several levels:
+
+* Widgets can specify custom template names.
+* Forms and widgets can specify custom renderer classes.
+* A widget's template can be overridden by a project. (Reusable applications
+  typically shouldn't override built-in templates because they might conflict
+  with a project's custom templates.)
+
+.. _low-level-widget-render-api:
+
+The low-level render API
+========================
+
+The rendering of form templates is controlled by a customizable renderer class.
+A custom renderer can be specified by updating the :setting:`FORM_RENDERER`
+setting. It defaults to
+``'``:class:`django.forms.renderers.DjangoTemplates`\ ``'``.
+
+You can also provide a custom renderer by setting the
+:attr:`.Form.default_renderer` attribute or by using the ``renderer`` argument
+of :meth:`.Widget.render`.
+
+Use one of the :ref:`built-in template form renderers
+<built-in-template-form-renderers>` or implement your own. Custom renderers
+must implement a ``render(template_name, context, request=None)`` method. It
+should return a rendered templates (as a string) or raise
+:exc:`~django.template.TemplateDoesNotExist`.
+
+.. _built-in-template-form-renderers:
+
+Built-in-template form renderers
+================================
+
+``DjangoTemplates``
+-------------------
+
+.. class:: DjangoTemplates
+
+This renderer uses a standalone
+:class:`~django.template.backends.django.DjangoTemplates`
+engine (unconnected to what you might have configured in the
+:setting:`TEMPLATES` setting). It loads templates first from the built-in form
+templates directory in ``django/forms/templates`` and then from the installed
+apps' templates directories using the :class:`app_directories
+<django.template.loaders.app_directories.Loader>` loader.
+
+If you want to render templates with customizations from your
+:setting:`TEMPLATES` setting, such as context processors for example, use the
+:class:`TemplatesSetting` renderer.
+
+``Jinja2``
+----------
+
+.. class:: Jinja2
+
+This renderer is the same as the :class:`DjangoTemplates` renderer except that
+it uses a :class:`~django.template.backends.jinja2.Jinja2` backend. Templates
+for the built-in widgets are located in ``django/forms/jinja2`` and installed
+apps can provide templates in a ``jinja2`` directory.
+
+To use this backend, all the widgets in your project and its third-party apps
+must have Jinja2 templates. Unless you provide your own Jinja2 templates for
+widgets that don't have any, you can't use this renderer. For example,
+:mod:`django.contrib.admin` doesn't include Jinja2 templates for its widgets
+due to their usage of Django template tags.
+
+``TemplatesSetting``
+--------------------
+
+.. class:: TemplatesSetting
+
+This renderer gives you complete control of how widget templates are sourced.
+It uses :func:`~django.template.loader.get_template` to find widget
+templates based on what's configured in the :setting:`TEMPLATES` setting.
+
+Using this renderer along with the built-in widget templates requires either:
+
+#. ``'django.forms'`` in :setting:`INSTALLED_APPS` and at least one engine
+   with :setting:`APP_DIRS=True <TEMPLATES-APP_DIRS>`.
+
+#. Adding the built-in widgets templates directory (``django/forms/templates``
+   or ``django/forms/jinja2``) in :setting:`DIRS <TEMPLATES-DIRS>` of one of
+   your template engines.
+
+Using this renderer requires you to make sure the form templates your project
+needs can be located.
+
+Context available in widget templates
+=====================================
+
+Widget templates receive a context from :meth:`.Widget.get_context`. By
+default, widgets receive a single value in the context, ``widget``. This is a
+dictionary that contains values like:
+
+* ``name``
+* ``value``
+* ``attrs``
+* ``is_hidden``
+* ``template_name``
+
+Some widgets add further information to the context. For instance, all widgets
+that subclass ``Input`` defines ``widget['type']`` and :class:`.MultiWidget`
+defines ``widget['subwidgets']`` for looping purposes.
+
+Overriding built-in widget templates
+====================================
+
+Each widget has a ``template_name`` attribute with a value such as
+``input.html``. Built-in widget templates are stored in the
+``django/forms/widgets`` path. You can provide a custom template for
+``input.html`` by defining ``django/forms/widgets/input.html``, for example.
+See :ref:`built-in widgets` for the name of each widget's template.
+
+If you use the :class:`TemplatesSetting` renderer, overriding widget templates
+works the same as overriding any other template in your project. You can't
+override built-in widget templates using the other built-in renderers.
diff --git a/docs/ref/forms/widgets.txt b/docs/ref/forms/widgets.txt
index c6148a5460..f0301c1eee 100644
--- a/docs/ref/forms/widgets.txt
+++ b/docs/ref/forms/widgets.txt
@@ -241,6 +241,28 @@ foundation for custom widgets.
             In older versions, this method is a private API named
             ``_format_value()``. The old name will work until Django 2.0.
 
+    .. method:: get_context(name, value, attrs=None)
+
+        .. versionadded:: 1.11
+
+        Returns a dictionary of values to use when rendering the widget
+        template. By default, the dictionary contains a single key,
+        ``'widget'``, which is a dictionary representation of the widget
+        containing the following keys:
+
+        * ``'name'``: The name of the field from the ``name`` argument.
+        * ``'is_hidden'``: A boolean indicating whether or not this widget is
+          hidden.
+        * ``'required'``: A boolean indicating whether or not the field for
+          this widget is required.
+        * ``'value'``: The value as returned by :meth:`format_value`.
+        * ``'attrs'``: HTML attributes to be set on the rendered widget. The
+          combination of the :attr:`attrs` attribute and the ``attrs`` argument.
+        * ``'template_name'``: The value of ``self.template_name``.
+
+        ``Widget`` subclasses can provide custom context values by overriding
+        this method.
+
     .. method:: id_for_label(self, id_)
 
         Returns the HTML ID attribute of this widget for use by a ``<label>``,
@@ -251,14 +273,16 @@ foundation for custom widgets.
         return an ID value that corresponds to the first ID in the widget's
         tags.
 
-    .. method:: render(name, value, attrs=None)
+    .. method:: render(name, value, attrs=None, renderer=None)
 
-        Returns HTML for the widget, as a Unicode string. This method must be
-        implemented by the subclass, otherwise ``NotImplementedError`` will be
-        raised.
+        Renders a widget to HTML using the given renderer. If ``renderer`` is
+        ``None``, the renderer from the :setting:`FORM_RENDERER` setting is
+        used.
 
-        The 'value' given is not guaranteed to be valid input, therefore
-        subclass implementations should program defensively.
+        .. versionchanged:: 1.11
+
+            The ``renderer`` argument was added. Support for subclasses that
+            don't accept it will be removed in Django 2.1.
 
     .. method:: value_from_datadict(data, files, name)
 
@@ -360,40 +384,21 @@ foundation for custom widgets.
             with the opposite responsibility - to combine cleaned values of
             all member fields into one.
 
-    Other methods that may be useful to override include:
-
-    .. method:: render(name, value, attrs=None)
-
-        Argument ``value`` is handled differently in this method from the
-        subclasses of :class:`~Widget` because it has to figure out how to
-        split a single value for display in multiple widgets.
-
-        The ``value`` argument used when rendering can be one of two things:
-
-        * A ``list``.
-        * A single value (e.g., a string) that is the "compressed" representation
-          of a ``list`` of values.
+    It provides some custom context:
 
-        If ``value`` is a list, the output of :meth:`~MultiWidget.render` will
-        be a concatenation of rendered child widgets. If ``value`` is not a
-        list, it will first be processed by the method
-        :meth:`~MultiWidget.decompress()` to create the list and then rendered.
+    .. method:: get_context(name, value, attrs=None)
 
-        When ``render()`` executes its HTML rendering, each value in the list
-        is rendered with the corresponding widget -- the first value is
-        rendered in the first widget, the second value is rendered in the
-        second widget, etc.
+        In addition to the ``'widget'`` key described in
+        :meth:`Widget.get_context`, ``MultiValueWidget`` adds a
+        ``widget['subwidgets']`` key.
 
-        Unlike in the single value widgets, method :meth:`~MultiWidget.render`
-        need not be implemented in the subclasses.
+        These can be looped over in the widget template:
 
-    .. method:: format_output(rendered_widgets)
+        .. code-block:: html+django
 
-        Given a list of rendered widgets (as strings), returns a Unicode string
-        representing the HTML for the whole lot.
-
-        This hook allows you to format the HTML design of the widgets any way
-        you'd like.
+            {% for subwidget in widget.subwidgets %}
+                {% include widget.template_name with widget=subwidget %}
+            {% endfor %}
 
     Here's an example widget which subclasses :class:`MultiWidget` to display
     a date with the day, month, and year in different select boxes. This widget
@@ -421,9 +426,6 @@ foundation for custom widgets.
                     return [value.day, value.month, value.year]
                 return [None, None, None]
 
-            def format_output(self, rendered_widgets):
-                return ''.join(rendered_widgets)
-
             def value_from_datadict(self, data, files, name):
                 datelist = [
                     widget.value_from_datadict(data, files, name + '_%s' % i)
@@ -442,11 +444,6 @@ foundation for custom widgets.
     The constructor creates several :class:`Select` widgets in a tuple. The
     ``super`` class uses this tuple to setup the widget.
 
-    The :meth:`~MultiWidget.format_output` method is fairly vanilla here (in
-    fact, it's the same as what's been implemented as the default for
-    ``MultiWidget``), but the idea is that you could add custom HTML between
-    the widgets should you wish.
-
     The required method :meth:`~MultiWidget.decompress` breaks up a
     ``datetime.date`` value into the day, month, and year values corresponding
     to each widget. Note how the method handles the case where ``value`` is
@@ -485,14 +482,18 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: TextInput
 
-    Text input: ``<input type="text" ...>``
+    * ``input_type``: ``'text'``
+    * ``template_name``: ``'django/forms/widgets/text.html'``
+    * Renders as: ``<input type="text" ...>``
 
 ``NumberInput``
 ~~~~~~~~~~~~~~~
 
 .. class:: NumberInput
 
-    Text input: ``<input type="number" ...>``
+    * ``input_type``: ``'number'``
+    * ``template_name``: ``'django/forms/widgets/number.html'``
+    * Renders as: ``<input type="number" ...>``
 
     Beware that not all browsers support entering localized numbers in
     ``number`` input types. Django itself avoids using them for fields having
@@ -503,21 +504,27 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: EmailInput
 
-    Text input: ``<input type="email" ...>``
+    * ``input_type``: ``'email'``
+    * ``template_name``: ``'django/forms/widgets/email.html'``
+    * Renders as: ``<input type="email" ...>``
 
 ``URLInput``
 ~~~~~~~~~~~~
 
 .. class:: URLInput
 
-    Text input: ``<input type="url" ...>``
+    * ``input_type``: ``'url'``
+    * ``template_name``: ``'django/forms/widgets/url.html'``
+    * Renders as: ``<input type="url" ...>``
 
 ``PasswordInput``
 ~~~~~~~~~~~~~~~~~
 
 .. class:: PasswordInput
 
-    Password input: ``<input type='password' ...>``
+    * ``input_type``: ``'password'``
+    * ``template_name``: ``'django/forms/widgets/password.html'``
+    * Renders as: ``<input type='password' ...>``
 
     Takes one optional argument:
 
@@ -531,7 +538,9 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: HiddenInput
 
-    Hidden input: ``<input type='hidden' ...>``
+    * ``input_type``: ``'hidden'``
+    * ``template_name``: ``'django/forms/widgets/hidden.html'``
+    * Renders as: ``<input type='hidden' ...>``
 
     Note that there also is a :class:`MultipleHiddenInput` widget that
     encapsulates a set of hidden input elements.
@@ -541,7 +550,9 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: DateInput
 
-    Date input as a simple text box: ``<input type='text' ...>``
+    * ``input_type``: ``'text'``
+    * ``template_name``: ``'django/forms/widgets/date.html'``
+    * Renders as: ``<input type='text' ...>``
 
     Takes same arguments as :class:`TextInput`, with one more optional argument:
 
@@ -558,7 +569,9 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: DateTimeInput
 
-    Date/time input as a simple text box: ``<input type='text' ...>``
+    * ``input_type``: ``'text'``
+    * ``template_name``: ``'django/forms/widgets/datetime.html'``
+    * Renders as: ``<input type='text' ...>``
 
     Takes same arguments as :class:`TextInput`, with one more optional argument:
 
@@ -579,7 +592,9 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: TimeInput
 
-    Time input as a simple text box: ``<input type='text' ...>``
+    * ``input_type``: ``'text'``
+    * ``template_name``: ``'django/forms/widgets/time.html'``
+    * Renders as: ``<input type='text' ...>``
 
     Takes same arguments as :class:`TextInput`, with one more optional argument:
 
@@ -598,7 +613,8 @@ These widgets make use of the HTML elements ``input`` and ``textarea``.
 
 .. class:: Textarea
 
-    Text area: ``<textarea>...</textarea>``
+    * ``template_name``: ``'django/forms/widgets/textarea.html'``
+    * Renders as: ``<textarea>...</textarea>``
 
 .. _selector-widgets:
 
@@ -610,7 +626,9 @@ Selector and checkbox widgets
 
 .. class:: CheckboxInput
 
-    Checkbox: ``<input type='checkbox' ...>``
+    * ``input_type``: ``'checkbox'``
+    * ``template_name``: ``'django/forms/widgets/checkbox.html'``
+    * Renders as: ``<input type='checkbox' ...>``
 
     Takes one optional argument:
 
@@ -624,7 +642,8 @@ Selector and checkbox widgets
 
 .. class:: Select
 
-    Select widget: ``<select><option ...>...</select>``
+    * ``template_name``: ``'django/forms/widgets/select.html'``
+    * Renders as: ``<select><option ...>...</select>``
 
     .. attribute:: Select.choices
 
@@ -637,6 +656,8 @@ Selector and checkbox widgets
 
 .. class:: NullBooleanSelect
 
+    * ``template_name``: ``'django/forms/widgets/select.html'``
+
     Select widget with options 'Unknown', 'Yes' and 'No'
 
 ``SelectMultiple``
@@ -644,6 +665,8 @@ Selector and checkbox widgets
 
 .. class:: SelectMultiple
 
+    * ``template_name``: ``'django/forms/widgets/select.html'``
+
     Similar to :class:`Select`, but allows multiple selection:
     ``<select multiple='multiple'>...</select>``
 
@@ -652,6 +675,8 @@ Selector and checkbox widgets
 
 .. class:: RadioSelect
 
+    * ``template_name``: ``'django/forms/widgets/radio.html'``
+
     Similar to :class:`Select`, but rendered as a list of radio buttons within
     ``<li>`` tags:
 
@@ -744,6 +769,8 @@ Selector and checkbox widgets
 
 .. class:: CheckboxSelectMultiple
 
+    * ``template_name``: ``'django/forms/widgets/checkbox_select.html'``
+
     Similar to :class:`SelectMultiple`, but rendered as a list of check
     buttons:
 
@@ -776,16 +803,18 @@ File upload widgets
 
 .. class:: FileInput
 
-    File upload input: ``<input type='file' ...>``
+    * ``template_name``: ``'django/forms/widgets/file.html'``
+    * Renders as: ``<input type='file' ...>``
 
 ``ClearableFileInput``
 ~~~~~~~~~~~~~~~~~~~~~~
 
 .. class:: ClearableFileInput
 
-    File upload input: ``<input type='file' ...>``, with an additional checkbox
-    input to clear the field's value, if the field is not required and has
-    initial data.
+    * ``template_name``: ``'django/forms/widgets/clearable_file_input.html'``
+    * Renders as: ``<input type='file' ...>`` with an additional checkbox
+      input to clear the field's value, if the field is not required and has
+      initial data.
 
 .. _composite-widgets:
 
@@ -797,7 +826,8 @@ Composite widgets
 
 .. class:: MultipleHiddenInput
 
-    Multiple ``<input type='hidden' ...>`` widgets.
+    * ``template_name``: ``'django/forms/widgets/multiple_hidden.html'``
+    * Renders as: multiple ``<input type='hidden' ...>`` tags
 
     A widget that handles multiple hidden widgets for fields that have a list
     of values.
@@ -813,6 +843,8 @@ Composite widgets
 
 .. class:: SplitDateTimeWidget
 
+    * ``template_name``: ``'django/forms/widgets/splitdatetime.html'``
+
     Wrapper (using :class:`MultiWidget`) around two widgets: :class:`DateInput`
     for the date, and :class:`TimeInput` for the time. Must be used with
     :class:`SplitDateTimeField` rather than :class:`DateTimeField`.
@@ -832,6 +864,8 @@ Composite widgets
 
 .. class:: SplitHiddenDateTimeWidget
 
+    * ``template_name``: ``'django/forms/widgets/splithiddendatetime.html'``
+
     Similar to :class:`SplitDateTimeWidget`, but uses :class:`HiddenInput` for
     both date and time.
 
@@ -840,6 +874,8 @@ Composite widgets
 
 .. class:: SelectDateWidget
 
+    * ``template_name``: ``'django/forms/widgets/select_date.html'``
+
     Wrapper around three :class:`~django.forms.Select` widgets: one each for
     month, day, and year.
 
diff --git a/docs/ref/settings.txt b/docs/ref/settings.txt
index 1d28c96137..10f60b3b2e 100644
--- a/docs/ref/settings.txt
+++ b/docs/ref/settings.txt
@@ -1517,6 +1517,18 @@ generate correct URLs when ``SCRIPT_NAME`` is not ``/``.
 
     The setting's use in :func:`django.setup()` was added.
 
+.. setting:: FORM_RENDERER
+
+``FORM_RENDERER``
+-----------------
+
+.. versionadded:: 1.11
+
+Default: ``'``:class:`django.forms.renderers.DjangoTemplates`\ ``'``
+
+The class that renders form widgets. It must implement :ref:`the low-level
+render API <low-level-widget-render-api>`.
+
 .. setting:: FORMAT_MODULE_PATH
 
 ``FORMAT_MODULE_PATH``
@@ -3351,6 +3363,10 @@ File uploads
 * :setting:`MEDIA_ROOT`
 * :setting:`MEDIA_URL`
 
+Forms
+-----
+* :setting:`FORM_RENDERER`
+
 Globalization (``i18n``/``l10n``)
 ---------------------------------
 * :setting:`DATE_FORMAT`
diff --git a/docs/releases/1.11.txt b/docs/releases/1.11.txt
index 48849f3b4d..f894c261f9 100644
--- a/docs/releases/1.11.txt
+++ b/docs/releases/1.11.txt
@@ -61,6 +61,15 @@ It can be subclassed to support different index types, such as
 :class:`~django.contrib.postgres.indexes.GinIndex`. It also allows defining the
 order (ASC/DESC) for the columns of the index.
 
+Template-based widget rendering
+-------------------------------
+
+To ease customizing widgets, form widget rendering is now done using the
+template system rather than in Python. See :doc:`/ref/forms/renderers`.
+
+You may need to adjust any custom widgets that you've written for a few
+:ref:`backwards incompatible changes <template-widget-incompatibilities-1-11>`.
+
 Minor features
 --------------
 
@@ -551,6 +560,21 @@ inside help text.
 Read-only fields are wrapped in ``<div class="readonly">...</div>`` instead of
 ``<p>...</p>`` to allow any kind of HTML as the field's content.
 
+.. _template-widget-incompatibilities-1-11:
+
+Changes due to the introduction of template-based widget rendering
+------------------------------------------------------------------
+
+Some undocumented classes in ``django.forms.widgets`` are removed:
+
+* ``SubWidget``
+* ``RendererMixin``, ``ChoiceFieldRenderer``, ``RadioFieldRenderer``,
+  ``CheckboxFieldRenderer``
+* ``ChoiceInput``, ``RadioChoiceInput``, ``CheckboxChoiceInput``
+
+The ``Widget.format_output()`` method is removed. Use a custom widget template
+instead.
+
 Miscellaneous
 -------------
 
@@ -754,3 +778,7 @@ Miscellaneous
   entries for search engines, for example. An alternative solution could be to
   create a :data:`~django.conf.urls.handler404` that looks for uppercase
   characters in the URL and redirects to a lowercase equivalent.
+
+* The ``renderer`` argument is added to the :meth:`Widget.render()
+  <django.forms.Widget.render>` method. Methods that don't accept that argument
+  will work through a deprecation period.
diff --git a/docs/spelling_wordlist b/docs/spelling_wordlist
index 8405db76dc..ce3c4c23ad 100644
--- a/docs/spelling_wordlist
+++ b/docs/spelling_wordlist
@@ -673,6 +673,7 @@ releasers
 reloader
 removetags
 renderer
+renderers
 repo
 reportable
 reprojection