Implemented auto-escaping of variable output in templates. Fully controllable by template authors and it's possible to write filters and templates that simulataneously work in both auto-escaped and non-auto-escaped environments if you need to. Fixed #2359

See documentation in templates.txt and templates_python.txt for how everything
works.

Backwards incompatible if you're inserting raw HTML output via template variables.

Based on an original design from Simon Willison and with debugging help from Michael Radziej.


git-svn-id: http://code.djangoproject.com/svn/django/trunk@6671 bcc190cf-cafb-0310-a4f2-bffc1f526a37

2 files changed, 282 insertions, 9 deletions

diff --git a/docs/templates.txt b/docs/templates.txt
index 68dbfa3e63..b85f108bbe 100644
--- a/docs/templates.txt
+++ b/docs/templates.txt
@@ -299,6 +299,104 @@ it also defines the content that fills the hole in the *parent*. If there were
 two similarly-named ``{% block %}`` tags in a template, that template's parent
 wouldn't know which one of the blocks' content to use.
 
+Automatic HTML escaping
+=======================
+
+**New in Django development version**
+
+A very real problem when creating HTML (and other) output using templates and
+variable substitution is the possibility of accidently inserting some variable
+value that affects the resulting HTML. For example, a template fragment such as
+::
+
+    Hello, {{ name }}.
+
+seems like a harmless way to display the user's name. However, if you are
+displaying data that the user entered directly and they had entered their name as ::
+
+    <script>alert('hello')</script>
+
+this would always display a Javascript alert box when the page was loaded.
+Similarly, if you were displaying some data generated by another process and it
+contained a '<' symbol, you couldn't just dump this straight into your HTML,
+because it would be treated as the start of an element. The effects of these
+sorts of problems can vary from merely annoying to allowing exploits via `Cross
+Site Scripting`_ (XSS) attacks.
+
+.. _Cross Site Scripting: http://en.wikipedia.org/wiki/Cross-site_scripting
+
+In order to provide some protection against these problems, Django
+provides automatic (but controllable) HTML escaping for data coming from
+tempate variables. Inside this tag, any data that comes from template
+variables is examined to see if it contains one of the five HTML characters
+(<, >, ', " and &) that often need escaping and those characters are converted
+to their respective HTML entities. It causes no harm if a character is
+converted to an entity when it doesn't need to be, so all five characters are
+always converted.
+
+Since some variables will contain data that is *intended* to be rendered
+as HTML, template tag and filter writers can mark their output strings as
+requiring no further escaping. For example, the ``unordered_list`` filter is
+designed to return raw HTML and we want the template processor to simply
+display the results as returned, without applying any escaping. That is taken
+care of by the filter. The template author need do nothing special in that
+case.
+
+By default, automatic HTML escaping is always applied. However, sometimes you
+will not want this to occur (for example, if you're using the templating
+system to create an email). To control automatic escaping inside your template,
+wrap the affected content in the ``autoescape`` tag, like so::
+
+    {% autoescape off %}
+        Hello {{ name }}
+    {% endautoescape %}
+
+The auto-escaping tag passes its effect onto templates that extend the
+current one as well as templates included via the ``include`` tag, just like
+all block tags.
+
+The ``autoescape`` tag takes either ``on`` or ``off`` as its argument. At times, you might want to force auto-escaping when it would otherwise be disabled. For example::
+
+    Auto-escaping is on by default. Hello {{ name }}
+
+    {% autoescape off %}
+        This will not be auto-escaped: {{ data }}.
+
+        Nor this: {{ other_data }}
+        {% autoescape on %}
+            Auto-escaping applies again, {{ name }}
+        {% endautoescape %}
+    {% endautoescape %}
+
+For individual variables, the ``safe`` filter can also be used to indicate
+that the contents should not be automatically escaped::
+
+    This will be escaped: {{ data }}
+    This will not be escaped: {{ data|safe }}
+
+Think of *safe* as shorthand for *safe from further escaping* or *can be
+safely interpreted as HTML*. In this example, if ``data`` contains ``'<a>'``,
+the output will be::
+
+    This will be escaped: &lt;a&gt;
+    This will not be escaped: <a>
+
+Generally, you won't need to worry about auto-escaping very much. View
+developers and custom filter authors need to think about when their data
+shouldn't be escaped and mark it appropriately. They are in a better position
+to know when that should happen than the template author, so it is their
+responsibility. By default, all output is escaped unless the template
+processor is explicitly told otherwise.
+
+You should also note that if you are trying to write a template that might be
+used in situations where automatic escaping is enabled or disabled and you
+don't know which (such as when your template is included in other templates),
+you can safely write as if you were in an ``{% autoescape off %}`` situation.
+Scatter ``escape`` filters around for any variables that need escaping. When
+auto-escaping is on, these extra filters won't change the output -- any
+variables that use the ``escape`` filter do not have further automatic
+escaping applied to them.
+
 Using the built-in reference
 ============================
 
@@ -374,6 +472,24 @@ available, and what they do.
 Built-in tag reference
 ----------------------
 
+autoescape
+~~~~~~~~~~
+
+**New in Django development version**
+
+Control the current auto-escaping behaviour. This tag takes either ``on`` or
+``off`` as an argument and that determines whether auto-escaping is in effect
+inside the block.
+
+When auto-escaping is in effect, all variable content has HTML escaping applied
+to it before placing the result into the output (but after any filters have
+been applied). This is equivalent to manually applying the ``escape`` filter
+attached to each variable.
+
+The only exceptions are variables that are already marked as 'safe' from
+escaping, either by the code that populated the variable, or because it has
+the ``safe`` or ``escape`` filters applied.
+
 block
 ~~~~~
 
@@ -452,7 +568,7 @@ just like in variable syntax.
 
 Sample usage::
 
-    {% filter escape|lower %}
+    {% filter force_escape|lower %}
         This text will be HTML-escaped, and will appear in all lowercase.
     {% endfilter %}
 
@@ -1076,6 +1192,10 @@ Returns true if the value is divisible by the argument.
 escape
 ~~~~~~
 
+**New in Django development version:** The behaviour of this filter has
+changed slightly in the development version (the affects are only applied
+once, after all other filters).
+
 Escapes a string's HTML. Specifically, it makes these replacements:
 
     * ``"&"`` to ``"&amp;"``
@@ -1084,6 +1204,16 @@ Escapes a string's HTML. Specifically, it makes these replacements:
     * ``'"'`` (double quote) to ``'&quot;'``
     * ``"'"`` (single quote) to ``'&#39;'``
 
+The escaping is only applied when the string is output, so it does not matter
+where in a chained sequence of filters you put ``escape``: it will always be
+applied as though it were the last filter. If you want escaping to be applied
+immediately, use the ``force_escape`` filter.
+
+Applying ``escape`` to a variable that would normally have auto-escaping
+applied to the result will only result in one round of escaping being done. So
+it is safe to use this function even in auto-escaping environments. If you want
+multiple escaping passes to be applied, use the ``force_escape`` filter.
+
 filesizeformat
 ~~~~~~~~~~~~~~
 
@@ -1140,6 +1270,17 @@ value     Template                      Output
 Using ``floatformat`` with no argument is equivalent to using ``floatformat``
 with an argument of ``-1``.
 
+force_escape
+~~~~~~~~~~~~
+
+**New in Django development version**
+
+Applies HTML escaping to a string (see the ``escape`` filter for details).
+This filter is applied *immediately* and returns a new, escaped string. This
+is useful in the rare cases where you need multiple escaping or want to apply
+other filters to the escaped results. Normally, you want to use the ``escape``
+filter.
+
 get_digit
 ~~~~~~~~~
 
@@ -1264,6 +1405,12 @@ Right-aligns the value in a field of a given width.
 
 **Argument:** field size
 
+safe
+~~~~
+
+Marks a string as not requiring further HTML escaping prior to output. When
+autoescaping is off, this filter has no effect.
+
 slice
 ~~~~~
 
diff --git a/docs/templates_python.txt b/docs/templates_python.txt
index bd105888ce..e4658f6461 100644
--- a/docs/templates_python.txt
+++ b/docs/templates_python.txt
@@ -219,13 +219,13 @@ be replaced with the name of the invalid variable.
 
     While ``TEMPLATE_STRING_IF_INVALID`` can be a useful debugging tool,
     it is a bad idea to turn it on as a 'development default'.
-    
+
     Many templates, including those in the Admin site, rely upon the
     silence of the template system when a non-existent variable is
     encountered. If you assign a value other than ``''`` to
     ``TEMPLATE_STRING_IF_INVALID``, you will experience rendering
     problems with these templates and sites.
-    
+
     Generally, ``TEMPLATE_STRING_IF_INVALID`` should only be enabled
     in order to debug a specific template problem, then cleared
     once debugging is complete.
@@ -722,6 +722,95 @@ decorator instead::
 If you leave off the ``name`` argument, as in the second example above, Django
 will use the function's name as the filter name.
 
+Filters and auto-escaping
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**New in Django development version**
+
+When you are writing a custom filter, you need to give some thought to how
+this filter will interact with Django's auto-escaping behaviour.  Firstly, you
+should realise that there are three types of strings that can be passed around
+inside the template code:
+
+ * raw strings are the native Python ``str`` or ``unicode`` types. On
+   output, they are escaped if auto-escaping is in effect and presented
+   unchanged, otherwise.
+
+ * "safe" strings are strings that are safe from further escaping at output
+   time. Any necessary escaping has already been done. They are commonly used
+   for output that contains raw HTML that is intended to be intrepreted on the
+   client side.
+
+   Internally, these strings are of type ``SafeString`` or ``SafeUnicode``,
+   although they share a common base class in ``SafeData``, so you can test
+   for them using code like::
+
+    if isinstance(value, SafeData):
+        # Do something with the "safe" string.
+
+ * strings which are marked as "needing escaping" are *always* escaped on
+   output, regardless of whether they are in an ``autoescape`` block or not.
+   These strings are only escaped once, however, even if auto-escaping
+   applies. This type of string is internally represented by the types
+   ``EscapeString`` and ``EscapeUnicode``. You will not normally need to worry
+   about these; they exist for the implementation of the ``escape`` filter.
+
+Inside your filter, you will need to think about three areas in order to be
+auto-escaping compliant:
+
+ 1. If your filter returns a string that is ready for direct output (it should
+    be considered a "safe" string), you should call
+    ``django.utils.safestring.mark_safe()`` on the result prior to returning.
+    This will turn the result into the appropriate ``SafeData`` type. This is
+    often the case when you are returning raw HTML, for example.
+
+ 2. If your filter is given a "safe" string, is it guaranteed to return a
+    "safe" string? If so, set the ``is_safe`` attribute on the function to be
+    ``True``. For example, a filter that replaced a word consisting only of
+    digits with the number spelt out in words is going to be
+    safe-string-preserving, since it cannot introduce any of the five dangerous
+    characters: <, >, ", ' or &. We can write::
+
+        @register.filter
+        def convert_to_words(value):
+            # ... implementation here ...
+            return result
+
+        convert_to_words.is_safe = True
+
+    Note that this filter does not return a universally safe result (it does
+    not return ``mark_safe(result)``) because if it is handed a raw string such
+    as '<a>', this will need further escaping in an auto-escape environment.
+    The ``is_safe`` attribute only talks about the the result when a safe
+    string is passed into the filter.
+
+ 3. Will your filter behave differently depending upon whether auto-escaping
+    is currently in effect or not? This is normally a concern when you are
+    returning mixed content (HTML elements mixed with user-supplied content).
+    For example, the ``ordered_list`` filter that ships with Django needs to
+    know whether to escape its content or not. It will always return a safe
+    string. Since it returns raw HTML, we cannot apply escaping to the
+    result -- it needs to be done in-situ.
+
+    For these cases, the filter function needs to be told what the current
+    auto-escaping setting is. Set the ``needs_autoescape`` attribute on the
+    filter to ``True`` and have your function take an extra argument called
+    ``autoescape`` with a default value of ``None``. When the filter is called,
+    the ``autoescape`` keyword argument will be ``True`` if auto-escaping is in
+    effect. For example, the ``unordered_list`` filter is written as::
+
+        def unordered_list(value, autoescape=None):
+            # ... lots of code here ...
+
+            return mark_safe(...)
+
+        unordered_list.is_safe = True
+        unordered_list.needs_autoescape = True
+
+By default, both the ``is_safe`` and ``needs_autoescape`` attributes are
+``False``. You do not need to specify them if ``False`` is an acceptable
+value.
+
 Writing custom template tags
 ----------------------------
 
@@ -840,6 +929,43 @@ Ultimately, this decoupling of compilation and rendering results in an
 efficient template system, because a template can render multiple context
 without having to be parsed multiple times.
 
+Auto-escaping considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The output from template tags is not automatically run through the
+auto-escaping filters. However, there are still a couple of things you should
+keep in mind when writing a template tag:
+
+If the ``render()`` function of your template stores the result in a context
+variable (rather than returning the result in a string), it should take care
+to call ``mark_safe()`` if appropriate. When the variable is ultimately
+rendered, it will be affected by the auto-escape setting in effect at the
+time, so content that should be safe from further escaping needs to be marked
+as such.
+
+Also, if your template tag creates a new context for performing some
+sub-rendering, you should be careful to set the auto-escape attribute to the
+current context's value. The ``__init__`` method for the ``Context`` class
+takes a parameter called ``autoescape`` that you can use for this purpose. For
+example::
+
+    def render(self, context):
+        # ...
+        new_context = Context({'var': obj}, autoescape=context.autoescape)
+        # ... Do something with new_context ...
+
+This is not a very common situation, but it is sometimes useful, particularly
+if you are rendering a template yourself. For example::
+
+    def render(self, context):
+        t = template.load_template('small_fragment.html')
+        return t.render(Context({'var': obj}, autoescape=context.autoescape))
+
+If we had neglected to pass in the current ``context.autoescape`` value to our
+new ``Context`` in this example, the results would have *always* been
+automatically escaped, which may not be the desired behaviour if the template
+tag is used inside a ``{% autoescape off %}`` block.
+
 Registering the tag
 ~~~~~~~~~~~~~~~~~~~
 
@@ -917,7 +1043,7 @@ current context, available in the ``render`` method::
         def __init__(self, date_to_be_formatted, format_string):
             self.date_to_be_formatted = date_to_be_formatted
             self.format_string = format_string
-        
+
         def render(self, context):
             try:
                 actual_date = resolve_variable(self.date_to_be_formatted, context)
@@ -934,26 +1060,26 @@ format it accordingly.
     ``template.resolve_variable()`` is still available, but has been deprecated
     in favor of a new ``template.Variable`` class. Using this class will usually
     be more efficient than calling ``template.resolve_variable``
-    
+
     To use the ``Variable`` class, simply instantiate it with the name of the
     variable to be resolved, and then call ``variable.resolve(context)``. So,
     in the development version, the above example would be more correctly
     written as:
-    
+
     .. parsed-literal::
-    
+
         class FormatTimeNode(template.Node):
             def __init__(self, date_to_be_formatted, format_string):
                 self.date_to_be_formatted = **Variable(date_to_be_formatted)**
                 self.format_string = format_string
-        
+
             def render(self, context):
                 try:
                     actual_date = **self.date_to_be_formatted.resolve(context)**
                     return actual_date.strftime(self.format_string)
                 except template.VariableDoesNotExist:
                     return ''
-    
+
     Changes are highlighted in bold.
 
 Variable resolution will throw a ``VariableDoesNotExist`` exception if it cannot