Fixed #17365, #17366, #18727 -- Switched to discovery test runner.

Thanks to Preston Timmons for the bulk of the work on the patch, especially
updating Django's own test suite to comply with the requirements of the new
runner. Thanks also to Jannis Leidel and Mahdi Yusuf for earlier work on the
patch and the discovery runner.

Refs #11077, #17032, and #18670.

4 files changed, 89 insertions, 240 deletions

diff --git a/docs/topics/testing/advanced.txt b/docs/topics/testing/advanced.txt
index 5f2fa65bed..cefb770469 100644
--- a/docs/topics/testing/advanced.txt
+++ b/docs/topics/testing/advanced.txt
@@ -165,7 +165,7 @@ environment first. Django provides a convenience method to do this::
 
 :func:`~django.test.utils.setup_test_environment` puts several Django features
 into modes that allow for repeatable testing, but does not create the test
-databases; :func:`django.test.simple.DjangoTestSuiteRunner.setup_databases`
+databases; :func:`django.test.runner.DiscoverRunner.setup_databases`
 takes care of that.
 
 The call to :func:`~django.test.utils.setup_test_environment` is made
@@ -178,27 +178,27 @@ tests via Django's test runner.
 Using different testing frameworks
 ==================================
 
-Clearly, :mod:`doctest` and :mod:`unittest` are not the only Python testing
-frameworks. While Django doesn't provide explicit support for alternative
-frameworks, it does provide a way to invoke tests constructed for an
-alternative framework as if they were normal Django tests.
+Clearly, :mod:`unittest` is not the only Python testing framework. While Django
+doesn't provide explicit support for alternative frameworks, it does provide a
+way to invoke tests constructed for an alternative framework as if they were
+normal Django tests.
 
 When you run ``./manage.py test``, Django looks at the :setting:`TEST_RUNNER`
 setting to determine what to do. By default, :setting:`TEST_RUNNER` points to
-``'django.test.simple.DjangoTestSuiteRunner'``. This class defines the default Django
+``'django.test.runner.DiscoverRunner'``. This class defines the default Django
 testing behavior. This behavior involves:
 
 #. Performing global pre-test setup.
 
-#. Looking for unit tests and doctests in the ``models.py`` and
-   ``tests.py`` files in each installed application.
+#. Looking for tests in any file below the current directory whose name matches
+   the pattern ``test*.py``.
 
 #. Creating the test databases.
 
 #. Running ``syncdb`` to install models and initial data into the test
    databases.
 
-#. Running the unit tests and doctests that are found.
+#. Running the tests that were found.
 
 #. Destroying the test databases.
 
@@ -215,15 +215,22 @@ process to satisfy whatever testing requirements you may have.
 Defining a test runner
 ----------------------
 
-.. currentmodule:: django.test.simple
+.. currentmodule:: django.test.runner
 
 A test runner is a class defining a ``run_tests()`` method. Django ships
-with a ``DjangoTestSuiteRunner`` class that defines the default Django
+with a ``DiscoverRunner`` class that defines the default Django
 testing behavior. This class defines the ``run_tests()`` entry point,
 plus a selection of other methods that are used to by ``run_tests()`` to
 set up, execute and tear down the test suite.
 
-.. class:: DjangoTestSuiteRunner(verbosity=1, interactive=True, failfast=True, **kwargs)
+.. class:: DiscoverRunner(pattern='test*.py', top_level=None, verbosity=1, interactive=True, failfast=True, **kwargs)
+
+    ``DiscoverRunner`` will search for tests in any file matching ``pattern``.
+
+    ``top_level`` can be used to specify the directory containing your
+    top-level Python modules. Usually Django can figure this out automatically,
+    so it's not necessary to specify this option. If specified, it should
+    generally be the directory containing your ``manage.py`` file.
 
     ``verbosity`` determines the amount of notification and debug information
     that will be printed to the console; ``0`` is no output, ``1`` is normal
@@ -238,11 +245,10 @@ set up, execute and tear down the test suite.
     If ``failfast`` is ``True``, the test suite will stop running after the
     first test failure is detected.
 
-    Django will, from time to time, extend the capabilities of
-    the test runner by adding new arguments. The ``**kwargs`` declaration
-    allows for this expansion. If you subclass ``DjangoTestSuiteRunner`` or
-    write your own test runner, ensure accept and handle the ``**kwargs``
-    parameter.
+    Django may, from time to time, extend the capabilities of the test runner
+    by adding new arguments. The ``**kwargs`` declaration allows for this
+    expansion. If you subclass ``DiscoverRunner`` or write your own test
+    runner, ensure it accepts ``**kwargs``.
 
     Your test runner may also define additional command-line options.
     If you add an ``option_list`` attribute to a subclassed test runner,
@@ -252,7 +258,7 @@ set up, execute and tear down the test suite.
 Attributes
 ~~~~~~~~~~
 
-.. attribute:: DjangoTestSuiteRunner.option_list
+.. attribute:: DiscoverRunner.option_list
 
     This is the tuple of ``optparse`` options which will be fed into the
     management command's ``OptionParser`` for parsing arguments. See the
@@ -261,20 +267,25 @@ Attributes
 Methods
 ~~~~~~~
 
-.. method:: DjangoTestSuiteRunner.run_tests(test_labels, extra_tests=None, **kwargs)
+.. method:: DiscoverRunner.run_tests(test_labels, extra_tests=None, **kwargs)
 
     Run the test suite.
 
     ``test_labels`` is a list of strings describing the tests to be run. A test
-    label can take one of three forms:
+    label can take one of four forms:
 
-    * ``app.TestCase.test_method`` -- Run a single test method in a test
+    * ``path.to.test_module.TestCase.test_method`` -- Run a single test method
+      in a test case.
+    * ``path.to.test_module.TestCase`` -- Run all the test methods in a test
       case.
-    * ``app.TestCase`` -- Run all the test methods in a test case.
-    * ``app`` -- Search for and run all tests in the named application.
+    * ``path.to.module`` -- Search for and run all tests in the named Python
+      package or module.
+    * ``path/to/directory`` -- Search for and run all tests below the named
+      directory.
 
-    If ``test_labels`` has a value of ``None``, the test runner should run
-    search for tests in all the applications in :setting:`INSTALLED_APPS`.
+    If ``test_labels`` has a value of ``None``, the test runner will search for
+    tests in all files below the current directory whose names match its
+    ``pattern`` (see above).
 
     ``extra_tests`` is a list of extra ``TestCase`` instances to add to the
     suite that is executed by the test runner. These extra tests are run
@@ -282,13 +293,13 @@ Methods
 
     This method should return the number of tests that failed.
 
-.. method:: DjangoTestSuiteRunner.setup_test_environment(**kwargs)
+.. method:: DiscoverRunner.setup_test_environment(**kwargs)
 
     Sets up the test environment by calling
     :func:`~django.test.utils.setup_test_environment` and setting
     :setting:`DEBUG` to ``False``.
 
-.. method:: DjangoTestSuiteRunner.build_suite(test_labels, extra_tests=None, **kwargs)
+.. method:: DiscoverRunner.build_suite(test_labels, extra_tests=None, **kwargs)
 
     Constructs a test suite that matches the test labels provided.
 
@@ -309,7 +320,7 @@ Methods
 
     Returns a ``TestSuite`` instance ready to be run.
 
-.. method:: DjangoTestSuiteRunner.setup_databases(**kwargs)
+.. method:: DiscoverRunner.setup_databases(**kwargs)
 
     Creates the test databases.
 
@@ -317,13 +328,13 @@ Methods
     that have been made. This data will be provided to the ``teardown_databases()``
     function at the conclusion of testing.
 
-.. method:: DjangoTestSuiteRunner.run_suite(suite, **kwargs)
+.. method:: DiscoverRunner.run_suite(suite, **kwargs)
 
     Runs the test suite.
 
     Returns the result produced by the running the test suite.
 
-.. method:: DjangoTestSuiteRunner.teardown_databases(old_config, **kwargs)
+.. method:: DiscoverRunner.teardown_databases(old_config, **kwargs)
 
     Destroys the test databases, restoring pre-test conditions.
 
@@ -331,11 +342,11 @@ Methods
     database configuration that need to be reversed. It is the return
     value of the ``setup_databases()`` method.
 
-.. method:: DjangoTestSuiteRunner.teardown_test_environment(**kwargs)
+.. method:: DiscoverRunner.teardown_test_environment(**kwargs)
 
     Restores the pre-test environment.
 
-.. method:: DjangoTestSuiteRunner.suite_result(suite, result, **kwargs)
+.. method:: DiscoverRunner.suite_result(suite, result, **kwargs)
 
     Computes and returns a return code based on a test suite, and the result
     from that test suite.
@@ -402,7 +413,7 @@ can be useful during testing.
     ``old_database_name``.
 
     The ``verbosity`` argument has the same behavior as for
-    :class:`~django.test.simple.DjangoTestSuiteRunner`.
+    :class:`~django.test.runner.DiscoverRunner`.
 
 .. _topics-testing-code-coverage:
 
diff --git a/docs/topics/testing/doctests.txt b/docs/topics/testing/doctests.txt
deleted file mode 100644
index 5036e946a9..0000000000
--- a/docs/topics/testing/doctests.txt
+++ /dev/null
@@ -1,81 +0,0 @@
-===================
-Django and doctests
-===================
-
-Doctests use Python's standard :mod:`doctest` module, which searches your
-docstrings for statements that resemble a session of the Python interactive
-interpreter. A full explanation of how :mod:`doctest` works is out of the scope
-of this document; read Python's official documentation for the details.
-
-.. admonition:: What's a **docstring**?
-
-    A good explanation of docstrings (and some guidelines for using them
-    effectively) can be found in :pep:`257`:
-
-        A docstring is a string literal that occurs as the first statement in
-        a module, function, class, or method definition.  Such a docstring
-        becomes the ``__doc__`` special attribute of that object.
-
-    For example, this function has a docstring that describes what it does::
-
-        def add_two(num):
-            "Return the result of adding two to the provided number."
-            return num + 2
-
-    Because tests often make great documentation, putting tests directly in
-    your docstrings is an effective way to document *and* test your code.
-
-As with unit tests, for a given Django application, the test runner looks for
-doctests in two places:
-
-* The ``models.py`` file. You can define module-level doctests and/or a
-  doctest for individual models. It's common practice to put
-  application-level doctests in the module docstring and model-level
-  doctests in the model docstrings.
-
-* A file called ``tests.py`` in the application directory -- i.e., the
-  directory that holds ``models.py``. This file is a hook for any and all
-  doctests you want to write that aren't necessarily related to models.
-
-This example doctest is equivalent to the example given in the unittest section
-above::
-
-    # models.py
-
-    from django.db import models
-
-    class Animal(models.Model):
-        """
-        An animal that knows how to make noise
-
-        # Create some animals
-        >>> lion = Animal.objects.create(name="lion", sound="roar")
-        >>> cat = Animal.objects.create(name="cat", sound="meow")
-
-        # Make 'em speak
-        >>> lion.speak()
-        'The lion says "roar"'
-        >>> cat.speak()
-        'The cat says "meow"'
-        """
-        name = models.CharField(max_length=20)
-        sound = models.CharField(max_length=20)
-
-        def speak(self):
-            return 'The %s says "%s"' % (self.name, self.sound)
-
-When you :ref:`run your tests <running-tests>`, the test runner will find this
-docstring, notice that portions of it look like an interactive Python session,
-and execute those lines while checking that the results match.
-
-In the case of model tests, note that the test runner takes care of creating
-its own test database. That is, any test that accesses a database -- by
-creating and saving model instances, for example -- will not affect your
-production database. However, the database is not refreshed between doctests,
-so if your doctest requires a certain state you should consider flushing the
-database or loading a fixture. (See the section on :ref:`fixtures
-<topics-testing-fixtures>` for more on this.) Note that to use this feature,
-the database user Django is connecting as must have ``CREATE DATABASE``
-rights.
-
-For more details about :mod:`doctest`, see the Python documentation.
diff --git a/docs/topics/testing/index.txt b/docs/topics/testing/index.txt
index 94e88bdf04..1a99a399b4 100644
--- a/docs/topics/testing/index.txt
+++ b/docs/topics/testing/index.txt
@@ -6,7 +6,6 @@ Testing in Django
    :hidden:
 
    overview
-   doctests
    advanced
 
 Automated testing is an extremely useful bug-killing tool for the modern
@@ -29,83 +28,13 @@ it should be doing.
 
 The best part is, it's really easy.
 
-Unit tests v. doctests
-======================
-
-There are two primary ways to write tests with Django, corresponding to the
-two test frameworks that ship in the Python standard library. The two
-frameworks are:
-
-* **Unit tests** -- tests that are expressed as methods on a Python class
-  that subclasses :class:`unittest.TestCase` or Django's customized
-  :class:`~django.test.TestCase`. For example::
-
-      import unittest
-
-      class MyFuncTestCase(unittest.TestCase):
-          def testBasic(self):
-              a = ['larry', 'curly', 'moe']
-              self.assertEqual(my_func(a, 0), 'larry')
-              self.assertEqual(my_func(a, 1), 'curly')
-
-* **Doctests** -- tests that are embedded in your functions' docstrings and
-  are written in a way that emulates a session of the Python interactive
-  interpreter. For example::
-
-      def my_func(a_list, idx):
-          """
-          >>> a = ['larry', 'curly', 'moe']
-          >>> my_func(a, 0)
-          'larry'
-          >>> my_func(a, 1)
-          'curly'
-          """
-          return a_list[idx]
-
-Which should I use?
--------------------
-
-Because Django supports both of the standard Python test frameworks, it's up to
-you and your tastes to decide which one to use. You can even decide to use
-*both*.
-
-For developers new to testing, however, this choice can seem confusing. Here,
-then, are a few key differences to help you decide which approach is right for
-you:
-
-* If you've been using Python for a while, :mod:`doctest` will probably feel
-  more "pythonic". It's designed to make writing tests as easy as possible,
-  so it requires no overhead of writing classes or methods. You simply put
-  tests in docstrings. This has the added advantage of serving as
-  documentation (and correct documentation, at that!). However, while
-  doctests are good for some simple example code, they are not very good if
-  you want to produce either high quality, comprehensive tests or high
-  quality documentation. Test failures are often difficult to debug
-  as it can be unclear exactly why the test failed. Thus, doctests should
-  generally be avoided and used primarily for documentation examples only.
-
-* The :mod:`unittest` framework will probably feel very familiar to
-  developers coming from Java. :mod:`unittest` is inspired by Java's JUnit,
-  so you'll feel at home with this method if you've used JUnit or any test
-  framework inspired by JUnit.
-
-* If you need to write a bunch of tests that share similar code, then
-  you'll appreciate the :mod:`unittest` framework's organization around
-  classes and methods. This makes it easy to abstract common tasks into
-  common methods. The framework also supports explicit setup and/or cleanup
-  routines, which give you a high level of control over the environment
-  in which your test cases are run.
-
-* If you're writing tests for Django itself, you should use :mod:`unittest`.
-
 Where to go from here
 =====================
 
-As unit tests are preferred in Django, we treat them in detail in the
+The preferred way to write tests in Django is using the :mod:`unittest` module
+built in to the Python standard library. This is covered in detail in the
 :doc:`overview` document.
 
-:doc:`doctests` describes Django-specific features when using doctests.
-
-You can also use any *other* Python test framework, Django provides an API and
+You can also use any *other* Python test framework; Django provides an API and
 tools for that kind of integration. They are described in the
 :ref:`other-testing-frameworks` section of :doc:`advanced`.
diff --git a/docs/topics/testing/overview.txt b/docs/topics/testing/overview.txt
index 9228a07b31..5023e099aa 100644
--- a/docs/topics/testing/overview.txt
+++ b/docs/topics/testing/overview.txt
@@ -17,7 +17,7 @@ Writing tests
 =============
 
 Django's unit tests use a Python standard library module: :mod:`unittest`. This
-module defines tests in class-based approach.
+module defines tests using a class-based approach.
 
 .. admonition:: unittest2
 
@@ -46,16 +46,6 @@ module defines tests in class-based approach.
 
 .. _unittest2: http://pypi.python.org/pypi/unittest2
 
-For a given Django application, the test runner looks for unit tests in two
-places:
-
-* The ``models.py`` file. The test runner looks for any subclass of
-  :class:`unittest.TestCase` in this module.
-
-* A file called ``tests.py`` in the application directory -- i.e., the
-  directory that holds ``models.py``. Again, the test runner looks for any
-  subclass of :class:`unittest.TestCase` in this module.
-
 Here is an example :class:`unittest.TestCase` subclass::
 
     from django.utils import unittest
@@ -71,21 +61,18 @@ Here is an example :class:`unittest.TestCase` subclass::
             self.assertEqual(self.lion.speak(), 'The lion says "roar"')
             self.assertEqual(self.cat.speak(), 'The cat says "meow"')
 
-When you :ref:`run your tests <running-tests>`, the default behavior of the test
-utility is to find all the test cases (that is, subclasses of
-:class:`unittest.TestCase`) in ``models.py`` and ``tests.py``, automatically
-build a test suite out of those test cases, and run that suite.
+When you :ref:`run your tests <running-tests>`, the default behavior of the
+test utility is to find all the test cases (that is, subclasses of
+:class:`unittest.TestCase`) in any file whose name begins with ``test``,
+automatically build a test suite out of those test cases, and run that suite.
 
-There is a second way to define the test suite for a module: if you define a
-function called ``suite()`` in either ``models.py`` or ``tests.py``, the
-Django test runner will use that function to construct the test suite for that
-module. This follows the `suggested organization`_ for unit tests. See the
-Python documentation for more details on how to construct a complex test
-suite.
+.. versionchanged:: 1.6
 
-For more details about :mod:`unittest`, see the Python documentation.
+   Previously, Django's default test runner only discovered tests in
+   ``tests.py`` and ``models.py`` files within a Python package listed in
+   :setting:`INSTALLED_APPS`.
 
-.. _suggested organization: http://docs.python.org/library/unittest.html#organizing-tests
+For more details about :mod:`unittest`, see the Python documentation.
 
 .. warning::
 
@@ -101,6 +88,7 @@ For more details about :mod:`unittest`, see the Python documentation.
 
 .. _running-tests:
 
+
 Running tests
 =============
 
@@ -109,46 +97,47 @@ your project's ``manage.py`` utility::
 
     $ ./manage.py test
 
-By default, this will run every test in every application in
-:setting:`INSTALLED_APPS`. If you only want to run tests for a particular
-application, add the application name to the command line. For example, if your
-:setting:`INSTALLED_APPS` contains ``'myproject.polls'`` and
-``'myproject.animals'``, you can run the ``myproject.animals`` unit tests alone
-with this command::
+Test discovery is based on the unittest module's `built-in test discovery`.  By
+default, this will discover tests in any file named "test*.py" under the
+current working directory.
 
-    $ ./manage.py test animals
+.. _built-in test discovery: http://docs.python.org/2/library/unittest.html#test-discovery
 
-Note that we used ``animals``, not ``myproject.animals``.
+You can specify particular tests to run by supplying any number of "test
+labels" to ``./manage.py test``. Each test label can be a full Python dotted
+path to a package, module, ``TestCase`` subclass, or test method. For instance::
 
-You can be even *more* specific by naming an individual test case. To
-run a single test case in an application (for example, the
-``AnimalTestCase`` described in the "Writing unit tests" section), add
-the name of the test case to the label on the command line::
+    # Run all the tests in the animals.tests module
+    $ ./manage.py test animals.tests
 
-    $ ./manage.py test animals.AnimalTestCase
+    # Run all the tests found within the 'animals' package
+    $ ./manage.py test animals
 
-And it gets even more granular than that! To run a *single* test
-method inside a test case, add the name of the test method to the
-label::
+    # Run just one test case
+    $ ./manage.py test animals.tests.AnimalTestCase
 
-    $ ./manage.py test animals.AnimalTestCase.test_animals_can_speak
+    # Run just one test method
+    $ ./manage.py test animals.tests.AnimalTestCase.test_animals_can_speak
 
-You can use the same rules if you're using doctests. Django will use the
-test label as a path to the test method or class that you want to run.
-If your ``models.py`` or ``tests.py`` has a function with a doctest, or
-class with a class-level doctest, you can invoke that test by appending the
-name of the test method or class to the label::
+You can also provide a path to a directory to discover tests below that
+directory::
 
-    $ ./manage.py test animals.classify
+    $ ./manage.py test animals/
 
-If you want to run the doctest for a specific method in a class, add the
-name of the method to the label::
+You can specify a custom filename pattern match using the ``-p`` (or
+``--pattern``) option, if your test files are named differently from the
+``test*.py`` pattern::
 
-    $ ./manage.py test animals.Classifier.run
+    $ ./manage.py test --pattern="tests_*.py"
 
-If you're using a ``__test__`` dictionary to specify doctests for a
-module, Django will use the label as a key in the ``__test__`` dictionary
-for defined in ``models.py`` and ``tests.py``.
+.. versionchanged:: 1.6
+
+   Previously, test labels were in the form ``applabel``,
+   ``applabel.TestCase``, or ``applabel.TestCase.test_method``, rather than
+   being true Python dotted paths, and tests could only be found within
+   ``tests.py`` or ``models.py`` files within a Python package listed in
+   :setting:`INSTALLED_APPS`. The ``--pattern`` option and file paths as test
+   labels are new in 1.6.
 
 If you press ``Ctrl-C`` while the tests are running, the test runner will
 wait for the currently running test to complete and then exit gracefully.
@@ -173,6 +162,7 @@ be reported, and any test databases created by the run will not be destroyed.
     flag areas in your code that aren't strictly wrong but could benefit
     from a better implementation.
 
+
 .. _the-test-database:
 
 The test database