Fixed #24522 -- Added a --shuffle option to DiscoverRunner.

5 files changed, 45 insertions, 9 deletions

diff --git a/docs/internals/contributing/writing-code/unit-tests.txt b/docs/internals/contributing/writing-code/unit-tests.txt
index d2d4044e7e..1d38d4f429 100644
--- a/docs/internals/contributing/writing-code/unit-tests.txt
+++ b/docs/internals/contributing/writing-code/unit-tests.txt
@@ -470,12 +470,13 @@ the first one:
 
     $ ./runtests.py --pair basic.tests.ModelTest.test_eq queries transactions
 
-You can also try running any set of tests in reverse using the ``--reverse``
-option in order to verify that executing tests in a different order does not
-cause any trouble:
+You can also try running any set of tests in a random or reverse order using
+the ``--shuffle`` and ``--reverse`` options. This can help verify that
+executing tests in a different order does not cause any trouble:
 
 .. console::
 
+    $ ./runtests.py basic --shuffle
     $ ./runtests.py basic --reverse
 
 Seeing the SQL queries run during a test
diff --git a/docs/ref/django-admin.txt b/docs/ref/django-admin.txt
index 078f019de3..89d945500d 100644
--- a/docs/ref/django-admin.txt
+++ b/docs/ref/django-admin.txt
@@ -1425,11 +1425,30 @@ subsequent run. Unless the :setting:`MIGRATE <TEST_MIGRATE>` test setting is
 ``False``, any unapplied migrations will also be applied to the test database
 before running the test suite.
 
+.. django-admin-option:: --shuffle [SEED]
+
+.. versionadded:: 4.0
+
+Randomizes the order of tests before running them. This can help detect tests
+that aren't properly isolated. The test order generated by this option is a
+deterministic function of the integer seed given. When no seed is passed, a
+seed is chosen randomly and printed to the console. To repeat a particular test
+order, pass a seed. The test orders generated by this option preserve Django's
+:ref:`guarantees on test order <order-of-tests>`. They also keep tests grouped
+by test case class.
+
+The shuffled orderings also have a special consistency property useful when
+narrowing down isolation issues. Namely, for a given seed and when running a
+subset of tests, the new order will be the original shuffling restricted to the
+smaller set. Similarly, when adding tests while keeping the seed the same, the
+order of the original tests will be the same in the new order.
+
 .. django-admin-option:: --reverse, -r
 
 Sorts test cases in the opposite execution order. This may help in debugging
 the side effects of tests that aren't properly isolated. :ref:`Grouping by test
-class <order-of-tests>` is preserved when using this option.
+class <order-of-tests>` is preserved when using this option. This can be used
+in conjunction with ``--shuffle`` to reverse the order for a particular seed.
 
 .. django-admin-option:: --debug-mode
 
diff --git a/docs/releases/4.0.txt b/docs/releases/4.0.txt
index 4c0efc236f..3ef008e5fe 100644
--- a/docs/releases/4.0.txt
+++ b/docs/releases/4.0.txt
@@ -325,6 +325,9 @@ Tests
 * The new :meth:`.DiscoverRunner.log` method allows customizing the way
   messages are logged.
 
+* Django test runner now supports a :option:`--shuffle <test --shuffle>` option
+  to execute tests in a random order.
+
 URLs
 ~~~~
 
diff --git a/docs/topics/testing/advanced.txt b/docs/topics/testing/advanced.txt
index a8a63e7b57..dcc1be7f44 100644
--- a/docs/topics/testing/advanced.txt
+++ b/docs/topics/testing/advanced.txt
@@ -510,7 +510,7 @@ behavior. This class defines the ``run_tests()`` entry point, plus a
 selection of other methods that are used by ``run_tests()`` to set up, execute
 and tear down the test suite.
 
-.. class:: DiscoverRunner(pattern='test*.py', top_level=None, verbosity=1, interactive=True, failfast=False, keepdb=False, reverse=False, debug_mode=False, debug_sql=False, parallel=0, tags=None, exclude_tags=None, test_name_patterns=None, pdb=False, buffer=False, enable_faulthandler=True, timing=True, **kwargs)
+.. class:: DiscoverRunner(pattern='test*.py', top_level=None, verbosity=1, interactive=True, failfast=False, keepdb=False, reverse=False, debug_mode=False, debug_sql=False, parallel=0, tags=None, exclude_tags=None, test_name_patterns=None, pdb=False, buffer=False, enable_faulthandler=True, timing=True, shuffle=False, **kwargs)
 
     ``DiscoverRunner`` will search for tests in any file matching ``pattern``.
 
@@ -539,7 +539,8 @@ and tear down the test suite.
     If ``reverse`` is ``True``, test cases will be executed in the opposite
     order. This could be useful to debug tests that aren't properly isolated
     and have side effects. :ref:`Grouping by test class <order-of-tests>` is
-    preserved when using this option.
+    preserved when using this option. This option can be used in conjunction
+    with ``--shuffle`` to reverse the order for a particular random seed.
 
     ``debug_mode`` specifies what the :setting:`DEBUG` setting should be
     set to prior to running tests.
@@ -576,6 +577,14 @@ and tear down the test suite.
     If ``timing`` is ``True``, test timings, including database setup and total
     run time, will be shown.
 
+    If ``shuffle`` is an integer, test cases will be shuffled in a random order
+    prior to execution, using the integer as a random seed. If ``shuffle`` is
+    ``None``, the seed will be generated randomly. In both cases, the seed will
+    be logged to the console and set to ``self.shuffle_seed`` prior to running
+    tests. This option can be used to help detect tests that aren't properly
+    isolated. :ref:`Grouping by test class <order-of-tests>` is preserved when
+    using this option.
+
     Django may, from time to time, extend the capabilities of the test runner
     by adding new arguments. The ``**kwargs`` declaration allows for this
     expansion. If you subclass ``DiscoverRunner`` or write your own test
@@ -590,6 +599,10 @@ and tear down the test suite.
 
         The ``enable_faulthandler`` and ``timing`` arguments were added.
 
+    .. versionadded:: 4.0
+
+        The ``shuffle`` argument was added.
+
 Attributes
 ~~~~~~~~~~
 
diff --git a/docs/topics/testing/overview.txt b/docs/topics/testing/overview.txt
index 7461fdaaf0..cc55224761 100644
--- a/docs/topics/testing/overview.txt
+++ b/docs/topics/testing/overview.txt
@@ -235,9 +235,9 @@ the Django test runner reorders tests in the following way:
     for quicker feedback. This includes things like test modules that couldn't
     be found or that couldn't be loaded due to syntax errors.
 
-You may reverse the execution order inside groups using the :option:`test
---reverse` option. This can help with ensuring your tests are independent from
-each other.
+You may randomize and/or reverse the execution order inside groups using the
+:option:`test --shuffle` and :option:`--reverse <test --reverse>` options. This
+can help with ensuring your tests are independent from each other.
 
 .. versionchanged:: 4.0