Fixed #1142 -- Added multiple database support.

This monster of a patch is the result of Alex Gaynor's 2009 Google Summer of Code project.
Congratulations to Alex for a job well done.

Big thanks also go to:
 * Justin Bronn for keeping GIS in line with the changes,
 * Karen Tracey and Jani Tiainen for their help testing Oracle support
 * Brett Hoerner, Jon Loyens, and Craig Kimmerer for their feedback.
 * Malcolm Treddinick for his guidance during the GSoC submission process.
 * Simon Willison for driving the original design process
 * Cal Henderson for complaining about ponies he wanted.

... and everyone else too numerous to mention that helped to bring this feature into fruition.

git-svn-id: http://code.djangoproject.com/svn/django/trunk@11952 bcc190cf-cafb-0310-a4f2-bffc1f526a37

6 files changed, 387 insertions, 51 deletions

diff --git a/docs/topics/db/index.txt b/docs/topics/db/index.txt
index 949f4e87e9..bf918eba6b 100644
--- a/docs/topics/db/index.txt
+++ b/docs/topics/db/index.txt
@@ -16,3 +16,4 @@ model maps to a single database table.
    managers
    sql
    transactions
+   multi-db
diff --git a/docs/topics/db/multi-db.txt b/docs/topics/db/multi-db.txt
new file mode 100644
index 0000000000..2e848f3430
--- /dev/null
+++ b/docs/topics/db/multi-db.txt
@@ -0,0 +1,265 @@
+.. _topics-db-multi-db:
+
+==================
+Multiple Databases
+==================
+
+.. versionadded:: 1.2
+
+This topic guide describes Django's support for interacting with multiple
+databases. Most of the rest of Django's documentation assumes you are
+interacting with a single database. If you want to interact with multiple
+databases, some additional steps must be taken.
+
+Defining your databases
+=======================
+
+The first step to using more than one database with Django is to tell
+Django about the database servers you'll be using. This is done using
+the :setting:`DATABASES` setting. This setting maps database aliases,
+which are a way to refer to a specific database throughout Django, to
+a dictionary of settings for that specific connection. The settings in
+the inner dictionaries are described fully in the :setting:`DATABASES`
+documentation.
+
+Regardless of how many databases you have, you *must* have a database
+named ``'default'``. Any additional databases you have can have
+whatever alias you choose.
+
+The following is an example ``settings.py`` snippet defining two
+databases - a default Postgres database, and a MySQL database called
+``users``::
+
+    DATABASES = {
+        'default': {
+            'NAME': 'app_data',
+            'BACKEND': 'django.db.backends.postgres_psycopg2',
+            'USER': 'postgres_user',
+            'PASSWORD': 's3krit'
+        },
+        'users': {
+            'NAME': 'user_data'
+            'BACKEND': 'django.db.backends.mysql',
+            'USER': 'mysql_user',
+            'PASSWORD': 'priv4te'
+        }
+    }
+
+If you attempt to access a database that you haven't defined in your
+:setting:`DATABASES` setting then Django will raise a
+``django.db.utils.ConnectionDoesNotExist`` exception.
+
+Selecting a database for a ``QuerySet``
+=======================================
+
+It is possible to select the database for a ``QuerySet`` at any point
+during it's construction. To choose the database that a query will be
+preformed against simply call the ``using()`` method on the
+``QuerySet``. ``using()`` takes a single argument: the alias of the
+database on which you want to run the query. For example::
+
+    # This will run on the 'default' database...
+    >>> Author.objects.all()
+    # So will this...
+    >>> Author.objects.using('default').all()
+    # This will run on the 'other' database
+    >>> Author.objects.using('other').all()
+
+Select a database to save to
+============================
+
+To choose what database to save a model to, provide a ``using`` keyword
+argument to ``Model.save()``. For example if you had a user model that you
+wanted to save to the ``'legacy_users'`` database you would save the user
+by calling::
+
+    >>> user_obj.save(using='legacy_users')
+
+Moving an object from one database to another
+---------------------------------------------
+
+If you have saved an instance to one database, it might be tempting to use
+``save(using=...)`` as a way to migrate the instance to a new database. However,
+if you don't take appropriate steps, this could have some unexpected consequences.
+
+Consider the following example::
+
+    >>> p = Person(name='Fred')
+    >>> p.save(using='first') # (1)
+    # some other processing ...
+    >>> p.save(using='second') # (2)
+
+In statement 1, a new Person object is saved to the ``first``
+database. At this time, ``p`` doesn't have a primary key, so Django
+issues a SQL ``INSERT`` statement. This creates a primary key, and
+Django assigns that primary key to ``p``.
+
+When the save occurs in statement 2, ``p`` already has a primary key
+value, and Django will attempt to use that primary key on the new
+database. If the primary key value isn't in use in the ``second``
+database, then you won't have any problems -- the object will be
+copied to the new databse.
+
+However, if the primary key of ``p`` is already in use on the
+``second`` database, the existing object on the ``second`` database
+will be lost when ``p`` is saved.
+
+There are two ways to avoid this outcome. Firstly, you can clear the
+primary key of the instance. If an object has no primary key, Django
+will treat it as a new object, avoiding any loss of data on the
+``second`` database::
+
+    >>> p = Person(name='Fred')
+    >>> p.save(using='first')
+    # some other processing ...
+    >>> p.pk = None # Clear the PK
+    >>> p.save(using='second') # Write a completely new object
+
+Secondly, you can use the ``force_insert`` option to ``save()`` to ensure that
+Django does a SQL ``INSERT``::
+
+    >>> p = Person(name='Fred')
+    >>> p.save(using='first')
+    # some other processing ...
+    >>> p.save(using='second', force_insert=True)
+
+This will ensure that the person named ``Fred`` will have the same
+primary key on both databases. If that primary key is already in use
+when you try to save onto the ``second`` database, an error will be
+raised.
+
+Select a database to delete from
+================================
+
+By default, a call to delete an existing object will be executed on the
+same database that was used to retrieve the object in the first place::
+
+    >>> user_obj = User.objects.using('legacy_users').get(username='fred')
+    >>> user_obj.delete() # will delete from the `legacy_users` database
+
+If you want to specify the database from which a model will be
+deleted, you can use a ``using`` keyword argument to the
+``Model.delete()`` method. This argument is analogous to the ``using``
+keyword argument to ``save()``. For example if you were migrating a
+user from the ``'legacy_users'`` database to the ``'new_users'``
+database you might use the commands::
+
+    >>> user_obj.save(using='new_users')
+    >>> user_obj.delete(using='legacy_users')
+
+Using ``Managers`` with multiple databases
+==========================================
+
+When you call ``using()`` Django returns a ``QuerySet`` that will be
+evaluated against that database. However, sometimes you want to direct
+a manager to use a specific database chain ``using()``. If you call
+``using()``, you won't have access to any of the methods on the
+manager.
+
+To overcome this limitation, managers provide a ``db_manager()``
+method. This method returns a copy of the *manager* bound to that
+specific database. So, if you want to load an object using it's
+natural key (using the ``get_by_natural_key()`` method on the manager,
+you can call::
+
+    >>> Book.objects.db_mamanger("other").get_by_natural_key(...)
+
+If you are overriding ``get_query_set()`` on your manager you must be sure to
+either, call the method on the parent (using ``super()``), or do the
+appropriate handling of the ``_db`` attribute on the manager. For example if
+you wanted to return a custom ``QuerySet`` class from the ``get_query_set``
+method you could do this::
+
+    class MyManager(models.Manager):
+        ...
+        def get_query_set(self):
+            qs = CustomQuerySet(self.model)
+            if self._db is not None:
+                qs = qs.using(self._db)
+            return qs
+
+Exposing multiple databases in Django's admin interface
+=======================================================
+
+Django's admin doesn't have any explicit support for multiple
+databases. If you want to provide an admin interface for a model on a
+database other than ``default``, you need to write custom
+:class:`~django.contrib.admin.ModelAdmin` classes that will direct the
+admin to use a specific database for content.
+
+There are four methods that require customization on a ModelAdmin
+object::
+
+    class MultiDBModelAdmin(admin.ModelAdmin):
+        # A handy constant for the name of the alternate database
+        using = 'other'
+
+        def save_model(self, request, obj, form, change):
+            # Tell Django to save objects to the 'other' database
+            obj.save(using=self.using)
+
+        def queryset(self, request):
+            # Tell Django to look for objects on the 'other' database
+            return super(MultiDBModelAdmin, self).queryset(request).using(self.using)
+
+        def formfield_for_foreignkey(self, db_field, request=None, **kwargs):
+            # Tell Django to populate ForeignKey widgets using a query
+            # on the 'other' database
+            return super(MultiDBModelAdmin, self).formfield_for_foreignkey(db_field, request=request, using=self.using, **kwargs)
+
+        def formfield_for_manytomany(self, db_field, request=None, **kwargs):
+            # Tell Django to populate ManyToMany widgets using a query
+            # on the 'other' database
+            return super(MultiDBModelAdmin, self).formfield_for_manytomany(db_field, request=request, using=self.using, **kwargs)
+
+The implementation provided here implements a multi-db strategy where
+all objects of a given type are stored on a specific database (e.g.,
+all ``User`` objects are on the ``other`` database). If your usage of
+multi-db is more complex, your ModelAdmin will need to reflect that
+strategy.
+
+Inlines can be handled in a similar fashion -- they require just three
+customized methods::
+
+    class MultiDBTabularInline(admin.TabularInline):
+        using = 'other'
+
+        def queryset(self, request):
+            # Tell Django to look for inline objects on the 'other' database
+            return super(MultiDBTabularInline, self).queryset(request).using(self.using)
+
+        def formfield_for_foreignkey(self, db_field, request=None, **kwargs):
+            # Tell Django to populate ForeignKey widgets using a query
+            # on the 'other' database
+            return super(MultiDBTabularInline, self).formfield_for_foreignkey(db_field, request=request, using=self.using, **kwargs)
+
+        def formfield_for_manytomany(self, db_field, request=None, **kwargs):
+            # Tell Django to populate ManyToMany widgets using a query
+            # on the 'other' database
+            return super(MultiDBTabularInline, self).formfield_for_manytomany(db_field, request=request, using=self.using, **kwargs)
+
+Once you have written your model admin definitions, they can be
+registered with any Admin instance::
+
+    from django.contrib import admin
+
+    # Specialize the multi-db admin objects for use with specific models
+    class BookInline(MultiDBTabularInline):
+        model = Book
+
+    class PublisherAdmin(MultiDBModelAdmin):
+        inlines = [BookInline]
+
+        admin.site.register
+
+    admin.site.register(Author, MultiDBModelAdmin)
+    admin.site.register(Publisher, PublisherAdmin)
+
+    othersite = admin.Site('othersite')
+    othersite.register(Publisher, MultiDBModelAdmin)
+
+This example sets up two admin sites. On the first site, the
+``Author`` and ``Publisher`` objects are exposed; ``Publisher``
+objects have an tabular inline showing books published by that
+publisher. The second site exposes just publishers, without the
+inlines.
diff --git a/docs/topics/db/sql.txt b/docs/topics/db/sql.txt
index 45aa4f950e..987fcb091f 100644
--- a/docs/topics/db/sql.txt
+++ b/docs/topics/db/sql.txt
@@ -23,7 +23,7 @@ Performing raw queries
 The ``raw()`` manager method can be used to perform raw SQL queries that
 return model instances:
 
-.. method:: Manager.raw(query, params=None, translations=None)
+.. method:: Manager.raw(raw_query, params=None, translations=None)
 
 This method method takes a raw SQL query, executes it, and returns model
 instances.
diff --git a/docs/topics/db/transactions.txt b/docs/topics/db/transactions.txt
index b374609ab9..8902f6506c 100644
--- a/docs/topics/db/transactions.txt
+++ b/docs/topics/db/transactions.txt
@@ -56,7 +56,10 @@ Controlling transaction management in views
 For most people, implicit request-based transactions work wonderfully. However,
 if you need more fine-grained control over how transactions are managed, you
 can use Python decorators to change the way transactions are handled by a
-particular view function.
+particular view function.  All of the decorators take an option ``using``
+parameter which should be the alias for a database connection for which the
+behavior applies to.  If no alias is specified then the ``"default"`` database
+is used.
 
 .. note::
 
@@ -79,9 +82,14 @@ Example::
     def viewfunc(request):
         ....
 
+    @transaction.autocommit(using="my_other_database")
+    def viewfunc2(request):
+        ....
+
 Within ``viewfunc()``, transactions will be committed as soon as you call
 ``model.save()``, ``model.delete()``, or any other function that writes to the
-database.
+database.  ``viewfunc2()`` will have this same behavior, but for the
+``"my_other_database"`` connection.
 
 ``django.db.transaction.commit_on_success``
 -------------------------------------------
@@ -95,6 +103,10 @@ all the work done in a function::
     def viewfunc(request):
         ....
 
+    @transaction.commit_on_success(using="my_other_database")
+    def viewfunc2(request):
+        ....
+
 If the function returns successfully, then Django will commit all work done
 within the function at that point. If the function raises an exception, though,
 Django will roll back the transaction.
@@ -127,6 +139,10 @@ Manual transaction management looks like this::
         else:
             transaction.commit()
 
+    @transaction.commit_manually(using="my_other_database")
+    def viewfunc2(request):
+        ....
+
 .. admonition:: An important note to users of earlier Django releases:
 
     The database ``connection.commit()`` and ``connection.rollback()`` methods
@@ -169,21 +185,25 @@ issue a rollback, the entire transaction is rolled back. Savepoints provide
 the ability to perform a fine-grained rollback, rather than the full rollback
 that would be performed by ``transaction.rollback()``.
 
+Each of these functions takes a ``using`` argument which should be the name of
+a database for which the behavior applies.  If no ``using`` argument is
+provided then the ``"default"`` database is used.
+
 Savepoints are controlled by three methods on the transaction object:
 
-.. method:: transaction.savepoint()
+.. method:: transaction.savepoint(using=None)
 
     Creates a new savepoint. This marks a point in the transaction that
     is known to be in a "good" state.
 
     Returns the savepoint ID (sid).
 
-.. method:: transaction.savepoint_commit(sid)
+.. method:: transaction.savepoint_commit(sid, using=None)
 
     Updates the savepoint to include any operations that have been performed
     since the savepoint was created, or since the last commit.
 
-.. method:: transaction.savepoint_rollback(sid)
+.. method:: transaction.savepoint_rollback(sid, using=None)
 
     Rolls the transaction back to the last point at which the savepoint was
     committed.
diff --git a/docs/topics/http/sessions.txt b/docs/topics/http/sessions.txt
index 3b1429b440..cfe53c514b 100644
--- a/docs/topics/http/sessions.txt
+++ b/docs/topics/http/sessions.txt
@@ -24,14 +24,6 @@ To enable session functionality, do the following:
       The default ``settings.py`` created by ``django-admin.py startproject`` has
       ``SessionMiddleware`` activated.
 
-    * Add ``'django.contrib.sessions'`` to your ``INSTALLED_APPS`` setting,
-      and run ``manage.py syncdb`` to install the single database table
-      that stores session data.
-
-.. versionchanged:: 1.0
-   This step is optional if you're not using the database session backend;
-   see `configuring the session engine`_.
-
 If you don't want to use sessions, you might as well remove the
 ``SessionMiddleware`` line from ``MIDDLEWARE_CLASSES`` and ``'django.contrib.sessions'``
 from your ``INSTALLED_APPS``. It'll save you a small bit of overhead.
@@ -46,6 +38,22 @@ By default, Django stores sessions in your database (using the model
 some setups it's faster to store session data elsewhere, so Django can be
 configured to store session data on your filesystem or in your cache.
 
+Using database-backed sessions
+------------------------------
+
+If you want to use a database-backed session, you need to add
+``'django.contrib.sessions'`` to your ``INSTALLED_APPS`` setting.
+
+If you want to store your session data on a database other than ``default``
+alias, you should set the :setting:`SESSION_DB_ALIAS` setting.
+
+Once you have configured your installation, run ``manage.py syncdb``
+to install the single database table that stores session data.
+
+.. versionadded:: 1.2
+   The :setting:`SESSION_DB_ALIAS` setting was added in Django 1.2. It
+   is not required in earlier versions.
+
 Using cached sessions
 ---------------------
 
@@ -86,6 +94,9 @@ disregards persistence. In most cases, the ``cached_db`` backend will be fast
 enough, but if you need that last bit of performance, and are willing to let
 session data be expunged from time to time, the ``cache`` backend is for you.
 
+If you use the ``cached_db`` session backend, you also need to follow the
+configuration instructions for the `using database-backed sessions`_.
+
 Using file-based sessions
 -------------------------
 
@@ -97,6 +108,7 @@ to output from ``tempfile.gettempdir()``, most likely ``/tmp``) to control
 where Django stores session files. Be sure to check that your Web server has
 permissions to read and write to this location.
 
+
 Using sessions in views
 =======================
 
diff --git a/docs/topics/testing.txt b/docs/topics/testing.txt
index a9ba66ece0..45f7f49999 100644
--- a/docs/topics/testing.txt
+++ b/docs/topics/testing.txt
@@ -278,33 +278,35 @@ The test database
 -----------------
 
 Tests that require a database (namely, model tests) will not use your "real"
-(production) database. A separate, blank database is created for the tests.
+(production) database. Separate, blank databases are created for the tests.
 
-Regardless of whether the tests pass or fail, the test database is destroyed
+Regardless of whether the tests pass or fail, the test databases are destroyed
 when all the tests have been executed.
 
-By default this test database gets its name by prepending ``test_`` to the
-value of the :setting:`DATABASE_NAME` setting. When using the SQLite database
-engine the tests will by default use an in-memory database (i.e., the database
-will be created in memory, bypassing the filesystem entirely!). If you want to
-use a different database name, specify the :setting:`TEST_DATABASE_NAME`
-setting.
+By default the test databases get their names by prepending ``test_``
+to the value of the :setting:`NAME`` settings for the databased
+defined in :setting:`DATABASES`. When using the SQLite database engine
+the tests will by default use an in-memory database (i.e., the
+database will be created in memory, bypassing the filesystem
+entirely!). If you want to use a different database name, specify
+``TEST_NAME`` in the dictionary for any given database in
+:setting:`DATABASES`.
 
-Aside from using a separate database, the test runner will otherwise use all of
-the same database settings you have in your settings file:
-:setting:`DATABASE_ENGINE`, :setting:`DATABASE_USER`, :setting:`DATABASE_HOST`,
-etc. The test database is created by the user specified by
-:setting:`DATABASE_USER`, so you'll need to make sure that the given user
-account has sufficient privileges to create a new database on the system.
+Aside from using a separate database, the test runner will otherwise
+use all of the same database settings you have in your settings file:
+:setting:`ENGINE`, :setting:`USER`, :setting:`HOST`, etc. The test
+database is created by the user specified by ``USER``, so you'll need
+to make sure that the given user account has sufficient privileges to
+create a new database on the system.
 
 .. versionadded:: 1.0
 
-For fine-grained control over the
-character encoding of your test database, use the
-:setting:`TEST_DATABASE_CHARSET` setting. If you're using MySQL, you can also
-use the :setting:`TEST_DATABASE_COLLATION` setting to control the particular
-collation used by the test database. See the :ref:`settings documentation
-<ref-settings>` for details of these advanced settings.
+For fine-grained control over the character encoding of your test
+database, use the :setting:`TEST_CHARSET` option. If you're using
+MySQL, you can also use the :setting:`TEST_COLLATION` option to
+control the particular collation used by the test database. See the
+:ref:`settings documentation <ref-settings>` for details of these
+advanced settings.
 
 Other test conditions
 ---------------------
@@ -1037,6 +1039,39 @@ URLconf for the duration of the test case.
 
 .. _emptying-test-outbox:
 
+Multi-database support
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. attribute:: TestCase.multi_db
+
+.. versionadded:: 1.2
+
+Django sets up a test database corresponding to every database that is
+defined in the :setting:``DATABASES`` definition in your settings
+file. However, a big part of the time taken to run a Django TestCase
+is consumed by the call to ``flush`` that ensures that you have a
+clean database at the start of each test run. If you have multiple
+databases, multiple flushes are required (one for each database),
+which can be a time consuming activity -- especially if your tests
+don't need to test multi-database activity.
+
+As an optimization, Django only flushes the ``default`` database at
+the start of each test run. If your setup contains multiple databases,
+and you have a test that requires every database to be clean, you can
+use the ``multi_db`` attribute on the test suite to request a full
+flush.
+
+For example::
+
+    class TestMyViews(TestCase):
+        multi_db = True
+
+        def testIndexPageView(self):
+            call_some_test_code()
+
+This test case will flush *all* the test databases before running
+``testIndexPageView``.
+
 Emptying the test outbox
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -1251,16 +1286,17 @@ utility methods in the ``django.test.utils`` module.
 .. function:: setup_test_environment()
 
     Performs any global pre-test setup, such as the installing the
-    instrumentation of the template rendering system and setting up the dummy
-    ``SMTPConnection``.
+    instrumentation of the template rendering system and setting up
+    the dummy ``SMTPConnection``.
 
 .. function:: teardown_test_environment()
 
-    Performs any global post-test teardown, such as removing the black magic
-    hooks into the template system and restoring normal e-mail services.
+    Performs any global post-test teardown, such as removing the black
+    magic hooks into the template system and restoring normal e-mail
+    services.
 
-The creation module of the database backend (``connection.creation``) also
-provides some utilities that can be useful during testing.
+The creation module of the database backend (``connection.creation``)
+also provides some utilities that can be useful during testing.
 
 .. function:: create_test_db(verbosity=1, autoclobber=False)
 
@@ -1268,27 +1304,29 @@ provides some utilities that can be useful during testing.
 
     ``verbosity`` has the same behavior as in ``run_tests()``.
 
-    ``autoclobber`` describes the behavior that will occur if a database with
-    the same name as the test database is discovered:
+    ``autoclobber`` describes the behavior that will occur if a
+    database with the same name as the test database is discovered:
 
-        * If ``autoclobber`` is ``False``, the user will be asked to approve
-          destroying the existing database. ``sys.exit`` is called if the user
-          does not approve.
+        * If ``autoclobber`` is ``False``, the user will be asked to
+          approve destroying the existing database. ``sys.exit`` is
+          called if the user does not approve.
 
-        * If autoclobber is ``True``, the database will be destroyed without
-          consulting the user.
+        * If autoclobber is ``True``, the database will be destroyed
+          without consulting the user.
 
     Returns the name of the test database that it created.
 
-    ``create_test_db()`` has the side effect of modifying
-    ``settings.DATABASE_NAME`` to match the name of the test database.
+    ``create_test_db()`` has the side effect of modifying the value of
+    :setting:`NAME` in :setting:`DATABASES` to match the name of the test
+    database.
 
     .. versionchanged:: 1.0
        ``create_test_db()`` now returns the name of the test database.
 
 .. function:: destroy_test_db(old_database_name, verbosity=1)
 
-    Destroys the database whose name is in the :setting:`DATABASE_NAME` setting
-    and restores the value of :setting:`DATABASE_NAME` to the provided name.
+    Destroys the database whose name is in stored in :setting:`NAME` in the
+    :setting:`DATABASES`, and sets :setting:`NAME` to use the
+    provided name.
 
     ``verbosity`` has the same behavior as in ``run_tests()``.