File storage refactoring, adding far more flexibility to Django's file handling. The new files.txt document has details of the new features.

This is a backwards-incompatible change; consult BackwardsIncompatibleChanges for details.

Fixes #3567, #3621, #4345, #5361, #5655, #7415.

Many thanks to Marty Alchin who did the vast majority of this work.


git-svn-id: http://code.djangoproject.com/svn/django/trunk@8244 bcc190cf-cafb-0310-a4f2-bffc1f526a37

6 files changed, 500 insertions, 63 deletions

diff --git a/docs/custom_model_fields.txt b/docs/custom_model_fields.txt
index 6b8f3c3ac6..a45e876fc6 100644
--- a/docs/custom_model_fields.txt
+++ b/docs/custom_model_fields.txt
@@ -596,3 +596,42 @@ smoothly:
        instance, not a ``HandField``). So if your ``__unicode__()`` method
        automatically converts to the string form of your Python object, you can
        save yourself a lot of work.
+
+Writing a ``FileField`` subclass
+=================================
+
+In addition to the above methods, fields that deal with files have a few other
+special requirements which must be taken into account. The majority of the
+mechanics provided by ``FileField``, such as controlling database storage and
+retrieval, can remain unchanged, leaving subclasses to deal with the challenge
+of supporting a particular type of file.
+
+Django provides a ``File`` class, which is used as a proxy to the file's
+contents and operations. This can be subclassed to customzie hwo the file is
+accessed, and what methods are available. It lives at
+``django.db.models.fields.files``, and its default behavior is explained in the
+`file documentation`_.
+
+Once a subclass of ``File`` is created, the new ``FileField`` subclass must be
+told to use it. To do so, simply assign the new ``File`` subclass to the special
+``attr_class`` attribute of the ``FileField`` subclass.
+
+.. _file documentation: ../files/
+
+A few suggestions
+------------------
+
+In addition to the above details, there are a few guidelines which can greatly
+improve the efficiency and readability of the field's code.
+
+    1. The source for Django's own ``ImageField`` (in
+       ``django/db/models/fields/files.py``) is a great example of how to
+       subclass ``FileField`` to support a particular type of file, as it
+       incorporates all of the techniques described above.
+
+    2. Cache file attributes wherever possible. Since files may be stored in
+       remote storage systems, retrieving them may cost extra time, or even
+       money, that isn't always necessary. Once a file is retrieved to obtain
+       some data about its content, cache as much of that data as possible to
+       reduce the number of times the file must be retrieved on subsequent
+       calls for that information.
diff --git a/docs/db-api.txt b/docs/db-api.txt
index 4f03a4810d..7e6406f334 100644
--- a/docs/db-api.txt
+++ b/docs/db-api.txt
@@ -2298,53 +2298,34 @@ For a full example, see the `lookup API sample model`_.
 get_FOO_filename()
 ------------------
 
-For every ``FileField``, the object will have a ``get_FOO_filename()`` method,
-where ``FOO`` is the name of the field. This returns the full filesystem path
-to the file, according to your ``MEDIA_ROOT`` setting.
-
-.. note::
-    It is only valid to call this method **after** saving the model when the
-    field has been set. Prior to saving, the value returned will not contain
-    the upload directory (the `upload_to` parameter) in the path.
-
-Note that ``ImageField`` is technically a subclass of ``FileField``, so every
-model with an ``ImageField`` will also get this method.
+**Deprecated in Django development version**; use ``object.FOO.name`` instead.
+See `managing files`_ for details.
 
 get_FOO_url()
 -------------
 
-For every ``FileField``, the object will have a ``get_FOO_url()`` method,
-where ``FOO`` is the name of the field. This returns the full URL to the file,
-according to your ``MEDIA_URL`` setting. If the value is blank, this method
-returns an empty string.
-
-.. note::
-    As with ``get_FOO_filename()``, it is only valid to call this method
-    **after** saving the model, otherwise an incorrect result will be
-    returned.
+**Deprecated in Django development version**; use ``object.FOO.url`` instead.
+See `managing files`_ for details.
 
 get_FOO_size()
 --------------
 
-For every ``FileField``, the object will have a ``get_FOO_size()`` method,
-where ``FOO`` is the name of the field. This returns the size of the file, in
-bytes. (Behind the scenes, it uses ``os.path.getsize``.)
+**Deprecated in Django development version**; use ``object.FOO.size`` instead.
+See `managing files`_ for details.
 
 save_FOO_file(filename, raw_contents)
 -------------------------------------
 
-For every ``FileField``, the object will have a ``save_FOO_file()`` method,
-where ``FOO`` is the name of the field. This saves the given file to the
-filesystem, using the given filename. If a file with the given filename already
-exists, Django adds an underscore to the end of the filename (but before the
-extension) until the filename is available.
+**Deprecated in Django development version**; use ``object.FOO.save()`` instead.
+See `managing files`_ for details.
 
 get_FOO_height() and get_FOO_width()
 ------------------------------------
 
-For every ``ImageField``, the object will have ``get_FOO_height()`` and
-``get_FOO_width()`` methods, where ``FOO`` is the name of the field. This
-returns the height (or width) of the image, as an integer, in pixels.
+**Deprecated in Django development version**; use ``object.FOO.width`` and
+``object.FOO.height`` instead. See `managing files`_ for details.
+
+.. _`managing files`: ../files/
 
 Shortcuts
 =========
diff --git a/docs/files.txt b/docs/files.txt
new file mode 100644
index 0000000000..9a8326806f
--- /dev/null
+++ b/docs/files.txt
@@ -0,0 +1,388 @@
+==============
+Managing files
+==============
+
+**New in Django development version**
+
+This document describes Django's file access APIs.
+
+By default, Django stores files locally, using the ``MEDIA_ROOT`` and
+``MEDIA_URL`` settings_. The examples below assume that you're using
+these defaults.
+
+However, Django provides ways to write custom `file storage systems`_ that
+allow you to completely customize where and how Django stores files. The
+second half of this document describes how these storage systems work.
+
+.. _file storage systems: `File storage`_
+.. _settings: ../settings/
+
+Using files in models
+=====================
+
+When you use a `FileField`_ or `ImageField`_, Django provides a set of APIs you can use to deal with that file.
+
+.. _filefield: ../model-api/#filefield
+.. _imagefield: ../model-api/#imagefield
+
+Consider the following model, using a ``FileField`` to store a photo::
+
+    class Car(models.Model):
+        name = models.CharField(max_length=255)
+        price = models.DecimalField(max_digits=5, decimal_places=2)
+        photo = models.ImageField(upload_to='cars')
+
+Any ``Car`` instance will have a ``photo`` attribute that you can use to get at
+the details of the attached photo::
+
+    >>> car = Car.object.get(name="57 Chevy")
+    >>> car.photo
+    <ImageFieldFile: chevy.jpg>
+    >>> car.photo.name
+    u'chevy.jpg'
+    >>> car.photo.path
+    u'/media/cars/chevy.jpg'
+    >>> car.photo.url
+    u'http://media.example.com/cars/chevy.jpg'
+    
+This object -- ``car.photo`` in the example -- is a ``File`` object, which means
+it has all the methods and attributes described below.
+
+The ``File`` object
+===================
+
+Internally, Django uses a ``django.core.files.File`` any time it needs to
+represent a file. This object is a thin wrapper around Python's `built-in file
+object`_ with some Django-specific additions.
+
+.. _built-in file object: http://docs.python.org/lib/bltin-file-objects.html
+
+Creating ``File`` instances
+---------------------------
+
+Most of the time you'll simply use a ``File`` that Django's given you (i.e. a
+file attached to an model as above, or perhaps an `uploaded file`_).
+
+.. _uploaded file: ../uploading_files/
+
+If you need to construct a ``File`` yourself, the easiest way is to create one
+using a Python built-in ``file`` object::
+
+    >>> from django.core.files import File
+
+    # Create a Python file object using open()
+    >>> f = open('/tmp/hello.world', 'w')
+    >>> myfile = File(f)
+    
+Now you can use any of the ``File`` attributes and methods defined below.
+    
+``File`` attributes and methods
+-------------------------------
+
+Django's ``File`` has the following attributes and methods:
+
+``File.path``
+~~~~~~~~~~~~~
+
+The absolute path to the file's location on a local filesystem. 
+
+Custom `file storage systems`_ may not store files locally; files stored on
+these systems will have a ``path`` of ``None``.
+
+``File.url``
+~~~~~~~~~~~~
+
+The URL where the file can be retrieved. This is often useful in templates_; for
+example, a bit of a template for displaying a ``Car`` (see above) might look
+like::
+
+    <img src='{{ car.photo.url }}' alt='{{ car.name }}' />
+
+.. _templates: ../templates/
+
+``File.size``
+~~~~~~~~~~~~~
+
+The size of the file in bytes.
+
+``File.open(mode=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Open or reopen the file (which by definition also does ``File.seek(0)``). The
+``mode`` argument allows the same values as Python's standard ``open()``.
+
+When reopening a file, ``mode`` will override whatever mode the file was
+originally opened with; ``None`` means to reopen with the original mode.
+
+``File.read(num_bytes=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Read content from the file. The optional ``size`` is the number of bytes to
+read; if not specified, the file will be read to the end.
+
+``File.__iter__()``
+~~~~~~~~~~~~~~~~~~~
+
+Iterate over the file yielding one line at a time.
+
+``File.chunks(chunk_size=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Iterate over the file yielding "chunks" of a given size. ``chunk_size`` defaults
+to 64 KB.
+
+This is especially useful with very large files since it allows them to be
+streamed off disk and avoids storing the whole file in memory.
+
+``File.multiple_chunks(chunk_size=None)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Returns ``True`` if the file is large enough to require multiple chunks to
+access all of its content give some ``chunk_size``.
+
+``File.write(content)``
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Writes the specified content string to the file. Depending on the storage system
+behind the scenes, this content might not be fully committed until ``close()``
+is called on the file.
+
+``File.close()``
+~~~~~~~~~~~~~~~~
+
+Close the file.
+
+.. TODO: document the rest of the File methods.
+
+Additional ``ImageField`` attributes
+------------------------------------
+
+``File.width`` and ``File.height``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These attributes provide the dimensions of the image. 
+
+Additional methods on files attached to objects
+-----------------------------------------------
+
+Any ``File`` that's associated with an object (as with ``Car.photo``, above)
+will also have a couple of extra methods:
+
+``File.save(name, content, save=True)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Saves a new file with the file name and contents provided. This will not replace
+the existing file, but will create a new file and update the object to point to
+it. If ``save`` is ``True``, the model's ``save()`` method will be called once
+the file is saved. That is, these two lines::
+
+    >>> car.photo.save('myphoto.jpg', contents, save=False)
+    >>> car.save()
+
+are the same as this one line::
+
+    >>> car.photo.save('myphoto.jpg', contents, save=True)
+
+``File.delete(save=True)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Remove the file from the model instance and delete the underlying file. The
+``save`` argument works as above.
+
+File storage
+============
+
+Behind the scenes, Django delegates decisions about how and where to store files
+to a file storage system. This is the object that actually understands things
+like file systems, opening and reading files, etc.
+
+Django's default file storage is given by the `DEFAULT_FILE_STORAGE setting`_;
+if you don't explicitly provide a storage system, this is the one that will be
+used. 
+
+.. _default_file_storage setting: ../settings/#default-file-storage
+
+The built-in filesystem storage class
+-------------------------------------
+
+Django ships with a built-in ``FileSystemStorage`` class (defined in
+``django.core.files.storage``) which implements basic local filesystem file
+storage. Its initializer takes two arguments:
+
+======================  ===================================================
+Argument                Description
+======================  ===================================================
+``location``            Optional. Absolute path to the directory that will
+                        hold the files. If omitted, it will be set to the
+                        value of your ``MEDIA_ROOT`` setting.
+``base_url``            Optional. URL that serves the files stored at this
+                        location. If omitted, it will default to the value
+                        of your ``MEDIA_URL`` setting.
+======================  ===================================================
+
+For example, the following code will store uploaded files under
+``/media/photos`` regardless of what your ``MEDIA_ROOT`` setting is::
+
+    from django.db import models
+    from django.core.files.storage import FileSystemStorage
+
+    fs = FileSystemStorage(base_url='/media/photos')
+
+    class Car(models.Model):
+        ...
+        photo = models.ImageField(storage=fs)
+
+`Custom storage systems`_ work the same way: you can pass them in as the
+``storage`` argument to a ``FileField``.
+
+.. _custom storage systems: `writing a custom storage system`_
+
+Storage objects
+---------------
+
+Though most of the time you'll want to use a ``File`` object (which delegates to
+the proper storage for that file), you can use file storage systems directly.
+You can create an instance of some custom file storage class, or -- often more
+useful -- you can use the global default storage system::
+
+    >>> from django.core.files.storage import default_storage
+
+    >>> path = default_storage.save('/path/to/file', 'new content')
+    >>> path
+    u'/path/to/file'
+
+    >>> default_storage.filesize(path)
+    11
+    >>> default_storage.open(path).read()
+    'new content'
+
+    >>> default_storage.delete(path)
+    >>> default_storage.exists(path)
+    False
+
+Storage objects define the following methods:
+
+``Storage.exists(name)``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+``True`` if a file exists given some ``name``.
+
+``Storge.path(name)``
+~~~~~~~~~~~~~~~~~~~~~
+
+The local filesystem path where the file can be opened using Python's standard
+``open()``. For storage systems that aren't accessible from the local
+filesystem, this will raise ``NotImplementedError`` instead.
+
+``Storage.size(name)``
+~~~~~~~~~~~~~~~~~~~~~~
+
+Returns the total size, in bytes, of the file referenced by ``name``.
+
+``Storage.url(name)``
+~~~~~~~~~~~~~~~~~~~~~
+
+Returns the URL where the contents of the file referenced by ``name`` can be
+accessed.
+
+``Storage.open(name, mode='rb')``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Opens the file given by ``name``. Note that although the returned file is
+guaranteed to be a ``File`` object, it might actually be some subclass. In the
+case of remote file storage this means that reading/writing could be quite slow,
+so be warned.
+
+``Storage.save(name, content)``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Saves a new file using the storage system, preferably with the name specified.
+If there already exists a file with this name ``name``, the storage system may
+modify the filename as necessary to get a unique name. The actual name of the
+stored file will be returned.
+
+``Storage.delete(name)``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Deletes the file referenced by ``name``. This method won't raise an exception if
+the file doesn't exist.
+
+Writing a custom storage system
+===============================
+
+If you need to provide custom file storage -- a common example is storing files
+on some remote system -- you can do so by defining a custom storage class.
+You'll need to follow these steps:
+
+#. Your custom storage system must be a subclass of 
+   ``django.core.files.storage.Storage``::
+
+        from django.core.files.storage import Storage
+    
+        class MyStorage(Storage):
+            ...
+        
+#. Django must be able to instantiate your storage system without any arguments. 
+   This means that any settings should be taken from ``django.conf.settings``::
+   
+        from django.conf import settings
+        from django.core.files.storage import Storage
+
+        class MyStorage(Storage):
+            def __init__(self, option=None):
+                if not option:
+                    option = settings.CUSTOM_STORAGE_OPTIONS
+                ...
+
+#. Your storage class must implement the ``_open()`` and ``_save()`` methods,
+   along with any other methods appropriate to your storage class. See below for
+   more on these methods. 
+   
+   In addition, if your class provides local file storage, it must override
+   the ``path()`` method.
+   
+Custom storage system methods
+-----------------------------
+
+Your custom storage system may override any of the storage methods explained
+above in `storage objects`_. However, it's usually better to use the hooks
+specifically designed for custom storage objects. These are:
+
+``_open(name, mode='rb')``
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Required**.
+
+Called by ``Storage.open()``, this is the actual mechanism the storage class
+uses to open the file. This must return a ``File`` object, though in most cases,
+you'll want to return some subclass here that implements logic specific to the
+backend storage system.
+
+``_save(name, content)``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Called by ``Storage.save()``. The ``name`` will already have gone through
+``get_valid_name()`` and ``get_available_name()``, and the ``content`` will be a
+``File`` object itself. No return value is expected.
+
+``get_valid_name(name)``
+------------------------
+
+Returns a filename suitable for use with the underlying storage system. The
+``name`` argument passed to this method is the original filename sent to the
+server, after having any path information removed. Override this to customize
+how non-standard characters are converted to safe filenames.
+
+The code provided on ``Storage`` retains only alpha-numeric characters, periods
+and underscores from the original filename, removing everything else.
+
+``get_available_name(name)``
+----------------------------
+
+Returns a filename that is available in the storage mechanism, possibly taking
+the provided filename into account. The ``name`` argument passed to this method
+will have already cleaned to a filename valid for the storage system, according
+to the ``get_valid_name()`` method described above.
+
+The code provided on ``Storage`` simply appends underscores to the filename
+until it finds one that's available in the destination directory.
\ No newline at end of file
diff --git a/docs/model-api.txt b/docs/model-api.txt
index 93b27b8c11..da5584e2bc 100644
--- a/docs/model-api.txt
+++ b/docs/model-api.txt
@@ -224,26 +224,64 @@ set to 75 by default, but you can specify it to override default behavior.
 ``FileField``
 ~~~~~~~~~~~~~
 
-A file-upload field. Has one **required** argument:
+A file-upload field. Has two special arguments, of which the first is
+**required**:
 
     ======================  ===================================================
     Argument                Description
     ======================  ===================================================
-    ``upload_to``           A local filesystem path that will be appended to
-                            your ``MEDIA_ROOT`` setting to determine the
-                            output of the ``get_<fieldname>_url()`` helper
-                            function.
+    ``upload_to``           Required. A filesystem-style path that will be
+                            prepended to the filename before being committed to
+                            the final storage destination.
+
+                            **New in Django development version**
+
+                            This may also be a callable, such as a function,
+                            which will be called to obtain the upload path,
+                            including the filename. See below for details.
+
+    ``storage``             **New in Django development version**
+
+                            Optional. A storage object, which handles the
+                            storage and retrieval of your files. See `managing
+                            files`_ for details on how to provide this object.
+    ======================  ===================================================
+
+.. _managing files: ../files/
+
+The ``upload_to`` path may contain `strftime formatting`_, which will be
+replaced by the date/time of the file upload (so that uploaded files don't fill
+up the given directory).
+
+**New in Django development version**
+
+If a callable is provided for the ``upload_to`` argument, that callable must be
+able to accept two arguments, and return a Unix-style path (with forward
+slashes) to be passed along to the storage system. The two arguments that will
+be passed are:
+
+    ======================  ===================================================
+    Argument                Description
     ======================  ===================================================
+    ``instance``            An instance of the model where the ``FileField`` is
+                            defined. More specifically, this is the particular
+                            instance where the current file is being attached.
 
-This path may contain `strftime formatting`_, which will be replaced by the
-date/time of the file upload (so that uploaded files don't fill up the given
-directory).
+                            **Note**: In most cases, this object will not have
+                            been saved to the database yet, so if it uses the
+                            default ``AutoField``, *it might not yet have a
+                            value for its primary key field*.
+
+    ``filename``            The filename that was originally given to the file.
+                            This may or may not be taken into account when
+                            determining the final destination path.
+    ======================  ===================================================
 
 The admin represents this field as an ``<input type="file">`` (a file-upload
 widget).
 
-Using a ``FileField`` or an ``ImageField`` (see below) in a model takes a few
-steps:
+Using a ``FileField`` or an ``ImageField`` (see below) in a model without a
+specified storage system takes a few steps:
 
     1. In your settings file, you'll need to define ``MEDIA_ROOT`` as the
        full path to a directory where you'd like Django to store uploaded
diff --git a/docs/settings.txt b/docs/settings.txt
index 1d1627d41e..3b3e908fab 100644
--- a/docs/settings.txt
+++ b/docs/settings.txt
@@ -426,6 +426,16 @@ Default content type to use for all ``HttpResponse`` objects, if a MIME type
 isn't manually specified. Used with ``DEFAULT_CHARSET`` to construct the
 ``Content-Type`` header.
 
+DEFAULT_FILE_STORAGE
+--------------------
+
+Default: ``'django.core.filestorage.filesystem.FileSystemStorage'``
+
+Default file storage class to be used for any file-related operations that don't
+specify a particular storage system. See the `file documentation`_ for details.
+
+.. _file documentation: ../files/
+
 DEFAULT_FROM_EMAIL
 ------------------
 
diff --git a/docs/upload_handling.txt b/docs/upload_handling.txt
index c0e8605686..488778a4e4 100644
--- a/docs/upload_handling.txt
+++ b/docs/upload_handling.txt
@@ -155,25 +155,8 @@ Three `settings`_ control Django's file upload behavior:
 ``UploadedFile`` objects
 ========================
 
-All ``UploadedFile`` objects define the following methods/attributes:
-
-    ``UploadedFile.read(self, num_bytes=None)``
-        Returns a byte string of length ``num_bytes``, or the complete file if
-        ``num_bytes`` is ``None``.
-
-    ``UploadedFile.chunks(self, chunk_size=None)``
-        A generator yielding small chunks from the file. If ``chunk_size`` isn't
-        given, chunks will be 64 KB.
-
-    ``UploadedFile.multiple_chunks(self, chunk_size=None)``
-        Returns ``True`` if you can expect more than one chunk when calling
-        ``UploadedFile.chunks(self, chunk_size)``.
-
-    ``UploadedFile.size``
-        The size, in bytes, of the uploaded file.
-
-    ``UploadedFile.name``
-        The name of the uploaded file as provided by the user.
+In addition to those inherited from `File`_, all ``UploadedFile`` objects define
+the following methods/attributes:
 
     ``UploadedFile.content_type``
         The content-type header uploaded with the file (e.g. ``text/plain`` or
@@ -186,13 +169,11 @@ All ``UploadedFile`` objects define the following methods/attributes:
         For ``text/*`` content-types, the character set (i.e. ``utf8``) supplied
         by the browser. Again, "trust but verify" is the best policy here.
 
-    ``UploadedFile.__iter__()``
-        Iterates over the lines in the file.
-
     ``UploadedFile.temporary_file_path()``
         Only files uploaded onto disk will have this method; it returns the full
         path to the temporary uploaded file.
 
+.. _File: ../files/
 
 Upload Handlers
 ===============