diff options
| author | François Freitag <mail@franek.fr> | 2017-06-01 16:56:51 -0400 |
|---|---|---|
| committer | Tim Graham <timograham@gmail.com> | 2017-06-01 17:50:41 -0400 |
| commit | edee5a8de6afb34a9247de2bb73ab66397a6e573 (patch) | |
| tree | e306f62cdbfc1e28ffb119bdbd7870a138e91dd9 /docs/ref | |
| parent | bf50ae821021c4f7cd608d2bd1f2dfff98f3ceb9 (diff) | |
Fixed #27639 -- Added chunk_size parameter to QuerySet.iterator().
Diffstat (limited to 'docs/ref')
| -rw-r--r-- | docs/ref/models/querysets.txt | 22 |
1 files changed, 21 insertions, 1 deletions
diff --git a/docs/ref/models/querysets.txt b/docs/ref/models/querysets.txt index cc80069894..48c547f514 100644 --- a/docs/ref/models/querysets.txt +++ b/docs/ref/models/querysets.txt @@ -2004,7 +2004,7 @@ If you pass ``in_bulk()`` an empty list, you'll get an empty dictionary. ``iterator()`` ~~~~~~~~~~~~~~ -.. method:: iterator() +.. method:: iterator(chunk_size=2000) Evaluates the ``QuerySet`` (by performing the query) and returns an iterator (see :pep:`234`) over the results. A ``QuerySet`` typically caches its results @@ -2033,6 +2033,11 @@ set into memory. The Oracle database driver always uses server-side cursors. +With server-side cursors, the ``chunk_size`` parameter specifies the number of +results to cache at the database driver level. Fetching bigger chunks +diminishes the number of round trips between the database driver and the +database, at the expense of memory. + On PostgreSQL, server-side cursors will only be used when the :setting:`DISABLE_SERVER_SIDE_CURSORS <DATABASE-DISABLE_SERVER_SIDE_CURSORS>` setting is ``False``. Read :ref:`transaction-pooling-server-side-cursors` if @@ -2048,10 +2053,25 @@ drivers load the entire result set into memory. The result set is then transformed into Python row objects by the database adapter using the ``fetchmany()`` method defined in :pep:`249`. +The ``chunk_size`` parameter controls the size of batches Django retrieves from +the database driver. Larger batches decrease the overhead of communicating with +the database driver at the expense of a slight increase in memory consumption. + +The default value of ``chunk_size``, 2000, comes from `a calculation on the +psycopg mailing list <https://www.postgresql.org/message-id/4D2F2C71.8080805%40dndg.it>`_: + + Assuming rows of 10-20 columns with a mix of textual and numeric data, 2000 + is going to fetch less than 100KB of data, which seems a good compromise + between the number of rows transferred and the data discarded if the loop + is exited early. + .. versionchanged:: 1.11 PostgreSQL support for server-side cursors was added. +.. versionchanged:: 2.0 + + The ``chunk_size`` parameter was added. ``latest()`` ~~~~~~~~~~~~ |