Fixed #12990, Refs #27694 -- Added JSONField model field.

Thanks to Adam Johnson, Carlton Gibson, Mariusz Felisiak, and Raphael
Michel for mentoring this Google Summer of Code 2019 project and
everyone else who helped with the patch.

Special thanks to Mads Jensen, Nick Pope, and Simon Charette for
extensive reviews.

Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>

6 files changed, 154 insertions, 92 deletions

diff --git a/docs/ref/checks.txt b/docs/ref/checks.txt
index daf651392f..37a3a572c9 100644
--- a/docs/ref/checks.txt
+++ b/docs/ref/checks.txt
@@ -190,6 +190,7 @@ Model fields
   ``<field data type>`` columns.
 * **fields.E170**: ``BinaryField``’s ``default`` cannot be a string. Use bytes
   content instead.
+* **fields.E180**: ``<database>`` does not support ``JSONField``\s.
 * **fields.E900**: ``IPAddressField`` has been removed except for support in
   historical migrations.
 * **fields.W900**: ``IPAddressField`` has been deprecated. Support for it
@@ -204,6 +205,9 @@ Model fields
   Django 3.1. *This check appeared in Django 2.2 and 3.0*.
 * **fields.W903**: ``NullBooleanField`` is deprecated. Support for it (except
   in historical migrations) will be removed in Django 4.0.
+* **fields.W904**: ``django.contrib.postgres.fields.JSONField`` is deprecated.
+  Support for it (except in historical migrations) will be removed in Django
+  4.0.
 
 File fields
 ~~~~~~~~~~~
diff --git a/docs/ref/contrib/postgres/fields.txt b/docs/ref/contrib/postgres/fields.txt
index baebba9c50..aeacc72e7c 100644
--- a/docs/ref/contrib/postgres/fields.txt
+++ b/docs/ref/contrib/postgres/fields.txt
@@ -16,8 +16,7 @@ Indexes such as :class:`~django.contrib.postgres.indexes.GinIndex` and
 :class:`~django.contrib.postgres.indexes.GistIndex` are better suited, though
 the index choice is dependent on the queries that you're using. Generally, GiST
 may be a good choice for the :ref:`range fields <range-fields>` and
-:class:`HStoreField`, and GIN may be helpful for :class:`ArrayField` and
-:class:`JSONField`.
+:class:`HStoreField`, and GIN may be helpful for :class:`ArrayField`.
 
 ``ArrayField``
 ==============
@@ -517,96 +516,14 @@ using in conjunction with lookups on
     of the JSON which allows indexing. The trade-off is a small additional cost
     on writing to the ``jsonb`` field. ``JSONField`` uses ``jsonb``.
 
-Querying ``JSONField``
-----------------------
-
-We will use the following example model::
-
-    from django.contrib.postgres.fields import JSONField
-    from django.db import models
-
-    class Dog(models.Model):
-        name = models.CharField(max_length=200)
-        data = JSONField()
-
-        def __str__(self):
-            return self.name
-
-.. fieldlookup:: jsonfield.key
-
-Key, index, and path lookups
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To query based on a given dictionary key, use that key as the lookup name::
-
-    >>> Dog.objects.create(name='Rufus', data={
-    ...     'breed': 'labrador',
-    ...     'owner': {
-    ...         'name': 'Bob',
-    ...         'other_pets': [{
-    ...             'name': 'Fishy',
-    ...         }],
-    ...     },
-    ... })
-    >>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': None})
-
-    >>> Dog.objects.filter(data__breed='collie')
-    <QuerySet [<Dog: Meg>]>
-
-Multiple keys can be chained together to form a path lookup::
+.. deprecated:: 3.1
 
-    >>> Dog.objects.filter(data__owner__name='Bob')
-    <QuerySet [<Dog: Rufus>]>
+    Use :class:`django.db.models.JSONField` instead.
 
-If the key is an integer, it will be interpreted as an index lookup in an
-array::
-
-    >>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
-    <QuerySet [<Dog: Rufus>]>
-
-If the key you wish to query by clashes with the name of another lookup, use
-the :lookup:`jsonfield.contains` lookup instead.
-
-If only one key or index is used, the SQL operator ``->`` is used. If multiple
-operators are used then the ``#>`` operator is used.
-
-To query for ``null`` in JSON data, use ``None`` as a value::
-
-    >>> Dog.objects.filter(data__owner=None)
-    <QuerySet [<Dog: Meg>]>
-
-To query for missing keys, use the ``isnull`` lookup::
-
-    >>> Dog.objects.create(name='Shep', data={'breed': 'collie'})
-    >>> Dog.objects.filter(data__owner__isnull=True)
-    <QuerySet [<Dog: Shep>]>
-
-.. warning::
-
-    Since any string could be a key in a JSON object, any lookup other than
-    those listed below will be interpreted as a key lookup. No errors are
-    raised. Be extra careful for typing mistakes, and always check your queries
-    work as you intend.
-
-Containment and key operations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. fieldlookup:: jsonfield.contains
-.. fieldlookup:: jsonfield.contained_by
-.. fieldlookup:: jsonfield.has_key
-.. fieldlookup:: jsonfield.has_any_keys
-.. fieldlookup:: jsonfield.has_keys
-
-:class:`~django.contrib.postgres.fields.JSONField` shares lookups relating to
-containment and keys with :class:`~django.contrib.postgres.fields.HStoreField`.
+Querying ``JSONField``
+----------------------
 
-- :lookup:`contains <hstorefield.contains>` (accepts any JSON rather than
-  just a dictionary of strings)
-- :lookup:`contained_by <hstorefield.contained_by>` (accepts any JSON
-  rather than just a dictionary of strings)
-- :lookup:`has_key <hstorefield.has_key>`
-- :lookup:`has_any_keys <hstorefield.has_any_keys>`
-- :lookup:`has_keys <hstorefield.has_keys>`
+See :ref:`querying-jsonfield` for details.
 
 .. _range-fields:
 
diff --git a/docs/ref/contrib/postgres/forms.txt b/docs/ref/contrib/postgres/forms.txt
index f559ac75cb..14a3ad61de 100644
--- a/docs/ref/contrib/postgres/forms.txt
+++ b/docs/ref/contrib/postgres/forms.txt
@@ -164,8 +164,8 @@ Fields
 .. class:: JSONField
 
     A field which accepts JSON encoded data for a
-    :class:`~django.contrib.postgres.fields.JSONField`. It is represented by an
-    HTML ``<textarea>``.
+    :class:`~django.db.models.JSONField`. It is represented by an HTML
+    ``<textarea>``.
 
     .. admonition:: User friendly forms
 
@@ -173,6 +173,10 @@ Fields
         it is a useful way to format data from a client-side widget for
         submission to the server.
 
+    .. deprecated:: 3.1
+
+        Use :class:`django.forms.JSONField` instead.
+
 Range Fields
 ------------
 
diff --git a/docs/ref/databases.txt b/docs/ref/databases.txt
index f01a054d51..a16f525d96 100644
--- a/docs/ref/databases.txt
+++ b/docs/ref/databases.txt
@@ -783,6 +783,22 @@ iterator. Your code must handle this.
 
 .. _`Isolation in SQLite`: https://sqlite.org/isolation.html
 
+.. _sqlite-json1:
+
+Enabling JSON1 extension on SQLite
+----------------------------------
+
+To use :class:`~django.db.models.JSONField` on SQLite, you need to enable the
+`JSON1 extension`_ on Python's :py:mod:`sqlite3` library. If the extension is
+not enabled on your installation, a system error (``fields.E180``) will be
+raised.
+
+To enable the JSON1 extension you can follow the instruction on
+`the wiki page`_.
+
+.. _JSON1 extension: https://www.sqlite.org/json1.html
+.. _the wiki page: https://code.djangoproject.com/wiki/JSON1Extension
+
 .. _oracle-notes:
 
 Oracle notes
diff --git a/docs/ref/forms/fields.txt b/docs/ref/forms/fields.txt
index 3d228e88ad..58db957512 100644
--- a/docs/ref/forms/fields.txt
+++ b/docs/ref/forms/fields.txt
@@ -776,6 +776,60 @@ For each field, we describe the default widget used if you don't specify
 
     These control the range of values permitted in the field.
 
+``JSONField``
+-------------
+
+.. class:: JSONField(encoder=None, decoder=None, **kwargs)
+
+    .. versionadded:: 3.1
+
+    A field which accepts JSON encoded data for a
+    :class:`~django.db.models.JSONField`.
+
+    * Default widget: :class:`Textarea`
+    * Empty value: ``''`` (an empty string)
+    * Normalizes to: A Python representation of the JSON value (usually as a
+      ``dict``, ``list``, or ``None``), depending on :attr:`JSONField.decoder`.
+    * Validates that the given value is a valid JSON.
+    * Error message keys: ``required``, ``invalid``
+
+    Takes two optional arguments:
+
+    .. attribute:: encoder
+
+        A :py:class:`json.JSONEncoder` subclass to serialize data types not
+        supported by the standard JSON serializer (e.g. ``datetime.datetime``
+        or :class:`~python:uuid.UUID`). For example, you can use the
+        :class:`~django.core.serializers.json.DjangoJSONEncoder` class.
+
+        Defaults to ``json.JSONEncoder``.
+
+    .. attribute:: decoder
+
+        A :py:class:`json.JSONDecoder` subclass to deserialize the input. Your
+        deserialization may need to account for the fact that you can't be
+        certain of the input type. For example, you run the risk of returning a
+        ``datetime`` that was actually a string that just happened to be in the
+        same format chosen for ``datetime``\s.
+
+        The ``decoder`` can be used to validate the input. If
+        :py:class:`json.JSONDecodeError` is raised during the deserialization,
+        a ``ValidationError`` will be raised.
+
+        Defaults to ``json.JSONDecoder``.
+
+    .. note::
+
+        If you use a :class:`ModelForm <django.forms.ModelForm>`, the
+        ``encoder`` and ``decoder`` from :class:`~django.db.models.JSONField`
+        will be used.
+
+    .. admonition:: User friendly forms
+
+        ``JSONField`` is not particularly user friendly in most cases. However,
+        it is a useful way to format data from a client-side widget for
+        submission to the server.
+
 ``GenericIPAddressField``
 -------------------------
 
diff --git a/docs/ref/models/fields.txt b/docs/ref/models/fields.txt
index 35b83c9b7a..452736dfa8 100644
--- a/docs/ref/models/fields.txt
+++ b/docs/ref/models/fields.txt
@@ -348,7 +348,7 @@ The default can't be a mutable object (model instance, ``list``, ``set``, etc.),
 as a reference to the same instance of that object would be used as the default
 value in all new model instances. Instead, wrap the desired default in a
 callable. For example, if you want to specify a default ``dict`` for
-:class:`~django.contrib.postgres.fields.JSONField`, use a function::
+:class:`~django.db.models.JSONField`, use a function::
 
     def contact_default():
         return {"email": "to1@example.com"}
@@ -1175,6 +1175,73 @@ are converted to lowercase.
 If you allow for blank values, you have to allow for null values since blank
 values are stored as null.
 
+``JSONField``
+-------------
+
+.. class:: JSONField(encoder=None, decoder=None, **options)
+
+.. versionadded:: 3.1
+
+A field for storing JSON encoded data. In Python the data is represented in its
+Python native format: dictionaries, lists, strings, numbers, booleans and
+``None``.
+
+``JSONField`` is supported on MariaDB 10.2.7+, MySQL 5.7.8+, Oracle,
+PostgreSQL, and SQLite 3.9.0+ (with the :ref:`JSON1 extension enabled
+<sqlite-json1>`).
+
+.. attribute:: JSONField.encoder
+
+    An optional :py:class:`json.JSONEncoder` subclass to serialize data types
+    not supported by the standard JSON serializer (e.g. ``datetime.datetime``
+    or :class:`~python:uuid.UUID`). For example, you can use the
+    :class:`~django.core.serializers.json.DjangoJSONEncoder` class.
+
+    Defaults to ``json.JSONEncoder``.
+
+.. attribute:: JSONField.decoder
+
+    An optional :py:class:`json.JSONDecoder` subclass to deserialize the value
+    retrieved from the database. The value will be in the format chosen by the
+    custom encoder (most often a string). Your deserialization may need to
+    account for the fact that you can't be certain of the input type. For
+    example, you run the risk of returning a ``datetime`` that was actually a
+    string that just happened to be in the same format chosen for
+    ``datetime``\s.
+
+    Defaults to ``json.JSONDecoder``.
+
+If you give the field a :attr:`~django.db.models.Field.default`, ensure it's an
+immutable object, such as a ``str``, or a callable object that returns a fresh
+mutable object each time, such as ``dict`` or a function. Providing a mutable
+default object like ``default={}`` or ``default=[]`` shares the one object
+between all model instances.
+
+To query ``JSONField`` in the database, see :ref:`querying-jsonfield`.
+
+.. admonition:: Indexing
+
+    :class:`~django.db.models.Index` and :attr:`.Field.db_index` both create a
+    B-tree index, which isn't particularly helpful when querying ``JSONField``.
+    On PostgreSQL only, you can use
+    :class:`~django.contrib.postgres.indexes.GinIndex` that is better suited.
+
+.. admonition:: PostgreSQL users
+
+    PostgreSQL has two native JSON based data types: ``json`` and ``jsonb``.
+    The main difference between them is how they are stored and how they can be
+    queried. PostgreSQL's ``json`` field is stored as the original string
+    representation of the JSON and must be decoded on the fly when queried
+    based on keys. The ``jsonb`` field is stored based on the actual structure
+    of the JSON which allows indexing. The trade-off is a small additional cost
+    on writing to the ``jsonb`` field. ``JSONField`` uses ``jsonb``.
+
+.. admonition:: Oracle users
+
+    Oracle Database does not support storing JSON scalar values. Only JSON
+    objects and arrays (represented in Python using :py:class:`dict` and
+    :py:class:`list`) are supported.
+
 ``NullBooleanField``
 --------------------