#32111: Django 3.1.2 JSONField handling error with an Postgres Foreign Data
Wrapper-based model
-------------------------------------+-------------------------------------
Reporter: Shaheed | Owner: nobody
Haque |
Type: Bug | Status: new
Component: Database | Version: 3.1
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Over at https://groups.google.com/g/django-users/c/Sr1TDVRpCLE, it was
suggested that I post an issue here...
I have a Django model working fine under Django 3.0 (i.e. "Django<3.1")
which looks like this:
{{{
class Job(models.Model):
id = models.CharField(max_length=36, primary_key=True)
queue = models.CharField(max_length=40)
args = JSONField()
kwargs = JSONField()
type = models.CharField(max_length=80)
...
class Meta:
managed = False # <------ The table is implemented as a
Postgres FDW wrapper.
db_table = 'jobs'
}}}
I am testing the update to Django 3.1.2 and hit an error in executing this
line:
jobs = list(models.Job.objects.filter(queue='celery',
state='scheduled'))
The error is as follows from pytest (i.e. stack trace with local variables
too):
{{{
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in
__iter__
self._fetch_all()
self = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in
_fetch_all
self._result_cache = list(self._iterable_class(self))
self = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in
__iter__
for row in compiler.results_iter(results):
annotation_col_map = {}
compiler = <django.db.models.sql.compiler.SQLCompiler object at
0x7f8685e49160>
db = 'fdw'
init_list = ['id', 'queue', 'args', 'kwargs', 'type', 'state',
...]
klass_info = {'model': <class 'paiyroll.models.batch.Job'>,
'select_fields': [0, 1, 2, 3, 4, 5, ...]}
known_related_objects = []
model_cls = <class 'paiyroll.models.batch.Job'>
model_fields_end = 9
model_fields_start = 0
queryset = <QuerySet []>
related_populators = []
results = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
'paiyroll.tasks.function_run', 'scheduled', ...)]]
select = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []),
None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None...,
paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs,
paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
select_fields = [0, 1, 2, 3, 4, 5, ...]
self = <django.db.models.query.ModelIterable object at
0x7f86836f3040>
/usr/local/lib/python3.8/dist-
packages/django/db/models/sql/compiler.py:1100: in apply_converters
value = converter(value, expression, connection)
connection = <django.db.backends.postgresql.base.DatabaseWrapper
object at 0x7f869a321670>
converter = <bound method JSONField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: args>>
converters = [(2, ([<bound method JSONField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs,
pa...NField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs,
paiyroll.Job.details)))]
convs = [<bound method JSONField.from_db_value of
<django.contrib.postgres.fields.jsonb.JSONField: args>>]
expression = Col(jobs, paiyroll.Job.args)
pos = 2
row = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery',
['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {},
'paiyroll.tasks.function_run', 'scheduled', ...]
rows = <itertools.chain object at 0x7f8683ae7520>
self = <django.db.models.sql.compiler.SQLCompiler object at
0x7f8685e49160>
value = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
[], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74:
in from_db_value
return json.loads(value, cls=self.decoder)
connection = <django.db.backends.postgresql.base.DatabaseWrapper
object at 0x7f869a321670>
expression = Col(jobs, paiyroll.Job.args)
self = <django.contrib.postgres.fields.jsonb.JSONField: args>
value = ['paiyroll.tasks', 'call_to_string', 'call_to_string',
[], {}]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls =
None
object_hook = None, parse_float = None, parse_int = None, parse_constant =
None
object_pairs_hook = None, kw = {}
def loads(s, *, cls=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, object_pairs_hook=None,
**kw):
"""Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray``
instance
containing a JSON document) to a Python object.
``object_hook`` is an optional function that will be called with
the
result of any object literal decode (a ``dict``). The return value
of
``object_hook`` will be used instead of the ``dict``. This feature
can be used to implement custom decoders (e.g. JSON-RPC class
hinting).
``object_pairs_hook`` is an optional function that will be called
with the
result of any object literal decoded with an ordered list of pairs.
The
return value of ``object_pairs_hook`` will be used instead of the
``dict``.
This feature can be used to implement custom decoders. If
``object_hook``
is also defined, the ``object_pairs_hook`` takes priority.
``parse_float``, if specified, will be called with the string
of every JSON float to be decoded. By default this is equivalent to
float(num_str). This can be used to use another datatype or parser
for JSON floats (e.g. decimal.Decimal).
``parse_int``, if specified, will be called with the string
of every JSON int to be decoded. By default this is equivalent to
int(num_str). This can be used to use another datatype or parser
for JSON integers (e.g. float).
``parse_constant``, if specified, will be called with one of the
following strings: -Infinity, Infinity, NaN.
This can be used to raise an exception if invalid JSON numbers
are encountered.
To use a custom ``JSONDecoder`` subclass, specify it with the
``cls``
kwarg; otherwise ``JSONDecoder`` is used.
The ``encoding`` argument is ignored and deprecated since Python
3.1.
"""
if isinstance(s, str):
if s.startswith('\ufeff'):
raise JSONDecodeError("Unexpected UTF-8 BOM (decode using
utf-8-sig)",
s, 0)
else:
if not isinstance(s, (bytes, bytearray)):
> raise TypeError(f'the JSON object must be str, bytes or
bytearray, '
f'not {s.__class__.__name__}')
E TypeError: the JSON object must be str, bytes or
bytearray, not list
cls = None
kw = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int = None
s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [],
{}]
}}}
As you can perhaps see from the section for /usr/local/lib/python3.8/dist-
packages/django/db/models/fields/json.py:74, the value being written into
the JSONField called "args" is a Python list, (shown as "s" on the last
line of the traceback-with-values). I am aware of documented changes
around serializers but did not think that affected me; however, given that
I am using an FDW to return the data, is it that I should now be
serialising returned data into a string?
Any pointers appreciated.
Thanks, Shaheed
--
Ticket URL: <https://code.djangoproject.com/ticket/32111>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/055.44740ef3f8286c3a882a54d8ec109578%40djangoproject.com.
