#18023: Stop bundling simplejson and deprecate the module
-------------------------------------+-------------------------------------
Reporter: ogier | Owner: aaugustin
Type: | Status: reopened
Cleanup/optimization | Version: master
Component: Core | Resolution:
(Serialization) | Triage Stage: Accepted
Severity: Release blocker | Needs documentation: 0
Keywords: | Patch needs improvement: 0
Has patch: 0 | UI/UX: 0
Needs tests: 0 |
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by ogier):
Replying to [comment:9 aaugustin]:
> [https://groups.google.com/forum/?fromgroups#!topic/django-
developers/JM4ndZbk_wY This thread on the mailing list] and
[https://github.com/django/django/commit/cec6bd5a59547dc97fe98975c570fc27a1e970be#L5R53
this comment on the commit] indicate that this change may be backwards
incompatible.
>
> Right now I don't know what the correct resolution is. In all cases it
should be documented here.
Here's what I believe is an accurate summary of the situation:
'''First issue: str vs. unicode'''
The simplejson API is documented as returning unicode strings for all JSON
keys and string values.
http://simplejson.readthedocs.org/en/latest/index.html#simplejson.JSONDecoder
In fact, the optional C library (installed by default when installing
simplejson) makes an API-changing optimization: if the value is entirely
ASCII characters, a `str` instance is returned instead of a `unicode`
instance.
https://github.com/simplejson/simplejson/blob/master/simplejson/_speedups.c#L527
The Python standard library `json` package was forked from simplejson at
version 1.9, and included this optimization. However, it was considered a
bug and fixed in python 2.7. http://bugs.python.org/issue11982
The net result is that it is possible that some code that runs on manually
installed versions of simplejson or python <= 2.6 depends on this
behavior. I think that Python core's approach is correct: we should
consider this a bug, because it is behavior that directly contradicts the
documentation. It's worth pointing out that any code that depends on this
behavior is already broken on python 2.7.
'''Second issue: namedtuple_as_object'''
This is a systemic problem with any shim that is supposed to be
transparent. In this case, `namedtuple_as_object` is a keyword argument to
`simplejson.JSONEncoder` that was added in simplejson 2.2. The latest
version of simplejson that was merged into the python standard library is
2.0.9 and Django 1.4 shipped with simplejson 2.0.7 bundled. This means
that nowhere in stock Django are there any references to this keyword.
This wouldn't be a huge problem, except that Django subclasses
`django.utils.simplejson.JSONEncoder` as
`django.core.serializers.json.DjangoJSONEncoder`, and naturally doesn't
know about this keyword argument.
This is a deeper problem than just passing along unrecognized keyword
arguments to the base class in `__init__()` to make sure that
`DjangoJSONEncoder` never breaks. If we want to deprecate the
`django.utils.simplejson` shim, then `DjangoJSONEncoder` must become a
subclass of `json.JSONEncoder`. This has the unfortunate effect that calls
to `simplejson.dumps({'hi':'there'}, cls=DjangoJSONEncoder)` will all fail
(this is the `simplejson` on PyPI I'm talking about). Basically
`DjangoJSONEncoder` will be incompatible with simplejson, which is a
backwards incompatibility.
The problem is that even leaving in the shim and subclassing as we do
isn't a good solution. It means that DjangoJSONEncoder needs to not only
be backwards compatible through all supported Python versions, but they
also must be forwards compatible with the latest simplejson releases,
which have already been shown to diverge in backwards incompatible ways.
(Adding an argument to a method on a base class breaks contravariance.)
If that paragraph seemed confusing, think of it this way: if
`DjangoJSONEncoder` is a `json.JSONEncoder` then passing it to
`simplejson.dumps()` is dangerous. If `DjangoJSONEncoder` is a
`simplejson.JSONEncoder` then passing it to `json.dumps()` is dangerous.
--
Ticket URL: <https://code.djangoproject.com/ticket/18023#comment:10>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To post to this group, send email to django-updates@googlegroups.com.
To unsubscribe from this group, send email to
django-updates+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-updates?hl=en.
