[issue18264] enum.IntEnum is not compatible with JSON serialisation
Roundup Robot added the comment: New changeset ae1a7c420f08 by Ethan Furman in branch 'default': Close #18264: int- and float-derived enums now converted to int or float. http://hg.python.org/cpython/rev/ae1a7c420f08 -- nosy: +python-dev resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Hopefully the final patch. :) -- Added file: http://bugs.python.org/file31172/issue18264.stoneleaf.04.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Forgot to back out core dump tests before creating previous patch. This one even passes the tests. :/ -- Added file: http://bugs.python.org/file31173/issue18264.stoneleaf.05.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: LGTM now. Make sure to run the test(s) in refleak mode and let's wait for a couple of days before committing, in case someone else has comments. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I'll plan on committing no sooner than Friday unless somebody else has comments/corrections. Thanks, Eli. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Eli Bendersky added the comment: Make sure to run the test(s) in refleak mode . . . How extensive should testing be? I plan on always running the refleak mode tests (now that I know how ;) . Should I also run non-refleak tests? Should I run tests with a python built without --with-pydebug? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: IMHO it's very much dependent on the change. When making C code changes, I usually run the relevant tests with refleaks, and then run the whole test suite; all of this --with-pydebug. Personally I've not encountered cases where non-debug builds failed where debug suceeded for CPython, but for very significant changes I may do some extra runs in the release mode as well (or try to test it on a specific OS). It's very important to watch the bots after committing, because they run a wide variety of configurations you'll be hard pressed to reproduce on your own - so they exercise the change pretty well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ezio Melotti added the comment: I plan on always running the refleak mode tests (now that I know how ;) FWIW `make patchcheck` should remind you about that. -- stage: - patch review type: - enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I don't understand. Type checks are already performed throughout the code (int, float, True, False, NaN, Inf, etc.). What will opereator.index buy us? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Nick Coghlan added the comment: The two isinstance checks that bother me are the ones for int and float. However, given that the JSON serialiser *already* includes those explicit checks, I now think it makes sense to just do the minimal fix of coercing subclasses to the base type for both of those cases, especially since such a change should be able to be ported to simplejson as well. Anything more elaborate (including making JSON explicitly enum aware, or else offering a new encoder that supports singledispatch) should be handled in a separate issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: I also think that exchanging the explicit type checks by __index__ merits more thought and is outside the scope of this local fix. The proposed patch does not add any new type checks, but acts within the bounds of code for which the type is already established. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: This patch handles both float and int subclasses, and includes fixes/improvements from the review. -- Added file: http://bugs.python.org/file31155/issue18264.stoneleaf.02.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Forgot to add float tests. Included in this patch. -- Added file: http://bugs.python.org/file31156/issue18264.stoneleaf.03.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Eli, your method is good. I thought I had tried something similar but I obviously had the wrong PyLong constructor. I'll get it implemented. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: Yes, AFAIU PyNumber_Long is the equivalent of Python-level int(obj). With other constructors of PyLong you are limited by long long (while Python integers may be arbitrarily large). Ethan - If you're still short on time I can pretty up this patch and put it for review with some tests. Let me know what you prefer. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Thanks for the offer, Eli, but I almost have the tests done. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Okay, patch attached with C code (thanks, Eli!), more python code, and some new tests. Only the `int` case is handled. -- Added file: http://bugs.python.org/file31141/issue18264.stoneleaf.01.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: Posted a Rietveld code review -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Nick Coghlan added the comment: It occurs to me that operator.index() (without a preceding type check) is likely the more ducktyping friendly option here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: I've been reading the discussion again to figure out what we need to move forward; it appears that a simple approach of using str(int(obj)) in json encoding when isinstance(obj, int) was acceptable for many of the participants, and it helps solve the most common problem - the one with IntEnum. We can solve it now, move on to converting socket.* and other constants to IntEnum and perhaps change the solution to something more grandiose later. I'm attaching a proof-of-concept (probably incomplete) patch that implements this for the Python and C implementations of json. With it applied, Nick's example goes like this: from enum import IntEnum import json class Example(IntEnum): x = 1 ... json.dumps(Example.x) '1' json.loads(json.dumps(Example.x)) 1 When thinking about the more generic approaches, we must consider what was raised here already - JSON interoperates with other languages, for which Python's enum is meaningless (think of JavaScript, for instance). So the only logical way to JSON-encode a IntEnum is str(int(...)). -- keywords: +patch Added file: http://bugs.python.org/file31134/issue18264.1-prelim.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: [please note that the patch is POC/hacky at best - it's likely to leak memory and be careless and incomplete in other ways. I'm just trying to demonstrate the approach] -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I'll check you patch later against big numbers (which is where I had difficulties). If it works I'll try to make it more complete. If it doesn't, I've been working on just extraction the Enum member's value and using that (works fine on the Python side ;) . Big Question: if the Enum member was used as a key, do we use .name or .value? The decision I went with was to compare the hashes of the member name vs the member itself -- if they're the same, use the .name, otherwise use the value. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I don't think my idea of always extracting .value is grandiose (and didn't take that long to implement on the Python side), but it is definitely forcing me to learn more about C and how Python is put together. It this point I have successfully imported Enum into the _json module; now I'm working on a function to quietly replace the object being encoded with its .value if it is, in fact, an Enum member. Just been real short on time lately. :/ Thank goodness for C API section of the docs! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Changes by Chris Rebert pyb...@rebertia.com: -- nosy: +cvrebert ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Guido van Rossum added the comment: The proposal to change json from using repr() to str() has unknown dangers. I don't want the str() of IntEnum to return just the decimal string (e.g. 42), since that breaks half of the usefulness of using the enum in the first place -- people will write print(x) and be confused. Unfortunately the json module doesn't have a way to define *in the object* how to customize its serialization -- this is always done in the json encoder/decoder. Maybe we can add something to the JSON encoder and decoder class that looks for a special method, e.g. __json_encode__ etc.? (OR maybe this would be a use case for PEP 443?) -- nosy: +gvanrossum ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Guido van Rossum added the comment: Modifying json to use str(int(value)) (if it detects isinstance(value, int)) is fine with me too. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Guido van Rossum added the comment: And similar for floats, really. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I'd be in favor of the __protocol__ route first and the PEP 443 route second. The problem with just tacking in `str(int(value))` or `str(float(value))` is where does it stop? StrEnum, TupleEnum, BytesEnum, ComplexEnum, SomeOtherInterestingConstantEnum, etc., etc.. If we go the __protocol__ route we add one method to Enum which returns the `str` of its `.value`, and we're set for every Enum.* As a bonus, any other object that needs special json (or yaml or whatever) consideration has a fairly easy way to make it happen. *Every normal Enum, anyway. ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Nick Coghlan added the comment: Can I vote for something like __builtin__ as the protocol, rather than something entirely specific to serialisation? As in return the most appropriate builtin type with the same value? Then a converter (operator.builtin?) could coerce builtin subclasses to their base classes by default, rather than needing to implement the protocol on every type. Such a design would need a PEP, of course. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Guido van Rossum added the comment: Hm. Then I prefer just calling the appropriate builtin, e.g. int(). --Guido van Rossum (sent from Android phone) On Jun 21, 2013 6:08 PM, Nick Coghlan rep...@bugs.python.org wrote: Nick Coghlan added the comment: Can I vote for something like __builtin__ as the protocol, rather than something entirely specific to serialisation? As in return the most appropriate builtin type with the same value? Then a converter (operator.builtin?) could coerce builtin subclasses to their base classes by default, rather than needing to implement the protocol on every type. Such a design would need a PEP, of course. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Barry A. Warsaw added the comment: On Jun 22, 2013, at 01:08 AM, Nick Coghlan wrote: Can I vote for something like __builtin__ as the protocol, rather than something entirely specific to serialisation? As in return the most appropriate builtin type with the same value? Then a converter (operator.builtin?) could coerce builtin subclasses to their base classes by default, rather than needing to implement the protocol on every type. Such a protocol needs a way to deserialize as well. You can't necessarily assume (or shouldn't impose) that the __init__() can do that conversion. In any case... Such a design would need a PEP, of course. ...yes, definitely. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eli Bendersky added the comment: On Fri, Jun 21, 2013 at 6:59 PM, Barry A. Warsaw rep...@bugs.python.orgwrote: Barry A. Warsaw added the comment: On Jun 22, 2013, at 01:08 AM, Nick Coghlan wrote: Can I vote for something like __builtin__ as the protocol, rather than something entirely specific to serialisation? As in return the most appropriate builtin type with the same value? Then a converter (operator.builtin?) could coerce builtin subclasses to their base classes by default, rather than needing to implement the protocol on every type. Such a protocol needs a way to deserialize as well. You can't necessarily assume (or shouldn't impose) that the __init__() can do that conversion. In any case... Such a design would need a PEP, of course. ...yes, definitely. Practically speaking, what should be done to make enum play well with JSON without writing new PEPs? I think we still want to convert those stdlib constants to IntEnums... Eli -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Guido van Rossum added the comment: Change json to call int() first. --Guido van Rossum (sent from Android phone) On Jun 21, 2013 7:45 PM, Eli Bendersky rep...@bugs.python.org wrote: Eli Bendersky added the comment: On Fri, Jun 21, 2013 at 6:59 PM, Barry A. Warsaw rep...@bugs.python.org wrote: Barry A. Warsaw added the comment: On Jun 22, 2013, at 01:08 AM, Nick Coghlan wrote: Can I vote for something like __builtin__ as the protocol, rather than something entirely specific to serialisation? As in return the most appropriate builtin type with the same value? Then a converter (operator.builtin?) could coerce builtin subclasses to their base classes by default, rather than needing to implement the protocol on every type. Such a protocol needs a way to deserialize as well. You can't necessarily assume (or shouldn't impose) that the __init__() can do that conversion. In any case... Such a design would need a PEP, of course. ...yes, definitely. Practically speaking, what should be done to make enum play well with JSON without writing new PEPs? I think we still want to convert those stdlib constants to IntEnums... Eli -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: On 06/21/2013 07:49 PM, Guido van Rossum wrote: Eli Bendersky added the comment: Practically speaking, what should be done to make enum play well with JSON without writing new PEPs? I think we still want to convert those stdlib constants to IntEnums... Change json to call int() first. Should we have json call float() and str() first as well? I'm thinking less of the stdlib and more of the unsuspecting user. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Guido van Rossum added the comment: Yes for float() -- but for str() it would seem redundant? (Or what's the context?) On Fri, Jun 21, 2013 at 8:23 PM, Ethan Furman rep...@bugs.python.org wrote: Ethan Furman added the comment: On 06/21/2013 07:49 PM, Guido van Rossum wrote: Eli Bendersky added the comment: Practically speaking, what should be done to make enum play well with JSON without writing new PEPs? I think we still want to convert those stdlib constants to IntEnums... Change json to call int() first. Should we have json call float() and str() first as well? I'm thinking less of the stdlib and more of the unsuspecting user. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Guido van Rossum added the comment: Yes for float() -- but for str() it would seem redundant? (Or what's the context?) If a user has class Color(StrEnum): red = 'ff' green = '00ff00' blue = 'ff' .. .. .. oh. `str()` isn't going to give is the `value`'s string, is it? Hmmm... Instead of calling int() or float() on an enum member, perhaps we could make json smart enough to use the `value`? That would also cover pure Enums. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Nick Coghlan added the comment: Whatever we do needs to be something third party serialisation libraries can also adopt with minimal compatibility risk for older versions of Python. Yes, that serialisation will lose the new debugging information. That's fine - if people want to map from a standard format like JSON to enums, they have to handle that themselves anyway. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
New submission from Nick Coghlan: Replacing an integer constant with the current incarnation of enum.IntEnum breaks JSON serialisation: from enum import Enum from enum import Enum, IntEnum class Example(IntEnum): ... x = 1 ... import json json.dumps(1) '1' json.loads(json.dumps(1)) 1 json.dumps(Example.x) 'Example.x' json.loads(json.dumps(Example.x)) Traceback (most recent call last): File stdin, line 1, in module File /home/ncoghlan/devel/py3k/Lib/json/__init__.py, line 316, in loads return _default_decoder.decode(s) File /home/ncoghlan/devel/py3k/Lib/json/decoder.py, line 344, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File /home/ncoghlan/devel/py3k/Lib/json/decoder.py, line 362, in raw_decode raise ValueError(errmsg(Expecting value, s, err.value)) from None ValueError: Expecting value: line 1 column 1 (char 0) It breaks for floats as well, but in a slightly different way: class FloatExample(float, Enum): ... x = 1.0 ... json.dumps(FloatExample.x) 'FloatExample.x: 1.0' Allowing __str__ to be inherited from Enum rather than from the concrete type will similarly break any serialisation protocol that relies on str() to handle integers. The float case is even trickier, since failing to inherit __repr__ would rather miss the point of the whole exercise... We're going to have to be *very* careful with swapping out existing constants with enums, and we should be warning about the pitfalls in the docs too (where we can't change enum to avoid them). -- components: Library (Lib) messages: 191467 nosy: ncoghlan priority: normal severity: normal status: open title: enum.IntEnum is not compatible with JSON serialisation versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Barry A. Warsaw added the comment: On Jun 19, 2013, at 01:45 PM, Nick Coghlan wrote: Allowing __str__ to be inherited from Enum rather than from the concrete type will similarly break any serialisation protocol that relies on str() to handle integers. The float case is even trickier, since failing to inherit __repr__ would rather miss the point of the whole exercise... Serialization isn't the only issue - you have to know how to deserialize as well. I use JSON extensively in Mailman's REST API. For enums I always encode them to just the member name, because I always know what enum class will be used to deserialize them. (TBH, I wish the json module had better hooks for extending both serialization and deserialization of non-basic types.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eric Snow added the comment: It's for times like this that I wonder if a simple serialization protocol might be worth it--something separate from __str__() but much simpler than the pickle protocol. __str__() could be the fallback. In the end, though, I always conclude that it's not worth adding yet another optional protocol to objects. Still, it would help in this case (at least for anything that knows to make use of the simple serialization protocol). Alternately, would it make sense to add special handling of enums to the json module? Do we already with pickle? -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Amaury Forgeot d'Arc added the comment: in encoder.py: ... elif instance(value, int): yield buf + str(value) ... What if we use int(str(value)) instead? -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Eric Snow added the comment: Serialization isn't the only issue - you have to know how to deserialize as well. +1 (TBH, I wish the json module had better hooks for extending both serialization and deserialization of non-basic types.) +1 There's at least one stdlib module that does this pretty well (sqlite3, I think). This is where a simple serialization (and deserialization of course) protocol would come in handy. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Nick, in your example it looks like json is using the __str__ for int, but the __repr__ for float -- is this correct? -- assignee: - ethan.furman nosy: +eli.bendersky, ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Barry A. Warsaw added the comment: On Jun 19, 2013, at 02:09 PM, Eric Snow wrote: There's at least one stdlib module that does this pretty well (sqlite3, I think). This is where a simple serialization (and deserialization of course) protocol would come in handy. Yeah, my database layer also has to be taught how to handle enums! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Antoine Pitrou added the comment: What if we use int(str(value)) instead? You mean str(int(value)). Sounds good to me. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Antoine Pitrou added the comment: Serialization isn't the only issue - you have to know how to deserialize as well. I think this is pretty much besides the point. IntEnums are meant to be substitutible with plain ints, so you can deserialize as a plain int. Moreoven, JSON doesn't fill the same use cases as pickle: it is meant to communicate with services written in any languages, and those services won't have the same enum mnemonics as your Python library. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Nick Coghlan added the comment: While I agree with forcing int subclasses to true integers in the JSON module, that may not be enough - the problem will affect third party serialisers as well. Whiel the debugging gains won't be as high, we may need to override __str__() in enum.IntEnum to give the raw form (leaving the debugging form solely for repr). As for the float behaviour, the JSON serialiser is almost certainly playing games to avoid the truncation that used to occur in float.__str__ (I don't recall if we actually got rid of that, or if it's just a lot harder to trigger these days - it definitely changed when float.__repr__ was updated to prefer human friendly representations that produce the same floating point number). I'm less concerned about that, since we don't plan to swap out any standard library floats for enums. We may still want to mention the potential compatibility issue somewhere in the docs, though. Backwards compatibility is such fun, isn't it? :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I have no problem with leaving __str__ as the inherited type's, and just keeping the __repr__ as the enum add-on; this could be one of the differences between a pure Enum and a hybrid Enum. Is it safe to change the json behaviour back to using __str__ for floats? If so, those two updates and we could be good to go. (Famous last words? ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: Eric, The pickle support is solely in Enum. No changes were made to pickles. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment: I was unable to find any references to previous problems with json and floats. A quick-n-dirty patch yields the following: -- from json import dumps, loads -- from enum import Enum -- class FE(float, Enum): ... pass ... -- class Test(FE): ... one = 1.0 ... -- Test.one Test.one: 1.0 -- str(Test.one) '1.0' -- dumps(Test.one) '1.0' -- loads(dumps(Test.one)) 1.0 All json and enum tests are still passing. If this is an acceptable solution I'll create a nicer patch and post for review. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Changes by Ethan Furman et...@stoneleaf.us: -- nosy: +ezio.melotti, rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18264 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18264] enum.IntEnum is not compatible with JSON serialisation
Ethan Furman added the comment:
Here's the relevant routine from _json.c:
-
static PyObject *
encoder_encode_float(PyEncoderObject *s, PyObject *obj)
{
/* Return the JSON representation of a PyFloat */
double i = PyFloat_AS_DOUBLE(obj);
if (!Py_IS_FINITE(i)) {
if (!s-allow_nan) {
PyErr_SetString(PyExc_ValueError, Out of range float values are
not JSON compliant);
return NULL;
}
if (i 0) {
return PyUnicode_FromString(Infinity);
}
else if (i 0) {
return PyUnicode_FromString(-Infinity);
}
else {
return PyUnicode_FromString(NaN);
}
}
/* Use a better float format here? */
return PyObject_Repr(obj);
}
Possible solutions
--
- Use PyObject_Str() instead of PyObject_Repr()
I was unable to find any references to why it isn't currently
PyObject_Str(), but switching over to it did not break any tests
- Coerce the obj to a PyFloat, and take the repr of that (just use the `i`)
float subclasses would always lose the subclass status, but that is lost on
deserialization anyway unless a custom decoder is used; and if a custom decoder
is being used I would think a custom encoder is also being used?
Summary
---
Having hybrid Enums not change __str__ would solve most of the json
serialization issues;
either of the above two changes will solve the json issue of enumerated floats.
Thoughts on which route to take for json?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18264
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
