[issue19837] Wire protocol encoding for the JSON module

Clay Gerrard added the comment: and for *encoding* case? Can you just add the encoding argument back to json.dumps? Have it default to None because of backwards compatibility in python3 and continue to return strings by default... ... and then *everyone* that ever wants to *serialize* an

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: For 3.6, the decoding case has been handled via Serhiy's autodetection patch in issue 17909 -- ___ Python tracker ___

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: After hitting this problem again in another nominally single-source compatible Python 2/3 project, I created #27765 to specifically cover accepting UTF-8 encoded bytes in json.loads() -- ___ Python tracker

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: dump_bytes() would be a binary counterpart to dumps() The dump() case is already handled more gracefully, as the implicit encoding to UTF-8 can live on the file-like object, rather than needing to be handled by the JSON encoder. I'm still not 100% sure on its

[issue19837] Wire protocol encoding for the JSON module

Serhiy Storchaka added the comment: Does dump_bytes() return bytes (similar to dumps()) or write to binary stream (similar to dump())? -- ___ Python tracker

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: I'm currently migrating a project that predates requests, and ended up needing to replace several "json.loads" calls with a "_load_json" helper that is just an alias for json.loads in Python 2, and defined as this in Python 3: def _load_json(data):

[issue19837] Wire protocol encoding for the JSON module

Марк Коренберг added the comment: In real life, I can confirm, that porting from Python2 to Python3 is almost automatic except JSON-related fixes. -- ___ Python tracker

[issue19837] Wire protocol encoding for the JSON module

Марк Коренберг added the comment: One of the problem, that decodeing JSON is FSM, where input is one symbol rather than one byte. AFAIK, Python still does not have FSM for decoding UTF-8 sequence, so iterative decoding of JSON will require more changes than expected. -- nosy: +mmarkk

[issue19837] Wire protocol encoding for the JSON module

Changes by Kevin Dwyer : -- nosy: +kdwyer ___ Python tracker ___ ___

[issue19837] Wire protocol encoding for the JSON module

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___ ___ Python-bugs-list

[issue19837] Wire protocol encoding for the JSON module

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___ ___ Python-bugs-list

[issue19837] Wire protocol encoding for the JSON module

Changes by Chris Rebert pyb...@rebertia.com: -- nosy: +cvrebert ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___ ___ Python-bugs-list

[issue19837] Wire protocol encoding for the JSON module

Changes by Josh Lee jlee...@gmail.com: -- nosy: +jleedev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___ ___ Python-bugs-list mailing list

[issue19837] Wire protocol encoding for the JSON module

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti type: - enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___

[issue19837] Wire protocol encoding for the JSON module

Gregory P. Smith added the comment: upstream simplejson (of which json is an earlier snapshot of) has an encoding parameter on its dump and dumps method. Lets NOT break compatibility with that API. Our users use these modules interchangeably today, upgrading from stdlib json to simplejson

[issue19837] Wire protocol encoding for the JSON module

Gregory P. Smith added the comment: So why not put a dump_bytes into upstream simplejson first, then pull in a modern simplejson? There might be some default flag values pertaining to new features that need changing for stdlib backwards compatible behavior but otherwise I expect it's a good

[issue19837] Wire protocol encoding for the JSON module

Changes by Éric Araujo mer...@netwok.org: -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___ ___ Python-bugs-list

[issue19837] Wire protocol encoding for the JSON module

Terry J. Reedy added the comment: Changing return type based on argument *values* is still a bad idea in general. I understand the proposal to be changing the return based on argument *presence*. It strikes me a a convenient abbreviation for making a separate encoding call and definitely

[issue19837] Wire protocol encoding for the JSON module

Antoine Pitrou added the comment: To give another data point: returning a different type based on argument value is also what the open() functions does, more or less. (that said, I would slightly favour a separate dump_bytes(), myself) -- ___

[issue19837] Wire protocol encoding for the JSON module

Serhiy Storchaka added the comment: Changing return type based on argument *values* is still a bad idea in general. However load() and loads() do this. ;) It also makes it hard to plug the API in to generic code that is designed to work with any dump/load based serialisation protocol. For

[issue19837] Wire protocol encoding for the JSON module

Antoine Pitrou added the comment: The problem with adding new APIs with different names to the JSON module is that it breaks symmetry with other wire protocols. The quartet of module level load, loads, dump and dumps functions has become a de facto standard API for wire protocols. Breaking

[issue19837] Wire protocol encoding for the JSON module

Serhiy Storchaka added the comment: I agree that adding a new module is very bad idea. I think that the reviving the encoding parameter is a lest wrong way. json.dumps() should return bytes when the encoding argument is specifiead and str otherwise. json.dump() should write binary data when

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: Changing return type based on argument *values* is still a bad idea in general. It also makes it hard to plug the API in to generic code that is designed to work with any dump/load based serialisation protocol. MvL suggested a json.bytes submodule (rather than a

[issue19837] Wire protocol encoding for the JSON module

Antoine Pitrou added the comment: MvL suggested a json.bytes submodule (rather than a separate top level module) in the other issue and that sounds reasonable to me, especially since json is already implemented as a package. I don't really find it reasonable to add a phantom module entirely

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: The parallel API would have to be: json.dump_bytes json.dumps_bytes json.load_bytes json.loads_bytes That is hardly an improvement over: json.bytes.dump json.bytes.dumps json.bytes.load json.bytes.loads It doesn't need to be documented as a completely separate

[issue19837] Wire protocol encoding for the JSON module

Antoine Pitrou added the comment: The parallel API would have to be: json.dump_bytes json.dumps_bytes json.load_bytes json.loads_bytes No, only one function dump_bytes() is needed, and it would return a bytes object (dumps meaning dump string, already). loads() can be polymorphic without

[issue19837] Wire protocol encoding for the JSON module

Antoine Pitrou added the comment: Still, JSON itself is not a wire protocol; HTTP is. http://www.json.org states that JSON is a text format and the grammar description talks UNICODE characters, not bytes. The ECMA spec states that JSON text is a sequence of Unicode code points. RFC 4627 is a

[issue19837] Wire protocol encoding for the JSON module

Serhiy Storchaka added the comment: I propose close this issue as a duplicate of issue10976. -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: Not sure yet if we should merge the two issues, although they're the serialisation and deserialisation sides of the same problem. Haskell seems to have gone with the approach of a separate jsonb API for the case where you want the wire protocol behaviour, such

[issue19837] Wire protocol encoding for the JSON module

Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837 ___ ___ Python-bugs-list mailing

[issue19837] Wire protocol encoding for the JSON module

Antoine Pitrou added the comment: I'm -1 for a new module doing almost the same thing. Let's add distinct APIs in the existing json module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19837

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: The problem with adding new APIs with different names to the JSON module is that it breaks symmetry with other wire protocols. The quartet of module level load, loads, dump and dumps functions has become a de facto standard API for wire protocols. If it wasn't

[issue19837] Wire protocol encoding for the JSON module

New submission from Nick Coghlan: In the Python 3 transition, we had to make a choice regarding whether we treated the JSON module as a text transform (with load[s] reading Unicode code points and dump[s] producing them), or as a text encoding (with load[s] reading binary sequences and

[issue19837] Wire protocol encoding for the JSON module

Nick Coghlan added the comment: The other simple solution would be to add nameb variants of the affected APIs. That's a bit ugly though, especially since it still has the problem of making it difficult to write correct cross-version code (although that problem is likely to exist regardless)