[Python-ideas] Re: Improvement: __json__
Hello, I'm not really sure how suggesting improvements to Python goes and I've found this mailing list, so I thought I'd email it. My suggestion is related to the json built in package, and more exactly about serialising custom classes. As of now, the process involves using a JSONEncoder class which, by default, supports the following types/values: dict, list, tuple, str, int, float, True, False, None and int- & float-derived Enums. In order to dump a custom class to JSON, the process involves creating a new class inheriting JSONEncoder and defining the default function in order to handle custom classes. I say that this approach is very complex and perhaps unpythonic, definitely very weird compared to other approaches. An alternative that comes to mind and would feel more natural is adding support for recognising a __json__ function. If a class would contain such a function, the default JSONEncoder should take that function's return value instead for encoding, assuming that such returned value will be able to be then serialised. If such a functionality would be added, many custom functions will most likely return a dict from their __json__ function, skipping the need for creating a class inheriting JSONEncoder altogether. I hope this idea is a good one and could perhaps even get implemented in the next version of Python. Respectfully, Dan Cojocaru. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B5RUXM2ENUWEISOSDMPIZF25OH4NUAZY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
This thread discusses a __json__ encoder protocol: "adding support for a "raw output" in JSON serializer" https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJDEZXKQ6OQLGAPB3OZ4OHCTPDU/ You actually don't have to subclass JSONEncoder; you can specify a default= method that uses isinstance(), hasattr(), or singledispatch. https://simplejson.readthedocs.io/en/latest/ - "Specializing JSON object encoding:" - https://simplejson.readthedocs.io/en/latest/#simplejson.dump > If for_json is true (not the default), objects with a for_json() method will use the return value of that method for encoding as JSON instead of the object. "use __json__ attribute to encode custom objects" https://github.com/simplejson/simplejson/issues/52 Simplejson supports object.for_json(), but only checks objects if for_json=True is specified. FWIW, Jupyter notebook supports object._repr_json_() and e.g. object._repr_pretty_() Dunder methods are name-mangled, and, by convention reserved for internal Python use: if CPython decides that object.__json__() should return a bool for whether a string is already JSON-encoded, and that breaks your excessively-presumptive dunder json method implementation, tough luck. Singledispatch (or, indeed, creating your own utils.json.dump that's a functools.partial that just specifies a default default=) is probably fastest. JSON5 allows things like ±inf and nan. On Mon, Apr 6, 2020, 9:32 AM Dan Cojocaru wrote: > Hello, > > I'm not really sure how suggesting improvements to Python goes and I've > found this mailing list, so I thought I'd email it. > > My suggestion is related to the json built in package, and more exactly > about serialising custom classes. > > As of now, the process involves using a JSONEncoder class which, by > default, supports the following types/values: dict, list, tuple, str, int, > float, True, False, None and int- & float-derived Enums. > > In order to dump a custom class to JSON, the process involves creating a > new class inheriting JSONEncoder and defining the default function in order > to handle custom classes. > > I say that this approach is very complex and perhaps unpythonic, > definitely very weird compared to other approaches. > > An alternative that comes to mind and would feel more natural is adding > support for recognising a __json__ function. If a class would contain such > a function, the default JSONEncoder should take that function's return > value instead for encoding, assuming that such returned value will be able > to be then serialised. > > If such a functionality would be added, many custom functions will most > likely return a dict from their __json__ function, skipping the need for > creating a class inheriting JSONEncoder altogether. > > I hope this idea is a good one and could perhaps even get implemented in > the next version of Python. > > Respectfully, > Dan Cojocaru. > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/B5RUXM2ENUWEISOSDMPIZF25OH4NUAZY/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NXZBVOWQA7Q2HHLY53LC7KRNZE3RGMK5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote: > This thread discusses a __json__ encoder protocol: > > "adding support for a "raw output" in JSON serializer" > > https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJDEZXKQ6OQLGAPB3OZ4OHCTPDU/ > and a numbe rof other issues :-) This would be a really good time for someone to go through this list and maybe others), and summarize all the discussions around __json__ FWIW, the idea of a __json__ protocol has support, but it's not clear to me if there wasn't interest by core devs, or if it's just that no one has cleaned the discussion up and made it into a clean proposal to be considered. Note that there IS notable overhead to adding a new dunder like that -- there will be (legitimate) resistance. > JSON5 allows things like ±inf and nan. For my part, I would REALLY like to see the stdlib json module support JSON5 -- it better matches Python. (inf, comments, and trailing commas -- ALL things I really miss with JSON) Second on my list would be support for Decimal, as that matches teh JSON data model (nothing in there about binary floating point). As for __json__ -- I get it, but I think I'm -0.5: because it's a one way street -- of you want to go from custom object => JSON, iot's great, but the other way doesn't work, so you'll need to do SOMETHING custom there anyway. So having the JSON module fully support a mapping between the JSON types and Python seems enough to me. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HKCKCOD6MY4CXN6OXAUGXLQK7H5AQTLK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
Addressing your last concern, about __json__ being only class -> JSON, not JSON -> class, classes implementing __str__ only go class -> str, not str -> class. Just because we can't fully solve a problem, it doesn't mean we shouldn't attempt to solve (or provide a more convenient to solve) half of it. --- Dan Cojocaru On 6 Apr 2020, 19:02 +0300, Christopher Barker , wrote: > On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote: > > > This thread discusses a __json__ encoder protocol: > > > > > > "adding support for a "raw output" in JSON serializer" > > > https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJDEZXKQ6OQLGAPB3OZ4OHCTPDU/ > > > > and a numbe rof other issues :-) > > > > This would be a really good time for someone to go through this list and > > maybe others), and summarize all the discussions around __json__ > > > > FWIW, the idea of a __json__ protocol has support, but it's not clear to me > > if there wasn't interest by core devs, or if it's just that no one has > > cleaned the discussion up and made it into a clean proposal to be > > considered. > > > > Note that there IS notable overhead to adding a new dunder like that -- > > there will be (legitimate) resistance. > > > > > JSON5 allows things like ±inf and nan. > > > > For my part, I would REALLY like to see the stdlib json module support > > JSON5 -- it better matches Python. (inf, comments, and trailing commas -- > > ALL things I really miss with JSON) > > > > Second on my list would be support for Decimal, as that matches teh JSON > > data model (nothing in there about binary floating point). > > > > As for __json__ -- I get it, but I think I'm -0.5: because it's a one way > > street -- of you want to go from custom object => JSON, iot's great, but > > the other way doesn't work, so you'll need to do SOMETHING custom there > > anyway. So having the JSON module fully support a mapping between the JSON > > types and Python seems enough to me. > > > > -CHB > > > > > > -- > Christopher Barker, PhD > > Python Language Consulting > - Teaching > - Scientific Software Development > - Desktop GUI and Web Development > - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JYQTHWN7XAANVZ6XFP7YR5LT5ISOHCTU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Mon, Apr 6, 2020 at 9:18 AM Dan Cojocaru wrote: > Addressing your last concern, about __json__ being only class -> JSON, not > JSON -> class, classes implementing __str__ only go class -> str, not str > -> class. Just because we can't fully solve a problem, it doesn't mean we > shouldn't attempt to solve (or provide a more convenient to solve) half of > it. > That's why I'm only -0.5 :-) But __str__ is pretty key to a lot of Python usage, for every type of object, etc. JSON is a far more specialized use case. Which I think is the argument against it: it's adding overhead to a lot, in order to support one particular special use case. Anyway, my primary suggestion at this point, if someone wants to pursue this, is to go back and gather all the arguments for and against that have already been made -- there really has been a LOT of discussion about this already! -CHB > > --- > Dan Cojocaru > On 6 Apr 2020, 19:02 +0300, Christopher Barker , > wrote: > > On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote: > >> This thread discusses a __json__ encoder protocol: >> >> "adding support for a "raw output" in JSON serializer" >> >> https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJDEZXKQ6OQLGAPB3OZ4OHCTPDU/ >> > > and a numbe rof other issues :-) > > This would be a really good time for someone to go through this list and > maybe others), and summarize all the discussions around __json__ > > FWIW, the idea of a __json__ protocol has support, but it's not clear to > me if there wasn't interest by core devs, or if it's just that no one has > cleaned the discussion up and made it into a clean proposal to be > considered. > > Note that there IS notable overhead to adding a new dunder like that -- > there will be (legitimate) resistance. > > > JSON5 allows things like ±inf and nan. > > For my part, I would REALLY like to see the stdlib json module support > JSON5 -- it better matches Python. (inf, comments, and trailing commas -- > ALL things I really miss with JSON) > > Second on my list would be support for Decimal, as that matches teh JSON > data model (nothing in there about binary floating point). > > As for __json__ -- I get it, but I think I'm -0.5: because it's a one way > street -- of you want to go from custom object => JSON, iot's great, but > the other way doesn't work, so you'll need to do SOMETHING custom there > anyway. So having the JSON module fully support a mapping between the JSON > types and Python seems enough to me. > > -CHB > > > > -- > Christopher Barker, PhD > > Python Language Consulting > - Teaching > - Scientific Software Development > - Desktop GUI and Web Development > - wxPython, numpy, scipy, Cython > > -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZYUBQCAFHL2WHEVFVEMOM5JS6AFZDDEK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
Python object > JSON > object requires type information to be serialized.
Jsonpickle is two-way; with support for numpy, pandas, JS, datetimes,
UUIDs, and IDK about e.g. complex fractions
https://github.com/jsonpickle/jsonpickle
JSONLD can specify type information in a space-efficient way with @context
(rather than having to specify e.g. @type=xsd:boolean for every boolean
value)
JSONLD includes @type URIs that can be e.g. XSD datatypes or
https://schema.org types.
- JSON is an ECMA spec.
- jsonpickle has a python and a js implementation
- JSON5 is an unofficial spec with many implementations
- JSONLD 1.1 is a W3C candidate recommendation with many implementations
(pyld, rdflib-jsonld, )
In order to optionally serialize to any of {JSON, JSON5, JSONLD,
JSON_future}, we could:
A) accept an optional parameter on a __json__ method: __json__(obj,
spec=None)
B) establish a convention for registering functions to call with
singledispatch in a default method
?) Other ideas?
On Mon, Apr 6, 2020, 12:22 PM Dan Cojocaru wrote:
> Addressing your last concern, about __json__ being only class -> JSON, not
> JSON -> class, classes implementing __str__ only go class -> str, not str
> -> class. Just because we can't fully solve a problem, it doesn't mean we
> shouldn't attempt to solve (or provide a more convenient to solve) half of
> it.
>
> ---
> Dan Cojocaru
> On 6 Apr 2020, 19:02 +0300, Christopher Barker ,
> wrote:
>
> On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote:
>
>> This thread discusses a __json__ encoder protocol:
>>
>> "adding support for a "raw output" in JSON serializer"
>>
>> https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJDEZXKQ6OQLGAPB3OZ4OHCTPDU/
>>
>
> and a numbe rof other issues :-)
>
> This would be a really good time for someone to go through this list and
> maybe others), and summarize all the discussions around __json__
>
> FWIW, the idea of a __json__ protocol has support, but it's not clear to
> me if there wasn't interest by core devs, or if it's just that no one has
> cleaned the discussion up and made it into a clean proposal to be
> considered.
>
> Note that there IS notable overhead to adding a new dunder like that --
> there will be (legitimate) resistance.
>
> > JSON5 allows things like ±inf and nan.
>
> For my part, I would REALLY like to see the stdlib json module support
> JSON5 -- it better matches Python. (inf, comments, and trailing commas --
> ALL things I really miss with JSON)
>
> Second on my list would be support for Decimal, as that matches teh JSON
> data model (nothing in there about binary floating point).
>
> As for __json__ -- I get it, but I think I'm -0.5: because it's a one way
> street -- of you want to go from custom object => JSON, iot's great, but
> the other way doesn't work, so you'll need to do SOMETHING custom there
> anyway. So having the JSON module fully support a mapping between the JSON
> types and Python seems enough to me.
>
> -CHB
>
>
>
> --
> Christopher Barker, PhD
>
> Python Language Consulting
> - Teaching
> - Scientific Software Development
> - Desktop GUI and Web Development
> - wxPython, numpy, scipy, Cython
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/JYQTHWN7XAANVZ6XFP7YR5LT5ISOHCTU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/KNJQ3NSLV42VINXGZQYY3EKR5G7RKCJ4/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Apr 6, 2020, at 09:22, Dan Cojocaru wrote: > > > Addressing your last concern, about __json__ being only class -> JSON, not > JSON -> class, classes implementing __str__ only go class -> str, not str -> > class. Sure, but str is inherently one-way. There’s no unambiguous mapping from human-readable text to data. That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now(). (Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/75MERXNMJGUUPJXHPRHHSZFSL4K4UGSH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
I totally agree, but the JSONEncoder/JSONDecoder mechanism already exist. I'm certainly removing stuff. I'm just proposing adding a simpler way to do one of the things - serialising. An example use case would be creating a simple class for something like a REST API and that class having a __json__ function. Python doesn't need to deserialise the JSON, so you only need one half of the story, the half that __json__ would make easier. --- Dan Cojocaru On 6 Apr 2020, 21:48 +0300, Andrew Barnert , wrote: > On Apr 6, 2020, at 09:22, Dan Cojocaru wrote: > > > > > > Addressing your last concern, about __json__ being only class -> JSON, not > > JSON -> class, classes implementing __str__ only go class -> str, not str > > -> class. > > Sure, but str is inherently one-way. There’s no unambiguous mapping from > human-readable text to data. > > That’s not true for JSON; the entire point of it is data interchange. You > expect to be able to dump an object, send it over the wire or store it to a > file, load it (or even parse it in JS or ObjC or Go or whatever) and get back > an equivalent object. It’s easy to come up with ways to build on top of JSON > to interchange things like time points or raw binary strings or higher-level > structured objects, but they require doing something on both the encode side > and the decode side. Just being able to encode them to something > human-readable is useless—if I encode a datetime object, I need to get back a > datetime (or Date or NSDate or whatever) on the other end, not a str (or > string or NSString or whatever) that a human could tell is probably meant to > be a datetime but will raise an exception when I try to subtract it from > now(). > > (Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, > int64 values discovers… but it’s good enough for many applications.) > > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6YUK427KZC7LFI5NNPYH7A622CR4OT4J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas wrote: > That’s not true for JSON; the entire point of it is data interchange. You > expect to be able to dump an object, send it over the wire or store it to a > file, load it (or even parse it in JS or ObjC or Go or whatever) and get back > an equivalent object. It’s easy to come up with ways to build on top of JSON > to interchange things like time points or raw binary strings or higher-level > structured objects, but they require doing something on both the encode side > and the decode side. Just being able to encode them to something > human-readable is useless—if I encode a datetime object, I need to get back a > datetime (or Date or NSDate or whatever) on the other end, not a str (or > string or NSString or whatever) that a human could tell is probably meant to > be a datetime but will raise an exception when I try to subtract it from > now(). > > (Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, > int64 values discovers… but it’s good enough for many applications.) > The trouble with using JSON to interchange things that aren't in the JSON spec is that you then have to build another layer on top of it - a non-standard layer. That means that whatever you do, it's inherently app-specific. For instance, I might choose to encode a datetime as a string in ISO format, but how is the other end going to know that it was meant to be a date? (Usually, if I'm sending something to some front-end JavaScript, I'll just hard-code the JS to know that thing[0].created is a timestamp, and should be parsed accordingly.) So if it's app-specific, then the best way to handle it is in your app, not in the data type you're encoding. Subclassing JSONEncoder works for this; adding a __json__ method doesn't really, unless there is some single canonical encoding for a particular object. The two become close to equivalent when you're only asking about your own custom classes. You can, in fact, create your own private __json__ protocol (although, since it's private to you, it'd probably be better to call the method "to_json" rather than "__json__"), and have a subclass of JSONEncoder that calls it. It'd work fine because you don't NEED to interoperate with other libraries. It's only when you try to standardize something that's inherently nonstandard that things get problematic :) ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4ZEQKTQ5CMZUXC6E6JVFHQS3VGYEDJ7C/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
I don't see why the need for standardisation exists. Here's another example because I can explain myself best by giving examples: I have an API. I can write some classes that correspond to JSON inputs my API expects and define __json__ in them to convey how they are expected to be serialised. For example, a Person class with first_name and last_name fields being serialised as a dict with the fields in camel case instead. (Sure, normally I'd write a serialisation function as well as part of my package in this case but I'm just coming up with examples on the spot). For non standard things like date and time, many implementations of JSON already added support for it using a format I can't recall at the moment. And even if a class defines __json__ in a particular format but you would need another one, the method of making a custom JSONEncoder still remains, as __json__ functions would be attempted only after the custom default implementation. Therefore, __json__ would enable sensible defaults to be chosen while not harming the possibility of using a custom format if it is so desired. --- Dan Cojocaru On 6 Apr 2020, 22:08 +0300, Chris Angelico , wrote: > On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas > wrote: > > That’s not true for JSON; the entire point of it is data interchange. You > > expect to be able to dump an object, send it over the wire or store it to a > > file, load it (or even parse it in JS or ObjC or Go or whatever) and get > > back an equivalent object. It’s easy to come up with ways to build on top > > of JSON to interchange things like time points or raw binary strings or > > higher-level structured objects, but they require doing something on both > > the encode side and the decode side. Just being able to encode them to > > something human-readable is useless—if I encode a datetime object, I need > > to get back a datetime (or Date or NSDate or whatever) on the other end, > > not a str (or string or NSString or whatever) that a human could tell is > > probably meant to be a datetime but will raise an exception when I try to > > subtract it from now(). > > > > (Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, > > int64 values discovers… but it’s good enough for many applications.) > > > > The trouble with using JSON to interchange things that aren't in the > JSON spec is that you then have to build another layer on top of it - > a non-standard layer. That means that whatever you do, it's inherently > app-specific. For instance, I might choose to encode a datetime as a > string in ISO format, but how is the other end going to know that it > was meant to be a date? (Usually, if I'm sending something to some > front-end JavaScript, I'll just hard-code the JS to know that > thing[0].created is a timestamp, and should be parsed accordingly.) > > So if it's app-specific, then the best way to handle it is in your > app, not in the data type you're encoding. Subclassing JSONEncoder > works for this; adding a __json__ method doesn't really, unless there > is some single canonical encoding for a particular object. > > The two become close to equivalent when you're only asking about your > own custom classes. You can, in fact, create your own private __json__ > protocol (although, since it's private to you, it'd probably be better > to call the method "to_json" rather than "__json__"), and have a > subclass of JSONEncoder that calls it. It'd work fine because you > don't NEED to interoperate with other libraries. It's only when you > try to standardize something that's inherently nonstandard that things > get problematic :) > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/4ZEQKTQ5CMZUXC6E6JVFHQS3VGYEDJ7C/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7LTMRW3IL4A4U3VYDZBZEXBLGOMTSFH4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Apr 6, 2020, at 11:54, Dan Cojocaru wrote: > > > I totally agree, but the JSONEncoder/JSONDecoder mechanism already exist. I'm > certainly removing stuff. I'm just proposing adding a simpler way to do one > of the things - serialising. An example use case would be creating a simple > class for something like a REST API and that class having a __json__ > function. Python doesn't need to deserialise the JSON, so you only need one > half of the story, the half that __json__ would make easier. For the vast majority of people writing web services, they’re already using a bunch of third-party libs. If they’re not using a framework that does its own JSON-izing, they can pick simplejson or some other third-party library they prefer. What about people who are writing a web service with no third-party libraries? Well, I think that’s a pretty rare use case, and a pretty specialized/expert one. And what’s the cost for them today? It takes 3 lines of code In your handler to call the __json__ method manually. And there’s a lot more flexibility. Have a bunch of already-written classes that you don’t want to edit? Use @singledispatch instead of a method protocol. Have a bunch of existing code from another project that used simplejson? Use its tojson protocol, and you don’t need to fork every class just to change the name. And so on. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NWEQYAIINMAX6ZOJKY2A7XDUPLJVK5YF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Mon, Apr 6, 2020, 3:37 PM Dan Cojocaru wrote: > I don't see why the need for standardisation exists. > > Here's another example because I can explain myself best by giving > examples: > > I have an API. I can write some classes that correspond to JSON inputs my > API expects and define __json__ in them to convey how they are expected to > be serialised. For example, a Person class with first_name and last_name > fields being serialised as a dict with the fields in camel case instead. > (Sure, normally I'd write a serialisation function as well as part of my > package in this case but I'm just coming up with examples on the spot). > DRF (Django REST API framework) serializers have a .to_representation() method: > Takes the object instance that requires serialization, and should return a primitive representation. Typically this means returning a structure of built-in Python datatypes. The exact types that can be handled will depend on the render classes you have configured for your API. > > May be overridden in order to modify the representation style https://www.django-rest-framework.org/api-guide/serializers/#overriding-serialization-and-deserialization-behavior > For non standard things like date and time, many implementations of JSON > already added support for it using a format I can't recall at the moment. > And even if a class defines __json__ in a particular format but you would > need another one, the method of making a custom JSONEncoder still remains, > as __json__ functions would be attempted only after the custom default > implementation. Therefore, __json__ would enable sensible defaults to be > chosen while not harming the possibility of using a custom format if it is > so desired. > In trying to do this (again), I've realized that you *can't* just check for hasattr(obj, '__json__') in a JSONEncoder.default method because default only get's called for types it doesn't know how to serialize; so if you e.g. subclass a dict, default() never gets called for the dict subclass. https://docs.python.org/3/library/json.html : > If specified, default should be a function that gets called for objects that can’t otherwise be serialized. https://github.com/python/cpython/blob/master/Lib/json/encoder.py https://stackoverflow.com/questions/16405969/how-to-change-json-encoding-behaviour-for-serializable-python-object/17684652#17684652 specifies how to overload _make_iterencode *in pure Python*, but AFAICS that NOPs use of the C-optimized json encoder. The python json module originally came from simplejson. Here's simplejson's for_json implementation in pure Python: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8ebcb82eee72/simplejson/encoder.py#L515 : for_json = _for_json and getattr(value, 'for_json', None) > if for_json and callable(for_json): > chunks = _iterencode(for_json(), _current_indent_level) > And simplejson's for_json implementation in C: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8ebcb82eee72/simplejson/_speedups.c#L2839 Is there a strong reason that the method would need to be called __json__ instead of 'for_json'? Passing a spec=None kwarg through to the __json__()/for_json() method would not be compatible with simplejson's existing implementation. Passing a spec=None kwarg would be necessary to support different JSON standards within the same method; which I think is desirable because JSON/JSON5/JSONLD/JSON_future. > > --- > Dan Cojocaru > On 6 Apr 2020, 22:08 +0300, Chris Angelico , wrote: > > On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas > wrote: > > That’s not true for JSON; the entire point of it is data interchange. You > expect to be able to dump an object, send it over the wire or store it to a > file, load it (or even parse it in JS or ObjC or Go or whatever) and get > back an equivalent object. It’s easy to come up with ways to build on top > of JSON to interchange things like time points or raw binary strings or > higher-level structured objects, but they require doing something on both > the encode side and the decode side. Just being able to encode them to > something human-readable is useless—if I encode a datetime object, I need > to get back a datetime (or Date or NSDate or whatever) on the other end, > not a str (or string or NSString or whatever) that a human could tell is > probably meant to be a datetime but will raise an exception when I try to > subtract it from now(). > > (Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, > int64 values discovers… but it’s good enough for many applications.) > > > The trouble with using JSON to interchange things that aren't in the > JSON spec is that you then have to build another layer on top of it - > a non-standard layer. That means that whatever you do, it's inherently > app-specific. For instance, I might choose to encode a datetime as a > string in ISO format, but how is the other end going to know that it > w
[Python-ideas] Re: Improvement: __json__
On 7/04/20 4:57 am, Wes Turner wrote: Python object > JSON > object requires type information to be serialized. Not necessarily -- Java's json library uses reflection on compile time type information to deserialise json into an object hierarchy. You tell it the class corresponding to the top level and it figures out the rest. Something similar could be done in Python using type annotations. -- Greg ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/C4DMLEWXAOQFDTKK7HXQSWQHW4MXDOEK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
Type information (the TBox) can be serialized along with instances or in a separate schema definition. https://en.wikipedia.org/wiki/Tbox On Tue, Apr 7, 2020, 12:17 AM Greg Ewing wrote: > On 7/04/20 4:57 am, Wes Turner wrote: > > Python object > JSON > object requires type information to be > serialized. > > Not necessarily -- Java's json library uses reflection on compile > time type information to deserialise json into an object hierarchy. > You tell it the class corresponding to the top level and it figures > out the rest. > > Something similar could be done in Python using type annotations. > > -- > Greg > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/C4DMLEWXAOQFDTKK7HXQSWQHW4MXDOEK/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/S7B24ISDYEEMEA6FKNK35XQA7WOLTNXQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
As the author of one of these third-party libraries, I feel like I can contribute to this discussion. It can indeed be done very elegantly with type annotations, and it should for sure be left to the ecosystem. The only things we need from core Python are good tools for dealing with run-time type information. For example, singledispatch doesn't really work with types (i.e. Optionals, Unions, Sequences as opposed to actual classes), but all of that can be worked around. In my experience, runtime type information is extremely useful exactly in cases of deserialization, and for big projects starts being useful much before the type information starts being useful in a static analysis context. > From: Greg Ewing > Subject: [Python-ideas] Re: Improvement: __json__ > To: python-ideas@python.org > Message-ID: > Content-Type: text/plain; charset=utf-8; format=flowed > > On 7/04/20 4:57 am, Wes Turner wrote: > > Python object > JSON > object requires type information to be > serialized. > > Not necessarily -- Java's json library uses reflection on compile > time type information to deserialise json into an object hierarchy. > You tell it the class corresponding to the top level and it figures > out the rest. > > Something similar could be done in Python using type annotations. > > -- > Greg > > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ISSUQVYI5OYYXKELUNCD5YCEDZ75LCEB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
Would you generate a schema from the type annotations so that other languages can use the data? JSON is really not the most efficient data representation for round-tripping to and from Python. A binary format (or pickle without arbitrary code execution) would solve for Python object > file > Python object. IMHO, JSON is most useful when the data needs to be read/written from (browser-side) JS, too. And then you need data validation for user-supplied input (which is not distinct from the deserialization problem). E.g. DRF has data validation. But (mypy) type annotations are insufficient for data validation: compare JSONschema and the maximum amount of information storable in mypy-compatible type annotations, for example. Data validation needs to return errors in a feedback loop with the user, so it's sort of more than just the preconditions that need to be satisfied before a function proceeds. - serialization from Python (TA useful) - deserialization with the same Python code (TA useful) - deserialization with other Python code (insufficient) - deserialization with other languages (insufficient) - form generation (insufficient) - data validation (insufficient) - preconditions (insufficient) So, IMHO type annotations are not insufficient and thus redundant and not elegant. On Tue, Apr 7, 2020, 11:45 AM Tin Tvrtković wrote: > As the author of one of these third-party libraries, I feel like I can > contribute to this discussion. It can indeed be done very elegantly with > type annotations, and it should for sure be left to the ecosystem. > > The only things we need from core Python are good tools for dealing with > run-time type information. For example, singledispatch doesn't really work > with types (i.e. Optionals, Unions, Sequences as opposed to actual > classes), but all of that can be worked around. > > In my experience, runtime type information is extremely useful exactly in > cases of deserialization, and for big projects starts being useful much > before the type information starts being useful in a static analysis > context. > > >> From: Greg Ewing >> Subject: [Python-ideas] Re: Improvement: __json__ >> To: python-ideas@python.org >> Message-ID: >> Content-Type: text/plain; charset=utf-8; format=flowed >> >> On 7/04/20 4:57 am, Wes Turner wrote: >> > Python object > JSON > object requires type information to be >> serialized. >> >> Not necessarily -- Java's json library uses reflection on compile >> time type information to deserialise json into an object hierarchy. >> You tell it the class corresponding to the top level and it figures >> out the rest. >> >> Something similar could be done in Python using type annotations. >> >> -- >> Greg >> >> ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/ISSUQVYI5OYYXKELUNCD5YCEDZ75LCEB/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LQKGBUGU4R5V2HYBKJHKBNFVR5VK7WCA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Mon, Apr 6, 2020 at 9:17 PM Greg Ewing wrote: > > > Something similar could be done in Python using type annotations. > sure can -- I've done this wrapping dataclasses -- each object in the hierarchy knows how to serialize/deserialize itself from JSON -- so there JSON can be totally standard. -CHB > > -- > Greg > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/C4DMLEWXAOQFDTKK7HXQSWQHW4MXDOEK/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6YDV54WX27MMV7W4WFQ2VBFG7LR7SBMA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Tue, Apr 7, 2020 at 11:17 AM Wes Turner wrote: > Would you generate a schema from the type annotations so that other > languages can use the data? > I haven't done this yet, but it would be pretty cool. > So, IMHO type annotations are not insufficient and thus redundant and not elegant. you mean are not sufficient / or are insufficient? But what is meant by "type annotations"? I'm using them via dataclasses -- really as a shorthand for assigning a type to every field -- the annotations are just a shorthand that auto-generates a schema, essentially. @dataclass class MyClass: x: A_Type = A_default So now I know that this class has a field names x that is the type int. So I use that type for validation, and serialization / deserialization. But if you mean "type annotations" in the sense of the types provided out of the box in the typing module and used by MyPy (so I've heard, never did it myself) -- no, they are not sufficient -- I need types that support my serialization / deserialization system, and my validation system. And I suppose we could have a standardized __json__ and __from_json__ protocol that I could use, but it seems a special case to me. Note: I don't need to do anything special for types with standard json representation, but that's only the basics -- I end up using custom types for anything nested. -CHB > > On Tue, Apr 7, 2020, 11:45 AM Tin Tvrtković wrote: > >> As the author of one of these third-party libraries, I feel like I can >> contribute to this discussion. It can indeed be done very elegantly with >> type annotations, and it should for sure be left to the ecosystem. >> >> The only things we need from core Python are good tools for dealing with >> run-time type information. For example, singledispatch doesn't really work >> with types (i.e. Optionals, Unions, Sequences as opposed to actual >> classes), but all of that can be worked around. >> >> In my experience, runtime type information is extremely useful exactly in >> cases of deserialization, and for big projects starts being useful much >> before the type information starts being useful in a static analysis >> context. >> >> >>> From: Greg Ewing >>> Subject: [Python-ideas] Re: Improvement: __json__ >>> To: python-ideas@python.org >>> Message-ID: >>> Content-Type: text/plain; charset=utf-8; format=flowed >>> >>> On 7/04/20 4:57 am, Wes Turner wrote: >>> > Python object > JSON > object requires type information to be >>> serialized. >>> >>> Not necessarily -- Java's json library uses reflection on compile >>> time type information to deserialise json into an object hierarchy. >>> You tell it the class corresponding to the top level and it figures >>> out the rest. >>> >>> Something similar could be done in Python using type annotations. >>> >>> -- >>> Greg >>> >>> ___ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-le...@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/ISSUQVYI5OYYXKELUNCD5YCEDZ75LCEB/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/LQKGBUGU4R5V2HYBKJHKBNFVR5VK7WCA/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PMF2GR6FTO2ZXY3DUP4MSSM4EZJ7WTIF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Apr 7, 2020, at 15:31, Christopher Barker wrote:
>
>> On Tue, Apr 7, 2020 at 11:17 AM Wes Turner wrote:
>
>> Would you generate a schema from the type annotations so that other
>> languages can use the data?
>
> I haven't done this yet, but it would be pretty cool.
This seems like one of the many things that’s impossible to do for Python
classes with full generality, but pretty easy to do if you only want to support
@dataclass. Either dynamically or statically, in fact. You could even write a
tool that generates dataclasses (statically or dynamically) from a schema, if
you wanted.
> But what is meant by "type annotations"? I'm using them via dataclasses --
> really as a shorthand for assigning a type to every field -- the annotations
> are just a shorthand that auto-generates a schema, essentially.
>
> @dataclass
> class MyClass:
> x: A_Type = A_default
>
> So now I know that this class has a field names x that is the type int. So I
> use that type for validation, and serialization / deserialization.
>
> But if you mean "type annotations" in the sense of the types provided out of
> the box in the typing module and used by MyPy (so I've heard, never did it
> myself) -- no, they are not sufficient -- I need types that support my
> serialization / deserialization system, and my validation system. And I
> suppose we could have a standardized __json__ and __from_json__ protocol that
> I could use, but it seems a special case to me.
A type annotation is just the “: whatever”. It doesn’t matter whether that
whatever is a real dynamic type like int or spam.Eggs or list, or a typing.type
like List[int], it’s still an annotation. And dataclass can handle
either—change that to “x: List[A_Type]” and at runtime, the dataclass will
treat that the same as if you just used list, but mypy can know that it’s only
supposed to have A_Type members in that list. (So if you initialize MyClass([1,
“spam”, open(“eggs.txt”)]) it’ll work at runtime, but mypy will flag it as a
type error.)
An “automated JSON serialization for dataclasses” library could do either. It
could be “dumb” like @dataclass and just treat x as a list of anything
serializable, or it could be “smart” and treat it as a list of A_Type objects
only. Either way seems like it makes sense.
A schema generator for dataclasses, I think you’d want it to use the typing
information. A MyClass property a doesn’t just have an attribute of {“type”:
“array”}; it has a {“type”: “array”, “contains”:
recursively_schematize(A_Type)}.___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/2DWJHNP76SXPZQWVCN5TX7QRL7VIXMB2/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
*That* should read as "are not sufficient".
Stuffing all of those into annotations is going to be cumbersome; resulting
in there being multiple schema definitions to keep synchronized and
validate data according to.
I think generating some amalgamation of JSONLD @context & SHACL and JSON
schema would be an interesting exercise. You'd certainly want to add more
information to the generated schema than just the corresponding Python
types :
- type URI(s) that correspond to the Python primitive types
- JSON schema format strings
- JSON schema length
- TODO JSON schema [...]
You could replace the first comma in this with a colon and call that a
validatable type annotation:
attrname, pytype, type URI, JSONschema format, validators
name, str, xsd:string, string
url, str, xsd:anyURI, uri
dateCreated, datetime.datetime, xsd:dateTime, date-time, regex_iso8601
author_email, str, xsd:string, email
username, str, xsd:string, string, {length: {minLength:3, maxLength:32}}
Stuffing all of those into annotations is going to be cumbersome; resulting
in there being multiple schema definitions to keep synchronized and
validate data according to.
https://docs.python.org/3/reference/datamodel.html
https://json-schema.org/understanding-json-schema/reference/string.html
https://www.w3.org/TR/json-ld11/#the-context
https://www.w3.org/TR/json-ld11/#advanced-context-usage
https://www.w3.org/TR/shacl/
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.3.md
(JSON schema)
Similarly unnecessarily redundant: having a models.py, forms.py, api.py,
and an OpenAPI specification that includes the JSON schema (that DRF
*tries* to generate from api.py)
https://www.django-rest-framework.org/api-guide/schemas/
> API schemas are a useful tool that allow for a range of use cases,
including generating reference documentation, or driving dynamic client
libraries that can interact with your API.
>
> Django REST Framework provides support for automatic generation of
OpenAPI schemas.
On Tue, Apr 7, 2020, 8:24 PM Andrew Barnert wrote:
> On Apr 7, 2020, at 15:31, Christopher Barker wrote:
>
>
>
> On Tue, Apr 7, 2020 at 11:17 AM Wes Turner wrote:
>
>> Would you generate a schema from the type annotations so that other
>> languages can use the data?
>>
>
> I haven't done this yet, but it would be pretty cool.
>
>
> This seems like one of the many things that’s impossible to do for Python
> classes with full generality, but pretty easy to do if you only want to
> support @dataclass. Either dynamically or statically, in fact. You could
> even write a tool that generates dataclasses (statically or dynamically)
> from a schema, if you wanted.
>
> But what is meant by "type annotations"? I'm using them via dataclasses --
> really as a shorthand for assigning a type to every field -- the
> annotations are just a shorthand that auto-generates a schema, essentially.
>
> @dataclass
> class MyClass:
> x: A_Type = A_default
>
> So now I know that this class has a field names x that is the type int. So
> I use that type for validation, and serialization / deserialization.
>
>
> But if you mean "type annotations" in the sense of the types provided out
> of the box in the typing module and used by MyPy (so I've heard, never did
> it myself) -- no, they are not sufficient -- I need types that support my
> serialization / deserialization system, and my validation system. And I
> suppose we could have a standardized __json__ and __from_json__ protocol
> that I could use, but it seems a special case to me.
>
>
> A type annotation is just the “: whatever”. It doesn’t matter whether that
> whatever is a real dynamic type like int or spam.Eggs or list, or a
> typing.type like List[int], it’s still an annotation. And dataclass can
> handle either—change that to “x: List[A_Type]” and at runtime, the
> dataclass will treat that the same as if you just used list, but mypy can
> know that it’s only supposed to have A_Type members in that list. (So if
> you initialize MyClass([1, “spam”, open(“eggs.txt”)]) it’ll work at
> runtime, but mypy will flag it as a type error.)
>
> An “automated JSON serialization for dataclasses” library could do either.
> It could be “dumb” like @dataclass and just treat x as a list of anything
> serializable, or it could be “smart” and treat it as a list of A_Type
> objects only. Either way seems like it makes sense.
>
> A schema generator for dataclasses, I think you’d want it to use the
> typing information. A MyClass property a doesn’t just have an attribute of
> {“type”: “array”}; it has a {“type”: “array”, “contains”:
> recursively_schematize(A_Type)}.
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/3W7MKXGOJ6W23OVMNWFK6MLNZPW3RYDI/
Code of
[Python-ideas] Re: Improvement: __json__
On Apr 7, 2020, at 18:10, Wes Turner wrote: > > > *That* should read as "are not sufficient". > > Stuffing all of those into annotations is going to be cumbersome; resulting > in there being multiple schema definitions to keep synchronized and validate > data according to. > > I think generating some amalgamation of JSONLD @context & SHACL and JSON > schema would be an interesting exercise. You'd certainly want to add more > information to the generated schema than just the corresponding Python types : > - type URI(s) that correspond to the Python primitive types > - JSON schema format strings > - JSON schema length > - TODO JSON schema [...] Not everything in the world has to be built around RDF semantic triples. In fact, most things don’t have to be. That’s why are a lot more things out there using plain old JSON Schema for their APIs and formats than using JSON-LD. And even more things just using free form JSON. And for either of those, type annotations are sufficient. You can serialize any instance of Spam to JSON, and deserialize JSON (that you know represents a Spam) to an equal Spam instance, as long as you know what the name and type of every attribute of Spam is (and all of those types are number/string/book/null, types that match the same qualifications as Spam, lists of such a type, and dicts mapping str to such a type). Which is guaranteed to be knowable for dataclasses even without any external information. Or any classes with a (correct) accompanying schema. Or just any classes you design around such a serialization system. The fact that you don’t have, e.g., a URI with metadata about Spam doesn’t in any way stop any of that from working, or being useful. Type annotations are sufficient for this purpose. In fact, even type annotations aren’t necessary. Any value that can pickle, you can just msg=json.dumps(b64_encode(pickle.dumps(obj and obj=pickle.loads(b64_decode(json.loads(msg and you’ve got working JSON serialization. What type annotations add is JSON serialization that’s human readable/editable, or computer verifiable, or both. You don’t need JSON-LD unless you’re not just building APIs, but meta-indexes of APIs or automatic API generators or something. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/P7PYTO663PJLJ45W6TPRDEDMA2NG4RXZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
You don't need JSON at all in order to serialize and deserialize instances of Python objects and primitives. Pickle (or e.g. Arrow + [parquet,]) handles nested, arbitrary complex types efficiently and without dataclasses and/or type annotations. I don't see the value in using JSON to round-trip from Python to the same Python code. External schema is far more useful than embedding part of an ad-hoc nested object schema in type annotations that can't also do or even specify data validations. You can already jsonpickle data classes. If you want to share or just publish data, external schema using a web standard is your best bet. On Wed, Apr 8, 2020, 3:30 AM Andrew Barnert wrote: > On Apr 7, 2020, at 18:10, Wes Turner wrote: > > > > > > *That* should read as "are not sufficient". > > > > Stuffing all of those into annotations is going to be cumbersome; > resulting in there being multiple schema definitions to keep synchronized > and validate data according to. > > > > I think generating some amalgamation of JSONLD @context & SHACL and JSON > schema would be an interesting exercise. You'd certainly want to add more > information to the generated schema than just the corresponding Python > types : > > - type URI(s) that correspond to the Python primitive types > > - JSON schema format strings > > - JSON schema length > > - TODO JSON schema [...] > > Not everything in the world has to be built around RDF semantic triples. > In fact, most things don’t have to be. That’s why are a lot more things out > there using plain old JSON Schema for their APIs and formats than using > JSON-LD. And even more things just using free form JSON. > > And for either of those, type annotations are sufficient. You can > serialize any instance of Spam to JSON, and deserialize JSON (that you know > represents a Spam) to an equal Spam instance, as long as you know what the > name and type of every attribute of Spam is (and all of those types are > number/string/book/null, types that match the same qualifications as Spam, > lists of such a type, and dicts mapping str to such a type). Which is > guaranteed to be knowable for dataclasses even without any external > information. Or any classes with a (correct) accompanying schema. Or just > any classes you design around such a serialization system. > > The fact that you don’t have, e.g., a URI with metadata about Spam doesn’t > in any way stop any of that from working, or being useful. Type annotations > are sufficient for this purpose. > > In fact, even type annotations aren’t necessary. Any value that can > pickle, you can just msg=json.dumps(b64_encode(pickle.dumps(obj and > obj=pickle.loads(b64_decode(json.loads(msg and you’ve got working JSON > serialization. What type annotations add is JSON serialization that’s human > readable/editable, or computer verifiable, or both. You don’t need JSON-LD > unless you’re not just building APIs, but meta-indexes of APIs or automatic > API generators or something. > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PZZC3LHS56NUVDK4I7WHXMUPD24ZX4J5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Wed, Apr 8, 2020 at 1:18 AM Wes Turner wrote: > > I don't see the value in using JSON to round-trip from Python to the same > Python code. > There is a bit of a value: it's a human-readable format that matches the Python Data model pretty well. I've used it for that reason. Maybe I should be using yaml or something instead, but it's nice to use something common. I have thought about using "PYSON" -- which would better match the Python data model, but never got around to formalizing that. But yes, the real advantage to JSON is interaction with non-python systems. > > External schema is far more useful than embedding part of an ad-hoc nested > object schema in type annotations that can't also do or even specify data > validations. > I suppose so -- I really need to see if I can make use of JSONSchema -- it would be nice to specify teh schema in one place, and be able to build Python objects, and Javascript objects, and theoretically all kinds of other implementations as well. Someone may have done that, I need to go look. NOTE: JSON-LD keeps coming up in these thrteads, but that really seems like an orthogonal issue. Maybe there should be JSON-LD support for Python (is there already?) but that shouldn't impact the core json library, nor a __json__ magic method. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/J3BCRNHEC3CLE2LLKB5UTAMSGTZUXZZN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Apr 8, 2020, at 01:18, Wes Turner wrote: > > I don't see the value in using JSON to round-trip from Python to the same > Python code. > > External schema is far more useful than embedding part of an ad-hoc nested > object schema in type annotations that can't also do or even specify data > validations. But dataclasses with type annotations can express a complete JSON Schema. Or, of course, in an ad hoc schema only published in human readable form, as most web APIs use today. > You can already jsonpickle data classes. If you want to share or just publish > data, external schema using a web standard is your best bet. Sure, but you don’t need JSON-LD for that. Again, the fact that type annotations are insufficient to represent semantic triples is irrelevant. They are sufficient for the case of writing code to parse what YouTube gives you, or to provide a documented API that you design to be consumed by other people in JS, or to generate a JSON Schema from your legacy collection of classes that you can then publish, or to validate that a dataclass hierarchy matches a published schema, and so on. All of which are useful. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XDJELGTCVSDHFNPQA6556CUWC5XUCXTW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
I'm aware of a couple Python implementations of JSON-LD: pyld [1] and rdflib-jsonld [2]. But you don't need a JSON-LD parser to parse or produce JSON-LD: You can just frame the data in the json document correctly such that other tools can easily parse your necessarily complex data types stored within the limited set of primitive datatypes supported by JSON. We should welcome efforts to support linked data in Python. TimBL created the web on top of the internet in order that we could share resources in order to collaborate on science. In order to collaborate on science, we need to be able to share, discover, merge, join, concatenate, analyze, and compare Datasets. TimBL's 5-star Open Data plan [3] justifies the costs and benefits of sharing LOD: Linked Open Data. ★ make your stuff available on the Web (whatever format) under an open license ★★ make it available as structured data (e.g., Excel instead of image scan of a table) ★★★ make it available in a non-proprietary open format (e.g., CSV instead of Excel) use URIs to denote things, so that people can point at your stuff ★ link your data to other data to provide context JSON is ★★★ data. JSON-LD, RDFa, and Microformats are or ★ data. We can link our data to other data with URIs in linked data formats. JSON-LD is one representation of RDF. RDF* (read as "RDF star") extends RDF for use with property graphs. No one cares whether you believe that "semantic web failed" or "those standards are useless": being able to share, discover, merge, join, concatenate, analyze, and compare Datasets is of significant value to the progress of the sciences and useful arts; so, I think that we should support the linked data use case with at least: 1. a __json__(obj, spec=None) method and 2. a more-easily modifiable make_iterencode/iterencode implementation in the json module of the standard library. [1] https://github.com/digitalbazaar/pyld [2] https://github.com/RDFLib/rdflib-jsonld [3] https://5stardata.info/en/ * Here's this that merges JSON-LD, SHACL, and JSON schema: It has 3 stars. Validating JSON documents is indeed somewhat orthogonal to the __json__ / iterencode implementation details we're discussing; but specifying types in type annotations and then creating an additional complete data validation specification is not DRY. https://github.com/mulesoft-labs/json-ld-schema * re: generating JSON schema from type annotations You can go from python:Str to jsonschema:format:string easily enough, but, again, going from python:str to jsonschema:format:email will require either extending the type annotation syntax or modifying a generated schema stub and then changes to which will then need to be ported back to the type annotations. I suppose if all you're working with are data classes, that generating part of the JSON schema from data class type annotations could be useful to you in your quest to develop a new subset of JSON thats supports deserializing (and validating) complex types in at least Python and JS (when there are existing standards for doing so). On Wed, Apr 8, 2020 at 3:08 PM Andrew Barnert wrote: > On Apr 8, 2020, at 01:18, Wes Turner wrote: > > > > I don't see the value in using JSON to round-trip from Python to the > same Python code. > > > > External schema is far more useful than embedding part of an ad-hoc > nested object schema in type annotations that can't also do or even specify > data validations. > > But dataclasses with type annotations can express a complete JSON Schema. > Or, of course, in an ad hoc schema only published in human readable form, > as most web APIs use today. > > > You can already jsonpickle data classes. If you want to share or just > publish data, external schema using a web standard is your best bet. > > Sure, but you don’t need JSON-LD for that. Again, the fact that type > annotations are insufficient to represent semantic triples is irrelevant. > They are sufficient for the case of writing code to parse what YouTube > gives you, or to provide a documented API that you design to be consumed by > other people in JS, or to generate a JSON Schema from your legacy > collection of classes that you can then publish, or to validate that a > dataclass hierarchy matches a published schema, and so on. All of which are > useful. > > > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/D7WZX6AGC2OP3G6IYI36LACABQCWTS4J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
On Apr 8, 2020, at 12:55, Wes Turner wrote: > > We should welcome efforts to support linked data in Python. Fine, but that’s doesn’t meant we should derail every proposal that has anything to do with JSON by turning it into a proposal for linked data, or consider a useful proposal to be useless because it doesn’t give us linked data support on top of whatever it was intended to give us. > No one cares whether you believe that "semantic web failed" or "those > standards are useless": But no one is saying either of those things. People are saying that the semantic web is irrelevant to this discussion. JSON is useful, JSON Schema is useful, improving Python to make working with ad hoc or Schema-specified JSON is useful, and that’s all equally true whether the semantic web is the one true future or a total failure. Would a __json__ method protocol improve Python? I don’t think so, but the reasons I don’t think so have nothing to do with LD. Other people do think so, and their reasons also have nothing to do with LD. And the same is true for most of the counter- and side-ideas that have come up in this thread and the last one. > re: generating JSON schema from type annotations > > You can go from python:Str to jsonschema:format:string easily enough, > but, again, going from python:str to jsonschema:format:email will require > either extending the type annotation syntax or modifying a generated schema > stub and then changes to which will then need to be ported back to the type > annotations. Sure, but who says you have to store email addresses as str? @dataclass class Person: name: str email: Email address: Address The fact that str is a builtin type, Address is a dataclass with a bunch of builtin-typed attributes, and Email is (say) a str subclass (or a dataclass with just a str member or whatever) that knows to render to and from type:string, format:email instead of just type:string doesn’t change the fact that they’re all perfectly good Python types and can all be used as type annotations and can all be mapped to a JSON Schema. How does it know that? Lots of things would work. Which one you use would be up to your JSON Schema-driven serialization library, or to your Python-code-from-JSON-Schema generator tool or your static Schema-from-code generator tool or.whatever. Maybe if we have multiple such libraries in wide use fighting it out, one of them will win. But even if there’s never a category killer, any of them will work fine for the applications that use it. And of course often just using str for email addresses is fine. There’s a reason the Formats section of the JSON Schema spec is optional and explicitly calls out that many implementations don’t include it. And in some applications it would make more sense to break an email address down into two parts (user and host) and store a dict of those two values instead, and that’s also fine. Would it better serve the needs of the semantic web if the email format were a mandatory rather than optional part of the spec and everyone were required by law to always use it when storing email addresses? Maybe. But neither of those things are true. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QTDOOH4ASKWNUFYRZHUOSNQB62MAL5YK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
I think that we should support the linked data use case with at least: 1. a __json__(obj, spec=None) method and 2. a more-easily modifiable make_iterencode/iterencode implementation in the json module of the standard library. On Wed, Apr 8, 2020 at 5:00 PM Andrew Barnert wrote: > On Apr 8, 2020, at 12:55, Wes Turner wrote: > > > > We should welcome efforts to support linked data in Python. > > Fine, but that’s doesn’t meant we should derail every proposal that has > anything to do with JSON by turning it into a proposal for linked data, or > consider a useful proposal to be useless because it doesn’t give us linked > data support on top of whatever it was intended to give us. > > > No one cares whether you believe that "semantic web failed" or "those > standards are useless": > > But no one is saying either of those things. People are saying that the > semantic web is irrelevant to this discussion. JSON is useful, JSON Schema > is useful, improving Python to make working with ad hoc or Schema-specified > JSON is useful, and that’s all equally true whether the semantic web is the > one true future or a total failure. > > Would a __json__ method protocol improve Python? I don’t think so, but the > reasons I don’t think so have nothing to do with LD. Other people do think > so, and their reasons also have nothing to do with LD. And the same is true > for most of the counter- and side-ideas that have come up in this thread > and the last one. > > > re: generating JSON schema from type annotations > > > > You can go from python:Str to jsonschema:format:string easily enough, > > but, again, going from python:str to jsonschema:format:email will > require either extending the type annotation syntax or modifying a > generated schema stub and then changes to which will then need to be ported > back to the type annotations. > > Sure, but who says you have to store email addresses as str? > > @dataclass > class Person: > name: str > email: Email > address: Address > > The fact that str is a builtin type, Address is a dataclass with a bunch > of builtin-typed attributes, and Email is (say) a str subclass (or a > dataclass with just a str member or whatever) that knows to render to and > from type:string, format:email instead of just type:string doesn’t change > the fact that they’re all perfectly good Python types and can all be used > as type annotations and can all be mapped to a JSON Schema. How does it > know that? Lots of things would work. Which one you use would be up to your > JSON Schema-driven serialization library, or to your > Python-code-from-JSON-Schema generator tool or your static Schema-from-code > generator tool or.whatever. Maybe if we have multiple such libraries in > wide use fighting it out, one of them will win. But even if there’s never a > category killer, any of them will work fine for the applications that use > it. > > And of course often just using str for email addresses is fine. There’s a > reason the Formats section of the JSON Schema spec is optional and > explicitly calls out that many implementations don’t include it. And in > some applications it would make more sense to break an email address down > into two parts (user and host) and store a dict of those two values > instead, and that’s also fine. Would it better serve the needs of the > semantic web if the email format were a mandatory rather than optional part > of the spec and everyone were required by law to always use it when storing > email addresses? Maybe. But neither of those things are true. > > > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EDWP54EOMECBKUIDWZMU5Q4ADSI36Z27/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Improvement: __json__
In trying to do this (again), I've realized that you *can't* just check for hasattr(obj, '__json__') in a JSONEncoder.default method because default only get's called for types it doesn't know how to serialize; so if you e.g. subclass a dict, default() never gets called for the dict subclass. https://docs.python.org/3/library/json.html : > If specified, default should be a function that gets called for objects that can’t otherwise be serialized. https://github.com/python/cpython/blob/master/Lib/json/encoder.py https://stackoverflow.com/questions/16405969/how-to-change-json-encoding-behaviour-for-serializable-python-object/17684652#17684652 specifies how to overload _make_iterencode *in pure Python*, but AFAICS that NOPs use of the C-optimized json encoder. The python json module originally came from simplejson. Here's simplejson's for_json implementation in pure Python: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8ebcb82eee72/simplejson/encoder.py#L515 : for_json = _for_json and getattr(value, 'for_json', None) if for_json and callable(for_json): chunks = _iterencode(for_json(), _current_indent_level) And simplejson's for_json implementation in C: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8ebcb82eee72/simplejson/_speedups.c#L2839 Is there a strong reason that the method would need to be called __json__ instead of 'for_json'? Passing a spec=None kwarg through to the __json__()/for_json() method would not be compatible with simplejson's existing implementation. Passing a spec=None kwarg would be necessary to support different JSON standards within the same method; which I think is desirable because JSON/JSON5/JSONLD/JSON_future. On Wed, Apr 8, 2020 at 5:03 PM Wes Turner wrote: > I think that we should support the linked data use case with at least: > 1. a __json__(obj, spec=None) method and > 2. a more-easily modifiable make_iterencode/iterencode implementation in > the json module of the standard library. > > On Wed, Apr 8, 2020 at 5:00 PM Andrew Barnert wrote: > >> On Apr 8, 2020, at 12:55, Wes Turner wrote: >> > >> > We should welcome efforts to support linked data in Python. >> >> Fine, but that’s doesn’t meant we should derail every proposal that has >> anything to do with JSON by turning it into a proposal for linked data, or >> consider a useful proposal to be useless because it doesn’t give us linked >> data support on top of whatever it was intended to give us. >> >> > No one cares whether you believe that "semantic web failed" or "those >> standards are useless": >> >> But no one is saying either of those things. People are saying that the >> semantic web is irrelevant to this discussion. JSON is useful, JSON Schema >> is useful, improving Python to make working with ad hoc or Schema-specified >> JSON is useful, and that’s all equally true whether the semantic web is the >> one true future or a total failure. >> >> Would a __json__ method protocol improve Python? I don’t think so, but >> the reasons I don’t think so have nothing to do with LD. Other people do >> think so, and their reasons also have nothing to do with LD. And the same >> is true for most of the counter- and side-ideas that have come up in this >> thread and the last one. >> >> > re: generating JSON schema from type annotations >> > >> > You can go from python:Str to jsonschema:format:string easily enough, >> > but, again, going from python:str to jsonschema:format:email will >> require either extending the type annotation syntax or modifying a >> generated schema stub and then changes to which will then need to be ported >> back to the type annotations. >> >> Sure, but who says you have to store email addresses as str? >> >> @dataclass >> class Person: >> name: str >> email: Email >> address: Address >> >> The fact that str is a builtin type, Address is a dataclass with a bunch >> of builtin-typed attributes, and Email is (say) a str subclass (or a >> dataclass with just a str member or whatever) that knows to render to and >> from type:string, format:email instead of just type:string doesn’t change >> the fact that they’re all perfectly good Python types and can all be used >> as type annotations and can all be mapped to a JSON Schema. How does it >> know that? Lots of things would work. Which one you use would be up to your >> JSON Schema-driven serialization library, or to your >> Python-code-from-JSON-Schema generator tool or your static Schema-from-code >> generator tool or.whatever. Maybe if we have multiple such libraries in >> wide use fighting it out, one of them will win. But even if there’s never a >> category killer, any of them will work fine for the applications that use >> it. >> >> And of course often just using str for email addresses is fine. There’s a >> reason the Formats section of the JSON Schema spec is optional and >> explicitly calls out that many implementations don’t include it. And
