Questioning the effects of multiple assignment
TLDR; if you are a Python 'Master' then feel free to skim the first part
(which you should know hands-down), until the excerpts from 'the manual'
and from there I'll be interested in learning from you...
Yesterday I asked a junior prog to expand an __init__() to accept either
a series of (>1) scalars (as it does now), or to take similar values but
presented as a tuple. He was a bit concerned that I didn't want to
introduce a (separate) keyword-argument, until 'we' remembered
starred-parameters. He then set about experimenting. Here's a dichotomy
that surfaced as part of our 'play':-
(my problem is: I can't (reasonably) answer his question...)
If you read this code:
NB The print-ing does not follow the input-sequence, because that's the
point to be made...
>>> def f( a, *b, c=0 ):
... print( a, type( a ) )
... print( c, type( c ) )
... print( b )
...
>>> f( 1, 'two', 3, 'four' )
[I had to force "c" to become a keyword argument, but other than that,
we'll be using these three parameters and four argument-values, again]
Question 1: did you correctly predict the output?
1
0
('two', 3, 'four')
Ahah, "c" went to default because there was no way to identify when the
"*b" 'stopped' and "c" started - so all the values 'went' to become "b"
(were all "consumed by"...).
Why did I also print "b" differently?
Building tension!
Please read on, gentle reader...
Let's make two small changes:
- amend the last line of the function to be similar:
... print( b, type( b ) )
- make proper use of the function's API:
>>> f( 1, 'two', 3, c='four' )
Question 2: can you predict the output of "a"? Well duh!
(same as before)
1
Question 3: has your predicted output of "c" changed? Yes? Good!
(Web.Refs below, explain; should you wish...)
four
Question 4: can you correctly predict the content of "b" and its type?
('two', 3)
That makes sense, doesn't it? The arguments were presented to the
function as a tuple, and those not assigned to a scalar value ("a" and
"c") were kept as a tuple when assigned to "b".
Jolly good show, chaps!
(which made my young(er) colleague very happy, because now he could see
that by checking the length of the parameter, such would reveal if the
arguments were being passed as scalars or as a tuple.
Aside: however, it made me think how handy it would be if the
newly-drafted PEP 622 -- Structural Pattern Matching were available
today (proposed for v3.10, https://www.python.org/dev/peps/pep-0622/)
because (YAGNI-aside) we could then more-easily empower the API to
accept other/more collections!
Why am I writing then?
Because during the same conversations I was
'demonstrating'/training/playing with some code that is (apparently)
very similar - and yet, it's not. Oops!
Sticking with the same, toy-data, let's code:
>>> a, *b, c = 1, 'two', 3, 'four'
>>> a, type( a )
>>> c, type( c )
>>> b, type( b )
Question 5: what do you expect "a" and "c" to deliver in this context?
(1, )
('four', )
Happy so far?
Question 6: (for maximum effect, re-read snippets from above, then) what
do you expect from "b"?
(['two', 3], )
List? A list? What's this "list" stuff???
When "b" was a parameter (above) it was assigned a tuple!
Are you as shocked as I?
Have you learned something?
(will it ever be useful?)
Has the world stopped turning?
Can you explain why these two (apparently) logical assignment processes
have been designed to realise different result-objects?
NB The list cf tuple difference is 'legal' - at least in the sense that
it is documented/expected behavior:-
Python Reference Manual: 7.2. Assignment statements
Assignment statements are used to (re)bind names to values and to modify
attributes or items of mutable objects:
...
An assignment statement evaluates the expression list (remember that
this can be a single expression or a comma-separated list, the latter
yielding a tuple) and assigns the single resulting object to each of the
target lists, from left to right.
...
A list of the remaining items in the iterable is then assigned to the
starred target (the list can be empty).
https://docs.python.org/3/reference/simple_stmts.html#assignment-statements
Python Reference Manual: 6.3.4. Calls
A call calls a callable object (e.g., a function) with a possibly empty
series of arguments:
...
If there are more positional arguments than there are formal parameter
slots, a TypeError exception is raised, unless a formal parameter using
the syntax *identifier is present; in this case, that formal parameter
receives a tuple containing the excess positional arguments (or an empty
tuple if there were no excess positional arguments).
https://docs.python.org/dev/reference/expressions.html#calls
--
Regards,
=dn
--
https://mail.python.org/mailman/listinfo/python-list
Re: [PSF-Community] Unable to login | fbchat.Client
Hi Shivam, Please be aware this mailing list is for PSF community-focused discussions -- see the list's purpose described at https://mail.python.org/mailman/listinfo/psf-community. I might suggest you participate in the discussion on on the `fbchat` project repository on GitHub -- the issue at https://github.com/carpedm20/fbchat/issues/598 seems to match the error you've pasted. Alternatively, you could try directing your question to the general Python list (python-list@python.org, CC'd here). Regards, Rami On Tue, Jul 7, 2020 at 07:52, Shivam Dutt Sharma wrote: Dear All, I am facing an error while trying to log-in to fbchat.Client. This is obviously despite me entering absolutely correct ID & P/W. It could be because of any latest update in the package, or a user_agent / session (cookies) issue too. Prima-facie, if you think a few adjustments need to be made in the client or state file/s, please let me know, else I shall share the complete code link for your perusal. Below's the error message. I will be highly grateful, if I can get an idea of where am I going wrong? Logging in xx.x.xx... Attempt #1 failed, retrying Traceback (most recent call last): File "C:\Users\my\Anaconda3\lib\site-packages\fbchat\_client.py", line 209, in login user_agent=user_agent, File "C:\Users\my\Anaconda3\lib\site-packages\fbchat\_state.py", line 149, in login return cls.from_session(session=session) File "C:\Users\my\Anaconda3\lib\site-packages\fbchat\_state.py", line 186, in from_session fb_dtsg = FB_DTSG_REGEX.search(r.text).group(1) AttributeError: 'NoneType' object has no attribute 'group' Best, Shivam Dutt Sharma LinkedIn : shivam-dutt-sharma-020456149 PGP in Machine Learning | NIT Warangal Fundraising Volunteer @ The AIM Foundation PCEP @ PythonInstitute -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
You can achieve round-tripping by maintaining a type mapping in code, for a single datatype it would look like: newloads(datetime, newdumps(datetime.now()) If those would rely on __dump__ and __load__ functions in the fashion of pickle then nested data structures would also be easy: @dataclass class YourStruct: dt = datetime children = [] @classmethod def __load__(cls, data): return cls( dt=datetime.fromisoformat(data['dt']), children=[cls.__load__(c) for c in data['children']]) ) def __dump__(self): return dict( dt=self.dt.isoformat(), children=[c.__dump__() for c in self.children], ) If your datetime is not being loaded from C-code you can even monkey patch it and add __load__ and __dump__ on it and data round-trip as long as you keep the type mapping in a method. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote: > On Tue, Jul 7, 2020 at 12:01 AM Jon Ribbens via Python-list > wrote: >> I think what you're saying is, if we do: >> >> json1 = json.dumps(foo) >> json2 = json.dumps(json.loads(json1)) >> assert json1 == json2 >> >> the assertion should never fail (given that Python dictionaries are >> ordered these days). I seems to me that should probably be true >> regardless of any 'strict mode' flag - I can't immediately think of >> any reason it wouldn't be. > > Right. But in strict mode, the stronger assertion would hold: > > assert obj == json.loads(json.dumps(obj)) > > Also, the intermediate text would be RFC-compliant. If this cannot be > done, ValueError would be raised. (Or maybe TypeError in some cases.) Yes, I agree (although you'd need to call it something other than 'strict' mode, since that flag already exists). But note nothing I am suggesting would involve JSONEncoder ever producing non-standard output (except in cases where it already would). -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On Tue, Jul 7, 2020 at 12:01 AM Jon Ribbens via Python-list wrote: > > On 2020-07-06, Chris Angelico wrote: > > I think that even in non-strict mode, round-tripping should be > > achieved after one iteration. That is to say, anything you can > > JSON-encode will JSON-decode to something that would create the same > > encoded form. Not sure if there's anything that would violate that > > (weak) guarantee. > > I think what you're saying is, if we do: > > json1 = json.dumps(foo) > json2 = json.dumps(json.loads(json1)) > assert json1 == json2 > > the assertion should never fail (given that Python dictionaries are > ordered these days). I seems to me that should probably be true > regardless of any 'strict mode' flag - I can't immediately think of > any reason it wouldn't be. Right. But in strict mode, the stronger assertion would hold: assert obj == json.loads(json.dumps(obj)) Also, the intermediate text would be RFC-compliant. If this cannot be done, ValueError would be raised. (Or maybe TypeError in some cases.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote: > I think that even in non-strict mode, round-tripping should be > achieved after one iteration. That is to say, anything you can > JSON-encode will JSON-decode to something that would create the same > encoded form. Not sure if there's anything that would violate that > (weak) guarantee. I think what you're saying is, if we do: json1 = json.dumps(foo) json2 = json.dumps(json.loads(json1)) assert json1 == json2 the assertion should never fail (given that Python dictionaries are ordered these days). I seems to me that should probably be true regardless of any 'strict mode' flag - I can't immediately think of any reason it wouldn't be. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Frank Millman wrote: > On 2020-07-06 3:08 PM, Jon Ribbens via Python-list wrote: >> On 2020-07-06, Frank Millman wrote: >>> On 2020-07-06 2:06 PM, Jon Ribbens via Python-list wrote: While I agree entirely with your point, there is however perhaps room for a bit more helpfulness from the json module. There is no sensible reason I can think of that it refuses to serialize sets, for example. Going a bit further and, for example, automatically calling isoformat() on date/time/datetime objects would perhaps be a bit more controversial, but would frequently be useful, and there's no obvious downside that occurs to me. >>> >>> I may be missing something, but that would cause a downside for me. >>> >>> I store Python lists and dicts in a database by calling dumps() when >>> saving them to the database and loads() when retrieving them. >>> >>> If a date was 'dumped' using isoformat(), then on retrieval I would not >>> know whether it was originally a string, which must remain as is, or was >>> originally a date object, which must be converted back to a date object. >>> >>> There is no perfect answer, but my solution works fairly well. When >>> dumping, I use 'default=repr'. This means that dates get dumped as >>> 'datetime.date(2020, 7, 6)'. I look for that pattern on retrieval to >>> detect that it is actually a date object. >> >> There is no difference whatsoever between matching on the repr output >> you show above and matching on ISO-8601 datetimes, except that at least >> ISO-8601 is an actual standard. So no, you haven't found a downside. > > I don't understand. As you say, ISO-8601 is a standard, so the original > object could well have been a string in that format. So how do you > distinguish between an object that started out as a string, and an > object that started out as a date/datetime object? With your method, how do you distinguish between an object that started out as a string, and an object that started out as a date/datetime object? The answer with both my method and your method is that you cannot - and therefore my method is not a "downside" compared to yours. Not to mention, I am not suggesting that your method should be disallowed if you want to continue using it - I am suggesting that your code could be simplified and your job made easier by my suggested improvement. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On Mon, Jul 6, 2020 at 11:39 PM Adam Funk wrote:
>
> Aha, I think the default=repr option is probably just what I need;
> maybe (at least in the testing stages) something like this:
>
> try:
> with open(output_file, 'w') as f:
> json.dump(f)
> except TypeError:
> print('unexpected item in the bagging area!')
> with open(output_file, 'w') as f:
> json.dump(f, default=repr)
>
> and then I'd know when I need to go digging through the output for
> bytes, sets, etc., but at least I'd have the output to examine.
>
Easier:
def proclaimed_repr():
seen = False
def show_obj(obj):
nonlocal seen
if not seen:
seen = True
print("unexpected item in the bagging area!")
return repr(obj)
return show_obj
json.dump(f, default=proclaimed_repr())
If you don't care about "resetting" the marker, you can just use a
global or a default-arg hack:
def show_obj(obj, seen=[]):
if not seen:
seen.push(True)
print("unexpected item in the bagging area!")
return repr(obj)
json.dump(f, default=show_obj)
Either way, you can stick this function off in a utilities collection,
and then use it without fiddling with try/except.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote: > On Mon, Jul 6, 2020 at 10:11 PM Jon Ribbens via Python-list > wrote: >> >> On 2020-07-06, Chris Angelico wrote: >> > On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote: >> >> Is there a "bulletproof" version of json.dump somewhere that will >> >> convert bytes to str, any other iterables to list, etc., so you can >> >> just get your data into a file & keep working? >> > >> > That's the PHP definition of "bulletproof" - whatever happens, no >> > matter how bad, just keep right on going. >> >> While I agree entirely with your point, there is however perhaps room >> for a bit more helpfulness from the json module. There is no sensible >> reason I can think of that it refuses to serialize sets, for example. > > Sets don't exist in JSON. I think that's a sensible reason. I don't agree. Tuples & lists don't exist separately in JSON, but both are serializable (to the same thing). Non-string keys aren't allowed in JSON, but it silently converts numbers to strings instead of barfing. Typically, I've been using sets to deduplicate values as I go along, & having to walk through the whole object changing them to lists before serialization strikes me as the kind of pointless labor that I expect when I'm using Java. ;-) >> Going a bit further and, for example, automatically calling isoformat() >> on date/time/datetime objects would perhaps be a bit more controversial, >> but would frequently be useful, and there's no obvious downside that >> occurs to me. > > They wouldn't round-trip without some way of knowing which strings > represent date/times. If you just want a one-way output format, it's > not too hard to subclass the encoder - there's an example right there > in the docs (showing how to create a representation for complex > numbers). The vanilla JSON encoder shouldn't do any of this. In fact, > just supporting infinities and nans is fairly controversial - see > other threads happening right now. > > Maybe what people want is a pretty printer instead? > > https://docs.python.org/3/library/pprint.html > > Resilient against recursive data structures, able to emit Python-like > code for many formats, is as readable as JSON, and is often > round-trippable. It lacks JSON's interoperability, but if you're > trying to serialize sets and datetimes, you're forfeiting that anyway. > > ChrisA -- "It is the role of librarians to keep government running in difficult times," replied Dramoren. "Librarians are the last line of defence against chaos." (McMullen 2001) -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On Mon, Jul 6, 2020 at 11:31 PM Jon Ribbens via Python-list
wrote:
>
> On 2020-07-06, Chris Angelico wrote:
> > On Mon, Jul 6, 2020 at 11:06 PM Jon Ribbens via Python-list
> > wrote:
> >> The 'json' module already fails to provide round-trip functionality:
> >>
> >> >>> for data in ({True: 1}, {1: 2}, (1, 2)):
> >> ... if json.loads(json.dumps(data)) != data:
> >> ... print('oops', data, json.loads(json.dumps(data)))
> >> ...
> >> oops {True: 1} {'true': 1}
> >> oops {1: 2} {'1': 2}
> >> oops (1, 2) [1, 2]
> >
> > There's a fundamental limitation of JSON in that it requires string
> > keys, so this is an obvious transformation. I suppose you could call
> > that one a bug too, but it's very useful and not too dangerous. (And
> > then there's the tuple-to-list transformation, which I think probably
> > shouldn't happen, although I don't think that's likely to cause issues
> > either.)
>
> That's my point though - there's almost no difference between allowing
> encoding of tuples and allowing encoding of sets. Any argument against
> the latter would also apply against the former. The only possible excuse
> for the difference is "historical reasons", and given that it would be
> useful to allow it, and there would be no negative consequences, this
> hardly seems sufficient.
>
> >> No. I want a JSON encoder to output JSON to be read by a JSON decoder.
> >
> > Does it need to round-trip, though? If you stringify your datetimes,
> > you can't decode it reliably any more. What's the purpose here?
>
> It doesn't need to round trip (which as mentioned above is fortunate
> because the existing module already doesn't round trip). The main use
> I have, and I should imagine the main use anyone has, for JSON is
> interoperability - to safely store and send data in a format in which
> it can be read by non-Python code. If you need, say, date/times to
> be understood as date/times by the receiving code they'll have to
> deal with that explicitly already. Improving Python to allow sending
> them at least gets us part way there by eliminating half the work.
That's fair.
Maybe what we need is to fork out the default JSON encoder into two,
or have a "strict=True" or "strict=False" flag. In non-strict mode,
round-tripping is not guaranteed, and various types will be folded to
each other - mainly, many built-in and stdlib types will be
represented in strings. In strict mode, compliance with the RFC is
ensured (so ValueError will be raised on inf/nan), and everything
should round-trip safely.
I think that even in non-strict mode, round-tripping should be
achieved after one iteration. That is to say, anything you can
JSON-encode will JSON-decode to something that would create the same
encoded form. Not sure if there's anything that would violate that
(weak) guarantee.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Frank Millman wrote:
> On 2020-07-06 2:06 PM, Jon Ribbens via Python-list wrote:
>> On 2020-07-06, Chris Angelico wrote:
>>> On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote:
Is there a "bulletproof" version of json.dump somewhere that will
convert bytes to str, any other iterables to list, etc., so you can
just get your data into a file & keep working?
>>>
>>> That's the PHP definition of "bulletproof" - whatever happens, no
>>> matter how bad, just keep right on going.
>>
>> While I agree entirely with your point, there is however perhaps room
>> for a bit more helpfulness from the json module. There is no sensible
>> reason I can think of that it refuses to serialize sets, for example.
>> Going a bit further and, for example, automatically calling isoformat()
>> on date/time/datetime objects would perhaps be a bit more controversial,
>> but would frequently be useful, and there's no obvious downside that
>> occurs to me.
>>
>
> I may be missing something, but that would cause a downside for me.
>
> I store Python lists and dicts in a database by calling dumps() when
> saving them to the database and loads() when retrieving them.
>
> If a date was 'dumped' using isoformat(), then on retrieval I would not
> know whether it was originally a string, which must remain as is, or was
> originally a date object, which must be converted back to a date object.
>
> There is no perfect answer, but my solution works fairly well. When
> dumping, I use 'default=repr'. This means that dates get dumped as
> 'datetime.date(2020, 7, 6)'. I look for that pattern on retrieval to
> detect that it is actually a date object.
>
> I use the same trick for Decimal objects.
>
> Maybe the OP could do something similar.
Aha, I think the default=repr option is probably just what I need;
maybe (at least in the testing stages) something like this:
try:
with open(output_file, 'w') as f:
json.dump(f)
except TypeError:
print('unexpected item in the bagging area!')
with open(output_file, 'w') as f:
json.dump(f, default=repr)
and then I'd know when I need to go digging through the output for
bytes, sets, etc., but at least I'd have the output to examine.
--
Well, we had a lot of luck on Venus
We always had a ball on Mars
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote:
> On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote:
>>
>> Hi,
>>
>> I have a program that does a lot of work with URLs and requests,
>> collecting data over about an hour, & then writing the collated data
>> to a JSON file. The first time I ran it, the json.dump failed because
>> there was a bytes value instead of a str, so I had to figure out where
>> that was coming from before I could get any data out. I've previously
>> run into the problem of collecting values in sets (for deduplication)
>> & forgetting to walk through the big data object changing them to
>> lists before serializing.
>>
>> Is there a "bulletproof" version of json.dump somewhere that will
>> convert bytes to str, any other iterables to list, etc., so you can
>> just get your data into a file & keep working?
>>
>
> That's the PHP definition of "bulletproof" - whatever happens, no
> matter how bad, just keep right on going. If you really want some way
Well played!
> to write "just anything" to your file, I recommend not using JSON -
> instead, write out the repr of your data structure. That'll give a
> decent result for bytes, str, all forms of numbers, and pretty much
> any collection, and it won't break if given something that can't
> safely be represented.
Interesting point. At least the TypeError message does say what the
unacceptable type is ("Object of type set is not JSON serializable").
--
"It is the role of librarians to keep government running in difficult
times," replied Dramoren. "Librarians are the last line of defence
against chaos." (McMullen 2001)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06 3:08 PM, Jon Ribbens via Python-list wrote: On 2020-07-06, Frank Millman wrote: On 2020-07-06 2:06 PM, Jon Ribbens via Python-list wrote: While I agree entirely with your point, there is however perhaps room for a bit more helpfulness from the json module. There is no sensible reason I can think of that it refuses to serialize sets, for example. Going a bit further and, for example, automatically calling isoformat() on date/time/datetime objects would perhaps be a bit more controversial, but would frequently be useful, and there's no obvious downside that occurs to me. I may be missing something, but that would cause a downside for me. I store Python lists and dicts in a database by calling dumps() when saving them to the database and loads() when retrieving them. If a date was 'dumped' using isoformat(), then on retrieval I would not know whether it was originally a string, which must remain as is, or was originally a date object, which must be converted back to a date object. There is no perfect answer, but my solution works fairly well. When dumping, I use 'default=repr'. This means that dates get dumped as 'datetime.date(2020, 7, 6)'. I look for that pattern on retrieval to detect that it is actually a date object. There is no difference whatsoever between matching on the repr output you show above and matching on ISO-8601 datetimes, except that at least ISO-8601 is an actual standard. So no, you haven't found a downside. I don't understand. As you say, ISO-8601 is a standard, so the original object could well have been a string in that format. So how do you distinguish between an object that started out as a string, and an object that started out as a date/datetime object? Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote:
> On Mon, Jul 6, 2020 at 11:06 PM Jon Ribbens via Python-list
> wrote:
>> The 'json' module already fails to provide round-trip functionality:
>>
>> >>> for data in ({True: 1}, {1: 2}, (1, 2)):
>> ... if json.loads(json.dumps(data)) != data:
>> ... print('oops', data, json.loads(json.dumps(data)))
>> ...
>> oops {True: 1} {'true': 1}
>> oops {1: 2} {'1': 2}
>> oops (1, 2) [1, 2]
>
> There's a fundamental limitation of JSON in that it requires string
> keys, so this is an obvious transformation. I suppose you could call
> that one a bug too, but it's very useful and not too dangerous. (And
> then there's the tuple-to-list transformation, which I think probably
> shouldn't happen, although I don't think that's likely to cause issues
> either.)
That's my point though - there's almost no difference between allowing
encoding of tuples and allowing encoding of sets. Any argument against
the latter would also apply against the former. The only possible excuse
for the difference is "historical reasons", and given that it would be
useful to allow it, and there would be no negative consequences, this
hardly seems sufficient.
>> No. I want a JSON encoder to output JSON to be read by a JSON decoder.
>
> Does it need to round-trip, though? If you stringify your datetimes,
> you can't decode it reliably any more. What's the purpose here?
It doesn't need to round trip (which as mentioned above is fortunate
because the existing module already doesn't round trip). The main use
I have, and I should imagine the main use anyone has, for JSON is
interoperability - to safely store and send data in a format in which
it can be read by non-Python code. If you need, say, date/times to
be understood as date/times by the receiving code they'll have to
deal with that explicitly already. Improving Python to allow sending
them at least gets us part way there by eliminating half the work.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, J. Pic wrote: > Well I made a suggestion on python-ideas and a PyPi lib came out of it, but > since you can't patch a lot of internal types it's not so useful. > > Feel free to try it out: > > https://yourlabs.io/oss/jsonlight/ While I applaud your experimentation, that is not suitable for any purpose. You would probably do better by starting off subclassing json.JSONEncoder. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On Mon, Jul 6, 2020 at 11:06 PM Jon Ribbens via Python-list
wrote:
>
> On 2020-07-06, Chris Angelico wrote:
> > On Mon, Jul 6, 2020 at 10:11 PM Jon Ribbens via Python-list
> > wrote:
> >> While I agree entirely with your point, there is however perhaps room
> >> for a bit more helpfulness from the json module. There is no sensible
> >> reason I can think of that it refuses to serialize sets, for example.
> >
> > Sets don't exist in JSON. I think that's a sensible reason.
>
> It is not. Tuples don't exist either, and yet they're supported.
Hmm, I didn't know that. Possibly it's as much a bug as the inf/nan issue.
> >> Going a bit further and, for example, automatically calling isoformat()
> >> on date/time/datetime objects would perhaps be a bit more controversial,
> >> but would frequently be useful, and there's no obvious downside that
> >> occurs to me.
> >
> > They wouldn't round-trip without some way of knowing which strings
> > represent date/times.
>
> The 'json' module already fails to provide round-trip functionality:
>
> >>> for data in ({True: 1}, {1: 2}, (1, 2)):
> ... if json.loads(json.dumps(data)) != data:
> ... print('oops', data, json.loads(json.dumps(data)))
> ...
> oops {True: 1} {'true': 1}
> oops {1: 2} {'1': 2}
> oops (1, 2) [1, 2]
There's a fundamental limitation of JSON in that it requires string
keys, so this is an obvious transformation. I suppose you could call
that one a bug too, but it's very useful and not too dangerous. (And
then there's the tuple-to-list transformation, which I think probably
shouldn't happen, although I don't think that's likely to cause issues
either.)
> > Maybe what people want is a pretty printer instead?
>
> No. I want a JSON encoder to output JSON to be read by a JSON decoder.
Does it need to round-trip, though? If you stringify your datetimes,
you can't decode it reliably any more. What's the purpose here?
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Frank Millman wrote: > On 2020-07-06 2:06 PM, Jon Ribbens via Python-list wrote: >> While I agree entirely with your point, there is however perhaps room >> for a bit more helpfulness from the json module. There is no sensible >> reason I can think of that it refuses to serialize sets, for example. >> Going a bit further and, for example, automatically calling isoformat() >> on date/time/datetime objects would perhaps be a bit more controversial, >> but would frequently be useful, and there's no obvious downside that >> occurs to me. > > I may be missing something, but that would cause a downside for me. > > I store Python lists and dicts in a database by calling dumps() when > saving them to the database and loads() when retrieving them. > > If a date was 'dumped' using isoformat(), then on retrieval I would not > know whether it was originally a string, which must remain as is, or was > originally a date object, which must be converted back to a date object. > > There is no perfect answer, but my solution works fairly well. When > dumping, I use 'default=repr'. This means that dates get dumped as > 'datetime.date(2020, 7, 6)'. I look for that pattern on retrieval to > detect that it is actually a date object. There is no difference whatsoever between matching on the repr output you show above and matching on ISO-8601 datetimes, except that at least ISO-8601 is an actual standard. So no, you haven't found a downside. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote:
> On Mon, Jul 6, 2020 at 10:11 PM Jon Ribbens via Python-list
> wrote:
>> While I agree entirely with your point, there is however perhaps room
>> for a bit more helpfulness from the json module. There is no sensible
>> reason I can think of that it refuses to serialize sets, for example.
>
> Sets don't exist in JSON. I think that's a sensible reason.
It is not. Tuples don't exist either, and yet they're supported.
>> Going a bit further and, for example, automatically calling isoformat()
>> on date/time/datetime objects would perhaps be a bit more controversial,
>> but would frequently be useful, and there's no obvious downside that
>> occurs to me.
>
> They wouldn't round-trip without some way of knowing which strings
> represent date/times.
The 'json' module already fails to provide round-trip functionality:
>>> for data in ({True: 1}, {1: 2}, (1, 2)):
... if json.loads(json.dumps(data)) != data:
... print('oops', data, json.loads(json.dumps(data)))
...
oops {True: 1} {'true': 1}
oops {1: 2} {'1': 2}
oops (1, 2) [1, 2]
> Maybe what people want is a pretty printer instead?
No. I want a JSON encoder to output JSON to be read by a JSON decoder.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
Well I made a suggestion on python-ideas and a PyPi lib came out of it, but since you can't patch a lot of internal types it's not so useful. Feel free to try it out: https://yourlabs.io/oss/jsonlight/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06 2:06 PM, Jon Ribbens via Python-list wrote: On 2020-07-06, Chris Angelico wrote: On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote: Is there a "bulletproof" version of json.dump somewhere that will convert bytes to str, any other iterables to list, etc., so you can just get your data into a file & keep working? That's the PHP definition of "bulletproof" - whatever happens, no matter how bad, just keep right on going. While I agree entirely with your point, there is however perhaps room for a bit more helpfulness from the json module. There is no sensible reason I can think of that it refuses to serialize sets, for example. Going a bit further and, for example, automatically calling isoformat() on date/time/datetime objects would perhaps be a bit more controversial, but would frequently be useful, and there's no obvious downside that occurs to me. I may be missing something, but that would cause a downside for me. I store Python lists and dicts in a database by calling dumps() when saving them to the database and loads() when retrieving them. If a date was 'dumped' using isoformat(), then on retrieval I would not know whether it was originally a string, which must remain as is, or was originally a date object, which must be converted back to a date object. There is no perfect answer, but my solution works fairly well. When dumping, I use 'default=repr'. This means that dates get dumped as 'datetime.date(2020, 7, 6)'. I look for that pattern on retrieval to detect that it is actually a date object. I use the same trick for Decimal objects. Maybe the OP could do something similar. Frank Millman -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On Mon, Jul 6, 2020 at 10:11 PM Jon Ribbens via Python-list wrote: > > On 2020-07-06, Chris Angelico wrote: > > On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote: > >> Is there a "bulletproof" version of json.dump somewhere that will > >> convert bytes to str, any other iterables to list, etc., so you can > >> just get your data into a file & keep working? > > > > That's the PHP definition of "bulletproof" - whatever happens, no > > matter how bad, just keep right on going. > > While I agree entirely with your point, there is however perhaps room > for a bit more helpfulness from the json module. There is no sensible > reason I can think of that it refuses to serialize sets, for example. Sets don't exist in JSON. I think that's a sensible reason. > Going a bit further and, for example, automatically calling isoformat() > on date/time/datetime objects would perhaps be a bit more controversial, > but would frequently be useful, and there's no obvious downside that > occurs to me. They wouldn't round-trip without some way of knowing which strings represent date/times. If you just want a one-way output format, it's not too hard to subclass the encoder - there's an example right there in the docs (showing how to create a representation for complex numbers). The vanilla JSON encoder shouldn't do any of this. In fact, just supporting infinities and nans is fairly controversial - see other threads happening right now. Maybe what people want is a pretty printer instead? https://docs.python.org/3/library/pprint.html Resilient against recursive data structures, able to emit Python-like code for many formats, is as readable as JSON, and is often round-trippable. It lacks JSON's interoperability, but if you're trying to serialize sets and datetimes, you're forfeiting that anyway. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On 2020-07-06, Chris Angelico wrote: > On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote: >> Is there a "bulletproof" version of json.dump somewhere that will >> convert bytes to str, any other iterables to list, etc., so you can >> just get your data into a file & keep working? > > That's the PHP definition of "bulletproof" - whatever happens, no > matter how bad, just keep right on going. While I agree entirely with your point, there is however perhaps room for a bit more helpfulness from the json module. There is no sensible reason I can think of that it refuses to serialize sets, for example. Going a bit further and, for example, automatically calling isoformat() on date/time/datetime objects would perhaps be a bit more controversial, but would frequently be useful, and there's no obvious downside that occurs to me. -- https://mail.python.org/mailman/listinfo/python-list
Re: Bulletproof json.dump?
On Mon, Jul 6, 2020 at 8:36 PM Adam Funk wrote: > > Hi, > > I have a program that does a lot of work with URLs and requests, > collecting data over about an hour, & then writing the collated data > to a JSON file. The first time I ran it, the json.dump failed because > there was a bytes value instead of a str, so I had to figure out where > that was coming from before I could get any data out. I've previously > run into the problem of collecting values in sets (for deduplication) > & forgetting to walk through the big data object changing them to > lists before serializing. > > Is there a "bulletproof" version of json.dump somewhere that will > convert bytes to str, any other iterables to list, etc., so you can > just get your data into a file & keep working? > That's the PHP definition of "bulletproof" - whatever happens, no matter how bad, just keep right on going. If you really want some way to write "just anything" to your file, I recommend not using JSON - instead, write out the repr of your data structure. That'll give a decent result for bytes, str, all forms of numbers, and pretty much any collection, and it won't break if given something that can't safely be represented. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Bulletproof json.dump?
Hi, I have a program that does a lot of work with URLs and requests, collecting data over about an hour, & then writing the collated data to a JSON file. The first time I ran it, the json.dump failed because there was a bytes value instead of a str, so I had to figure out where that was coming from before I could get any data out. I've previously run into the problem of collecting values in sets (for deduplication) & forgetting to walk through the big data object changing them to lists before serializing. Is there a "bulletproof" version of json.dump somewhere that will convert bytes to str, any other iterables to list, etc., so you can just get your data into a file & keep working? (I'm using Python 3.7.) Thanks! -- Slade was the coolest band in England. They were the kind of guys that would push your car out of a ditch. ---Alice Cooper -- https://mail.python.org/mailman/listinfo/python-list
Re: Assign Excel cell value from Datepicker widget Selection using Python
On 5/07/20 5:20 AM, narenchund...@gmail.com wrote: I am trying to assign a widget to an excel cell. Convertion wont help me.Thanks If you are following this post, you may be interested to view:- Using Widgets in Jupyter Notebook (Video) July 5, 2020 in the Mouse vs Python series. Whilst I haven't watched it myself, Mike's stuff is usually a solid investment! https://www.blog.pythonlibrary.org/2020/07/05/using-widgets-in-jupyter-notebook-video/ -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
