[Python-ideas] Re: Conditional dict declarations

[Guido van Rossum]

On Thu, Sep 5, 2019 at 8:37 PM Joao S. O. Bueno 
wrote:

> Also, if one have in mind that dict addition with the `+` operator had been
> recelently approved
>

Where did you hear this? I am not aware that this was even made into a PEP
let alone that such a PEP was approved.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UWQJAAQCDPPK7HMQOWYXDYN32VQ4TGHT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Joao S. O. Bueno]

Also, if one have in mind that dict addition with the `+` operator had been
recelently approved, as of Python 3.9 it will be ok to write:

```
 return ({
'user_id': user_id,} +
({'max_results': max_results} if max_results else {}) +
({'active': active} if active is not None else {}) +
({'deleted': deleted} if deleted is not None else {})
)

And actually, on a second though, this is already valid as of current
Python  - and is potentially more
readable than a magic "dict.skip" singleton.
```
 return {
 'user_id': user_id,
 **({'max_results': max_results} if max_results else {}),
 **({'active': active} if active is not None else {}),
 **({'deleted': deleted} if deleted is not None else {})
 }
```

On Thu, 5 Sep 2019 at 10:22,  wrote:

> I thought of something similar, but where the dict-literal construction is
> desired:
>
> >>> foo = True
> >>> bar = False
> >>> baz = False
> >>> d = {
> ...   'foo' if foo else None: 1,
> ...   'bar' if bar else None: 2,
> ...   'baz' if baz else None: 3,
> ... }
> >>> d
> {'foo': 1, None: 3}
> >>> d.pop(None)
> 3
> >>> d
> {'foo': 1}
>
> The natural helper function for the "'foo' if foo else None" snip would
> then be something like:
>
> >>> def key_flag(key, flag, default=None):
> ... return key if flag else default
> ...
> >>> dd = {
> ...   key_flag('foo', foo): 1,
> ...   key_flag('bar', bar): 2,
> ...   key_flag('baz', baz): 3,
> ... }
> >>> dd
> {'foo': 1, None: 3}
> >>> dd.pop(None)
> 3
> >>> dd
> {'foo': 1}
>
> -Brian
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/SZZVOFRPJEWTK5BJLEZVVSAZAL5F3OT7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2MRYBEHOWRE6YCGNHKXCPAVUITVPNETF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Andrew Barnert via Python-ideas]

Two more bits of bikeshedding…

On Sep 5, 2019, at 12:12, Andrew Barnert via Python-ideas 
 wrote:
> 
>> On Sep 5, 2019, at 04:24, Steven D'Aprano  wrote:
>> 
>> If Union is a built-in, we could have something like this:
>> 
>>   def isinstance(obj, class_or_tuple):
>>   if type(class_or_tuple) is Union:
>>   class_or_tuple = class_or_tuple.__union_params__
>>   # followed by the same code as above
>> 
>> typing.Union already defines .__union_params__ which returns a tuple of 
>> the classes used to construct the union, so in principle at least, there 
>> need be no significant performance hit from supporting Unions.
> 
> That’s a great point.
> 
> And if type.__or__ is going to return a union type, we probably don’t want 
> that to go lazily importing typing and pulling something out of it to call 
> __getitem__ on, so we probably want that result to be builtin.
> 
> I don’t think we need Union itself to be a builtin. But 
> typing.Union.__getitem__ needs to return the same kind of builtin as 
> type.__or__ (which is presumably exposed as something like types.UnionType, 
> not only exposed in the typing module) instead of returning a 
> typing._GenericAlias, or the whole point of this proposal (that `int|str == 
> Union[int, str]`) breaks.
> 
> That does raise some more bikeshedding  questions (what the constructor for 
> types.union accepts, or whether it refuses to be constructed and forces you 
> to use type.__or__ or Union.__getitem__; what its repr looks like; etc.).

Also:

Are runtime union types actually types, unlike the things in typing, or are 
they still non-type values that just have special handling as the second 
argument of isinstance and issubclass and maybe except statements?

I’d expect issubclass(int|str, int|str|bytes) to be true, and 
issubclass(int|str, int) to be false, not for both of them to raise exceptions 
about the first argument not being a type.

And I don’t see any reason that “things designed to be used as types for 
runtime type checks” shouldn’t be types. And their type (types.UnionType or 
whatever) a perfectly normal metaclass that inherits from type.

But, other than the issubclass question, I’m having a hard time imagining 
anywhere that it would make a difference.

While we’re at it:

issubclass(int|str, types.UnionType)

I think this should be false because UnionType is not like typing.Union (which 
is a typing.Generic, and therefore on its own it has to have the useless 
meaning of “the type that includes all values of any unions of any 1 or more 
types), it’s just a normal metaclass (with the normal meaning “the type of all 
union types”).

Finally, do we still need the existing Generic, typing.Union, at all? If 
types.UnionType defines a __getitem__, we could just do Union = 
types.UnionType. Would this do the right thing in every case, or could it break 
anything? I don’t know; I think it’s safer to leave typing.Union as-is (except 
for defining its __getitem__ to return the | of all of its arguments, instead 
of inheriting the _SpecialForm.__getitem__ behavior).

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PRKCGCX4X2GPP4JMZGWIKGG2ZF5IAURF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Steven D'Aprano]

On Tue, Sep 03, 2019 at 12:19:15PM -0700, Andrew Barnert via Python-ideas wrote:
> On Sep 2, 2019, at 23:50, Philippe Prados  wrote:
> > 
> > Add a new operator for `Union[type1|type2]` ?
> 
> Hold on. Are you proposing `Union[t1 | t2]` as a new spelling for 
> `Union[t1, t2]`? That seems pointless. I thought you were proposing 
> just `t1 | t2`, which seems a whole lot more useful (and no more 
> disruptive)?
> 
> I’m going to assume this is just a typo?

I too assume it was a typo, but for the record, since Union already 
flattens nested Unions, Union[str|int] would be equivalent to 
Union[Union[str|int]] which flattens to just str|int.


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XZAWTH6FCO3PUJ325NY6CP52KRWVOUHS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Steven D'Aprano]

On Thu, Sep 05, 2019 at 12:12:05PM -0700, Andrew Barnert wrote:
> On Sep 5, 2019, at 04:24, Steven D'Aprano  wrote:
> > 
> > But having said all that, I'm not sure that we should be rejecting this 
> > proposal on the basis of performance when we haven't got any working 
> > code to measure performance of :)
> 
> Good point. But then if we never get any realistic use cases, it’ll 
> probably already be rejected for that reason. :)

Sorry, I don't understand your point about realistic use-cases.

The realistic use-case for int|str (or Union[int, str]) in isinstance is 
exactly the same as the use-case for (int, str) and we've had that for 
many, many releases. I didn't think we still have to prove the utility 
of checking whether an object was an instance of class A or B.

This is about making the syntax look pretty, not adding new 
functionality.

There are two proposals here:

* allow str|int in type annotations, as a nicer-looking and shorter
  equivalent for Union[str, int]

* allow Unions (whether spelled explicitly or with the | operator)
  to have runtime effects, specifically in isinstance and I presume
  issubclass, as a nicer-looking alternative to a tuple of classes.

(The first is a necessary but not sufficient condition for the second.)

Both are cosmetic changes. Neither involves new functionality which 
doesn't already exist. I don't think use-cases come into it. This seems 
to me to be a pair of purely design questions:

* are Unions important enough to get an operator? (I think so.)

* is it reasonable to relax the prohibition on isinstance(obj, Union)?
  (I think so.)

although there are subtleties that need to be considered, as your post 
goes on to mention.


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/74PMWNRZLW2BJ2M6AXLKF5NMUPFF5CR2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [OT] Designing for Concurrency (was: Add a "block" option to Executor.submit)

[Andrew Barnert via Python-ideas]

One thing I think is worth raising here: If my experience with 
concurrent.futures is far from universal, my expectations for this change may 
not be helpful, so let me lay that all out explicitly.

In my experience, concurrent.futures has two very good, but very different, 
uses.

First, there’s trivial concurrency. The last stage of my processing is an 
obviously parallelizable pure function? Just toss it in an executor and I’m 
done. My asyncio code needs to call something blocking that doesn’t have an 
async equivalent? Toss it in an executor and I’m done.

On the other extreme, there are cases where your data flow is complicated, and 
organizing it all around the composability of futures helps rein in that 
complexity.

In between those two extremes, it’s often easier to use something 
different—e.g., a multiprocessing.Pool has a lot more options, and a lot more 
convenience functions for different ways of mapping; I’ve never missed Java’s 
thread-per-task executor because it’s easier to just use a Thread directly; you 
wouldn’t want a timer scheduling queue that hid all the time information under 
the covers; etc.

I’m expect this change will mostly be helpful for the trivial case. The last 
stage of my processing is obviously parallelizable? Just stick it in an 
executor and … wait, it’s still too slow, so the caller is wasting too many 
resources in the queue? Just stick it in a _bounded_ executor and I’m done.

In the really complicated cases, the backpressure is usually going to be in the 
future waiting, not in the task queuing, even if that’s more complicated to set 
up. Because otherwise the flow isn’t 100% composable anymore, which is the 
whole reason I was using concurrent.futures.

Anything in the moderately complicated range, and I‘m probably going to want an 
explicitly-managed queue rather than one hidden behind an abstraction, or a 
separate semaphore, or whatever. But then in those cases, I’m probably not even 
using concurrent.futures in the first place.

So, max_queue_len with blocking submit (and maybe also with raising 
submit_nowait) handles virtually all of the simple cases that I can think of or 
that anyone else has suggested, and I don’t think it matters much how many of 
the moderate or extremely complicated cases it handles.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CE56FZQODIWEP3PUSK4I2R2S4LESEJLB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [OT] Designing for Concurrency (was: Add a "block" option to Executor.submit)

[Andrew Barnert via Python-ideas]

On Sep 5, 2019, at 11:13, Barry Scott  wrote:
> 
> 
> 
>> On 5 Sep 2019, at 12:06, Dan Sommers <2qdxy4rzwzuui...@potatochowder.com> 
>> wrote:
>> 
>> Yes, you need some way to produce "back pressure" from downstream to
>> upstream, and to stop making new work (with new memory consumption)
>> until there's a place to put it.
> 
> It seems that this is the important idea.
> 
> How does the back pressure work in different use cases.
> 
> 1) I don't care.
> 
> Just add items to the queue. (The current API?)
> 
> 2) I can be blocked when the queue is at a limit.
> 
> Need an API to allows the block to happen
> and resumes the code when there is space in the queue.
> 
> 3) I cannot be blocked as this is async code, but I need a bound on the queue.
> 
> Maybe call "queue has space" callback when the queue has space?
> Maybe submit returns an "queue full" status?
> 
> I have needed all 3 uses cases.

Have you actually needed case 3 with Executor, or only with other kinds of 
async models?

Anyway, failing makes sense, is usable for backpressure, and is trivial to 
implement. Although I think it should raise an exception rather than return a 
special value in place of a tuple. Here’s a design (borrowed from Queue of 
course):

 * submit always succeeds, blocking if necessary
 * submit_nowait always completes immediately, raising executor.Full if 
necessary

And of course if you don’t pass a max_queue_len, they both always succeed 
immediately.

I’m not sure submit_nowait is needed. And if we add max_queue_len in 3.x and 
people start asking for submit_nowait, it would be trivial to add it in 3.x+1. 
But if someone has an actual use case for it now, I don’t see any problem at 
all adding it now. Do you have real code or realistic sample code that needs it?

I don’t think the queue-has-room callback, on the other hand, is very useful. 
That’s just clunkily simulating an unbounded queue on top of a bounded queue 
plus an implicit unbounded queue of callbacks.

Anyway, in theory, there are a zillion different things you could conceivably 
want to happen on failure, which is why Java and Ruby have a zillion different 
fail protocols, and Ruby lets you set a fail method if even that isn’t good 
enough. But in practice, I’m pretty sure either that people only need “block”, 
or that they only need “block” and “raise”. I’m just not sure which of those 
two.___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NJT3AWQTDIZWFUKDF6MURWRL5RB75X75/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Andrew Barnert via Python-ideas]

On Sep 5, 2019, at 04:24, Steven D'Aprano  wrote:
> 
> But having said all that, I'm not sure that we should be rejecting this 
> proposal on the basis of performance when we haven't got any working 
> code to measure performance of :)

Good point. But then if we never get any realistic use cases, it’ll probably 
already be rejected for that reason. :)

So this is just something we should keep in mind when considering examples, not 
something we should use as an open-and-shut argument on its own, but I think it 
was still worth raising by [whoever raised it that I quoted in snipped text].

> isinstance is a wrapper around PyObject_IsInstance(obj, class_or_tuple), 
> and if I'm reading the C code correctly, PyObject_IsInstance is roughly 
> equivalent to this Python pseudo-code:
> 
># Except in C, not Python
>def isinstance(obj, class_or_tuple):
>if type(class_or_tuple) is tuple:
>for C in class_or_tuple:
>if isinstance(obj, C):
>return True
>else:
>...

Wow, I didn’t realize it handled tuples recursively, but the docs say it does, 
and of course the docs and the code are right:

>>> isinstance(2, (dict, (int, str)))
True

Not really relevant to anything (Unions already explicitly handle that same 
thing at construction time: `Union[dict, Union[int, str]]` just returns 
`Union[dict, int, str]`), I’m just surprised that I never noticed that.

> If Union is a built-in, we could have something like this:
> 
>def isinstance(obj, class_or_tuple):
>if type(class_or_tuple) is Union:
>class_or_tuple = class_or_tuple.__union_params__
># followed by the same code as above
> 
> typing.Union already defines .__union_params__ which returns a tuple of 
> the classes used to construct the union, so in principle at least, there 
> need be no significant performance hit from supporting Unions.

That’s a great point.

And if type.__or__ is going to return a union type, we probably don’t want that 
to go lazily importing typing and pulling something out of it to call 
__getitem__ on, so we probably want that result to be builtin.

I don’t think we need Union itself to be a builtin. But 
typing.Union.__getitem__ needs to return the same kind of builtin as 
type.__or__ (which is presumably exposed as something like types.UnionType, not 
only exposed in the typing module) instead of returning a typing._GenericAlias, 
or the whole point of this proposal (that `int|str == Union[int, str]`) breaks.

That does raise some more bikeshedding  questions (what the constructor for 
types.union accepts, or whether it refuses to be constructed and forces you to 
use type.__or__ or Union.__getitem__; what its repr looks like; etc.).

And I suppose it also helps answer the question of why typing.Union is special 
for isinstance: it’s returning a completely different thing than every other 
generic’s __getitem__, so it’s not really the same kind of generic, so maybe 
nobody expects it to follow the same rules in the first place?

More generally, it changes the proposal to “create a new runtime union type 
that works the way you’d expect, then add syntax for that, and change 
typing.Union to take advantage of it”, which sounds conceptually better than 
“add syntax to creating typing.Union static types, and then add special support 
to just this one kind of static type to make it usable at runtime”, even if 
they’re close to equivalent in practice.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HAN5SB7QSG5TJJC7Q6ZNYIKDPAOWUJPO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Chris Angelico]

On Fri, Sep 6, 2019 at 4:12 AM Andrew Barnert  wrote:
>
> On Sep 5, 2019, at 02:15, Chris Angelico  wrote:
> >
> > On Thu, Sep 5, 2019 at 7:08 PM Andrew Barnert via Python-ideas
> >  wrote:
> >>
> >> Which means if they run into a situation where they’re checking types of a 
> >> zillion objects, they’re definitely not going to guess that it’s 5x as 
> >> slow, so there definitely going to misuse it.
> >>
> >
> > Hang on hang on what's this situation where you're checking types
> > of a zillion objects?
>
> Exception was the first one I thought of, but, as you argue, that’s a bit of 
> a special case. And my next thought was the kind of thing you’d do with 
> pattern matching in a different language but which would be more naturally 
> (and often more efficiently) done with method calls (or maybe 
> @singledispatch) in Python.
>
> But here’s an example from the stdlib, in the json module:
>
> `elif isinstance(value, …):` ... with a tuple today (to special-case list and 
> tuple).
>
> But the fact that I found an example in the stdlib in 2 minutes of searching 
> implies that this probably isn’t nearly as rare as you’d at first expect.
>

We need an implementation to benchmark before we can be sure, but part
of my "hang on" was that, even when there ARE lots of objects to
check, a single isinstance per object is a vanishingly small part of
the overall job. I suppose you might find a performance regression on
json.dumps([[]]*1) but the other costs are normally going to
dominate it.

In any case, though, the Union type can itself be special-cased inside
isinstance to make this efficient again (as Steven showed), which
means this is highly unlikely to be "5x as slow", and making this
entire subtread fairly moot :)

(As a side effect of this change, I wouldn't be sorry to bypass the
grammatical ambiguity (from history) of the comma in an except clause.
Currently "except Exc1, Exc2:" is a syntax error, but "except
Exc1|Exc2:" would be perfectly valid.)

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/E5BY2SOFPA7F5FOG7LJTIL2PK4U32F4C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: [OT] Designing for Concurrency (was: Add a "block" option to Executor.submit)

[Barry Scott]

> On 5 Sep 2019, at 12:06, Dan Sommers <2qdxy4rzwzuui...@potatochowder.com> 
> wrote:
> 
> Yes, you need some way to produce "back pressure" from downstream to
> upstream, and to stop making new work (with new memory consumption)
> until there's a place to put it.

It seems that this is the important idea.

How does the back pressure work in different use cases.

1) I don't care.

Just add items to the queue. (The current API?)

2) I can be blocked when the queue is at a limit.

Need an API to allows the block to happen
and resumes the code when there is space in the queue.

3) I cannot be blocked as this is async code, but I need a bound on the queue.

Maybe call "queue has space" callback when the queue has space?
Maybe submit returns an "queue full" status?

I have needed all 3 uses cases.

Barry

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/H625GQU2HYJ2WQFH6I4O5L3UCAYTQWBW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Andrew Barnert via Python-ideas]

On Sep 5, 2019, at 02:15, Chris Angelico  wrote:
> 
> On Thu, Sep 5, 2019 at 7:08 PM Andrew Barnert via Python-ideas
>  wrote:
>> 
>> On Sep 4, 2019, at 20:51, Inada Naoki  wrote:
>> 
 And you’re right, because int|str _looks_ better than (int, str) here, 
 many people will be encouraged to use it even though it’s slower, which 
 could potentially be a bad thing for some programs.
>>> 
>>> That's exactly my point.  If we say "you can use `isinstance(x, int |
>>> str)` for now", people
>>> may think it is a new and recommended way to write it.
>> 
>> Right; I was agreeing, and saying it may even be worse than you’re 
>> suggesting. There’s always the possibility that any shiny new feature looks 
>> like the One True Way and gets overused. But in this case, it looks more 
>> obviously right even to someone who’s not interested in shiny new features 
>> and has never heard the word “pythonic” or read a new version’s release 
>> notes. So they’re going to use it. Which means if they run into a situation 
>> where they’re checking types of a zillion objects, they’re definitely not 
>> going to guess that it’s 5x as slow, so there definitely going to misuse it.
>> 
> 
> Hang on hang on what's this situation where you're checking types
> of a zillion objects?

Exception was the first one I thought of, but, as you argue, that’s a bit of a 
special case. And my next thought was the kind of thing you’d do with pattern 
matching in a different language but which would be more naturally (and often 
more efficiently) done with method calls (or maybe @singledispatch) in Python.

But here’s an example from the stdlib, in the json module:

Deep inside the recursive encoder function, there’s a string of `elif 
isinstance(value, …):` calls. And at least one of them does a check with a 
tuple today (to special-case list and tuple for encoding as JSON arrays without 
needing to call any user callbacks). In a JSON structure with zillions of 
nodes, this check will happen zillions of times—once for every node that’s a 
list or tuple or any type that gets checked after those types.

This isn’t the most compelling example. Most nodes are probably captured before 
this check, and the ones lower on the chain are all the ones that are 
inherently slow to process so the added cost of getting there is less 
important, and so on. Plus, the very fact that the author thought of 
optimizations like that implies that the author would have had no problem 
profiling and replacing `list|tuple` with `(list, tuple)` if it makes a 
difference.

But the fact that I found an example in the stdlib in 2 minutes of searching 
implies that this probably isn’t nearly as rare as you’d at first expect.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7EGJJSIVZSD5M4DNSUO242FCS2LDU26U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Andrew Barnert via Python-ideas]

> On Sep 5, 2019, at 07:46, Sebastian Kreft  wrote:
> 
> How is this different to the discussion 
> https://mail.python.org/archives/list/python-ideas@python.org/thread/MFSL3U6V74PZN4AT4JFQMQXSMGOJ6F27/#MFSL3U6V74PZN4AT4JFQMQXSMGOJ6F27
>  ?

Well, that discussion started with an unworkable proposal, and generated a 
variety of different alternatives, some of which were workable, but all of 
which were buried deep in the discussion of the unworkable idea.

Polishing up one of those alternatives and extracting it out into its own new 
thread doesn’t seem like a bad thing to do.

A link back to the old thread might have been nice, but now you’ve fixed that. 
:)___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5EQHNJYPJLSI3DFU4XS73652UNHGYZMU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Jan Verbeek]

A problem with `(int, str)` that I believe hasn't been brought up yet is that 
it interferes with the existing use of subscription syntax, particularly by 
`Tuple`.

`Tuple[int, str]` is equivalent to `Tuple[(int, str)]` but not to 
`Tuple[Union[int, str]]`, because `__getitem__` receives a single tuple instead 
of multiple arguments.

I think all other subscriptable types could resolve the ambiguity in principle 
because they aren't variadic.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J3VM5PEV6BX5DH36QFUJFOWNM5QPKNKU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Brandt Bucher]

Also worth noting is that dict.skip would live in dict.__dict__... which is a 
dict.

Brandt

> On Sep 5, 2019, at 08:51, Andrew Barnert via Python-ideas 
>  wrote:
> 
>> On Sep 4, 2019, at 15:25, r...@hwyl.org wrote:
>> 
>> ..."skip" being a unique object defined in and accessible from dict (there 
>> may be better places to make it available). The dict insertion code then 
>> simply ignores those entries when creating the dict.
> 
> I’m a bit confused about exactly when this does and doesn’t come into play.
> 
> You say here “the dict insertion code” does it. Would they mean d[key]=d.skip 
> doesn’t insert anything? If so, then how could I put skip in a dict at all? 
> If there’s already a d[key] does that get left alone, or deleted? Wouldn’t 
> this even affect dict inserts that happen behind the scenes, like any time 
> you tried to assign dict.skip to a global variable or a class or instance 
> attribute, or pass it as a keyword argument? If an implementation doesn’t use 
> dict.insert to build dict literals, does that mean it doesn’t work in dict 
> literals on that implementation, even though that was the whole motivation 
> for the feature?
> 
> Maybe instead of starting with how it could be implemented, start with where 
> you ideally do and don’t want it to take effect, and then we can try to work 
> out how it could be implemented to do that, and how to specify it to require 
> such an implementation.
> 
>  * Obviously you want it in non-comprehension dict displays, because that’s 
> your example.
>  * What about comprehensions?
>  * Constructor calls with the keyword-arg API like dict(a=1, b=dict.skip, 
> **d2}?
>  * Constructor calls from iterables of pairs, like dict((1, 2), (3, 
> dict.skip))?
>  * Subscription assignments?
>  * Calls to insert?
>  * Calls to update?
>  * Capturing ** keyword args?
>  * Assignments to globals, attributes, etc. when those use a dict under the 
> covers (as they usually do!?
>  * Assignments to such things when they don’t use a dict (as with a slots 
> class instance)?
> 
> Also, what should happen with this case:
> 
>flag0, flag1 = True, False
>d = {'k': 0 if flag0 else dict.skip,
>'k': 1 if flag1 else dict.skip}
> 
> Do we keep the last value for 'k' as usual, and because that’s dict.skip, 
> therefore d['k'] ends up as a KeyError? Or do we keep the last non-skip value 
> and therefore d['k'] ends up as 0?
> 
> What about this:
> 
>class D(dict):
>skip = 0
>d = D()
>d.skip = 1
>d[0] = 0
>d[1] = 1
>d[dict.skip] = dict.skip
> 
> Do we skip the one that uses d.skip, the one that uses type(d).skip (like a 
> special method lookup), or the one that uses dict.skip?
> 
> 
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/GH3KENFHG6B2PXLV7JYZHQKHSUZ56NQ5/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TEDX373AAYX32OARYVABFUDOUDHYBKDT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Andrew Barnert via Python-ideas]

On Sep 4, 2019, at 15:25, r...@hwyl.org wrote:
> 
> ..."skip" being a unique object defined in and accessible from dict (there 
> may be better places to make it available). The dict insertion code then 
> simply ignores those entries when creating the dict.

I’m a bit confused about exactly when this does and doesn’t come into play.

You say here “the dict insertion code” does it. Would they mean d[key]=d.skip 
doesn’t insert anything? If so, then how could I put skip in a dict at all? If 
there’s already a d[key] does that get left alone, or deleted? Wouldn’t this 
even affect dict inserts that happen behind the scenes, like any time you tried 
to assign dict.skip to a global variable or a class or instance attribute, or 
pass it as a keyword argument? If an implementation doesn’t use dict.insert to 
build dict literals, does that mean it doesn’t work in dict literals on that 
implementation, even though that was the whole motivation for the feature?

Maybe instead of starting with how it could be implemented, start with where 
you ideally do and don’t want it to take effect, and then we can try to work 
out how it could be implemented to do that, and how to specify it to require 
such an implementation.

  * Obviously you want it in non-comprehension dict displays, because that’s 
your example.
  * What about comprehensions?
  * Constructor calls with the keyword-arg API like dict(a=1, b=dict.skip, 
**d2}?
  * Constructor calls from iterables of pairs, like dict((1, 2), (3, 
dict.skip))?
  * Subscription assignments?
  * Calls to insert?
  * Calls to update?
  * Capturing ** keyword args?
  * Assignments to globals, attributes, etc. when those use a dict under the 
covers (as they usually do!?
  * Assignments to such things when they don’t use a dict (as with a slots 
class instance)?

Also, what should happen with this case:

flag0, flag1 = True, False
d = {'k': 0 if flag0 else dict.skip,
'k': 1 if flag1 else dict.skip}

Do we keep the last value for 'k' as usual, and because that’s dict.skip, 
therefore d['k'] ends up as a KeyError? Or do we keep the last non-skip value 
and therefore d['k'] ends up as 0?

What about this:

class D(dict):
skip = 0
d = D()
d.skip = 1
d[0] = 0
d[1] = 1
d[dict.skip] = dict.skip

Do we skip the one that uses d.skip, the one that uses type(d).skip (like a 
special method lookup), or the one that uses dict.skip?


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GH3KENFHG6B2PXLV7JYZHQKHSUZ56NQ5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Sebastian Kreft]

How is this different to the discussion
https://mail.python.org/archives/list/python-ideas@python.org/thread/MFSL3U6V74PZN4AT4JFQMQXSMGOJ6F27/#MFSL3U6V74PZN4AT4JFQMQXSMGOJ6F27
 ?

On Thu, Sep 5, 2019 at 9:24 AM  wrote:

> Okie, looks like my code got munched in the web view -- how do I make
> it not do that?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/JNUMEIHLYVLC64MOVZ4P7ABMOOY3SJC4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Sebastian Kreft
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6JJWWUBAARDBCMW2BD2DWDMDHXT52S2V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[brian . skinn]

Okie, looks like my code got munched in the web view -- how do I make it 
not do that?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JNUMEIHLYVLC64MOVZ4P7ABMOOY3SJC4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[brian . skinn]

I thought of something similar, but where the dict-literal construction is 
desired:

>>> foo = True
>>> bar = False
>>> baz = False
>>> d = {
...   'foo' if foo else None: 1,
...   'bar' if bar else None: 2,
...   'baz' if baz else None: 3,
... }
>>> d
{'foo': 1, None: 3}
>>> d.pop(None)
3
>>> d
{'foo': 1}

The natural helper function for the "'foo' if foo else None" snip would then be 
something like:

>>> def key_flag(key, flag, default=None):
... return key if flag else default
...
>>> dd = {
...   key_flag('foo', foo): 1,
...   key_flag('bar', bar): 2,
...   key_flag('baz', baz): 3,
... }
>>> dd
{'foo': 1, None: 3}
>>> dd.pop(None)
3
>>> dd
{'foo': 1}

-Brian
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SZZVOFRPJEWTK5BJLEZVVSAZAL5F3OT7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Richard Musil]

I was thinking more in this direction:

d = dict()
d[action in not None and 'action'] = action
d[minval > 0 and 'minval'] = minval
...
del d[False]

There are few things I do not particularly like about this (assigning to
dict, even when value is discarded later, or necessity of the del
d[False]), but I guess it is as expressive as it could be and I could
imagine using it in my app easily. If needed one can even define another
key instead of False.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3IQ7GYOYKSSGKSWOGCFABF2KGHNY4PSY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Chris Angelico]

On Thu, Sep 5, 2019 at 9:31 PM Steven D'Aprano  wrote:
>
> On Thu, Sep 05, 2019 at 07:15:27PM +1000, Chris Angelico wrote:
>
> > Hang on hang on what's this situation where you're checking types
> > of a zillion objects?
>
> An earlier version of the statistics module used lots of isinstance
> checks in order to support arbitrary numeric types, and was a lot
> slower. The current version avoids most of those at the cost of being a
> lot less clear and elegant, but it improved performance somewhat.

That's fair, although if someone's concerned about squeaking the
performance as hard as possible, they'll probably be using numpy.

> On my PC, an isinstance check against a single concrete type (not an
> ABC) is about three times as expensive as a float arithmetic operation,
> so in a tight loop it may not be an insignificant cost.

How often do you check against a union type (or, in current code,
against a tuple of types)? This proposal wouldn't affect anything that
checks against a single type.

> > I think there's a bigger problem there than
> > whether isinstance(x, int|str) is slower than isinstance(x, (int,str))
> > ! Even if this change DOES have a measurable impact on the time to do
> > those checks, it only applies to unions, and if that's a notable
> > proportion of your total run time, maybe there's a better way to
> > architect this.
>
> Maybe... but in my experience, only at the cost of writing quite ugly
> code.

Perhaps. But for this to matter, you would need:
1) some sort of complicated dispatch handler that has to handle
subclasses (so you can't just look up type(x) in a dict)
2) handling of multiple types the same way (so you want to do union
isinstances rather than each one being done individually)
3) little enough other code that a performance regression in
isinstance makes a measurable difference
4) clean code that you don't want to disrupt for the sake of performance

Seems like a fairly uncommon case to me. Maybe I'm wrong.

> But having said all that, I'm not sure that we should be rejecting this
> proposal on the basis of performance when we haven't got any working
> code to measure performance of :)

Definitely. I don't think performance should be a major consideration
until code cleanliness has been proven or disproven.

> isinstance is a wrapper around PyObject_IsInstance(obj, class_or_tuple),
> and if I'm reading the C code correctly, PyObject_IsInstance is roughly
> equivalent to this Python pseudo-code:
>
> # Except in C, not Python
> def isinstance(obj, class_or_tuple):
> if type(class_or_tuple) is tuple:
> for C in class_or_tuple:
> if isinstance(obj, C):
> return True
> else:
> ...
>
>
> If Union is a built-in, we could have something like this:
>
> def isinstance(obj, class_or_tuple):
> if type(class_or_tuple) is Union:
> class_or_tuple = class_or_tuple.__union_params__
> # followed by the same code as above
>
> typing.Union already defines .__union_params__ which returns a tuple of
> the classes used to construct the union, so in principle at least, there
> need be no significant performance hit from supporting Unions.

That seems like a pretty good optimization!

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AROEQ5Q74PC6MWBUBQQY2BP4HNTFP65A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Conditional dict declarations

[Paul Moore]

On Thu, 5 Sep 2019 at 11:19,  wrote:
> I find a common requirement I have is to include one or more entries in a 
> dict only if some condition is met. Currently, the usual way of doing that is 
> to add one or more if statements.
>
> Simple (but hopefully relatable-to-real-world-code) example.
[...]
> This is a fairly common (tedious) pattern in my code, but I feel it could be 
> avoided. In my view it would be better if it was possible to conditionally 
> add the entry in the dict declaration.
>
> My proposal is that there should be a special value defined that results in a 
> dict entry not being added to the dict if either the key or the value equals 
> this value. There may be several ways to achieve this, but here's one way it 
> could work.

This is pretty easy to write as a helper function:

>>> def dct(**kw):
... return {k:v for k, v in kw.items() if v is not dct.skip}
...
>>> dct.skip = object()
>>>
>>> dct(a=12, b=None, c=dct.skip, d="Hello", e=dct.skip)
{'a': 12, 'b': None, 'd': 'Hello'}

Given this, I'm not sure it's a sufficiently useful improvement to be
worth adding to the builtin dictionary (although thanks for the
suggestion, I wouldn't have ever thought about writing a helper to
avoid the boilerplate without this posting to prompt me!)

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4B2LALDNOUN2AII7OPZM4TUURJHFF6NO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Steven D'Aprano]

On Thu, Sep 05, 2019 at 07:15:27PM +1000, Chris Angelico wrote:

> Hang on hang on what's this situation where you're checking types
> of a zillion objects?

An earlier version of the statistics module used lots of isinstance 
checks in order to support arbitrary numeric types, and was a lot 
slower. The current version avoids most of those at the cost of being a 
lot less clear and elegant, but it improved performance somewhat.

On my PC, an isinstance check against a single concrete type (not an 
ABC) is about three times as expensive as a float arithmetic operation, 
so in a tight loop it may not be an insignificant cost.


> I think there's a bigger problem there than
> whether isinstance(x, int|str) is slower than isinstance(x, (int,str))
> ! Even if this change DOES have a measurable impact on the time to do
> those checks, it only applies to unions, and if that's a notable
> proportion of your total run time, maybe there's a better way to
> architect this.

Maybe... but in my experience, only at the cost of writing quite ugly 
code.

But having said all that, I'm not sure that we should be rejecting this 
proposal on the basis of performance when we haven't got any working 
code to measure performance of :)

isinstance is a wrapper around PyObject_IsInstance(obj, class_or_tuple), 
and if I'm reading the C code correctly, PyObject_IsInstance is roughly 
equivalent to this Python pseudo-code:

# Except in C, not Python
def isinstance(obj, class_or_tuple):
if type(class_or_tuple) is tuple:
for C in class_or_tuple:
if isinstance(obj, C):
return True
else:
...


If Union is a built-in, we could have something like this:

def isinstance(obj, class_or_tuple):
if type(class_or_tuple) is Union:
class_or_tuple = class_or_tuple.__union_params__
# followed by the same code as above

typing.Union already defines .__union_params__ which returns a tuple of 
the classes used to construct the union, so in principle at least, there 
need be no significant performance hit from supporting Unions.



-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TJ65FQCNDOSCV365QQHUCRNQOEYA7GMC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] [OT] Designing for Concurrency (was: Add a "block" option to Executor.submit)

[Dan Sommers]

On 9/4/19 5:38 PM, Andrew Barnert via Python-ideas wrote:

> On Sep 4, 2019, at 08:54, Dan Sommers 
<2qdxy4rzwzuui...@potatochowder.com> wrote:


>> How does blocking the submit call differ from setting max_workers in
>> the call to ThreadPoolExecutor?

> Here’s a concrete example from my own code:

Aha.  Thanks.

> I need to create thousands of images, each of which is about 1MB
> uncompressed, but compressed down to a 40KB PNG that I save to disk.

> Compressing and saving takes 20-80x as long as creating, so I want to
> do that in parallel, so my program runs 16x as fast.

Without knowing anything else, I would wonder why you've combined
compressing (an apparently CPU bound operation) and saviing (an
apparently I/O bound operation) togther, but left creating separate.

Please don't answer that; it's not related to Python.

> But since 16 < 20, the main thread will still get ahead of the
> workers, and eventually I’ll have a queue with thousands of 1MB
> pixmaps in it, at which point my system goes into swap hell and slows
> to a crawl.

Yes, you need some way to produce "back pressure" from downstream to
upstream, and to stop making new work (with new memory consumption)
until there's a place to put it.

> If I bound the queue at length 16, the main thread automatically
> blocks whenever it gets too far ahead, and now I have a fixed memory
> use of about 33MB instead of unbounded GB, and my program really does
> run almost 16x as fast as the original serial version.

> And the proposal in this thread would allow me to do that with just a
> couple lines of code: construct an executor with a max queue length at
> the top, and replace the call to the compress-and-write function with
> a submit of that call, and I’m done.

> Could I instead move the pixmap creation into the worker tasks and
> rearrange the calculations and add locking so they could all share the
> accumulator state correctly? Sure, but it would be a lot more
> complicated, and probably a bit slower (since parallelizing code that
> isn’t in a bottleneck, and then adding locks to it, is a
> pessimization).

"[T]he accumulator?"  Is there only one data store to manage multiple
instances of three different/separate operations?  Ouch.  At least
that's my reaction to the limited amount of insight I have right now.

Now we're discussing how to desgign a concurrent application to maximize
resources and minimize complexity and overall run time.  Yes, if you
build your application one way, and run into issues, then some ways of
addressing the issues (e.g., going back to the design phase) will cost
more to implement than others (tweaking a supporting libraries).  It's
happened to all of us.  :-)

I'm not against tweaking the standard library (and even if I were, my
vote shouldn't count for much).  For *this case*, it seemed to me that
changing the standard library was Less Better™ than considering the
concurrency issues earlier on in the development process.

There's also an old software engineer inside me that wants most of this
control up near the top (where I can see it) as opposed to way down
inside the supporting libraries (where it becomes magic that has to be
debugged when it's not doing what I thought it would).  That's great for
new applications, but it doesn't always stay that way over time.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BW4NUVID6J4TVWZRFKQDYMNPG7SSAY4S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Conditional dict declarations

[rls]

Hi list

I have a proposal for a minor (improvement?) to dict declarations.

I find a common requirement I have is to include one or more entries in a dict 
only if some condition is met. Currently, the usual way of doing that is to add 
one or more if statements.

Simple (but hopefully relatable-to-real-world-code) example. Here we're 
constructing a query for a database based on the arguments passed in. In some 
cases we don't want to include some fields in the search at all depending on 
what the values passed in are.


###

def construct_query(user_id,
max_results=1,
active=None,
deleted=None):

query = {
'user_id': user_id
}

if max_results > 0:
query['max_results'] = max_results

if active is not None:
query['active'] = active

if deleted is not None:
query['deleted'] = deleted

return query

###


This is OK-ish, but seems unnecessarily verbose, and it gets worse if you have 
more conditional entries. In addition, you're separating the dict setup code 
into two or more places so it is slower to grasp mentally.

This is a fairly common (tedious) pattern in my code, but I feel it could be 
avoided. In my view it would be better if it was possible to conditionally add 
the entry in the dict declaration.

My proposal is that there should be a special value defined that results in a 
dict entry not being added to the dict if either the key or the value equals 
this value. There may be several ways to achieve this, but here's one way it 
could work.


###

def construct_query(user_id,
max_results=1,
active=None,
deleted=False):

return {
'user_id': user_id,
'max_results': max_results or dict.skip,
'active': active if active is not None else dict.skip,
'deleted': deleted if deleted is not None else dict.skip
}

###


..."skip" being a unique object defined in and accessible from dict (there may 
be better places to make it available). The dict insertion code then simply 
ignores those entries when creating the dict.

I think this would be relatively easy to achieve. No significant modification 
to the language, just a minor modification to the dict code (as far as I know).
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OYYCR7E5SPRRVBRYQYPZXSUAVLB7BZSA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Chris Angelico]

On Thu, Sep 5, 2019 at 7:08 PM Andrew Barnert via Python-ideas
 wrote:
>
> On Sep 4, 2019, at 20:51, Inada Naoki  wrote:
>
> >> And you’re right, because int|str _looks_ better than (int, str) here, 
> >> many people will be encouraged to use it even though it’s slower, which 
> >> could potentially be a bad thing for some programs.
> >
> > That's exactly my point.  If we say "you can use `isinstance(x, int |
> > str)` for now", people
> > may think it is a new and recommended way to write it.
>
> Right; I was agreeing, and saying it may even be worse than you’re 
> suggesting. There’s always the possibility that any shiny new feature looks 
> like the One True Way and gets overused. But in this case, it looks more 
> obviously right even to someone who’s not interested in shiny new features 
> and has never heard the word “pythonic” or read a new version’s release 
> notes. So they’re going to use it. Which means if they run into a situation 
> where they’re checking types of a zillion objects, they’re definitely not 
> going to guess that it’s 5x as slow, so there definitely going to misuse it.
>

Hang on hang on what's this situation where you're checking types
of a zillion objects? I think there's a bigger problem there than
whether isinstance(x, int|str) is slower than isinstance(x, (int,str))
! Even if this change DOES have a measurable impact on the time to do
those checks, it only applies to unions, and if that's a notable
proportion of your total run time, maybe there's a better way to
architect this.

The only situation I can think of would be exception handling. As a
special case, BaseException could perhaps have an __or__ method that
returns a tuple, but even there, I'd want to see a macrobenchmark that
shows that the difference actually affects a larger program
(especially since most exception checks are by hierarchy, not by
union).

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VNBDJPQDY767LMBUHFBJYMZ55HJWU5DG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "block" option to Executor.submit

[Andrew Barnert via Python-ideas]

On Sep 4, 2019, at 21:53, Anders Hovmöller  wrote:
> 
> 
>>> On 4 Sep 2019, at 22:58, Andrew Barnert  wrote:
>>> 
 On Sep 4, 2019, at 10:17, Anders Hovmöller  wrote:
 
 .
>>> 
>>> Doesn't all that imply that it'd be good if you could just pass it the 
>>> queue object you want?
>> 
>> Pass it a queue object that you construct? Or a queue factory (which would 
>> default to the SimpleQueue class, but you could pass, e.g., partial(Queue, 
>> max_len=10)? While either of those would be more flexible, it also breaks 
>> the abstraction, and the simplicity of the API.
> 
> Well we are talking about the case where the abstraction is broken already so 
> that seems like it's reasonable. 

No it’s not. Thread pools are a perfectly good abstraction, executors are a 
perfectly good further abstraction on that idea, with or without blocking (as 
evidenced by the many APIs that do have that feature, like Ruby’s, and the many 
APIs that don’t, like Win32’s, all of which are usable without having to know 
what’s inside them, and without confusion, or even disconcerting “wtf? Oh, I 
get it… I guess…” moments).

Being able to block on a fixed number of queued-up tasks is a useful feature. 
And if the only way to get it were to crack a seam in a solid abstraction, that 
would be a tradeoff worth discussing.

But in this case, there’s a perfectly good, and obvious, API that fits the 
abstraction seamlessly (and that already exists in a number of APIs with the 
same lineage of Python’s) that gives us that feature. So why not do it that way?

>> It might still be worth it if anyone had a use case for anything but a fixed 
>> queue length, or a custom type of queue, etc. But I suspect nobody does.
> 
> Well if the API is changed and we just add the fixed length parameter we'd 
> feel right stupid when the other valid use cases did show up :)

No, we’d feel appropriately conservative. :)

Seriously, this is exactly why we have the YAGNI principle. We naturally always 
want to design the most flexible and powerful and abstract thing possible even 
though there’s only one simple and concrete use for it. But it’s always easy to 
add more flexibility later if it turns out to be needed; taking flexibility 
away if it turns out to be confusing or buggy or whatever usually means 
breaking existing code. (Don’t take that past “take that into account and think 
twice” into “reject all flexibility without a second thought”, or you’ll end up 
in extreme programming dogma land, and then you’ll be forced to write Ruby 
instead of Python.)

This thread was started by someone who wanted blocking. I’ve had coworkers ask 
me how to do that, and seen dozens of questions on Stack Overflow, and 
implemented it myself two different ways, and I’m pretty sure lots of other 
real-world programs have done the same. So that’s a real need. Conversely, I 
don’t think I’ve ever seen anyone who wanted to use a different kind of queue 
for anything else, or anyone who used the more flexible in other libraries like 
Ruby’s to add anything but a queue bound and maybe a fail handler flag. So I’m 
not sure the flexibility buys us anything here.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SW5ZNOZZXZ27VPKTU5V5TNMG4IF2NWDE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Inspired by Scala, a new syntax for Union type

[Andrew Barnert via Python-ideas]

On Sep 4, 2019, at 20:51, Inada Naoki  wrote:

>> And you’re right, because int|str _looks_ better than (int, str) here, many 
>> people will be encouraged to use it even though it’s slower, which could 
>> potentially be a bad thing for some programs.
> 
> That's exactly my point.  If we say "you can use `isinstance(x, int |
> str)` for now", people
> may think it is a new and recommended way to write it.

Right; I was agreeing, and saying it may even be worse than you’re suggesting. 
There’s always the possibility that any shiny new feature looks like the One 
True Way and gets overused. But in this case, it looks more obviously right 
even to someone who’s not interested in shiny new features and has never heard 
the word “pythonic” or read a new version’s release notes. So they’re going to 
use it. Which means if they run into a situation where they’re checking types 
of a zillion objects, they’re definitely not going to guess that it’s 5x as 
slow, so there definitely going to misuse it.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/X7FZ3P6OJ3TM6OFS527N27DNKYWENOOU/
Code of Conduct: http://python.org/psf/codeofconduct/