Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

On 30.06.2017 00:57, Jan Kaliszewski wrote:

Please, note that it can be upturned: maybe they are not so common as
they could be because of all that burden with importing from separate
module -- after all we are saying about somewhat very simple operation,
so using lists and `+` just wins because of our (programmers')
laziness. :-)


Is it laziness or is it just "the best way to do it"™? ;-)


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Wes Turner]

On Saturday, July 1, 2017, Steven D'Aprano  wrote:

> On Sat, Jul 01, 2017 at 01:35:29AM -0500, Wes Turner wrote:
> > On Saturday, July 1, 2017, Steven D'Aprano  > wrote:
> >
> > > On Fri, Jun 30, 2017 at 01:09:51AM +0200, Jan Kaliszewski wrote:
> > >
> > > [...]
> > >
> > > But the more I think about it the more I agree with Nick. Let's start
> > > by moving itertools.chain into built-ins, with zip and map, and only
> > > consider giving it an operator after we've had a few years of
> experience
> > > with chain as a built-in. We might even find that an operator doesn't
> > > add any real value.
> >
> >
> > - Would that include chain.from_iterable?
>
> Yes.
>
> > - So there's then a new conditional import (e.g. in a compat package)?
> What
> > does this add?
>
> try:  chain
> except NameError:  from itertools import chain
>
> Two lines, if and only if you both need chain and want to support
> versions of Python older than 3.7.
>
> There's no need to import it if you aren't going to use it.


Or, can I just continue to import the same function from the same place:

  from itertools import chain

Nice, simple, easy. There's even (for all you functional lovers):

  from itertools import *

And, again, this works today:

  from fn import Stream
  itr = Stream() << my_generator() << (8,9,0)

-
https://github.com/kachayev/fn.py/blob/master/README.rst#streams-and-infinite-sequences-declaration
  - https://github.com/kachayev/fn.py/blob/master/fn/stream.py

- AFAIU, + doesn't work because e.g. numpy already defines + and & for
Iterable arrays.


>
>
> > > > ¹ Preferably using the existing `yield from` mechanism -- because, in
> > > > case of generators, it would provide a way to combine ("concatenate")
> > > > *generators*, preserving semantics of all that their __next__(),
> send(),
> > > > throw() nice stuff...
> > >
> > > I don't think that would be generally useful.
> >
> > Flatten one level?
>
> Flattening typically applies to lists and sequences.
>
> I'm not saying that chain shouldn't support generators. That would be
> silly: a generator is an iterable and chaining supports iterables. I'm
> saying that it wouldn't be helpful to require chain objects to support
> send(), throw() etc.


So the argspec is/shouldbe Iterables with __iter__ (but not necessarily
__len__)?


>
> > > If you're sending values
> > > into an arbitrary generator, who knows what you're getting? chain()
> will
> > > operate on arbitrary iterables, you can't expect to send values into
> > > chain([1, 2, 3], my_generator(), "xyz") and have anything sensible
> > > occur.
> >
> >
> > - is my_generator() mutable (e.g. before or during iteration)?
>
> It doesn't matter. Sending into a chain of arbitrary iterators isn't a
> useful thing to do.


So, with a generator function, I get a traceback at the current yield
statement. With chain() I get whatever line the chain call is on.


>
>
> > - https://docs.python.org/2/reference/expressions.html#generator.send
>
> Why are you linking to the 2 version of the docs? We're discusing a
> hypotheticial new feature which must go into 3, not 2.


In your opinion, has the send() functionality changed at all?


>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org 
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Paul Moore]

On 1 July 2017 at 07:13, Steven D'Aprano  wrote:
> But the more I think about it the more I agree with Nick. Let's start
> by moving itertools.chain into built-ins, with zip and map, and only
> consider giving it an operator after we've had a few years of experience
> with chain as a built-in. We might even find that an operator doesn't
> add any real value.

I'm struck here by the contrast between this and the "let's slim down
the stdlib" debates we've had in the past.

How difficult is it really to add "from itertools import chain" at the
start of a file? It's not even as if itertools is a 3rd party
dependency.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Chris Angelico]

On Sat, Jul 1, 2017 at 6:11 PM, Steven D'Aprano  wrote:
>> - So there's then a new conditional import (e.g. in a compat package)? What
>> does this add?
>
> try:  chain
> except NameError:  from itertools import chain
>
> Two lines, if and only if you both need chain and want to support
> versions of Python older than 3.7.
>
> There's no need to import it if you aren't going to use it.
>

It'd be even simpler. If you want to support <3.7 and 3.7+, you write:

from itertools import chain

At least, I presume it isn't going to be *removed* from itertools.
Promotion to builtin shouldn't break pre-existing code, so the way to
be compatible with pre-promotion Pythons is simply to code for those
and not take advantage of the new builtin.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Sat, Jul 01, 2017 at 01:35:29AM -0500, Wes Turner wrote:
> On Saturday, July 1, 2017, Steven D'Aprano  wrote:
> 
> > On Fri, Jun 30, 2017 at 01:09:51AM +0200, Jan Kaliszewski wrote:
> >
> > [...]
> >
> > But the more I think about it the more I agree with Nick. Let's start
> > by moving itertools.chain into built-ins, with zip and map, and only
> > consider giving it an operator after we've had a few years of experience
> > with chain as a built-in. We might even find that an operator doesn't
> > add any real value.
> 
> 
> - Would that include chain.from_iterable?

Yes.

> - So there's then a new conditional import (e.g. in a compat package)? What
> does this add?

try:  chain
except NameError:  from itertools import chain

Two lines, if and only if you both need chain and want to support 
versions of Python older than 3.7.

There's no need to import it if you aren't going to use it.


> > > ¹ Preferably using the existing `yield from` mechanism -- because, in
> > > case of generators, it would provide a way to combine ("concatenate")
> > > *generators*, preserving semantics of all that their __next__(), send(),
> > > throw() nice stuff...
> >
> > I don't think that would be generally useful.
> 
> Flatten one level?

Flattening typically applies to lists and sequences.

I'm not saying that chain shouldn't support generators. That would be 
silly: a generator is an iterable and chaining supports iterables. I'm 
saying that it wouldn't be helpful to require chain objects to support 
send(), throw() etc.

> > If you're sending values
> > into an arbitrary generator, who knows what you're getting? chain() will
> > operate on arbitrary iterables, you can't expect to send values into
> > chain([1, 2, 3], my_generator(), "xyz") and have anything sensible
> > occur.
> 
> 
> - is my_generator() mutable (e.g. before or during iteration)?

It doesn't matter. Sending into a chain of arbitrary iterators isn't a 
useful thing to do.


> - https://docs.python.org/2/reference/expressions.html#generator.send

Why are you linking to the 2 version of the docs? We're discusing a 
hypotheticial new feature which must go into 3, not 2.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Wes Turner]

On Saturday, July 1, 2017, Steven D'Aprano  wrote:

> On Fri, Jun 30, 2017 at 01:09:51AM +0200, Jan Kaliszewski wrote:
>
> [...]
>
> But the more I think about it the more I agree with Nick. Let's start
> by moving itertools.chain into built-ins, with zip and map, and only
> consider giving it an operator after we've had a few years of experience
> with chain as a built-in. We might even find that an operator doesn't
> add any real value.


- Would that include chain.from_iterable?
- So there's then a new conditional import (e.g. in a compat package)? What
does this add?


>
> > ¹ Preferably using the existing `yield from` mechanism -- because, in
> > case of generators, it would provide a way to combine ("concatenate")
> > *generators*, preserving semantics of all that their __next__(), send(),
> > throw() nice stuff...
>
> I don't think that would be generally useful.


Flatten one level?


>
> If you're sending values
> into an arbitrary generator, who knows what you're getting? chain() will
> operate on arbitrary iterables, you can't expect to send values into
> chain([1, 2, 3], my_generator(), "xyz") and have anything sensible
> occur.


- is my_generator() mutable (e.g. before or during iteration)?

- https://docs.python.org/2/reference/expressions.html#generator.send
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Fri, Jun 30, 2017 at 01:09:51AM +0200, Jan Kaliszewski wrote:

> But implementation of the OP's proposal does not need to be based on
> __add__ at all.  It could be based on extending the current behaviour of
> the `+` operator itself.
> 
> Now this behavior is (roughly): try left side's __add__, if failed try
> right side's __radd__, if failed raise TypeError.
> 
> New behavior could be (again: roughly): try left side's __add__, if
> failed try right side's __radd__, if failed try __iter__ of both sides
> and chain them (creating a new iterator¹), if failed raise TypeError.

That's what I suggested earlier, except using & instead of + as the 
operator.

The reason I suggested & instead of + is that there will be fewer 
clashes between iterables that already support the operator and hence 
fewer surprises.

Using + will be a bug magnet. Consider:


it = x + y  # chain two iterables
first = next(it, "default")


That code looks pretty safe, but it's actually a minefield waiting to 
blow you up. It works fine if you pass (let's say) a generator object 
and a string, or a list and an iterator, but if x and y happen to both 
be strings, or both lists, or both tuples, the + operator will 
concatenate them instead of chaining them, and the call to next will 
blow up.

So you would have to write:

it = iter(x + y)  # chain two iterables, and ensure the result is an iterator

to be sure. Which is not a huge burden, but it does take away the 
benefit of having an operator. In that case, you might as well do:

it = chain(x, y)

and be done with it.

It's true that exactly the same potential problem occurs with & but its 
less likely. Strings, tuples, lists and other sequences don't typically 
support __(r)and__ and the & operator, so you're less likely to be 
burned. Still, the possibility is there. Maybe we should use a different 
operator. ++ is out because that already has meaning, so that leaves 
either && or inventing some arbitrary symbol.

But the more I think about it the more I agree with Nick. Let's start 
by moving itertools.chain into built-ins, with zip and map, and only 
consider giving it an operator after we've had a few years of experience 
with chain as a built-in. We might even find that an operator doesn't 
add any real value.



> ¹ Preferably using the existing `yield from` mechanism -- because, in
> case of generators, it would provide a way to combine ("concatenate")
> *generators*, preserving semantics of all that their __next__(), send(),
> throw() nice stuff...

I don't think that would be generally useful. If you're sending values 
into an arbitrary generator, who knows what you're getting? chain() will 
operate on arbitrary iterables, you can't expect to send values into 
chain([1, 2, 3], my_generator(), "xyz") and have anything sensible 
occur.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[David Mertz]

We HAVE spellings for these things:

c from:
>>yield from a
>>yield from b
>>
>
c = chain(a, b)


> c as list from:
>>yield from a
>>yield from b
>>
>
c = list(chain(a, b))


> c as sum from:
>>yield from a
>>yield from b
>>
>
c = sum(chain(a, b))

Those really are not "crazy generator expressions."


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Cameron Simpson]

On 28Jun2017 09:54, Paul Moore  wrote:

On 28 June 2017 at 05:30, Terry Reedy  wrote:

On 6/27/2017 10:47 PM, Nick Coghlan wrote:


While I haven't been following this thread closely, I'd like to note
that arguing for a "chain()" builtin has the virtue that would just be
arguing for the promotion of the existing itertools.chain function
into the builtin namespace.

Such an approach has a lot to recommend it:

1. It has precedent, in that Python 3's map(), filter(), and zip(),
are essentially Python 2's itertools.imap(), ifilter(), and izip()
2. There's no need for a naming or semantics debate, as we'd just be
promoting an established standard library API into the builtin
namespace



A counter-argument is that there are other itertools that deserve promotion,
by usage, even more.  But we need to see comparisons from more that one
limited corpus.


Indeed. I don't recall *ever* using itertools.chain myself. I'd be
interested in seeing some usage stats to support this proposal. As an
example, I see 8 uses of itertools.chain in pip and its various
vendored packages, as opposed to around 30 uses of map (plus however
many list comprehensions are used in place of maps). On a very brief
scan, it looks like the various other itertools are used less than
chain, but with only 8 uses of chain, it's not really possible to read
anything more into the relative frequencies.


I don't use it often, but when I do it is very handy. While I'm not arguing for 
making it a builtin on the basis of my own use (though I've no objections 
either), a quick grep shows:


My maildb kit uses chain to assemble multiple related header values:

   *chain( msg.get_all(hdr, [])
   for hdr
   in ('to', 'cc', 'bcc', 'resent-to', 'resent-cc') )

Two examples where I use it to insert items in front of an iterable:

   chunks = chain( [data], chunks )

   blocks = indirect_blocks(chain( ( topblock, nexttopblock ), blocks ))

Neither of these is amenable to list rephrasings because the tail iterables 
("chunks" and "blocks") are of unknown and potentially large size.


And a few other cases whose uses are harder to succinctly describe, but 
generally "iterable flattening".


So it is uncommon for me, but very useful when I want it.

Just some (small) data points.

Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Koos Zevenhoven]

On Jun 30, 2017 2:23 PM, "Erik Bray"  wrote:

I actually really like this proposal, in additional to the original
proposal of using '+' to chain generators--I don't think it
necessarily needs to be extended to *all* iterables.  But this
proposal goes one better.  I just have to wonder what kind of strange
unexpected bugs would result.  For example now you could add a list to
a string:

>>> list(['a', 'b', 'c'] + 'def')
['a', 'b', 'c', 'd', 'e', 'f']

Personally, I really like this and find it natural.  But it will break
anything expecting this to be a TypeError.


Note that you can already do:

[*iterable1, *iterable2]

Or like in your example:

>>> [*['a', 'b', 'c'], *'def']
['a', 'b', 'c', 'd', 'e', 'f']

At least I think you can do that in 3.6 ;)

-- Koos (mobile)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Jan Kaliszewski]

2017-06-25 Serhiy Storchaka  dixit:

> 25.06.17 15:06, lucas via Python-ideas пише:

> > I often use generators, and itertools.chain on them.
> > What about providing something like the following:
> > 
> >  a = (n for n in range(2))
> >  b = (n for n in range(2, 4))
> >  tuple(a + b)  # -> 0 1 2 3
[...]
> It would be weird if the addition is only supported for instances of
> the generator class, but not for other iterators. Why (n for n in
> range(2)) 
> + (n for n in range(2, 4)) works, but iter(range(2)) + iter(range(2,
> 4)) and iter([0, 1]) + iter((2, 3)) don't? itertools.chain() supports 
> arbitrary iterators. Therefore you will need to implement the __add__ 
> method for *all* iterators in the world.
> 
> However itertools.chain() accepts not just *iterators*.
[...]

But implementation of the OP's proposal does not need to be based on
__add__ at all.  It could be based on extending the current behaviour of
the `+` operator itself.

Now this behavior is (roughly): try left side's __add__, if failed try
right side's __radd__, if failed raise TypeError.

New behavior could be (again: roughly): try left side's __add__, if
failed try right side's __radd__, if failed try __iter__ of both sides
and chain them (creating a new iterator¹), if failed raise TypeError.

And similarly, for `+=`: try __iadd__..., try __add__..., try
__iter__..., raise TypeError.

Cheers.
*j

¹ Preferably using the existing `yield from` mechanism -- because, in
case of generators, it would provide a way to combine ("concatenate")
*generators*, preserving semantics of all that their __next__(), send(),
throw() nice stuff...
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Jan Kaliszewski]

2017-06-27 Stephan Houben  dixit:

> Is "itertools.chain" actually that common?
> Sufficiently common to warrant its own syntax?

Please, note that it can be upturned: maybe they are not so common as
they could be because of all that burden with importing from separate
module -- after all we are saying about somewhat very simple operation,
so using lists and `+` just wins because of our (programmers')
laziness. :-)

Cheers.
*j
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

On 28.06.2017 20:37, Koos Zevenhoven wrote:

Oh, I've been very close to getting one of those. But then I should 
probably get a pair of glasses too ;).


:D

​​That pattern annoys people and negates the benefits of views and 
generators.​


Sure, that's why I am in favor of this proposal. It would remove the 
necessity to do that in various places. :)




Sure, but you may want to turn your unknown sources into something 
predictable as soon as possible. You'll need to deal with the errors 
in the input anyway.


That's a good point.

Very often one doesn't really need a list, but just something that has 
indexing, slicing and/or len(). Wrapping things with list() can be ok, 
but uses memory and is O(n). Generating lists from all kinds of 
iterables all the time is just a whole bunch of unnecessary overhead. 
But yes, it happens, because that's the convenient way of doing it 
now. That's like going back to Python 2, but with additional calls to 
list() required. Maybe you're lucky that your iterables are small and 
not a bottle neck and/or you just don't feel guilty every time you 
call list() where you shouldn't have to ;).


Yep, exactly. That's why I like an easier way of concating them with no 
bells and whistles. Preferably like lists today. ;)



Cheers,
Sven

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Koos Zevenhoven]

On Wed, Jun 28, 2017 at 9:01 PM, Sven R. Kunze  wrote:

> Good man. Today, a colleague of mine showed me a mobile mini-keyboard with
> a phone bracket (not even a dock). So, having his 7'' smartphone, he can
> work from his vacations and answer emails as well. ;) Cheap notebook
> replacement, if you don't prefer large screens and keyboards. :D
>
>
​Oh, I've been very close to getting one of those. But then I should
probably get a pair of glasses too ;).
​

> It all depends on what you need to do with the result of the
> concatenation. When all you need is something to iterate over, a
> generator-like thingy is fine. But when you need something for indexing and
> slicing or len etc., you want to be sure that that is what you're getting.
> But maybe someone passed you an argument that is not a sequence, or you
> forgot if a function returns a sequence or a generator. In that case, you
> want the error right away, instead of from some completely different piece
> of code somewhere that thinks it's getting a sequence. I don't think Python
> should depend on a static type checker to catch the error early. After all,
> type hints are optional.
>
>
> I understand that. In the end, I remember people on this mailing-list
> recommending me to use "list(...)" to make sure you got one in your hands.
> I remember this being necessary in the conversion process from Python2 to
> 3. The pattern is already here.
>
>
​That pattern annoys people and negates the benefits of views and
generators.​


> PS: I consider this proposal to be like allowing adding floats and ints
>> together. If I don't know if there was a float in the sum, don't know if my
>> result will be a float.
>>
>
> ​Not to say that the float/int case is never problematic, but the
> situation is still different. Often when a float makes any sense, you can
> work with either floats or ints and it doesn't really matter. But if you
> specifically *need* an int, you usually don't call functions that return
> floats. But if you do use division etc., you probably need to think about
> floor/ceil/closest anyway. And yes, there have probably been Python 2->3
> porting bugs where / division was not appropriately replaced with //.
>
>
> Division is one thing, numeric input parameters from unknown sources is
> another. In this regard, calling "int(...)" or "list(...)" follows the same
> scheme IMO.
>
>
​Sure, but you may want to turn your unknown sources into something
predictable as soon as possible. You'll need to deal with the errors in the
input anyway. Returning sequences vs generators is a different matter.​ You
don't want to turn generators into lists if you don't have to.

>
​[...]​

Just my 2 cents on this:
> At work, we heavily rely on Django. Django provides a so-called QuerySet
> type, its db-result abstraction. Among those querysets, our functions
> return lists and sets with no indication of whatsoever type it may be. It
> works quite well and we didn't had any issues with that. If we need a list,
> we wrap it with "list(...)". It's as simple as that. The valid concern,
> that it could be confusing which type the return value might have, is
> usually an abstract one. I can tell you that in practice it's not really an
> issue to talk about.
>

Very often one doesn't really need a list, but just something that has
indexing, slicing and/or len(). Wrapping things with list() can be ok, but
uses memory and is O(n). Generating lists from all kinds of iterables all
the time is just a whole bunch of unnecessary overhead. But yes, it
happens, because that's the convenient way of doing it now. That's like
going back to Python 2, but with additional calls to list() required. Maybe
you're lucky that your iterables are small and not a bottle neck and/or you
just don't feel guilty every time you call list() where you shouldn't have
to ;).

​-- Koos​

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Brendan Barnwell]

On 2017-06-28 09:16, Steven D'Aprano wrote:

On Tue, Jun 27, 2017 at 01:53:37PM -0700, Mike Miller wrote:

>
>On 2017-06-25 20:23, Steven D'Aprano wrote:

> >I have a counter-proposal: introduce the iterator chaining operator "&":
> >
> > iterable & iterable --> itertools.chain(iterable, iterable)
> >

>
>I like this suggestion.  Here's another color that might be less
>controversial:
>
> iterable3 = iterable1.chain(iterable2)

That requires every iterable class to add its own reimplementation of
chain, or else it will surprisingly not be chainable -- or at least it
*sometimes*  won't be chainable.

chain(iterable1, iterable2) would be more acceptable. The reason why a
function would be better here than a method is explained in the FAQ for
why len() is a function.


	I still think a good middle ground would be to have such a function, 
but have the return type of that function be an iterator that provides a 
.chain method or (better) defines __add__ to allow adding it to other 
iterables.  Then a single call to "chain" (or whatever the global 
function was called) would be enough to give you a nice readable syntax 
if you later want to chain other iterables on.  This behavior would not 
need to be stipulated for any other kinds of iterators; "chain" would 
just be a function that converts any iterable into a nicely chainable 
one, similar to how pathlib.Path converts a string into a nicely 
manipulable Path object that allows various handy path operations.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

On 28.06.2017 16:01, Koos Zevenhoven wrote:
For a moment, I was wondering what the double emphasis was for, but 
then I realized you are simply calling `statement.__why__()`​ directly 
instead of the recommended `spoiler(statement)`.


Doing this for years now. Sometimes, when 'statement.__why__()' returns 
None, 'spoiler(statement)' returns some thought-terminating cliché. ;)


But sure, I just got on vacation and I even found a power extension 
cord to use my laptop at the pool, so what else would I do ;).


Good man. Today, a colleague of mine showed me a mobile mini-keyboard 
with a phone bracket (not even a dock). So, having his 7'' smartphone, 
he can work from his vacations and answer emails as well. ;) Cheap 
notebook replacement, if you don't prefer large screens and keyboards. :D


It all depends on what you need to do with the result of the 
concatenation. When all you need is something to iterate over, a 
generator-like thingy is fine. But when you need something for 
indexing and slicing or len etc., you want to be sure that that is 
what you're getting. But maybe someone passed you an argument that is 
not a sequence, or you forgot if a function returns a sequence or a 
generator. In that case, you want the error right away, instead of 
from some completely different piece of code somewhere that thinks 
it's getting a sequence. I don't think Python should depend on a 
static type checker to catch the error early. After all, type hints 
are optional.


I understand that. In the end, I remember people on this mailing-list 
recommending me to use "list(...)" to make sure you got one in your 
hands. I remember this being necessary in the conversion process from 
Python2 to 3. The pattern is already here.



PS: I consider this proposal to be like allowing adding floats and
ints together. If I don't know if there was a float in the sum,
don't know if my result will be a float.


​Not to say that the float/int case is never problematic, but the 
situation is still different. Often when a float makes any sense, you 
can work with either floats or ints and it doesn't really matter. But 
if you specifically *need* an int, you usually don't call functions 
that return floats. But if you do use division etc., you probably need 
to think about floor/ceil/closest anyway. And yes, there have probably 
been Python 2->3 porting bugs where / division was not appropriately 
replaced with //.


Division is one thing, numeric input parameters from unknown sources is 
another. In this regard, calling "int(...)" or "list(...)" follows the 
same scheme IMO.


But regarding containers, it often makes just as much sense for a 
function to return a generator as it does to return a sequence. The 
name or purpose of a function may give no hint about whether an 
iterable or sequence is returned, and you can't expect everyone to 
prefix their function names with iter_ and seq_ etc. And it's not just 
function return values, it's also arguments passed into your function.


Exactly. Neither want I those prefixes. And I can tell you they aren't 
necessary in practice at all.


Just my 2 cents on this:
At work, we heavily rely on Django. Django provides a so-called QuerySet 
type, its db-result abstraction. Among those querysets, our functions 
return lists and sets with no indication of whatsoever type it may be. 
It works quite well and we didn't had any issues with that. If we need a 
list, we wrap it with "list(...)". It's as simple as that. The valid 
concern, that it could be confusing which type the return value might 
have, is usually an abstract one. I can tell you that in practice it's 
not really an issue to talk about.


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Wed, Jun 28, 2017 at 12:47:21PM +1000, Nick Coghlan wrote:

> While I haven't been following this thread closely, I'd like to note
> that arguing for a "chain()" builtin has the virtue that would just be
> arguing for the promotion of the existing itertools.chain function
> into the builtin namespace.

Yes, given the difficulties in choosing a operator, I'm leaning towards 
this approach now: promote itertools.chain to a builtin, and re-visit 
the operator suggestions in 3.8 or 3.9.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Tue, Jun 27, 2017 at 01:53:37PM -0700, Mike Miller wrote:
> 
> On 2017-06-25 20:23, Steven D'Aprano wrote:
> >I have a counter-proposal: introduce the iterator chaining operator "&":
> >
> > iterable & iterable --> itertools.chain(iterable, iterable)
> >
> 
> I like this suggestion.  Here's another color that might be less 
> controversial:
> 
> iterable3 = iterable1.chain(iterable2)

That requires every iterable class to add its own reimplementation of 
chain, or else it will surprisingly not be chainable -- or at least it 
*sometimes* won't be chainable.

chain(iterable1, iterable2) would be more acceptable. The reason why a 
function would be better here than a method is explained in the FAQ for 
why len() is a function.

The itertools chain function accepts *any* iterable. For this proposal 
to make sense, we can do no less. It isn't acceptable to:

- only support a subset of iterables (unless there's an easy 
  work-around);

- expect everyone to add a chain() method to their iterable classes;

- change the iterator protocol to require extra methods;

- conflict with the + operator that already works with sequences;

- choose an arbitrary operator that doesn't have at least some 
  association with concatenation or chaining or addition (e.g. "?" 
  would be unacceptible).

Ideally, we should also be able to avoid conflicting with other 
operators as well, but given that there's only a small set of ASCII 
symbols to choose from, we should at least consider using an existing 
operator so long as:

- we don't break existing code by changing the existing behaviour;

- any clashes between the new and old behaviour should be uncommon 
  (which is why + is unacceptible: using + on lists, tuples and 
  strings is too common);

- in any clash, the existing behaviour has priority over new behaviour
  (to avoid breaking existing code);

- and there's an easy work-around to get the new behaviour (say, call
  iter() on the object first).



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Koos Zevenhoven]

On Wed, Jun 28, 2017 at 3:18 PM, Sven R. Kunze  wrote:

> On 28.06.2017 14:00, Koos Zevenhoven wrote:
>
> The programmer needs to be well aware of whether the resulting object is a
> Sequence or 'just' a generator.
>
>
> Could you elaborate more on **why**?
>
>
​For a moment, I was wondering what the double emphasis was for, but then I
realized you are simply calling `statement.__why__()`​ directly instead of
the recommended `spoiler(statement)`. But sure, I just got on vacation and
I even found a power extension cord to use my laptop at the pool, so what
else would I do ;).

It all depends on what you need to do with the result of the concatenation.
When all you need is something to iterate over, a generator-like thingy is
fine. But when you need something for indexing and slicing or len etc., you
want to be sure that that is what you're getting. But maybe someone passed
you an argument that is not a sequence, or you forgot if a function returns
a sequence or a generator. In that case, you want the error right away,
instead of from some completely different piece of code somewhere that
thinks it's getting a sequence. I don't think Python should depend on a
static type checker to catch the error early. After all, type hints are
optional.

PS: I consider this proposal to be like allowing adding floats and ints
> together. If I don't know if there was a float in the sum, don't know if my
> result will be a float.
>

​Not to say that the float/int case is never problematic, but the situation
is still different. Often when a float makes any sense, you can work with
either floats or ints and it doesn't really matter. But if you specifically
*need* an int, you usually don't call functions that return floats. But if
you do use division etc., you probably need to think about
floor/ceil/closest anyway. And yes, there have probably been Python 2->3
porting bugs where / division was not appropriately replaced with //.

But regarding containers, it often makes just as much sense for a function
to return a generator as it does to return a sequence. The name or purpose
of a function may give no hint about whether an iterable or sequence is
returned, and you can't expect everyone to prefix their function names with
iter_ and seq_ etc. And it's not just function return values, it's also
arguments passed into your function.
​​
​-- Koos​


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

On 28.06.2017 14:00, Koos Zevenhoven wrote:
The programmer needs to be well aware of whether the resulting object 
is a Sequence or 'just' a generator.


Could you elaborate more on **why**?

Regards,
Sven

PS: I consider this proposal to be like allowing adding floats and ints 
together. If I don't know if there was a float in the sum, don't know if 
my result will be a float.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Koos Zevenhoven]

On Wed, Jun 28, 2017 at 11:54 AM, Paul Moore  wrote:

> Indeed. I don't recall *ever* using itertools.chain myself.

​
In fact, me neither. Or maybe a couple of times. For such a basic task, it
feels more natural to write a generator function, or even turn it into a
list, if you can be sure that the 'unnecessary' lists will be small and
that the code won't be a performance bottle neck.

To reiterate on this some more: One of the nice things of Python 3 is (or
could be) the efficiency of not making unnecessary lists by default. But
for the programmer/beginner it's not nearly as convenient with the views as
it is with lists. Beginners quickly need to learn about

Also generators are really nice, and chaining them is just as
useful/necessary as extending or concatenating lists. Chaining generators
with other iterables is certainly useful, but when all the parts of the
chained object are iterable but not sequences, that seems like an
invitation to use list() at some point in the code.

So whatever the outcome of this discussion (I hope there is one, whether it
is by adding iterator-related builtins or something more sophisticated), it
should probably take into account possible future ways of dealing with some
kind of "lazy lists". However, I'm actually not sure the syntax of chaining
generators/iterables should necessarily be the same as for chaining
arbitrary sequences. The programmer needs to be well aware of whether the
resulting object is a Sequence or 'just' a generator.

-- Koos

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

On 28.06.2017 11:09, Nick Coghlan wrote:

The other thing to look for would be list() and list.extend() calls. I
know I use those quite a bit in combination with str.join, where I
don't actually *need* a list, it's just currently the most convenient
way to accumulate all the components I'm planning to combine. And if
you're converting from Python 2 code, then adding a few list() calls
in critical places in order to keep the obj.extend() calls working is
likely to be easier in many cases than switching over to using
itertools.chain.


This is exactly the reason why I also doubt that Stephan's Sagemath 
stats are telling anything beyond "chain isn't used that much".


Iterators are only nice to have if you work with simple lists up to 1000 
items. Current hardware is able to fix that for you.


There are simply more readable ways of doing "chaining of sequences" for 
many cases. Even if you are already on Python 3.


In the end, list and "+" operator are the best way of "doing sequences".

Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Nick Coghlan]

On 28 June 2017 at 18:54, Paul Moore  wrote:
> On 28 June 2017 at 05:30, Terry Reedy  wrote:
>> A counter-argument is that there are other itertools that deserve promotion,
>> by usage, even more.  But we need to see comparisons from more that one
>> limited corpus.
>
> Indeed. I don't recall *ever* using itertools.chain myself. I'd be
> interested in seeing some usage stats to support this proposal. As an
> example, I see 8 uses of itertools.chain in pip and its various
> vendored packages, as opposed to around 30 uses of map (plus however
> many list comprehensions are used in place of maps). On a very brief
> scan, it looks like the various other itertools are used less than
> chain, but with only 8 uses of chain, it's not really possible to read
> anything more into the relative frequencies.

The other thing to look for would be list() and list.extend() calls. I
know I use those quite a bit in combination with str.join, where I
don't actually *need* a list, it's just currently the most convenient
way to accumulate all the components I'm planning to combine. And if
you're converting from Python 2 code, then adding a few list() calls
in critical places in order to keep the obj.extend() calls working is
likely to be easier in many cases than switching over to using
itertools.chain.

For simple cases, that's fine (since a list of direct references will
be lower overhead than accumulating a chain of short iterables), but
without builtin support for iterator concatenation, it's currently a
nonlocal refactoring (to add the "from itertools import chain" at the
top of the file) to switch completely to the "pervasive iterators"
model when folks actually do want to do that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Paul Moore]

On 28 June 2017 at 05:30, Terry Reedy  wrote:
> On 6/27/2017 10:47 PM, Nick Coghlan wrote:
>
>> While I haven't been following this thread closely, I'd like to note
>> that arguing for a "chain()" builtin has the virtue that would just be
>> arguing for the promotion of the existing itertools.chain function
>> into the builtin namespace.
>>
>> Such an approach has a lot to recommend it:
>>
>> 1. It has precedent, in that Python 3's map(), filter(), and zip(),
>> are essentially Python 2's itertools.imap(), ifilter(), and izip()
>> 2. There's no need for a naming or semantics debate, as we'd just be
>> promoting an established standard library API into the builtin
>> namespace
>
>
> A counter-argument is that there are other itertools that deserve promotion,
> by usage, even more.  But we need to see comparisons from more that one
> limited corpus.

Indeed. I don't recall *ever* using itertools.chain myself. I'd be
interested in seeing some usage stats to support this proposal. As an
example, I see 8 uses of itertools.chain in pip and its various
vendored packages, as opposed to around 30 uses of map (plus however
many list comprehensions are used in place of maps). On a very brief
scan, it looks like the various other itertools are used less than
chain, but with only 8 uses of chain, it's not really possible to read
anything more into the relative frequencies.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Nick Coghlan]

On 28 June 2017 at 14:30, Terry Reedy  wrote:
> On 6/27/2017 10:47 PM, Nick Coghlan wrote:
>> Such an approach has a lot to recommend it:
>>
>> 1. It has precedent, in that Python 3's map(), filter(), and zip(),
>> are essentially Python 2's itertools.imap(), ifilter(), and izip()
>> 2. There's no need for a naming or semantics debate, as we'd just be
>> promoting an established standard library API into the builtin
>> namespace
>
> A counter-argument is that there are other itertools that deserve promotion,
> by usage, even more.  But we need to see comparisons from more that one
> limited corpus.
>
> On the other hand, there might be a theory argument that chain is somehow
> more basic, akin to map, etc, in a way that others are not.

The main rationale I see is the one that kicked off the most recent
discussion, which is that in Python 2, you could readily chain the
output of map(), filter(), zip(), range(), dict.keys(), dict.values(),
dict.items(), etc together with "+", simply because they all returned
concrete lists.

In Python 3, you can't do that as easily anymore, since they all
return either iterators or computed containers that don't support "+".

While there are good reasons not to implement "+" on those iterators
and custom containers, we *can* fairly easily restore builtin
concatentation support for their outputs, and we can do it in a way
that's friendly to all implementations that already provide the
itertools.chain API.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Terry Reedy]

On 6/27/2017 10:47 PM, Nick Coghlan wrote:


While I haven't been following this thread closely, I'd like to note
that arguing for a "chain()" builtin has the virtue that would just be
arguing for the promotion of the existing itertools.chain function
into the builtin namespace.

Such an approach has a lot to recommend it:

1. It has precedent, in that Python 3's map(), filter(), and zip(),
are essentially Python 2's itertools.imap(), ifilter(), and izip()
2. There's no need for a naming or semantics debate, as we'd just be
promoting an established standard library API into the builtin
namespace


A counter-argument is that there are other itertools that deserve 
promotion, by usage, even more.  But we need to see comparisons from 
more that one limited corpus.


On the other hand, there might be a theory argument that chain is 
somehow more basic, akin to map, etc, in a way that others are not.



3. Preserving compatibility with older versions is straightforward:
just do an unconditional "from itertools import chain"
4. As an added bonus, we'd also get "chain.from_iterable" as a builtin API

So it would be good to have a short PEP that argued that since
chaining arbitrary iterables is at least as important as mapping,
filtering, and zipping them, itertools.chain should be added to the
builtin namespace in 3.7+ (but no, I'm not volunteering to write that
myself).

As a *separate* discussion, folks could then also argue for the
additional of a `__lshift__` operator implementation specifically to
iterator chains that let you write:

 full_chain = chain(it1) << it2 << it3 # Incrementally create new chains
 full_chain <<= it4  # Extend an existing chain

I'd be surprised if such a proposal got accepted for 3.7, but it would
make a good follow-up discussion for 3.8 (assuming chain() made it
into the 3.7 builtins).


--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Nick Coghlan]

On 28 June 2017 at 07:13, Mike Miller  wrote:
>
> On 2017-06-27 14:05, Brendan Barnwell wrote:
>>
>>  Even if this "chain" only took one argument, you could do
>> it1.chain(it2).chain(it3).  But I don't see why it couldn't take multiple
>> arguments as you suggest.
>
> Right, and as I forgot to mention, making it a built-in is an uphill battle
> with higher backward compatibility concerns.

While I haven't been following this thread closely, I'd like to note
that arguing for a "chain()" builtin has the virtue that would just be
arguing for the promotion of the existing itertools.chain function
into the builtin namespace.

Such an approach has a lot to recommend it:

1. It has precedent, in that Python 3's map(), filter(), and zip(),
are essentially Python 2's itertools.imap(), ifilter(), and izip()
2. There's no need for a naming or semantics debate, as we'd just be
promoting an established standard library API into the builtin
namespace
3. Preserving compatibility with older versions is straightforward:
just do an unconditional "from itertools import chain"
4. As an added bonus, we'd also get "chain.from_iterable" as a builtin API

So it would be good to have a short PEP that argued that since
chaining arbitrary iterables is at least as important as mapping,
filtering, and zipping them, itertools.chain should be added to the
builtin namespace in 3.7+ (but no, I'm not volunteering to write that
myself).

As a *separate* discussion, folks could then also argue for the
additional of a `__lshift__` operator implementation specifically to
iterator chains that let you write:

full_chain = chain(it1) << it2 << it3 # Incrementally create new chains
full_chain <<= it4  # Extend an existing chain

I'd be surprised if such a proposal got accepted for 3.7, but it would
make a good follow-up discussion for 3.8 (assuming chain() made it
into the 3.7 builtins).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Wes Turner]

On Monday, June 26, 2017, Wes Turner  wrote:

>
>
> On Sunday, June 25, 2017, Wes Turner  > wrote:
>
>>
>>
>> On Sunday, June 25, 2017, Danilo J. S. Bellini 
>> wrote:
>>
>>> On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas <
>>> python-ideas@python.org> wrote:
>>>
 I often use generators, and itertools.chain on them.
 What about providing something like the following:

 a = (n for n in range(2))
 b = (n for n in range(2, 4))
 tuple(a + b)  # -> 0 1 2 3
>>>
>>>
>>> AudioLazy does that: https://github.com/danilobellini/audiolazy
>>>
>>
>> - http://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.concat
>> and concatv
>>
>> - https://github.com/kachayev/fn.py#streams-and-infinite-seque
>> nces-declaration
>>  - Stream() << obj
>>
>
> << is __lshift__()
> <<= is __ilshift__()
>
> https://docs.python.org/2/library/operator.html
>
> Do Stream() and __lshift__() from fn.py not solve here?
>

In this syntax example, iter1 is mutated before iteration:

  iter_x = Stream(iter1) << iter2 << iter3
  iter1 <<= iter5 # IDK if fn.py yet has <<=
  list(iter1)
  list(iter_x)



>
>
>>
>>
>>>
>>>
>>> --
>>> Danilo J. S. Bellini
>>> ---
>>> "*It is not our business to set up prohibitions, but to arrive at
>>> conventions.*" (R. Carnap)
>>>
>>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Mike Miller]

On 2017-06-27 14:02, David Mertz wrote:

 iterable3 = iterable1.chain(iterable2)


How do you chain it1, it2, it3, etc?


Why not:

iterable5 = iterable1.chain(iterable2, iterable3, iterable4)

?  i.e. Couldn't a class method do this with itertools.chain() under the hood?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Brendan Barnwell]

On 2017-06-27 14:02, David Mertz wrote:

On Tue, Jun 27, 2017 at 1:57 PM, Mike Miller > wrote:

I like this suggestion.  Here's another color that might be less
controversial:

 iterable3 = iterable1.chain(iterable2)


How do you chain it1, it2, it3, etc?

I guess `it1.chain(it2.chain(it3)))` ... but that starts to become
distinctly less readable IMO.  I'd much rather spell `chain(it1, it2, it3)`.


	Even if this "chain" only took one argument, you could do 
it1.chain(it2).chain(it3).  But I don't see why it couldn't take 
multiple arguments as you suggest.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[David Mertz]

On Tue, Jun 27, 2017 at 1:57 PM, Mike Miller 
wrote:

> I like this suggestion.  Here's another color that might be less
> controversial:
>
> iterable3 = iterable1.chain(iterable2)
>

How do you chain it1, it2, it3, etc?

I guess `it1.chain(it2.chain(it3)))` ... but that starts to become
distinctly less readable IMO.  I'd much rather spell `chain(it1, it2, it3)`.


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Mike Miller]

On 2017-06-25 20:23, Steven D'Aprano wrote:
> I have a counter-proposal: introduce the iterator chaining operator "&":
>
>  iterable & iterable --> itertools.chain(iterable, iterable)
>

I like this suggestion.  Here's another color that might be less controversial:

iterable3 = iterable1.chain(iterable2)


Perhaps more obvious than &, easier to use than "from itertools import 
chain...".

-Mike
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

On 27.06.2017 21:27, David Mertz wrote:
And I would like a language change that made a number of common 
iterable objects "chainable" without the wrapper.  This wrapper could 
of course be used as a decorator too.


E.g. generator comprehensions, things returned by itertools functions, 
range(), enumerate(), zip(), etc.  This wouldn't promise that EVERY 
iterable or iterator had that "chainable" behavior, but it would cover 
90% of the use cases.  And I wouldn't find it confusing because the 
leftmost object would be the one determining the behavior, which feels 
more intuitive and predictable.


I think that most people in favor of this proposal agree with you here. 
Let's start with something simple which can be extended bit by bit to 
cover more and more use-cases.


I for one would include also right-handed iterators/generator because I 
definitely know real-world usage (just recently) but that can wait until 
you feel comfortable with it as well.


If it's a language change, I would like the plus operator to be it as it 
integrated well with lists, such as


generator + (list1 + list2)

It can be sometimes necessary to group things up like this. Using a 
different operator symbol here, I would find confusing. Plus looks like 
concat to me.


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Brendan Barnwell]

On 2017-06-27 00:47, David Mertz wrote:

Maybe if you are confident a and b are exactly NumPy arrays it is
obvious, but what about:

 from some_array_library import a
 from other_array_library import b

What do you think `a & b` will do under your proposal? Yes, I understand
it's deterministic... but it's far from *obvious*.  This isn't even
doing something pathological like defining both `.__iter__()` and
`.__and__()` on the same class... which honestly, isn't even all that
pathological; I can imagine real-world use cases.


	Hmmm, is the proposal really meant to include behavior that global and 
non-overridable?  My understanding was that the proposal would 
effectively be like defining a default __and__ (or whatever) on some 
basic iterator types.  Individual iterables (or iterators) could still 
define their own magic methods to define their own behavior.  Because of 
that, I wouldn't expect it to be obvious what would happen in your case. 
 If I import types from two random libraries, I can't expect to know 
what any operator does on them without reading their docs.


	Also because of that, I think it might be a bit much to expect this new 
concat operator to work on ALL iterables/iterators/  Nothing else really 
works that way; types have to define their own operator behavior. 
Iterators can be instances of any class that defines a __next__, so I 
don't see how we could support the magic-concat-everything operator 
without interfering there.


	So. . . wouldn't it be somewhat more reasonable to define this concat 
operator only on actual generators, and perhaps instances of the common 
iterator types returned from zip, enumerate, etc.?  Someone earlier in 
the thread said that would be "weird" because it would be less generic 
than itertools.chain, but it seems to me it would cover most of the 
needed use cases (including the one that was initially given as a 
motivating example).  Also, this generator.__add__ could be smart about 
handling other iterables on the right of the operator, so you could do


(x for x in blah) + [1, 2, 3] + "hello"

. . .and, as long as you started with a regular generator, it could 
work, by having the magic method return a new instance of a type that 
also has this behavior.


	This would be a bit odd because I think most existing Python types 
don't try to do this kind of thing (subsuming many different types of 
right-operands).  But I think it would be useful.  In the interim, it 
could be played with by just making a class that implements the __add__ 
(or whatever), so you could do


cool_iterator(x for x in blah) + [1, 2, 3] + "hello"

. . . and just wrapping the leftmost operand would be enough to give you 
nice syntax for chaining all the rest.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Joshua Morton]

Argh, you're correct. Thanks for the catch.

On Tue, Jun 27, 2017 at 10:06 AM Stephan Houben 
wrote:

> Unfortunately this is existing syntax:
>
> a++b
> is parsed as
> a+(+b)
>
> Stephan
>
>
> Op 27 jun. 2017 6:03 p.m. schreef "Joshua Morton" <
> joshua.morto...@gmail.com>:
>
> Just another syntactical suggestion: the binary ++ operator is used as
> concat in various contexts in various languages, and is probably less
> likely to confuse people as being either a logical or binary &.
>
> On Tue, Jun 27, 2017 at 6:53 AM Stephan Houben 
> wrote:
>
>> > Its the applications where it *is* important that
>> > we should be looking at.
>>
>> Um, yes, but given our relative positions in this debate,
>> the onus is not really on *me* to demonstrate such an application, right?
>> That would just confuse everbody ;-)
>>
>> (FWIW, Sagemath is not mostly "numerical processing", it is mostly
>> *symbolic* calculations and involves a lot of complex algorithms and
>> datastructures, including sequences.)
>>
>> Stephan
>>
>> 2017-06-27 13:48 GMT+02:00 Steven D'Aprano :
>> > On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:
>> >> Hi Steven,
>> >>
>> >> To put this into perspective, I did some greps on Sagemath,
>> >> being the largest Python project I have installed on this machine
>> >> (1955 .py files).
>> >
>> > And one which is especially focused on numerical processing, not
>> > really the sort of thing that does a much iterator chaining. That's
>> > hardly a fair test -- we know there are applications where chaining is
>> > not important at all. Its the applications where it *is* important that
>> > we should be looking at.
>> >
>> >
>> > --
>> > Steve
>> > ___
>> > Python-ideas mailing list
>> > Python-ideas@python.org
>> > https://mail.python.org/mailman/listinfo/python-ideas
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Stephan Houben]

Unfortunately this is existing syntax:

a++b
is parsed as
a+(+b)

Stephan


Op 27 jun. 2017 6:03 p.m. schreef "Joshua Morton" :

Just another syntactical suggestion: the binary ++ operator is used as
concat in various contexts in various languages, and is probably less
likely to confuse people as being either a logical or binary &.

On Tue, Jun 27, 2017 at 6:53 AM Stephan Houben  wrote:

> > Its the applications where it *is* important that
> > we should be looking at.
>
> Um, yes, but given our relative positions in this debate,
> the onus is not really on *me* to demonstrate such an application, right?
> That would just confuse everbody ;-)
>
> (FWIW, Sagemath is not mostly "numerical processing", it is mostly
> *symbolic* calculations and involves a lot of complex algorithms and
> datastructures, including sequences.)
>
> Stephan
>
> 2017-06-27 13:48 GMT+02:00 Steven D'Aprano :
> > On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:
> >> Hi Steven,
> >>
> >> To put this into perspective, I did some greps on Sagemath,
> >> being the largest Python project I have installed on this machine
> >> (1955 .py files).
> >
> > And one which is especially focused on numerical processing, not
> > really the sort of thing that does a much iterator chaining. That's
> > hardly a fair test -- we know there are applications where chaining is
> > not important at all. Its the applications where it *is* important that
> > we should be looking at.
> >
> >
> > --
> > Steve
> > ___
> > Python-ideas mailing list
> > Python-ideas@python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Joshua Morton]

Just another syntactical suggestion: the binary ++ operator is used as
concat in various contexts in various languages, and is probably less
likely to confuse people as being either a logical or binary &.

On Tue, Jun 27, 2017 at 6:53 AM Stephan Houben  wrote:

> > Its the applications where it *is* important that
> > we should be looking at.
>
> Um, yes, but given our relative positions in this debate,
> the onus is not really on *me* to demonstrate such an application, right?
> That would just confuse everbody ;-)
>
> (FWIW, Sagemath is not mostly "numerical processing", it is mostly
> *symbolic* calculations and involves a lot of complex algorithms and
> datastructures, including sequences.)
>
> Stephan
>
> 2017-06-27 13:48 GMT+02:00 Steven D'Aprano :
> > On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:
> >> Hi Steven,
> >>
> >> To put this into perspective, I did some greps on Sagemath,
> >> being the largest Python project I have installed on this machine
> >> (1955 .py files).
> >
> > And one which is especially focused on numerical processing, not
> > really the sort of thing that does a much iterator chaining. That's
> > hardly a fair test -- we know there are applications where chaining is
> > not important at all. Its the applications where it *is* important that
> > we should be looking at.
> >
> >
> > --
> > Steve
> > ___
> > Python-ideas mailing list
> > Python-ideas@python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Stephan Houben]

> Its the applications where it *is* important that
> we should be looking at.

Um, yes, but given our relative positions in this debate,
the onus is not really on *me* to demonstrate such an application, right?
That would just confuse everbody ;-)

(FWIW, Sagemath is not mostly "numerical processing", it is mostly
*symbolic* calculations and involves a lot of complex algorithms and
datastructures, including sequences.)

Stephan

2017-06-27 13:48 GMT+02:00 Steven D'Aprano :
> On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:
>> Hi Steven,
>>
>> To put this into perspective, I did some greps on Sagemath,
>> being the largest Python project I have installed on this machine
>> (1955 .py files).
>
> And one which is especially focused on numerical processing, not
> really the sort of thing that does a much iterator chaining. That's
> hardly a fair test -- we know there are applications where chaining is
> not important at all. Its the applications where it *is* important that
> we should be looking at.
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:
> Hi Steven,
> 
> To put this into perspective, I did some greps on Sagemath,
> being the largest Python project I have installed on this machine
> (1955 .py files).

And one which is especially focused on numerical processing, not 
really the sort of thing that does a much iterator chaining. That's 
hardly a fair test -- we know there are applications where chaining is 
not important at all. Its the applications where it *is* important that 
we should be looking at.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

TL;DR

If people really object to & doing double-duty as bitwise-AND and 
chaining, there's always && as a possibility.


On Tue, Jun 27, 2017 at 12:47:40AM -0700, David Mertz wrote:

> > - keep the existing __and__ and __rand__ behaviour;
> > - if they are not defined, and both operands x, y are iterable,
> >   return chain(x, y);
> >
> 
> I understand.  But it invites confusion about just what the `&` operator
> will do for a given iterable.

With operator overloading, that's a risk for any operator, and true for 
everything except literals.

What will `x * y` do? How about `y * x`? They're not even guarenteed to 
call the same dunder method.

But this is a problem in theory far more than in practice. In practice, 
you know what types you are expecting, and if you don't get them, you 
either explicitly raise an exception, or wait for something to fail. 
"Consenting adults" applies, and we often put responsibility back on the 
caller to do the right thing.

If you expect to use & on iterable arguments, it is reasonable 
to put the responsibility on the caller to only provide "sensible" 
iterables, not something wacky like an infinite generator (unless your 
code is explicitly documented as able to handle them) or one that 
overrides __(r)and__. Or you could check for it yourself, if you don't 
trust the argument:

if hasattr(arg, '__and__') or hasattr(arg, '__rand__'):
raise Something

But that strikes me as overkill. You don't normally check for dunders 
before using an operator, and we already have operators that can return 
different types depending on the operands:

% can mean modulo division or string interpolation
* can mean sequence repetition or numeric multiplication
+ can mean numeric addition or sequence concatenation

Why is 

& can mean ierable chaining or bitwise-AND

uniquely confusing?


> For NumPy itself, you don't really want to
> spell `chain(a, b)` so much.  But you CAN, they are iterables.  The
> idiomatic way is:
> 
> >>> np.concat((a,b))
> array([1, 2, 3, 4, 5, 6])

That's not really chaining, per say -- it is concatenating two arrays to 
create a third array. (Probably by copying the array elements.)

If you wanted to chain a numpy array, you would either use 
itertools.chain directly, or call iter(myarray) before using the & 
operator.


> However, for "any old iterable" it feels very strange to need to inspect
> the .__and__() and .__rand__ ()  methods of the things on both sides before
> you know WHAT operator it is.

Do you inspect the dunder methods of objects before calling + or * or & 
currently? Why would you need to start now?

Since there's no way of peering into an object's dunder methods and 
seeing what they do (short of reading the source code), you always have 
an element of trust and hope whenever you call an operator on anything 
but a literal.


> Maybe if you are confident a and b are exactly NumPy arrays it is obvious,
> but what about:
> 
> from some_array_library import a
> from other_array_library import b
> 
> What do you think `a & b` will do under your proposal? Yes, I understand
> it's deterministic... but it's far from *obvious*.

It's not "obvious" now, since a and b can do anything they like in their 
dunder methods. They could even inspect the call stack from __next__ and 
if they see "chain" in the stack, erase your hard drive. Does that mean 
we don't dare call chain(a, b)?

I don't think this proposal adds any new risk. All the risk already 
exists as soon as you allow method overloading. All we're doing is 
saying is "try the method overloading first, and if that isn't 
supported, try iterable chaining second before raising TypeError". 



> This isn't even doing
> something pathological like defining both `.__iter__()` and `.__and__()` on
> the same class... which honestly, isn't even all that pathological; I can
> imagine real-world use cases.

Indeed -- numpy arrays probably do that, as do sets and frozensets. I 
didn't say it was pathological, I said it was uncommon.


> I think that chaining iterators is common enough and important enough in
> > Python 3 to deserve an operator. While lists are still important, a lot
> > of things which were lists are now lazily generated iterators, and we
> > often need to concatenate them. itertools.chain() is less convenient
> > than it should be.
> >
> 
> I actually completely agree! I just wish I could think of a good character
> that doesn't have some very different meaning in other well-known contexts
> (even among iterables).

There's always && for iterator chaining.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Stephan Houben]

Hi Steven,

To put this into perspective, I did some greps on Sagemath,
being the largest Python project I have installed on this machine
(1955 .py files).

Occurrences:
enumerate: 922
zip: 585
itertools.product: 67
itertools.combinations: 18
itertools.islice: 17
chain: 14 (with or without itertools. prefix)

This seems to confirm my gut feeling that "chain" just isn't that
common an operation;
even among itertools functions, product, combinations and islice are
more common.

Based on this I would say there is little justification to even put
"chain" in builtins,
let alone to give it dedicated syntax.

I also note that * for repetition is only supported for a few
iterables (list, tuple),
incidentally the same ones which support + for sequence chaining.

Stephan




2017-06-27 12:38 GMT+02:00 Steven D'Aprano :
> On Tue, Jun 27, 2017 at 11:01:32AM +0200, Stephan Houben wrote:
>> Hi all,
>>
>> Is "itertools.chain" actually that common?
>> Sufficiently common to warrant its own syntax?
>
> I think it's much more common than (say) sequence repetition:
>
> a = [None]*5
>
> which has had an operator for a long, long time.
>
>> In my experience, "enumerate" is far more common
>> among the iterable operations.
>> And that doesn't have special syntax either.
>
> True. But enumerate is a built-in, and nearly always used in a single
> context:
>
> for i, x in enumerate(sequence):
> ...
>
> A stranger to Python could almost be forgiven for thinking that
> enumerate is part of the for-loop syntax.
>
> In contrast, chaining (while not as common as, say, numeric addition)
> happens in variable contexts: in expressions, as arguments to function
> calls, etc.
>
> It is absloutely true that this proposal brings nothing new to the
> language that cannot already be done. It's syntactic sugar. So I guess
> the value of it depends on whether or not you chain iterables enough
> that you would rather use an operator rather than a function.
>
>> A minimal  proposal would be to promote "chain" to builtins.
>
> Better than nothing, I suppose.
>
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Mon, Jun 26, 2017 at 01:23:36PM +1000, Steven D'Aprano wrote:

> The downside to this proposal is that it adds some conceptual complexity 
> to Python operators. Apart from `is` and `is not`, all Python operators 
> call one or more dunder methods. This is (as far as I know) the first 
> operator which has fall-back functionality if the dunder methods aren't 
> defined.

I remembered there is a precedent here. The == operator tries __eq__ 
before falling back on object identity, at least in Python 2.

py> getattr(object(), '__eq__')
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'object' object has no attribute '__eq__'
py> object() == object()
False


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Tue, Jun 27, 2017 at 11:01:32AM +0200, Stephan Houben wrote:
> Hi all,
> 
> Is "itertools.chain" actually that common?
> Sufficiently common to warrant its own syntax?

I think it's much more common than (say) sequence repetition:

a = [None]*5

which has had an operator for a long, long time.

> In my experience, "enumerate" is far more common
> among the iterable operations.
> And that doesn't have special syntax either.

True. But enumerate is a built-in, and nearly always used in a single 
context:

for i, x in enumerate(sequence):
...

A stranger to Python could almost be forgiven for thinking that 
enumerate is part of the for-loop syntax.

In contrast, chaining (while not as common as, say, numeric addition) 
happens in variable contexts: in expressions, as arguments to function 
calls, etc.

It is absloutely true that this proposal brings nothing new to the 
language that cannot already be done. It's syntactic sugar. So I guess 
the value of it depends on whether or not you chain iterables enough 
that you would rather use an operator rather than a function.

> A minimal  proposal would be to promote "chain" to builtins.

Better than nothing, I suppose.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Tue, Jun 27, 2017 at 08:40:23PM +1200, Greg Ewing wrote:
> David Mertz wrote:
> >I just wish I could think of a good 
> >character that doesn't have some very different meaning in other 
> >well-known contexts (even among iterables).
> 
>(a;b)
> 
> Should be unambiguous as long as the parens are required.

Except to the human reader, who can be forgiven for thinking "What the 
fudge is that semicolon doing there???" (or even less polite).

I don't know of any language that uses semi-colon as an operator.

That looks like a bug magnet to me. Consider what happens when (not if) 
you write (a,b) instead, or when you write

items = (x; y)

and it happens to succeed because x and y are iterable.

To be perfectly honest, and no offence is intended, this suggestion 
seems so wacky to me that I'm not sure if you intended for it to be 
taken seriously or not.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Stephan Houben]

Hi all,

Is "itertools.chain" actually that common?
Sufficiently common to warrant its own syntax?

In my experience, "enumerate" is far more common
among the iterable operations.
And that doesn't have special syntax either.

A minimal  proposal would be to promote "chain" to builtins.

Stephan

2017-06-27 10:40 GMT+02:00 Greg Ewing :
> David Mertz wrote:
>>
>> I just wish I could think of a good character that doesn't have some very
>> different meaning in other well-known contexts (even among iterables).
>
>
>(a;b)
>
> Should be unambiguous as long as the parens are required.
>
> --
> Greg
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Greg Ewing]

David Mertz wrote:
I just wish I could think of a good 
character that doesn't have some very different meaning in other 
well-known contexts (even among iterables).


   (a;b)

Should be unambiguous as long as the parens are required.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[David Mertz]

On Tue, Jun 27, 2017 at 12:12 AM, Steven D'Aprano 
wrote:

> > > I have a counter-proposal: introduce the iterator chaining operator
> "&":
> > > iterable & iterable --> itertools.chain(iterable, iterable)
> >
> > In [1]: import numpy as np
> > In [2]: import itertools
> > In [3]: a, b = np.array([1,2,3]), np.array([4,5,6])
> > In [4]: a & b
> > Out[4]: array([0, 0, 2])
> > In [5]: a + b
> > Out[5]: array([5, 7, 9])
> > In [6]: list(itertools.chain(a, b))
> > Out[6]: [1, 2, 3, 4, 5, 6]
> >
> > These are all distinct, useful, and well-defined behaviors.
>
> - keep the existing __and__ and __rand__ behaviour;
> - if they are not defined, and both operands x, y are iterable,
>   return chain(x, y);
>

I understand.  But it invites confusion about just what the `&` operator
will do for a given iterable.  For NumPy itself, you don't really want to
spell `chain(a, b)` so much.  But you CAN, they are iterables.  The
idiomatic way is:

>>> np.concat((a,b))
array([1, 2, 3, 4, 5, 6])

However, for "any old iterable" it feels very strange to need to inspect
the .__and__() and .__rand__ ()  methods of the things on both sides before
you know WHAT operator it is.

Maybe if you are confident a and b are exactly NumPy arrays it is obvious,
but what about:

from some_array_library import a
from other_array_library import b

What do you think `a & b` will do under your proposal? Yes, I understand
it's deterministic... but it's far from *obvious*.  This isn't even doing
something pathological like defining both `.__iter__()` and `.__and__()` on
the same class... which honestly, isn't even all that pathological; I can
imagine real-world use cases.

I think that chaining iterators is common enough and important enough in
> Python 3 to deserve an operator. While lists are still important, a lot
> of things which were lists are now lazily generated iterators, and we
> often need to concatenate them. itertools.chain() is less convenient
> than it should be.
>

I actually completely agree! I just wish I could think of a good character
that doesn't have some very different meaning in other well-known contexts
(even among iterables).

Some straw men:

both = a ⊕ b
both = a ⇢ b

Either of those look pretty nice to me, but neither is easy to enter on
most keyboards.

I think I wouldn't mind `&` if it only worked on iteraTORS.  But then it
loses many of the use cases.  I'd like this, after all:

for i in range(10)⇢[20,19,18]⇢itertools.count(100):
if i>N: break
...


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Mon, Jun 26, 2017 at 09:55:19AM -0700, David Mertz wrote:
> On Sun, Jun 25, 2017 at 8:23 PM, Steven D'Aprano 
> wrote:
> 
> > On Sun, Jun 25, 2017 at 02:06:54PM +0200, lucas via Python-ideas wrote:
> >
> > I have a counter-proposal: introduce the iterator chaining operator "&":
> >
> > iterable & iterable --> itertools.chain(iterable, iterable)
> >
> 
> In [1]: import numpy as np
> In [2]: import itertools
> In [3]: a, b = np.array([1,2,3]), np.array([4,5,6])
> In [4]: a & b
> Out[4]: array([0, 0, 2])
> In [5]: a + b
> Out[5]: array([5, 7, 9])
> In [6]: list(itertools.chain(a, b))
> Out[6]: [1, 2, 3, 4, 5, 6]
> 
> These are all distinct, useful, and well-defined behaviors.

Um... yes? I don't understand what point you are making.


Did you read all of my post? I know it was long, but if you stopped 
reading at the point you replied, you might not realise that my proposal 
keeps the existing bitwise-AND behaviour of & and so the numpy array 
behaviour won't change.

TL;DR

- keep the existing __and__ and __rand__ behaviour;

- if they are not defined, and both operands x, y are iterable, 
  return chain(x, y);

- raise TypeError for operands which neither support __(r)and__ 
  nor are iterable.


I think that chaining iterators is common enough and important enough in 
Python 3 to deserve an operator. While lists are still important, a lot 
of things which were lists are now lazily generated iterators, and we 
often need to concatenate them. itertools.chain() is less convenient 
than it should be.

If we decide that chaining deserves an operator, it shouldn't be + 
because that clashes with existing sequence addition. & has the 
advantage that it means "concatenation" in some other languages, it 
means "and" in English which can be read as "add or concatenate", and it 
is probably unsupported by most iterables.

I didn't think of numpy arrays as an exception (I was mostly thinking of 
sets) but I don't think people chain numpy arrays together very often. 
If they do, it's easy enough to call iter() first.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Wes Turner]

On Sunday, June 25, 2017, Wes Turner  wrote:

>
>
> On Sunday, June 25, 2017, Danilo J. S. Bellini  > wrote:
>
>> On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas <
>> python-ideas@python.org> wrote:
>>
>>> I often use generators, and itertools.chain on them.
>>> What about providing something like the following:
>>>
>>> a = (n for n in range(2))
>>> b = (n for n in range(2, 4))
>>> tuple(a + b)  # -> 0 1 2 3
>>
>>
>> AudioLazy does that: https://github.com/danilobellini/audiolazy
>>
>
> - http://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.concat
> and concatv
>
> - https://github.com/kachayev/fn.py#streams-and-infinite-
> sequences-declaration
>  - Stream() << obj
>

<< is __lshift__()
<<= is __ilshift__()

https://docs.python.org/2/library/operator.html

Do Stream() and __lshift__() from fn.py not solve here?


>
>
>>
>>
>> --
>> Danilo J. S. Bellini
>> ---
>> "*It is not our business to set up prohibitions, but to arrive at
>> conventions.*" (R. Carnap)
>>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Koos Zevenhoven]

On Mon, Jun 26, 2017 at 4:53 PM, Serhiy Storchaka 
wrote:

> 26.06.17 13:47, Joao S. O. Bueno пише:
>
>> On 25 June 2017 at 20:55, Danilo J. S. Bellini > > wrote:
>>
>> On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas
>> > >wrote:
>>
>> I often use generators, and itertools.chain on them.
>> What about providing something like the following:
>>
>>  a = (n for n in range(2))
>>  b = (n for n in range(2, 4))
>>  tuple(a + b)  # -> 0 1 2 3
>>
>>
>> You know you can do `tuple(*a, *b)` , right?
>>
>> The problem with the "*" notation is that it will actually render the
>> iterable
>> contents eagerly - unlike something that would just chain them.
>> But for creating tuples, it just works.
>>
>
> Even the tuple constructor is not needed.
>
> >>> *a, *b
> (0, 1, 2, 3)
>

​And you can also do

def a_and_b():
yield from a
yield from b

c = a_and_b() # iterable that yields 0, 1, 2, 3


I sometimes wish there was something like

c from:
yield from a
yield from b

​...or to get a list:

c as list from:
yield from a
yield from b

...or a sum:

c as sum from:
yield from a
yield from b

These would be great for avoiding crazy oneliner generator expressions.
They would also be equivalent to things like:

@list
@from
def c():
yield from a
yield from b

@sum
@from
def c():
yield from a
yield from b

the above, given:
def from(genfunc):
return genfunc()

Except of course `from` is a keyword and it should probably just be `call`.
​
But this still doesn't naturally extend to allow indexing and slicing, like
c[2] and c[1:3], for the case where the concatenated iterables are
Sequences.

-- Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Sven R. Kunze]

Personally, I find syntactic sugar for concating interators would come 
in handy.


The purpose of iterators and generators is performance and efficiency.  
So, lowering the bar of using them is a good idea IMO. Also hoping back 
and forth a generator/iterator-based solution and a, say, 
list-based/materialized solution would become a lot easier.



On 25.06.2017 16:04, Stephan Houben wrote:

I would like to add that for example numpy ndarrays are iterables, but
they have an __add__ with completely different semantics, namely 
element-wise ( numerical) addition.


So this proposal would conflict with existing libraries with iterable 
objects.


I don't see a conflict.



Op 25 jun. 2017 2:51 p.m. schreef "Serhiy Storchaka" 
>:


It would be weird if the addition is only supported for instances
of the generator class, but not for other iterators. Why (n for n
in range(2)) + (n for n in range(2, 4)) works, but iter(range(2))
+ iter(range(2, 4)) and iter([0, 1]) + iter((2, 3)) don't?
itertools.chain() supports arbitrary iterators. Therefore you will
need to implement the __add__ method for *all* iterators in the world.



I don't think it's necessary to start with *all* iterators in the world.

So, adding iterators and/or generators, should be possible without any 
problems. It's a start and could already help a lot if I have my 
use-cases correctly.



However itertools.chain() accepts not just *iterators*. It works
with *iterables*. Therefore you will need to implement the __add__
method also for all iterables in the world. But __add__ already is
implemented for list and tuple, and many other sequences, and your
definition conflicts with this.



As above, I don't see a conflict.


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Serhiy Storchaka]

26.06.17 13:47, Joao S. O. Bueno пише:
On 25 June 2017 at 20:55, Danilo J. S. Bellini > wrote:


On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas
>wrote:

I often use generators, and itertools.chain on them.
What about providing something like the following:

 a = (n for n in range(2))
 b = (n for n in range(2, 4))
 tuple(a + b)  # -> 0 1 2 3


You know you can do `tuple(*a, *b)` , right?

The problem with the "*" notation is that it will actually render the 
iterable

contents eagerly - unlike something that would just chain them.
But for creating tuples, it just works.


Even the tuple constructor is not needed.

>>> *a, *b
(0, 1, 2, 3)

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Joao S. O. Bueno]

On 25 June 2017 at 20:55, Danilo J. S. Bellini 
wrote:

> On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas <
> python-ideas@python.org> wrote:
>
>> I often use generators, and itertools.chain on them.
>> What about providing something like the following:
>>
>> a = (n for n in range(2))
>> b = (n for n in range(2, 4))
>> tuple(a + b)  # -> 0 1 2 3
>
>
> You know you can do `tuple(*a, *b)` , right?

The problem with the "*" notation is that it will actually render the
iterable
contents eagerly - unlike something that would just chain them.
But for creating tuples, it just works.


> AudioLazy does that: https://github.com/danilobellini/audiolazy
>
> --
> Danilo J. S. Bellini
> ---
> "*It is not our business to set up prohibitions, but to arrive at
> conventions.*" (R. Carnap)
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Steven D'Aprano]

On Sun, Jun 25, 2017 at 02:06:54PM +0200, lucas via Python-ideas wrote:

> What about providing something like the following:
> 
> a = (n for n in range(2))
> b = (n for n in range(2, 4))
> tuple(a + b)  # -> 0 1 2 3

As Serhiy points out, this is going to conflict with existing use of + 
operator for string and sequence concatenation.

I have a counter-proposal: introduce the iterator chaining operator "&":

iterable & iterable --> itertools.chain(iterable, iterable)

The reason I choose & rather than + is that & is less likely to conflict 
with any existing string/sequence types. None of the built-in or std lib 
sequences that I can think of support the & operator.

Also, & is used for (string?) concatenation in some languages, such as 
VB.Net, some BASIC dialects, Hypertalk, AppleScript, and Ada. Iterator 
chaining is more like concatenation than (numeric) addition.

However, the & operator is already used for bitwise-AND. Under my 
proposal that behaviour will continue, and will take priority over 
chaining. Currently, the & operator does something similar to (but 
significantly more complex) to this:


# simplified pseudo-code of existing behaviour
if hasattr(x, '__and__'):
return x.__and__(y)
elif hasattr(y, '__rand__'):
return y.__rand__(x)
else:
raise TypeError


The key is to insert the new behaviour after the existing __(r)and__ 
code, just before TypeError is raised:


attempt existing __(r)and__ behaviour
if and only if that fails to apply:
return itertools.chain(iter(x), iter(y))


So classes that define a __(r)and__ method will keep their existing 
behaviour.

This implies that we cannot use & to chain sets and frozen sets, since 
they already define __(r)and__. This has an easy work-around: just call 
iter() on the set first.

Applying & to objects which don't define __(r)and__ and aren't iterable 
will continue to raise TypeError, just as it does now. The only 
backwards incompatibility this proposal introduces is for any code which 
relies on `iterable & iterable` to raise TypeError. Frankly I can't 
imagine that there is any such code, outside of the Python test suite, 
but if there is, and people think it is worth it, we could make this a 
__future__ import. But I think that's overkill.

The downside to this proposal is that it adds some conceptual complexity 
to Python operators. Apart from `is` and `is not`, all Python operators 
call one or more dunder methods. This is (as far as I know) the first 
operator which has fall-back functionality if the dunder methods aren't 
defined.

Up to now, I've talked about & chaining being equivalent to the 
itertools.chain function. That glosses over one difference which 
needs to be mentioned. The chain function currently doesn't attempt 
to iterate over its arguments until needed:


py> x = itertools.chain("a", 1, "c")
py> next(x)
'a'
py> next(x)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'int' object is not iterable


Any proposal to change this behaviour for the itertools.chain 
function should be kept separate from this one.

But for the & chaining operator, I think that behaviour must change: if 
we have an operand that is neither iterable nor defines __(r)and__, the 
& operator should fail early:

[1, 2, 3] & None

should raise TypeError immediately, unlike itertools.chain().



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Wes Turner]

On Sunday, June 25, 2017, Danilo J. S. Bellini 
wrote:

> On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas <
> python-ideas@python.org
> > wrote:
>
>> I often use generators, and itertools.chain on them.
>> What about providing something like the following:
>>
>> a = (n for n in range(2))
>> b = (n for n in range(2, 4))
>> tuple(a + b)  # -> 0 1 2 3
>
>
> AudioLazy does that: https://github.com/danilobellini/audiolazy
>

- http://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.concat and
concatv

-
https://github.com/kachayev/fn.py#streams-and-infinite-sequences-declaration

 - Stream() << obj


>
>
> --
> Danilo J. S. Bellini
> ---
> "*It is not our business to set up prohibitions, but to arrive at
> conventions.*" (R. Carnap)
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Danilo J. S. Bellini]

On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas <
python-ideas@python.org> wrote:

> I often use generators, and itertools.chain on them.
> What about providing something like the following:
>
> a = (n for n in range(2))
> b = (n for n in range(2, 4))
> tuple(a + b)  # -> 0 1 2 3


AudioLazy does that: https://github.com/danilobellini/audiolazy

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Koos Zevenhoven]

On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas <
python-ideas@python.org> wrote:

> I often use generators, and itertools.chain on them.
> What about providing something like the following:
>
> a = (n for n in range(2))
> b = (n for n in range(2, 4))
> tuple(a + b)  # -> 0 1 2 3
>
> This, from user point of view, is just as how the
> __add__ operator works on lists and tuples.
> Making generators works the same way could be a great way to avoid calls
> to itertools.chain everywhere, and to limits the differences between
> generators and other "linear" collections.
>
>
​I think a convenient syntax for chaining iterables and sequences would be
very usef​ul in Python 3, because there has been a shift from using lists
by default to using views to dict keys and values, range objects etc.
Having to add an import for a basic operation that used to just work with
the + operator feels like a regression to many.

It's not really clear if you will be able to implement this, but if you can
find a syntax that gets accepted, I think using the same type as
itertools.chain might be a good starting point, although the docs should
not promise to return that exact type so that support for __getitem__ etc.
could be added in the future for cases where the chained iterables are
Sequences.

-- Koos



> I do not know exactly how to implement that (i'm not that good at C, nor
> CPython source itself), but by seeing the sources,
> i imagine that i could do something like the list_concat function at
> Objects/listobject.c:473, but in the Objects/genobject.c file,
> where instead of copying elements i'm creating and initializing a new
> chainobject as described at Modules/itertoolsmodule.c:1792.
>
> (In pure python, the implementation would be something like `def
> __add__(self, othr): return itertools.chain(self, othr)`)
>
> Best regards,
> --lucas
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Stephan Houben]

I would like to add that for example numpy ndarrays are iterables, but
they have an __add__ with completely different semantics, namely
element-wise ( numerical) addition.

So this proposal would conflict with existing libraries with iterable
objects.

Stephan

Op 25 jun. 2017 2:51 p.m. schreef "Serhiy Storchaka" :

> 25.06.17 15:06, lucas via Python-ideas пише:
>
>> I often use generators, and itertools.chain on them.
>> What about providing something like the following:
>>
>>  a = (n for n in range(2))
>>  b = (n for n in range(2, 4))
>>  tuple(a + b)  # -> 0 1 2 3
>>
>> This, from user point of view, is just as how the
>> __add__ operator works on lists and tuples.
>> Making generators works the same way could be a great way to avoid calls
>> to itertools.chain everywhere, and to limits the differences between
>> generators and other "linear" collections.
>>
>> I do not know exactly how to implement that (i'm not that good at C, nor
>> CPython source itself), but by seeing the sources,
>> i imagine that i could do something like the list_concat function at
>> Objects/listobject.c:473, but in the Objects/genobject.c file,
>> where instead of copying elements i'm creating and initializing a new
>> chainobject as described at Modules/itertoolsmodule.c:1792.
>>
>> (In pure python, the implementation would be something like `def
>> __add__(self, othr): return itertools.chain(self, othr)`)
>>
>
> It would be weird if the addition is only supported for instances of the
> generator class, but not for other iterators. Why (n for n in range(2)) +
> (n for n in range(2, 4)) works, but iter(range(2)) + iter(range(2, 4)) and
> iter([0, 1]) + iter((2, 3)) don't? itertools.chain() supports arbitrary
> iterators. Therefore you will need to implement the __add__ method for
> *all* iterators in the world.
>
> However itertools.chain() accepts not just *iterators*. It works with
> *iterables*. Therefore you will need to implement the __add__ method also
> for all iterables in the world. But __add__ already is implemented for list
> and tuple, and many other sequences, and your definition conflicts with
> this.
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] + operator on generators

[Serhiy Storchaka]

25.06.17 15:06, lucas via Python-ideas пише:

I often use generators, and itertools.chain on them.
What about providing something like the following:

 a = (n for n in range(2))
 b = (n for n in range(2, 4))
 tuple(a + b)  # -> 0 1 2 3

This, from user point of view, is just as how the
__add__ operator works on lists and tuples.
Making generators works the same way could be a great way to avoid calls
to itertools.chain everywhere, and to limits the differences between
generators and other "linear" collections.

I do not know exactly how to implement that (i'm not that good at C, nor
CPython source itself), but by seeing the sources,
i imagine that i could do something like the list_concat function at
Objects/listobject.c:473, but in the Objects/genobject.c file,
where instead of copying elements i'm creating and initializing a new
chainobject as described at Modules/itertoolsmodule.c:1792.

(In pure python, the implementation would be something like `def
__add__(self, othr): return itertools.chain(self, othr)`)


It would be weird if the addition is only supported for instances of the 
generator class, but not for other iterators. Why (n for n in range(2)) 
+ (n for n in range(2, 4)) works, but iter(range(2)) + iter(range(2, 4)) 
and iter([0, 1]) + iter((2, 3)) don't? itertools.chain() supports 
arbitrary iterators. Therefore you will need to implement the __add__ 
method for *all* iterators in the world.


However itertools.chain() accepts not just *iterators*. It works with 
*iterables*. Therefore you will need to implement the __add__ method 
also for all iterables in the world. But __add__ already is implemented 
for list and tuple, and many other sequences, and your definition 
conflicts with this.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/