Re: Python Iterables struggling using map() built-in
On Wed, Dec 10, 2014 at 9:01 PM, Steven D'Aprano st...@pearwood.info wrote: On Wed, 10 Dec 2014 09:46:55 -0700, Ian Kelly wrote: I don't particularly have a problem with functions having attributes, e.g. I think itertools.chain.from_iterable is just peachy. There is a downside though, which is that making those functions attributes of another function rather than of the module defeats the dir() function for that module. I think you are missing the point of namespaces :-) When I call dir(os), I see os.path, but I don't see the names in os.path. That is working as designed. I don't think I'm missing the point. os.path is a module. That makes it well suited for a namespace, because that's what modules are designed and expected to provide: the implementation of namespaces. A function, on the other hand, is not well suited to be a namespace, because it's not expected to provide one. When I see a new function, I don't think I should check what attributes it has. Instead, I wonder about what parameters it takes and what it returns. I expect to be able to open up my web browser and go to the library documentation for the os.path module. I don't expect to find a separate page for statistics.median, because the docs are organized around modules, not functions. Would the median function itself be documented in the statistics page, in the statistics.median page, or redundantly in both? I expect to be able to import modules, and by extension, namespaces. import os.path works. import statistics.median.low would return an error, as would from statistics.median import low as median, if that were the only function I wanted. In the latter case I would have to do something like from statics import median; median = median.low. That's not just ugly; it also demonstrates how the programmer needs to remember and incorporate the detail that this namespace does not work like other namespaces. As the designer of the module I thought that median_low and median_high were important enough to include but not important enough to stick in the main module namespace. To me this sounds backward. Namespaces are a great idea, but at the same time they also create cognitive burden. One creates a namespace because the contents are important enough to be referenced as a group, not because they aren't as important as the other things they're grouped with. The reason the collections ABCs were moved into collections.abc wasn't because they weren't important enough to be in the top-level collections module. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Ian Kelly wrote:
A function, on the
other hand, is not well suited to be a namespace, because it's not
expected to provide one.
And that is exactly the point I am making about the inherent
conservativeness of Python developers.
Functions ARE namespaces, like instances of user-defined classes they have a
__dict__ and can carry per-instance state. (If you think that only modules
are namespaces, you are badly mistaken. Instances are namespaces too.) This
is *not* an accident of implementation, it is a deliberate design choice.
Contrast functions with builtin functions:
py def f(): pass
...
py vars(f)
{}
py vars(len)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: vars() argument must have __dict__ attribute
Although functions are written in pure Python, their type is implemented in
C and could easily have been designed to be __dict__-less just like the
builtin_function_or_method type. But they weren't. It is a language feature
that functions are namespaces.
So why don't we use functions as namespaces?
We don't use functions as namespaces because nobody thinks of them as
namespaces, and we don't think of them as namespaces because nobody uses
them as namespaces -- and when somebody tries to break out of that vicious
circle, the conservativeness kicks in and the very idea is rejected.
I think that there is a legitimate debate to be had as to whether this
conservativeness and resistance to change is a good thing or a bad thing,
but I don't think that there should be any debate about the reality of the
Python community being strongly conservative, compared to the anything
goes attitude of (say) Ruby, Perl and Lisp programmers. We don't like code
that does anything we haven't seen a hundred times before.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Thu, Dec 11, 2014 at 4:28 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Ian Kelly wrote: A function, on the other hand, is not well suited to be a namespace, because it's not expected to provide one. And that is exactly the point I am making about the inherent conservativeness of Python developers. Functions ARE namespaces, like instances of user-defined classes they have a __dict__ and can carry per-instance state. (If you think that only modules are namespaces, you are badly mistaken. Instances are namespaces too.) I never said that functions can't be used as namespaces. I said that functions are *bad* namespaces, and I gave reasons why I think this is true. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Fri, Dec 12, 2014 at 10:28 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I think that there is a legitimate debate to be had as to whether this conservativeness and resistance to change is a good thing or a bad thing, but I don't think that there should be any debate about the reality of the Python community being strongly conservative, compared to the anything goes attitude of (say) Ruby, Perl and Lisp programmers. We don't like code that does anything we haven't seen a hundred times before. I agree that Python is a lot more conservative than many languages, and personally, I think it's good. When it comes to monkeypatching, less is *definitely* better - less places where you have to figure out what import changed the meanings of your built-ins, and so on. (Case in point: A logger replacement tampered with the log.info()/log.error() family of functions, adding a keyword parameter. Took me a fair amount of digging to find out exactly where that happened, and it made REPL importing of the module's functions rather awkward.) The same is true of obscure namespaces, specifically because they're obscure; if I do a JSON decode of something, I'm not going to check to see if numbers came through as a subclass of float with a .warnings attribute to tell me about loss of numeric precision, even though that would be perfectly possible. Since almost nobody does this, nobody will expect it to be done, ergo nobody will go looking for it, and the feature may as well not even have been implemented if it's never seen. Popularity *is* sometimes a legitimate argument. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
In article mailman.16880.1418342293.18130.python-l...@python.org,
Ian Kelly ian.g.ke...@gmail.com wrote:
I never said that functions can't be used as namespaces. I said that
functions are *bad* namespaces, and I gave reasons why I think this is true.
An excellent example of functions acting as namespaces is nosetest's
@attr() decorator. We use this, for example, to tag certain test cases
as being reliant on facebook being up(*):
@attr('facebook', 'services')
def test_some_facebook_thing():
# whatever
this lets us turn all those tests on or off with a single switch. The
way @attr() is implemented, it sets attributes on the decorated
function. It's the most logical and obvious place to store a piece of
information about a test case -- right on the test case itself.
Similarly, we've got tests that we annotate as being dependent on itunes
being reachable, depending on certain data existing in the database(**),
being too slow to want to run all the time(**), etc.
(*) and please don't tell me that tests shouldn't depend on external
services.
(**) see first footnote
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Thu, Dec 11, 2014 at 6:55 PM, Roy Smith r...@panix.com wrote:
In article mailman.16880.1418342293.18130.python-l...@python.org,
Ian Kelly ian.g.ke...@gmail.com wrote:
I never said that functions can't be used as namespaces. I said that
functions are *bad* namespaces, and I gave reasons why I think this is
true.
An excellent example of functions acting as namespaces is nosetest's
@attr() decorator. We use this, for example, to tag certain test cases
as being reliant on facebook being up(*):
@attr('facebook', 'services')
def test_some_facebook_thing():
# whatever
this lets us turn all those tests on or off with a single switch. The
way @attr() is implemented, it sets attributes on the decorated
function. It's the most logical and obvious place to store a piece of
information about a test case -- right on the test case itself.
I agree this is a great use of function attributes, but this is an example
of tags or annotations, not namespaces. The purpose of namespaces is to
distinguish between different entities of the same name, e.g. 'pow' and
'math.pow' are two distinct functions that both happen to be named pow. We
can tell them apart because they're in separate namespaces.
In the nosetest example, the 'facebook' attribute always means the same
thing no matter which functions you apply it to. Conceptually speaking,
each instance of the attribute is the same annotation. The expression
test_some_facebook_thing.facebook doesn't perform the role of a namespace
because it's used to signify the presence of that annotation, not to
identify something.
A similar example from the standard library is the functools.lru_cache
decorator, which adds a cache_info function to the wrapped function.
However, the cache_info function always performs the same task in relation
to the particular function it decorates, so in this regard it acts more
like an object method than like a name in a namespace.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Tue, 09 Dec 2014 21:44:54 -0500, Roy Smith wrote: In article 54878f8a$0$13010$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I really think you guys are trying too hard to make this function seem more complicated than it is. If you find it so hard to understand a simple function with four short lines, one wonders how you would possibly cope with real code. Well, look at it this way. You've got several folks here (I count Terry, Ned, Chris, and myself) all saying, We find this confusing, and you're saying, nobody should find this confusing. I suppose one possible explanation for this dichotomy is that we're all incapable of dealing with real code. Yeah, that must be it. I know you guys deal with *much* more complicated, obfuscated and difficult code. Just look at the stuff posted here by newbies :-) But you insist that you're confused by while iters. I don't get it. I think you're protesting too much -- like the weightlifter who routinely bench- presses 400 lbs, but then insists he can't take the trash out because it's too heavy. Or you're insisting on an unrealistic ideal of readable, that if it takes *any time at all* to think about the code, that makes it bad. I've noticed this deep-seated conservatism in Python programmers before. Parts of the language are deeply under-utilised, because there are simple idioms that people refuse to use because they're confusing even though they are a trivial generalisation of things that we use all the time. Or at least, they should be common idioms. They're not common idioms because nobody uses them, and nobody uses them because they're not common idioms. Example: In the statistics module in Python 3.4, I added a `median` function to calculate the median by the traditional schoolbook algorithm. But that's only one out of a number of ways to calculate medium, and inspired by similar syntax from R I proposed adding additional methods to that `median` object: median.low(alist) median.high(alist) etc. We all know that functions are first class objects with attributes and methods, they have a __dict__ so you can attach per-function data to them. And there's the last line of the Zen, about namespaces being awesome. Every instance with a __dict__ is a namespace. This ought to be a no-brainer, but apparently nobody has any experience with callable attributes attached to instances: math.sin(x) some string.split(s) We make it a point of pride that functions in Python are not just first class values but that they are objects with attributes and methods and a per-instance __dict__. But when somebody proposes making use of that, it is rejected. Not that I'm bitter :-) But I digress. I'm truly sorry that I have been unable to express in words just how elegant and beautiful the short version of the myzip() function is. It is vanishingly rare to have such a perfect match between an algorithm and real working code. And if anyone has got the impression that I'm calling you a dummy because you don't see it my way, I'm not. I'm calling you nekulturny and somebody who can't recognise elegant code when it's staring you right in the face :-P -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Wed, Dec 10, 2014 at 8:24 PM, Steven D'Aprano st...@pearwood.info wrote: And if anyone has got the impression that I'm calling you a dummy because you don't see it my way, I'm not. I'm calling you nekulturny and somebody who can't recognise elegant code when it's staring you right in the face :-P I love it when a post sends me to a dictionary. Though in this case, the word actually has Russian origin, so it ought perhaps to be spelled некультурны. Good word, anyhow. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Steven D'Aprano st...@pearwood.info: I've noticed this deep-seated conservatism in Python programmers before. Parts of the language are deeply under-utilised, because there are simple idioms that people refuse to use because they're confusing even though they are a trivial generalisation of things that we use all the time. IMO, large parts of the language are deservedly under-utilized. This whole dunder thicket: __complex__(), __ifloordiv__(), __rxor__() etc. Or this metaclass attribute. Or decorators. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Wed, Dec 10, 2014 at 2:24 AM, Steven D'Aprano st...@pearwood.info wrote: Example: In the statistics module in Python 3.4, I added a `median` function to calculate the median by the traditional schoolbook algorithm. But that's only one out of a number of ways to calculate medium, and inspired by similar syntax from R I proposed adding additional methods to that `median` object: median.low(alist) median.high(alist) etc. We all know that functions are first class objects with attributes and methods, they have a __dict__ so you can attach per-function data to them. And there's the last line of the Zen, about namespaces being awesome. Every instance with a __dict__ is a namespace. This ought to be a no-brainer, but apparently nobody has any experience with callable attributes attached to instances: math.sin(x) some string.split(s) We make it a point of pride that functions in Python are not just first class values but that they are objects with attributes and methods and a per-instance __dict__. But when somebody proposes making use of that, it is rejected. Not that I'm bitter :-) I don't particularly have a problem with functions having attributes, e.g. I think itertools.chain.from_iterable is just peachy. There is a downside though, which is that making those functions attributes of another function rather than of the module defeats the dir() function for that module. Likewise the generated help for the help() function, unless care is taken to explicitly mention the existence of those functions in either the doc string for the module or the standard median function. Thirdly, IDEs with code completion features may simply fail to notice that these alternate versions of the function exist. For these reasons I think that this is an anti-pattern and is best avoided in the standard library. Having to type _ instead of . is only one extra key stroke. Now if there were an established pattern to such function attributes, e.g. if f.len(parrots) was a standard alternative to len(f(parrots)) for functions that return sequences, then that could be a useful feature, especially if such properties could be chained arbitrarily. But one-offs like median.low would only serve to make the function more difficult to find than necessary. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 12/10/2014 11:46 AM, Ian Kelly wrote: I don't particularly have a problem with functions having attributes, e.g. I think itertools.chain.from_iterable is just peachy. There is a downside though, which is that making those functions attributes of another function rather than of the module Itertools.chain is a class, not a function. Chain.from_iterable is an *alternate constructor* for instances of the class, just as dict.fromkeys is an alternate constructor for dict instances. It is true that if itertools were written in python, chain and the other classes would likely be written as generator functions, so tha analogy would not apply. defeats the dir() function for that module. And it is also true that dir(__builtins__) does not show dict.fromkeys. Likewise the generated help for the help() function, unless care is taken to explicitly mention the existence of those functions in either the doc string for the module help(it.chain) lists | from_iterable(...) from builtins.type | chain.from_iterable(iterable) -- chain object | | Alternate chain() contructor taking a single iterable argument | that evaluates lazily. help(dict) | fromkeys(iterable, value=None, /) from builtins.type | Returns a new dict with keys from iterable and values equal to value. Thirdly, IDEs with code completion features may simply fail to notice that these alternate versions of the function exist. I don't know what you mean here. In Idle it.chain. -- completion box with from_iterable dict. -- completion box that includes fromkeys None of the above is to say that alternate constructors for classed are not a bit awkward. I believe this was recognized when dict.fromkeys was added. But it was considered less awkward than the alternatives: 1. do not add the functionality; 2. further overload the main (__init__) constructor; 3. add a independent function. Anyway, they exist, are an accepted pattern among core devs, and will probably increase in number. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Wed, Dec 10, 2014 at 10:48 AM, Terry Reedy tjre...@udel.edu wrote: Likewise the generated help for the help() function, unless care is taken to explicitly mention the existence of those functions in either the doc string for the module help(it.chain) lists | from_iterable(...) from builtins.type | chain.from_iterable(iterable) -- chain object | | Alternate chain() contructor taking a single iterable argument | that evaluates lazily. As you point out, chain is a class, not a function. When you invoke help() on a function, it lists the members, of which a static alternate constructor would be one. Try it with a function, though: import statistics statistics.median.low = statistics.median_low help(statistics.median) Help on function median in module statistics: median(data) Return the median (middle value) of numeric data. When the number of data points is odd, return the middle data point. When the number of data points is even, the median is interpolated by taking the average of the two middle values: median([1, 3, 5]) 3 median([1, 3, 5, 7]) 4.0 It only prints out the doc string, nothing about the attributes. So the doc string for median would need to explicitly document the existence of median.low and median.high in addition to its own usage. Thirdly, IDEs with code completion features may simply fail to notice that these alternate versions of the function exist. I don't know what you mean here. In Idle it.chain. -- completion box with from_iterable dict. -- completion box that includes fromkeys So Idle gets it right. At least for static methods of classes, which isn't very surprising. Does it complete a function attribute of a function? I don't have it installed to test. Having used Komodo IDE for a number of years and been occasionally frustrated by its code completion, it would not surprise me in the least if it failed to pull attributes of functions into its completion database. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Wed, 10 Dec 2014 09:46:55 -0700, Ian Kelly wrote: I don't particularly have a problem with functions having attributes, e.g. I think itertools.chain.from_iterable is just peachy. There is a downside though, which is that making those functions attributes of another function rather than of the module defeats the dir() function for that module. I think you are missing the point of namespaces :-) When I call dir(os), I see os.path, but I don't see the names in os.path. That is working as designed. Likewise the generated help for the help() function, unless care is taken to explicitly mention the existence of those functions in either the doc string for the module or the standard median function. Yes. I don't have a problem with saying See also ``median.low`` in the docstring for median. I would also have raised a feature request (and/or patch) for having help() automatically display functions and methods attached to a function, much as it already does for classes. As the designer of the module I thought that median_low and median_high were important enough to include but not important enough to stick in the main module namespace. I wanted to delegate them to secondary status by putting them in a separate lightweight namespace, without going all the way to making statistics be a package. I even considered doing the same for the population variance and standard deviation, but decided against it as most people are familiar with them from scientific calculators and school. Python gives us the functionality to include multiple namespaces in a single module. Despite the Zen of Python, hardly anyone makes use of the fact that functions are namespaces. I wish to rail against that :-) Thirdly, IDEs with code completion features may simply fail to notice that these alternate versions of the function exist. Hmmm, that would be a bad bug in the IDE then. If I type foo. I would expect the IDE to complete on whatever attributes foo has, regardless of whether it is a function or some other object. I have tab completion enabled in Python 2.7, and it works for me: py def spam(): pass ... py spam. Display all 31 possibilities? (y or n) spam.__call__( spam.__globals__spam.__str__( spam.__class__( spam.__hash__( spam.__subclasshook__( spam.__closure__spam.__init__( spam.func_closure spam.__code__ spam.__module__ spam.func_code spam.__defaults__ spam.__name__ spam.func_defaults spam.__delattr__( spam.__new__( spam.func_dict spam.__dict__ spam.__reduce__(spam.func_doc spam.__doc__spam.__reduce_ex__( spam.func_globals spam.__format__(spam.__repr__( spam.func_name spam.__get__( spam.__setattr__( spam.__getattribute__( spam.__sizeof__( I expect that most IDEs will Just Work in this case. [...] Now if there were an established pattern to such function attributes, But there's the rub. That's the conservatism I'm referring to. Somebody has to be the first to do such a thing, and if we reject it because nobody does it, nobody will do it because when they try they're always rejected. We know namespaces are great and useful. We use namespaces in the form of modules, packages and classes, and the Zen says we should use more of them. But we don't, because we're trapped in a vicious circle where attempts to use more namespaces get dismissed because nobody uses them, and nobody uses them because when they try it gets dismissed :-( Oh well, there's always my own non-stdlib modules :-) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 12/10/2014 3:32 PM, Ian Kelly wrote: So Idle gets it right. At least for static methods of classes, which isn't very surprising. Does it complete a function attribute of a function? def f(): pass f.a='attr' f. box with 'a' as possible completion. Having used Komodo IDE for a number of years and been occasionally frustrated by its code completion, it would not surprise me in the least if it failed to pull attributes of functions into its completion database. I do not think Idle has a static database. But I have not looked at the completion code yet to see exactly how it works (and how it might be improved). -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 12/9/2014 12:03 AM, Terry Reedy wrote: Roy Smith wrote: Chris Angelico wrote: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. What is nasty to me is that to understand the loop, one must do a whole program analysis to determine both that 'iters' is not rebound and that the list it is bound to is not mutated. To do the later, one must not only read the loop body, but also preceding code to make sure the list is not aliased. iters is empty if and only if args is empty. If args is empty, iters should not be created. if args: iters = ... while True ... (return on exception) makes the logic clear. Once the logic is clear and 'localized', even a simple compiler like CPython's can see that this is a loop-forever construct and that the loop test is unnecessary. So it can be removed. dis(while a: b+=1) 1 0 SETUP_LOOP 20 (to 23) 3 LOAD_NAME0 (a) 6 POP_JUMP_IF_FALSE 22 9 LOAD_NAME1 (b) 12 LOAD_CONST 0 (1) 15 INPLACE_ADD 16 STORE_NAME 1 (b) 19 JUMP_ABSOLUTE3 22 POP_BLOCK 23 LOAD_CONST 1 (None) 26 RETURN_VALUE dis(while True: b+=1) 1 0 SETUP_LOOP 13 (to 16) 3 LOAD_NAME0 (b) 6 LOAD_CONST 0 (1) 9 INPLACE_ADD 10 STORE_NAME 0 (b) 13 JUMP_ABSOLUTE3 16 LOAD_CONST 1 (None) 19 RETURN_VALUE 'while 1' and 'while exception not raised' are similarly condensed. This leaves only the initial test of the argument. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Terry Reedy wrote: On 12/9/2014 12:03 AM, Terry Reedy wrote: Roy Smith wrote: Chris Angelico wrote: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. What is nasty to me is that to understand the loop, one must do a whole program analysis to determine both that 'iters' is not rebound and that the list it is bound to is not mutated. When people say whole program analysis, they usually mean the entire application including all its modules and libraries, not a four line function, excluding the def header. (Or three lines if you get rid of the unnecessary temporary variable 'res'.) Did it take you a long time to read all two lines of the while loop to determine that iters is not modified or rebound? I really think you guys are trying too hard to make this function seem more complicated than it is. If you find it so hard to understand a simple function with four short lines, one wonders how you would possibly cope with real code. Purely by coincidence, I have the source to the pyclbr module from the standard library open in a text editor. I see a _readmodule function that looks, in part, like this: try: for tokentype, token, start, _end, _line in g: if ... while ... elif ... while.. if ... if ... if ... else ... elif ... while ... if ... if ... while ... if ... if ... else ... if ... if ... if ... if ... elif ... if ... elif ... elif ... at which point I'm about halfway through the try block and I'm giving up. https://hg.python.org/cpython/file/3.4/Lib/pyclbr.py This, presumably, is good enough for the standard library, but the four line version of zip is supposed to be too hard for mortal man to comprehend. That's funny :-) To do the later, one must not only read the loop body, but also preceding code to make sure the list is not aliased. The preceding code is exactly *one* line, a single assignment binding the name iters to the list. The while loop body is exactly two lines, one if you dump the unnecessary 'res' temporary variable: yield tuple([next(i) for i in iters]) Quite frankly Terry, I do not believe for a second that somebody like you who can successfully maintain IDLE is struggling to understand this myzip() function. Wait... is this like the Four Yorkshire Men sketch from Monty Python, only instead of complaining about how hard you had it as children, you're all trying to outdo each other about how difficult you find it to read this function? If so, well done, you really had me for a while. Once the logic is clear and 'localized', even a simple compiler like CPython's can see that this is a loop-forever construct and that the loop test is unnecessary. So it can be removed. Ah, now that's nice. You're suggesting that by moving the loop condition outside of the while statement, the compiler can generate more efficient byte code. It only needs to test iters once, not at the start of every loop. That is the first interesting argument I've seen so far! On the one hand, as micro-optimizations go, it will be pretty micro. Particularly compared to the cost of starting and stopping a generator, I doubt that will save any meaningful time, at least not enough to make up for the extra effort in having to read and comprehend one more line of code. On the other hand, *premature optimization*. In general, one shouldn't write more complex code so the compiler can optimize it, one should write simpler code and have a smarter compiler. If *we* are capable of recognising that iters is not modified in the body of the loop, then the compiler should be capable of it too. (If it isn't, it is because nobody has bothered to give the compiler sufficient smarts, not because it can't be done.) So a good compiler should be able to compile while iters into if iters: while True so long as iters is not modified in the body of the loop. Still, that's a nice observation: sometimes more complex source code can lead to simpler byte code. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Wed, Dec 10, 2014 at 11:10 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On the other hand, *premature optimization*. In general, one shouldn't write more complex code so the compiler can optimize it, one should write simpler code and have a smarter compiler. If *we* are capable of recognising that iters is not modified in the body of the loop, then the compiler should be capable of it too. (If it isn't, it is because nobody has bothered to give the compiler sufficient smarts, not because it can't be done.) So a good compiler should be able to compile while iters into if iters: while True so long as iters is not modified in the body of the loop. In general, one can't expect the boolification of a Python object to be consistent, so the compiler can't optimize this. How can it be sure the list will never become empty? I'm still of the opinion that a while loop's header implies something about the code; while iters: implies that iters might be able to become false. Sure, you can verify easily enough that it never will... but why should you have to verify at all? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 10/12/2014 00:10, Steven D'Aprano wrote: Wait... is this like the Four Yorkshire Men sketch from Monty Python, only instead of complaining about how hard you had it as children, you're all trying to outdo each other about how difficult you find it to read this function? If so, well done, you really had me for a while. The Four Yorkshiremen was not actually a Monty Python sketch. Get it right, lad :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
In article 54878f8a$0$13010$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I really think you guys are trying too hard to make this function seem more complicated than it is. If you find it so hard to understand a simple function with four short lines, one wonders how you would possibly cope with real code. Well, look at it this way. You've got several folks here (I count Terry, Ned, Chris, and myself) all saying, We find this confusing, and you're saying, nobody should find this confusing. I suppose one possible explanation for this dichotomy is that we're all incapable of dealing with real code. Yeah, that must be it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, 08 Dec 2014 11:35:36 +1100, Chris Angelico wrote: On Mon, Dec 8, 2014 at 11:27 AM, Roy Smith r...@panix.com wrote: Although, to be honest, I'm wondering if this is more straight-forward (also not tested): def myzip37(*args): if not args: return iters = list(map(iter, args)) Yes, I prefer this too. It's explicit and clear that passing no arguments will yield no values. The first version is explicit and clear too. I'm sorry to say this, but it is true: if you (generic you) don't recognise that while iters: ... skips the while block if iters is an empty list, then *you* have a problem, not the code. You're simply not fluent with the language. (Or you have a strange blind-spot, in which case you have my sympathy but that's your problem to deal with, not mine.) This is not an obscure corner of some rarely-used library that has a strange and weird API, it is a fundamental part of Python. I *guarantee* that there are people who will bitch and moan about your example too, and insist that the only right way to write it is to be explicit that an empty tuple is falsey: if not args == (): return and there will be some who are convinced that operator precedence is too implicit: if not (args == ()): # args != () is okay too return Dumbing down code is an anti-pattern, because there's always somebody just a little less fluent in the language who will complain that it isn't dumbed down enough. Sometimes we have this meme that if you can't take in code at a glance and instantly understand it, it's bad code. But that meme is untrue, and we all know that it is untrue: real world code is often hard to understand because it can't be any simpler. Try understanding the code for a web server, or code that calculates the square root of a number, or code for importing a module. Occasionally, very occasionally, we have a combination of programming language and algorithm which combines in such a way that you can actually implement a useful algorithm in a simple, elegant, minimalist way. As programmers, we all know how fecking rare this is: we start with an elegant five line function, and by the time we cover all the corners and fix the bugs it's twenty lines and you can't tell what it does any more without studying it for ten minutes. zip() is an exception, you can write zip() in Python beautifully. It breaks my heart that there are people who think that it is improved by adding unnecessary guard clauses to it. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 8:40 PM, Steven D'Aprano st...@pearwood.info wrote: The first version is explicit and clear too. I'm sorry to say this, but it is true: if you (generic you) don't recognise that while iters: ... skips the while block if iters is an empty list, then *you* have a problem, not the code. Of course it skips the body if iters starts out empty. The argument is whether or not it makes sense to use this to mean if iters: while True: because iters will never be changed. Is it abusing syntax or a valid way to spell that condition? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
In article 5485721c$0$2817$c3e8da3$76491...@news.astraweb.com, Steven D'Aprano st...@pearwood.info wrote: On Mon, 08 Dec 2014 11:35:36 +1100, Chris Angelico wrote: On Mon, Dec 8, 2014 at 11:27 AM, Roy Smith r...@panix.com wrote: Although, to be honest, I'm wondering if this is more straight-forward (also not tested): def myzip37(*args): if not args: return iters = list(map(iter, args)) Yes, I prefer this too. It's explicit and clear that passing no arguments will yield no values. The first version is explicit and clear too. I'm sorry to say this, but it is true: if you (generic you) don't recognise that while iters: ... skips the while block if iters is an empty list, then *you* have a problem, not the code. The problem is not that the while body is skipped if iters is falsey. Thats quite clear. The problem is that the looping termination isn't actually controlled by the control variable being exhausted. It's controlled by an exception getting thrown. I'm OK with breaking out of a loop by throwing an exception, but if that's what you're going to do, then make it clear that's the case by doing while 1 or while True. Those screams out, Hey, look at me, I'm an infinite loop, which is your clue that there's something else going on. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Roy Smith wrote:
Chris Angelico wrote:
I'm actually glad PEP 479 will break this kind of code. Gives a good
excuse for rewriting it to be more readable.
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote:
What kind of code is that? Short, simple, Pythonic and elegant? :-)
Here's the code again, with indentation fixed:
def myzip(*args):
iters = map(iter, args)
while iters:
res = [next(i) for i in iters]
yield tuple(res)
Ugh. When I see while foo, my brain says, OK, you're about to see a
loop which is controlled by the value of foo being changed inside the
loop.
Yes. Me too. 99% of the time when you see while foo, that's what you'll
get, so it's the safe assumption. But it's only an assumption, not a
requirement. When you read a bit more of the code and see that iters isn't
being modified, your reaction ought to be closer oh wow, that's neat
than oh noes it's different from what I expected.
while foo is logically equivalent to if foo: while foo:. The if is
completely redundant.
That's not at all what's happening here, so my brain runs into a
wall.
I hope you are exaggerating for effect, because if you genuinely mean that
reading that code causes major mental trauma (perhaps the equivalent of a
mental BSOD) then you've probably picked the wrong industry to be working
in. Imagine how you would cope reading genuinely obfuscated code. You would
probably have a nervous breakdown :-)
It's okay to read code which forces you to reevaluate your initial
assumption about the code. People, especially (allegedly) smart people like
programmers, are intelligent and flexible. If you can't do that, you're
going to hate Python:
- Python has no repeat N times loop, we have to use for i in range(...)
instead, so seeing a for-loop doesn't necessarily mean that the loop
variable will be used. Sometimes it isn't.
- I cannot count the number of times I've read, or written, a method that
doesn't use self, but doesn't bother to declare it as a staticmethod.
- The official way to get a single arbitrary value from a set without
removing it is:
for value in the_set:
return value
GvR recently gave an example of how to process a single element in a
possibly-empty iterator:
for x in it:
print(x)
break
else:
print('nothing')
so there are two examples of using a for-loop to *not* loop over something.
- Ducktyping. Just because some code is using a goose, doesn't mean that a
goose is required. Perhaps a duck is required but a goose is close enough.
Next problem, what the heck is res? We're not back in the punch-card
days. We don't have to abbreviate variable names to save columns.
*shrug* I didn't pick the name. But res is a standard abbreviation
for result or resource, and from context it clearly should be result.
[...]
I think this function makes a good test to separate the masters from the
apprentices.
The goal of good code is NOT to separate the masters from the
apprentices. The goal of good code is to be correct
So far I agree.
and easy to
understand by the next guy who comes along to maintain it.
No. That is *one secondary goal*. Efficiency is another secondary goal.
Sometimes education is even more important, and in this specific case the
function is being used to teach people.
Another secondary goal ought to be beauty and elegance over ugliness. It's
not often a beautiful function actually is good enough for production use.
It's usually surrounded by an inelegant if not downright ugly pile of code
testing arguments, checking for error conditions, handling corner cases,
etc. That ugliness can obfuscate the underlying algorithm and make the
function harder to understand. If it isn't *necessary*, take it out.
Perfection is achieved, not when there is nothing more to add,
but when there is nothing left to take away.
While ease of maintenance is an important goal, think about what we are
discussing. zip() is a primitive function. Once you have decided on the
public API, it will probably never need any maintenance. It's not like the
requirements will change -- yeah, we used to want to zip the items
together, but now we need to add a 9% superannuation surcharge to them
first.
The beauty of this code is that it is so simple that it cannot fail to be
bug-free. (Am I wrong? Have I missed a corner-case?) There's no unnecessary
code in the function that can hide bugs and obfuscate what it does.
And it is so simple that it's hard to see anything written in pure Python
being more efficient. So as far as maintenance goes, that's irrelevant.
That function is simply finished, done, complete. Anything you do to
improve it can only make it slower, buggier, or uglier.
(And that's a rare and beautiful thing in code.)
If you can read this function and instantly tell how it works, that it is
bug-free and duplicates the behaviour of the built-in zip(), you're
probably Raymond Hettinger. If
Re: Python Iterables struggling using map() built-in
On 12/8/2014 9:50 PM, Steven D'Aprano wrote: Roy Smith wrote: Chris Angelico wrote: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. Yes. Me too. 99% of the time when you see while foo, that's what you'll get, so it's the safe assumption. But it's only an assumption, not a requirement. When you read a bit more of the code and see that iters isn't being modified, your reaction ought to be closer oh wow, that's neat To me it is a code smell. iters is empty if and only if args is empty. If args is empty, iters should not be created. if args: iters = ... while True ... (return on exception) makes the logic clear. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Tue, 09 Dec 2014 00:03:33 -0500, Terry Reedy wrote: On 12/8/2014 9:50 PM, Steven D'Aprano wrote: Roy Smith wrote: Chris Angelico wrote: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. Yes. Me too. 99% of the time when you see while foo, that's what you'll get, so it's the safe assumption. But it's only an assumption, not a requirement. When you read a bit more of the code and see that iters isn't being modified, your reaction ought to be closer oh wow, that's neat To me it is a code smell. iters is empty if and only if args is empty. If args is empty, iters should not be created. if args: iters = ... while True ... (return on exception) makes the logic clear. When you understand why this body of code is inelegant and ugly, you should understand why the above is inelegant and ugly: def double(x): if x != 0: x *= 2 return x -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Python Iterables struggling using map() built-in
Hello everyone,
I'm currently in the process of self-study journey, so I have some questions
arisen from time to time.
Today I would like to talk about iterables and iterators,(ask for your help
actually ^_^).
Before I'll continue, just wanted to draw your attention to the fact that I
did some RTFM before posting.
Links:
1. map() built-in definitions:
https://docs.python.org/3/library/functions.html#map -for Python 3.X
https://docs.python.org/2.6/library/functions.html#map - for Python 2.6.X
2. Glossary definitions of iterable and iterator:
https://docs.python.org/3/glossary.html?highlight=glossary
3. Iterator Types:
https://docs.python.org/2/library/stdtypes.html#typeiter
4. iter() definition:
https://docs.python.org/2/library/functions.html#iter
5. Some StackOverflow links, related to the topic:
http://stackoverflow.com/questions/13054057/confused-with-python-lists-are-t
hey-or-are-they-not-iterators
http://stackoverflow.com/questions/9884132/understanding-pythons-iterator-it
erable-and-iteration-protocols-what-exact
http://stackoverflow.com/questions/19523563/python-typeerror-int-object-is-n
ot-iterable
http://stackoverflow.com/questions/538346/iterating-over-a-string
6. And of course, re-read couple of times a relevant parts of the
book ('Learning Python by Mark Lutz).
But the questions still persist, maybe because those examples look too
esoteric though.
Another warning: Despite all my attempts to make my questions as short as
possible it still looks huge. My apologies.
The problem:
Here is the book's example:
Consider the following clever alternative coding for this chapter's zip
emulation examples,
adapted from one in Python's manuals at the time I wrote these words:
def myzip(*args):
iters = map(iter, args)
while iters:
res = [next(i) for i in iters]
yield tuple(res)
Because this code uses iter and next, it works on any type of iterable. Note
that there
is no reason to catch the StopIteration raised by the next(it) inside the
comprehension
here when any one of the arguments' iterators is exhausted-allowing it to
pass ends
this generator function and has the same effect that a return statement
would. The
while iters: suffices to loop if at least one argument is passed, and avoids
an infinite
loop otherwise (the list comprehension would always return an empty list).
This code works fine in Python 2.X as is:
list(myzip('abc', 'lmnop'))
[('a', 'l'), ('b', 'm'), ('c', 'n')]
But it falls into an infinite loop and fails in Python 3.X, because the 3.X
map returns a
one-shot iterable object instead of a list as in 2.X. In 3.X, as soon as
we've run the list
comprehension inside the loop once, iters will be exhausted but still True
(and res will
be []) forever. To make this work in 3.X, we need to use the list built-in
function to
create an object that can support multiple iterations:
def myzip(*args):
iters = list(map(iter, args)) # Allow multiple scans
...rest as is...
Run this on your own to trace its operation. The lesson here: wrapping map
calls in
list calls in 3.X is not just for display!
*END OF THE BOOK EXAMPLE
According to the book , in order to get things work properly in Python 3.X,
I should write this code:
def myzip(*args):
iters = list(map(iter, args))
while iters:
res = [next(i) for i in iters]
yield tuple(res)
And all seemed to be clear here, but, when I tried to run this thing:
k= myzip(1, 2, 3, 4)
next(k)
I got this result:
Traceback (most recent call last):
File pyshell#73, line 1, in module
next(k)
File pyshell#65, line 2, in myzip
iters = list(map(iter, args))
TypeError: 'int' object is not iterable
Finding the problem:
I started to investigate further in order to catch the bug:
What I've tried?
1. L = [1, 2, 3, 4]
iter(L) is L
False --- According to the theory it's OK, because list doesn't have
self iterators.
k = iter(L)
print(k)
list_iterator object at 0x03233F90
next(k)
1
print(list(map(iter, L)))
Traceback (most recent call last):
File pyshell#88, line 1, in module
print(list(map(iter, L)))
TypeError: 'int' object is not iterable
2. I went to strings:
S = 'spam'
iter(S) is S
False ---Same about strings
string = iter(S)
string
str_iterator object at 0x02E24F30
next(string)
's'
and so on.
And then just tried this one:
print(list(map(iter, S)))
[str_iterator object at 0x02E24FF0,
str_iterator object at 0x02E24CF0,
str_iterator object at 0x02E24E10,
str_iterator object at 0x02E24DF0] - At this moment Wrecking
Ball song played in my head %)))
k = list(map(iter,S))
next(k[0])
's'
next(k[0])
Traceback (most recent call
Re: Python Iterables struggling using map() built-in
On Sun, Dec 7, 2014 at 3:44 AM, Ivan Evstegneev webmailgro...@gmail.com wrote: (quoting from the book) Because this code uses iter and next, it works on any type of iterable. Note that there is no reason to catch the StopIteration raised by the next(it) inside the comprehension here when any one of the arguments’ iterators is exhausted—allowing it to pass ends this generator function and has the same effect that a return statement would. I'd just like to point out something here that's about to change. In future versions of Python (starting with 3.5 with a governing directive, and always happening in 3.7), it *will* be important to catch the StopIteration. You can read more about it in PEP 479: https://www.python.org/dev/peps/pep-0479/ By the way, your email would have been far better sent as plain text. It's a bit messy here. k= myzip(1, 2, 3, 4) next(k) I got this result: Traceback (most recent call last): File pyshell#73, line 1, in module next(k) File pyshell#65, line 2, in myzip iters = list(map(iter, args)) TypeError: 'int' object is not iterable We'll get back to this later. 1. L = [1, 2, 3, 4] iter(L) is L False --- According to the “theory” it’s OK, because list doesn’t have self iterators. That's because L is a list, not an iterator. A list is iterable, but since you can iterate over it more than once, it returns a separate list_iterator whenever you call iter() on it. S = 'spam' iter(S) is S False ---Same about strings Again, strings are iterable, they are not iterators. L = [1, 2, 3, 4] k = iter(L) That's calling iter on L list(map(iter, L)) That's calling iter on *every element of* L. 2. Why strings are allowed “to become” an iterators(self-iterators)? Maybe because of files(reading from file) ? Because a string is a sequence of characters. You can iterate over a string to work with its characters. Why the infinite loop would be there and why should list() to make it finite? o_0 If you play around with it, you'll see why the infinite loop happens. Actually, even simpler: try calling that zip function with no arguments, and see what happens. I would suggest not trying to rewrite zip(), but develop your own iterator workings, in order to better understand what's going on. In fact, you can completely avoid map(), using only a longhand form of generator expression; and if you do that, you can easily switch in a list comprehension and see exactly what it's doing. But regardless of the above, I suggest reading through the section Explanation of generators, iterators, and StopIteration in PEP 479. It explains some important concepts which are best kept separate in your head, as conflating them will only confuse. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
RE: Python Iterables struggling using map() built-in
Awesome Ned, Believe it or not, but I was browsing web for the answer about a half an hour ago. Guess what? I found your web page with the explanations you provided there. ))) Finally, I was ready to send this question to you directly, cause I didn't know that you subscribed to this mailing list too. ^_^ But, you was a bit faster. What a surprise. ))) ))) ))) Thanks a lot for your answer. Ivan. -Original Message- From: Python-list [mailto:python-list-bounces+webmailgroups=gmail@python.org] On Behalf Of Ned Batchelder Sent: Sunday, December 7, 2014 17:29 To: python-list@python.org Subject: Re: Python Iterables struggling using map() built-in On 12/6/14 11:44 AM, Ivan Evstegneev wrote: And as I've promised the question section: 1.What actually map() trying to do in Python 3.X? I mean, why is this works fine: L = [1, 2, 3, 4] k = iter(L) next(k) 1 and so on. But not this: list(map(iter, L)) Traceback (most recent call last): File pyshell#88, line 1, in module print(list(map(iter, L))) TypeError: 'int' object is not iterable Let's unpack the code. You are running: map(iter, L) which is equivalent to: map(iter, [1, 2, 3, 4]) which executes: iter(1), iter(2), iter(3), iter(4) If you try iter(1), you get the error you are seeing. Integers are not iterable. What values would it produce? You ask what this is doing in Python 3, but it happens in any Python version, because integers are not iterable. 2.Why strings are allowed to become an iterators(self-iterators)? Maybe because of files(reading from file) ? I mean why, is this possible: print(list(map(iter, S))) [str_iterator object at 0x02E24FF0, str_iterator object at 0x02E24CF0, str_iterator object at 0x02E24E10, str_iterator object at 0x02E24DF0] This is a confusing thing in Python: strings are iterable, they produce a sequence of 1-character strings: list(hello) ['h', 'e', 'l', 'l', 'o'] This isn't because of reading from files. Open files are iterable, they produce a sequence of strings, one for each line in the file. This is why you can do: for line in file: process(line) Many times, it would be more convenient if strings were not iterable, but they are, and you need to keep it in mind when writing general-purpose iteration. 3.The last question Author says: /But it falls into an infinite loop and fails in Python 3.X, because the 3.X map returns a / /one-shot iterable object instead of a list as in 2.X. In 3.X, as soon as we've run the list / /comprehension inside the loop once, iters will be exhausted but still True/. /To make this / /work in 3.X, we need to use the list built-in function to create an object that can support / /multiple iterations. /(Like:Wat?! ^_^)// Why the infinite loop would be there and why should list() to make it finite? o_0 OK, let's go slowly. There are a few foundational concepts to get under our belt. *** CONCEPTS 1. An iterable is an object that you can pass to iter() to get an iterator. 2. An iterator is an object that can provide you with values one after the other, by using next(). next() will either return the next value, or raise a StopIteration exception, indicating that there are no more values. 3. The only operation supported on iterators is next(). You cannot start them over, you cannot ask if there will be more values, you cannot find out how many values there will be, you can't ask what the last value was, etc. By supporting only one operation, they allow the broadest possible set of implementations. 4. You can ask an iterator for an iterator, and it will return itself. That is: iter(some_iterator) is some_iterator. 5. The for NAME in EXPR construct is equivalent to this: expr_iter = iter(EXPR) try: while True: NAME = next(expr_iter) ..DO_SOMETHING.. except StopIteration: pass 6. In Python 2, map() produces a list of values. Lists are iterable. In Python 3, map() produces a map object, which is an iterator. *** PYTHON 2 EXECUTION OK, now, here is the code in question: 1. def myzip(*args): 2. iters = map(iter, args) 3. while iters: 4. res = [next(i) for i in iters] 5. yield tuple(res) Let's cover the Python 2 execution first. At line 2, map produces a list of iterators. Line 3 will loop forever. Nothing ever changes the list. In fact, this is a very confusing part of this code. The code should have said while True here, because it would work exactly the same. At line 4, we loop over our list of iterators, and pull the next value from each one. HERE'S THE IMPORTANT PART: because iters is a list, it is an iterable, and the for loop on this line will make a new list iterator to visit each iterator in turn. Every time this line is executed, every iterator in the list will be next'ed. Now res
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 2:29 AM, Ned Batchelder n...@nedbatchelder.com wrote: 3. The only operation supported on iterators is next(). You cannot start them over, you cannot ask if there will be more values, you cannot find out how many values there will be, you can't ask what the last value was, etc. By supporting only one operation, they allow the broadest possible set of implementations. Technically, this is one of only two operations *guaranteed to be* supported on iterators (the other being that `iter(iterator) is iterator`). There are plenty of iterators which do more than that, but all iterators are guaranteed to support next() and nothing more. (For instance, a generator object is an iterator, and it supports a lot more operations.) 5. The for NAME in EXPR construct is equivalent to this: expr_iter = iter(EXPR) try: while True: NAME = next(expr_iter) ..DO_SOMETHING.. except StopIteration: pass Small subtlety: The body of the for block is _not_ guarded by the try/except. It's more like this: expr_iter = iter(EXPR) while True: try: NAME = next(expr_iter) except StopIteration: break ..DO_SOMETHING.. NOTE: THIS EXAMPLE IS HORRIBLE. This code is crazy-confusing, and should never have been used as an example of iteration. It layers at least three iterations on top of each other, making it very difficult to see what is going on. It uses while iters where while True would do exactly the same thing (iters will never be false). There's one way for iters to be false, and that's if you give it no arguments at all. I've only just noticed this now, as I responded earlier with a suggestion to try passing it no args, which won't work because of that (or at least, won't work in Py2; in Py3, iters will indeed never be false, unless you use list() to coalesce the map). This is something which definitely ought to have been given a comment. Or, more usefully, a guarding 'if' before the loop, rather than needlessly checking at every iteration - if you want an infinite loop guarded by a precondition, write it as such so the subsequent reader can see that that's your intention. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Dec 7, 2014 8:31 AM, Ned Batchelder n...@nedbatchelder.com wrote: NOTE: THIS EXAMPLE IS HORRIBLE. This code is crazy-confusing, and should never have been used as an example of iteration. It layers at least three iterations on top of each other, making it very difficult to see what is going on. It uses while iters where while True would do exactly the same thing (iters will never be false). That's not quite correct; the while iters actually guards against the case where the passed args are empty. With no sub-iterators, no StopIteration would ever be raised, and the result would be an infinite generator of empty tuples. The while iters makes it return immediately instead. So it seems this example is even more confusing than you thought. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 5:27 AM, Ian Kelly ian.g.ke...@gmail.com wrote: On Dec 7, 2014 8:31 AM, Ned Batchelder n...@nedbatchelder.com wrote: NOTE: THIS EXAMPLE IS HORRIBLE. This code is crazy-confusing, and should never have been used as an example of iteration. It layers at least three iterations on top of each other, making it very difficult to see what is going on. It uses while iters where while True would do exactly the same thing (iters will never be false). That's not quite correct; the while iters actually guards against the case where the passed args are empty. With no sub-iterators, no StopIteration would ever be raised, and the result would be an infinite generator of empty tuples. The while iters makes it return immediately instead. So it seems this example is even more confusing than you thought. I'm actually glad PEP 479 will break this kind of code. Gives a good excuse for rewriting it to be more readable. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Chris Angelico wrote: On Mon, Dec 8, 2014 at 5:27 AM, Ian Kelly ian.g.ke...@gmail.com wrote: On Dec 7, 2014 8:31 AM, Ned Batchelder n...@nedbatchelder.com wrote: NOTE: THIS EXAMPLE IS HORRIBLE. This code is crazy-confusing, and should never have been used as an example of iteration. It layers at least three iterations on top of each other, making it very difficult to see what is going on. It uses while iters where while True would do exactly the same thing (iters will never be false). That's not quite correct; the while iters actually guards against the case where the passed args are empty. With no sub-iterators, no StopIteration would ever be raised, and the result would be an infinite generator of empty tuples. The while iters makes it return immediately instead. So it seems this example is even more confusing than you thought. I'm actually glad PEP 479 will break this kind of code. Gives a good excuse for rewriting it to be more readable. What kind of code is that? Short, simple, Pythonic and elegant? :-) Here's the code again, with indentation fixed: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) That is *beautiful code*. It's written for Python 2, where map returns a list, so the while iters line is morally equivalent to: while iters != [] and True It would be even more beautiful if we get rid of the unnecessary temporary variable: def myzip(*args): iters = map(iter, args) while iters: yield tuple([next(i) for i in iters]) I think this function makes a good test to separate the masters from the apprentices. No offence intended to Ned, who is a master, anyone can have a bad day or a blind spot. If you can read this function and instantly tell how it works, that it is bug-free and duplicates the behaviour of the built-in zip(), you're probably Raymond Hettinger. If you can tell what it does but you have to think about it for a minute or two before you understand why it works, you can call yourself a Python master. If you have to sit down with the interactive interpreter and experiment for a bit to understand it, you're doing pretty well. I do not believe that good code must be obviously right. It's okay for code to be subtly right. Either is better than complicated code which contains no obvious bugs. How would we re-write this to work in the future Python 3.7? Unless I have missed something, I think we could write it like this: def myzip37(*args): iters = list(map(iter, args)) while iters: try: yield tuple([next(i) for i in iters]) except StopIteration: return which I guess is not too horrible. If Python had never supported the current behaviour, I'd probably be happy with this. But having seen how elegant generators *can* be, the post-PEP 479 version will always look bloated and clumsy to me, like Garfield next to a cheetah. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 10:33 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: How would we re-write this to work in the future Python 3.7? Unless I have missed something, I think we could write it like this: def myzip37(*args): iters = list(map(iter, args)) while iters: try: yield tuple([next(i) for i in iters]) except StopIteration: return which I guess is not too horrible. It's not horrible, and there are other ways it could be written too, which also aren't horrible. Yes, it's not quite as short as the other version; but more importantly, it's explicit about how StopIteration affects it. It's clear that this exception, if raised by _any_ of the iterators (even after consuming values from some of them, perhaps), will silently terminate the generator. The current behaviour favours a handful of cases like this, although personally I think the termination of zip() is simply this is what happens if we have no code here, so let's document it rather than being something inherently ideal; the new behaviour favours the debugging of many obscure cases and some less-obscure ones as well. Most importantly, if you run the old version of myzip on Python 3.7, you'll get an immediate and noisy RuntimeError when it terminates, and you'll know exactly where to go fix stuff; if you run a buggy generator on Python 3.4, you simply see no more results, without any explanation of why. I'd rather debug the RuntimeError that can be easily and trivially fixed in 99% of cases. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Chris Angelico wrote: I'm actually glad PEP 479 will break this kind of code. Gives a good excuse for rewriting it to be more readable. Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: What kind of code is that? Short, simple, Pythonic and elegant? :-) Here's the code again, with indentation fixed: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. That's not at all what's happening here, so my brain runs into a wall. Next problem, what the heck is res? We're not back in the punch-card days. We don't have to abbreviate variable names to save columns. Variable names are supposed to describe what they hold, and thus help you understand the code. I have no idea what res is supposed to be. Residue? Result? Rest_of_items? Response? None of these make much sense here, so I'm just left befuddled. It would be even more beautiful if we get rid of the unnecessary temporary variable: def myzip(*args): iters = map(iter, args) while iters: yield tuple([next(i) for i in iters]) Well, that's one way to solve the mystery of what res means, but it doesn't actually make it easier to understand. I think this function makes a good test to separate the masters from the apprentices. The goal of good code is NOT to separate the masters from the apprentices. The goal of good code is to be correct and easy to understand by the next guy who comes along to maintain it. If you can read this function and instantly tell how it works, that it is bug-free and duplicates the behaviour of the built-in zip(), you're probably Raymond Hettinger. If you can tell what it does but you have to think about it for a minute or two before you understand why it works, you can call yourself a Python master. If you have to sit down with the interactive interpreter and experiment for a bit to understand it, you're doing pretty well. That pretty much is the point I'm trying to make. If the code is so complicated that masters can only understand it after a couple of minutes of thought, and those of us who are just doing pretty well need to sit down and puzzle it out in the REPL, then it's too complicated for most people to understand. KISS beats elegant. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
In article mailman.16689.1417996247.18130.python-l...@python.org, Chris Angelico ros...@gmail.com wrote: On Mon, Dec 8, 2014 at 10:33 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: How would we re-write this to work in the future Python 3.7? Unless I have missed something, I think we could write it like this: def myzip37(*args): iters = list(map(iter, args)) while iters: try: yield tuple([next(i) for i in iters]) except StopIteration: return I'm still not liking this use of while. Yes, of course, it handles the special case of no arguments, but I'd be in-your-face about that (not tested): def myzip37(*args): iters = list(map(iter, args)) if not iters: return None while True: try: yield tuple([next(i) for i in iters]) except StopIteration: return This makes it really obvious that there's something going on inside the loop other than exhausting the control variable to cause it to exit. Although, to be honest, I'm wondering if this is more straight-forward (also not tested): def myzip37(*args): if not args: return iters = list(map(iter, args)) while True: try: yield tuple(map(next, iters)) except StopIteration: return -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 11:12 AM, Roy Smith r...@panix.com wrote: Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. That's not at all what's happening here, so my brain runs into a wall. I agree, with the caveat that this kind of thing makes a fine infinite loop: while No exception raised: # do stuff that can raise an exception while Playing more games: play_game() if not still_playing: break reset_game_board() Nobody expects a string literal to actually become false inside the loop. With a local name, yes, I would expect it to at least have a chance of becoming false. Next problem, what the heck is res? We're not back in the punch-card days. We don't have to abbreviate variable names to save columns. Variable names are supposed to describe what they hold, and thus help you understand the code. I have no idea what res is supposed to be. Residue? Result? Rest_of_items? Response? None of these make much sense here, so I'm just left befuddled. I take it as result, which makes plenty of sense to me. It's the thing that's about to be yielded. Given that there's not much else you can say in a meta-function like zip(), I have no problem with that. Here's a slightly different example: def mark_last(it): it = iter(it) lastres = sentinel = object() while more values coming: res = next(it, sentinel) if lastres is not sentinel: yield (lastres, res is sentinel) if res is sentinel: return lastres = res Use of res for result and lastres to mean res as of the previous iteration of the loop seems fine to me. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 11:27 AM, Roy Smith r...@panix.com wrote: Although, to be honest, I'm wondering if this is more straight-forward (also not tested): def myzip37(*args): if not args: return iters = list(map(iter, args)) Yes, I prefer this too. It's explicit and clear that passing no arguments will yield no values. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 12/7/2014 7:12 PM, Roy Smith wrote: Chris Angelico wrote: I'm actually glad PEP 479 will break this kind of code. Gives a good excuse for rewriting it to be more readable. Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: What kind of code is that? Short, simple, Pythonic and elegant? :-) Here's the code again, with indentation fixed: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. That's not at all what's happening here, so my brain runs into a wall. I agree. Too tricky. The code should have been def myzip(*args): if args: iters = map(iter, args) while True: res = [next(i) for i in iters] yield tuple(res) However, this 'beautiful' code has a trap. If one gets rid of the seemingly unneeded temporary list res by telescoping the last two lines into a bit too much into yield tuple(next(i) for i in iters) we now have an infinite generator, because tuple() swallows the StopIteration raised as a side-effect of next calls. def myzip(*args): if args: iters = map(iter, args) while True: try: result = [next(i) for i in iters] except StopIteration return yield tuple(res) makes the side-effect dependence of stopping clearer. Putting yield tuple([next(i) for i in iters]) in the try would also work. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
In article mailman.16690.1417998873.18130.python-l...@python.org, Chris Angelico ros...@gmail.com wrote: Next problem, what the heck is res? We're not back in the punch-card days. We don't have to abbreviate variable names to save columns. Variable names are supposed to describe what they hold, and thus help you understand the code. I have no idea what res is supposed to be. Residue? Result? Rest_of_items? Response? None of these make much sense here, so I'm just left befuddled. I take it as result, which makes plenty of sense to me. OK, so spell it out. Three more keystrokes (well, plus another three when you use it on the next line). And one of them is a vowel; they don't even cost much. The next guy who has to read your code will thank you for it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On Mon, Dec 8, 2014 at 11:45 AM, Roy Smith r...@panix.com wrote: I take it as result, which makes plenty of sense to me. OK, so spell it out. Three more keystrokes (well, plus another three when you use it on the next line). And one of them is a vowel; they don't even cost much. The next guy who has to read your code will thank you for it. Maybe. Personally, I don't mind the odd abbreviation; they keep the code small enough to eyeball, rather than spelling everything out everywhere. Using cur (or curr) for current, next for next, prev for previous, as prefixes to a short word saying *what* they're the current/next/previous of, is sufficiently obvious IMO to justify the repeated use of the abbreviation. Why does Python have int and str rather than integer and string? Or, worse, arbitrary_precision_integer and unicode_codepoint_string? Common words get shortened - it's a legit form of Huffman compression. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 12/7/14 7:12 PM, Roy Smith wrote: Chris Angelico wrote: I'm actually glad PEP 479 will break this kind of code. Gives a good excuse for rewriting it to be more readable. Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: What kind of code is that? Short, simple, Pythonic and elegant? :-) Here's the code again, with indentation fixed: def myzip(*args): iters = map(iter, args) while iters: res = [next(i) for i in iters] yield tuple(res) Ugh. When I see while foo, my brain says, OK, you're about to see a loop which is controlled by the value of foo being changed inside the loop. That's not at all what's happening here, so my brain runs into a wall. Next problem, what the heck is res? We're not back in the punch-card days. We don't have to abbreviate variable names to save columns. Variable names are supposed to describe what they hold, and thus help you understand the code. I have no idea what res is supposed to be. Residue? Result? Rest_of_items? Response? None of these make much sense here, so I'm just left befuddled. It would be even more beautiful if we get rid of the unnecessary temporary variable: def myzip(*args): iters = map(iter, args) while iters: yield tuple([next(i) for i in iters]) Well, that's one way to solve the mystery of what res means, but it doesn't actually make it easier to understand. I think this function makes a good test to separate the masters from the apprentices. The goal of good code is NOT to separate the masters from the apprentices. The goal of good code is to be correct and easy to understand by the next guy who comes along to maintain it. If you can read this function and instantly tell how it works, that it is bug-free and duplicates the behaviour of the built-in zip(), you're probably Raymond Hettinger. If you can tell what it does but you have to think about it for a minute or two before you understand why it works, you can call yourself a Python master. If you have to sit down with the interactive interpreter and experiment for a bit to understand it, you're doing pretty well. That pretty much is the point I'm trying to make. If the code is so complicated that masters can only understand it after a couple of minutes of thought, and those of us who are just doing pretty well need to sit down and puzzle it out in the REPL, then it's too complicated for most people to understand. KISS beats elegant. Now that I understand all the intricacies (thanks everyone!), this is how I would write it: def zip(*args): if not args: return iters = list(map(iter, args)) while True: try: result = [next(it) for it in iters] except StopIteration: return yield tuple(result) The implicit use of StopIteration to end the entire generator is far too implicit for my taste. This code expresses the intent much better. And good call on not being able to use: tuple(next(it) for it in iters) Again, the tricky implicit hidden StopIterations are confusing. One last tweak: why do we use map to make iters, but a list comprehension to make result? OK, let's try this: result = map(next, iters) Oops, another infinite loop on Py3, right, because map is lazy, so the StopIteration doesn't happen until the tuple() call. OK, try this: result = list(map(next, iters)) Nope, still infinite because now list() consumes the StopIteration again. Ugh. -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
On 2014-12-08 01:00, Chris Angelico wrote: On Mon, Dec 8, 2014 at 11:45 AM, Roy Smith r...@panix.com wrote: I take it as result, which makes plenty of sense to me. OK, so spell it out. Three more keystrokes (well, plus another three when you use it on the next line). And one of them is a vowel; they don't even cost much. The next guy who has to read your code will thank you for it. Maybe. Personally, I don't mind the odd abbreviation; they keep the code small enough to eyeball, rather than spelling everything out everywhere. Using cur (or curr) for current, next for next, prev for previous, as prefixes to a short word saying *what* they're the current/next/previous of, is sufficiently obvious IMO to justify the repeated use of the abbreviation. Why does Python have int and str rather than integer and string? Or, worse, arbitrary_precision_integer and unicode_codepoint_string? Common words get shortened - it's a legit form of Huffman compression. Not to mention len, def, iter, etc. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Steven D'Aprano wrote: I do not believe that good code must be obviously right. It's okay for code to be subtly right. If you write code as subtly as you can, you're not subtle enough to debug it... -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Iterables struggling using map() built-in
Terry Reedy wrote: However, this 'beautiful' code has a trap. If one gets rid of the seemingly unneeded temporary list res by telescoping the last two lines into a bit too much into yield tuple(next(i) for i in iters) we now have an infinite generator, because tuple() swallows the StopIteration raised as a side-effect of next calls. An excellent example of the kind of thing that PEP 479 is designed to catch! There couldn't be a better advertisement for it. :-) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
