Re: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin
> With the increased emphasis on iterators and generators in Python 3.x, > the lack of a simple expression level equivalent to "for item in > iterable: pass" is occasionally irritating, especially when > demonstrating behaviour at the interactive prompt. I've sometimes thought that exhaust(iterator) or iterator.exhaust() would be a good thing to have - I've often wrote code doing basically "call this function for every element in this container, and idc about return values", but find myself using a list comprehension instead of generator. I guess it's such an edge case that exhaust(iterator) as builtin would be overkill (but perhaps itertools could have it ?), and most people don't pass around iterators, so (f(x) for x in y).exhaust() might not look natural to most people. It could return the value for the last() semantics, but I think exhaustion would often be more important than the last value. 2018-04-09 0:58 GMT+02:00 Greg Ewing : > Kyle Lahnakoski wrote: > >> Consider Serhiy Storchaka's elegant solution, which I reformatted for >> readability >> >>> smooth_signal = [ >>> average >>>for average in [0] >>>for x in signal >>> for average in [(1-decay)*average + decay*x] >>> ] > > > "Elegant" isn't the word I would use, more like "clever". > Rather too clever, IMO -- it took me some head scratching > to figure out how it does what it does. > > And it would have taken even more head scratching, except > there's a clue as to *what* it's supposed to be doing: > the fact that it's assigned to something called > "smooth_signal" -- one of those "inaccurate names" that > you disparage so much. :-) > > -- > Greg > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
> On Apr 8, 2018, at 6:43 PM, Tim Peters wrote: > >> My other common case for accumulate() is building cumulative >> probability distributions from probability mass functions (see the >> code for random.choice() for example, or typical code for a K-S test). > > So, a question: why wasn't itertools.accumulate() written to accept > iterables of _only_ numeric types? Akin to `sum()`. I gather from > one of Nick's messages that it was so restricted in 3.2. Then why was > it generalized to allow any 2-argument function? Prior to 3.2, accumulate() was in the recipes section as pure Python code. It had no particular restriction to numeric types. I received a number of requests for accumulate() to be promoted to a real itertool (fast, tested, documented C code with a stable API). I agreed and accumulate() was added to itertools in 3.2. It worked with anything supporting __add__, including str, bytes, lists, and tuples. More specifically, accumulate_next() called PyNumber_Add() without any particular type restriction. Subsequently, I got requests to generalize accumulate() to support any arity-2 function (with operator.mul offered as the motivating example). Given that there were user requests and there were ample precedents in other languages, I acquiesced despite having some reservations (if used with a lambda, the function call overhead might make accumulate() slower than a plain Python for-loop without the function call). So, that generalized API extension went into 3.3 and has remained unchanged ever since. Afterwards, I was greeted with the sound of crickets. Either it was nearly perfect or no one cared or both ;-) It remains one of the least used itertools. > Given that it was, `sum()` is no longer particularly relevant: the > closest thing by far is now `functools.reduce()`, which does support > an optional `initial` argument. Which it really should, because it's > impossible for the implementation to guess a suitable starting value > for an arbitrary user-supplied dyadic function. > > My example using accumulate() to generate list prefixes got snipped, > but same thing there: it's impossible for that snippet to work unless > an empty list is supplied as the starting value. And it's impossible > for the accumulate() implementation to guess that. Honestly, I couldn't immediately tell what this code was doing: list(accumulate([8, 4, "k"], lambda x, y: x + [y], first_result=[])) This may be a case where a person would be better-off without accumulate() at all. > In short, for _general_ use `accumulate()` needs `initial` for exactly > the same reasons `reduce()` needed it. The reduce() function had been much derided, so I've had it mentally filed in the anti-pattern category. But yes, there may be wisdom there. > BTW, the type signatures on the scanl (requires an initial value) and > scanl1 (does not support an initial value) implementations I pasted > from Haskell's Standard Prelude give a deeper reason: without an > initial value, a list of values of type A can only produce another > list of values of type A via scanl1. The dyadic function passed must > map As to As. But with an initial value supplied of type B, scanl can > transform a list of values of type A to a list of values of type B. > While that may not have been obvious in the list prefix example I > gave, that was at work: a list of As was transformed into a list _of_ > lists of As. That's impossible for scanl1 to do, but easy for scanl. Thanks for pointing that out. I hadn't considered that someone might want to transform one type into another using accumulate(). That is pretty far from my mental model of what accumulate() was intended for. Also, I'm still not sure whether we would want code like that buried in an accumulate() call rather than as a regular for-loop where I can see the logic and trace through it with pdb. As for scanl, I'm not sure what this code means without seeing some python equivalent. scanl:: (a -> b -> a) -> a -> [b] -> [a] scanl f q xs = q : (case xs of [] -> [] x:xs -> scanl f (f q x) xs) scanl1 :: (a -> a -> a) -> [a] -> [a] scanl1 f (x:xs) = scanl f x xs scanl1 _ [] = [] > Or, in short, someone coming from a typed functional language > background sees all sorts of things that rarely (if ever) come up in > number-crunching languages. Their sensibilities should count too - > although not much ;-) They should get _some_ extra consideration in > this context, though, because `itertools` is one of the first things > they dig into when they give Python a try. I concur. >> and it would have been distracting to even had the option. > > Distracting for how long? One second or two? ;-) Possibly forever. In my experience, if a person initially frames a problem wrong (or perhaps in a hard to solve way), it can take them a long time
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
Raymond Hettinger wrote: For neither of those use case categories did I ever want an initial value and it would have been distracting to even had the option. For example, when doing a discounted cash flow analysis, I was taught to model the various flows as a single sequence of up and down arrows rather than thinking of the initial balance as a distinct concept¹ There's always an initial value, even if it's implicit. The way accumulate() works can be thought of in two ways: (1) The initial value is implicitly the identity of whatever operation the function performs. (2) The first item in the list is the initial value, and the rest are items to be accumulated. Both of these are somewhat dodgy, IMO. The first one works only if the assumed identity is what you actually want, *and* there is always at least one item to accumulate. If those conditions don't hold, you need to insert the initial value as the first item. But this is almost certainly going to require extra code. The initial value and the items are conceptually different things, and are unlikely to start out in the same list together. What's more, the first thing the implementation of accumulate() does is extract the first item and treat it differently from the rest. So your code goes out of its way to insert the initial value, and then accumulate() goes out of its way to pull it out again. Something smells wrong about that. As an example, suppose you have a list [1, 2, 3] and you want to construct [], [1], [1, 2], [1, 2, 3]. To do that with accumulate() you need to write something like: accumulate([[], 1, 2, 3], lambda x, y: x + [y]) The fact that the first element of the list doesn't even have the same *type* as the rest should be a strong hint that forcing them to occupy the same list is an unnatural thing to do. -- Greg Because of this background, I was surprised to have the question ever come up at all (other than the symmetry argument that sum() has "start" so accumulate() must as well). When writing itertools.accumulate(), I started by looking to see what other languages had done. Since accumulate() is primarily a numerical tool, I expected that the experience of numeric-centric languages would have something to teach us. My reasoning was that if the need hadn't arisen for APL, R, Numpy, Matlab², or Mathematica, perhaps it really was just noise. My views may be dated though. Looking at the wheel sieve and collatz glide record finder, I see something new, a desire to work with lazy, potentially infinite accumulations (something that iterators do well but almost never arises in the world of fixed-length sequences or cumulative probability distributions). So I had been warming up to the idea, but got concerned that Nick could have had such a profoundly different idea about what the code should do. That cooled my interest a bit, especially when thinking about two key questions, "Will it create more problems than it solves?" and "Will anyone actually use it?". Raymond ¹ http://www.chegg.com/homework-help/questions-and-answers/solve-present-worth-cash-flow-shown-using-three-interest-factors-10-interest-compounded-an-q878034 ² https://www.mathworks.com/help/matlab/ref/accumarray.html ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
> >>> list(accumulate([1, 2, 3])) > [11, 13, 16] Wow! I would have sworn that said [1, 3, 6] when I sent it. Damn Gmail ;-) > >>> list(accumulate([1, 2, 3], initial=10)) > [10, 11, 13, 16] ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
[Raymond] > The Bayesian world view isn't much different except they would > prefer "prior" instead of "initial" or "start" ;-) > > my_changing_beliefs = accumulate(stream_of_new_evidence, bayes_rule, > prior=what_i_used_to_think) > > Though the two analogies are cute, I'm not sure they tell us much. In ... They're not intended to. They're just intended to shake people loose from picturing nothing subtler than adding a list of 3 integers ;-) > My own experience with actually using accumulations in algorithmic > code falls neatly into two groups. Many years ago, I used APL > extensively in accounting work and my recollection is that a part > of the convenience of "\+" was that the sequence length didn't change > (so that the various data arrays could interoperate with one another). Sure. > My other common case for accumulate() is building cumulative > probability distributions from probability mass functions (see the > code for random.choice() for example, or typical code for a K-S test). So, a question: why wasn't itertools.accumulate() written to accept iterables of _only_ numeric types? Akin to `sum()`. I gather from one of Nick's messages that it was so restricted in 3.2. Then why was it generalized to allow any 2-argument function? Given that it was, `sum()` is no longer particularly relevant: the closest thing by far is now `functools.reduce()`, which does support an optional `initial` argument. Which it really should, because it's impossible for the implementation to guess a suitable starting value for an arbitrary user-supplied dyadic function. My example using accumulate() to generate list prefixes got snipped, but same thing there: it's impossible for that snippet to work unless an empty list is supplied as the starting value. And it's impossible for the accumulate() implementation to guess that. In short, for _general_ use `accumulate()` needs `initial` for exactly the same reasons `reduce()` needed it. BTW, the type signatures on the scanl (requires an initial value) and scanl1 (does not support an initial value) implementations I pasted from Haskell's Standard Prelude give a deeper reason: without an initial value, a list of values of type A can only produce another list of values of type A via scanl1. The dyadic function passed must map As to As. But with an initial value supplied of type B, scanl can transform a list of values of type A to a list of values of type B. While that may not have been obvious in the list prefix example I gave, that was at work: a list of As was transformed into a list _of_ lists of As. That's impossible for scanl1 to do, but easy for scanl. Or, in short, someone coming from a typed functional language background sees all sorts of things that rarely (if ever) come up in number-crunching languages. Their sensibilities should count too - although not much ;-) They should get _some_ extra consideration in this context, though, because `itertools` is one of the first things they dig into when they give Python a try. > For neither of those use case categories did I ever want an initial value As above, in all your related experiences "0" was a suitable base value, so you had no reason to care. > and it would have been distracting to even had the option. Distracting for how long? One second or two? ;-) > ... > Because of this background, I was surprised to have the question ever > come up at all (other than the symmetry argument that sum() has "start" > so accumulate() must as well). As above, the real parallel here is to reduce(). `sum()` became an historical red herring when `accumulate()` was generalized. With a different background, you may just as well have been surprised if the question _hadn't_ come up. For example, this is a standard example in the Haskell world for how to define an infinite Fibonacci sequence with the initial two values f0 and f1: fibs = f0 : scanl (+) f1 fibs The part from `scanl` onward would be spelled in Python as accumulate(fibs, initial=f1) although it requires some trickery to get the recursive reference to work (details on request, but I'm sure you already know how to do that). > When writing itertools.accumulate(), I started by looking to see what > other languages had done. Since accumulate() is primarily a > numerical tool, I expected that the experience of numeric-centric > languages would have something to teach us. My reasoning was > that if the need hadn't arisen for APL, R, Numpy, Matlab², or Mathematica, > perhaps it really was just noise. In the itertools context, I also would have looked hard at Haskell experience. BTW, whoever wrote the current `accumulate()` docs also found a use for `initial`, but hacked around it: """ First-order recurrence relations can be modeled by supplying the initial value in the iterable and using only the accumulated total in func argument: """ followed by: >>> logistic_map = lambda x, _: r * x * (1 - x) >>> r = 3.8 >>> x0 = 0.4 >>> inp
Re: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin
Kyle Lahnakoski wrote: Consider Serhiy Storchaka's elegant solution, which I reformatted for readability smooth_signal = [ average for average in [0] for x in signal for average in [(1-decay)*average + decay*x] ] "Elegant" isn't the word I would use, more like "clever". Rather too clever, IMO -- it took me some head scratching to figure out how it does what it does. And it would have taken even more head scratching, except there's a clue as to *what* it's supposed to be doing: the fact that it's assigned to something called "smooth_signal" -- one of those "inaccurate names" that you disparage so much. :-) -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three!
# Dict display
data = {
key_a: local_a := 1,
key_b: local_b := 2,
key_c: local_c := 3,
}
Isn’t this a set display with local assignments and type annotations? :o)
(I’m -1 on all of these ideas, btw. None help readability for me, and I read
much more code than I write.)
Top-posted from my Windows phone
From: Nick Coghlan
Sent: Sunday, April 8, 2018 6:27
To: Chris Angelico
Cc: python-ideas
Subject: Re: [Python-ideas] PEP 572: Statement-Local Name Bindings,take three!
On 23 March 2018 at 20:01, Chris Angelico wrote:
> Apologies for letting this languish; life has an annoying habit of
> getting in the way now and then.
>
> Feedback from the previous rounds has been incorporated. From here,
> the most important concern and question is: Is there any other syntax
> or related proposal that ought to be mentioned here? If this proposal
> is rejected, it should be rejected with a full set of alternatives.
I was writing a new stdlib test case today, and thinking about how I
might structure it differently in a PEP 572 world, and realised that a
situation the next version of the PEP should discuss is this one:
# Dict display
data = {
key_a: 1,
key_b: 2,
key_c: 3,
}
# Set display with local name bindings
data = {
local_a := 1,
local_b := 2,
local_c := 3,
}
# List display with local name bindings
data = {
local_a := 1,
local_b := 2,
local_c := 3,
}
# Dict display
data = {
key_a: local_a := 1,
key_b: local_b := 2,
key_c: local_c := 3,
}
# Dict display with local key name bindings
data = {
local_a := key_a: 1,
local_b := key_b: 2,
local_c := key_c: 3,
}
I don't think this is bad (although the interaction with dicts is a
bit odd), and I don't think it counts as a rationale either, but I do
think the fact that it becomes possible should be noted as an outcome
arising from the "No sublocal scoping" semantics.
Cheers,
Nick.
P.S. The specific test case is one where I want to test the three
different ways of spelling "the current directory" in some sys.path
manipulation code (the empty string, os.curdir, and os.getcwd()), and
it occurred to me that a version of PEP 572 that omits the sublocal
scoping concept will allow inline naming of parts of data structures
as you define them.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
> On Apr 8, 2018, at 12:22 PM, Tim Peters wrote: > > [Guido] >> Well if you can get Raymond to agree on that too I suppose you can go ahead. >> Personally I'm -0 but I don't really write this kind of algorithmic code >> enough to know what's useful. > > Actually, you do - but you don't _think_ of problems in these terms. > Neither do I. For those who do: consider any program that has state > and responds to inputs. When you get a new input, the new state is a > function of the existing state and the input. The Bayesian world view isn't much different except they would prefer "prior" instead of "initial" or "start" ;-) my_changing_beliefs = accumulate(stream_of_new_evidence, bayes_rule, prior=what_i_used_to_think) Though the two analogies are cute, I'm not sure they tell us much. In running programs or bayesian analysis, we care more about the result rather than the accumulation of intermediate results. My own experience with actually using accumulations in algorithmic code falls neatly into two groups. Many years ago, I used APL extensively in accounting work and my recollection is that a part of the convenience of "\+" was that the sequence length didn't change (so that the various data arrays could interoperate with one another). My other common case for accumulate() is building cumulative probability distributions from probability mass functions (see the code for random.choice() for example, or typical code for a K-S test). For neither of those use case categories did I ever want an initial value and it would have been distracting to even had the option. For example, when doing a discounted cash flow analysis, I was taught to model the various flows as a single sequence of up and down arrows rather than thinking of the initial balance as a distinct concept¹ Because of this background, I was surprised to have the question ever come up at all (other than the symmetry argument that sum() has "start" so accumulate() must as well). When writing itertools.accumulate(), I started by looking to see what other languages had done. Since accumulate() is primarily a numerical tool, I expected that the experience of numeric-centric languages would have something to teach us. My reasoning was that if the need hadn't arisen for APL, R, Numpy, Matlab², or Mathematica, perhaps it really was just noise. My views may be dated though. Looking at the wheel sieve and collatz glide record finder, I see something new, a desire to work with lazy, potentially infinite accumulations (something that iterators do well but almost never arises in the world of fixed-length sequences or cumulative probability distributions). So I had been warming up to the idea, but got concerned that Nick could have had such a profoundly different idea about what the code should do. That cooled my interest a bit, especially when thinking about two key questions, "Will it create more problems than it solves?" and "Will anyone actually use it?". Raymond ¹ http://www.chegg.com/homework-help/questions-and-answers/solve-present-worth-cash-flow-shown-using-three-interest-factors-10-interest-compounded-an-q878034 ² https://www.mathworks.com/help/matlab/ref/accumarray.html ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Accepting multiple mappings as positional arguments to create dicts
Hi!
I thought that maybe dict could accept several mappings as positional
arguments, like this:
class Dict4(dict):
> def __init__(self, *args, **kwargs):
> if len(args) > 1:
> if not all([isinstance(arg, dict) for arg in args]):
> raise TypeError('Dict4 expected instances of dict since
> multiple positional arguments were passed')
>
> temp = args[0].copy()
>
> for arg in args[1:]:
> temp.update(arg)
>
> super().__init__(temp, **kwargs)
> else:
> super().__init__(*args, **kwargs)
>
AFAIK, this wouldn't create compatibility problems, since you can't pass
two positional arguments now anyways.
It would be useful to solve the "sum/union dicts" discussion, for example:
requests.get(url, params=dict(params, {'foo': bar})
Whar are your thoughts?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
[Guido] > Well if you can get Raymond to agree on that too I suppose you can go ahead. > Personally I'm -0 but I don't really write this kind of algorithmic code > enough to know what's useful. Actually, you do - but you don't _think_ of problems in these terms. Neither do I. For those who do: consider any program that has state and responds to inputs. When you get a new input, the new state is a function of the existing state and the input. That's `accumulate`! It generates a sequence of new states as the sequence of incrementally updated states derived from a sequence of inputs according to the function passed to accumulate. In that view of the world, specifying a starting value is merely specifying the program's initial state - and from that view of the world, not allowing to specify a starting value is bizarre. While Will Ness's wheel sieve program, and my Collatz glide-record-finder, don't _derive_ from that view of the world, it turns out that specifying (and returning) an initial value is also useful for them, and despite that they're just doing integer running sums. A simple example from the broader view of the world is generating all the prefixes of a list: >>> from itertools import * >>> list(accumulate(chain([[]], [8, 4, "k"]), lambda x, y: x + [y])) [[], [8], [8, 4], [8, 4, 'k']] That's obviously easier to follow if written, e.g., list(accumulate([8, 4, "k"], lambda x, y: x + [y], first_result=[])) > I do think that the new parameter name ["first_result"] is ugly. But maybe > that's > the point. I noted later that the `accumulate` in the Python itertoolz package names its optional argument "initial". That's not ugly, and works just as well for me. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
2018-04-08 8:19 GMT+03:00 Nick Coghlan : > A name like "first_result" would also make it clearer to readers that > passing that parameter has an impact on the length of the output > series (since you're injecting an extra result), and also that the > production of the first result skips calling func completely (as can > be seen in Tim's str coercion example). > > So where I'd be -1 on: > > >>> list(accumulate(1, 2, 3)) > [1, 3, 6] > >>> list(accumulate(1, 2, 3, start=0)) > [0, 1, 3, 6] > >>> list(accumulate(1, 2, 3, start=1)) > [1, 2, 4, 7] > > I'd be +1 on: > > >>> list(accumulate(1, 2, 3)) > [1, 3, 6] > >>> list(accumulate(1, 2, 3, first_result=0)) > [0, 1, 3, 6] > >>> list(accumulate(1, 2, 3, first_result=1)) > [1, 2, 4, 7] > > It is a fair point! But the usual way to understand how to use an additional argument, is to try it or to look examples in the documentation. Concerning the topic of relationship between `sum` and `accumulate` I have another point of view. If `start` means something to the `sum`, there are no grounds for believing that it should mean the same for `accumulate`, these functions are not really comparable and fundamentally different. The closest `sum`s friend is `functools.reduce` which uses `initial` instead of `start`. ( the documentation uses `initializer` and the docstring uses `initial`, as for me I prefer the `initial`) and so there is already a discrepancy. Having said this I think that it is no matter how it will be named `start` or `initial` or `first_result` or `first`, which is more suitable. I would prefer `initial` to be the same as in `itertoolz` package. Regarding the second point, should it yield one more element if provided - I think everyone here agrees that yes. With kind regards, -gdg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin
On 2018-04-05 21:18, Steven D'Aprano wrote: > (I don't understand why so many people have such an aversion to writing > functions and seek to eliminate them from their code.) > I think I am one of those people that have an aversion to writing functions! I hope you do not mind that I attempt to explain my aversion here. I want to clarify my thoughts on this, and maybe others will find something useful in this explanation, maybe someone has wise words for me. I think this is relevant to python-ideas because someone with this aversion will make different language suggestions than those that don't. Here is why I have an aversion to writing functions: Every unread function represents multiple unknowns in the code. Every function adds to code complexity by mapping an inaccurate name to specific functionality. When I read code, this is what I see: > x = you_will_never_guess_how_corner_cases_are_handled(a, b, c) > y = you_dont_know_I_throw_a_BaseException_when_I_do_not_like_your_arguments(j, k, l) Not everyone sees code this way: I see people read method calls, make a number of wild assumptions about how those methods work, AND THEY ARE CORRECT! How do they do it!? It is as if there are some unspoken convention about how code should work that's opaque to me. For example before I read the docs on itertools.accumulate(list_of_length_N, func), here are the unknowns I see: * Does it return N, or N-1 values? * How are initial conditions handled? * Must `func` perform the initialization by accepting just one parameter, and accumulate with more-than-one parameter? * If `func` is a binary function, and `accumulate` returns N values, what's the Nth value? * if `func` is a non-cummutative binary function, what order are the arguments passed? * Maybe accumulate expects func(*args)? * Is there a window size? Is it equal to the number of arguments of `func`? These are not all answered by reading the docs, they are answered by reading the code. The code tells me the first value is a special case; the first parameter of `func` is the accumulated `total`; `func` is applied in order; and an iterator is returned. Despite all my questions, notice I missed asking what `accumulate` returns? It is the unknown unknowns that get me most. So, `itertools.accumulate` is a kinda-inaccurate name given to a specific functionality: Not a problem on its own, and even delightfully useful if I need it often. What if I am in a domain where I see `accumulate` only a few times a year? Or how about a program that uses `accumulate` in only one place? For me, I must (re)read the `accumulate` source (or run the caller through the debugger) before I know what the code is doing. In these cases I advocate for in-lining the function code to remove these unknowns. Instead of an inaccurate name, there is explicit code. If we are lucky, that explicit code follows idioms that make the increased verbosity easier to read. Consider Serhiy Storchaka's elegant solution, which I reformatted for readability > smooth_signal = [ > average > for average in [0] > for x in signal > for average in [(1-decay)*average + decay*x] > ] We see the initial conditions, we see the primary function, we see how the accumulation happens, we see the number of returned values, and we see it's a list. It is a compact, easy read, from top to bottom. Yes, we must know `for x in [y]` is an idiom for assignment, but we can reuse that knowledge in all our other list comprehensions. So, in the specific case of this Reduce-Map thread, I would advocate using the list comprehension. In general, all functions introduce non-trivial code debt: This debt is worth it if the function is used enough; but, in single-use or rare-use cases, functions can obfuscate. Thank you for your time. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
Another bit of prior art: the Python itertoolz package also supplies
`accumulate()`, with an optional `initial` argument. I stumbled into
that when reading a Stackoverflow "how can I do Haskell's scanl in
Python?" question.
https://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.accumulate
On Fri, Apr 6, 2018 at 8:02 PM, Raymond Hettinger
wrote:
>> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote:
>> On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor wrote:
>>> So some more humble proposals would be:
>>>
>>> 1) An initializer to itertools.accumulate
>>> functools.reduce already has an initializer, I can't see any controversy to
>>> adding an initializer to itertools.accumulate
>>
>> See if that's accepted in the bug tracker.
>
> It did come-up once but was closed for a number reasons including lack of use
> cases. However, Peter's signal processing example does sound interesting, so
> we could re-open the discussion.
>
> For those who want to think through the pluses and minuses, I've put together
> a Q&A as food for thought (see below). Everybody's design instincts are
> different -- I'm curious what you all think think about the proposal.
>
>
> Raymond
>
> -
>
> Q. Can it be done?
> A. Yes, it wouldn't be hard.
>
> _sentinel = object()
>
> def accumulate(iterable, func=operator.add, start=_sentinel):
> it = iter(iterable)
> if start is _sentinel:
> try:
> total = next(it)
> except StopIteration:
> return
> else:
> total = start
> yield total
> for element in it:
> total = func(total, element)
> yield total
>
> Q. Do other languages do it?
> A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes.
>
> *
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.html
> * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html
> * http://microapl.com/apl/apl_concepts_chapter5.html
> \+ 1 2 3 4 5
> 1 3 6 10 15
> * https://reference.wolfram.com/language/ref/Accumulate.html
> * https://www.haskell.org/hoogle/?hoogle=mapAccumL
>
>
> Q. How much work for a person to do it currently?
> A. Almost zero effort to write a simple helper function:
>
>myaccum = lambda it, func, start: accumulate(chain([start], it), func)
>
>
> Q. How common is the need?
> A. Rare.
>
>
> Q. Which would be better, a simple for-loop or a customized itertool?
> A. The itertool is shorter but more opaque (especially with respect
>to the argument order for the function call):
>
> result = [start]
> for x in iterable:
> y = func(result[-1], x)
> result.append(y)
>
> versus:
>
> result = list(accumulate(iterable, func, start=start))
>
>
> Q. How readable is the proposed code?
> A. Look at the following code and ask yourself what it does:
>
> accumulate(range(4, 6), operator.mul, start=6)
>
>Now test your understanding:
>
> How many values are emitted?
> What is the first value emitted?
> Are the two sixes related?
> What is this code trying to accomplish?
>
>
> Q. Are there potential surprises or oddities?
> A. Is it readily apparent which of assertions will succeed?
>
> a1 = sum(range(10))
> a2 = sum(range(10), 0)
> assert a1 == a2
>
> a3 = functools.reduce(operator.add, range(10))
> a4 = functools.reduce(operator.add, range(10), 0)
> assert a3 == a4
>
> a4 = list(accumulate(range(10), operator.add))
> a5 = list(accumulate(range(10), operator.add, start=0))
> assert a5 == a6
>
>
> Q. What did the Python 3.0 Whatsnew document have to say about reduce()?
> A. "Removed reduce(). Use functools.reduce() if you really need it; however,
> 99 percent of the time an explicit for loop is more readable."
>
>
> Q. What would this look like in real code?
> A. We have almost no real-world examples, but here is one from a
> StackExchange post:
>
> def wsieve(): # wheel-sieve, by Will Ness.
> ideone.com/mqO25A->0hIE89
> wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2,
> 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10]
> cs = accumulate(cycle(wh11), start=11)
> yield( next( cs)) # cf. ideone.com/WFv4f
> ps = wsieve() #
> codereview.stackexchange.com/q/92365/9064
> p = next(ps)# 11
> psq = p*p # 121
> D = dict( zip( accumulate(wh11, start=0), count(0))) # start
> from
> sieve = {}
> for c in cs:
> if c in sieve:
> wheel = sieve.pop(c)
> for m in wheel:
> if
Re: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three!
On Sun, Apr 8, 2018 at 8:01 AM, Steven D'Aprano wrote: > If we really wanted to keep the sublocal scoping, we could make > list/set/dict displays their own scope too. > > Personally, that's the only argument for sublocal scoping that I like > yet: what happens inside a display should remain inside the display, and > not leak out into the function. > That sounds like a reasonable proposal that we could at least consider. But I think it will not fly. Presumably it doesn't apply to tuple displays, because of reasonable examples like ((a := f(), a+1), a+2), and because it would create an ugly discontinuity between (a := f()) and (a := f(),). But then switching between [a := f(), a] and (a := f(), a) would create a discontinuity. For comprehensions and generator expressions there is no such discontinuity in the new proposal, since these *already* introduce their own scope. -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
Well if you can get Raymond to agree on that too I suppose you can go ahead. Personally I'm -0 but I don't really write this kind of algorithmic code enough to know what's useful. I do think that the new parameter name is ugly. But maybe that's the point. On Sat, Apr 7, 2018 at 10:26 PM, Tim Peters wrote: > Top-posting just to say I agree with Nick's bottom line (changing the > name to `first_result=`). I remain just +0.5, although that is up a > notch from yesterday's +0.4 ;-) > > --- nothing new below --- > > On Sun, Apr 8, 2018 at 12:19 AM, Nick Coghlan wrote: > > On 8 April 2018 at 14:31, Guido van Rossum wrote: > >> Given that two respected members of the community so strongly disagree > >> whether accumulate([], start=0) should behave like accumulate([]) or > like > >> accumulate([0]), maybe in the end it's better not to add a start > argument. > >> (The disagreement suggests that we can't trust users' intuition here.) > > > > The potential ambiguity I see is created mainly by calling the > > proposed parameter "start", while having it do more than just adjust > > the individual partial sum calculations by adding an extra partial > > result to the output series. > > > > If it's called something else (e.g. "first_result"), then the > > potential "sum(iterable, start=start)" misinterpretation goes away, > > and it can have Tim's desired effect of defining the first output > > value (effectively prepending it to the input iterable, without the > > boilerplate and overhead of actually doing so). > > > > A name like "first_result" would also make it clearer to readers that > > passing that parameter has an impact on the length of the output > > series (since you're injecting an extra result), and also that the > > production of the first result skips calling func completely (as can > > be seen in Tim's str coercion example). > > > > So where I'd be -1 on: > > > > >>> list(accumulate(1, 2, 3)) > > [1, 3, 6] > > >>> list(accumulate(1, 2, 3, start=0)) > > [0, 1, 3, 6] > > >>> list(accumulate(1, 2, 3, start=1)) > > [1, 2, 4, 7] > > > > I'd be +1 on: > > > > >>> list(accumulate(1, 2, 3)) > > [1, 3, 6] > > >>> list(accumulate(1, 2, 3, first_result=0)) > > [0, 1, 3, 6] > > >>> list(accumulate(1, 2, 3, first_result=1)) > > [1, 2, 4, 7] > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three!
On Sun, Apr 08, 2018 at 09:25:33PM +1000, Nick Coghlan wrote:
> I was writing a new stdlib test case today, and thinking about how I
> might structure it differently in a PEP 572 world, and realised that a
> situation the next version of the PEP should discuss is this one:
>
> # Dict display
> data = {
> key_a: 1,
> key_b: 2,
> key_c: 3,
> }
>
> # Set display with local name bindings
> data = {
> local_a := 1,
> local_b := 2,
> local_c := 3,
>}
I don't understand the point of these examples. Sure, I guess they would
be legal, but unless you're actually going to use the name bindings,
what's the point in defining them?
data = {
1,
(spam := complex_expression),
spam+1,
spam*2,
}
which I think is cleaner than the existing alternative of defining spam
outside of the set.
And for dicts:
d = {
'key': 'value',
(spam := calculated_key): (eggs := calculated_value),
spam.lower(): eggs.upper(),
}
> I don't think this is bad (although the interaction with dicts is a
> bit odd), and I don't think it counts as a rationale either, but I do
> think the fact that it becomes possible should be noted as an outcome
> arising from the "No sublocal scoping" semantics.
If we really wanted to keep the sublocal scoping, we could make
list/set/dict displays their own scope too.
Personally, that's the only argument for sublocal scoping that I like
yet: what happens inside a display should remain inside the display, and
not leak out into the function.
So that has taken me from -1 on sublocal scoping to -0.5 if it applies
to displays.
--
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three!
On 23 March 2018 at 20:01, Chris Angelico wrote:
> Apologies for letting this languish; life has an annoying habit of
> getting in the way now and then.
>
> Feedback from the previous rounds has been incorporated. From here,
> the most important concern and question is: Is there any other syntax
> or related proposal that ought to be mentioned here? If this proposal
> is rejected, it should be rejected with a full set of alternatives.
I was writing a new stdlib test case today, and thinking about how I
might structure it differently in a PEP 572 world, and realised that a
situation the next version of the PEP should discuss is this one:
# Dict display
data = {
key_a: 1,
key_b: 2,
key_c: 3,
}
# Set display with local name bindings
data = {
local_a := 1,
local_b := 2,
local_c := 3,
}
# List display with local name bindings
data = {
local_a := 1,
local_b := 2,
local_c := 3,
}
# Dict display
data = {
key_a: local_a := 1,
key_b: local_b := 2,
key_c: local_c := 3,
}
# Dict display with local key name bindings
data = {
local_a := key_a: 1,
local_b := key_b: 2,
local_c := key_c: 3,
}
I don't think this is bad (although the interaction with dicts is a
bit odd), and I don't think it counts as a rationale either, but I do
think the fact that it becomes possible should be noted as an outcome
arising from the "No sublocal scoping" semantics.
Cheers,
Nick.
P.S. The specific test case is one where I want to test the three
different ways of spelling "the current directory" in some sys.path
manipulation code (the empty string, os.curdir, and os.getcwd()), and
it occurred to me that a version of PEP 572 that omits the sublocal
scoping concept will allow inline naming of parts of data structures
as you define them.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
FYI: [Raymond] > ... > Q. Do other languages do it? > A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. > >... > * https://www.haskell.org/hoogle/?hoogle=mapAccumL Haskell has millions of functions ;-) `mapAccumL` is a God-awful mixture of Python's map(), reduce(), and accumulate() :-( The examples here should convince you it's nearly incomprehensible: http://zvon.org/other/haskell/Outputlist/mapAccumL_f.html A more-than-less direct equivalent to Python's `accumulate` is Haskell's `scanl1`: http://zvon.org/comp/r/ref-Haskell.html#Functions~Prelude.scanl1 That doesn't allow specifying an initial value. But, Haskell being Haskell, the closely related `scanl` function requires specifying an initial value, which is also the first element of the list it returns: http://zvon.org/comp/r/ref-Haskell.html#Functions~Prelude.scanl Of the two, `scanl` is more basic - indeed, the Standard Prelude defines scanl1 by peeling off the first element of the list and passing it as the initial value for scanl to use scanl:: (a -> b -> a) -> a -> [b] -> [a] scanl f q xs = q : (case xs of [] -> [] x:xs -> scanl f (f q x) xs) scanl1 :: (a -> a -> a) -> [a] -> [a] scanl1 f (x:xs) = scanl f x xs scanl1 _ [] = [] There are also analogous `scanr` and `scanr1` functions for doing "right-to-left" accumulation. So, bottom line: as usual, when looking to Haskell for prior art,, "all of the above - for a start" applies ;-) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
