Re: Pre-pep discussion material: in-place equivalents to map and filter
I understand that, the cost of change is such that it's very unlikely something like this ever goes into Python, but I feel like the interest of the proposition is being underestimated here, that's why I'm going to argue a few points and give a bit more context as needed. > While mapping and filtering are common operations, I'm not sure mapping > and filtering and then reassigning back to the original sequence is > especially common. It depends of your context. On the last 3 months, I stumbled across this at least 3 times, which is 3 times more than I used a lambda or a metaclass or a decorator or other fancy language feature that we simply avoid whenever possible. It also happened to my collegue. I remember these examples because we had a bit of humour about how nice can be inlined ternaries inside comprehensions, but I could be missing a lot more. The reason I'm being especially impacted by this is because I am maintainer of a decent-sized Python application (~50-100K lines of code) that extensively uses lists and dictionaries. We value "low level" data manipulation and efficiency a lot more than complex, non-obvious constructs. In other contexts, it may be very different. Note that my context is only relevant for illustration here, I don't expect a feature to save me since we are currently shipping to Centos 6 and thus will not see the light of Python 3.7 in the next 10 years (optimistic estimation). > Arthur, I would suggest looking at what numpy and pandas do. In my example context, their benefits can't apply, because I'm not going to rewrite code that uses lists for them to uses np.array instead for example. Although the performance boost is likely to be bigger if used properly, I would have prefered a lesser boost that comes for (almost) free. Like most Python programmers, I'm not in the case of needing a performance boost very bad, but that does not mean I disregard performance entirely. The need of performance is not so binary that it either don't matter at all or is enough to motivate a rewrite. > To my eyes, this proposed syntax is completely foreign I must admit I don't have much imagination for syntax proposals...all that mattered to me here was to make it clear you are doing an in-place modification. Feel free to completely ignore that part. Any proposal welcomed of course. About Readability & Redundancy I have misused the terms here, but I wasn't expecting so much nitpicking. I should have used the term maintenability, because that one is bland and subjective enough that nobody would have noticed :D How about "I find that cooler." Good enough ? In a less sarcastic tone: What I truely meant here is that when you contain the behavior of your code inside a specific keyword or syntax, you are making your intentions clear to the reader. It may be harder for him to gain access to the knowledge in the first place, but makes it easier over time. Famous example: When you learned programming, you may have had no idea what "+=" was doing, but now you do, you probably rate "a += 2" syntax to be much better than "a = a + 2". You make the economy of a token, but more important, you make your intentions clearer because "+=" rings a bell, wihle "=" is a more generic syntax with a broader meaning. > So map() here is less than a factor of two slower. I wouldn't call > that "especially bad" -- often times, a factor of two is not important. > What really hurts is O(N**2) performance, or worse. When you evaluate your application bottleneck or your average daily algorithmic evaluation, perhaps. When regarding language core features we are not on the same scale. If a language X is 2 times faster than a language Y to do the same task, that's a huge seller, and is of real importance. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
> If slice assignment is done as I hope it will optimize remain memory
operations.
Bad news.
http://stackoverflow.com/questions/4948293/python-slice-assignment-memory-usage/4948508#4948508
> If you want something like C++ move semantics, use C++.
I don't see anything like this in my proposal. If any in-place operation is
"C++ semantics" how do you explain there is already a bunch of in-place
operator stuffed in Python that have even less justification to their
existance than map or filter does such as sort/sorted, reverse/reversed ?
Especially troubling since the optimisation in a sort operation is likely
to be less significant than in a linear algorithm.
2016-11-04 1:03 GMT+01:00 Terry Reedy :
> On 11/3/2016 2:56 AM, arthurhavli...@gmail.com wrote:
>
> lst = [ item for item in lst if predicate(item) ]
>> lst = [ f(item) for item in lst ]
>>
>> Both these expressions feature redundancy, lst occurs twice and item at
>> least twice. Additionally, the readability is hurt, because one has to dive
>> through the semantics of the comprehension to truely understand I am
>> filtering the list or remapping its values.
>>
> ...
>
>> A language support for these operations to be made in-place could improve
>> the efficiency of this operations through reduced use of memory.
>>
>
> We already have that: slice assignment with an iterator.
>
> lst[:] = (item for item in list if predicate(item))
> lst[:] = map(f, lst) # iterator in 3.x.
>
> To save memory, stop using unneeded temporary lists and use iterators
> instead. If slice assignment is done as I hope it will optimize remain
> memory operations. (But I have not read the code.) It should overwrite
> existing slots until either a) the iterator is exhausted or b) existing
> memory is used up. When lst is both source and destination, only case a)
> can happen. When it does, the list can be finalized with its new contents.
>
> As for timings.
>
> from timeit import Timer
> setup = """data = list(range(1))
> def func(x):
> return x
> """
> t1a = Timer('data[:] = [func(a) for a in data]', setup=setup)
> t1b = Timer('data[:] = (func(a) for a in data)', setup=setup)
> t2a = Timer('data[:] = list(map(func, data))', setup=setup)
> t2b = Timer('data[:] = map(func, data)', setup=setup)
>
> print('t1a', min(t1a.repeat(number=500, repeat=7)))
> print('t1b', min(t1b.repeat(number=500, repeat=7)))
> print('t2a', min(t2a.repeat(number=500, repeat=7)))
> print('t2b', min(t2b.repeat(number=500, repeat=7)))
> #
> t1a 0.5675313005414555
> t1b 0.7034254675598604
> t2a 0.518128598520
> t2b 0.5196112759726024
>
> If f does more work, the % difference among these will decrease.
>
>
>
> --
> Terry Jan Reedy
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > > I don't know who you are quoting there. It is considered impolite to quote > people without giving attribution, and makes it harder to respond. > My bad. I was unaware of that. This was quoted from Ned Batchelder's mali. 2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > I trust that isn't actually the case, but by describing decorators and > lambda as too fancy to use, you're giving a good impression of somebody who > doesn't know what they're doing. > I was a bit humorous here, making fun about the fact I do not write anything sophisticated. However, the fact that we do avoid them is entirely true ! We do so because these concepts are foreign to the average developer (lambda being a bit apart: it's easily understood, but functional constructs may not, so its a result of us avoiding functional constructs as well). Pick 10 programmers for hire and count how many know how to write a decorator. If you have specified you needed python specialists, you may have 3-4. If not, you are lucky to find even one. And where I live we don't have the luxury of finding competent Python experts by kicking a tree. In our open space, we are 6 devs (+ one ex-dev being my manager), only 2 had previous Python experience: me and the CEO. And the other 4 are doing fine ! Locking ourselves in Python-specific semantics, over time, will make us loose precious dev time. The best code is the one everyone can understand. That is why I point this as "complex, non-obvious", and my manager wouldn't have so kind words (but he is a Java fan, so he doesn't count.) 2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > - you would have to demonstrate a good reason to think that this new > feature > will actually be more efficient and faster > This is easy but time-consuming, I could roll my implementation and showcase a few benchs. I am very confident that is the case. I would have bet that I would need to do it at some point, but I wanted to poll opinions a bit beforehand. 2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > In other words, in my opinion: > > - you would need to prove that there is a widespread need for in-place > modification of lists, where the lists are big enough that using a > temporary list is not practical > However this is tricky, I may very well be an exception. About your bench: I don't really know why you are surprised the for loop is slower. That's the result of comprehension being native while for loop isn't. That does not mean writing to a copy would save time for exchange of memory. In my opinion, the fact that we will win something is guaranteed because we save a copy call and do the exact same operation. There is nothing like cache magic optimization that could happen because the mapping needs to read the first list anyway. Nor we need a temporary buffer to cache operations since they are independent. Really, I am ready to make a serious bet. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
2016-11-05 12:47 GMT+01:00 Chris Angelico : > On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek > > But here's the thing. For everyone who writes a decorator function, > there could be dozens who use it. > The day that one guy leaves the team, suddenly you have code that's become a bit tricky to maintain. Our philosophy is that everyone should be able to contribute anywhere although I agree with you: there is such thing as using a decorator as a black box, and even diving into it isn't that hard. Notice I didn't say we don't use these features at all, we just tend to avoid them when a more familiar approach exist. Even so, maybe we are wrong to do so, and the code would be clearer if we used more features. Maybe we just lack the expertise and we are indeed stuck in an old mindset, after all. Anyway, I have some directives to make my code easy to access and must compose with what I would call industrial constraints. As a gambler I can't refuse that bet proposal, no sir :) Expect me to come back with a piece of code and/or questions about CPython engine. -- https://mail.python.org/mailman/listinfo/python-list
Re: Rawest raw string literals
No escaping is not something possible, in your suggested syntax ") is
ambigous. E.g. raw("abcd")") is ambigous.
Any sequence delimited string involves escaping, the only thing that
wouldnt would be size-defined strings but they are impractical.
Le 20/04/2017 à 18:03, Mikhail V a écrit :
On 20 April 2017 at 17:55, Chris Angelico wrote:
On Fri, Apr 21, 2017 at 1:44 AM, Mikhail V wrote:
What I think: why there is no some built-in function, for example like:
s = raw("ffmpeg -i "\\server-01\D\SER_Bigl__"")
which would just need *one* quote sign in the beginning and on the end.
Would it be useful, what do you think? I think looks better than triple quote.
In the past there were quite a lot additions to string manipulation,
probably there is already something like this.
What would be the argument passed to this function?
ChrisA
Yeah that is right, I cant imagine how this would work.
But I think you get the idea- I'd want something dedicated
to input raw strings with cute syntax and *no* escaping
at all.
Mikhail
--
https://mail.python.org/mailman/listinfo/python-list
