Re: [Tutor] List comprehensions to search a list--amazing!
On Thu, Mar 19, 2015 at 4:52 AM, Peter Otten __pete...@web.de wrote: Dave Angel wrote: [...] By the way, if you were to use a plain old loop the expected speedup over the listcomp would be 2. You can break out of the loop when you have found the gap, after iterating over one half of the list on average. So for-loops are twice as amazing ;) Actually, this was my first approach to solving the problem. The catch to a list comprehension is it has to visit all the elements, while a binary search would visit log-base-2 of them. So instead of 1 elements, you'd be searching about 14 items. For large lists, it'd probably be much quicker to use the bisect module. https://docs.python.org/3.4/library/bisect.html Check out bisect.bisect_left() and bisect.bisect_right() I don't see how to directly use those functions on a list which is reverse-sorted, but the source is available. On my install, it's located at: /usr/lib/python3.4/bisect.py To back Dave's suggestion with some empirical data here are two ways to make bisect() work with a descending list (though if possible I would recommend that you change your script to use ascending lists). I could easily do this, though the data naturally presents itself as I stated originally. $ cat reverse_bisect2.py import bisect import random def break_listcomp(L, Vt): S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] return S[0] def break_bisect_reverse(L, Vt): L.reverse() result = bisect.bisect(L, Vt) L.reverse() return len(L) - result -1 class Items: def __init__(self, items): self.items = items def __len__(self): return len(self.items) def __getitem__(self, index): return self.items[len(self.items) - 1 - index] def break_bisect_virt(L, Vt): return len(L) - 1 - bisect.bisect(Items(L), Vt) random.seed(42) N = 10**6 data = [random.randrange(N) for i in range(10**5)] data = data + data data.sort(reverse=True) sample = random.sample(data, 10) $ break_bisect_reverse() reverses the list before and after applying bisect. This is still O(N), but the actual work is done in C. break_bisect_virt() wraps the actual list in a class that translates items[0] to items[len(items)-1] items[1] to items[len(items)-2] Thank you for taking time to write this. I may have questions later. and so on, thus providing a reversed view of the list without moving any values. This severely slows down access to a single value, but as bisect needs much fewer lookups than the listcomp the overall result is still a massive speedup. The actual timings: $ python3 -m timeit -s 'from reverse_bisect2 import data, sample, break_listcomp as f' '[f(data, v) for v in sample]' 10 loops, best of 3: 781 msec per loop $ python3 -m timeit -s 'from reverse_bisect2 import data, sample, break_bisect_reverse as f' '[f(data, v) for v in sample]' 100 loops, best of 3: 15 msec per loop $ python3 -m timeit -s 'from reverse_bisect2 import data, sample, break_bisect_virt as f' '[f(data, v) for v in sample]' 1000 loops, best of 3: 221 usec per loop So reverse/bisect is 50 times faster than the listcomp, and bisect/virt is 3500 times faster than the listcomp. You present a compelling case! I expect that a prepackaged linear interpolation function from numpy/scipy can still do better, and also handle the corner cases correctly. To use such a function you may have to reverse order of the values. This is not an option for me as I would not be allowed to install numpy/scipy. -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On 03/23/2015 10:17 PM, Dave Angel wrote: On 03/23/2015 09:42 PM, boB Stepp wrote: Not really. See Steve's OOPS. Peter's response for some numbers. If I had to guess, I'd say that for lists over 100 items, you should use bisect or equivalent. But I'd also say you should have one algorithm in your final code, even if it's sub-optimal for tiny lists. If even a fraction of the searches are going to be on a list of 10k items, you should switch to a bisect approach. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On Mon, Mar 23, 2015 at 08:42:23PM -0500, boB Stepp wrote: On Thu, Mar 19, 2015 at 12:10 AM, Dave Angel da...@davea.name wrote: The catch to a list comprehension is it has to visit all the elements, while a binary search would visit log-base-2 of them. So instead of 1 elements, you'd be searching about 14 items. I suspected as much, but had not verified this. Nonetheless, this may prove sufficiently fast. I will have to test this with my final data files. Right now I am using test cases, while I continue to design, check, rewrite, etc. For large lists, it'd probably be much quicker to use the bisect module. https://docs.python.org/3.4/library/bisect.html Can you give me a ballpark number for large, where this would start making a meaningful difference? Tell us what you consider a meaningful difference :-) What counts as too slow will depend on: - what you are doing - how often you are doing it - what else you're doing at the same time - what hardware you have to run it on - whether you are running the program only once, or repeatedly etc. E.g. taking 30 seconds to iterate over a billion items in a list is insignificant if this is part of a bigger program which takes twenty minutes to run, but critical if you are hoping to run the program thirty times a second, hundreds of times a day. But I've never heard anyone complain that a program was too fast :-) (Actually, that's not quite true. Sometimes if the user is expecting a program function to take a while, say Rebuild database, and it actually runs near instantaneously, it is indeed *too fast* because it can give the user the idea that the rebuild function isn't working. But that's a UI issue, not a speed issue.) But if you twist my arm, and force me to pluck some round numbers from thin air, I would guess: - for under ten items, a linear search will be insignificantly faster; - for under a hundred items, there's no meaningful difference; - for under a million items, binary search will be typically better but a linear search still acceptable (everything is fast for small N); - for over a million items, linear search will typically be unacceptably slow. For certain definitions of what's acceptable or not :-) -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On Thu, Mar 19, 2015 at 12:10 AM, Dave Angel da...@davea.name wrote: The catch to a list comprehension is it has to visit all the elements, while a binary search would visit log-base-2 of them. So instead of 1 elements, you'd be searching about 14 items. I suspected as much, but had not verified this. Nonetheless, this may prove sufficiently fast. I will have to test this with my final data files. Right now I am using test cases, while I continue to design, check, rewrite, etc. For large lists, it'd probably be much quicker to use the bisect module. https://docs.python.org/3.4/library/bisect.html Can you give me a ballpark number for large, where this would start making a meaningful difference? Check out bisect.bisect_left() and bisect.bisect_right() It looks like this should work. Thanks! I will investigate. I don't see how to directly use those functions on a list which is reverse-sorted, but the source is available. On my install, it's located at: /usr/lib/python3.4/bisect.py And I see this is available on my oldest Python installlation, 2.4.4, too. -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On 03/23/2015 09:42 PM, boB Stepp wrote: On Thu, Mar 19, 2015 at 12:10 AM, Dave Angel da...@davea.name wrote: The catch to a list comprehension is it has to visit all the elements, while a binary search would visit log-base-2 of them. So instead of 1 elements, you'd be searching about 14 items. I suspected as much, but had not verified this. Nonetheless, this may prove sufficiently fast. I will have to test this with my final data files. Right now I am using test cases, while I continue to design, check, rewrite, etc. For large lists, it'd probably be much quicker to use the bisect module. https://docs.python.org/3.4/library/bisect.html Can you give me a ballpark number for large, where this would start making a meaningful difference? Not really. See Steve's response for some numbers. If I had to guess, I'd say that for lists over 100 items, you should use bisect or equivalent. But I'd also say you should have one algorithm in your final code, even if it's sub-optimal for tiny lists. If even a fraction of the searches are going to be on a list of 10k items, you should switch to a bisect approach. I'd have to measure it, same as anyone. And because of the reverse-ordering problem, you have to weigh the advantages of using a standard library (which is unlikely to be buggy), versus making an edited version which works directly on reversed lists. It also can matter how many times you're searching the same list. If you're going to be many lookups, it's worth keeping a reversed copy. You can reverse as simply as rlist = mylist[::-1] -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On Mon, Mar 23, 2015 at 10:17:11PM -0400, Dave Angel wrote: On 03/23/2015 09:42 PM, boB Stepp wrote: Can you give me a ballpark number for large, where this would start making a meaningful difference? Not really. See Steve's response for some numbers. o_O Have you borrowed Guido's Time Machine? I hadn't even finished writing my post??? -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
Patrick Thunstrom wrote: The generalized problem: L = [V0, V1, ..., Vn], where V0 = V1 = V2 = ... = Vn . Find index i, such that V[i] = Vt = V[i + 1], where Vt is the test value being searched for. I need to know the indices i and i + 1, which I need to interpolate based on where Vt falls. The solution (As a sublist, S) I worked out tonight after experimenting with comprehension syntax is: S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] And, of course, the index i I need is: i = S[0] I tested this out with concrete examples in the interpreter, such as with a list, L: L = [item for item in range(1, 0, -1)] and trying different test values. It was blazingly fast, too! All I can say is: WOW!!! By the way, if you were to use a plain old loop the expected speedup over the listcomp would be 2. You can break out of the loop when you have found the gap, after iterating over one half of the list on average. So for-loops are twice as amazing ;) For the same basic speed up, a generator expression with a call to next() will produce similar results. Without the additional code to get the indexed item. index = (i for i, _ in enumerate(original_list) if original_list[i] = target = original_list[i + 1]).next() The only catch is it raises a StopIteration exception if no element fits the test. In new code you shouldn't call the next() method (renamed __next__() in Python 3) explicitly. Use the next() builtin instead which also allows you to provide a default: exhausted = iter(()) next(exhausted) Traceback (most recent call last): File stdin, line 1, in module StopIteration next(exhausted, default_value) 'default_value' That said, rather than polishing the implementation of the inferior algorithm pick the better one. Ceterum censeo bisect is the way to go here ;) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
Dave Angel wrote: On 03/19/2015 12:20 AM, boB Stepp wrote: I hope extolling the beauty and power of Python on this list is allowed, because I have had a large WOW!!! moment tonight. I had a problem I was working on at work this afternoon. I have a list of ~ 10,000 floating point numbers, which run from largest to smallest. There are duplicates scattered throughout, so I might have something like: [5942.6789, 5942.6789, 5941.03, 5941.01, 5941.01, ... ], etc. I wanted to search the list for a test value, which, depending on the particular list (I can have many such lists, each different from the other.), could conceivably be anywhere within the given list. I needed to return the index where the list values change from being just greater than the test value to just less than the test value at the very next index position. I spent a good chunk of my afternoon writing a binary search function and wondering what theoretically the optimum search algorithm would be, got interrupted (as usual on this project), and decided to look at my books at home to see if a better solution would be staring at me from some book (Like there usually is!). I haven't studied list comprehensions formally yet, but a snippet of code in a book caught my eye where the author was discussing filtering data in a list. This led me to try: The generalized problem: L = [V0, V1, ..., Vn], where V0 = V1 = V2 = ... = Vn . Find index i, such that V[i] = Vt = V[i + 1], where Vt is the test value being searched for. I need to know the indices i and i + 1, which I need to interpolate based on where Vt falls. The solution (As a sublist, S) I worked out tonight after experimenting with comprehension syntax is: S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] And, of course, the index i I need is: i = S[0] I tested this out with concrete examples in the interpreter, such as with a list, L: L = [item for item in range(1, 0, -1)] and trying different test values. It was blazingly fast, too! All I can say is: WOW!!! By the way, if you were to use a plain old loop the expected speedup over the listcomp would be 2. You can break out of the loop when you have found the gap, after iterating over one half of the list on average. So for-loops are twice as amazing ;) That's very innovative. The catch to a list comprehension is it has to visit all the elements, while a binary search would visit log-base-2 of them. So instead of 1 elements, you'd be searching about 14 items. For large lists, it'd probably be much quicker to use the bisect module. https://docs.python.org/3.4/library/bisect.html Check out bisect.bisect_left() and bisect.bisect_right() I don't see how to directly use those functions on a list which is reverse-sorted, but the source is available. On my install, it's located at: /usr/lib/python3.4/bisect.py To back Dave's suggestion with some empirical data here are two ways to make bisect() work with a descending list (though if possible I would recommend that you change your script to use ascending lists). $ cat reverse_bisect2.py import bisect import random def break_listcomp(L, Vt): S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] return S[0] def break_bisect_reverse(L, Vt): L.reverse() result = bisect.bisect(L, Vt) L.reverse() return len(L) - result -1 class Items: def __init__(self, items): self.items = items def __len__(self): return len(self.items) def __getitem__(self, index): return self.items[len(self.items) - 1 - index] def break_bisect_virt(L, Vt): return len(L) - 1 - bisect.bisect(Items(L), Vt) random.seed(42) N = 10**6 data = [random.randrange(N) for i in range(10**5)] data = data + data data.sort(reverse=True) sample = random.sample(data, 10) $ break_bisect_reverse() reverses the list before and after applying bisect. This is still O(N), but the actual work is done in C. break_bisect_virt() wraps the actual list in a class that translates items[0] to items[len(items)-1] items[1] to items[len(items)-2] and so on, thus providing a reversed view of the list without moving any values. This severely slows down access to a single value, but as bisect needs much fewer lookups than the listcomp the overall result is still a massive speedup. The actual timings: $ python3 -m timeit -s 'from reverse_bisect2 import data, sample, break_listcomp as f' '[f(data, v) for v in sample]' 10 loops, best of 3: 781 msec per loop $ python3 -m timeit -s 'from reverse_bisect2 import data, sample, break_bisect_reverse as f' '[f(data, v) for v in sample]' 100 loops, best of 3: 15 msec per loop $ python3 -m timeit -s 'from reverse_bisect2 import data, sample, break_bisect_virt as f' '[f(data, v) for v in sample]' 1000 loops, best of 3: 221 usec per loop So reverse/bisect is 50 times faster than the listcomp, and bisect/virt is 3500 times faster than the listcomp. I expect
Re: [Tutor] List comprehensions to search a list--amazing!
The generalized problem: L = [V0, V1, ..., Vn], where V0 = V1 = V2 = ... = Vn . Find index i, such that V[i] = Vt = V[i + 1], where Vt is the test value being searched for. I need to know the indices i and i + 1, which I need to interpolate based on where Vt falls. The solution (As a sublist, S) I worked out tonight after experimenting with comprehension syntax is: S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] And, of course, the index i I need is: i = S[0] I tested this out with concrete examples in the interpreter, such as with a list, L: L = [item for item in range(1, 0, -1)] and trying different test values. It was blazingly fast, too! All I can say is: WOW!!! By the way, if you were to use a plain old loop the expected speedup over the listcomp would be 2. You can break out of the loop when you have found the gap, after iterating over one half of the list on average. So for-loops are twice as amazing ;) For the same basic speed up, a generator expression with a call to next() will produce similar results. Without the additional code to get the indexed item. index = (i for i, _ in enumerate(original_list) if original_list[i] = target = original_list[i + 1]).next() The only catch is it raises a StopIteration exception if no element fits the test. P ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On 19/03/2015 04:20, boB Stepp wrote: I hope extolling the beauty and power of Python on this list is allowed, because I have had a large WOW!!! moment tonight. I had a problem I was working on at work this afternoon. I have a list of ~ 10,000 floating point numbers, which run from largest to smallest. There are duplicates scattered throughout, so I might have something like: [5942.6789, 5942.6789, 5941.03, 5941.01, 5941.01, ... ], etc. I wanted to search the list for a test value, which, depending on the particular list (I can have many such lists, each different from the other.), could conceivably be anywhere within the given list. I needed to return the index where the list values change from being just greater than the test value to just less than the test value at the very next index position. I spent a good chunk of my afternoon writing a binary search function and wondering what theoretically the optimum search algorithm would be, got interrupted (as usual on this project), and decided to look at my books at home to see if a better solution would be staring at me from some book (Like there usually is!). I haven't studied list comprehensions formally yet, but a snippet of code in a book caught my eye where the author was discussing filtering data in a list. This led me to try: The generalized problem: L = [V0, V1, ..., Vn], where V0 = V1 = V2 = ... = Vn . Find index i, such that V[i] = Vt = V[i + 1], where Vt is the test value being searched for. I need to know the indices i and i + 1, which I need to interpolate based on where Vt falls. The solution (As a sublist, S) I worked out tonight after experimenting with comprehension syntax is: S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] And, of course, the index i I need is: i = S[0] I tested this out with concrete examples in the interpreter, such as with a list, L: L = [item for item in range(1, 0, -1)] and trying different test values. It was blazingly fast, too! All I can say is: WOW!!! How does the performance of your code compare to the bisect module https://docs.python.org/3/library/bisect.html#module-bisect ? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] List comprehensions to search a list--amazing!
I hope extolling the beauty and power of Python on this list is allowed, because I have had a large WOW!!! moment tonight. I had a problem I was working on at work this afternoon. I have a list of ~ 10,000 floating point numbers, which run from largest to smallest. There are duplicates scattered throughout, so I might have something like: [5942.6789, 5942.6789, 5941.03, 5941.01, 5941.01, ... ], etc. I wanted to search the list for a test value, which, depending on the particular list (I can have many such lists, each different from the other.), could conceivably be anywhere within the given list. I needed to return the index where the list values change from being just greater than the test value to just less than the test value at the very next index position. I spent a good chunk of my afternoon writing a binary search function and wondering what theoretically the optimum search algorithm would be, got interrupted (as usual on this project), and decided to look at my books at home to see if a better solution would be staring at me from some book (Like there usually is!). I haven't studied list comprehensions formally yet, but a snippet of code in a book caught my eye where the author was discussing filtering data in a list. This led me to try: The generalized problem: L = [V0, V1, ..., Vn], where V0 = V1 = V2 = ... = Vn . Find index i, such that V[i] = Vt = V[i + 1], where Vt is the test value being searched for. I need to know the indices i and i + 1, which I need to interpolate based on where Vt falls. The solution (As a sublist, S) I worked out tonight after experimenting with comprehension syntax is: S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] And, of course, the index i I need is: i = S[0] I tested this out with concrete examples in the interpreter, such as with a list, L: L = [item for item in range(1, 0, -1)] and trying different test values. It was blazingly fast, too! All I can say is: WOW!!! -- boB ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions to search a list--amazing!
On 03/19/2015 12:20 AM, boB Stepp wrote: I hope extolling the beauty and power of Python on this list is allowed, because I have had a large WOW!!! moment tonight. I had a problem I was working on at work this afternoon. I have a list of ~ 10,000 floating point numbers, which run from largest to smallest. There are duplicates scattered throughout, so I might have something like: [5942.6789, 5942.6789, 5941.03, 5941.01, 5941.01, ... ], etc. I wanted to search the list for a test value, which, depending on the particular list (I can have many such lists, each different from the other.), could conceivably be anywhere within the given list. I needed to return the index where the list values change from being just greater than the test value to just less than the test value at the very next index position. I spent a good chunk of my afternoon writing a binary search function and wondering what theoretically the optimum search algorithm would be, got interrupted (as usual on this project), and decided to look at my books at home to see if a better solution would be staring at me from some book (Like there usually is!). I haven't studied list comprehensions formally yet, but a snippet of code in a book caught my eye where the author was discussing filtering data in a list. This led me to try: The generalized problem: L = [V0, V1, ..., Vn], where V0 = V1 = V2 = ... = Vn . Find index i, such that V[i] = Vt = V[i + 1], where Vt is the test value being searched for. I need to know the indices i and i + 1, which I need to interpolate based on where Vt falls. The solution (As a sublist, S) I worked out tonight after experimenting with comprehension syntax is: S = [i for i, V in enumerate(L) if L[i] = Vt = L[i + 1]] And, of course, the index i I need is: i = S[0] I tested this out with concrete examples in the interpreter, such as with a list, L: L = [item for item in range(1, 0, -1)] and trying different test values. It was blazingly fast, too! All I can say is: WOW!!! That's very innovative. The catch to a list comprehension is it has to visit all the elements, while a binary search would visit log-base-2 of them. So instead of 1 elements, you'd be searching about 14 items. For large lists, it'd probably be much quicker to use the bisect module. https://docs.python.org/3.4/library/bisect.html Check out bisect.bisect_left() and bisect.bisect_right() I don't see how to directly use those functions on a list which is reverse-sorted, but the source is available. On my install, it's located at: /usr/lib/python3.4/bisect.py -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehensions
On Fri, Oct 9, 2009 at 1:21 AM, wesley chun wes...@gmail.com wrote: [generator expressions] are lazy because you iterate over the values one at a time instead of creating the entire list. they are slightly slower but save memory. I don't think you can make a blanket statement comparing speed of list comp vs genexp. For example: kent $ py -m timeit 'sum(x for x in xrange(1,1000) if x%3==0 or x%5==0)' 1000 loops, best of 3: 295 usec per loop kent $ py -m timeit 'sum([x for x in xrange(1,1000) if x%3==0 or x%5==0])' 1000 loops, best of 3: 281 usec per loop Here list comp is a bit faster. kent $ py -m timeit 'sum(x for x in xrange(1,1) if x%3==0 or x%5==0)' 100 loops, best of 3: 2.83 msec per loop kent $ py -m timeit 'sum([x for x in xrange(1,1) if x%3==0 or x%5==0])' 100 loops, best of 3: 2.85 msec per loop Here they are neck and neck. kent $ py -m timeit 'sum(x for x in xrange(1,10) if x%3==0 or x%5==0)' 10 loops, best of 3: 28.9 msec per loop kent $ py -m timeit 'sum([x for x in xrange(1,10) if x%3==0 or x%5==0])' 10 loops, best of 3: 29.4 msec per loop genexp pulls in to the lead. Kent ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] list comprehensions
I've been studying python now for a few weeks and I've recently come into list comprehensions. Some of the examples that I've found make sense, and I find them readable and concise. In particular I'm referring to the python docs on the topic (http://docs.python.org/tutorial/datastructures.html#list-comprehensions). Those make sense to me. The way I understand them is: do something to x for each x in list, with an optional qualifier. On the other hand I've seen a few examples that look similar, but there is no action done to the first x, which confuses me. An example: print sum(x for x in xrange(1,1000) if x%3==0 or x%5==0) In this case is sum() acting as the action on the first x? Can anyone shed some light on what it is I'm not quite grasping between the two? Christer ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehensions
Christer Edwards schrieb: I've been studying python now for a few weeks and I've recently come into list comprehensions. Some of the examples that I've found make sense, and I find them readable and concise. In particular I'm referring to the python docs on the topic (http://docs.python.org/tutorial/datastructures.html#list-comprehensions). Those make sense to me. The way I understand them is: do something to x for each x in list, with an optional qualifier. On the other hand I've seen a few examples that look similar, but there is no action done to the first x, which confuses me. An example: print sum(x for x in xrange(1,1000) if x%3==0 or x%5==0) In this case is sum() acting as the action on the first x? No. (x for x in xrange(1,100) if x%3==0 or x%5==0) generator object genexpr at 0x011C1990 Is a generator expression. If you want to see what it generates, create e.g. a tuple tuple(x for x in xrange(1,100) if x%3==0 or x%5==0) (3, 5, 6, 9, 10, 12, 15, 18, 20, 21, 24, 25, 27, 30, 33, 35, 36, 39, 40, 42, 45, 48, 50, 51, 54, 55, 57, 60, 63, 65, 66, 69, 70, 72, 75, 78, 80, 81, 84, 85, 87, 90, 93, 95, 96, 99) So you see that the generator selects those integers that are divisible by 3 or 5 sum(x for x in xrange(1,100) if x%3==0 or x%5==0) 2318 This calculates the some of those. There are quite a couple of functions in Python that accept generator expressions as arguments Regards, Gregor Can anyone shed some light on what it is I'm not quite grasping between the two? Christer ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehensions
Hello, On Wed, Oct 7, 2009 at 6:57 AM, Christer Edwards christer.edwa...@gmail.com wrote: do something to x for each x in list, with an optional qualifier. To be more precise: http://docs.python.org/tutorial/datastructures.html#list-comprehensions Each list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses On the other hand I've seen a few examples that look similar, but there is no action done to the first x, which confuses me. An example: print sum(x for x in xrange(1,1000) if x%3==0 or x%5==0) So, it's not an 'action' as such, but an expression, x in this case is a variable containing a number, thus stating 'x' by itself is an expression that yields x's value and adds it to the sequence being generated without doing anything else to it. The built-in function sum (on the interpreter, type: help(sum)) takes a sequence of (numeric) values and returns their... sum :-). In this case is sum() acting as the action on the first x? Can anyone shed some light on what it is I'm not quite grasping between the two? Hence, sum does not know about x at all, just about the end result of the generator expression: the generated sequence first evaluated, then passed into it as an argument. -- Kamal -- There is more to life than increasing its speed. -- Mahatma Gandhi ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehensions
I've been studying python now for a few weeks and I've recently come into list comprehensions. [...] Those make sense to me. The way I understand them is: do something to x for each x in list, with an optional qualifier. that's pretty much correct. On the other hand I've seen a few examples that look similar, but there is no action done to the first x, which confuses me. An example: print sum(x for x in xrange(1,1000) if x%3==0 or x%5==0) In this case is sum() acting as the action on the first x? the do something can also include do nothing! in other words, you can simply just select certain elements that meet your particular criteria. for example, let's say you just want all the odd numbers from 0 to 19: odd = [x for x in range(20) if x % 2 != 0] you just wanted the numbers but not to really do anything with them other than have them as elements in a newly-created list. your question also brought up generator expressions which are lazier, more memory-friendly alternative to listcomps. the syntax is nearly identical, with the only difference being that they're enclosed by parentheses instead of square brackets: odd_gen = (x for x in range(20) if x % 2 != 0) that's the only difference syntactically. however, there is a bigger difference under the covers: odd [1, 3, 5, 7, 9, 11, 13, 15, 17, 19] odd_gen generator object genexpr at 0x012F44E0 notice that memory is allocated and the entire list created in memory for the listcomp but just a generic object for the genexp. they are lazy because you iterate over the values one at a time instead of creating the entire list. they are slightly slower but save memory. similar constructs for dicts and sets appear in Python 3: dictionary and set comprehensions. back to your question regarding sum(). sum() just iterates over the genexp and adds up all the individual numbers iterated over. hope this helps! -- wesley - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Core Python Programming, Prentice Hall, (c)2007,2001 Python Fundamentals, Prentice Hall, (c)2009 http://corepython.com wesley.j.chun :: wescpy-at-gmail.com python training and technical consulting cyberweb.consulting : silicon valley, ca http://cyberwebconsulting.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Dinesh B Vadhia [EMAIL PROTECTED] wrote i I'm using a Javascript autocomplete plugin for an online web application/service. Each time a user inputs a character, the character is sent to the backend Python program which searches for the character in a list of 10,000 string items. Eeek! That will be incredibly slow over any kind of network other than fast gigabit! try doing a ping from the client to the server. If it over the internet you will be lucky to get less that 100ms, over a fast LAN it might be around 30ms. Add in the transmission time for your data - (ie 150*ave string length*10/average bandwidth) That network delay will swamp any optimisation of a for loop. It is likely to come out around 10ms so your total delay between each character becomes 40-110ms or a maximum typing rate of 9-25 characters per second. The latter will feel slightly clunky but the former will feel very sluggish. And on anything less than a good internet connection the rate could drop off to around 1 character per second! And anyone using a mobile GPRS connection for access would be crippled. Alan G. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
also if you need to go for 2 results I propose you use filters interactive menus which will help you tailor the query to the users desires thus limit the query results. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
i think you are using ajax ... which undoubdetly uses an sql database since its based on queries run from whithin the application in the browser whithout the need for refreshing the page ... i would suggest you try serching internet for something like google autocomplete feature i guess the queries are also no that long they have a limit of the results ... for example not more than 20 results per query. than said there would be no loop. just a query (with a limit of 20 rersults) each time a use inputs a character. hope this helps. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
My guess, though I'm not sure, is that google uses hashes... why? Because they're a *ahem* load faster than loops, and the reason is they replace the repetitive nature of a loop by using some type of formula. Exactly /how/ this is implemented, I'm not sure. A simple example of the speed difference: 11 * 1 = 11 The program spent 0.000810146331787 seconds. 11 * 1 = 11 The program spent 6.19888305664e-05 seconds. (btw, that means .06... ) The difference? The first was a loop: 1 from time import time 2 start_time = time() 3 4 x = 0 5 while x 11: 6 x +=1 7 8 print 11 * 1 = , x 9 10 end_time = time() 11 print The program spent, end_time - start_time, seconds. And the second, a algebraic statement 14 start_time = time() 15 16 x = 11 / 1 17 print 11 * 1 = , x 18 19 end_time = time() 20 print The program spent, end_time - start_time, seconds. It would be simple to replace the 11 with a variable supplied by something like variable = int(raw_input(Enter a number: )) and you would come out with similar output. That's basically the reason a dictionary finds dictionary[foo] faster than a for loop: the key, foo, is transformed into some value (As I understand it, the hashtable refers to some memory location, i.e 0x08f or some such), and there, sitting in that location, is the value for the key foo. so rather than comparing each value a list, it would be like having some formula to grab that value. I hope this wasn't too confusing, and if anyone has any corrections or clarifications, feel free to muck about. But yeah, a hash is probably the way you want to go (from what I know) -Wayne On Thu, Apr 10, 2008 at 5:03 AM, linuxian iandsd [EMAIL PROTECTED] wrote: also if you need to go for 2 results I propose you use filters interactive menus which will help you tailor the query to the users desires thus limit the query results. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor -- To be considered stupid and to be told so is more painful than being called gluttonous, mendacious, violent, lascivious, lazy, cowardly: every weakness, every vice, has found its defenders, its rhetoric, its ennoblement and exaltation, but stupidity hasn't. - Primo Levi ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Dinesh B Vadhia wrote: Kent I'm using a Javascript autocomplete plugin for an online web application/service. Each time a user inputs a character, the character is sent to the backend Python program which searches for the character in a list of 10,000 string items. Once it finds the character, the backend will return that string and N other adjacent string items where N can vary from 20 to 150. Each string item is sent back to the JS in separate print statements. Hence, the for loop. Ok, this sounds a little closer to a real spec. What kind of search are you doing? Do you really just search for individual characters or are you looking for the entire string entered so far as a prefix? Is the list of 10,000 items sorted? Can it be? You need to look at your real problem and find an appropriate data structure, rather than showing us what you think is the solution and asking how to make it faster. For example, if what you have a sorted list of strings and you want to find the first string that starts with a given prefix and return the N adjacent strings, you could use the bisect module to do a binary search rather than a linear search. Binary search of 10,000 items will take 13-14 comparisons to find the correct location. Your linear search will take an average of 5,000 comparisons. You might also want to use a trie structure though I'm not sure if that will let you find adjacent items. http://www.cs.mcgill.ca/~cs251/OldCourses/1997/topic7/ http://jtauber.com/blog/2005/02/10/updated_python_trie_implementation/ I haven't done any profiling yet as we are still building the system but it seemed sensible that replacing the for loop with a built-in would help. Maybe not? Not. An algorithm with poor big O performance should be *replaced*, not optimized. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Kent Johnson [EMAIL PROTECTED] wrote application/service. Each time a user inputs a character, the character is sent to the backend Python program which searches for the character in a list of 10,000 string items. Once it finds the character, the backend will return that string and N other adjacent string items where N can vary from 20 to 150. Each string item is sent back to the JS in separate print statements. Hence, the for loop. You need to look at your real problem and find an appropriate data structure, rather than showing us what you think is the solution and asking how to make it faster. One possibility is that the javascript fetches the list back on the first few characters and caches it on the browser, it can then do the search locally and only go back to the server if the user deletes enough characters to invalidate the cache. That would make a big difference to the overall speed by eliminating several network lookups. I am assuming the server lookup list does not change significantly over the duration of a form submission? Alan G ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Alan Gauld wrote: One possibility is that the javascript fetches the list back on the first few characters and caches it on the browser Here is an autocomplete widget I use that can do exactly that: http://www.dyve.net/jquery/?autocomplete Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Dinesh B Vadhia wrote: Kent I'm using a Javascript autocomplete plugin for an online web application/service. Each time a user inputs a character, the character is sent to the backend Python program which searches for the character in a list of 10,000 string items. Once it finds the character, the backend will return that string and N other adjacent string items where N can vary from 20 to 150. So if I had these strings ape, bee, cat, dog, eel, fly, gnu, hex, imp, jut, kit, lox and N were 2 and the user entered g the program finds dog and sends back cat, dog, eel. OK so far or not? The user then enters y, the program finds fly and sends back eel, fly, gnu. OK so far or not? The user then enters x, the program finds no match and sends back what?? Each string item is sent back to the JS in separate print statements. IIRC you don't need separate print statements. Just put \n between the strings and print one big string. Hence, the for loop. Now, N = 20 to 150 is not a lot (for a for loop) but this process is performed each time the user enters a character. Plus, there will be thousands (possibly more) users at a time. There is also the searching of the 10,000 string items using the entered character. All of this adds up in terms of performance. I haven't done any profiling yet as we are still building the system but it seemed sensible that replacing the for loop with a built-in would help. Maybe not? Hope that helps. Dinesh - Original Message - *From:* Kent Johnson mailto:[EMAIL PROTECTED] *To:* Dinesh B Vadhia mailto:[EMAIL PROTECTED] *Cc:* tutor@python.org mailto:tutor@python.org *Sent:* Wednesday, April 09, 2008 1:48 PM *Subject:* Re: [Tutor] List comprehensions Dinesh B Vadhia wrote: Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = some operation on item print result I want to replace the for loop with another structure to improve performance (as the data list will contain 10,000 string items]. At each iteration of the for loop the result is printed (in fact, the result is sent from the server to a browser one result line at a time) Any savings you have from optimizing this loop will be completely swamped by the network time. Why do you think this is a bottleneck? You could use [ sys.stdout.write(some operation on item) for item in data ] but I consider this bad style and I seriously doubt you will see any difference in performance. The for loop will be called continuously and this is another reason to look for a potentially better structure preferably a built-in. What do you mean 'called continuously'? Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor -- Bob Gailer 919-636-4239 Chapel Hill, NC ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] List comprehensions
Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = item + \n print result I want to replace the for loop with a List Comrehension (or whatever) to improve performance (as the data list will be 10,000]. At each stage of the for loop I want to print the result ie. [print (item + \n) for item in data] But, this doesn't work as the inclusion of the print causes an invalid syntax error. Any thoughts? Dinesh ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
On Wed, Apr 9, 2008 at 7:12 AM, Dinesh B Vadhia [EMAIL PROTECTED] wrote: I want to replace the for loop with a List Comrehension (or whatever) to improve performance (as the data list will be 10,000]. At each stage of the for loop I want to print the result ie. List comprehensions are for building lists, not consuming them. If you want to do something with every element in a list, other than building a new list with it, you should be using a for loop. [print (item + \n) for item in data] But, this doesn't work as the inclusion of the print causes an invalid syntax error. Any thoughts? Don't do this. Perhaps you could share a small piece of code that you think is too slow, and ask for advice in speeding it up? If you're not sure which small pieces of code are too slow, you need to profile your application to find out. See the documentation for python's profile module. If you don't have enough code written to profile, then it's probably too early to be doing these optimizations. -- Jerry ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
On Wed, Apr 9, 2008 at 7:44 PM, Jerry Hill [EMAIL PROTECTED] wrote: On Wed, Apr 9, 2008 at 7:12 AM, Dinesh B Vadhia [EMAIL PROTECTED] wrote: I want to replace the for loop with a List Comrehension (or whatever) to improve performance (as the data list will be 10,000]. At each stage of the for loop I want to print the result ie. List comprehensions are for building lists, not consuming them. If you want to do something with every element in a list, other than building a new list with it, you should be using a for loop. [print (item + \n) for item in data] But, this doesn't work as the inclusion of the print causes an invalid syntax error. Any thoughts? Don't do this. Perhaps you could share a small piece of code that you think is too slow, and ask for advice in speeding it up? If you're not sure which small pieces of code are too slow, you need to profile your application to find out. See the documentation for python's profile module. If you don't have enough code written to profile, then it's probably too early to be doing these optimizations. -- Jerry ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor if you explain the source of the list, what do you want to change in it, what destination will it take, i m sure the guys here will help a lot. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Sorry, let's start again. Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = some operation on item print result I want to replace the for loop with another structure to improve performance (as the data list will contain 10,000 string items]. At each iteration of the for loop the result is printed (in fact, the result is sent from the server to a browser one result line at a time) The for loop will be called continuously and this is another reason to look for a potentially better structure preferably a built-in. Hope this makes sense! Thank-you. Dinesh - Original Message - From: Kent Johnson To: Dinesh B Vadhia Cc: tutor@python.org Sent: Wednesday, April 09, 2008 12:40 PM Subject: Re: [Tutor] List comprehensions Dinesh B Vadhia wrote: Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = item + \n print result I'm not sure what your goal is here. Do you mean to be accumulating all the values in data into result? Your sample code does not do that. I want to replace the for loop with a List Comrehension (or whatever) to improve performance (as the data list will be 10,000]. At each stage of the for loop I want to print the result ie. [print (item + \n) for item in data] But, this doesn't work as the inclusion of the print causes an invalid syntax error. You can't include a statement in a list comprehension. Anyway the time taken to print will swamp any advantage you get from the list comp. If you just want to print the items, a simple loop will do it: for item in data: print item + '\n' Note this will double-space the output since print already adds a newline. If you want to create a string with all the items with following newlines, the classic way to do this is to build a list and then join it. To do it with the print included, try result = [] for item in data: newItem = item + '\n' print newItem result.append(newItem) result = ''.join(result) Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
i can't think of anything but a loop here UNLESS you take the list from its source one element at a time, process it then print the result. example of this would be : list comes in from standard input. list comes from a database list is read from a file. so again where the list comes from is important. if its standard input then you program will be easy won't use any memory i guess. import sys # cgi cgitb if going web data = sys.stdin.readline() do what ever you like with data print data On Wed, Apr 9, 2008 at 8:15 PM, Dinesh B Vadhia [EMAIL PROTECTED] wrote: Sorry, let's start again. Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = some operation on item print result I want to replace the for loop with another structure to improve performance (as the data list will contain 10,000 string items]. At each iteration of the for loop the result is printed (in fact, the result is sent from the server to a browser one result line at a time) The for loop will be called continuously and this is another reason to look for a potentially better structure preferably a built-in. Hope this makes sense! Thank-you. Dinesh - Original Message - *From:* Kent Johnson [EMAIL PROTECTED] *To:* Dinesh B Vadhia [EMAIL PROTECTED] *Cc:* tutor@python.org *Sent:* Wednesday, April 09, 2008 12:40 PM *Subject:* Re: [Tutor] List comprehensions Dinesh B Vadhia wrote: Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = item + \n print result I'm not sure what your goal is here. Do you mean to be accumulating all the values in data into result? Your sample code does not do that. I want to replace the for loop with a List Comrehension (or whatever) to improve performance (as the data list will be 10,000]. At each stage of the for loop I want to print the result ie. [print (item + \n) for item in data] But, this doesn't work as the inclusion of the print causes an invalid syntax error. You can't include a statement in a list comprehension. Anyway the time taken to print will swamp any advantage you get from the list comp. If you just want to print the items, a simple loop will do it: for item in data: print item + '\n' Note this will double-space the output since print already adds a newline. If you want to create a string with all the items with following newlines, the classic way to do this is to build a list and then join it. To do it with the print included, try result = [] for item in data: newItem = item + '\n' print newItem result.append(newItem) result = ''.join(result) Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Dinesh B Vadhia wrote: Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = item + \n print result I'm not sure what your goal is here. Do you mean to be accumulating all the values in data into result? Your sample code does not do that. I want to replace the for loop with a List Comrehension (or whatever) to improve performance (as the data list will be 10,000]. At each stage of the for loop I want to print the result ie. [print (item + \n) for item in data] But, this doesn't work as the inclusion of the print causes an invalid syntax error. You can't include a statement in a list comprehension. Anyway the time taken to print will swamp any advantage you get from the list comp. If you just want to print the items, a simple loop will do it: for item in data: print item + '\n' Note this will double-space the output since print already adds a newline. If you want to create a string with all the items with following newlines, the classic way to do this is to build a list and then join it. To do it with the print included, try result = [] for item in data: newItem = item + '\n' print newItem result.append(newItem) result = ''.join(result) Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Dinesh B Vadhia wrote: Here is a for loop operating on a list of string items: data = [string 1, string 2, string 3, string 4, string 5, string 6, string 7, string 8, string 9, string 10, string 11] result = for item in data: result = some operation on item print result I want to replace the for loop with another structure to improve performance (as the data list will contain 10,000 string items]. At each iteration of the for loop the result is printed (in fact, the result is sent from the server to a browser one result line at a time) Any savings you have from optimizing this loop will be completely swamped by the network time. Why do you think this is a bottleneck? You could use [ sys.stdout.write(some operation on item) for item in data ] but I consider this bad style and I seriously doubt you will see any difference in performance. The for loop will be called continuously and this is another reason to look for a potentially better structure preferably a built-in. What do you mean 'called continuously'? Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
On Wed, Apr 9, 2008 at 4:15 PM, Dinesh B Vadhia [EMAIL PROTECTED] wrote: Sorry, let's start again. This version really isn't any more helpful than the first one. I know you corrected the sample code, but you haven't addressed any of the fundamental questions that Kent or I asked. I want to replace the for loop with another structure to improve performance (as the data list will contain 10,000 string items]. At each iteration of the for loop the result is printed (in fact, the result is sent from the server to a browser one result line at a time) Are you looking for a different data structure to hold your list of strings, or are you looking for a replacement for the for loop? How long does it take to generate your list? How long does each iteration of your for loop take? What are acceptable times for each of these? You need to know these things before you can seriously optimize anything. If building up the list takes a long time and you only use it once, consider using a generator instead of building the whole list and then processing it. This spreads out the time to create the list and operates on each piece of data as soon as it's available. I don't think you're going to find a replacement for a for loop that is inherently faster. You keep talking about list comprehensions, but I don't see how a list comprehension is even appropriate for the problem you're describing. The for loop will be called continuously and this is another reason to look for a potentially better structure preferably a built-in. Again, it would be helpful to discuss actual bits of code, so we can see if there are places you can gain some performance. It's hard to optimize psuedocode, because there are sometimes very minor changes you can make which affect performance quite a bit. For instance, attribute lookup in python is relatively slow. If you can hoist any attribute lookups out of your loop, you will get some performance increases. Also, you should mention what version of python you're using and what platform it's running on. -- Jerry ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
On Wed, Apr 9, 2008 at 4:48 PM, Kent Johnson [EMAIL PROTECTED] wrote: You could use [ sys.stdout.write(some operation on item) for item in data ] but I consider this bad style and I seriously doubt you will see any difference in performance. This really isn't a good idea. It will take just as long as the for loop, plus it builds a list with the return code for each call to sys.stdout.write() call (which is probably None). Then after building the list with 10,000 entries of None, it throws it away. That can't be good for performance. -- Jerry ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] List comprehensions
Hi, Is there a way to apply multiple actions within one list comprehension? i.e. instead of a = [] for i in x: i.pop(3) g = [ int(item) for item in i] a.append(g) Is there any guides to this (possibly obtuse) tool? Regards, Liam Clarke PS I can see how nested list comprehensions can quickly lose readability. -- 'There is only one basic human right, and that is to do as you damn well please. And with it comes the only basic human duty, to take the consequences. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
RE: [Tutor] List comprehensions
I'm not sure if you can or want to do this solely one list comprehension. You could make a helper function and use map: ### def helper(i): i.pop(3) return map(int, i) a = map(helper, x) ### Description of map: http://docs.python.org/lib/built-in-funcs.html I think map is a little cleaner is some cases. Not sure if its more Pythonic, I'm still trying to figure out exactly what that means. The need to pop(3) off the list makes a pure functional solution kinda hard. Thanks, Ryan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Liam Clarke Sent: Wednesday, March 23, 2005 5:44 AM To: Tutor Tutor Subject: [Tutor] List comprehensions Hi, Is there a way to apply multiple actions within one list comprehension? i.e. instead of a = [] for i in x: i.pop(3) g = [ int(item) for item in i] a.append(g) Is there any guides to this (possibly obtuse) tool? Regards, Liam Clarke PS I can see how nested list comprehensions can quickly lose readability. -- 'There is only one basic human right, and that is to do as you damn well please. And with it comes the only basic human duty, to take the consequences. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Liam Clarke wrote: Is there any guides to this (possibly obtuse) tool? Think of it this way. A list comprehension generates a new list. So, you should think about list comps whenever you have old_list - new_list style behavior. There are two advantages to list comps over map: 1) a list comp can use a predicate to filter results [ x for x in mylist if is Prime(x) ] for instance. that would be map(lambda x: x, filter(isPrime, mylist)) which is a little more dense. 2) usually there is no need for a lambda. Not that there is anything wrong with lambdas mind you (-: ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Liam Clarke wrote: Is there any guides to this (possibly obtuse) tool? http://docs.python.org/tut/node7.html#SECTION00714 http://www.amk.ca/python/2.0/index.html#SECTION00060 Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Ryan Davis wrote: I think map is a little cleaner is some cases. Not sure if its more Pythonic, I'm still trying to figure out exactly what that means. map is (probably) going to be removed in Python3000 :-( So it's probably better to not get into the habit of using it. -- John. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
On Jan 13, 2005, at 04:13, Bob Gailer wrote: I like Kent's response. foobar(item)/0 is a valid expression. It fits the grammar of expressions. The fact that it raises an exception does not make it an invalid expression. Consider foobar(item)/xyz. It is valid. If xyz == 0 then it will also raise an exception. You have a point, I hadn't thought of it that way. -- Max maxnoel_fr at yahoo dot fr -- ICQ #85274019 Look at you hacker... A pathetic creature of meat and bone, panting and sweating as you run through my corridors... How can you challenge a perfect, immortal machine? ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Blake Winton wrote: Kent Johnson wrote: If you mean for j to be a list of foobar(item) then use j=[foobar(item) for item in x] The first part of the list comp can be any valid expression. Does that mean that there are invalid expressions? I'd enjoy seeing an example. I suppose if it's an expression, it must be valid, eh? Otherwise it's something else. I don't think I entirely agree... What about x === item It's certainly not a statement, and I would wager that Python was in the middle of its expression parsing code when it threw the SyntaxError. (Or how about x == item =?) Perhaps someone more in touch with the compiler internals will chip in here. We're talking about angels and pins, here, but I would say that the only meaningful interpretation of 'x is an expression' is 'x is parseable according to the syntax rules for an expression'. So if it doesn't parse it's not an expression. How can you say x === item is not a statement? Because it doesn't parse as a statement. By the same logic it isn't an expression either. Kent It is an interesting philosophical question, though. Later, Blake. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Quoting Liam Clarke [EMAIL PROTECTED]: As a comprehension, if you get what I mean... Can I build a for loop with a function in for x in x part? Ack. Having difficulty summing it up properly. I'm not sure exactly what you mean, so my answer is probably. from math import sin arr = range(10) arr [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [sin(float(x)/10) for x in arr] [0.0, 0.099833416646828155, 0.19866933079506122, 0.29552020666133955, 0.38941834230865052, 0.47942553860420301, 0.56464247339503537, 0.64421768723769102, 0.71735609089952279, 0.78332690962748341] def biggerThanHalf(x): ... return x 0.5 ... [biggerThanHalf(sin(float(x)/10)) for x in arr] [False, False, False, False, False, False, True, True, True, True] [biggerThanHalf(sin(float(x)/10)) for x in map(lambda y: 2*y + 3, arr)] [False, False, True, True, True, True, True, True, True, True] Hopefully one of these examples answers your question :-) -- John. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
If you mean for j to be a list of foobar(item) then use j=[foobar(item) for item in x] The first part of the list comp can be any valid expression. Kent Liam Clarke wrote: Hi, Am I able to achieve something like this - def foobar(); # stuff return x=[1,1000] for j=foobar(item) for item in x: As a comprehension, if you get what I mean... Can I build a for loop with a function in for x in x part? Ack. Having difficulty summing it up properly. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
Aah, thank you both - I knew there was a way to do it. Regards, Liam Clarke PS John - another Kiwi, seems to be a lot of NZers involved in Python Python projects. Is it the No. 8 wire freedom of Python? ; ) On Wed, 12 Jan 2005 18:48:39 -0500, Kent Johnson [EMAIL PROTECTED] wrote: If you mean for j to be a list of foobar(item) then use j=[foobar(item) for item in x] The first part of the list comp can be any valid expression. Kent Liam Clarke wrote: Hi, Am I able to achieve something like this - def foobar(); # stuff return x=[1,1000] for j=foobar(item) for item in x: As a comprehension, if you get what I mean... Can I build a for loop with a function in for x in x part? Ack. Having difficulty summing it up properly. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor -- 'There is only one basic human right, and that is to do as you damn well please. And with it comes the only basic human duty, to take the consequences. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
At 04:48 PM 1/12/2005, Kent Johnson wrote: If you mean for j to be a list of foobar(item) then use j=[foobar(item) for item in x] The first part of the list comp can be any valid expression. Does that mean that there are invalid expressions? I'd enjoy seeing an example. Bob Gailer mailto:[EMAIL PROTECTED] 303 442 2625 home 720 938 2625 cell ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] List comprehensions
I suppose if it's an expression, it must be valid, eh? Otherwise it's something else. Bob Gailer wrote: At 04:48 PM 1/12/2005, Kent Johnson wrote: If you mean for j to be a list of foobar(item) then use j=[foobar(item) for item in x] The first part of the list comp can be any valid expression. Does that mean that there are invalid expressions? I'd enjoy seeing an example. Bob Gailer mailto:[EMAIL PROTECTED] 303 442 2625 home 720 938 2625 cell ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
