Re: [Tutor] fast list traversal

2008-10-31 Thread Kent Johnson
On Fri, Oct 31, 2008 at 8:34 AM, Dinesh B Vadhia [EMAIL PROTECTED] wrote: Hi Kent The code is very simple: dict_long_lists = defaultdict(list) for long_list in dict_long_lists.itervalues() for element in long_list: array_a[element] = m + n + p# m,n,p are

[Tutor] fast list traversal

2008-10-30 Thread Dinesh B Vadhia
I need to process a large number ( 20,000) of long and variable length lists ( 5,000 elements) ie. for element in long_list: do something with element# the result of this operation is not a list The performance is reasonable but I wonder if there are faster Python methods? Dinesh

Re: [Tutor] fast list traversal

2008-10-30 Thread bob gailer
Dinesh B Vadhia wrote: I need to process a large number ( 20,000) of long and variable length lists ( 5,000 elements) ie. for element in long_list: do something with element# the result of this operation is not a list The performance is reasonable but I wonder if there are

Re: [Tutor] fast list traversal

2008-10-30 Thread Shawn Milochik
On Thu, Oct 30, 2008 at 2:40 PM, bob gailer [EMAIL PROTECTED] wrote: Dinesh B Vadhia wrote: I need to process a large number ( 20,000) of long and variable length lists ( 5,000 elements) ie. for element in long_list: do something with element# the result of this operation is not

Re: [Tutor] fast list traversal

2008-10-30 Thread Dinesh B Vadhia
Bob: Nothing special is being done on the elements of the list - additions/subtractions/ - and storing the results in an array. That's it. Dinesh From: bob gailer Sent: Thursday, October 30, 2008 11:40 AM To: Dinesh B Vadhia Cc: tutor@python.org Subject: Re: [Tutor] fast list traversal

Re: [Tutor] fast list traversal

2008-10-30 Thread Kent Johnson
On Thu, Oct 30, 2008 at 2:36 PM, Dinesh B Vadhia [EMAIL PROTECTED] wrote: I need to process a large number ( 20,000) of long and variable length lists ( 5,000 elements) ie. for element in long_list: do something with element# the result of this operation is not a list The

Re: [Tutor] fast list traversal

2008-10-30 Thread Kent Johnson
On Thu, Oct 30, 2008 at 2:46 PM, Shawn Milochik [EMAIL PROTECTED] wrote: You might try using dictionaries instead. I've had phenomenal speed gains by switching lists to dictionaries before, although that may have had more to do with the fact that I needed to access certain values, rather than

Re: [Tutor] fast list traversal

2008-10-30 Thread Shawn Milochik
On Thu, Oct 30, 2008 at 4:05 PM, Kent Johnson [EMAIL PROTECTED] wrote: On Thu, Oct 30, 2008 at 2:46 PM, Shawn Milochik [EMAIL PROTECTED] wrote: You might try using dictionaries instead. I've had phenomenal speed gains by switching lists to dictionaries before, although that may have had more

Re: [Tutor] fast list traversal

2008-10-30 Thread wesley chun
based on the all the performance questions, i would say agree that dictionary access is faster than lists (hashes exist cuz they're fast) but they use up more memory, as shown in shawn's numbers. also, one of the reasons why slots was added to classes was because the attribute dictionary began to

Re: [Tutor] fast list traversal

2008-10-30 Thread Shawn Milochik
On Thu, Oct 30, 2008 at 6:06 PM, wesley chun [EMAIL PROTECTED] wrote: based on the all the performance questions, i would say agree that dictionary access is faster than lists (hashes exist cuz they're fast) but they use up more memory, as shown in shawn's numbers. also, one of the reasons why

Re: [Tutor] fast list traversal

2008-10-30 Thread Dinesh B Vadhia
with a for-loop. Btw, cannot move to Python 2.6 or 3.0 until Numpy/Scipy catches up. Dinesh From: wesley chun Sent: Thursday, October 30, 2008 3:06 PM To: Dinesh B Vadhia Cc: tutor@python.org Subject: Re: [Tutor] fast list traversal based on the all the performance questions, i would say agree

Re: [Tutor] fast list traversal

2008-10-30 Thread wesley chun
They seem pretty similar. Here are two tests (code follows). Perhaps I could have loaded them differently and it would have made more of a difference. In this case I just made a list and populated the sets from it. Results for 999 iterations: Set: Load: 1.24

Re: [Tutor] fast list traversal

2008-10-30 Thread Eike Welk
Hello Dinesh! On Thursday 30 October 2008, Dinesh B Vadhia wrote: Bob: Nothing special is being done on the elements of the list - additions/subtractions/ - and storing the results in an array. That's it. You could convert the list into a numpy array first, and you could try to express the

Re: [Tutor] fast list traversal

2008-10-30 Thread Kent Johnson
On Thu, Oct 30, 2008 at 4:55 PM, Shawn Milochik [EMAIL PROTECTED] wrote: I just ran a very crude test. Results: Lists load more quickly but iterate more slowly. Dictionaries take longer to load but iteration takes about half the time. Here are my results using timeit and Python 2.6: