Efficiently iterating over part of a list
If I want to iterate over part of the list, the normal Python idiom is to do something like this: alist = range(50) # first item is special x = alist[0] # iterate over the rest of the list for item in alist[1:] x = item The important thing to notice is that alist[1:] makes a copy. What if the list has millions of items and duplicating it is expensive? What do people do in that case? Are there better or more Pythonic alternatives to this obvious C-like idiom? for i in range(1, len(alist)): x = alist[i] -- Steven D'Aprano -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficiently iterating over part of a list
Steven D'Aprano wrote: If I want to iterate over part of the list, the normal Python idiom is to do something like this: alist = range(50) # first item is special x = alist[0] # iterate over the rest of the list for item in alist[1:] x = item The important thing to notice is that alist[1:] makes a copy. What if the list has millions of items and duplicating it is expensive? What do people do in that case? Are there better or more Pythonic alternatives to this obvious C-like idiom? for i in range(1, len(alist)): x = alist[i] I think this is a job for iterators: listiter = iter(alist) first_item_is_special = listiter.next() for not_special_item in listiter: do_stuff_with(not_special_item) Other solutions might involve enumerators: special = [i for i in xrange(50) if not i%13] for i,item in alist: if i in special: do_something_special_with(item) else: do_other_stuff_with(item) James James -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficiently iterating over part of a list
James Stroud wrote: Steven D'Aprano wrote: If I want to iterate over part of the list, the normal Python idiom is to do something like this: alist = range(50) # first item is special x = alist[0] # iterate over the rest of the list for item in alist[1:] x = item The important thing to notice is that alist[1:] makes a copy. What if the list has millions of items and duplicating it is expensive? What do people do in that case? Are there better or more Pythonic alternatives to this obvious C-like idiom? for i in range(1, len(alist)): x = alist[i] I think this is a job for iterators: listiter = iter(alist) first_item_is_special = listiter.next() for not_special_item in listiter: do_stuff_with(not_special_item) Other solutions might involve enumerators: special = [i for i in xrange(50) if not i%13] for i,item in alist: if i in special: do_something_special_with(item) else: do_other_stuff_with(item) James James I mean for i,item in enumerate(alist): -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficiently iterating over part of a list
Steven D'Aprano wrote: [snip] The important thing to notice is that alist[1:] makes a copy. What if the list has millions of items and duplicating it is expensive? What do people do in that case? Are there better or more Pythonic alternatives to this obvious C-like idiom? for i in range(1, len(alist)): x = alist[i] for x in itertools.islice(alist, 1, len(alist)): HTH Ziga -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficiently iterating over part of a list
Steven D'Aprano wrote: Are there better or more Pythonic alternatives to this obvious C-like idiom? for i in range(1, len(alist)): x = alist[i] For small start values you can use itertools.islice(), e. g: for x in islice(alist, 1, None): # use x You'd have to time at what point the C-like idiom (which I would have no qualms using throughout) becomes faster. Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficiently iterating over part of a list
Steven D'Aprano [EMAIL PROTECTED] wrote: The important thing to notice is that alist[1:] makes a copy. What if the list has millions of items and duplicating it is expensive? What do people do in that case? I think you are worrying prematurely. On my system slicing one element off the front of a 10,000,000 element list takes 440mS. The same operation on 1,000,000 elements taks 41mS. Iterating through the sliced list: for x in r[1:]: y = x+1 takes 1.8s and 157mS respectively, so the slicing is only a quarter of the time for even this minimal loop. As soon as you do anything much inside the loop you can forget the slice cost. Remember that copying the list never copies the elements in the list, it just copies pointers and bumps ref counts. Copying a list even if it has millions of items is not usually expensive compared with the costs of manipulating all the items in the list. So the first thing you do is not to worry about this until you know it is an issue. Once you know for a fact that it is a problem, then you can look at optimising it with fancy lazy slicing techniques, but not before. -- http://mail.python.org/mailman/listinfo/python-list
Re: Efficiently iterating over part of a list
Steven D'Aprano [EMAIL PROTECTED] writes: for i in range(1, len(alist)): x = alist[i] a2 = iter(alist) a2.next() # throw away first element for x in a2: ... -- http://mail.python.org/mailman/listinfo/python-list
