Re: [Tutor] Alternatives to append() for growing a list
On Sun, Dec 08, 2013 at 04:10:45PM +1000, Amit Saha wrote: It didn't have to do with strings. It was a basic example of using append() which is to start with an empty list and and then build it incrementally: l = [ ] l.append(1) # append more If this is literally what the code does, then it's fat and slow and should be replaced with this: # not this l = [] l.append(1) l.append(2) l.append(x) l.append(y) # this is even worse, despite being shorter l = [] for item in [1, 2, x, y]: l.append(item) # this is the way to do it l = [1, 2, x, y] But without seeing the specific code in question, it is impossible to judge whether you are using append appropriately or not. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On 12/08/2013 07:14 AM, Amit Saha wrote: If all of the bitN are strings: bit1 + bit2 + bit3 + bit4 + bit5 actually constructs: bit1+bit2 bit1+bit2+bit3 bit1+bit2+bit3+bit4 bit1+bit2+bit3+bit4+bit5 A number of unneeded string object, and a very big useless memory weight. Thanks Denis, good to know this is how it is done. I hadn't thought about it simply because, never have probably concatenated strings beyond the simple adding a new line to a string. This is only due to catenation being defined in python as a _binary_ operation. In principle, there is no reason for this. In a hypothetical language (I will here use Lisp-like syntax just to make a visual difference), one may write: (cat bit1 bit2 bit3 bit4 bit5) Then cat would catenate all bits in one go, into a sigle total string (sum the sizes and make a single whole string of this size). As a side-note, this is also true for common arthmetic operators like + or *, which are not conceptually binary, but we have this impression *due to infix notation*: 1 + 2 + 3 + 4 + 5 is usually interpreted as a series of binary operations: 1 + 2) + 3) + 4) + 5) but there could well be (and in some languages this is the case): (+ 1 2 3 4 5) in one go (even if behind the stage there are binary ops, just because the machine also knows that). Think at manual sum for illustration: 123 456 789 --- result Maybe infix notation is just wrong for some operations and misleads into wrong thinking. We should reserve for actual binary ops, namely - and /; but prefix notation, as illustrated above, would nicely do the job for binary pos as well. (The same point applies to logical operators or and, which are not binary, while the case of relational ones [comparisons] is more obscure.) However, unlike the case of catenation, unjustified binary arithmetic operations do not cause any other harm than needlessly multiplying function or method lookups calls (if there is a call, which is the case in python because such operators can be overloaded). Denis ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On Tue, Dec 3, 2013 at 6:32 AM, Danny Yoo d...@hashcollision.org wrote: I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: I think this thread demonstrates: we don't need an excuse to talk your ears off. :P hehe, indeed. Using append() is fine. If anything, the comment might be referring to an issue with appending strings in a naive way. But without further information, can't say for sure. If you can get more information about what your friend was talking about, that would be helpful. It didn't have to do with strings. It was a basic example of using append() which is to start with an empty list and and then build it incrementally: l = [ ] l.append(1) # append more But yeah, I think that bit of talking my ears off is now no more valid and I have written a good rebuttal to the comment :) Best, Amit. -- http://echorand.me ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On Sun, Dec 1, 2013 at 7:27 PM, spir denis.s...@gmail.com wrote: On 12/01/2013 05:32 AM, Amit Saha wrote: Hello, I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: mylist = [] mylist.append(1) # a number of times over I have some ideas that on an append() the list's internal size increases (doubled?) since CPython over-allocates memory, and such. So, the question is: what is the alternative to growing a list when I have no idea what items or how many may be there? Maybe you are confusing with catenating _strings_, rather than lists. Python's concat is problematic because it is a binary operation. So, catenating n bits makes n-1 operations, each with intermediate results, of growing sizes, all thrown away except for the last one, the actual result. If all of the bitN are strings: bit1 + bit2 + bit3 + bit4 + bit5 actually constructs: bit1+bit2 bit1+bit2+bit3 bit1+bit2+bit3+bit4 bit1+bit2+bit3+bit4+bit5 A number of unneeded string object, and a very big useless memory weight. Thanks Denis, good to know this is how it is done. I hadn't thought about it simply because, never have probably concatenated strings beyond the simple adding a new line to a string. Best, Amit. -- http://echorand.me ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
It didn't have to do with strings. It was a basic example of using append() which is to start with an empty list and and then build it incrementally: l = [ ] l.append(1) # append more Hi Amit, Ok, good. This context helps! If you do know all the values of the list up front, then defining 'f' with those values as part of the list literal is idiomatic: l = [1, ## fill me in... ] and in this way, we probably wouldn't use append() for this situation. The reason for this can be based on readability arguments: a programmer who sees this will be more inclined to know that the list won't change. Symmetrically, the presence of 'l.append()' in a program is often a hint a reader to anticipate the need for the list to have some dynamic, runtime-dependent size. Good luck! ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: I think this thread demonstrates: we don't need an excuse to talk your ears off. :P Using append() is fine. If anything, the comment might be referring to an issue with appending strings in a naive way. But without further information, can't say for sure. If you can get more information about what your friend was talking about, that would be helpful. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On Sun, Dec 1, 2013 at 5:04 PM, Steven D'Aprano st...@pearwood.info wrote: On Sun, Dec 01, 2013 at 02:32:38PM +1000, Amit Saha wrote: Hello, I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: There is absolutely nothing wrong with append. Either you have misunderstood, or the person who told you this was smoking crack. heh, no I literally quote the person above, so I think I understood him alright. mylist = [] mylist.append(1) # a number of times over However, growing a list one item at a time using append like this is too much hard work. Some better solutions: # You have an existing list, and want to append a bunch of things to it mylist.extend([x, y+1, z*2]) # You want a list consisting of the same objects repeated many times mylist = [0, 1, 2]*100 # like [0, 1, 2, 0, 1, 2, 0, 1, 2, ...] # You want to create a list containing known objects mylist = [1, 2, 4, 8, 16, 32, 64] # Like above, but using a list comprehension mylist = [2**n for n in range(7)] # You don't know how many things you want to add. mylist = [] x = 1 while x 100: mylist.append(x) x = 3*x-1 and so on. As you can see, append has its job to do. The last case is the closest to the context in which I used the above code construct. I have some ideas that on an append() the list's internal size increases (doubled?) since CPython over-allocates memory, and such. Not just on append. Any operation that adds or deletes items from a list might trigger a re-size. If the list needs to increase, it will double in size up to some maximum, then it will grow by a smaller amount. The purpose of this is that *on average* appending to the list will take a fixed amount of time. So, the question is: what is the alternative to growing a list when I have no idea what items or how many may be there? Don't worry about growing the list. That's all handled for you as part of the list interface. The whole point of using Python is to not have to worry about internal details like that. Right, that's what I thought. Good that I checked here, now I can write a nice reply to the comment :-) Thanks Steven and Dave. Best, Amit. -- http://echorand.me ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On Sun, Dec 1, 2013 at 2:04 AM, Steven D'Aprano st...@pearwood.info wrote: might trigger a re-size. If the list needs to increase, it will double in size up to some maximum, then it will grow by a smaller amount. The purpose of this is that *on average* appending to the list will take a fixed amount of time. Here's the over-allocation rule: if newsize 0: const = 3 if newsize 9 else 6 newsize += newsize // 8 + const Starting from empty and appending: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88 CPython's list uses realloc, which tries to resize without copying. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On 12/01/2013 05:32 AM, Amit Saha wrote: Hello, I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: mylist = [] mylist.append(1) # a number of times over I have some ideas that on an append() the list's internal size increases (doubled?) since CPython over-allocates memory, and such. So, the question is: what is the alternative to growing a list when I have no idea what items or how many may be there? Maybe you are confusing with catenating _strings_, rather than lists. Python's concat is problematic because it is a binary operation. So, catenating n bits makes n-1 operations, each with intermediate results, of growing sizes, all thrown away except for the last one, the actual result. If all of the bitN are strings: bit1 + bit2 + bit3 + bit4 + bit5 actually constructs: bit1+bit2 bit1+bit2+bit3 bit1+bit2+bit3+bit4 bit1+bit2+bit3+bit4+bit5 A number of unneeded string object, and a very big useless memory weight. Example Python code showing good/bad usage below (advanced pythonistas, please correct if needed, I don't know the latest nice Python idioms): # === case of a series of explicite text sections # don't do this: text = intro = intro... text += intro body = body... text+= '\n' + body concl = concl... text += '\n' + concl print(text) ; print() # also do not do: text = intro + '\n' + body + '\n' + concl print(text) ; print() # do that...: intro = intro... body = body... concl = concl... text = '\n'.join([intro, body, concl]) # but creates a list print(text) ; print() # ...or that: text = %s\n%s\n%s % (intro, body, concl) # no such list print(text) ; print() # === case of a list of sections of arbitrary size # (simulation code for demo) def make_section (n): return %s\n % n section_data = [13, 5, 79, 4, 268, 0, 987654321] # don't do this: text = for dat in section_data: section = make_section(dat) text += section print(text) # do that...: sections = [] for dat in section_data: section = make_section(dat) sections.append(section) text = ''.join(sections) print(text) # ... or that: sections = (make_section(dat) for dat in section_data) text = ''.join(sections) print(text) # ... or even that: text = ''.join(make_section(dat) for dat in section_data) # no need for ((...)) print(text) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On Sun, 1 Dec 2013 14:32:38 +1000, Amit Saha amitsaha...@gmail.com wrote: I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: mylist = [] mylist.append(1) Nothing evil about append. Many times the list can be built more prettily with a list comprehension, but if you're building it in a for loop, append does the job. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Alternatives to append() for growing a list
On Sun, Dec 01, 2013 at 02:32:38PM +1000, Amit Saha wrote: Hello, I was told by someone (as a comment) that a code snippet such as this would make Pythonistas talk my ear off about how evil the append() function is: There is absolutely nothing wrong with append. Either you have misunderstood, or the person who told you this was smoking crack. mylist = [] mylist.append(1) # a number of times over However, growing a list one item at a time using append like this is too much hard work. Some better solutions: # You have an existing list, and want to append a bunch of things to it mylist.extend([x, y+1, z*2]) # You want a list consisting of the same objects repeated many times mylist = [0, 1, 2]*100 # like [0, 1, 2, 0, 1, 2, 0, 1, 2, ...] # You want to create a list containing known objects mylist = [1, 2, 4, 8, 16, 32, 64] # Like above, but using a list comprehension mylist = [2**n for n in range(7)] # You don't know how many things you want to add. mylist = [] x = 1 while x 100: mylist.append(x) x = 3*x-1 and so on. As you can see, append has its job to do. I have some ideas that on an append() the list's internal size increases (doubled?) since CPython over-allocates memory, and such. Not just on append. Any operation that adds or deletes items from a list might trigger a re-size. If the list needs to increase, it will double in size up to some maximum, then it will grow by a smaller amount. The purpose of this is that *on average* appending to the list will take a fixed amount of time. So, the question is: what is the alternative to growing a list when I have no idea what items or how many may be there? Don't worry about growing the list. That's all handled for you as part of the list interface. The whole point of using Python is to not have to worry about internal details like that. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
