Re: [Tutor] I am having difficulty grasping 'generators'
On 27May2014 15:27, Degreat Yartey yarteydegre...@gmail.com wrote: I am studying python on my own (i.e. i am between the beginner and intermediate level) and i haven't met any difficulty until i reached the topic 'Generators and Iterators'. I need an explanation so simple as using the expression 'print ()', in this case 'yield'. Python 2.6 here! Thank you. Generators are functions that do a bit of work and then yield a value, then a bit more and so on. This means that you call them once. What you get back is an iterator, not the normal function return value. Whenever you use the iterator, the generator function runs until it hits a yield statement, and the value in theyield statement is what you get for that iteration. Next time you iterate, the function runs a bit more, until it yields again, or returns (end of function, and that causes end of iteration). So the function doesn't even run until you ask for a value, and then it only runs long enough to find the next value. Example (all code illstrative only, untested): Suppose you need to process every second line of a file. You might write it directly like this: def munge_lines(fp): ''' Do stuff with every second line of the already-open file `fp`. ''' lineno = 0 for line in fp: lineno += 1 if lineno % 2 == 0: print lineno, line, That should read lines from the file and print every second one with the line number. Now suppose you want something more complex than every second line, especially something that requires keeping track of some state. In the example above you only need the line number, and using it still consumes 2 of the 3 lines in the loop body. A more common example might be lines between two markers. The more of that you embed in the munge_lines function, the more it will get in the way of seeing what the function actually does. So a reasonable thing might be to write a function that gets the requested lines: def wanted_lines(fp): wanted = [] between = False for line in fp: if between: if 'end_marker' in line: between = False else: wanted.append(line) elif 'start_maker' in line: between = True return wanted This reads the whole file and returns a line of the wanted lines, and munge_lines: might then look like this: for line in wanted_lines(fp): print line However: - that reads the whole file before returning anything - has to keep all the lines in the list wanted Slow in response, heavy in memory cost, and unworkable if fp actually doesn't end (eg reading from a terminal, or a pipeline, or...) What you'd really like is to get each line as needed. We can rewrite wanted_lines as a generator: def wanted_lines(fp): between = False for line in fp: if between: if 'end_marker' in line: between = False else: yield line elif 'start_maker' in line: between = True All we've done is used yield instead of the append and removed the wanted list and the return statement. The calling code is the same. To see the difference, put a print in wanted_lines as the first line of the for loop. With the list version you will see all the prints run before you get the array back. With the generator you will see the print run just before each value you get back. Cheers, Cameron Simpson c...@zip.com.au ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
I am completely new to programming! On May 27, 2014 10:54 PM, R. Alan Monroe amon...@columbus.rr.com wrote: I need an explanation so simple as using the expression 'print ()', in this case 'yield'. Python 2.6 here! Ever write any C programs with static variables? Generators can be explained in those terms if you have experience with them. Alan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
I really love this explanation... that means functions just run till it finishes its duty, then return...and generators just generate one at a time until the 'for' statement asks for __next__(). On May 28, 2014 8:37 AM, Cameron Simpson c...@zip.com.au wrote: On 27May2014 15:27, Degreat Yartey yarteydegre...@gmail.com wrote: I am studying python on my own (i.e. i am between the beginner and intermediate level) and i haven't met any difficulty until i reached the topic 'Generators and Iterators'. I need an explanation so simple as using the expression 'print ()', in this case 'yield'. Python 2.6 here! Thank you. Generators are functions that do a bit of work and then yield a value, then a bit more and so on. This means that you call them once. What you get back is an iterator, not the normal function return value. Whenever you use the iterator, the generator function runs until it hits a yield statement, and the value in theyield statement is what you get for that iteration. Next time you iterate, the function runs a bit more, until it yields again, or returns (end of function, and that causes end of iteration). So the function doesn't even run until you ask for a value, and then it only runs long enough to find the next value. Example (all code illstrative only, untested): Suppose you need to process every second line of a file. You might write it directly like this: def munge_lines(fp): ''' Do stuff with every second line of the already-open file `fp`. ''' lineno = 0 for line in fp: lineno += 1 if lineno % 2 == 0: print lineno, line, That should read lines from the file and print every second one with the line number. Now suppose you want something more complex than every second line, especially something that requires keeping track of some state. In the example above you only need the line number, and using it still consumes 2 of the 3 lines in the loop body. A more common example might be lines between two markers. The more of that you embed in the munge_lines function, the more it will get in the way of seeing what the function actually does. So a reasonable thing might be to write a function that gets the requested lines: def wanted_lines(fp): wanted = [] between = False for line in fp: if between: if 'end_marker' in line: between = False else: wanted.append(line) elif 'start_maker' in line: between = True return wanted This reads the whole file and returns a line of the wanted lines, and munge_lines: might then look like this: for line in wanted_lines(fp): print line However: - that reads the whole file before returning anything - has to keep all the lines in the list wanted Slow in response, heavy in memory cost, and unworkable if fp actually doesn't end (eg reading from a terminal, or a pipeline, or...) What you'd really like is to get each line as needed. We can rewrite wanted_lines as a generator: def wanted_lines(fp): between = False for line in fp: if between: if 'end_marker' in line: between = False else: yield line elif 'start_maker' in line: between = True All we've done is used yield instead of the append and removed the wanted list and the return statement. The calling code is the same. To see the difference, put a print in wanted_lines as the first line of the for loop. With the list version you will see all the prints run before you get the array back. With the generator you will see the print run just before each value you get back. Cheers, Cameron Simpson c...@zip.com.au ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
This means that '...' should generally contain a manipulator then yield generates from where it stopped...*getting it* Thanks for the explanation though! Its seems so simple to digest. Thank you... On May 28, 2014 1:09 AM, Danny Yoo d...@hashcollision.org wrote: On Tue, May 27, 2014 at 12:27 PM, Degreat Yartey yarteydegre...@gmail.com wrote: I am studying python on my own (i.e. i am between the beginner and intermediate level) and i haven't met any difficulty until i reached the topic 'Generators and Iterators'. I need an explanation so simple as using the expression 'print ()', in this case 'yield'. You can think of a generator as almost like a function, except it can return, not just once, but multiple times. Because it can return multiple times, if we squint at it enough, it acts like a _sequence_, just like the other sequence-like things in Python like files and lists and tuples. That is, as a sequence, it's something that we can walk down, element by element. We can loop over it. For example, let's say that we wanted to represent the same sequences as that of range(5). Here's one way we can do it with a generator: # def upToFive(): yield 0 yield 1 yield 2 yield 3 yield 4 # Let's try it. # sequence = upToFive() next(sequence) 0 next(sequence) 1 next(sequence) 2 next(sequence) 3 next(sequence) 4 next(sequence) Traceback (most recent call last): File stdin, line 1, in module StopIteration for x in upToFive(): ... print(I see %d % x) ... I see 0 I see 1 I see 2 I see 3 I see 4 # Now this is a toy example. If we wanted range(5), we'd just say range(5) and be done with it. What's neat about generators is that they make it easy to build these sequences while pretending that we're writing a plain function. All of the even numbers, for examples, is a sequence that we can make with a generator: # def onlyEvens(): n = 0 while True: yield n n = n + 2 # Let's try running it: # sequence = onlyEvens() next(sequence) 0 next(sequence) 2 next(sequence) 4 next(sequence) 6 # And note that this sequence doesn't stop! We can keep calling next() on it and it will continue to run. We _can_ write a loop to run over such infinite sequences, but we'll also have to make sure to stop it manually: it won't exhaust otherwise, so doing something like: # for n in onlyEvens(): ... # better have something in the ... that interrupts or returns, or else that loop will never end. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
On 28/05/14 11:52, Degreat Yartey wrote: This means that '...' should generally contain a manipulator then yield generates from where it stopped...*getting it* It would help if you deleted the irrelevent bits so we can see which '...' you mean. I'm guessing it was this comment, right at the end, by Danny: We _can_ write a loop to run over such infinite sequences, but we'll also have to make sure to stop it manually: it won't exhaust otherwise, so doing something like: # for n in onlyEvens(): ... # better have something in the ... that interrupts or returns, or else that loop will never end. Notice that the ... here is not part of the generator. So the ... here references to how you process the output of the generator - the yielded values. So in this example case you'd have something like # for n in onlyEvens(): if n 1000: break # prevent infinite loop # now process the evens that we are interested in. # The 'break' could also be a 'return' if the loop were inside a function. And it doesn't have to be a direct value check as shown the condition could be based on some external condition or even user input. The important point is to make sure that you have some way to exit the loop if the generator is infinite. hth -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
Dear all, i'd like to thank every answer in this list. Alan Gauld is a fine writer of excellent introductory material on Pyton, and so are a few other members of this list. So, it is always enlightening to read what you all write. Keep up the good work. All the best, hilton On Wed, May 28, 2014 at 8:57 AM, Alan Gauld alan.ga...@btinternet.comwrote: On 28/05/14 11:52, Degreat Yartey wrote: This means that '...' should generally contain a manipulator then yield generates from where it stopped...*getting it* It would help if you deleted the irrelevent bits so we can see which '...' you mean. I'm guessing it was this comment, right at the end, by Danny: We _can_ write a loop to run over such infinite sequences, but we'll also have to make sure to stop it manually: it won't exhaust otherwise, so doing something like: # for n in onlyEvens(): ... # better have something in the ... that interrupts or returns, or else that loop will never end. Notice that the ... here is not part of the generator. So the ... here references to how you process the output of the generator - the yielded values. So in this example case you'd have something like # for n in onlyEvens(): if n 1000: break # prevent infinite loop # now process the evens that we are interested in. # The 'break' could also be a 'return' if the loop were inside a function. And it doesn't have to be a direct value check as shown the condition could be based on some external condition or even user input. The important point is to make sure that you have some way to exit the loop if the generator is infinite. hth -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
I'm not going to add too much more to all the replies here already, but one of my students did record a quick 6-minute video in one of my courses where i explained generators. hopefully you find it useful! It's about halfway down the page at http://cyberwebconsulting.com. (Also for those learning Python and in the San Francisco area, I'm offering another intensive 3-day course mid-summer -- more info on the same page. Ping me privately for more details or if you have questions!) Cheers, --Wesley On Tue, May 27, 2014 at 12:27 PM, Degreat Yartey yarteydegre...@gmail.comwrote: I am studying python on my own (i.e. i am between the beginner and intermediate level) and i haven't met any difficulty until i reached the topic 'Generators and Iterators'. I need an explanation so simple as using the expression 'print ()', in this case 'yield'. Python 2.6 here! Thank you. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - A computer never does what you want... only what you tell it. +wesley chun http://google.com/+WesleyChun : wescpy at gmail : @wescpyhttp://twitter.com/wescpy Python training consulting : http://CyberwebConsulting.com Core Python books : http://CorePython.com Python blog: http://wescpy.blogspot.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
I need an explanation so simple as using the expression 'print ()', in this case 'yield'. Python 2.6 here! Ever write any C programs with static variables? Generators can be explained in those terms if you have experience with them. Alan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
On Tue, May 27, 2014 at 12:27 PM, Degreat Yartey yarteydegre...@gmail.com wrote: I am studying python on my own (i.e. i am between the beginner and intermediate level) and i haven't met any difficulty until i reached the topic 'Generators and Iterators'. I need an explanation so simple as using the expression 'print ()', in this case 'yield'. You can think of a generator as almost like a function, except it can return, not just once, but multiple times. Because it can return multiple times, if we squint at it enough, it acts like a _sequence_, just like the other sequence-like things in Python like files and lists and tuples. That is, as a sequence, it's something that we can walk down, element by element. We can loop over it. For example, let's say that we wanted to represent the same sequences as that of range(5). Here's one way we can do it with a generator: # def upToFive(): yield 0 yield 1 yield 2 yield 3 yield 4 # Let's try it. # sequence = upToFive() next(sequence) 0 next(sequence) 1 next(sequence) 2 next(sequence) 3 next(sequence) 4 next(sequence) Traceback (most recent call last): File stdin, line 1, in module StopIteration for x in upToFive(): ... print(I see %d % x) ... I see 0 I see 1 I see 2 I see 3 I see 4 # Now this is a toy example. If we wanted range(5), we'd just say range(5) and be done with it. What's neat about generators is that they make it easy to build these sequences while pretending that we're writing a plain function. All of the even numbers, for examples, is a sequence that we can make with a generator: # def onlyEvens(): n = 0 while True: yield n n = n + 2 # Let's try running it: # sequence = onlyEvens() next(sequence) 0 next(sequence) 2 next(sequence) 4 next(sequence) 6 # And note that this sequence doesn't stop! We can keep calling next() on it and it will continue to run. We _can_ write a loop to run over such infinite sequences, but we'll also have to make sure to stop it manually: it won't exhaust otherwise, so doing something like: # for n in onlyEvens(): ... # better have something in the ... that interrupts or returns, or else that loop will never end. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] I am having difficulty grasping 'generators'
On Tue, May 27, 2014 at 12:27 PM, Degreat Yartey yarteydegre...@gmail.com wrote: I am studying python on my own (i.e. i am between the beginner and intermediate level) and i haven't met any difficulty until i reached the topic 'Generators and Iterators'. I need an explanation so simple as using the expression 'print ()', in this case 'yield'. Python 2.6 here! Thank you. Simply put, a generator is just like a function - except that a function returns a value and then dies, but a generator yields a value and sticks around, waiting to yield another one - and another, and another, etc. I had some difficulty grasping the concept of generators at first because the examples you get in the tutorials are either trivial (you could do the same thing some other way, and it would be simpler) or horribly complex (and you get bogged down in the problem, rather than getting to grips with generators themselves.) So here's a real-life example - the first time I actually used generators... and they actually saved me a LOT of hard work and confusion. I needed to read some database files (from a very old database, for which I don't have an API - so I basically needed to read each record and interpret it byte-by-byte). I wrote a bunch of classes for the different tables in the database (I'll use Patient() as my example), but the heart of the operation is a generator. def RecordGenerator(office=None, fileType=None): obj = fTypes[fileType]()# create an empty record so we can read its attributes recLen = obj.RecordLength# such as the record length, the filename, etc. with open(getattr(office, obj.TLA) + '.dat','rb') as inFile: tmpIn = inFile.read() buf = StringIO.StringIO(tmpIn) inRec = buf.read(recLen)# initialize inRec and burn the header record while not (len(inRec) recLen): inRec = buf.read(recLen) obj = fTypes[fileType](inRec) if (obj.Valid): yield obj buf.close() Now I'm writing a program that needs to process all the patients in office 01. I write the following: for obj in RecordGenerator(01, Patient): doStuffPart1 doStuffPart2 doStuffPart3 etc. The generator creates an empty Patient() object, looks at its attributes, and finds that the filename is 01PAT.dat and the records are 1024 bytes long. It opens the file and reads it into tmpIn, then grabs 1024-byte chunks of tmpIn and passes them to Patient() until it finds a valid (not deleted, not corrupted) patient record. It yields that record to the caller, and waits. When doStuffPart3 has completed, I go back to the top of the for loop and ask for the next patient. RecordGenerator steps through tmpIn, 1024 bytes at a time, until it finds a valid patient, then yields it and waits. When it hits the end of the file (or, more precisely, when the remaining length of tmpIn is less than a full record length), it quits. Essentially, you write a generator as if it were a function - but you use it as if it were a sequence. Best of both worlds, baby! ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
