Re: PEP 288 ponderings

[Michael Sparks]

On Sun, 2 Jan 2005, Ian Bicking wrote:

> Steven Bethard wrote:
> > PEP 288 was mentioned in one of the lambda threads and so I ended up
> > reading it for the first time recently.  I definitely don't like the
> > idea of a magical __self__ variable that isn't declared anywhere.  It
> > also seemed to me like generator attributes don't really solve the
> > problem very cleanly.  An example from the PEP[1]:
> >
> > def mygen():
> > while True:
> > print __self__.data
> > yield None
> >
> > g = mygen()
> > g.data = 1
> > g.next()# prints 1
> > g.data = 2
> > g.next()# prints 2
>
> I don't get why this isn't good enough:
>
>  def mygen(data):
>  while True:
>  print data[0]
>  yield None

It's OK, but rather limited. IMHO a nicer approach is to use decorators to
decorate a generator with extra attributes. Specifically to embed the
generator inside an iterator inside an anonymous class. (We're exploring
this as a alternative approach to the system we currently use at work)
(Hmm... Now I describe it that sounds rather hideous, but the result is
nice)

First of all the decorator:

import copy
def wrapgenerator(bases=object, **attrs):
   def decorate(func):
   class statefulgenerator(bases):
  __doc__ = func.__doc__
  def __init__(self,*args):
 super(statefulgenerator, self).__init__(*args)
 self.func=func(self,*args)
 for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k])
 self.next=self.__iter__().next
  def __iter__(self): return iter(self.func)
   return statefulgenerator
   return decorate

This would be used thus:

@wrapgenerator(component)
def forwarder(self):
   # The generator, note the explicit self for local state
   # for this generator
   "Simple data forwarding generator"
   while 1:
  if self.dataReady("inbox"):
 self.send("outbox",self.recv("inbox"))
  elif self.dataReady("control"):
 if self.recv("control") == "shutdown":
break
  yield 1
   self.send("signal","shutdown")
   yield 0

This clearly assumes a set of attributes, local methods etc.

In this case our local methods, attribute etc are from this class:
class component(object):
   def __init__(self, *args):
   # Default queues
   self.queues = {"inbox":[],"control":[],"outbox":[],"signal":[]}
   def send(self, box, object): self.queues[box].append(object)
   def dataReady(self,box): return len(self.queues[box])>0
   def recv(self, box): # NB. Exceptions aren't caught
  X=self.queues[box][0]
  del self.queues[box][0]
  return X

This then gets used in the usual way:
>>> f = forwarder()
>>> f
<__main__.statefulgenerator object at 0x402d>
>>> f.send("inbox", "some data")
>>> f.send("inbox", "some more data")
>>> f.next()
1
>>> f.next()
1
>>> f.recv("outbox")
'some data'
>>> f.recv("outbox")
'some more data'

If you "just" want (arbitrary) generator attributes, then that's a lot
easier (since we don't need to support an arbitrary base class):

import copy
def add_attrs_to_generator(**attrs):
   def decorate(func):
   class statefulgenerator:
  __doc__ = func.__doc__
  def __init__(self,*args):
 self.func=func(self,*args)
 for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k])
 self.next=self.__iter__().next
  def __iter__(self): return iter(self.func)
   return statefulgenerator
   return decorate

@add_attrs_to_generator()
def simpleGen(self):
   while True:
  print dir(self)
  yield None

We can also add default attributes:

@add_attrs_to_generator(test=1,simple=2)
def simpleGen(self):
   while True:
  print dir(self)
  yield None

>>> g=simpleGen()
>>> g.next()
['__doc__', '__init__', '__iter__', '__module__', 'func', 'next',
'simple', 'test']

We've been using this general approach of giving generators local state
for communications for a while now, and it's been rather interesting/fun
to use. However we've been using explicit declaration of a class, and then
a standard fixed name for the generator (main) instead. (we've been
thinking about using decorators as syntactic sugar)

Being able to pass exception in would be very nice, however we've used a
"control"  attribute, queue in practice, to allow for need to be able to
shutdown an arbitrary generator. However, I don't think the __self__ part
of the PEP is explicitly required - since you can build that onto
generators using python quite happily now (@add_attrs_to_generator).

The code for the stuff we're playing with (in case anyone's interested :)
is at http://kamaelia.sourceforge.net/. The specific subsystem that uses
this approach has been released as a separate download (Axon[1]) on the
sourceforge project page[2], and the main system (obviously) uses this
heavily and can be browsed via viewcvs. [3]
   [1] http://kamae

Re: PEP 288 ponderings

[Steven Bethard]

Raymond Hettinger wrote:
[Steven Bethard]
(2) Since in all the examples there's a one-to-one correlation between
setting a generator attribute and calling the generator's next function,
aren't these generator attribute assignments basically just trying to
define the 'next' parameter list?
They are not the same.  The generator needs some way to receive the values. 
 The
function arguments cannot be used because they are needed to create the
generator-iterator.  The yield statements likewise won't work because the first
yield is not encountered until well after the first next() call.
Yeah, I wasn't trying to claim that passing the arguments to .next() is 
equivalent to generator attributes, only that the point at which new 
values for the generator state variables are provided correspond with 
calls to .next().  So if there was a means within a generator of getting 
access to the arguments passed to .next(), generator attributes would be 
unnecessary for the examples provided.

The given examples are minimal and are intended only to demonstrate the idea.
Do you have an example where the generator state isn't updated in 
lock-step with .next() calls?  I'd be interested to look at an example 
of this...

I definitely don't like the
idea of a magical __self__ variable that isn't declared anywhere.
It is no more magical than f.__name__ or f.__doc__ for functions.
I'm not sure this is quite a fair parallel.  The difference here is that 
 f.__name__ and f.__doc__ are accessed as attributes of the f object, 
and the __name__ and __doc__ attributes are created as a result of 
function creation.  The proposed __self__ is (1) not an attribute that 
becomes available, rather, a new binding local to the function, and (2) 
not created as a result of generator object creation but created as a 
result of calling .next() on the generator object.

Also, the __self__ argument is a non-issue because there are other alternate
approaches such as providing a function that retrieves the currently
running generator.
Is there a discussion of some of these alternate suggested approaches 
somewhere you could point me to?

The more important part of the PEP is the idea for generator exceptions.  The
need arises in the context of flushing/closing resources upon generator
termination.
I wonder if maybe it would be worth moving this part to a separate PEP. 
 It seems like it could probably stand on its own merit, and having it 
in with the generator attributes PEP means it isn't likely to be 
accepted separately.

Of course, I would probably declare a class and provide a .close() 
method. =)

Steve
--
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 288 ponderings

[Raymond Hettinger]

[Steven Bethard]
> (1) What's the benefit of the generator versions of these functions over
> the class-based versions?

Generators are easier to write, are clearer, and run faster.

They automatically
* create a distinct generator-iterator object upon each invocation
* create the next() and idempotent __iter__() methods.
* raise StopIteration  upon termination
* remain stopped if next() is called too many times
* save their own local variable state, avoiding the need for self.var references
* resume execution at the point of the last yield



> (2) Since in all the examples there's a one-to-one correlation between
> setting a generator attribute and calling the generator's next function,
> aren't these generator attribute assignments basically just trying to
> define the 'next' parameter list?

They are not the same.  The generator needs some way to receive the values.  The
function arguments cannot be used because they are needed to create the
generator-iterator.  The yield statements likewise won't work because the first
yield is not encountered until well after the first next() call.

The given examples are minimal and are intended only to demonstrate the idea.



> I definitely don't like the
> idea of a magical __self__ variable that isn't declared anywhere.

It is no more magical than f.__name__ or f.__doc__ for functions.  The concept
is almost identical to that for threading.local().  Also, the __self__ argument
is a non-issue because there are other alternate approaches such as providing a
function that retrieves the currently running generator.  Which approach is
ultimately taken is a matter of aesthetics -- the PEP itself concentrates on the
core idea instead of debating syntax.

The real issue with PEP 288's idea for generator attributes is that the current
C implementation doesn't readily accommodate this change.  Major surgery would
be required :-(

The more important part of the PEP is the idea for generator exceptions.  The
need arises in the context of flushing/closing resources upon generator
termination.



Raymond Hettinger


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 288 ponderings

[Nick Coghlan]

Ian Bicking wrote:
Using a one-element list is kind of annoying, because it isn't clear out 
of context that it's just a way of creating shared state.  But it's 
okay, work right now, and provides the exact same functionality.
Uh, isn't shared state what classes were invented for?
Py> class mygen(object):
...   def __init__(self, data):
... self.data = data
...   def __iter__(self):
... while 1:
...   print self.data
...   yield None
...
Py> g = mygen(0)
Py> giter = iter(g)
Py> giter.next()
0
Py> g.data = 1
Py> giter.next()
1
Cheers,
Nick.
--
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
http://boredomandlaziness.skystorm.net
--
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 288 ponderings

[Ian Bicking]

Steven Bethard wrote:
PEP 288 was mentioned in one of the lambda threads and so I ended up 
reading it for the first time recently.  I definitely don't like the 
idea of a magical __self__ variable that isn't declared anywhere.  It 
also seemed to me like generator attributes don't really solve the 
problem very cleanly.  An example from the PEP[1]:

def mygen():
while True:
print __self__.data
yield None
g = mygen()
g.data = 1
g.next()# prints 1
g.data = 2
g.next()# prints 2
I don't get why this isn't good enough:
def mygen(data):
while True:
print data[0]
yield None
data = [None]
g = mygen(data)
data[0] = 1
g.next()
data[0] = 1
g.next()
Using a one-element list is kind of annoying, because it isn't clear out 
of context that it's just a way of creating shared state.  But it's 
okay, work right now, and provides the exact same functionality.  The 
exception part of PEP 288 still seems interesting.

--
Ian Bicking  /  [EMAIL PROTECTED]  / http://blog.ianbicking.org
--
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 288 ponderings

[Jp Calderone]

On Sun, 02 Jan 2005 01:04:06 GMT, Steven Bethard <[EMAIL PROTECTED]> wrote:
>PEP 288 was mentioned in one of the lambda threads and so I ended up 
> reading it for the first time recently.  I definitely don't like the 
> idea of a magical __self__ variable that isn't declared anywhere.  It 
> also seemed to me like generator attributes don't really solve the 
> problem very cleanly.  An example from the PEP[1]:
> 
> [snip]

  You may be interested in greenlets.  They solve this problem in a way 
which is conceptually much simpler and cleaner.

  They are available as a third party extension, and unfortunately are 
not terribly well documented yet (but luckily they are quite simple).

  http://codespeak.net/svn/user/arigo/greenlet/

  Jp
-- 
http://mail.python.org/mailman/listinfo/python-list

PEP 288 ponderings

[Steven Bethard]

PEP 288 was mentioned in one of the lambda threads and so I ended up 
reading it for the first time recently.  I definitely don't like the 
idea of a magical __self__ variable that isn't declared anywhere.  It 
also seemed to me like generator attributes don't really solve the 
problem very cleanly.  An example from the PEP[1]:

def mygen():
while True:
print __self__.data
yield None
g = mygen()
g.data = 1
g.next()# prints 1
g.data = 2
g.next()# prints 2
I looked in the archives but couldn't find a good discussion of why 
setting an attribute on the generator is preferable to passing the 
argument to next.  Isn't this example basically equivalent to:

class mygen(object):
def next(self, data):
print data
return None
g = mygen()
g.next(1)   # prints 1
g.next(2)   # prints 2
Note that I didn't even define an __iter__ method since it's never used 
in the example.

Another example from the PEP:
def filelike(packagename, appendOrOverwrite):
data = []
if appendOrOverwrite == 'w+':
data.extend(packages[packagename])
try:
while True:
data.append(__self__.dat)
yield None
except FlushStream:
packages[packagename] = data
ostream = filelike('mydest','w')
ostream.dat = firstdat; ostream.next()
ostream.dat = firstdat; ostream.next()
ostream.throw(FlushStream)
This could be rewritten as:
class filelike(object):
def __init__(self, packagename, appendOrOverwrite):
self.data = []
if appendOrOverwrite == 'w+':
self.data.extend(packages[packagename])
def next(self, dat):
self.data.append(dat)
return None
def close(self):
packages[packagename] = self.data
ostream = filelike('mydest','w')
ostream.next(firstdat)
ostream.next(firstdat)
ostream.close()
So, I guess I have two questions:
(1) What's the benefit of the generator versions of these functions over 
the class-based versions?

(2) Since in all the examples there's a one-to-one correlation between 
setting a generator attribute and calling the generator's next function, 
aren't these generator attribute assignments basically just trying to 
define the 'next' parameter list?

If this is true, I would have expected that a more useful idiom would 
look something like:

def mygen():
while True:
data, = nextargs()
print data
yield None
g = mygen()
g.next(1)   # prints 1
g.next(2)   # prints 2
where the nextargs function retrieves the arguments of the most recent 
call to the generator's next function.

With a little sys._getframe hack, you can basically get this behavior now:
py> class gen(object):
... def __init__(self, gen):
... self.gen = gen
... def __iter__(self):
... return self
... def next(self, *args):
... return self.gen.next()
... @staticmethod
... def nextargs():
... return sys._getframe(2).f_locals['args']
...
py> def mygen():
... while True:
... data, = gen.nextargs()
... print data
... yield None
...
py> g = gen(mygen())
py> g.next(1)
1
py> g.next(2)
2
Of course, it's still a little magical, but I think I like it a little 
better because you can see, looking only at 'mygen', when 'data' is 
likely to change value...

Steve
[1] http://www.python.org/peps/pep-0288.html
--
http://mail.python.org/mailman/listinfo/python-list