Re: [Python-Dev] Active Objects in Python

2005-10-01 Thread Michael Sparks 
On Friday 30 September 2005 22:13, Michael Sparks (home address) wrote:
> I wrote a white paper based on my Python UK talk, which is here:
>     * http://www.bbc.co.uk/rd/pubs/whp/whp11.shtml

Oops that URL isn't right. It should be:
   * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

Sorry! (Thanks to LD 'Gus' Landis for pointing that out!)

Regards,


Michael.
--
"Though we are not now that which in days of old moved heaven and earth, 
   that which we are, we are: one equal temper of heroic hearts made 
 weak by time and fate but strong in will to strive, to seek, 
  to find and not to yield" -- "Ulysses", Tennyson
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-30 Thread Michael Sparks (home address) 
[ I don't post often, but hopefully the following is of interest in this
  discussion ]

Bruce Eckel wrote:

> Yes, defining an class as "active" would:
> 1) Install a worker thread and concurrent queue in each object of that
> class.
> 2) Automatically turn method calls into tasks and enqueue them
> 3) Prevent any other interaction other than enqueued messages

Here I think the point is an object that has some form of thread with
enforced communications preventing (or limiting) shared data. This
essentially describes the project we've been working on with Kamaelia.
Kamaelia is divided into two halves:

   * A simple framework (Axon) for building components with the following
 characteristics:
  * Have a main() method which is a generator.
  * Have inbound and outbound named queues implemented using lists.
 (akin to stdin/out)
  * Access to an environmental system which is key/value lookup (similar
 purpose to the unix environment). (This could easily grow into a
 much more linda like system)
   * A collection of components which form a library.

We also have a threaded component which uses a thread and queues rather than
a generator & lists.

With regard to the criteria you listed in a later post:
   1) Has been tested against (so far) 2 novice programmers, who just picked
  up the framework and ran with in. (learnt python one week, axon the
  next, then implemented interesting things rapidly)
   2) We don't deal with distribution (yet), but this is on the cards.
  Probably before EuroOSCON. (I want to have a program with two top
  level pygame windows)
   3) We use python generators and have experimented initially with several
  tens of thousands of them, which works suprisingly well, even on an
  old 500Mhz machine.
   4) Self guarding is difficult. However generators are pretty opaque, and
  getting at values inside their stack frame is beyond my python skills.
   5) Deadlock can be forced in all scenarios, but you would actually have
  to work at it. In practice if we yield at all points or use the
  threaded components for everything then we'd simply livelock.
   6) Regarding an actor. Our pygame sprite component actually is very,
  very actor like. This is to the extent we've considered implementing a
  script reading and director component for orchestrating them. (Time
  has prevented that though)
   7) Since our target has been novices, this has come by default.
   8) We don't do mobility of components/tasks as yet, however a colleague
  at work interested in mobility & agent software has proposed looking
  at that using Kamaelia for the coming year. We also ran the framework
  by people at the WoTUG (*) conference recently, and the general
  viewpoint came back that it's tractable from a formal analysis
  and mobility perspective(if we change x,y,z).
 (*) http://www.wotug.org/

I wrote a white paper based on my Python UK talk, which is here:
* http://www.bbc.co.uk/rd/pubs/whp/whp11.shtml

(BBC R&D white papers are aimed at imparting info in a such a way that it's
useful and readable, unlike some white papers :)

So far we have components for building things from network servers through
to audio/video decoders and players, and presentation tools. (Using pygame
and tkinter) Also we have a graphical editting tool for putting together
systems and introspecting systems.

An aim in terms of API was to enable novices to implement a microcosm
version of the system. Specifically we tested this idea on a pre-university
trainee who built a fairly highly parallel system which took video from an
mpg file (simulating a PVR), and sent periodic snapshots to a mobile phone
based client. He had minimal programming experience at the start, and
implemented this over a 3 months period.

We've since repeated that experience with a student who has had 2 years at
university, but no prior concurrency, python or networks experience. In his
6 weeks with us he was able to do some interesting things. (Such as
implement a system for joining multicast islands and sending content over a
simple reliable multicast protocol).

The tutorial we use to get people up to speed rapidly is here:
* http://kamaelia.sourceforge.net/MiniAxon/

However that doesn't really cover the ideas of how to write components, use
pipelines, graphlines and services.

At the moment we're adding in the concept of a chassis into which you plug
in existing components. The latest concept we're bouncing round is this:

# (This was fleshed out in respect to a question by a novice on c.l.p -
# periodically send the the same message to all connected clients)
# start
from Kamaelia.Chassis.ConnectedServer import SimpleServer
from Kamaelia.Chassis.Splitter import splitter, publish, Subscriber

class message_source(component):
   [ ... snippety ... ]
   def main(self):
  last = self.scheduler.time
  while 1:
 yield 1
 if

Re: [Python-Dev] Active Objects in Python

2005-09-28 Thread Martin v. Löwis 
Bruce Eckel wrote:
> Since the enqueuing process serializes all requests to active objects,
> and since each message-task runs to completion before the next one
> starts, the problem of threads interfering with each other is
> eliminated. Also, the enqueuing process will always happen, so even if
> the queue blocks it will be for a very short, deterministic time. So
> the theory is that Active Object systems cannot deadlock (although I
> believe they can livelock).

My experience with active objects is with SDL (the Specification and
Description Language). There, the primitive communication between
objects is through one-way messages (called signals). Such a system
can deadlock if an object doesn't produce "enough" new outgoing
messages in response to incoming messages.

If all communication is through RPCs, i.e. you owe your caller
a response, then the only way to deadlock such a system is that
all active objects terminate. The interesting question then is
whether an object can terminate while processing a message, and
whether this will cause a response to be sent back.

> But what's not clear is whether this would
> require knowledge of the innards of the Python interpreter (which I
> don't have) or if it could be built using libraries.

As others have pointed out, you have to find a way to deal with the
standard library. In principle, each active object should have its
own copy of the standard library. For some modules/objects, this is
not reasonable (e.g. stdout). In OCCAM, host IO itself is (conceptually)
an occam process, so sys.stdout would have to become an active
object. In that approach, you would essentially have to rewrite
parts of the standard library, wrapping the underlying modules
appropriately.

> BTW: I think that Hoare's Communicating Sequential Processes (CSP)
> basically describes the idea of active objects but without objects.
> That is, I think active objects is CSP applied to OO.

I don't believe this is true. In CSP, communication between processes
is through rendezvous, which is quite different from active objects,
as there is no queue for incoming calls - instead, the receiving process
must expect a specific communication partner just like the sending
process; there is also no inherent request/response mechanism in
CSP.

In CSP, there is plenty of chance for deadlocks. If a process becomes
STOP (i.e. the process that will never again interact), then any
communication attempt with that process also cause the partner to
become STOP.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-28 Thread Nick Coghlan 
Greg Ewing wrote:
> Nick Coghlan wrote:
> 
> 
>>PEP 342's yield expressions can probably be used to help address that 
>>problem, 
>>though:
>>
>>   class SomeAO(ActiveObject):
>> def processSomeMessage(self):
>>   msg = yield
>>   # Do something with the message
>>   next_msg = yield makeSomeBlockingCall(self)
>>   # Do something with the next message
> 
> 
> I don't see how that helps, since makeSomeBlockingCall()
> is evaluated (and therefore blocks) *before* the yield
> happens.

Chris got it right - I just named the function badly. The idea is that there 
would be an interaction with the Active Object's event loop to get the actual 
result back once it was available from the thread that actually made the 
blocking call. In the meantime, *this* object could continue to handle other 
messages.

What the approach allows is to have a single physical thread with a number of 
coroutiney-type event handlers running inside it.

Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-27 Thread Bruce Eckel 
Oops. I forgot to add that to the list. Yes, in the working example
of Active Objects that I've written in Java J2SE5, when you send a
message to an active object, you get back a Future, which
I suspect would be the same as your Deferred.

Tuesday, September 27, 2005, 7:41:27 PM, Christopher Armstrong wrote:

> On 9/28/05, Greg Ewing <[EMAIL PROTECTED]> wrote:
>> Nick Coghlan wrote:
>>
>> > PEP 342's yield expressions can probably be used to help address that 
>> > problem,
>> > though:
>> >
>> >class SomeAO(ActiveObject):
>> >  def processSomeMessage(self):
>> >msg = yield
>> ># Do something with the message
>> >next_msg = yield makeSomeBlockingCall(self)
>> ># Do something with the next message
>>
>> I don't see how that helps, since makeSomeBlockingCall()
>> is evaluated (and therefore blocks) *before* the yield
>> happens.

> Sounds like makeSomeBlockingCall is just misnamed (probably depending
> who you ask).

> I wrote a small library recently that wraps Twisted's Deferreds and
> asynchronous Failure objects such that you can do stuff like

> try:
>  x = yield remoteObject.getSomething()
> except Foo:
>  print "Oh no!"

> This is just a 2.5-ification of defgen, which is at
> twisted.internet.defer.{deferredGenerator,waitForDeferred}. So anyway,
> if your actor messages always return Deferreds, then this works quite
> nicely.


> --
>   Twisted   |  Christopher Armstrong: International Man of Twistery
>Radix|-- http://radix.twistedmatrix.com
> |  Release Manager, Twisted Project
>   \\\V///   |-- http://twistedmatrix.com
>|o O||
> wvw-+
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/bruceeckel-python3234%40mailblocks.com


Bruce Eckelhttp://www.BruceEckel.com   mailto:[EMAIL PROTECTED]
Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e"
Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel
Subscribe to my newsletter:
http://www.mindview.net/Newsletter
My schedule can be found at:
http://www.mindview.net/Calendar



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-27 Thread Christopher Armstrong 
On 9/28/05, Greg Ewing <[EMAIL PROTECTED]> wrote:
> Nick Coghlan wrote:
>
> > PEP 342's yield expressions can probably be used to help address that 
> > problem,
> > though:
> >
> >class SomeAO(ActiveObject):
> >  def processSomeMessage(self):
> >msg = yield
> ># Do something with the message
> >next_msg = yield makeSomeBlockingCall(self)
> ># Do something with the next message
>
> I don't see how that helps, since makeSomeBlockingCall()
> is evaluated (and therefore blocks) *before* the yield
> happens.

Sounds like makeSomeBlockingCall is just misnamed (probably depending
who you ask).

I wrote a small library recently that wraps Twisted's Deferreds and
asynchronous Failure objects such that you can do stuff like

try:
 x = yield remoteObject.getSomething()
except Foo:
 print "Oh no!"

This is just a 2.5-ification of defgen, which is at
twisted.internet.defer.{deferredGenerator,waitForDeferred}. So anyway,
if your actor messages always return Deferreds, then this works quite
nicely.


--
  Twisted   |  Christopher Armstrong: International Man of Twistery
   Radix|-- http://radix.twistedmatrix.com
|  Release Manager, Twisted Project
  \\\V///   |-- http://twistedmatrix.com
   |o O||
wvw-+
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-27 Thread Greg Ewing 
Nick Coghlan wrote:

> PEP 342's yield expressions can probably be used to help address that 
> problem, 
> though:
> 
>class SomeAO(ActiveObject):
>  def processSomeMessage(self):
>msg = yield
># Do something with the message
>next_msg = yield makeSomeBlockingCall(self)
># Do something with the next message

I don't see how that helps, since makeSomeBlockingCall()
is evaluated (and therefore blocks) *before* the yield
happens.

-- 
Greg Ewing, Computer Science Dept, +--+
University of Canterbury,  | A citizen of NewZealandCorp, a   |
Christchurch, New Zealand  | wholly-owned subsidiary of USA Inc.  |
[EMAIL PROTECTED]  +--+
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-27 Thread Nick Coghlan 
Bruce Eckel wrote:
> Since the enqueuing process serializes all requests to active objects,
> and since each message-task runs to completion before the next one
> starts, the problem of threads interfering with each other is
> eliminated. Also, the enqueuing process will always happen, so even if
> the queue blocks it will be for a very short, deterministic time. So
> the theory is that Active Object systems cannot deadlock (although I
> believe they can livelock).

I've done this at work (in C++, not Python), and it works very well. I had my 
doubts when my boss first put the architecture for it in place, but a couple 
of years of working with this architecture (and that boss!) persuaded me 
otherwise.

Livelock is a definite issue, particular if you design the individual active 
objects as finite state machines - it is fairly easy to get into a situation 
where Object A is waiting for a particular message from Object B, while Object 
B is waiting for a particular message from Object A. This chain is generally 
indirect, which makes it hard to spot. It is, however, still easier to figure 
this out than it is to figure out normal threading bugs, because you can find 
it just by analysing a sequence diagram showing message exchange and state 
transitions, rather than caring about the actual underlying code. Threading 
bugs, on the other hand, can turn up any time you access a shared data 
structure.

> BTW: I think that Hoare's Communicating Sequential Processes (CSP)
> basically describes the idea of active objects but without objects.
> That is, I think active objects is CSP applied to OO.

Or to put it another way, applying OO to CSP is a matter of hiding the 
addressing and sending of messages behind method calls. It becomes almost like 
doing in-process remote procedure calls.

I think there's a definite synergy with PEP 342 here as well - one of the 
problems with handling Active Objects is how to allow other messages to be 
processed while making a blocking call, without losing your current state. One 
way is to hold any state that persists between messages in the object itself, 
but that's a seriously unnatural style of programming (it can be done, it's 
just a pain).

PEP 342's yield expressions can probably be used to help address that problem, 
though:

   class SomeAO(ActiveObject):
 def processSomeMessage(self):
   msg = yield
   # Do something with the message
   next_msg = yield makeSomeBlockingCall(self)
   # Do something with the next message

Regards,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://boredomandlaziness.blogspot.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-27 Thread Benji York 
Phillip J. Eby wrote:
> Well, you could pickle and unpickle the objects you send from one function 
> to another, and for cross-process communication, you'll need to do 
> something like that anyway, or else use that shared-memory objects 
> thing.  PySHM?  I don't remember its name, but it's an extension that lets 
> you store Python objects in shared memory and use them from multiple 
> processes, modulo certain strict limitations.

"POSH": http://poshmodule.sourceforge.net/posh/html/
--
Benji York


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Active Objects in Python

2005-09-27 Thread Phillip J. Eby 
At 11:18 AM 9/27/2005 -0600, Bruce Eckel wrote:
>Yes, defining an class as "active" would:
>1) Install a worker thread and concurrent queue in each object of that
>class.
>2) Automatically turn method calls into tasks and enqueue them
>3) Prevent any other interaction other than enqueued messages

#3 is the sticky bit.  Enforcing such restrictions in Python is 
*hard*.  Each active object would have to have its own sys.modules, for 
example, because modules and classes in Python are mutable and use 
dictionaries, and modifying them from two thread simultaneously would be fatal.

For built-in types it's not so bad, so they could be shared...  except for 
the fact that they have reference counts, which need lock protection as 
well to avoid fouling up GC.  So really, this idea doesn't really help with 
GIL removal at all.

What it *does* help with, is effective use of multiprocessor machines on 
platforms where fork() is available, if the API works across processes as 
well as threads.


>So yes, the best way for this to work might be some kind of enforced
>implementation -- but forcing you to do something is not particularly
>Pythonic,

But keeping the interpreter from dumping core due to program bugs *is* 
Pythonic, which is why the GIL would need to stay.  :)


>so I would think that it would be possible to build an
>active object framework along with some checking tools to warn you if
>you are breaking the rules.

Well, you could pickle and unpickle the objects you send from one function 
to another, and for cross-process communication, you'll need to do 
something like that anyway, or else use that shared-memory objects 
thing.  PySHM?  I don't remember its name, but it's an extension that lets 
you store Python objects in shared memory and use them from multiple 
processes, modulo certain strict limitations.


>  But what's not clear is whether this would
>require knowledge of the innards of the Python interpreter (which I
>don't have) or if it could be built using libraries.

If you're not trying to get rid of the GIL or truly "enforce" things, you 
can do everything you need for CSP in plain old Python.  peak.events has a 
generator-based microthread facility that's CSP-ish, for example, although 
it doesn't have decorators for async methods.  They're not hard to write, 
though; Chandler has some for methods that get called across thread 
barriers now.  That is, methods that get called from Chandler's UI thread 
but are run in the Twisted reactor thread.  I've occasionally also thought 
about implementing async C#'s "chord" features in Python for peak.events, 
but I haven't actually had a use case that needed that much generality yet.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Active Objects in Python

2005-09-27 Thread Bruce Eckel 
According to this list's welcome message, I should introduce myself.
I'm Bruce Eckel, I write, consult and give seminars about computer
programming, I use Python whenever I can get away with it, and I've
spoken at Pycon a number of times. There are further URLs in my
signature at the bottom.

I'm joining this list because I was having a conversation with Guido
about concurrency support in Python (in particular, using an
actor/active-object approach) and he suggested I bring it over
here. Here are the highlights of my most recent reply to him (Guido's
remarks are marked by the '>' signs):

> Agreed. IMO the pthreads-style solution doesn't work well no matter
> *what* the programming model.

Exactly. I think the problem may be that low-level threads people are
one step behind (much like embedded systems programmers, which is
where I began). They may be just now catching up to objects, but as
far as concurrency goes their brains still think in terms of threads,
so it seems natural to apply thread concepts to objects.

But from an OO standpoint, pthread-style thinking is two steps
backwards. You effectively throw open the innards of the object that
you just spent time decoupling from the rest of your system, and the
coupling is now unpredictable. It's the worst of all possible worlds.

> I worked on Mobile Agents at CNRI from 1995 till 2000, but they were
> very heavy-weight. Yet, I think that any solution that moves objects
> between CPUs on a regular basis is going to suffer the same problem
> that process switching in general gets you -- the move is so expensive
> that it's hard to decide when it's justified.

The examples I've seen haven't relied on mobility, so that's a
question: how important is mobility really to an agent system? And
could it be done in a trivial fashion in Python, by sending source
code. And if it was really necessary sometimes, could it be something
where the cost of mobility only occurred when moving an object?

It's possible that either an argument can be made against mobility
and/or some kind of trivial mobility system could be developed that
would work when necessary. Possibly a Linda-like put-take system with
pickles (off the top of my head).

I don't know that much about mobility, but it does seem like mobile
agents could be a powerful solution to the problem solved by
enterprise messaging systems.

> I believe you pointed me to Active Objects before, right? (Years ago,
> maybe when you visited us at Zope.)

I may have mentioned them in the past.

> If multiple active objects can co-exist in the same *process*, but
> nevertheless the language implementation prevents them from sharing
> data except via channels, and in addition allows dynamic reallocation
> of active objects across multiple CPUs, they just might be the ticket.
> But it would require a complete implementation to prove it.

Yes, defining an class as "active" would:
1) Install a worker thread and concurrent queue in each object of that
class.
2) Automatically turn method calls into tasks and enqueue them
3) Prevent any other interaction other than enqueued messages

At that point, the only time the GIL might be needed is to lock the
queue at the point of putting ant taking objects. But there has been
work done on lock-free data structures, which has been incorporated
into Java 5 libraries, so it might even be possible to use a lock-free
queue to do this and thus eliminate the need for the GIL at all.

Since the enqueuing process serializes all requests to active objects,
and since each message-task runs to completion before the next one
starts, the problem of threads interfering with each other is
eliminated. Also, the enqueuing process will always happen, so even if
the queue blocks it will be for a very short, deterministic time. So
the theory is that Active Object systems cannot deadlock (although I
believe they can livelock).

So yes, the best way for this to work might be some kind of enforced
implementation -- but forcing you to do something is not particularly
Pythonic, so I would think that it would be possible to build an
active object framework along with some checking tools to warn you if
you are breaking the rules. But what's not clear is whether this would
require knowledge of the innards of the Python interpreter (which I
don't have) or if it could be built using libraries.

BTW: I think that Hoare's Communicating Sequential Processes (CSP)
basically describes the idea of active objects but without objects.
That is, I think active objects is CSP applied to OO.

Bruce Eckelhttp://www.BruceEckel.com
Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e"
Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com