On Monday 08 June 2009 18:38:40 Gloria W wrote:
> I am still, however, stuck on the last series of messages regardin
> interprocess communication. It wasn't clear to me if you would support the
> Python 2.6 processing module,
At some point, yes. (Hopefully sooner rather than later, but I don't want to
pin a date on it.)
> or if your interprocess communication is DIY.
For interprocess communication inside a system (ie single source file) you use
one of these from Axon.experimental.Process :
* ProcessGraphline
* ProcessPipeline
* ProcessPipelineComponent
* ProcessWrapComponent
Under the hood these are implemented using Paul Boddie's pprocess library from
here:
* http://pypi.python.org/pypi/pprocess
These all allow you to use multiple processes (and hence processors), because
in the words of Paul's description, pprocess:
"The pprocess module provides elementary support for parallel
programming in Python using a fork-based process creation model in
conjunction with a channel-based communications model implemented
using socketpair and poll."
Under the hood, this boils down to:
* os.fork for splitting into multiple processes
* os.pipe's for communication between them
* cPickle for passing objects between processes.
This naturally allows a container component to fork to distribute
subcomponents to other processes.
> Can you please point me to some more recent examples of IPC in Kamaelia?
It's important to realise something here. This allows you to put *ANY*
components in these containers. As a result, almost _any_ Kamaelia system is
a candidate for demonstrating this. For example taking this from yesterday:
Pipeline(
Textbox(size = (800, 300), position = (100,380)),
InterpreterTransformer(),
TextDisplayer(size = (800, 300), position = (100,40)),
).run()
Turning this into a system that uses 4 processes (one container + 3 sub
processes), is as simple as this:
from Axon.experimental.Process import ProcessPipelineComponent
ProcessPipelineComponent(
Textbox(size = (800, 300), position = (100,380)),
InterpreterTransformer(),
TextDisplayer(size = (800, 300), position = (100,40)),
).run()
I've checked this in here:
http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Kamaelia/Examples/PythonInterpreter/MultiProcessPygamePythonInterpreter.py
The proof of this is naturally the fact that pygame is limited to one SDL
window per process at present and of course this opens 2 such windows.
It's worth mentioning though that the process behaviour of this example is
exactly equivalent to a normal unix pipeline. (each part in it's own sub
process, communicating serialised data over an os.pipe).
... and just like a normal unix pipeline, the components *in* the pipeline
neither know, or nor need to care about this.
> This is the most common question asked of me when I speak about Kamaelia
> publicly.
I do agree we need more examples in this area, but once you have the model
right, this does mean that every example is an example of IPC...
I doubt this is a very satisfactory answer though (unless it causes a
lightbulb moment... :) so I will say this: I'm likely to get this question
myself in just under 3 weeks time at Europython since I'll be giving a
Kamaelia tutorial there (in Birmingham UK), so I'll try to provide a nicer
example for it there, and hence for here on the list too.
There isn't however an example at present for this example:
* Multiple machines come together in a server farm
* Register themselves as processing slaves
* Then a system is able to distribute their work over this system.
... with the primary reason being *I don't have a server farm* :-)
This would probably make a nice sprint to do at some point, since we have
most of the components you'd need to make this happen. (eg multicast your
presence on the network, allows nodes to autodiscover each other, then
perform some validation of nodes, and then use a model a bit like
ProcessPipeline, but linked in with the PythonInterpreter code build a job
server/handler.) Drawing a diagram would probably a good starting point for
that :)
Thanks for reminding me that it does need addressing, reminders of the
questions that are most often asked are very useful! :)
Regards,
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamaelia@googlegroups.com
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~----------~----~----~----~------~----~------~--~---
