RE: [kamaelia-list] Using Kamaelia in Python Qt applications.
Hi Jim,
Many thanks for this - I'll add this to the github repository shortly.
A few questions below on namespaces/naming, and how to use etc. (based on
reading the docs!)
In terms of slotting this into the namespace, I presume that putting into here:
Kamaelia/UI/Qt.py
Would make sense? ie allowing use like:
- from Kamaelia.UI.Qt import background, KamaeliaQt
Sketching out from the docs in the file, it looks like the hypothetical video
player application would look like this:
from MyPackageOfQTThings import FileSelector
from BigBucketKamaeliaComponents import Pipeline, FileReader, MPEGDecoder
import Kamaelia.UI.Qt # To be explicit below
from Kamaelia.UI.Qt import KamaeliaQt
class Display(ProbablyAQTThing):
...
def new_frame(self, data):
"display the frame"
#exercise for reader
Kamaelia.UI.Qt.background().start()
display = Display(...)
file_selector = FileSelector(...)
pipe = KamaeliaQt(
Pipeline(
FileReader(),
MPEGDecoder(),
)
pipe.signals['outbox'].connect(display.new_frame)
file_selector.new_file.connect(pipe.slots['inbox'])
pipe.activate()
pipe.slots['inbox']('~/Videos/TheMatrix.mpg')
Does that make sense?
This is a little unclear though - if I'm reading this right it does the
following:
Start the axon scheduler under Qt's command.
Create a display
Create a file selector
Create a file reading component - that when it recieves a filename opens that
file for reading and sending data to it's outbox.
Creates an MPEG Decoder that expects raw file / mpeg data on its inbox, and
spits out video frames to it's outbox, and hence the pipeline's outbox.
The arrival of a frame on the pipeline's outbox results in the data being
passed to the new_frame method of the display component, resulting in it being
displayed.
The file_selector is wired in, such that the selection of a file results in a
filename being sent to the pipeline, via the KamaeliaQT wrapper.
As a default starting file, the system starts up reading the hypothetical file
'~/Videos/TheMatrix.mpg' . However, once that completes, the next file can be
selected via the file selector, resulting in the FileReader starting to read
that new file, and therefore playing back the next video.
Is that all more or less correct?
I guess that you could actually make the filereader more intelligent, and only
read/playback based on information in the file, and furthermore change the file
its reading based on a new message, but that's all random thinking that's
irrelevant to the logic of the component.
If it does, I'll add the component to the github repository later there.
Regards,
Michael.
--
| Michael Sparks, Senior Research Engineer
| BBC R&D, 5th Floor, Dock House, MediaCityUK, Salford, M50 2LH
| michael.spa...@bbc.co.uk, 03030409510, http://www.bbc.co.uk/rd/
From: kamaelia@googlegroups.com [kamaelia@googlegroups.com] on behalf of Jim
Easterbrook [jim.easterbr...@bbc.co.uk]
Sent: 28 November 2012 09:44
To: kamaelia@googlegroups.com
Subject: [kamaelia-list] Using Kamaelia in Python Qt applications.
As part of my recent work (see
http://www.bbc.co.uk/rd/publications/whitepaper191.shtml for details)
I've written code to connect Kamaelia components' outboxes and inboxes
to Qt signals and slots. I'd like to make this available to all Kamaelia
users, so here it is. (Michael, can you add it to the Kamaelia repos?)
There are two parts - a background scheduler and a 'wrapper' component.
The background scheduler is a simple replacement for Kamaelia's
background scheduler that uses the Qt event loop to schedule Kamaelia
processes. The wrapper can be passed any Kamaelia component (such as a
Pipeline) which it adopts as a child. All the component's inboxes and
outboxes (apart from names beginning with an underscore) are connected
to Qt slots and signals. This allows objects to be passed between
Kamaelia and Qt very easily.
--
Jim Easterbrook
Senior Research Engineer
BBC Research & Development South Lab
BBC Centre House
56 Wood Lane
London W12 7SB
Phone: +44 303 040 9705
Mobile: +44 7590 307002
FAX:+44 20 8811 8815
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamaelia@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
-
http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and
may contain personal views which are not the views of the BBC unless
specifically stated.
If you have received it in
error, please delete it from your system.
Do
[kamaelia-list] Re: TwoWaySplitter and string exception
Hi, On Jul 15, 2:28 pm, lgiordani wrote: > Since using Util/TwoWaySplitter.py results in > > raise "STOP" > TypeError: exceptions must be old-style classes or derived from > BaseException, not str This was fixed on /trunk in June 2009: http://code.google.com/p/kamaelia/source/diff?spec=svn6103&r=6103&format=side&path=/trunk/Code/Python/Kamaelia/Kamaelia/Util/TwoWaySplitter.py&old_path=/trunk/Code/Python/Kamaelia/Kamaelia/Util/TwoWaySplitter.py&old=3882 I'm guessing therefore that you don't know we have a newer (stable) release Kamaelia-1.0.12.0.tar.gz : - - http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.12.0.tar.gz There's also a comprehensive tutorial these days as well here: - http://www.kamaelia.org/Europython09/A4KamaeliaEuroPython09.FINAL.pdf The website needs a fair amount of TLC, of which finding time has been hard, but the doc strings are up to date, as are examples, and that tutorial PDF :-) > [patch] Patches like this are always very much appreciated! > Since this is my very first post on this list I do not know if there > is a bug tracker with patch management active. I prefer to have bugs / patches to be sent to the list like this, so this is fine. I'm thinking about how to revamp Kamaelia at the moment, so suggestions are welcome - both technologically, website wise and process wise. In case that's worrying, please be assured that I think breaking backwards compatibility is a bad idea. Best Regards and thanks! Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Axon: Python 3 Support
Hi, I've spent this evening working through Kamaelia's core - ie through Axon - and have converted all of it, including the examples - to be valid python 3 code *and* valid python 2 code. This includes the test suite, and the Axon test suite passes under both python 2 and python 3. This is the second necessary/important step for supporting python 3 taken in recent months. (the first was switching from Pyrex to Cython - which claims it can generate python 3 extension modules as well as python 2) Anyway, I'm feeling rather pleased with this, and I've bumped the version number of Axon to 1.7. If there's anyone interested in playing with *just* Axon 1.7, then it can be grabbed from here: * http://www.kamaelia.org/release/Axon-1.7.0.tar.gz There's been a few little additions to make things simpler for python3 support, which leverage off this fact: try: import Queue as queue version = 2 except ImportError: import queue version = 3 Based on this there's some definitions in Axon.util for the following: * vrange is an alias for xrange under python 2 and an alias for range under python 3 * next(X) is a wrapper around x.next() in python and an alias for next(X) in python 3 Other: * g.next() becomes either g.__next__() or next(g) Exceptions handling: * Syntax between 2 and 3 is different In python 2 you put except Exception, e: In python 3 you put except Exception as e: In order to work with both python 2 AND python 3 you put except Exception: e = sys.exc_info()[1] The exception handling is a little mucky as a result IMO, but it does mean you get to use a single codebase. Also, I use the import trick above to handle the syntactic difference between python 2 and 3 for metaclasses for AxonType/AxonObject (which should probably be ditched now anyway really, thinking about it). Anyway, as noted, code here: * http://www.kamaelia.org/release/Axon-1.7.0.tar.gz Have fun, Michael -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: Kamaelia's Next Release - 1.1.2.0 or 1.1.1.0 ?
On Dec 30 2010, 1:53 pm, Michael Sparks wrote: > Hi, > > Just to note that I'm planning the next release at the moment, and I'm > planning on the following stuff: > > * Finishing off the Dirac 1.0 bindings. > * Finishing Packaging up the Macro work as a standalone > application > * Cherry Picking, fixing and merging the "Jam" work as a > standalone app (insofar as possible). > > This relates to the following 3 branches: > * private_MPS_Dirac1_0_Update/ > * private_MPS_Macro/ - this will probably end up having a name > change at some point too, and I'd be interested in suggestions > * private_JT_JamDev/ > > If anyone has any code they wish to contribute, feedback or plain help > out with, it's very much appreciated. This can be anything from code > related, through docs, through to help with the website. (I've got > some plans for improving the website, but the traditional problem of > only 1 pair of hands!) > > That said, if there's any specific feature requests that people have > as well, now is the time to ask (in the next week or two). ... > Anyway, I'm not sure on timeline for the next release at the moment, > but aside from a december release it seems that a bi-monthly release > schedule works quite well. Given the versioning of Y.Y.M.r, this would > suggest that the next release would be 1.1.2.0. No replies, so I'll make my life easy and go for even numbered releases this year :-) First release this year will be 1.1.2.0 then. Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Website changes - beta.kamaelia.org -> www.kamaelia.org
On Thursday 30 Dec 2010 13:53:33 Michael Sparks wrote: > through to help with the website. (I've got > some plans for improving the website, but the traditional problem of > only 1 pair of hands!) As a secondary note to this, I've today shifted the beta site over to being the main kamaelia site. That is http://beta.kamaelia.org/ has been shifted over to http://www.kamaelia.org/ . If anyone notices any problems with this, please do let me know. Whilst this covers a fair amount of thought and planning, this should actually either result in no change or less than noticable improvement. This is primarily due to the change being about making maintenance and future improvment simpler, rather than anything else! As a bit more detail, as to what's been done... Part of these plans have been tested for a while, and revolve around putting the website back into SVN recognising the whole "let's edit the website online" idea was more of a pain than a benefit. Seemed like a good idea. It wasn't so I'm changing to something better. I placed pretty much the whole website into subversion, here: * http://code.google.com/p/kamaelia/source/browse/website/ In terms of a subversion repository, it means the root of the subversion repository looks like this: # branches # mirror (contains a copy of http://code.google.com/p/kamaelia-aws/) # tags # trunk * Website # website ( /trunk/Website will be restired/deleted at some point ) Inside website, you'll find the following structure: website * as_published - Website as published at present on www.kamaelia.org, for better or worse. This isn't ideal by any stretch of imagination. * source - This is currently empty, however, the aim is to shift copies of "as_published" into this directory such that "as_published" then becomes generated code from this source directory. That means there'll be templates etc in here. My aim is to change the website source over to something like markdown, so that it can be published based on that. (rather than the custom stuff I had before) * docs - This is the actual docroot for the website. It actually contains the following: ./release -> ../downloads/release/ ./index.html ./.htaccess This isn't likely to be extended. This means that the website itself is served via a 404 handler, which means it can be made to shift over from using "as_published" to using "source" at some point, or even replaced completely with something else. In particular this means that the 404 handler can undo the mistake of /Home vs /Home.html. The former is a clean URL (perhaps) but the latter can he handled statically by a random hosting in case of an emergency (The former can't). The .htaccess sets the 404 handler and sets up a redirect to /Home.html for /index.html for the one file that the 404 handler won't kick in for. * cgi - Contains a simple 404 stub handler. If someone asks for /Home.html for example, it serves "as_published/Home.html". For legacy links, if sent a request for /Home, it looks in "as_published" for "Home" first and "Home.html" second. - Also contains a simple support library kamweb.py * downloads - Contains releases/etc. Generally linked into the main site. Whilst lots of (good) arguments could be put in place for using something like django and friends these days, my primary aim here is to NOT break existing links, and fix some which are currently bust, whilst at the same time trying to: a) Make the website more maintainable b) Maintainable by people than just me. (Since even it just remains me, this will make my life simpler anyway :-) Any suggestions for improvement, etc, gratefully taken. Offers to make it happen also very welcome. Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Kamaelia's Next Release - 1.1.2.0 or 1.1.1.0 ?
Hi, Just to note that I'm planning the next release at the moment, and I'm planning on the following stuff: * Finishing off the Dirac 1.0 bindings. * Finishing Packaging up the Macro work as a standalone application * Cherry Picking, fixing and merging the "Jam" work as a standalone app (insofar as possible). This relates to the following 3 branches: * private_MPS_Dirac1_0_Update/ * private_MPS_Macro/ - this will probably end up having a name change at some point too, and I'd be interested in suggestions * private_JT_JamDev/ If anyone has any code they wish to contribute, feedback or plain help out with, it's very much appreciated. This can be anything from code related, through docs, through to help with the website. (I've got some plans for improving the website, but the traditional problem of only 1 pair of hands!) That said, if there's any specific feature requests that people have as well, now is the time to ask (in the next week or two). Heck if someone wants to come in and take over one of these bits because they think they can do it or work on stablising or extending any of the existing apps, that's also very welcome. I'd find it quite fun to see someone scavenge a simple video recording system based on ripping code out the whiteboard, and perhaps autosegments it using the shotchange detection code, but a simple direct to disk video recorder would be fun. Just say what you'd like to do, I'll create you a space in /Sketches with your initials on it, and let you know how to go from random exploration in /Sketches to deployable code in /trunk via hand off in / branches. :-) Anyway, I'm not sure on timeline for the next release at the moment, but aside from a december release it seems that a bi-monthly release schedule works quite well. Given the versioning of Y.Y.M.r, this would suggest that the next release would be 1.1.2.0. Either way, in advance of the new year - Happy New Year :-) Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: Merge of branch private_MPS_CL_Topology3D (nee private_CL_Topology3D)
Hi Gloria, On Dec 27, 9:29 pm, Gloria W wrote: > This sounds awesome, thank you for doing this! I hope to get some time > in the next three months to actually play with this. The FOAF viewer in > particular is really handy. It would be so great to bind such a tool to > Diaspora. Sounds like a fun idea, and thanks for the kind words :-) The thing interesting about the 3D visualisation is that unlike the 2D graph visualisation is that it's actually nested, which opens up some interesting options. Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Kamaelia 1.0.12.0 released
Hi, I've built the 1.0.12.0 release and linked it from the website front page. This is the 4th and final release of 2010. I think it also represents a sustainable rhythm of development - bi-monthly releases, with work primarily on branches, and new work primarily inside applications which can be merged, and the components from apps slowly merged into the main namespace. In particular it provides a good balance between "I need to write this asap to solve a problem I have" and "I need to merge this in a stable way against mainline for reuse". Direct link: * http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.12.0.tar.gz This rolls up (primarily) 3 application and examples branches. The core functionality for these, as ever, is in the main Kamaelia.Apps namespace, meaning these applications and examples are designed for inclusion or extraction into other applications relatively easily. As a result they act as exemplars for things like 3D visualisation, video and audio communications, twitter mining, database interactionn and analysis and django integration. They're also useful (and used) as standalone apps in their own right. Overview of Changes in 1.0.12.0 * Examples (and application components) added for using the 3D graph visualisation - one based on visualising collaborations, another based on viewed FOAF networks. * Whiteboard application extended such that: * It supports multiway video comms as well as multiway audio comms. * Adds support for "decks" (collections of slides which can be downloaded, saved, loaded, emailed, encrypted, etc) * Removes pymedia dependency * Change audio over to us PyAlsaAudio directly. * Adds support for calibrated touch screen displays to Pygame Display. - For example large digital whiteboards in addition to existing tablets etc. * Adds in a "Social Bookmarking system" that does the following: * Harvests a semantic web/RDF data store for realtime search terms (relating to live television broadcast) * Uses these search terms to search twitter, to identify conversations around the semantic web data. * Takes the resulting tweets, and stores them in a DB * Analyses the tweets (including fixing language for analysis using NLTK) for a variety of aspects, storing these in the DB * Presents the results (graphs of buzz/popularity around the content) * Additionally the system attempts to identify particularly interesting/notable moments based on audience conversations, and provides links back to the actual broadcast programmes. * Additionally provides an API for data, generates word clouds etc. * Front end uses Django and web graph APIs to presnet data. Changes not described elsewhere === Kamaelia-1.0.12.0/README * Revision Bumped for 1.0.12.0 release Kamaelia-1.0.12.0/AUTHORS * Added in Chong Liu who should've been there already (not sure why he wasn't there) Kamaelia-1.0.12.0/Kamaelia/setup.py * Revision Bumped for 1.0.12.0 release * Added in namespace for Kamaelia.Apps.CL * Add in namespaces for applications/examples: Kamaelia.Apps.CL.CollabViewer, Kamaelia.Apps.CL.FOAFViewer * Added in namespace for Kamaelia.Apps.SocialBookmarks Kamaelia-1.0.12.0/Kamaelia/Kamaelia/Support/DVB/Descriptors.py * Spelling fix, based on feedback from "Terence" on the mailing list Changes which have more detail described in the mailing list archives: == * Merge of private_MPS_CL_Topology3D (nee private_CL_Topology3D) http://groups.google.com/group/kamaelia/browse_frm/thread/8b79e79ae8ca6b25 * Merge of branch private_AB_Whiteboard http://groups.google.com/group/kamaelia/browse_frm/thread/aca7ebdf4176dd40 * Merge of branch private_AB_SocialBookmarks http://groups.google.com/group/kamaelia/browse_frm/thread/f7b818fafe8099f6 Overview of merged branches === Merged Branch: private_MPS_CL_Topology3D (nee private_CL_Topology3D) This is a collection of components and examples using the 3D graph visualisation code, and was the hardest part of Chong Liu's code to merge. It is however now merged :-) (Seems that this is the season I find it easiest to find time to merge things :-) In particular, this adds two non-trivial examples: * "CollabViewer" - which is a hypothetical example based on visualising collaborations based on orgs or projects. It shows how to switch between 2 visualisations which both contain nesting. * FOAFViewer - visualises FOAF networks. Directories added: Kamaelia-1.0.12.0/Kamaelia/Examples/GraphVisualisation: CollabViewer Kamaelia-1.0
[kamaelia-list] Merge of branch private_MPS_CL_Topology3D (nee private_CL_Topology3D)
Hi,
Just a note to say that I've merged branch private_MPS_CL_Topology3D.
This was a more or less complete restructure of the branch
private_CL_Topology3D.
This is a collection of components and examples using the 3D graph
visualisation code, and was the hardest part of Chong Liu's code to
merge. It is however now merged :-)
(Seems that this is the season I find it easiest to find time to merge
things :-)
In particular, this adds two non-trivial examples:
* "CollabViewer" - which is a hypothetical example based on
visualising collaborations based on orgs or projects. It shows
how to switch between 2 visualisations which both contain
nesting.
* FOAFViewer - visualises FOAF networks.
The details of the merge can be found here:
* http://code.google.com/p/kamaelia/source/detail?r=7264
Actual examples have been added here:
* /trunk/Code/Python/Kamaelia/Examples/GraphVisualisation/
CollabViewer/CollabViewer.py
The collaboration file example.
* /trunk/Code/Python/Kamaelia/Examples/GraphVisualisation/
CollabViewer/Data/collab.json
Data file for the collaboration example
* /trunk/Code/Python/Kamaelia/Examples/GraphVisualisation/
FOAFViewer/FOAFViewer.py
The FOAF viewer example
App support code can be found here:
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/__init__.py
Base file to allow import
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/CollabViewer/
__init__.py
Base file to allow import
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/FOAFViewer/
__init__.py
Base file to allow import
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/JSON.py
Adds JSON serialisation/deserialisation components:
* JSONEncoder
* JSONDecoder
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/DictChooser.py
Adds a "DictChooser" component that can be used as follows:
imageOptions = { "image1" : "image1.png",
"image2" : "image2.png",
"image3" : "image3.png" }
Graphline( CHOOSER = DictChooser(options=imageOptions),
image1 = Button(position=(300,16), msg="image1",
caption="image1"),
image2 = Button(position=(16,16), msg="image2",
caption="image2"),
image3 = Button(position=(16,16), msg="image3",
caption="image3"),
DISPLAY = Image(position=(16,64), size=(640,480)),
linkages = { ("image1" ,"outbox") :
("CHOOSER","inbox"),
("image2","outbox") :
("CHOOSER","inbox"),
("image3","outbox") :
("CHOOSER","inbox"),
("CHOOSER" ,"outbox") :
("DISPLAY","inbox"),
}
).run()
In this example, the chooser is driven by the 'image1', 'image2'
and
'image3' Button components. Chooser then sends filenames to an
Image
component to display them.
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/
SimpleFileWriterWithOutput.py
Adds a subclass of SimpleFileWriter that outputs the name of the
file being written.
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/CollabViewer/
CollabParsing.py
Contains components - CollabParser, CollabWithViewParser
CollabParser
Kamaelia component to parse collaboration data between
organizations received as dictionary.
CollabWithViewParser
Kamaelia component to parse collaboration data between
organizations received as dictionary into different views'
TopologyViewer commands.
* /trunk/Code/Python/Kamaelia/Kamaelia/Apps/CL/FOAFViewer/
RDFParsing.py
Contains components - RDFParser
A component to parse RDF data received from a uri to
TopologyViewer3D command.
Regards,
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Merge of branch private_AB_Whiteboard
Hi, Just a note to say that I've merged branch private_AB_Whiteboard. This is again a branch by Andrew Bonney and modifies the whiteboard code in the following way: * Adds support for "decks" (collections of slides which can be downloaded, saved, loaded,emailed etc) * Removes pymedia dependency * Change audio over to us PyAlsaAudio directly. * Adds in multiway video support. * Adds support for calibrated touch screen displays to Pygame Display. - For example large digital whiteboards in addition to existing tablets etc. The actual merge details are here: * http://code.google.com/p/kamaelia/source/detail?r=7231 I'd expect some future work to be to re-architect the whiteboard to pull out the obvious apps as standalon apps as well as the merged composite app. Details of modifications (along with some todos which were not considered blockers - primarily because they're inside the app, not inside the mainline): Files modified in the main area: * /trunk/Code/Python/Kamaelia/Kamaelia/UI/Pygame/Display.py Added support for calibrated displays * /trunk/Code/Python/Kamaelia/Kamaelia/UI/Pygame/Ticker.py Adds support for controlled clearing with "out of band" messages. Distribution Modifications: * /trunk/Code/Python/Apps/Whiteboard/App/Decks Default directory for decks * /trunk/Code/Python/Apps/Whiteboard/App/README.txt Basic Decks * /trunk/Code/Python/Apps/Whiteboard/App/Whiteboard.py Contains the modifications * /trunk/Code/Python/Apps/Whiteboard/App/whiteboard.conf.dist New default configuration * /trunk/Code/Python/Apps/Whiteboard/DistBuild/setup.py.src Version, license & authors updated Files added in the mainline of the whiteboard app: * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Decks.py Adds initial support for decks. Decks are collections of whiteboard slides. They can be loaded, saved, etc. * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Email.py Support for emailing decks. * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Webcam.py Webcam support (derived from the Europython tutorial) * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/ProperSurfaceDisplayer.py Webcam playback support * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Record.py Alsa Audio recording support * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Play.py Alsa Audio playback support * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/SmartBoard.py Initial support for directly supporting smartboards. (nonfunctional) Files modified in the mainline of the whiteboard app: * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/__init__.py License block updated * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Canvas.py Transparency modified to enable it to work better with open gl display Changes Pygame Display caption to Kamaelia Whiteboard Adds in support for a "NEW" message as well as "CLEAR" - via a "remotenew" message Fileloading tidied up TODO: imports need cleanup admittedly, due to some changes in deck handling ] TODO: Shutdown handling needs improving TODO: Caption needs to be set via Kamaelia.UI.Pygame.Display rather than directly * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/CheckpointSequencer.py Modified to support deletions. TODO: Modify to remove decks dependency, and change back to callbacks for deletions as well * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/TwoWaySplitter.py Whitespace change * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/UI.py Capitalisation of buttons Graphlines modified to change usage of "" to "self" Added buttons for SaveDeck, LoadDeck, ClearScribbles, Delete, Quit Modifications that were on the branch that would make sense to revert: * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Entuple.py Shutdown handling change TODO: revert back * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Painter.py Shutdown handling change TODO: revert back * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Router.py Shutdown handling change TODO: revert back * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/Routers.py * /trunk/Code/Python/Kamaelia/Kamaelia/Apps/Whiteboard/TagFiltering.py Shutdown handling change TODO: revert back Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Merging Branch: private_AB_SocialBookmarks
Hi, As an FYI, I'm merging the above branch. It's a branch of work by an Industrial Trainee at work. It contains an application, the purpose of which was to answer these two questions: * Is there enough data in a social network such as twitter to identify what programmes the audience are watching and how interesting (engaging) they find the programme from moment to moment? * If "yes", can you identify bookmarkable moments in a programme for the audience to be able to look at and watch at a later point in time? More detail on findings will be published elsewhere as time goes on. Functionality wise, it does the following: * Grabs semantic web/RDF data from bbc.co.uk/programmes * Uses the pristine data therein to generate - a collection of normal search terms - a collection of people search terms * Sends those searches to twitter, over the streaming API * Waits for results (tweets) * Correlates results back to search terms (and hence to programmes) * Maps these back to programme time using the information from /programmes and a broadcast bridge (which I hope to talk about at some point, but isn't public). In the absence of a broadcast bridge, it just uses /programmes data. * This can then be used to analyse the data, and give tweet volumes per minute (peaking for example at around 2000 tweets/min) and generate: - Bare volume per minute graphs - word frequency analysis and tag cloud of what people are saying The graphs give a measure of how much the audience reacted to a moment in a programme (which does correlate with actual programme events) and an attempt at identifying bookmarks is also made. The data is published via a django frontend. Code files to be committed: * Kamaelia/setup.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/TweetFixer.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/LiveAnalysis.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/BBCProgrammes.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/PureFilter.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/DataCollector.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/__init__.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/Requester.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/TwitterStream.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/URLGetter.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/ComponentBoxTracer.py * Kamaelia/Kamaelia/Apps/SocialBookmarks/TwitterSearch.py * Apps/SocialBookmarks/Doc/README * Apps/SocialBookmarks/Config/twitter-login.conf.dist * Apps/SocialBookmarks/Config/README * Apps/SocialBookmarks/Config/django.wsgi.dist * Apps/SocialBookmarks/Config/settings.py.dist * Apps/SocialBookmarks/Config/twitter_bookmarks_schema.sql * Apps/SocialBookmarks/App/web/django.wsgi.dist * Apps/SocialBookmarks/App/web/bookmarks/urls.py * Apps/SocialBookmarks/App/web/bookmarks/settings.py.dist * Apps/SocialBookmarks/App/web/bookmarks/api/urls.py * Apps/SocialBookmarks/App/web/bookmarks/api/handlers.py * Apps/SocialBookmarks/App/web/bookmarks/api/__init__.py * Apps/SocialBookmarks/App/web/bookmarks/__init__.py * Apps/SocialBookmarks/App/web/bookmarks/output/models.py * Apps/SocialBookmarks/App/web/bookmarks/output/tests.py * Apps/SocialBookmarks/App/web/bookmarks/output/__init__.py * Apps/SocialBookmarks/App/web/bookmarks/output/views.py * Apps/SocialBookmarks/App/web/bookmarks/output/admin.py * Apps/SocialBookmarks/App/web/bookmarks/manage.py * Apps/SocialBookmarks/App/web/media/channels/worldservice.gif * Apps/SocialBookmarks/App/web/media/channels/6music.gif * Apps/SocialBookmarks/App/web/media/channels/asiannetwork.gif * Apps/SocialBookmarks/App/web/media/channels/bbcparliament.gif * Apps/SocialBookmarks/App/web/media/channels/bbchd.gif * Apps/SocialBookmarks/App/web/media/channels/cbeebies.gif * Apps/SocialBookmarks/App/web/media/channels/1xtra.gif * Apps/SocialBookmarks/App/web/media/channels/sportsextra.gif * Apps/SocialBookmarks/App/web/media/channels/cbbc.gif * Apps/SocialBookmarks/App/web/media/channels/bbcone.gif * Apps/SocialBookmarks/App/web/media/channels/radio7.gif * Apps/SocialBookmarks/App/web/media/channels/5live.gif * Apps/SocialBookmarks/App/web/media/channels/bbcnews.gif * Apps/SocialBookmarks/App/web/media/channels/radio4.gif * Apps/SocialBookmarks/App/web/media/channels/bbcfour.gif * Apps/SocialBookmarks/App/web/media/channels/radio3.gif * Apps/SocialBookmarks/App/web/media/channels/radio2.gif * Apps/SocialBookmarks/App/web/media/channels/bbctwo.gif * Apps/SocialBookmarks/App/web/media/channels/radio1.gif * Apps/SocialBookmarks/App/web/media/channels/bbcthree.gif * Apps/SocialBookmarks/App/web/media/bbclogo.gif * Apps/SocialBookmarks/App/Bookmarks.py Additionally, the branch contai
Re: [kamaelia-list] Backplane notifications
Hi,
On 2 December 2010 15:14, Jim Easterbrook wrote:
> How practical would it be to add a 'notification' service to the
> backplane component? What I want is for a component of mine to be woken
> up when a subscriber is added or removed, e.g. to stop doing any work
> when the backplane has no subscribers.
What sort of interface would you want ? Something like this?
Pipeline(
Backplane("MYBACKPLANE"),
MyComponentThatGetsNotificationsWhenNewClients(),
)
That in itself leads rather naturally to:
Pipeline(
SendNotificationsToBackplane(),
Backplane("MYBACKPLANE"),
MyComponentThatGetsNotificationsWhenNewClients(),
)
Or:
Pipeline(
Backplane("MYBACKPLANE"),
BackplaneMonitor(),
circular = True,
)
(Which would enable BackplaneMonitor to shutdown the Backplane)
This could also lead to the rather recursive:
Pipeline(
SubscribeTo("UpdateBackplane"),
Backplane("MYBACKPLANE"),
PublishTo("ThingsMyBackplanDidToday"),
)
Adding a notification service like this would certainly be doable, so
if you had an API in mind, I'd be curious.
> At the moment I'm sending a message to my component whenever I add a
> subscriber, and using a timer loop to look at the backplane's splitter's
> outboxsinks count every 5 seconds to see if it's dropped to zero. There
> has to be a better way than this.
You could have a notification of:
* New subscriber
* New publisher
* Back to zero subscribers
* Back to zero publishers
(Publishers are actually harder, but should be doable safely now given
the existence of the STM code)
I'm not sure what notifications you might want to send to the
backplane itself though either. (suggestions/comments welcome)
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Kamaelia 1.011.0 Released
Hi, Minor update this time, but I've uploaded the bi-monthly release to the server, updated links etc. Major points: * All bindings have been changed over to use Cython rather than Pyrex. This is primarily because Cython seems to be more practical (things like better warnings, import hooks to enable importing .pyx files directly, etc). The aim here is to update the Dirac bindings incidentally, which is happening on a branch at the moment. (Along with packaging up the kamaelia macro code seperately - at which point I'll probably rename it) * Introspection Tutorial added. * Small but useful improvements to the DVB examples. * TCPClient changed to support (better - it did work before) fire and forget HTTP style requests. Linked from homepage, but also here: http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.11.0.tar.gz Michael. -- Detailed changelist +++ Kamaelia-1.0.11.0/Axon/Axon/ThreadedComponent.py2010-11-26 22:58:30.0 + Remove legacy __future__ import +++ Kamaelia-1.0.11.0/Bindings/Dirac-0.6.0/setup.py 2010-11-19 18:06:01.0 + Changed to build build using Cython rather than Pyrex. This is a bridging change towards updating properly to Dirac 1.0 +++ Kamaelia-1.0.11.0/Bindings/MpegTsDemux/MpegTsDemux.pyx 2010-11-19 18:02:22.0 + Bugfix - detected due to change to Cython +++ Kamaelia-1.0.11.0/Bindings/MpegTsDemux/setup.py 2010-11-19 18:02:22.0 + Use Cython, not Pyrex. +++ Kamaelia-1.0.11.0/Bindings/python-dvb3/dvb3/dmx.pyx 2010-11-19 17:52:26.0 + Bugfix - detected due to change to Cython +++ Kamaelia-1.0.11.0/Bindings/python-dvb3/dvb3/frontend.pyx 2010-11-19 17:52:26.0 + Bugfix - detected due to change to Cython +++ Kamaelia-1.0.11.0/Bindings/python-dvb3/setup.py 2010-11-19 17:52:26.0 + Use Cython, not Pyrex. +++ Kamaelia-1.0.11.0/Bindings/SupportCode/Kamaelia/Support/Optimised/ Kamaelia.Support.Optimised.MpegTsDemux.pyx 2010-11-19 17:57:56.0 + Bugfix - detected due to change to Cython +++ Kamaelia-1.0.11.0/Bindings/SupportCode/setup.py 2010-11-19 17:57:56.0 + Use Cython, not Pyrex. +++ Kamaelia-1.0.11.0/Bindings/vorbissimple/python/setup.py 2010-11-19 18:00:41.0 + Use Cython, not Pyrex. +++ Kamaelia-1.0.11.0/Kamaelia/Examples/DVB_Systems/channels.conf 2010-10-16 12:21:51.0 +0100 Added other channels from the Winter Hill transmitter. +++ Kamaelia-1.0.11.0/Kamaelia/Examples/DVB_Systems/ChannelsConf.py 2010-10-16 12:25:31.0 +0100 Minor improvement to capture TDT and EIT information as well. This allows recordings to be decoded and split up based on programmes easier. (Which is useful for an ad-hoc PVR. It's akin to including exif information in a JPEG IMO). Binary files Kamaelia-1.0.9.0/Kamaelia/Examples/Introspection/ IntrospectionTutorial/01.gif and Kamaelia-1.0.11.0/Kamaelia/Examples/ Introspection/IntrospectionTutorial/01.gif differ Introspection tutorial/overview added. +++ Kamaelia-1.0.11.0/Kamaelia/Kamaelia/Internet/TCPClient.py 2010-10-26 16:06:17.0 +0100 Added "wait_for_serverclose" flag. If true ignores a producerFinished from the component sending data to the connection, and rather waits for the client to break the connection. This avoids sending data to a server and exitting before receiving the response. +++ Kamaelia-1.0.11.0/Kamaelia/setup.py 2010-11-19 18:07:40.0 + Version number bumped. +++ Kamaelia-1.0.11.0/README2010-11-27 21:19:26.0 + Version number bumped. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] yield newComponent(xxx) or xxx.activate() ?
On 5 October 2010 15:48, Jim Easterbrook wrote: > I've been tearing my hair out this afternoon :-( > trying to get > Kamaelia.Internet.Selector.Selector to work. Finally I twigged that the > newly created Selector component was never being activated. I didn't > immediately spot that the TCPClient example code uses 'yield > newComponent()' so its parent component will activate the Selector. > > The component I'm altering to use Selector was a threaded component, so > it didn't have any yield statements. Putting in explicit activation of > the newly created Selector made everything magically spring to life. Now > I've got Selector working I've converted it to a non-threaded component. > > Which brings me on to my question. Is there any reason to prefer > yielding a newComponent message over explicit activation when creating > new child components? .activate() is preferred. yielding a newComponent was viewed as an easy/sensible way of passing back a message to the scheduler in Kamaelia's early days, long before the introduction of threaded components, whilst it still works, it's not the preferred approach. Care is taken to make sure that .activate() is threadsafe, so it can be happily and safely called. Creation doesn't cause activation because it often causes more problems than it's worth. (eg components being created inside a pipeline, starting executing before the linkages have been created) However, components rarely standalone, so .activate() returns the object. This means you can type this: foo = MyComponent().activate() And that has the same effect as this: foo = MyComponent() foo.activate() In threaded components, .activate() starts the main thread running *and* the generator to pump the data queues. For generator based components .activate() adds the component to a Queue.Queue which the scheduler looks inside and adds to an internal runqueue. (Hence why .activate() is threadsafe) The reason that code is still in TCPClient is based on the "if it isn't broke, don't fix it principle". (aka the "life's too short" principle :-) That said, I do periodically go through rip out bad code, and replace with new better idioms. TCPClient predates kamaelia having threaded components. (Indeed the first threaded component was a threaded version of TCPClient that was written for python on Series 60 mobile phones before "select" became available on Series 60 phones) Hope that helps, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] Answered one of my questions
On Monday 04 October 2010 08:23:10 Matt Hammond wrote: > > I tried this again, and found it hadn't changed much in the past couple > > of years. It still needs > > > > a scrolling mechanism, > > iirc arrow keys do this? I think this changed recently when I changed the Pygame Display such that clients have to explicitly ask for event requests. (I'm pretty sure that Kamaelia.UI.MH.PygameApp and hence the topology vis, doesn't do that since it was early on) I'd not sat down and thought the why through yet. In general a zooming mechanism is really the needed thing IMO. (Since then the vis could be based on child components) > > a way to limit the > > > > amount of nested detail, > > The nesting in Kamaelia is not strict containment. A component created by > another component is free to connect to other components not created by > the same 'parent'. For example, a TCP Client or server will create > ConnectedSocketAdaptor components to handle the individual socket > connections. Is this a problem? Something appearing in 2 places is not a problem in practice. My direct equivalent here would be a filesystem. Symbolic links and hard links don't prevent the system being comprehensible, even though the filesystem is a graph rather than a strict tree. (despite the "visualisation" of an FS being a tree ... ?) > These might appear nested, but they use teh CAT service to > instantiate a globally shared Selector component which they all directly > communicate with. Similarly, when you use the backplane publish-subscribe > model, the publish and subscribe components are locating the backplane > component and directly forming linkages with it. Diagrams of backplanes et al could be ... intriguing. > > moves from one component to the next, > > For that to be useful, I guess what's actually needed is a kind of logger > and stepping playback mechanism, or break-point stepping mode? It's possible to switch on dumping of all messages flowing through the system now - much more so than there used to be. Changing this to send it's information to an introspection subsystem rather than "print" is ... a good idea, to say the least. (Providing opportunity for analysis of all sorts) > > and a cleanup mechanism when > > > > components go away. > > Components should disappear when they shut down. If they don't disappear > then that usually implies they're still running. This might be because > they weren't sent a shutdownMicroprocess or producerFinished message to > their "control" inbox. +1 One of the delays in getting that document written was the fact that when doing the pictures, components weren't disappearing correctly when I expected to, meaning the components weren't shutting down and needed fixing. Conversely the recent bug that required adding the single word "break" was noticed by looking at the diagrams because *too much* disappeared at once. > > I am going to try and find a volunteer to help with this task. It would > > be incredibly useful to also have a running audit trail of what > > messages were sent/received by components, in another window, while > > the visualizer is running. This combined with a cleaner visual will be > > so powerful. > > That would be extremely useful and also quite cool to see working :-) It'd be utterly fantastic. Also, any improvements to the visualiser have knockons elsewhere (eg to the ER Modeller) so anything done here would be very welcome indeed. Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] Two questions
On Monday 27 September 2010 04:16:23 Gloria W wrote: > Hi again, > > I have two questions, one is possibly silly, but here goes anyway. I view "silly" questions as useful. If one person has a misunderstanding it is possible/likely that several people may do. If no-one asks them such misunderstandings persist. If someone asks on list and I answer on the list, at minimum the answer is googleable and referenceable. > (1) When using FastRestartServer(), I want to prevent more than one TCP > client from connecting. When a second client connects, I get a nasty > error, as expected: This isn't expected. The FastRestartServer is designed to handle many many concurrent connections. Having an error upon client connect is normally a symptom that Kamaelia is being used wrong. In particular this error: > > Axon.AxonExceptions.BoxAlreadyLinkedToDestination: ([], None, []) Is symptomatic of something like this: from Kamaelia.Apps.SA.Time import PeriodicTick from Kamaelia.Chassis.Pipeline import Pipeline from Kamaelia.Chassis.ConnectedServer import FastRestartServer X = Pipeline( PeriodicTick(delay=0.5, tick_mesg="Yes"), ).activate() def myProtocol(**argd): return X FastRestartServer(protocol= myProtocol, port=1500).run() If you run this server: $ ./BadReuse.py And then connet a client: $ telnet 127.0.0.1 1500 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. YesYesYesYesYesYesYesYesYesYesYesYesYesYesYesYesYesConnection closed by foreign host. When you connect a second client: $ telnet 127.0.0.1 1500 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. Connection closed by foreign host. ... the server crashes, and all the clients disconnect, and the server crashes with the message: $ ./BadReuse.py ...[snip]... packages/Kamaelia/Chassis/ConnectedServer.py", line 381, in handleNewConnection self.link((protocolHandler,"outbox"),(connectedSocket,"inbox")) File "/usr/local/lib/python2.6/dist-packages/Axon/Component.py", line 576, in link return self.postoffice.link(source, sink, *optionalargs, **kwoptionalargs) File "/usr/local/lib/python2.6/dist-packages/Axon/Postoffice.py", line 147, in link raise e Axon.AxonExceptions.BoxAlreadyLinkedToDestination: ([], None, []) To be clear, this crash is expected because it is not the correct way to use FastRestartServer. Why? FastRestartServer's "protocol" argument is expected to be a factory type function. Clearly this an __init__ method, or an actual factory function can achieve this. (Consider that this is somewhat like how CGIs work on a webserver) More precisely, it expects to be provided with a component it can do the following: * Link its inboxes to the outputs from a connected socket adapter, so that data from the client connection can flow in to the component. * Link its outboxes to the inputs to a connected socket adapter, so that data from the component can flow in to the client connection. In the above example, * In the first connection, since X has never been linked into anything else, the first client succeeds. * In the second connection, since the first connection is still active, this means X is still wired into the first connection. So when the second connection tries to wire X into the second connection, you get an error causing a crash. The reason why it will not crash if you have one at a time should be clear from the introspection tutorial pdf referenced earlier, but for completeness is as follows: * Assume X is slightly different. In particular, it is different in that it ignores shutdown and producerFinished messages. * In the first connection, since X has never been linked into anything else, the first client succeeds. * The first client disconnects. This causes X to be unlinked from the connection. ie the boxes of X are no longer linked to anything. * The second client connects. As in the first connection, since X is no longer linked into anything else, the second connection succeeds. * The second client disconnects. This causes X to be unlinked from the connection. If you genuinely need the behaviour: * Background resource creating data * Client connects - wants SOLE output of resource. Gets it or fails gracefully. Then this is different from the above accidental behaviour, and should be managed as such. I've jokingly referred to this as the Dining Philosophers Gravy Jug problem (perhaps because it's lunchtime here!), but it's about right: each philosopher wants to grab the shared resource, take some data from it, and then move on. (And a conflict causes gravy to go everywhere :-) I suspect that this *may* be what you're after. It may not. I'll need to code something up for that to be clear I suspect. > I want t
Re: [kamaelia-list] Answered one of my questions
Hi Gloria,
[ been a bit time constrained recently ]
On Tuesday 28 September 2010 00:21:43 Gloria W wrote:
> So I found the answer to the silly question: This is not a
> single-connection model, it is a fork-exec model, and postboxes are
> properly isolated by thread. Based on my bug, I thought this was either
> a non-functioning single connection model, or a fork-exec model without
> the isolation I needed, and I did not know how to inherently prevent
> more than one connection.
>
> My bug was that I had a globally scoped component, and of course, all
> threads were reusing the already connected component. My bad!
>
> So scratch the first question. My other question still stands: How do I
> write data back to the TCP socket client? Kamaelia doesn't directly give
> me access to a connected socket descriptor. I could dig for it and
> expose it, but there must be a better way.
I'm a big believer in "If you can *see* what the code is doing it
becomes more obvious". I'd noticed a number of q's - which I've been
trying to get a chance to answer - might have a *better* answer if I
wrote a tutorial about the Introspector.
Specifically allowing you to *see* inside systems.
It's generally only useful for small systems, but will hopefully
explain why you don't do the above.
I'd been trying to get it "done", but as mentioned I've been time constrained.
However, I have checked in the current version into here:
http://kamaelia.googlecode.com/svn/trunk/Code/Python/Kamaelia/Examples/Introspection/IntrospectionTutorial/
Whilst direct URLs into svn do allow direct download, since svn
doesn't set a MIME type, I've mirrored the parts of /trunk that go
into the release onto the Kamaelia website here:
http://www.kamaelia.org/release/KamaeliaTrunk/
This means the above tutorial can be found here:
http://www.kamaelia.org/release/KamaeliaTrunk/Kamaelia/Examples/Introspection/IntrospectionTutorial/
In particular a version specifically for reading is here:
http://www.kamaelia.org/release/KamaeliaTrunk/Kamaelia/Examples/Introspection/IntrospectionTutorial/IntrospectorWalkthrough.pdf
(Simply skipping through the doc looking at the diagrams generated
alone probably has significant value)
These are in effect tutorials about building (basic) network systems
with kamaelia, but in such a way that you can see what happens in
terms of components created and shutdown when clients connect and
disconnect.
The diagrams are generated live, at run time, based on introspection
of the system looking at what components exist, what in/out-boxes they
have and how linkages are formed.
They have some limitations - for example, the diagrams are "flat" - in
that child components are not shown nested inside parent components,
leading to some slight oddities. (the contents of a pipeline are shown
beside the pipeline component rather than inside for example)
Nonetheless, it will hopefully help explain far better what's actually
going on. In particular, it helps with debugging this sort of issue:
> My bug was that I had a globally scoped component, and of course, all
> threads were reusing the already connected component. My bad!
And also hopefully makes it clear why this ...
> So scratch the first question. My other question still stands: How do I
> write data back to the TCP socket client? Kamaelia doesn't directly give
> me access to a connected socket descriptor. I could dig for it and
> expose it, but there must be a better way.
... is very unlikely to be what you want. I say this because when
building network systems with kamaelia, the need for access to the
socket is normally a symptom of one of two things: something bust in
Kamaelia, or a misunderstanding of kamaelia)
Writing back to a client is simple - you simply send data out an
outbox of the component created when a client connects:
import Axon
from Kamaelia.Chassis.ConnectedServer import FastRestartServer
class HelloWorld(Axon.Component.component):
def main(self):
self.send("Hello ", "outbox")
self.send("World", "outbox")
self.send("\r\n", "outbox") # Network end of line
yield 1 # To ensure a generator
self.send(Axon.Ipc.producerFinished(), "signal")
FastRestartServer(protocol=HelloWorld, port=1500).run()
If you want to have the output of a pipeline instead, you return a
pipeline, and the output from the last component gets sent to the
connection. Picking a spurious example using simplejson to send the
current system time.
It'll have 5 components :
1 A pipeline component containing:
1 One as a datasource of the current time in seconds.
2 One that transforms it to a time structure with hours, minutes, etc
3 One that turns that into something json'able
4 One that encodes that as json
The output from the outbox "outbox" of 5 falls out the outbox "outbox"
of 1, and that data gets sent to the connected socket adapter handling
the actual connected socket.
import simp
[kamaelia-list] Kamaelia 1.0.9.0 released
Hi, I've built the 1.0.9.0 release and linked it from the website front page. Direct link: * http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.9.0.tar.gz This rolls up some bug fixes, a number of improvements aimed at debugging (tracing of messages, profiling running systems), and quiessence for open gl systems, and a handful of rewritten components. Additionally the SoC2006 example code has been split out into more sensible directories. (The place they lived then made sense then, but would be clearer elsewhere!) For those that missed it, Kamaelia's license changed in July 2010 to the Apache 2 license. (with explicit permission to use the old license scheme if needed - basically to cover any weird edge cases with a "yes" :-) Summary Changelog === Axon Improvments Summary * Better tracing of message passing for debug purposes Kamaelia Improvements Summary * Added namespace Kamaelia.Apps.MH * Kamaelia/Apps/MH/Profiling.py -- provides unix "top" like functionality for components * Bugfix for Kamaelia/Chassis/PAR.py -- handle shutdown messages better/more correctly * Bugfix for Kamaelia/Internet/TCPServer.py -- eliminate random shutdowns under load * Kamaelia/Internet/SingleServer.py -- arguments made more in keeping with ConnectedServer, added FastRestartSingleServer * Support multiple multicast transcievers in the same app on more OSs cleaner * CSA.stop changed to handle multiple stop requests more robustly. (Shouldn't be called twice, this makes it safe if it happens) * Kamaelia/Visualisation/PhysicsGraph/TopologyViewerServer.py changed to allow fast restart * Rewrites of EchoProtocol.py, NullSinkComponent.py * CPU hogging behaviour of OpenGL code reduced slightly in Interactor.py, OpenGLDisplay.py * Bugfix in OpenGLDisplay.py to handle change in PyOpenGL API * CPU improvements in Pygame/Display.py, Pygame/Ticker.py * GSOC 2006 examples moved out from SoC2006 directory into directories with better names. Detailed Changelog AUTHORS * Added Andrew Bonney as contributor README * Bumped version number Axon/Axon/Component.py * Encoding line added * Added two global flag variables for debugging: TraceAllSends = False TraceAllRecvs = False * If TraceAllSends is set, replaces .send with a version that traces all sends * If TraceAllRecvs is set, replaces .recv with a version that traces all recvs * Added _debug_recv to trace recvs * Added _debug_send to trace sends Axon/Axon/Linkage.py * Encoding line added * Imports Box to enable improving storage tags if Box.ShowAllTransits is set - These aren't perfect - they should probably *append* to the storage tag rather than replace * Adds method short_str - which summarises a linkage in a shorter fashion Kamaelia/setup.py * Bumped version number * Added Kamaelia.Apps.MH Kamaelia/Examples/UsingChassis/PAR/par_shutdown_simple.py * Added example showing clean shutdown of PAR in the case of producerFinished messages, in simple and pathological examples. Kamaelia/Kamaelia/Apps/MH/__init__.py * Added to allow this namespace to exist Kamaelia/Kamaelia/Apps/MH/Profiling.py * Added for debugging purposes. Allows profiling of CPU usage in a manner somewhat like unix "top". Kamaelia/Kamaelia/Apps/Whiteboard/Canvas.py * Added some notes regarding code quality... Kamaelia/Kamaelia/Chassis/PAR.py * While loop regarding pausing/shutdown turned into an if. This solves a bug where the flag shutdown can't become true and sleeps through child shutdowns Kamaelia/Kamaelia/Internet/TCPServer.py * 1 line bugfix, but critical. Beforehand, depending on shutdown approach, the TCPServer would search through the socket handlers for whichever component was handling the socket. However, rather than that component being picked for shutdown, the last one in the list would be picked instead. This is a pretty critical problem for any major server and would cause "erratic" behaviour in certain limited circumstances. It would mean that many servers would look correct with basic testing and have unexpected intermittent issues under load from buggy clients. Kamaelia/Kamaelia/Internet/SingleServer.py * Removed reference to python 2.3 from script executable path * Added comment about FastRestartSingleServer * imports socket to enable this * Changed to use default arguments style * Default arguments added largely mirror ConnectedServer.ServerCore, sans protocol * Allows socketoptions to be set for the TCPServer. * In theory allows TCPServer to be overridden * Adds FastRestartSingleServer as a subclass of SingleServer - this is identical to SingleServer, except with appropriate socket options * Demo example now shows usage of socketOptions for fast restarting Kamaelia/Kam
Re: [kamaelia-list] Shut down problem with Kamaelia.Chassis.PAR
Hi Jim, On 9 September 2010 09:02, Jim Easterbrook wrote: > I'm having a problem with PAR in a pipeline of other components. If I > shut down the pipeline with a producerFinished() signal, the PAR > component does not exit. Using a shutdownMicroprocess() signal works OK > though. Thanks for this, I'm trying to create a reproducible test case for this at the moment. Re-reading the PAR component, I think I can see why this behaviour happens, but I want to check first before checking in a fix. In particular it relates the the things that can cause a component to unpause: * Message received on an inbox. * Message taken from an outbox. * Child component termination. Unfortunately, the loop here... while not self.anyReady() and not shutdown: self.pause() yield 1 ... ignores that final case. If it looked like this: while not self.anyReady() and not self.childrenDone(): self.pause() yield 1 That would be OK. My preferred fix will probably actually just be to change this: while not self.anyReady() and not shutdown: self.pause() yield 1 To this: if not self.anyReady(): self.pause() But I want to make a minimal reproducible test case first. The shutdown variable is almost certainly extraneous as well. > (As an aside, I'm not a fan of having loops within loops like this. I > like my components to have a simple main loop with an if ... elif ... > elif ... else self.pause() chain.) Most of mine end up like that as well. The while loop inside here is often a symptom that I'm not interested in being woken up when someone takes a message out of my component's outbox - ie that I'm only interested in working when I recieve data on an inbox of some kind. As a further piece of random useful debugging info, recently I improved (on /trunk) the output that you get from this code: Axon.Box.ShowAllTransits = True I also added hooks enabling the following: Axon.Component.TraceAllSends = True Axon.Component.TraceAllRecvs = True For these 3 debug hooks to work correctly, they need to be done immediately after your imports Axon. ie something like this: import Axon from Kamaelia.Protocol.EchoProtocol import EchoProtocol from Kamaelia.Chassis.ConnectedServer import FastRestartServer Axon.Box.ShowAllTransits = True Axon.Component.TraceAllSends = True Axon.Component.TraceAllRecvs = True FastRestartServer(protocol=EchoProtocol, port=1500).run() Anyway, I'll create that test case, then check in the fix - thanks for the bug report! Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Probably need to break this into a handful of topics Re: [kamaelia-list] TaggingPluggableProcessor usage examples, also bigger design question
On Monday 06 September 2010 17:33:44 Michael Sparks wrote: > Fundamentally, yes. The longer answer to this was delayed. I've just noticed that this has a certain irony given that mail ended up being pretty long itself. I think what this really means is that this is really a collection of topics that are all related, and what I really meant is that I'll be chopping my replies into a collection of docs focussed on an area. Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] TaggingPluggableProcessor usage examples, also bigger design question
Hi, On Saturday 28 August 2010 22:30:16 Gloria W wrote: > If I send it a message via a SimpleServer, can components be added after > run() is executed? Fundamentally, yes. The longer answer to this was delayed. The reason it was delayed is because I started writing something about using the visualiser which can be useful for explaining this better, and then discovering a latent bug in the TCPServer code. OK, I didn't know that *then* but I do now. Anyway, I'll be posting that later today, along with an explanation of *how* I found the bug, since I think that's perhaps as much, if not more useful. In particular, I've enhanced on part of linkage tracing, and also more in depth system tracing. Fundamentally though the answer to the above question is "yes". > Right, I have my own components with which I did this, and it makes > sense. Indeed. > I end up converting things that I can post to this list to > generic examples which will not violate the NDA I signed when I started > this work. I guessed that might be the case. Violating NDA's falls into my list of bad ideas, so I approve :-) (Not that approval is required, but hey :-) > Instead of doing this from this point forward, I'm first > developing the generic weather handling model, then rewriting it to meet > the needs of the specific project. Sounds sensible. > The ConditionalTransformer rocks. Passing function factories to filter > out messages is a beautiful implementation. :-) > > However, if I did want access to instance storage space, or to add in my > > own defaults, I can do that too. For example, to change the above > > "Divisible3" example into something generic, I want to shift the divisor > > value out into the > > > > class definition: > > class Divisible(ConditionalTransformer): > > divisor = 4 > > > > def condition(self, message): # condition callback with state > > > > return (message % self.divisor) == 0 > > > > def process(self, message): # process callback with state > > > > return "Divisible by "+str(self.divisor) + " ! " + > > str(message) > > > > And then I can use this as follows: > > Divisible(TPP, "FOUR").activate() > > Right, all without having to call super(). This is one of my favorite > features of Python. Indeed, it's a very nice aspect of python. > > Finally, I since "divisor = 4" acts as a Kamaelia inheritable default > > value, I > > > > can override it in other instances: > > Divisible(TPP, "FIVE", divisor = 5).run() > > Ah! Oh wow... > > Then define 2 inheritable default values, which subclasses can override > > explicitly. > > > > condition=None > > process=None > > > > Then we declare our __init__ method signature. As usual we include **argd > > to enable someone subclassing us to also define inheritable defaults. > > > > def __init__(self, processor=None, tag=None, condition=None, process=None,**argd): > This is really slick. I like it, but is it considered bad practice, > because the inheritable values are implicit? It's debateable. The normal approach we've been using looks like this: class ServerCore(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent): port = 1601 protocol = None socketOptions=None TCPS=TCPServer def __init__(self, **argd): super(ServerCore, self).__init__(**argd) This then allows override elsewhere. However, the __init__ method now has entirely implicit arguments, which is a little ugly. I'd personally wondered whether this would be viewed as pythonic or unpythonic by someone experienced, but figured that the **argd (or **kwargs) form was so useful that utility overrode things. ie in this case pragmatism beat purity. At Europython Raymond Hettinger, with David Jones was running a code clinic where people could bring up there code for critique, etc. After seeing the format, I figured why not and thought it worth seeing what Raymond would think of it. We went through all sorts, and if I was the sort of person to pull quotes out of context, there'd be some really amazingly cool ones I could put here :-) (The videos are on europythonvideos.blip.tv) Anyway, it transpired that he thought that this: class ServerCore(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent): port = 1601 protocol = None socketOptions=None TCPS=TCPServer def __init__(self, **argd): super(ServerCore, self).__init__(**argd) Was pretty close to the best thing he'd seen for this and managing defaults - since that's effectively done here: class component(microprocess): def __init__(self, *args, **argd): super(component, self).__init__() self.__dict__.update(argd) In particular, the reason being that the **argd form plays particularly well with super() which in turn is a good combination of things in a library. However, he also noted that you're then left wit
Re: [kamaelia-list] Found it:Bug in setsockopt call: Only in fedora 12/13
Hi Gloria, On Sunday 29 August 2010 22:35:59 Gloria W wrote: > I tested this on RHE 64 bit 2.6.18 kernel, and all is good. I then > more carefully looked at my iptables set up and discovered it was not > shutting down when it claimed it was. Now it works. > Thanks for the sanity test! That's great to hear :-) Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] TaggingPluggableProcessor usage examples, also bigger design question
On 22 Aug, 18:32, Gloria W wrote:
> Hi again Michael,
>
> Firstly, many many thanks once again for this. I am really appreciative
> of the in-depth explanations.
You're welcome. :-)
> Secondly, I am trying to use the TaggingPluggableProcessor you sent
> earlier, as follows:
For reference I didn't expect that to be a useful component. But I guess that
that form of component would be. As a result, I've checked that example into
SVN here:
/trunk/Sketches/MPS/List/TaggingPluggableProcessor.py
(svn url)
http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/List
(pretty version)
http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/List/
There's a couple of other examples that use it in that directory, which I'll
describe below.
> t = TaggingPluggableProcessor()
> t.add_processor(["Test",None,MyComponent()])
>
> and I got this error:
>
> [glori...@localhost Kamaelia]$ !py
> python tagging_pluggable_orig.py
> Traceback (most recent call last):
>File "tagging_pluggable_orig.py", line 190, in
> t.add_processor(["Test",None,Seq( Pauser(),DataSource([1,2,3,4,5,6]))])
>File "tagging_pluggable_orig.py", line 100, in add_processor
> self.lookup[filter_component] = bundle
> AttributeError: 'TaggingPluggableProcessor' object has no attribute 'lookup'
OK, the key problem here is the fact that you're calling t.add_processor
directly, but you can't know what the state of "t" is externally. When dealing
with Kamaelia components, you *always* send a message to it, with very few
exceptions.
Those exceptions are:
* __init__
* activate
* run
As a result, the above call is an error. In particular, the error you're
seeing also demonstrates why - if you call a bare method like the above, then
you have no idea what the current state is of the component.
> I didn't want to define the lookup dictionary before the constructor,
> because I don't think it should be shared among instances. Am I using
> the TaggingPluggableProcessor correctly?
As you can see usage of any component by calling a method it has is incorrect,
so that would be "no" :-)
> Can you please give an example where you add processors to it after it
> is instantiated?
OK, I'll break this into two halves. First a toy example, and then something
more practical.
At the absolute core, the core of what needs to happen is this:
def main(self):
# Assume TPP is a TaggingPluggableProcessor
# We first have to make a linkage to it
self.link((self, "toprocessor"), (TPP, "processor"))
# We then need to send it a message which consists of
# ( TAG, CONDITION_OR_FILTER_CALLBACK,
COMPONENT_THAT_PERFORMS_PROCESSING)
#
self.send( ("five", self.condition, ), "toprocessor")
So, let's build a toy example - something adds *itself* as a processor to the
tagging pluggable processor, that emits "Johnny 5 is Alive?" when it recieves
the number "5" on its main inbox.
This toy example is in svn here:
http://kamaelia.googlecode.com/svn/trunk/Sketches/MPS/List/SimpleCondTransformer.py
http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/List/SimpleCondTransformer.py
Let's walk through this.
First we need some basic imports, and to create (but NOT activate) a
processor:
import Axon
from TaggingPluggableProcessor import TaggingPluggableProcessor
TPP = TaggingPluggableProcessor()
We'll use a global variable here because it's a toy example.
Then we need to subclass Axon.Component.component, and have an additional
outbox "toprocessor" for sending messages to the TaggingPluggableProcessor. We
don't need to change the defaults for Inboxes, so we don't declare it.
class SimpleCondTransformer(Axon.Component.component):
# Inboxes not listed, since as default
Outboxes = ["outbox", "signal", "toprocessor"]
I've chosen to use a list here rather than dictionary, primarily for terseness
in this toy example.
Now we need to define a condition function and process function - one is to
select messages to process, the latter is to process them. These are static
methods for the moment, and just inside SimpleCondTransformer for convenience:
@staticmethod
def condition(message): return int(message) == 5
@staticmethod
def process(message): return "Is Johnny 5 Alive ?"
>>>
We can then start defining our main method. The first thing this wants to do
is use the global name "TPP" as something to link TO. We'll link out
"toprocessor" outbox to the processor's "processor" inbox. We'll then send a
message of format ("tag, condition function, component)
def main(self):
self.link((self, "toprocessor"), (TPP, "processor"))
self.send( ("five", self.condition, self), "toprocessor")
... when this is handled by the TaggingPluggableProcessor, this wiil cause our
SimpleCondTransformer to be added as a processor processing va
Re: [kamaelia-list] Bug in setsockopt call
On Wednesday 25 August 2010 22:55:10 Gloria W wrote:
> Hi again,
> I am trying to run this example:
> amaelia-1.0.7.0/Kamaelia/Test/Internet/test_MulticastTransceiverSystem.py
>
> and I'm getting this error:
>
> Traceback (most recent call last):
>File "test_MulticastTransceiverSystem.py", line 59, in
> tests()
>File "test_MulticastTransceiverSystem.py", line 55, in tests
> scheduler.run.runThreads(slowmo=0.1)
>File "/usr/lib/python2.6/site-packages/Axon/Scheduler.py", line 579,
> in runThreads
> for i in self.main(slowmo,canblock=True): pass
>File "/usr/lib/python2.6/site-packages/Axon/Scheduler.py", line 473,
> in main
> result = mprocess.next()
>File "/usr/lib/python2.6/site-packages/Axon/Microprocess.py", line
> 398, in next
> return self.__thread.next()
>File "/usr/lib/python2.6/site-packages/Axon/Microprocess.py", line
> 507, in _microprocessGenerator
> v = pc.next()
>File
> "/usr/lib/python2.6/site-packages/Kamaelia/Internet/Multicast_transceiver.p
> y", line 130, in main
> socket.inet_aton(self.remote_addr) + socket.inet_aton("0.0.0.0"))
>File "", line 1, in setsockopt
> socket.error: [Errno 22] Invalid argument
>
> I checked the docs on setsockopt, and the parameters under
> IP_ADD_MEMBERSHIP look correct:
>
> The ip_mreq structure (taken from |/usr/include/linux/in.h|) has the
> following members:
>
>
>
> struct ip_mreq
> {
> struct in_addr imr_multiaddr; /* IP multicast address of
> group */ struct in_addr imr_interface; /* local IP address of interface
> */ };
>
> What is going wrong here?
I've just tried taking down all my interfaces, except the loopback one, and
only having a default route saying "send all traffic that's not otherwise
routed out this way":
$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
$ ifconfig
loLink encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:4863 errors:0 dropped:0 overruns:0 frame:0
TX packets:4863 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1477186 (1.4 MB) TX bytes:1477186 (1.4 MB)
$ sudo route add -net default dev lo
$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 lo
And it still runs:
$ ./test_MulticastTransceiverSystem.py
(('0.0.0.0', 38028), 'Hello World')(('0.0.0.0', 38028), 'Hello World')
(('0.0.0.0', 38028), 'Hello World')(('0.0.0.0', 38028), 'Hello World')
(('0.0.0.0', 38028), 'Hello World')(('0.0.0.0', 38028), 'Hello World')
(('0.0.0.0', 38028), 'Hello World')(('0.0.0.0', 38028), 'Hello World')
(('0.0.0.0', 38028), 'Hello World')(('0.0.0.0', 38028), 'Hello
World')^CTraceback (most recent call last):
If I remove all default routes like this:
$ sudo route del -net default
And try again, I get a different error from yourself:
$ ./test_MulticastTransceiverSystem.py
Traceback (most recent call last):
File "./test_MulticastTransceiverSystem.py", line 59, in
tests()
File "./test_MulticastTransceiverSystem.py", line 55, in tests
scheduler.run.runThreads(slowmo=0.1)
File "/usr/local/lib/python2.6/dist-packages/Axon/Scheduler.py", line 580,
in runThreads
for i in self.main(slowmo,canblock=True): pass
File "/usr/local/lib/python2.6/dist-packages/Axon/Scheduler.py", line 474,
in main
result = mprocess.next()
File "/usr/local/lib/python2.6/dist-packages/Axon/Microprocess.py", line
399, in next
return self.__thread.next()
File "/usr/local/lib/python2.6/dist-packages/Axon/Microprocess.py", line
508, in _microprocessGenerator
v = pc.next()
File "/usr/local/lib/python2.6/dist-
packages/Kamaelia/Internet/Multicast_transceiver.py", line 128, in main
socket.inet_aton(self.remote_addr) + socket.inet_aton("0.0.0.0"))
File "", line 1, in setsockopt
socket.error: [Errno 19] No such device
Which suggests to me that something firewally. If you're running something
like SuSE linux it defaults to having a personal firewall. For reference, my
output from iptables looks like this:
$ sudo iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
I'd suggest as a test trying this from a command line before running the test
code:
sudo iptables --flush
That will effectively switch off any local personal firewall by zapping on the
local firewall rules.
Other than those guesses though, which might not be right I'll need more hints
as to the OS you're using and its config to be
[kamaelia-list] Re: Bug in setsockopt call
On Aug 25, 10:55 pm, Gloria W wrote:
> Hi again,
> I am trying to run this example:
> amaelia-1.0.7.0/Kamaelia/Test/Internet/test_MulticastTransceiverSystem.py
>
> and I'm getting this error:
>
> Traceback (most recent call last):
> File "test_MulticastTransceiverSystem.py", line 59, in
> tests()
> File "test_MulticastTransceiverSystem.py", line 55, in tests
> scheduler.run.runThreads(slowmo=0.1)
> File "/usr/lib/python2.6/site-packages/Axon/Scheduler.py", line 579,
> in runThreads
> for i in self.main(slowmo,canblock=True): pass
> File "/usr/lib/python2.6/site-packages/Axon/Scheduler.py", line 473,
> in main
> result = mprocess.next()
> File "/usr/lib/python2.6/site-packages/Axon/Microprocess.py", line
> 398, in next
> return self.__thread.next()
> File "/usr/lib/python2.6/site-packages/Axon/Microprocess.py", line
> 507, in _microprocessGenerator
> v = pc.next()
> File
> "/usr/lib/python2.6/site-packages/Kamaelia/Internet/Multicast_transceiver.py",
> line 130, in main
> socket.inet_aton(self.remote_addr) + socket.inet_aton("0.0.0.0"))
> File "", line 1, in setsockopt
> socket.error: [Errno 22] Invalid argument
OK, that's weird. When I run it, I get this (expected) behaviour:
kamael...@kamaelian-laptop:~/Checkouts/kamaelia/trunk/Code/Python/
Kamaelia/Test/Internet$ ./test_MulticastTransceiverSystem.py
(('192.168.2.2', 54741), 'Hello World')(('192.168.2.2', 54741), 'Hello
World')(('192.168.2.2', 54741), 'Hello World')(('192.168.2.2', 54741),
'Hello World')(('192.168.2.2', 54741), 'Hello World')(('192.168.2.2',
54741), 'Hello World')(('192.168.2.2', 54741), 'Hello World')
(('192.168.2.2', 54741), 'Hello World')(('192.168.2.2', 54741), 'Hello
World')(('192.168.2.2', 54741), 'Hello World')(('192.168.2.2', 54741),
'Hello World')(('192.168.2.2', 54741), 'Hello World')^CTraceback (most
recent call last):
> I checked the docs on setsockopt, and the parameters under
> IP_ADD_MEMBERSHIP look correct:
Given this:
> The ip_mreq structure (taken from |/usr/include/linux/in.h|)
I'm assuming you're running linux. For reference, I'm running Ubuntu
10.04.1 LTS. Some relevant local output: (with some IPs changed to
protect the innocent :-)
kamael...@kamaelian-laptop:~ $ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window
irtt Iface
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0
0 wlan0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0
0 wlan0
0.0.0.0 192.168.2.1 0.0.0.0 UG0 0
0 wlan0
kamael...@kamaelian-laptop:~ $ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:1b:24:8c:24:a3
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:17
loLink encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2709 errors:0 dropped:0 overruns:0 frame:0
TX packets:2709 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1051823 (1.0 MB) TX bytes:1051823 (1.0 MB)
pan0 Link encap:Ethernet HWaddr d6:d0:e8:f3:0c:b7
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vboxnet0 Link encap:Ethernet HWaddr 0a:00:27:00:00:00
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
wlan0 Link encap:Ethernet HWaddr 00:1b:77:78:b4:c6
inet addr:192.168.2.2 Bcast:192.168.2.255 Mask:
255.255.255.0
inet6 addr: fe80::21b:77ff:fe78:b4c6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:123630 errors:0 dropped:0 overruns:0 frame:0
TX packets:108907 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:109866599 (109.8 MB) TX bytes:20077933 (20.0 MB)
And for completeness:
kamael...@kamaelian-laptop:~ $ uname -a
Linux kamaelian-laptop 2.6.32-24-generic #41-Ubuntu SMP Thu Aug 19
01:12:52 UTC 2010 i686 GNU/Linux
kamael...@kamaelian-laptop:~ $ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes
Re: [kamaelia-list] Re: How do I force the OS to let go of the socket with this model?
Hi Gloria,
That code fragment is clearer regarding what you're trying to do.
A Kamaelia.Protocol component is just a component that expects to
recieve bytes from a network transport of some kind on its inbox
"inbox", and send bytes out its outbox "outbox" which should be sent
to the network.
In particular, rather than writing this:
# listing 1
Pipeline(
ServerCore(port=1500, protocol = EchoProtocol,
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)),
DeJSONizer(),
PublishTo("RAINDATA")
).activate()
You want to write this:
# listing 2
def MyRainDataProtocol(**argd):
return Pipeline(
DeJSONizer(),
PublishTo("RAINDATA")
) # Gets activated by ServerCore
ServerCore(port=1500, protocol = EchoProtocol,
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR,
1)).activate()
Which can also be rewritten as:
# listing 3
def MyRainDataProtocol(**argd):
return Pipeline(
DeJSONizer(),
PublishTo("RAINDATA")
) # Gets activated by ServerCore
FastRestartServer(port=1500, protocol = MyRainDataProtocol).activate()
FastRestartServer is a TCP based server core. Under the hood, it
creates a TCP based socket, and sets it listening. It then waits for
connections. When connections occur, the argument passed in in the
"protocol" argument is called as follows:
# Inside the code, protocol gets stored as mkProtocolHandler
protocolHandler = self.mkProtocolHandler(peer=peer,
peerport=peerport,localip=localip,localport=localport)
So in particular, if I connect from 192.168.2.5, port 32122 to a
server running on 192.168.2.10, port 1500 - in particular to the
server code in listing 3 - then MyRainDataProtocol gets called as
follows:
MyRainDataProtocol(peer="192.168.2.5",
peerport=32122,localip="192.168.2.10",localport=1500)
ie what happens is whatever argument is passed in as the "protocol"
argument to ServerCore should be a callable that is able to create a
component to handle the connection. ie a factory function. Of course
the __init__ method of any class is such a beast, but you aren't
limited to that as you can see above.
If I rewrote listing 3 such that it didn't accept any arguments like
this, then it would fail:
# listing 4 - This will fail horribly, don't do this,
# since you should handle the 4 named arguments, even
# if its simply throwing them away.
def MyRainDataProtocol():
return Pipeline(
DeJSONizer(),
PublishTo("RAINDATA")
) # Gets activated by ServerCore
FastRestartServer(port=1500, protocol = MyRainDataProtocol).activate()
In particular, this would fail when someone tries to connect to the
server, and the end of the traceback would look something like this:
...
protocolHandler = self.mkProtocolHandler(peer=peer,
peerport=peerport,localip=localip,localport=localport)
File "/usr/lib/python2.6/site-packages/Kamaelia/Chassis/ConnectedServer.py",
line 343, in mkProtocolHandler
localport = sock_info["localport"])
TypeError: __init__() got an unexpected keyword argument 'peer'
Which you'll note is the traceback you saw. ie the bug you're seeing
isn't with your code per se, but with the EchoProtocol you're
importing and using with ServerCore. The reason for this is because
ServerCore is a generalisation of an earlier piece of code called
"SimpleServer", and SimpleServer expected to be passed callbacks that
don't need arguments.
In particular, it's a bit of a red herring here, but useful nonetheless.
You can make Listing 4 valid again as follows:
from Kamaelia.Chassis.ConnectedServer import SimpleServer
# listing 5 - Protocol will ONLY work with SimpleServer
def MyRainDataProtocol():
return Pipeline(
DeJSONizer(),
PublishTo("RAINDATA")
) # Gets activated by SimpleServer
SimpleServer(port=1500, protocol = MyRainDataProtocol).activate()
... and that will work, and NOT do what you want because it's not the fast
restart version.
So, in summary, you've already written the protocol you want in
listing 1, but it just needs to be able to be created on demand when
someone connects. In order to do that it needs to be returned from a
function that can be called when someone connects. Hence listings 2
and 3. Listing 4 is an error. Listing 5 works with the protocol from
listing 4, but really isn't what you want.
So in short you either want the sort of thing in listing 2 or 3. If
you change listing 3 to dump out diagnostics:
# listing 3
def MyRainDataProtocol(**argd):
print "New connection info", argd
return Pipeline(
DeJSONizer(),
PublishTo("RAINDATA")
) # Gets activated by ServerCore
FastRestartServer(port=1500, protocol = MyRainDataProtocol).activate()
... you'll see why this is useful - it's about NOT hiding connection
info from the client. (Something useful in a greylisting server for
example)
Wh
[kamaelia-list] Re: How do I force the OS to let go of the socket with this model?
Hi Gloria,
On 20 Aug, 03:46, Gloria W wrote:
> Hi all,
> I am usign SingleServer like this:
>
> s = SingleServer(1500)
> s.socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>
> Pipeline(s,
> DeJSONizer(),
> PublishTo("Blah")
> ).activate()
>
> Pipeline(
> SubscribeTo("Blah"),
> Seq("Sequence1",
> stuff...
> )
> ).activate()
>
> Backplane("Blah").run()
>
> And when I kill it and restart, the blasted socket is still tied up, and
> my telnet fails for about a minute or so.
>
> Why is the socket option not working? Is there a new way to solve this
> problem now?
The tl;dr answer would be "SingleServer doesn't do that sort of thing,
use ServerCore
instead, also look for FastRestart* versions of things" :-)
Longer (more useful) answer:
SingleServer is sometimes useful for knocking things together, but
personally I always tend to use ServerCore:
# listing 1
from Kamaelia.Chassis.ConnectedServer import ServerCore
from Kamaelia.Protocol.EchoProtocol import EchoProtocol
def HandleConnection(**argd):
print "Client connected", argd
return EchoProtocol()
ServerCore(port=1500,
protocol = EchoProtocol).run()
Additionally, ServerCore can take a socketOptions argument:
# listing 2
import socket
from Kamaelia.Chassis.ConnectedServer import ServerCore
from Kamaelia.Protocol.EchoProtocol import EchoProtocol
def HandleConnection(**argd):
print "Client connected", argd
return EchoProtocol()
ServerCore(port=1500,
protocol = EchoProtocol,
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR,
1)
).run()
Since the above is a common need, FastRestartServer was added as well.
This
is *precisely* the same as ServerCore *except* it's socketOptions is
the
above as per listing 2 above.
# listing 3 - has exact same functionality as listing 2
from Kamaelia.Chassis.ConnectedServer import FastRestartServer
from Kamaelia.Protocol.EchoProtocol import EchoProtocol
def HandleConnection(**argd):
print "Client connected", argd
return EchoProtocol()
FastRestartServer(port=1500,
protocol = EchoProtocol,
).run()
By contrast, SingleServer has always been (deliberately) pretty dumb -
it's
a "netcat" type single shot server that can only accept 1 connection,
and
not accepted setting socket options. I've just modified it on /trunk
to
match the options that ServerCore takes.
In particular, it now has the following default settings:
# listing 4
class SingleServer(_Axon.Component.component):
port = 1601
socketOptions=None
TCPS=TCPServer
Whilst this means its core behaviour remains unchanged, it means it
now has
extra functionality. In particular, you can cause it to restart
straight
away:
# listing 5
SingleServer(port=1501, socketOptions=(socket.SOL_SOCKET,
socket.SO_REUSEADDR, 1))
Additionally, I've also added FastRestartSingleServer, which means
listing 5 can be rewritten thus:
# listing 6 - works *exactly* as per listing 5
from Kamaelia.Internet.SingleServer import FastRestartSingleServer
FastRestartSingleServer(port=1501)
That said, I personally find ServerCore and FastRestartServer much
more convenient to use than SingleServer these days. (Also looking at
SingleServer's code - I note that it could do with some modernisation,
in that it works, but could work better)
Regards,
Michael
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: Key error on Dynamic component example
On Aug 11, 3:04 pm, Gloria W wrote:
> Hi Michael,
> In the fourth example you sent, I am getting a key error here:
>
> # Similarly, we also need to notify the filter/processor components that
> # we've ceased sending them any data. This idiom of reusing an outbox is
> # a convenience that tidies up code.
> for c in self.childComponents():
> L = self.link( (self, "_cs"), (c, "control"))
> self.send( Axon.Ipc.producerFinished(), "_cs")
> self.unlink(thelinkage=L)
>
> self.link fails, claiming that "_cs" is not defined. Should the box name
> be "signal" instead?
Ah, that confused me for a moment, primarily because I tested the code
before mailing it. If you compare the version with comments:
* http://pastebin.com/HQcDesx4
To the one without:
* http://pastebin.com/mpcsSKeu
You'll note that the former has the component signature of:
class
TaggingPluggableProcessor(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent):
Inboxes = {
"inbox": "Stream of data to process",
"processor": "Processor requests are sent here",
"control" : "So we can shutdown"
}
def main(self):
Whereas the latter has the component signature of:
class
TaggingPluggableProcessor(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent):
Inboxes = {
"inbox": "Stream of data to process",
"processor": "Processor requests are sent here",
"control" : "So we can shutdown"
}
Outboxes = {
"outbox": "Stream of tagged processed data",
"signal": "To pass on shutdown",
"_cs" : "For shutting down internal components"
}
So, please use this latter signature instead!
Incidentally, as a convention, it's useful to name inboxes and
outboxes with leading underscores like _this when they're intended to
talk to subcomponents rather than the outside world. Whilst they don't
have to be used that way, it can simplify things for people looking at
the component signature.
Also that should make it clearer why "signal" isn't used for sending
signals to subcomponents. Lack of having a leading underscore implies
it's public, not private.
Sample components that do this incidentally include:
Chassis/Carousel.py: "_signal" : "internal use:
for sending 'shutdownMicroprocess' to child",
Chassis/ConnectedServer.py:Inboxes = { "_socketactivity" :
"Messages about new and closing connections here",
Chassis/ConnectedServer.py:Outboxes = { "_serversignal" : "we send
shutdown messages to the TCP server here",
Chassis/Graphline.py: Outboxes = {"outbox":"", "signal":"", "_cs":
"For signaling to subcomponents shutdown"}
Chassis/PAR.py: "_co": "For passing data to
subcomponents based on a policy (unusued at present)",
Chassis/PAR.py: "_cs": "For signaling to subcomponents
shutdown",
Experimental/Services.py: "_toService" : "request to
service",
File/BetterReading.py:"_selectorready" : "ready to read"
File/BetterReading.py:"_selectorask" : "ask the Selector to
notify readiness to read on a file"
File/UnixProcess2.py: "_shutdownPipes" : "For shutting
down any named pipes used for output"
Internet/ConnectedSocketAdapter.py:"_selectorSignal" :
"For communication to the selector",
Internet/SingleServer.py: "_oobinfo" : "We receive
notification of connection on this inbox"
Internet/TCPClient.py:"_socketFeedback" :
"notifications from the ConnectedSocketAdapter",
Internet/TCPClient.py:"_selectorSignal" : "For
registering and deregistering ConnectedSocketAdapter components with a
selector service",
Internet/TCPServer.py: Inboxes = { "_feedbackFromCSA" : "for
feedback from ConnectedSocketAdapter (shutdown messages)",
Internet/TCPServer.py:"_selectorSignal" : "For
registering newly created ConnectedSocketAdapter components with a
selector service",
Internet/TCPServer.py:"_selectorShutdownSignal" : "To
deregister our interest with the selector",
Internet/UDP_ng.py: "_selectorSignal" : "For
communication to the selector"
Util/Backplane.py: "_outbox" : "To backplane if a
forwarder",
Util/Splitter.py:"_inbox": "Internal inbox for
receiving from the child source component (if it exists)",
Util/Splitter.py:"_control" : "Internal inbox for
receiving from the child source component (if it exists)",
Probably should've mentioned that before. It's mentioned here:
* http://www.kamaelia.org/Docs/NotationForVisualisingAxon.html
But as I say, the website does need an overhaul!
Thanks for point out the typo!
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
[kamaelia-list] Re: threadedcomponent's main function not starting
On Aug 11, 4:40 am, Joesan wrote: > Hi Matt & Michael, > > The yield thing was the exact problem. Thank you! Guess we missed it > in the documentation. Cool. 2 thoughts: * Put an error handler in to catch the case that main() is a generator and tell the user what they're doing wrong. * Put an error handler in to catch the case that main() is a generator and make it do what the user expects. I'm tempted to do the latter, but my instinct is for the former. Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] Dynamic Graphline components?
[warning, this is a little on the long side, and really covers 3 - 4
major topics ]
Hi Gloria,
On Mon, Aug 9, 2010 at 4:32 AM, Gloria W wrote:
> Hi Michael et al,
>
> I am pleased to say that I finally have a freelance contract where I can
> make great use of Kamaelia. I am really happy about transitioning my
> personal Kamaelia work from toy to tool. This is exciting. The license
> change helps greatly.
That's really good to hear :-)
> I have a situation where I want to use a Graphline, but the components are
> dynamic. They can become invalid, and new ones can be required, in realtime.
> Am I forced to add them as subcomponents? Is using __setattr__ in this
> capacity considered a kludge?
> How do I add and remove linkages? Should I destroy and recreate the
> Graphline object each time, instead of all of this manipulation? If so, is
> this slow/inefficient?
You've had a few answers to various questions, so I'll try not to
retread the same ground.
As noted, graphline declares linkages statically and wires things up.
This isn't a limitation of Axon but a limitation of graphline - which
is after all intended to be a convenience for managing linkages, but
relatively statically.
Stepping back, a component that takes other components and wires them
into itself is a chassis component. Some chassis components are
obvious:
* Pipeline
* Graphline
Some are more powerful since they can handle more interesting
topologies or behaviours, but can be slightly confusing:
* PAR - Run all subcomponents in parallel, collate outboxes into one outbox
* Seq - Pipeline(A, Seq(B,C,D), E) is like "A | (B; C; D) | E" in the shell.
* Carousel - Repeatedly restarts the component mentioned
And some are useful in terms of saying "run this, then this, then
this" or "run all of these":
Then there's chassis type components that manage resources of some kind:
* Selector
* DemuxerService
* Backplane (manages plug splitters)
- It's well worth looking at backplanes incidentally. I tend to
use them extensively as opposed to creating new chassis style
components.
* PygameDisplay
* OpenGLDisplay
* ServerCore (& hence FastRestartServer & SimpleServer)
Such components tend to want to advertise to the rest of the system that
they're managing services. It was this need to advertise services that led
to the creation of the Co-ordinating assistant tracker. The tracker was
always expected to be created, but was expected to be used for statistics
collection (hence the key/value store). In practice, it was realising that
it can be used more like the unix environment that made a difference.
For example most version control systems will launch $EDITOR rather than a
particular editor.
As you'd expect, as time has gone on we've figured out better ways of
writing these things, and better idioms. I'll try and cover these here.
First of all there's the idea of a basic static chassis. It takes one or
more components and wires it up as a subcomponent. If we assume that it
does not know in advance how many subcomponents it has, it needs to create
inboxes and outboxes dynamically.
It may need to pass data through from it's inbox through to a subcomponent's
inbox efficiently, and likewise pass data outwards efficiently. Once it's
done this, if it's a static chassis, it probably wants to go to sleep and
only wake up when components shutdown in order to unwire them, and shutdown
when all the children have shutdown.
First of all, let's look at how you would re-build the pipeline component.
I'm also posting this next chunk on a pastebin, where colorising make
make it clearer:
http://pastebin.com/FiuJWapm -- No comments & runnable example usage
http://pastebin.com/ctgG0vQC -- inline comments
Having identified this lot, the code looks like this - declare the basic
class, and capture config.
class MyPipeline(Axon.Component.component):
def __init__(self, *comps, **argv):
super(MyPipeline, self).__init__(**argv)
if len(comps) < 1:
raise ValueError("MyPipeline must have at least 1 subcomponent!")
self.comps = comps
# Then in the main method...
def main(self):
# Create the linkages _inside_ the chassis:
middle_components_indices = range(0,len(self.comps[0:-1]))
for i in middle_components_indices:
self.link( (self.comps[i],"outbox"), (self.comps[i+1], "inbox") )
self.link( (self.comps[i],"signal"), (self.comps[i+1], "control") )
# Create linkages that pass data through from the outside world into the
# subcomponents. These need to be marked as "passthrough":
self.link( (self,"inbox"), (self.comps[0], "inbox"), passthrough=1 )
self.link( (self,"control"), (self.comps[0], "control"), passthrough=1 )
# Create linkages that pass data through from the subcomponents to the
# outside world . These need to be marked as "passthrough", but outbound:
self.link( (self.comps[-1], "outbox"), (self,"outbox"),
[kamaelia-list] Re: threadedcomponent's main function not starting
On Aug 10, 11:11 pm, Joesan wrote:
> We started working with kamaelia recently at work and we're having
> trouble right now working with threadedcomponents.
...
> The new component gets created just fine but for some reason
> it seems it's main function is never actually called.
At a random guess, does your main method contain a yield ? (This is
the
most common cause of this problem)
Threadedcomponents do NOT have a generator as the main method. (I've
been
wondering if this was a good decision, but it's a little late now)
So, for example this will cause a message to be emitted once per
second:
import time
import Axon
from Kamaelia.Util.Console import ConsoleEchoer
from Kamaelia.Chassis.Pipeline import Pipeline
class Prodder(Axon.Threadedcomponent.threadedcomponent):
def main(self):
while not self.dataReady("control"):
time.sleep(1)
self.send("ping", "outbox")
self.send(self.recv("control"), "signal")
Pipeline( Prodder(), ConsoleEchoer()).run()
However this version will not work because main is a generator not a
regular method.
class Prodder(Axon.Threadedcomponent.threadedcomponent):
def main(self):
while not self.dataReady("control"):
time.sleep(1)
self.send("ping", "outbox")
yield 1
self.send(self.recv("control"), "signal")
> PS - This may not have any effect on anything, but we're not using
> graphline right now and are calling link several times to actually
> link together the mailboxes of various components. Could this keep
> the threadedcomponent's main from starting?
Almost certainly not - I would expect that the issue is that the main
method is a generator. (I've often wondered in recent months whether
this was the right decision)
The reason you'd see this behaviour incidentally is that under the
hood
what actually happens is the main method is used as the new thread
of control inside the thread. If it's a generator method this would
simply create the generator (inside the new thread) and then exit.
Hence confusion.
Hopefully that helps! (If it doesn't please post a fragment showing
the confusing behaviour)
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] Re: Announcement : Kamaelia License Change : Apache 2
On Monday 26 July 2010 13:43:23 Sylvain Hellegouarch wrote: .. > Probably not a surprise but I would definitely suggest WebFaction. They've > always been responsive and reliable. > > http://www.webfaction.com/ On Monday 26 July 2010 14:38:21 Jim Burnes wrote: > If you're just trying to host the source, binaries and Wiki documentation, > bitbucket has worked very well for me. You can get a mercurial account > there for free as long as you don't exceed certain maximum storage > requirements. > > Google code is of course and option. Oh, sorry. That's not what I meant. I'm using the Gandi VPS [1] service at the moment for hosting kamaelia.org, and that's actually working out very nicely, with a similar cost to web faction's prices. [1] http://www.gandi.net/hosting/ What I'm more talking about is that the website is a bit of a mess at the moment IMO, due to doing some emergency stuff. For example, this URL * http://www.kamaelia.org/Home Used to be generated and fed by a error handler which generated HTML on the fly. Since I ended up having less time for the move than intended, I spidered kamaelia.org, to capture the generated pages, and renamed them .html like this: * http://www.kamaelia.org/Home.html I'd not looked into making that switch nicer, primarily because I'd expected to switch back to the original host. Since that's not going to happen, I may as well treat the current gandi hosting as the "final" move, and move on. If I'm going to fix the website anyway, any comments on improving it would be welcome. Every website has annoyances where people say "it needs X". I don't claim to be a web developer (though clearly I can do that) and suggestions for improvement are welcome. (and I'll evaluate based on practicality :-) Any comments welcome :-) Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: Announcement : Kamaelia License Change : Apache 2
Many thanks everyone. Further to this, I've now also updated the entirety of Kamaelia's SVN repository (excluding branches - I'll update those on merge), and also refreshed the Kamaelia homepage. I've also rebuilt the Kamaelia release, which given the version scheme of Y.Y.M.c , means the current release number of 1.0.7.0. I'll be revisiting the website in coming days, and comments and suggestions for improving it would be welcome. In particular there have been issues with hosting, which led to the current solution which I'd considered temporary. Since they look like they're not temporary I may as well decide they're not temporary, and firm things up. To cut a long story short, the machine the site was hosted on was being shut down, moved and should've been switched back on - meaning the current host was intended to be temporary. Since the original host was never switched back on it's left the site in limbo. Deciding to bless the new host as permanent brings some opportunities, which I'll come back to another night. However, any comments and suggestions would be welcome in the meantime would be welcome. Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: Announcement : Kamaelia License Change : Apache 2
On Jul 22, 12:09 am, Michael Sparks wrote: [snip] Re-reading that, sorry for the poor grammar in places. Just excited to be able to say finally this has happened :-) Michael -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Announcement : Kamaelia License Change : Apache 2
Hi, This has been discussed many times, and something I've been pushing forward slowly, and has now come to fruition - we're changing the license of Kamaelia. Rather than being the (as it turns out) difficult to understand Mozilla tri-license, we're changing the license on new releases to the Apache 2.0 license. For those wondering "what sort of license is the Apache 2 license - it's a BSD-like license, and considered by the FSF a free software license. Basically, you can use the code under sensible terms that don't force you to release your code in any particular way. Since all code is either copyright BBC or under contributer agreements allowing this, there aren't any legal issues. Also I've been upfront with everyone over the past couple of years I've been slowing prodding this forward that this was the plan. I'm pretty sure I've discussed this with everyone relevant. If by any chance I've missed you, or you've changed your mine or you've contributed code and have a problem with this, please get in contact offline. And I'll work with you to resolve it. (Even if it means removal of code - just because legally we wouldn't have to, if someone objects, morally I think discussing it and then removal of really strong feelings are felt is appropriate) The key reasons for the change are as follows: * To simplify things for people taking code from us. You no longer need to worry about whether you need to release your code or not. * To simplify contribution. Clause 5 of the apache 2 license is pretty awesome: 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. Contributor agreements are still preferable, but pragmatically the above clause helps in trivial cases. * Since 2004 (Kamaelia's original release), the BBC has released several more projects, and now understands much better the importance of releasing code that matches the community. As a result, the apache 2 license is well known and understood, and used by the BBC glow library for example. (Personally, I think that's a huge step) * Finally, the existance of the Mini Axon tutorial has always led to questions regarding what "import Axon" at the top of a piece of code means. Does it mean our Axon or yours? If it's yours the the GPL can't apply, since you didn't copy our code. Switching to the Apache 2 license that makes it clear that our license doesn't change your licensing. Anyway, I'm delighted to be able to make this announcement. I'm making changes to the repository now to reflect this. Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Changing signature Kamaelia.Device.DVB.Core.DVB_Multiplex
Hi,
I just want to assess the damage of changing the signature of
Kamaelia.Device.DVB.Core.DVB_Multiplex.
Specifically, it would be useful to change this:
def __init__(self, freq, pids, feparams={},card=0):
Such that frequency can embedded inside feparams, and such that
frequency is actually in Hz not MHz.
I can think of some relatively nasty ways of doing this, without
changing the signature, but changing it is cleaner.
This is to simplify dealing with channel.conf lines like this:
CBeebies:
801833000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:
401:402:4672
... which use frequency as an absolute number rather than a float
representing MHz.
This would probably cause breakage to code that uses this component -
in that the way feparams etc is put together would change, so if this
would cause pain for anyone I'm happy to just do this instead:
@classmethod
def from_feparams( new args )
However, before I choose I'd be interested to hear whom this may cause
breakage for.
Regards,
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: First Version of Dendrite Flow-Based Programming System Released
On Thu, Apr 1, 2010 at 11:41 PM, Jim Burnes wrote: > First Version of Dendrite Flow-Based Programming System Released > > The 0.9 Release of the D-Based, Flow-Based Programming System has been > released. > > FBP allows a developer to write simple, cooperative components that exchange > messages via mailboxes to create pluggable, concurrent, modular applications. > > FBP is useful for modelling: > > o Streams > o Pipelines > o Concurrency > o Simulation > o Pluggable Workbenches > > Dendrite was inspired by various FBP systems including the Unix > command shell and > the Python-based Axon system written by Michael Sparks of BBC Research The thing about catching up on email after being away from connectivity is missing really cool announcements. Really pleased you could get this released, and hope it goes well. I'll take a look as soon as I can - sounds neat :-) Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] Dendrite Almost Ready
Hi Jim, On Mon, Mar 29, 2010 at 9:27 PM, Jim Burnes wrote: > Kamaelia People, > > I've already sent this to Michael, but I thought I'd forward it to the group > because we're all in different time zones... I've been away from connectivity for the past couple of weeks, but now that I'm back I'd just like to say congratulations! :-) I'll try and take a look at my earliest opportunity. I've not used D, but this will be a nice introduction :-) Best Regards, Michael -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Re: Any mirrors for tutorials? Main seems down
Incidentally, I've patched the website until DNS changes ripple
through correctly, meaning the site is available, but in a slightly
odd way. Once DNS changes ripple through (next few hours - will be
quicker in different places), things will be back to their usual,
stable, normal.
Regards,
Michael.
On Mar 28, 1:20 pm, Michael Sparks wrote:
> Hi,
>
> The website was briefly down for a few hours when you looked last
> week, and is down again at the moment. (There's, shall we summarise, a
> hosting transition going on at the moment)
>
> That said, there are tutorial materials still online, including the
> Kamaelia tutorial from Europython last year.
> * Slides:http://www.slideshare.net/kamaelian/kamaelia-europython-tutorial
> * Tutorial Notes:
>
> http://www.lulu.com/product/download/kamaelia-pragmatic-concurrency/5...
> (Link to PDF download - free naturally :-)
> (It's hosted on lulu.com because I used them to print up nice
> copies of the tutorial
> notes, which you can also get direct from them. You can
> obviously print your own
> quite happily)
> * Video: (has lip sync issue due to ffmpeg not being fanastic at
> transcoding, but may
> be useful) Tutorial is about 3 hours in length.
>
> http://europython09.blip.tv/file/2323925/- Part 1
> http://europython09.blip.tv/file/2327817/- Part 2
> http://europython09.blip.tv/file/2328583/- Part 3
> http://europython09.blip.tv/file/2328810/- Part 4
>
> Obviously all 3 cover the same basic material, but do so slightly
> differently, due to being different media.
>
> Whilst the site is down, it does mean that you can't get at the
> current release. However you can get a bundle containing the same
> stuff by doing this:
>
> svn checkouthttp://kamaelia.googlecode.com/svn/trunk/Code/Python
> kamaelia
>
> Inside that you'll find 4 directories:
> Apps
> Axon
> Bindings
> Kamaelia
>
> Once you've done that you'll want to do this:
>
> cd Axon # Kamaelia's core
> sudo python setup.py install
> cd ..
> cd Kamaelia # Where a number of tools, examples and Kamaelia
> components live
> sudo python setup.py install
> cd ..
> cd Bindings
> # Inside the various subdirectories, decide whether you want to
> install any of those.
> # Clearly the ones you choose will influence what you can play with
>
> Having pygame installed is highly recommended incidentally. (Given the
> topology visualiser code is quite useful) PyOpenGL is likewise
> recommended for the 3D version, if that's interesting.
>
> Inside the Apps directory, you'll find a bunch of applications, of
> varying states of utility, in there. This includes however Apps/
> Europython09/App which includes ALL the supporting files for the
> Europython tutorial.
>
> This includes:
>
> MiniAxon- { 0 .. 5 }.py - various versions of a miniaxon, from a basic
> run through of generators through to a simple app for pushing a file
> out over a multicast socket.
>
> BB { 1 .. 15 } .py - a walkthrough of how to grow and evolve an old
> school style bulletin board from a simple request/response protocol,
> to something you can login to, and then rummage around inside menus
> over a network connection.
>
> simplegame-{1 .. 2 }.py - 2 simple files showing how to control 2
> sprites, and also how to get them to update a shared game state.
> (There are other ways, but this also shows how to use the concept of
> "backplanes" in a nice way)
>
> Video-{ 1 .. 5 } .py - Shows how to use the pygame webcam module, and
> then how to use that to create a Kamaelia based video recorder that
> stores the video in Dirac format. Early versions only requires
> Kamaelia & Pygame (latest version), later versions of this require
> Dirac & Dirac bindings installed.
>
> SimpleDiracPlayer.py - A simple Dirac based video player. Requires
> dirac & the dirac bindings to be
> installed and working.
>
> Finally, you'll probably want to look at the STM, which you can view
> as being like version control for values for components to check out/
> in.
>
> Hopefully the site will be back up (permanently) in the next couple of
> days. In the meantime, hopefully this message is useful!
>
> Regards,
>
> Michael.
>
> On Mar 22, 1:35 am, "hb,cse" wrote:
>
> > Hello.
>
> > I'm an absolute newbie, and I need to get started. I understand your
> > MiniAxon tutorial is geared to my level, but I can't get to it -
> > kamaelia.org seems dead, and I cannot find your tutorials anywhere
> > else, either. Is there a mirror somewhere, wh
[kamaelia-list] Re: Any mirrors for tutorials? Main seems down
Hi,
The website was briefly down for a few hours when you looked last
week, and is down again at the moment. (There's, shall we summarise, a
hosting transition going on at the moment)
That said, there are tutorial materials still online, including the
Kamaelia tutorial from Europython last year.
* Slides: http://www.slideshare.net/kamaelian/kamaelia-europython-tutorial
* Tutorial Notes:
http://www.lulu.com/product/download/kamaelia-pragmatic-concurrency/5043299
(Link to PDF download - free naturally :-)
(It's hosted on lulu.com because I used them to print up nice
copies of the tutorial
notes, which you can also get direct from them. You can
obviously print your own
quite happily)
* Video: (has lip sync issue due to ffmpeg not being fanastic at
transcoding, but may
be useful) Tutorial is about 3 hours in length.
http://europython09.blip.tv/file/2323925/ - Part 1
http://europython09.blip.tv/file/2327817/ - Part 2
http://europython09.blip.tv/file/2328583/ - Part 3
http://europython09.blip.tv/file/2328810/ - Part 4
Obviously all 3 cover the same basic material, but do so slightly
differently, due to being different media.
Whilst the site is down, it does mean that you can't get at the
current release. However you can get a bundle containing the same
stuff by doing this:
svn checkout http://kamaelia.googlecode.com/svn/trunk/Code/Python
kamaelia
Inside that you'll find 4 directories:
Apps
Axon
Bindings
Kamaelia
Once you've done that you'll want to do this:
cd Axon # Kamaelia's core
sudo python setup.py install
cd ..
cd Kamaelia # Where a number of tools, examples and Kamaelia
components live
sudo python setup.py install
cd ..
cd Bindings
# Inside the various subdirectories, decide whether you want to
install any of those.
# Clearly the ones you choose will influence what you can play with
Having pygame installed is highly recommended incidentally. (Given the
topology visualiser code is quite useful) PyOpenGL is likewise
recommended for the 3D version, if that's interesting.
Inside the Apps directory, you'll find a bunch of applications, of
varying states of utility, in there. This includes however Apps/
Europython09/App which includes ALL the supporting files for the
Europython tutorial.
This includes:
MiniAxon- { 0 .. 5 }.py - various versions of a miniaxon, from a basic
run through of generators through to a simple app for pushing a file
out over a multicast socket.
BB { 1 .. 15 } .py - a walkthrough of how to grow and evolve an old
school style bulletin board from a simple request/response protocol,
to something you can login to, and then rummage around inside menus
over a network connection.
simplegame-{1 .. 2 }.py - 2 simple files showing how to control 2
sprites, and also how to get them to update a shared game state.
(There are other ways, but this also shows how to use the concept of
"backplanes" in a nice way)
Video-{ 1 .. 5 } .py - Shows how to use the pygame webcam module, and
then how to use that to create a Kamaelia based video recorder that
stores the video in Dirac format. Early versions only requires
Kamaelia & Pygame (latest version), later versions of this require
Dirac & Dirac bindings installed.
SimpleDiracPlayer.py - A simple Dirac based video player. Requires
dirac & the dirac bindings to be
installed and working.
Finally, you'll probably want to look at the STM, which you can view
as being like version control for values for components to check out/
in.
Hopefully the site will be back up (permanently) in the next couple of
days. In the meantime, hopefully this message is useful!
Regards,
Michael.
On Mar 22, 1:35 am, "hb,cse" wrote:
> Hello.
>
> I'm an absolute newbie, and I need to get started. I understand your
> MiniAxon tutorial is geared to my level, but I can't get to it -
> kamaelia.org seems dead, and I cannot find your tutorials anywhere
> else, either. Is there a mirror somewhere, where I can find docs and a
> tutorial ? (Preferably a complete one - I need concurrency to write a
> simulator, network functionality not needed.)
>
> Sincerely,
> HB
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Revisiting the idea of path/autoload
Hi,
I raised this idea last summer which I've bounced around for a while,
which was to enable this:
from Kamaelia.Util.Backplane import Backplane, PublishTo,
SubscribeTo
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Chassis.ServerCore import ServerCore
from Kamaelia.File.Append import Append
from Kamaelia.Util.PureTransformer import PureTransformer
To be also said/written
from Kamaelia.Path import Backplane, PublishTo, SubscribeTo, \
Pipeline, ServerCore, Append, PureTransformer
What I've realised since then is that the primary objections were:
* Knowing where things are coming from and never worry about
name clashing
* Ability to query this statically, clearly
* How to deal with overrides/name clashes
* How to change the search order (eg for experimental versions and/
or
testing purposes)
* The idea that the library should do no magic, leaving the library
user in charge.
Therefore my suggestion is this:
* You can spell the imports above either way. People who do not
wish
to use this functionality would be in no way expected or required
to do so.
* Code that enters into the main repository (ie into the Kamaelia.*
namespace) would be required NOT to use Kamaelia.Path code (at
least initially - to see how things go). ie whilst a user of the
library can choose to enable "magic" for *their* script, the
library itself refrains from doing so.
* for Kamaelia.Path, importing a CamelCase or Caps'dname ALWAYS
imports a Component, no exceptions.
* Kamaelia.Path ONLY affects loads from Kamaelia.Path & nowhere
else
* Kamaelia.Path would also contain utility functions & attributes,
which are always lower case including:
* Kamaelia.Path.path would be modifiable, ala sys.path -
extendable
or replaceable. this would be a list of namespaces to search
in
what order. These would default to Kamaelia, but /could/
include
other namespaces.
* Kamaelia.Path.which would return the fully qualified
component
name. For example Kamaelia.Path.which("Pipeline") would return
"Kamaelia.Chassis.Pipeline.Pipeline"
* Kamaelia.Path.trace - flag which if set to True dumps what
components are being loaded from what files, and _why_ .
* Kamaelia.Path.doc would return the docstring for a given
Kamaelia
component (formatted to take into account docs for inboxes and
outboxes).
* Kamaelia.Path.webdoc would return the weblink for the same
docs.
I think that this combination would together be both useful for small
testing scripts, and incremental development. It would simplify cases
that really ought to be simple, and not get in the way of current
development styles.
FWIW, you could also think of Kamaelia.Path as being similar to the
unix command "env", as used like this:
#!/usr/bin/env python
ie a tool explicitly there for making it simpler to pick "the right"
thing, based on a known, defined path. Some people like that tool,
some hate it - which is what I suspect would be the reaction to be
for Kamaelia.Path
Any thoughts welcome.
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Not quite on or off topic: Chapel - a high level parallel language from Cray
Hiya, Saw this today, and its syntax is intriguing. * http://chapel.cray.com/ Description: Chapel is a new parallel programming language being developed by Cray Inc. as part of the DARPA-led High Productivity Computing Systems program (HPCS). Chapel is designed to improve the productivity of high-end computer users while also serving as a portable parallel programming model that can be used on commodity clusters or desktop multicore systems. Chapel strives to vastly improve the programmability of large-scale parallel computers while matching or beating the performance and portability of current programming models like MPI. Chapel supports a multithreaded execution model via high-level abstractions for data parallelism, task parallelism, concurrency, and nested parallelism. Chapel's locale type enables users to specify and reason about the placement of data and tasks on a target architecture in order to tune for locality. Chapel supports global-view data aggregates with user-defined implementations, permitting operations on distributed data structures to be expressed in a natural manner. In contrast to many previous higher-level parallel languages, Chapel is designed around a multiresolution philosophy, permitting users to initially write very abstract code and then incrementally add more detail until they are as close to the machine as their needs require. Chapel supports code reuse and rapid prototyping via object-oriented design, type inference, and features for generic programming. Chapel was designed from first principles rather than by extending an existing language. It is an imperative block-structured language, designed to be easy to learn for users of C, C++, Fortran, Java, Perl, Matlab, and other popular languages. While Chapel builds on concepts and syntax from many previous languages, its parallel features are most directly influenced by ZPL, High-Performance Fortran (HPF), and the Cray MTA's extensions to C and Fortran. Or more usefully - copied from the download, two examples: * Generic linked list : http://pastebin.com/f75623d12 * Producer Consumer : http://pastebin.com/f6f69a91d Interesting to note the use of a yield keyword used in a very python like way. The fact it supports generic containers cleanly, and yielding behaviour strikes me as an interesting example to play with making a mini axon for. (Adapting the two above exampes would be where I'd personally start) Michael -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Kamaelia.org moved
Hi, I've now moved the contents of Kamaelia.org from one server to another. There are some differences to the set up - specifically, rather than running from a wiki engine, this is now a static dump. Whilst there's some positives to that, there are also negatives - such as bust a few bust links, and missing ".html" bits meaning that some old links will render dumbly. I'll be patching those as things go. The reason for this change BTW, is due to the fact that I've had to shift the site off a BBC owned & hosted machine (at least temporarily) to a virtual machine of mine (which I rent form gandi.net). The reason for that is because the BBC owned & hosted machine I was using will be switched off very shortly because the site it's on is closing and everyone & everthing there is changing sites. If there's any q's, please ask away. If you find a problem, please let me know (here, rather than privately). One major one I'm aware of at present is that this: ... http://www.kamaelia.org/Home Does the wrong thing at present, and should be this: ... http://www.kamaelia.org/Home.html I'll probably put in an apache redirect directive for that specific one before doing a general solution. (Via an error handler) Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Rejected: private_JMB_MPS_XMPP_WSGI
Hi, Last year I fought the above branch for many months seeking to make it something that a) worked b) was usable. I have decided, regretfully, to cease spending any of my time on it. Rationale: * In practical terms it was sucking up far too much of my time, since the code was non-idiomatic for either python or Kamaelia. * It didn't work, and still largely doesn't work. * It provided no way of testing the system as a whole, nor in part. * It was not documented, or where documentation was included it was either unclear, patchy or wrong. (no docs are better than broken docs) * The approach taken to inclusion of a seperately library appeared to be copy and paste, rather than wrap and use. Thereby making the code significantly more fragile. This goes rather heavily against Kamaelia's approach of modularity. * The coding style is extremely monolithic, rather than modular. Kamaelia systems are generally testable in small fragments. Crucially though: * The original developer dumped the code and ignored it. (This isn't necessarily a problem for code that is documented and works, if the code is useful enough - see above on this point). * The original branch proposer did not assist in fixing the code, nor documenting it. * It is largely code for its own sake - it has no user & no potential user. For these reasons I'm rejecting the branch. I should have done this 4 months ago when I actually stopped working on fighting this branch, but like many people I don't like giving up on something. Also, as many people know, much of my development of Kamaelia itself as a project is now done in my own time, so the time I have to fight LARGE branches which don't work, aren't documented, aren't maintained, and don't have a developer taking ownership of their own code, is minimal. As a result, I've decided that the potential benefits of working with this code to get it working are outweight by the time cost incurred (to me, personally) of doing so. I'm sorry to the original developer who will read this and probably be disappointed, but if you don't care enough to make your code work and use it, I don't have to either. Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] envbuilder file for Kamaelia
On Saturday 30 January 2010 13:40:31 Jason Baker wrote: > PS. Note to ms: this does use setuptools. :-) Rejected then. Regards, Michael. -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] What is the state of multicore in Kamaelia?
Jason, This is very bad timing. Ask this question again when it's not holiday season. Regards, Michael. On Thu, Dec 24, 2009 at 2:00 PM, Jason Baker wrote: > I know it's been a while since I've looked at Kamaelia, but I might > have a usecase for it at my current job. Performance is an area > that's really hurting us right now, and I've been asked to parallelize > some of our data loads. > > Last time I checked, the Process Pipeline and Process Graphline where > highly experimental (and that's still where they are listed). Are > they any more stable or mature now? > > Alternatively, I could also work with a ProcessComponent using > multiprocessing. It looks like it wouldn't be *too* difficult to > adapt a ThreadedComponent to do this (although there may be some > "devil's in the details" type things). The only issue that I forsee > is that multiprocessing has been backported to python 2.4/2.5. Is > Kamaelia still aiming to be compatible with Python 2.3? > > -- > > You received this message because you are subscribed to the Google Groups > "kamaelia" group. > To post to this group, send email to kamae...@googlegroups.com. > To unsubscribe from this group, send email to > kamaelia+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/kamaelia?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
[kamaelia-list] Fwd: Re: Re: Tuning
Bouncing this mail to the mailing list. Regarding the comment "the mailing list is deleted?" - no the mailing list moved to google groups a long while back - __as that page states__ . Incidentally, Mailing me off list repeatedly is not a good way to guarantee a response. When people have license queries I'll often happily answer them off list, since really whilst relevant isn't particularly interesting to a wider audience IMO. But when there's a technical q, asking only one person your question limits the people who can answer your question - reducing the chances of a useful answer. (sorry if this comes over as overly terse to anyone, I'm a bit pressed for time at the moment) (translated to italian using google translate - for luca : Tradotto in italiano con Google Translate - per luca: Bouncing questa mail alla mailing list. Per quanto riguarda il commento "la mailing list è cancellato?" - La mailing list non si trasferì a Google Gruppi a lungo indietro - __as questa pagina states__. Per inciso, a me off Mailing list volte non è un buon modo per garantire una risposta. Quando la gente ha domande di licenza io spesso felicemente rispondere fuori dalla lista, perché in realtà pur rilevanti, non è particolarmente interessante per un pubblico più ampio IMO. Ma quando c'è un q tecnico, chiedendo solo una persona alla tua domanda limita le persone che possono rispondere alla tua domanda - la riduzione della possibilità di una risposta utile. Ci dispiace se questo si avvicina eccessivamente laconico a nessuno, io sono un po 'poco tempo in questo momento. ) Michael. -- Forwarded Message -- Subject: R: Re: Tuning Date: Monday 30 November 2009 From: "lucabe...@libero.it" To: spark...@gmail.com Hello and sorry if i reply to your mail, but the mailing list is deleted? http://www.kamaelia.org/mailinglists second point really this is my file : import dvb3 from dvb3 import frontend scheda = dvb3.frontend.Frontend(0) frequency = 11977 parametri = dvb3.frontend.QPSKParameters(frequency=frequency, inversion=dvb3.frontend.INVERSION_AUTO , symbol_rate=27500, fec_inner=dvb3. frontend.FEC_AUTO) params = (parametri.frequency, parametri.inversion, parametri.symbol_rate, parametri.fec_inner) scheda.set_frontend(params) And finally yes i'm Italian and my english is bad. (sorry), buti understand what you write. Thanks Luca >Messaggio originale >Da: spark...@gmail. com >Data: 28/11/2009 15.34 >A: "luca72" >Ogg: Re: Tuning > >Luca, > > >On Saturday 28 November 2009 00:06:08 luca72 wrote: >> Have you any idea > >I have asked you to post the entirety of the files you are using. If you don't >post the entirety of the files you are using I cannot debug your problem. >(That's why I have not replied further) > >Seriously, I want to help, but if you want my help, you do need to provide me >with full information. This means posting: > * A complete file which you are running > * If you are changing Kamaelia files, either: > - Posting a diff created by: > svn diff > - Post a diff using the "diff" command > - Post a copy of the *complete* file you are changing. > >Finally, using full sentences help. > >I note from your email address that you may be italian and english may not be >your first language. As a result I am also posting this using a translation >from google translate. > >Also, please post queries to the kamaelia mailing list. Mailing me direct >risks me not seeing your mail and it going unanswered. > > >Italian version translated using translate.google.com: >Versione italiana tradotta con translate.google.com: > > >Ho chiesto di pubblicare l'insieme dei file che si sta utilizzando. Se non >inserisci la totalità dei file che si sta utilizzando non riesco a eseguire >il debug del problema. (Ecco perché non ho più risposto) > >Scherzi a parte, voglio aiutare, ma se si vuole il mio aiuto, si ha bisogno di >fornirmi tutte le informazioni. Questo significa distacco: > * Un file __completo__, che si esegue > * Se si stanno modificando i file Kamaelia, sia: > - Registrazione di una diff creato da: > svn diff > - Invia un diff usando il diff "comando" > - Invia una copia del * completa * file che si sta cambiando. > >Infine, usando frasi di aiuto. > >Prendo atto dal tuo indirizzo e-mail che vi possono essere italiano e in >inglese non può essere la vostra prima lingua. Come risultato sono anche >postando usando una traduzione da Google Translate. > >Inoltre, si prega di inviare domande alla mailing list kamaelia. Mailing me >rischi diretti a me non vedere la posta e andare senza risposta. > > >Regards, > > > > >Michael. >-- >http://yeoldeclue. com/blog >http://twitter.com/kamaelian >http://www.kamaelia.org/Home > > -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this gro
Re: [kamaelia-list] Re: simple tuning operation
On Monday 30 November 2009 20:34:34 luca72 wrote: > No one can help me please > Luca > > On 29 Nov, 12:03, luca72 wrote: > > Hello again : > > this is my file: > > > > import dvb3 ... I haven't had time yet to reply. Sorry. (I've been busy doing work today, and spending yesterday with family) That was the first time you'd posted something to the list which looks debuggable though. I'll take at some point this week. I don't have a DVB-S card though, so I won't be able to fully test, but I'll take a look. Michael -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en.
Re: [kamaelia-list] Re: Hello
On Monday 23 November 2009 13:29:13 luca72 wrote: > I reply here because if i reply to the previous message i get the > error and i can't post What error are you seeing? I'm not seeing anything in the google groups moderation interface. The first post of someone is moderated to ensure that the group doesn't get spammed - it gets about 1/2 dozen spams per day which I don't forward. Once I identify posts as coming from a person their posts are auto-moderated. Your settings in the system are: * lucabe...@libero.it luca72 Nov 17 Email Allowed .. and there's no pending messages so you shouldn't be having issues. Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=.
Re: [kamaelia-list] Re: Tuning
Luca, On Friday 20 November 2009 09:36:20 Michael Sparks wrote: > until I see the full file you're *currently* > editting and the *current* sourcefile, I have no chance of reconstructing > exactly what's happening. Third time lucky, hopefully. Please can you post: * The code that you have tried running into pastebin.com * The full stack trace/error that you are seeing into pastebin.com I can't really help you properly unless you do this. (I'll be making random stabs in the dark which help no-one) Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=.
Re: [kamaelia-list] Re: Tuning
Hi Luca,
On Friday 20 November 2009 07:39:12 luca72 wrote:
> now i can't test it
>
> Many thanks i will inform you.
> PS:I think that i'm your nightmare (sorry)
No, not nightmare. Someone testing current code in a scenario we haven't tried
since that code was written is actually a dream scenario. What is making it
hard to help is not posting a copy of the actual code you're running.
Fragments are good, and you have the context of the fragments, so they make
sense to you. However, simply posting this by itself...
from Kamaelia.Device.DVB.Core import DVB_Multiplex
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.File.Writing import SimpleFileWriter
import dvb3
freq = 119.77
feparams = {"symbol_rate" : 27500,
}
(for example - some your mail at 13:41:15 -0800 (PST) )
... doesn't give me the full picture.
When you then post this:
and i get the following error
The file frontend.pyx could not be opened.
...
and after i gat :
The debugged program raised the exception unhandled IOError
"(22, 'Invalid argument')"
File: frontend.pyx, Line: 364
These two errors look contradictory, which is confusing.
I'm explaining this because taking the time to report what you're trying, and
the results you're seeing is fantastic feedback, and I suspect looking at the
original early code that we may need to change some code with regard to
configuring the front end, but until I see the full file you're *currently*
editting and the *current* sourcefile, I have no chance of reconstructing
exactly what's happening.
In order to allow me to help you, I simply need that information.
Once again, regarding this:
> PS:I think that i'm your nightmare (sorry)
Please let me put your mind at rest - someone (anyone) taking the time to
report back and try using code that hasn't been tested in the scenario you're
using it, is not a nightmare, it's closer to a dream scenario - since it's
useful feedback.
At present I think, that the current code is not configuring the satellite
frontend before passing in the fe.params. It looks like with DVB-T & -C you
can more or less "just use it", whereas with DVB-S the setting of horizontal
matters. This quite possibly requires a change to our code as well as yours,
hence me asking for information which you may feel repetitious, however what
I'm simply doing is trying to see precisely what's changing in the system as
a whole :-)
Thanks for persevering! :-)
Regards,
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=.
Re: [kamaelia-list] Re: Tuning
On Thursday 19 November 2009 21:41:15 luca72 wrote: > i make a little change but i get another error: Could you post the whole file into a pastebin ? eg http://pastebin.com/ Along with the full trace back? One thing that does look wrong to me is given this: * Frequency 11977 * pid 1296 * Horizontal * 27500 Shouldn't translate as: freq = 119.77 .. DVB_Multiplex(freq, [0x1296] That should be: freq = 11977 ... DVB_Multiplex(freq, [1296], ... The numbers look higher than DVB-T because DVB-S does work at a higher frequency. It will be easier to help with a copy of the full file & traceback in a pastebin though. Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=.
Re: [kamaelia-list] Re: Tuning
On Thursday 19 November 2009 07:25:56 luca72 wrote: > Thanks for the linkd page, in the example there are dvb-t example but > i con also use for dvb-s making some change is it correct? You'll be pleased to hear that the dvb bindings were first used with a DVB-S card. Most current testing has been with DVB-T, but they should still work with DVB-S. (We've made changes to support DVB-C as well when a patch came our way BTW :) Michael -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=.
Re: [kamaelia-list] Tuning
On Wednesday 18 November 2009 20:31:34 luca72 wrote: > Where i can find a tuning example The component documentation generally has an example of usage for most components. So for example this page has an example + explanation: http://www.kamaelia.org/Components/pydoc/Kamaelia.Device.DVB.Tuner.html More generally if you scroll down to Kamaelia.Device.DVB on this page you can click through to modules & components: * http://www.kamaelia.org/Components There are also example DVB systems in the distribution, which you can also find here: http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Kamaelia/Examples/DVB_Systems/ Finally there's also some examples describing how to use the DVB code here: * http://www.kamaelia.org/Cookbook That has some small examples, as well as some larger systems such as this one which records programmes by name: * http://www.kamaelia.org/Cookbook/DVB/PersonalVideoRecorder There's also a more substantial DVB system described here which can make your timeshifting decisions retroactive: * http://www.kamaelia.org/KamaeliaMacro Are those the sort of examples you've been looking for ? Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=.
Re: [kamaelia-list] Help for installation
Hi, On Tuesday 17 November 2009 20:52:18 luca72 wrote: > Pls Help Good news: I had exactly this problem the other day when reinstalling a new machine from scratch with ubuntu 9.10 - and solved it happily Bad news: I'm nowhere near that machine at the moment to tell you precisely what I did. Hopefully the following from memory helps: Outline though: (would want to double check .bash_histories really though) I made sure that the ubuntu version of Pyrex was NOT installed. I made sure the python-devel package was installed I grabbed this: http://www.kamaelia.org/release/deps/Pyrex-0.9.3.1.tar.gz Installed that using the usual "python setup.py install" dance I used a svn check out of /trunk - eg: svn co http://kamaelia.googlecode.com/svn/trunk kamaelia-trunk Installed Axon Installed Kamaelia Went into Bindings/python-dvb3 Installed that. Things then worked as expected. Hopefully that helps. (I won't be near the machine I was doing this on until thursday or friday I'm afraid) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home -- You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamae...@googlegroups.com. To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/kamaelia?hl=.
Fwd: Re: [kamaelia-list] Re: Bidirectional Json-Rpc Server
(forwarding as per final line :)
-- Forwarded Message --
Subject: Re: [kamaelia-list] Re: Bidirectional Json-Rpc Server
Date: Friday 13 November 2009
From: Jim Burnes
To: Michael Sparks
Very interesting discussion, Michael. When I first considered the concept
of stream / pipe polymorphism it really rattled my brains. Basically they
are objects that conform to the shape of their current component. It blew
my mind and took me a while to consider the impact. At the time I was
thinking about this I was designing a malware analysis pipeline modeled
after the human immune system and the conformable aspect had many
similarities to how immune system cells can morph depending on their current
function.
I've not had a chance for detailed comment, but two brief comments:
>* Yes, sending components through Axon/Kamaelia pipelines does already
> happen. (Indeed, we'll often send around ("inbox", component) pairs
> because that's defined as a service.
>
Heh. I hadn't even yet considered pipelining Axon's own core objects
(mailboxes, microprocesses etc). Quite a bit of potential there for danger
:-)
Python-Axon pipelines are fully polymorphic, and as you suspected,
> fun.
> It's possible that typed inboxes may be useful though, but that's a
> different discussion.
>
Sure. If nothing else they carry type-safety and certainly performance
improvements by avoiding the serialization issues, but you still have the
capability to go all string or whatever. The utility of doing this is
another question. When everything is a string you have more flexibility in
what you can read or connect to. Everything is trade offs.
>
>* If you pass in-active/passive objects - ie not components and not
> microprocesses - through the pipeline, because they don't have a local
> thread of control, this is generally thread safe.
>
Good. I had considered this and it's nice to know that someone has
considered it.
>
> After all, when you put an object in an outbox, you no longer own it,
> and if you take something from an inbox you own it.
Yes.
This ensures the idea of single reader/single writer for any single piece
> of data, which
> eliminates whole classes of concurrency bugs. (You don't necessarily
> make a system much simpler, but you at least you know one sort of bug
> doesn't exist, and one that's normally awkward to sort out/find :-)
>
Good point.
>
> BUT, if you put a component in the pipeline, you need to be careful.
>
>
No doubt playing with serious FM at that point.
> Specifically, this means a component is expecting a component on an
> inbox. In that case, your recipient component cannot know in advance
> whether the recipient is a generator-type component, threaded
> component,
> or similar. (eg a generator inside a threaded chassis, which has been
> proposed)
>
>
Bottom line really:
>* If possible send passive objects.
>
>
Certainly.
>* Finally, consider building something actor-like on top. Everyone who's
> built a mini-axon has actually built something actor-like, so this
> shouldn't be that bad, but there's no real direct support for ad-hoc
> actor like stuff as yet.
>
> Michael -- since I haven't used Actor I'm not sure what that implies.
>
> [Incidentally Jim, I suspect you may be aware of these issues, but I'm also
> aware that the thread is archived, and may be read in future :-) ]
>
> > I still think that you may need some sort of "bus" or "kernel" object for
> > mailbox creation, but that may be a orthogonal issue.
>
> That's the purpose behind adaptive comms components really - since they can
> all create mailboxes.
>
>
I'm thinking from a single-responsibility standpoint, keeping the mailbox
and process factories in the kernel clean up a lot of coupling issues.
> Interestingly, whilst we don't do this in Kamaelia/Axon, MASCOT's
> equivalents
> to out inboxes/outboxes can exist independently of what we call components.
> It may be that upgrading inboxes/outboxes to fully fledged things may have
> some benefits.
>
>
Yes. My mailboxes are independent, but because parts of their code (at
least mine) require them to know their parent component it's hard for them
to exist independently unless you use an abstract class for inboxes and then
use a concrete class of inbox that "belongs" to a component. Right now I
think I'll try to hide those implementation details and let the kernel worry
about ownership and parent/child issues. Part of the refactoring that I'm
trying to complete.
BTW: Here's an i
[kamaelia-list] Re: Bidirectional Json-Rpc Server
On Friday 13 November 2009 04:36:56 Jim Burnes wrote:
> Good luck and let us know how it works. It sounds like a good idea. The
> objects encapsulate all their state, components press the button methods
> and events are pushed down the pipeline.
I've not had a chance for detailed comment, but two brief comments:
* Yes, sending components through Axon/Kamaelia pipelines does already
happen. (Indeed, we'll often send around ("inbox", component) pairs
because that's defined as a service.
Components that send components through inboxes include the TCP
subsystem and pygame subsystem.
Python-Axon pipelines are fully polymorphic, and as you suspected, fun.
It's possible that typed inboxes may be useful though, but that's a
different discussion.
* If you pass in-active/passive objects - ie not components and not
microprocesses - through the pipeline, because they don't have a local
thread of control, this is generally thread safe.
After all, when you put an object in an outbox, you no longer own it,
and if you take something from an inbox you own it. This ensures the
idea of single reader/single writer for any single piece of data, which
eliminates whole classes of concurrency bugs. (You don't necessarily
make a system much simpler, but you at least you know one sort of bug
doesn't exist, and one that's normally awkward to sort out/find :-)
BUT, if you put a component in the pipeline, you need to be careful.
Specifically, this means a component is expecting a component on an
inbox. In that case, your recipient component cannot know in advance
whether the recipient is a generator-type component, threaded component,
or similar. (eg a generator inside a threaded chassis, which has been
proposed)
This means the instant you start calling methods on components rather
than sending them messages, you have to start thinking about thread
safety. Now, if you're doing "read only", and not worried about the data
going stale, that's one way of dealing with it. (I'm actually doing this
in a project at the moment :-) that I can't really talk about
publically )-:
Picking a neutral example though if you had a component that kept track
of a piece of information that changes often - such as directory size,
the current time, how many users are connected, then that's probably OK.
If you do need to change the component's state in some fashion though you
have an issue. For example if you do this:
somecomponent.somestore.append(something)
And in its thread of control "somecomponent" does this in a loop: (say)
self.somestore = []
You have a clear issue depending on ordering of instructions.
There are two ways round this:
* Build something actor like based on threadsafe queues. This is what
the scheduler does incidentally for handling .activate() calls in
a threadsafe manner. ie accept the call, append the request to an
inbound queue, and then inside the component's main thread service
these calls there. (This cries out for decorator support :-)
* The other is to use the STM code, which is probably far more better
suited to this sort of task. http://www.kamaelia.org/STM describes
it, but effectively you do this:
1 Create an STM store.
2 Allocate some names in it
3 Grab a checkout containing those names/values
4 Update the values in that checkout
5 Check in/commit those values
6 If you get a conflict, go back to 4
ie the same as databases or version control (sans versions)
Bottom line really:
* If possible send passive objects.
* If sending components, ideally the safest way of dealing with them is to
create a link to the component (using self.link) from a local outbox,
send a message to it and get a message back. (by telling it where to
respond to.)
eg: (for one offs)
somecomponent = self.recv("somebox")
link = self.link((self,"adhoc"), (somecomponent, "inbox"))
self.send( { "req" : "some request",
"callback" : (self, "someinbox") }, "adhoc")
self.unlink(link)
(Incidentally in Axon2 I'm planning on calling this "post" :-)
* If that's inappropriate for whatever reason, you need to consider
threadsafety, at that point, seriously consider using the STM code -
it's what it's there for.
* Finally, consider building something actor-like on top. Everyone who's
built a mini-axon has actually built something actor-like, so this
shouldn't be that bad, but there's no real direct support for ad-hoc
actor like stuff as yet.
* If you're merely reading values via a method call,
[kamaelia-list] Re: Subclassing Kamaelia.Util.Backplane
On Tuesday 10 November 2009 14:36:38 Jim Easterbrook wrote: > Is it possible to subclass Backplane() in order to create a backplane > that knows when a subscriber is added or removed? > > I have a CPU intensive process creating stuff that is published to a > backplane, and I'd like to suspend processing when there aren't any > subscribers, to minimise load on the system. My first thought was to > subclass Backplane(), but I'm not getting anywhere with that. > > Is this a silly idea? Is there another way of doing it that I haven't > discovered? It's not a silly thought - the backplane code was written incredibly quickly, and is really quite simple so changing it to support this would be useful. I'll have a brief ponder. (The way to do it would be to also change the primary data inbox to a null sink so that it can throw away data no one is listening to without reactivating) An overview of how the backplane works inside is described incidentally on pages 49 - 52 of the kamaelia tutorial which can be downloaded from here: * http://www.kamaelia.org/PragmaticConcurrency (Incidentally, if you'd like a paper version, I can send you a copy in the the internal mail :) Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Bidirectional Json-Rpc Server
On Sun, Nov 1, 2009 at 2:05 PM, Rasjid Wilcox wrote: > 2009/7/15 Rasjid Wilcox >> >> 2009/7/12 Michael Sparks : >> >> Due to other work commitments it will be a couple of weeks before I >> get to do much with this. >> > > Sigh. It has been a very long 'couple of weeks'. Work and personal life > has been pretty intense the last few months. But I'm still here and have > started working on a revised version of the JsonRPC server and client. Will > post back when I have something to show. :-) I have a feeling that many engineers who rely on working on projects on their own time a) estimate relatively accurately the length of time before something will happen b) but inaccurately assume that the duration relates to absolute time :-) Since it's quite late here I'll indulge myself by explaining a touch. If I have an average of 8 hours per week available for coding on my own time (say), and "I'll do that in 2 weeks" means "I'll do that in 7*24*2 hours time - 336 hours time". However, factoring in that 8 hours passes for every 168 hours of real time, means that actually it's (168/8)*336 - which is 7056 hours of real time. That's actually approximately 42 weeks. Considering this thread actually started roundabout in April, 42 weeks added onto that is Jan/Feb next year, so I wouldn't worry too much about what 2 weeks is here :-) More seriously, I understand what it's like when work & personal life go into overdrive & glad to hear that you're still planning on this - as usual, if you need anything from me to help, let me know :-) Regards, Michael. (he says wondering what the metric is here - maybe we should measure things in tuits :-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: What's Cooking?
On Sunday 11 October 2009 23:52:42 Steve wrote: > I was wondering if you could take just a couple minutes to tell us > what's cooking and maybe how much work is left to do. Just a quick note to say that I'm writing a reply to this, and thank you for asking the question. It's a really nice question, and it deserves a decent answer, if only because I can make a plea for help that way :-) Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Feedback requested: Decorating Generators to make components, perhaps pythonic ?
On Sunday 11 October 2009 23:32:42 unetright.thebas...@xoxy.net wrote: > On Sun, Oct 11, 2009 at 8:09 AM, Michael Sparks - spark...@gmail.com wrote: > > No positive responses. > > > > Probably not worth the effort in to clean up and merge this in that > > case. > > The intrawebs are slow don't you know. ;) Also a good reason not to rush :-) > > However, this looked like one step too far towards voodoo for Steve & > > IIRC John agreed. In retrospect, I agree, though it could be a > > convenient shortcut. > > Thanks for remembering. My brain is slow when coming back to a > project I haven't worked with in 6 months. And decorators that take > parameters tend to be so much harder for me to reacquaint myself with. It was a good argument/rarionale. It's also an easy one to accept, so that's why I accepted it :-) > I do really like the ease of construction that you're shooting for > with these decorators. I would probably use them too for very simple > unix shell style transformation components. As you might expect, I'm not 100% convinced at this stage. (If someone else is though, they could always refine the code further) The difference though is 4 months ago I couldn't even see how to go about writing these decorators (well the TransformGenComponent one at least). > Most of the kamaelia > components I've written have been more involved though. Likewise. Not all, but many. > I haven't > personally yet needed a utility where I wanted to do some simple > composition functions (tail, grep, print, etc) AND thought that I > should use a concurrency system like kamaelia instead of a plain > python script. I'm sure others have though. I can see precisely where you're coming from with this. I'll often write a shell script where a shell script will suffice, and if I need more, then I'l write a python program. If it turns out to be useful, I'll probably componentise that program. This is potentially a halfway house. I'm not sure how useful a one at present. > > In this case concept comes first, followed by an implementation. Point is > > taken though :-) Suggestions for better names welcome. (As noted before) > > Anyway, I do like the ease of use. That's useful to know. > And I hate to be another person to > harp on the names. Well, I did ask for better names. :-) > Honestly though, it's only easier for me if when I > come back after 6 months, it's immediately obvious what's going on in > the hidden decorator functionality. Comprehending what _my_ function > did and holding in my head the code from somewhere else about what > your decorator is doing is just too much for me dull brain. On the > otherhand, if the name of the decorator is extremely clear, I don't > have to lookup the decorator code as I just take it on faith that it > does what it's name says. Some examples would be like: > @run_in_transaction > @retry_on_exception(max_tries=3) > @wrap_with_context_manager > > So, that said, I'd personally +1 your sketch decorators if they could > be given names that were not "what they are" but instead were "what > they do". Perhaps: > @TransformerGenComponent -> @kComponent_from_generator > @blockingProducer -> @kComponent_from_blocking_generator > > I really prefer the explicit nature of telling me what's going on. Interesting. I've often noticed a certain terseness in python code, sometimes to the point of obscurity, whereas I tend to favour more descriptive names. In this case I hadn't thought of the idea of describing the generators in terms of "what they do", because the most common decorators I use are @classmethod, @staticmethod - and similar - things that describe what the thing is, rather than what it's doing. (Though the other canonical decorator is "memoise", which of course is saying what it's doing) Interesting thought. > I can remember what "TransformerGenComponent" means no problem while I'm > coding. When I come back to the project in April though, I'm going to > have to lookup the docs for that on what it does and how it does it > before I can even read what my code is doing with it. Yep. That's my biggest concern with this sort of thing. Kamaelia is designed after all with maintainers in mind rather than initial development (It's harder to debug someone else's code than your own IMO). These two aims can support each other, but equally can conflict. In this case, my suspicion is that a decorator based component is a crutch - something that helps someone get started. (Crutch in a good sense rather than bad though) Much like a WSGI systems can st
[kamaelia-list] Re: Axon.ThreadedComponent.threadedcomponent and internal queues
On Monday 12 October 2009 07:56:36 Jim Easterbrook wrote: > True, but (in my application) only when a component is being shut down. > I don't use signal/control for anything else. I'm not suggesting it is > applicable more generally. Indeed. However, it's definitely the sort of effect on user level code of Axon that I really wouldn't want to see if it can be helped :-) > I didn't expect to start such a long thread, let alone trigger a code > rewrite. Well, the bugfix I've merged which makes your threaded examples work as you expect isn't a rewrite. It is however a very good usecase for extending some work which I'd played with, to flesh it out. Whilst you may not have expected this thread, I find it useful for that reason. (I prefer to do work on the core when there's a need. In a way a bit like the difference between "a priori" and "a posteriori" :-) I've got some other things I want to do first before that rewrite, which is why I was looking for a reasonable bugfix, hence that bugfix/patch onto /trunk at the weekend, but that rewrite is something I feel is useful to do. Again, many thanks for raising it! :-) Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Feedback requested: Decorating Generators to make components, perhaps pythonic ?
No positive responses. Probably not worth the effort in to clean up and merge this in that case. /me goes off to focus on something else :-) Michael. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Axon.ThreadedComponent.threadedcomponent and internal queues
On Saturday 10 October 2009 12:43:13 Michael Sparks wrote: > I think whilst the above works, making Axon work the way you expected it to > in the first place is probably more beneficial. > > It's a bug in Axon IMO, thanks for discussing it here :-) Now fixed on /trunk Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Axon.ThreadedComponent.threadedcomponent and internal queues
On Thursday 08 October 2009 14:28:58 Jim Easterbrook wrote:
> On 08/10/2009 13:45, Matt Hammond wrote:
> > Ignoring threaded component behaviour for a moment. Axon delivers
> > messages to the destination component immediately. So if you send a
> > message(s) to the "inbox" inbox of a component, then a message to the
> > "control" inbox of the same component; then if that component notices the
> > message in its "control" inbox, then it is guaranteed that the messages
> > sent, earlier, to its "inbox" inbox will already be there. Ie. (in the
> > current
> > implementation at least) message delivery is guaranteed in-order.
>
> Good. I, for one, would like this to be made part of the "specification"
> so future implementations preserve this aspect.
It's guaranteed that inboxes and outboxes are FIFO queues, and guaranteed to
have in order delivery. You could define a special outbox which didn't
guarantee delivery or order - ala UDP - but that would be more sensible to
implement as a component so someone knew explicitly what they're getting
into.
> > It does cause queues to be flushed. And the documentation (I wrote) is
> > poor in that it does not explicitly state what actually happens (or what
> > is guaranteed from the perspective of a component writer). Yes, its
> > purpose when I coded it was to ensure queues are flushed.
>
> That's reassuring. I'd sort of worked out from the code what it was up
> to, but it's nice to have confirmation from the author.
It's the maintainer you need to worry about. Given I'm working on redoing the
scheduler, there wasn't any guarantee this trick would work in future. I'll
add a self.flushboxes() method to threadedcomponent that (for the moment)
calls sync().
Whilst Matth says the purpose of sync was to flush the queues, the code
doesn't really agree - it looks more like an accident of implementation.
That's not to say that wasn't the intent, but you'd be hard pressed to take
away that interpretation from the code.
cf:
Regulating speed
In addition to being able to pause (with an optional timeout), a threaded
component can also regulate its speed by briefly synchronising with the
rest of the system. Calling the sync() method simply briefly blocks until
the rest of the system can acknowledge.
...
def sync(self):
"""\
Call this from main() to synchronise with the main scheduler's thread.
You may wish to do this to throttle your component's behaviour
This is akin to posix.sched_yield or shoving extra "yield"
statements into a component's generator.
"""
return self._do_threadsafe( lambda:None, [], {} )
Which means that the anonymous function "lambda: None" (called here "anon" for
sake of discussion) is called inside the scheduler.
_do_threadsafe's purpose is essentially described as follows:
For other methods such as link(), unlink() and (in the case of
threadedadaptivecommscomponent) addInbox(), deleteInbox(),
addOutbox() and deleteOutbox(), the _localmain() microprocess
also acts on the thread's behalf.
...
This is implemented by the _do_threadsafe() method.
Crucially, in order for this to flush the queues, it has to work on the
assumption that this can and will cause _localmain to have a full cycle
before returning. This relies on a side effect of the way the scheduler is
implemented today - which if left relied upon makes the system brittle.
> > However, even inserting self.sync() calls doesn't eliminate all race
> > conditions: a fresh message could arrive at the "control" inbox just
> > after sync() returns, and be delivered first (before any other messages
> > pending at the "inbox" inbox) meaning the thread might still miss them.
> > Hmm.
>
> I'd not given much thought to inboxes when I wrote my original, being
> more concerned with the producer's outboxes. However, testing my
> threaded consumer with the unthreaded producer revealed that inboxes are
> also a problem.
That's because as I noted the issue is in the fact that movement of data from
inboxes to inqueues results in data from the control inbox being passed over
by the localmain first.
Thinking about it a flush() method in both threaded/non-threaded components
would solve this, as would changing the message delivery system. The latter
would would be backwards compatible with existing components though, and make
the system work essentially the way you want it to. (ie without making
components more complex and without needing to alter existing logic).
I'll implement the latter.
> Here's my solution (tested) which I think achieves the same thing:
>
> class ConsumerT(Axon.ThreadedComponent.threadedcomponent):
> def main(self):
> count = 0
> while 1:
> if self.dataReady('inbox'):
> msg = self.recv('inbox')
> count += 1
> elif self.dataReady('control'):
> self.sync()
[kamaelia-list] Re: Axon.ThreadedComponent.threadedcomponent and internal queues
On Thursday 08 October 2009 15:23:12 Matt Hammond wrote: > """ > > > Deliveries to a given single inbox will be in-order. But I don't think > > you can guarantee delivery is in order when considering, as a whole, a > > sequence of messages sent to two or more inboxes. > > """ I completely missed that comment. Yes. Incidentally, my plan is as I noted yesterday to do directly delivery - but isn't an immediate solution. (The Axon2 rewrite is starting down that path) Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Axon.ThreadedComponent.threadedcomponent and internal queues
On Thursday 08 October 2009 13:45:08 Matt Hammond wrote:
> Yes, its
> purpose when I coded it was to ensure queues are flushed.
It does not guarantee in order delivery though, when you chain 2 threaded
components together
Specifically, given this sequence of messages being *logically* sent:
ProducerThread
( 1, "outbox")
( 2, "outbox")
( 3, "outbox")
( 4, "outbox")
( 5, "outbox")
( 6, "outbox")
( 7, "outbox")
( 8, "outbox")
( producerFinished, "signal")
Note that I've divided those into 3 chunks of messages.
These get appended to outqueues in that order. Due to the way this loop...
for box in self.outboxes:
... works this ensures that these will get sent to the actual outboxes as
follows:
( 1, "outbox")
( 2, "outbox")
( 3, "outbox")
( 4, "outbox")
( 5, "outbox")
( 6, "outbox")
( 7, "outbox")
( 8, "outbox")
( producerFinished, "signal")
This arrives in the inboxes of the recipient:
( 1, "inbox")
( 2, "inbox")
( 3, "inbox")
( 4, "inbox")
( 5, "inbox")
( 6, "inbox")
( 7, "inbox")
( 8, "inbox")
( producerFinished, "control")
Now we hit the problem point. We hit this loop:
for box in self.inboxes:
This causes this delivery order - control box, then inbox. This comes in fits
and starts - due to timeslice interleaving:
{ no control messages }
( 1, "inbox")
( 2, "inbox")
( 3, "inbox")
{ no control messages }
( 4, "inbox")
( 5, "inbox")
( 6, "inbox")
( producerFinished, "control")
( 7, "inbox")
( 8, "inbox")
The consumer then exits early. This is because order is not guaranteed in this
way by the implementation. It's a rather subtle bug, one only really visible
if you are connecting two threads together.
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamaelia@googlegroups.com
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~--~~~~--~~--~--~---
[kamaelia-list] Re: Axon.ThreadedComponent.threadedcomponent and internal queues
Hi Jim, On Monday 05 October 2009 12:45:47 Jim Easterbrook wrote: > I'm worried about cleanly shutting down pipelines of threaded > components. > [snip] OK, I've read this, copied the generator version into /Sketches/MPS/List/JE.py and run that, and as expected get the same result. ~/Checkouts/kamaelia/trunk/Sketches/MPS/List> ./JE.py 1000 messages received Similarly I've placed the threaded version in JE_T.py. Running that 10 times results in this: ~/Checkouts/kamaelia/trunk/Sketches/MPS/List> for i in `seq 10`; do ./JE_T.py; done 213 messages received 696 messages received 676 messages received 205 messages received 232 messages received 870 messages received 952 messages received 1000 messages received 646 messages received 1000 messages received Which is also the sort of behaviour you describe. Adding the self.sync() call does this: ~/Checkouts/kamaelia/trunk/Sketches/MPS/List> for i in `seq 10`; do ./JE_TS.py; done 1000 messages received 1000 messages received 1000 messages received 1000 messages received 1000 messages received 1000 messages received 1000 messages received 1000 messages received 1000 messages received 1000 messages received The reason the latter works as you expect, but the former doesn't is tricky to explain, and I'm not actually sure if this is a bug we can do something about directly at this stage, or if it's something we have to chalk up as a race hazard. I think it can be chalked up as a race hazard for which we can create a (better) work around. I'll try and summarise what's happening. For safety reasons, threadedcomponent was implemented as simply as possible originally. Specifically, the following happens: * Someone creates a threadedcomponent - C * They call .activate() on it. This does 2 relevant things: - C.main() is activated inside a thread - C._localmain() is activated as a Axon.Microprocess.microprocess as a normal microprocess. Essentially, when you activate a threaded component, 2 pieces of code are activated - the component you're running, and essentially a local postman. * This shadow postman runs in the same thread as the scheduler. * The thread runs inside it's own thread. Communication between this shadow postman in the scheduler's thread and the thread containing the component logic is via Queue.Queue objects - inqueues and outqueues. So when you send messages between 2 threaded components, the following occurs: In the sender: * self.send(something, "outbox") - something is added to a Queue.Queue called self.outqueues[boxname] - The _localmain / local postman is woken up * _localmain/postman: - loops through self.outqueues[boxname] taking any values it finds, and delivers those into self.outboxes[boxname] In the recipient: * _localmain/postman: - loops through self.inboxes[boxname] taking any values it finds, and delivers those into self.inqueues[boxname] Summarising that, the following happens in the following threads: a Add to outqueue [ in producer thread] b move from outqueue to outbox [ in scheduler thread ] c move from inbox to inqueue [ in scheduler thread ] d Collect from inqueue [ in consumer thread ] Now, the interesting step here is "c" * Step c has an interesting issue. The way "c" is implemented is as follows: for box in self.inboxes: # copy data from inboxes[box] to inqueue[box] If we edit this (inside threadedcomponent._localmain) slightly to: for box in self.inboxes: print box # copy data from inboxes[box] to inqueue[box] Then something jumps out: ~/Checkouts/kamaelia/trunk/Sketches/MPS/List> ./JE_T.py control inbox control inbox In essence, this means that in some circumstance data can be pushed into the consumer thread faster via "control" than via "inbox". If the consumer thread empties the inqueue, then it can then collect from control, and cause the thread to exit "early". Now, what self.sync() is doing in this scenario essentially is actually essentially adding in a delay. We can find out how long that is: ~/Checkouts/kamaelia/trunk/Sketches/MPS/List> diff -u JE_TS.py JE_TSt.py --- JE_TS.py2009-10-06 23:54:21.0 +0100 +++ JE_TSt.py 2009-10-07 00:48:02.0 +0100 @@ -4,13 +4,17 @@ from Axon.Ipc import producerFinished from Kamaelia.Chassis.Pipeline import Pipeline import sys +import time class ProducerT(Axon.ThreadedComponent.threadedcomponent): def main(self): for i in range(1000): self.send(i, 'outbox') +t = time.time() self.sync() self.send(producerFinished(), 'signal') +t2 = time.time() +print "D", t2-t class Consu
[kamaelia-list] Re: Feedback requested: Decorating Generators to make components, perhaps pythonic ?
On Oct 5, 1:46 pm, Jason Baker wrote:
> Grammar nazi alert:
>
> "To stop this generator, you need to call it's .stop() method. The
> wrapper could do this"
It's a sketch - rather than a proposal for immediate merge - hence the
request for feedback. Pedantry at this stage does not fall into the
category of helpful feedback. (It makes people, me included, think
twice about posting a sketch something for feedback on the concepts)
> Moving on to subjects with a little bit more substance, suppose I have
> a generator function Foo that I wrap using one of these decorators.
> What exactly will happen if I call Foo?
Like all decorators, Foo then no longer binds to the original version
of Foo. In this case, rather than returning a generator, it returns a
component.
I suspected this:
Pipeline(
follow('/var/log/system.log'),
grep(None, ".*pants.*"),
printer(None)
).run()
would be relatively obvious.
After all, Pipeline links and activates components, not generators,
therefore the decorated generator functions return components that use
the generator functions for their logic, rather than the generator
functions directly.
In this case, the 2 decorators are used to use the logic embodied by
the generator to control a component. This opens up a variety of
possibilities.
> This isn't entirely clear to me by tracing the sourcecode.
This is why I split into two portions, at least when I reposted.
Specifically, one part shows a working illustration of how this would
be used:
http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/example.py
... and I noted that the names "blockingProducer" &
"TransformerGenComponent" are non ideal - but then it is inside /
Sketches...
The other was an initial implementation - which is primarily proof of
concept that this can even be done - which can be found here:
http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/decorators.py
blockingProducer is a fairly traditional/simple decorator. For anyone
who've written lots of decorators, there's a handful of issues with
that bare approach (cf the Michele Simionato's decorator module for
more details), but it's fine as a basic proof of concept.
The decorator TransformerGenComponent is slightly odd in that it
returns a class rather than a function which reuses/configures a
function. This is primarily because that was the more obvious approach
at the time. I'll be refactoring it to match the blocking producer
format.
Out of these two files though, the most important one is not the
implementation, but the idea & API as used in the example file:
http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/example.py
For example, in an earlier thread, I posted this syntax/decorator
illustration:
@GeneratorComponent(Inboxes = ["inbox", "control"],
Outboxes = ["outbox", "signal"])
def Tracer(self):
while 1:
for msg in self.Inbox("inbox"):
print self.tag, str(msg)
self.send(msg, "outbox")
self.pause()
yield 1
However, this looked like one step too far towards voodoo for Steve &
IIRC John agreed. In retrospect, I agree, though it could be a
convenient shortcut.
Furthermore a suggested alternative syntax like this:
# Becomes a subsumed generator by one in the wrapper that handles
control and
# moving yielded values to the outbox. Looks like a standard simple
generator.
@GeneratorPipelineComponent
def Tracer(inbox):
for msg in inbox:
print "> ", str(msg)
yield msg
Has some unresolvable problems without massively complicating
GeneratorPipelineComponent, or without reformulating that example like
this:
@TransformerGenComponent
def Tracer(inbox):
while 1:
for msg in inbox():
print "> ", str(msg)
yield msg
yield
My question was fundamentally this: does the following code look clear
enough to follow /roughly/ what's going on at a glance, or is it too
unclear.
@blockingProducer
def follow(fname):
f = file(fname)
f.seek(0,2) # go to the end
while True:
l = f.readline()
if not l: # no data
time.sleep(.1)
else:
yield l
@TransformerGenComponent
def grep(lines, pattern):
regex = re.compile(pattern)
while 1:
for l in lines():
if regex.search(l):
yield l
yield
@TransformerGenComponent
def printer(lines):
while 1:
for line in lines():
sys.stdout.write(line)
sys.stdout.flush()
yield
Pipeline(
follow('/var/log/system.log'),
grep(None, ".*pants.*"),
printer(None)
).run()
If the above is clear & readable - ie you read this as it follows a
system file, greps for a pattern and prints it, that's good. If it
isn't the implementation is irrelevant. If it is clear, the goalposts
change to making the implementation nicer, maintainable and ideally
unsurprising.
I'
[kamaelia-list] Re: Feedback requested: Decorating Generators to make components, perhaps pythonic ?
On Monday 05 October 2009 16:07:50 Nathan Davis wrote: > 1. You really should use a consistent naming convention -- either call the > decorators blockingProducer and transformGenComponent or BlockingProducer > and TransformGenComponent. This is currently a sketch - basically a proof of concept, written when I had time. In this case concept comes first, followed by an implementation. Point is taken though :-) Suggestions for better names welcome. (As noted before) This is after all also the point of /Sketches - it's not intended to be "perfect" or great - just a place to play with ideas until they become clear whether they're useful or not. It's a place for putting ideas into, in concrete form for discussion. It's far easier to point at code after all :) As you'd expect several things have been renamed on their way out from Sketches, along with implementation. > 2. What are these decorators supposed to do, exactly? When would I use > one over the other? blockingProducer wraps a generator that contains calls to blocking operations, and generates data, and transforms it into a component that sends the yielded values out the default outbox "outbox". Literally, it results in producer component which uses blocking operations. Likewise "TransformGenComponent" wraps a generator that performs some transformation, and returns a component that has that logic. This is actually the more awkward case - because it requires the generator be push and pull. For example, currently I essentially do this: argv = self.argv if argv[0] == None: argv = (self.Inbox,) + argv[1:] Which gets passed to this: @TransformerGenComponent def grep(lines, pattern): regex = re.compile(pattern) while 1: for l in lines(): if regex.search(l): yield l Resulting in lines() meaning self.Inbox() is called. This means this: for l in lines(): Iterates over messages waiting in the component's inbox "inbox". (The yielded values get sent out the outbox "outbox"). If instead of this, we created the generator self.Inbox() outside the generator like this: argv = self.argv if argv[0] == None: argv = (self.Inbox(),) + argv[1:] @TransformerGenComponent def grep(lines, pattern): regex = re.compile(pattern) while 1: for l in lines: # Change if regex.search(l): yield l Then the problem here is that as soon as the inbox was empty, self.Inbox() - ie "lines" would raise StopIteration, and not be reusable. Fundamentally, both wrappers are suggested to be used like this: @wrapper def someFunction(arg1, otherargs): do something, possible iterating over arg1, yielding values The upshot of this being that this transforms the generator function, so that rather than returning a generator, it returns a component, which can then be used like any other component. http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1254693316 may make this more obvious. > Adding some descriptive docstrings would go a long way. It's a proof of concept - there's no way it'd get in the core without doc strings :-) > 3. Why did you choose to return a factory function for a single class for > blockingProducer, but return a dynamically generated class for > TransformGenComponent? Again, same reason. 4 months ago I couldn't see how to do this sensibly in either case. Yesterday it became clear to me that it could be done sensibly, and so wrote each version in the most directly obvious way to me at the time. Obviously there's a schism form an implementation perspective, but from a concept perspective, they're equivalent: * You decorate a generator function containing your logic * The resulting function when called returns a component. * The "input generator" callback gets values from the component's primary inbox. * The values yielded are sent out the component's primary outbox. Does that help? Incidentally, I generally have the viewpoint of "get it working; get it working properly/nicely/correctly; make it faster". The stage this code is at is "get it working". Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Feedback requested: Decorating Generators to make components, perhaps pythonic ?
On Sunday 04 October 2009 19:21:19 Michael Sparks wrote: ... > They're currently highly experimental, and located here: > http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/decorato >rs.py > > so I'm skipping those imports :-) > > Anyway, code: I've shifted a copy of the code here: http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/decorators.py http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/MPS/AxonDecorators/example.py Which may make it clearer. Especially the second of these. Changed subject line in case it helps flag what I'm hoping for from anyone willing to give feedback. Many thanks to anyone willing to reply in advance, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Decorating Generators to make components, perhaps pythonic ?
Hi,
OK, I've revisited this, and come to the conclusion that this is possible. I
hadn't been thinking quite as clearly as I could've done :-)
To cut to the chase, I've written some decorators that when they wrap a
generator, they turn said generator into something we can use as a component.
The sticking point in the past when discussing this was that it created
something like this:
@GeneratorComponent(Inboxes = ["inbox", "control"],
Outboxes = ["outbox", "signal"])
def Tracer(self):
while 1:
for msg in self.Inbox("inbox"):
print self.tag, str(msg)
self.send(msg, "outbox")
self.pause()
yield 1
The biggest, and best objections to this were:
"Michael, I can appreciate the goal of trying to make basic things
simple. Your suggestion however feels really ... magical to me. I
like decorators, really. I use them a lot. But I think decorators
are best used to apply one flavor change to the decorated function /
class without significantly changing the decorated item's signature."
"To my brain, that is easier to process one line at a time than trying
to grok the mind-bending:"
-- Steve,
And this:
"Though as I've said, I am not really fond of his example either. IMO it
is coming at it from the wrong direction. It seems like it is starting
with the class-based component and just abstracting out some bits.
Instead I think it would work better to come at it from the starting
point of a standard python generator containing your logic that you want
to fit into the framework."
-- John,
Both from this thread:
http://groups.google.com/group/kamaelia/browse_thread/thread/e6adc81dff852429
This was partly prompted by this discussion on IRC:
http://www.kamaelia.org/logs/kamaelia2009-06-16_log.html
...
[16:59] < eikenberry> Does Kamaelia have decorators for wrapping generators.
[16:59] < eikenberry> ?
[16:59] < eikenberry> Making them into components?
[17:09] < eikenberry> cool. I agree. IMO the best system would be one that
would look almost just like a system based on generators/coroutines, that you
would just decorate into a more capable system. Something akin to beazley's
coroutine talk... taking the basic blocks then hooking them into a scheduler
and allowing for more easily setting up complex pipelines.
[17:09] < MS-> That's nice, until you realise how the control flow of
beazley's coroutines work.
[17:10] < MS-> Specifically they seem to be limited to pipelines.
[17:10] *** eikenberry nods. They are more an example than a working system. I
just really liked how they built on idiomatic python rather than a
framework.
[17:10] < MS-> The reason is because the last coroutine *pulls* rather than
pushes values.
[17:11] < MS-> Mixing push and pull - which kamaelia does - seems to be
somewhat harder to do that way
[17:12] *** eikenberry nods.
[17:13] < MS-> I've just posted to the list a piece of code that's a syntax
I'm toying with.
[17:14] < MS-> But more ideas as to what you'd like to type would be v useful
It took me a while to realise it, but this CAN be done. But you need to use
python generators in a slightly different way from the way David Beazley
does, but ironically slightly more like a unix program uses a filename.
If I do this:
* tail -f /var/log/messages
* grep hello /var/log/messages
Then tail & grep both open the files provided, and use the contents in a well
defined way.
Conversely, the way David Beazley was using generators was not quite
equivalent. The way he was using them was equivalent to passing an open file
handle.
Specifically, grep would look something like this:
def grep(lines, pattern):
regex = re.compile(pattern)
for l in lines:
if regex.search(l):
yield l
In that scenario, this works given either a list or a generator as an
argument. By changing this slightly to this:
def grep(lines, pattern):
"""To stop this generator, you need to call it's .stop() method. The
wrapper could do this"""
regex = re.compile(pattern)
while 1:
for l in lines(): # foo
if regex.search(l):
yield l
yield
We gain something which is only slightly more complex, but IS something we can
decorate as a component. Specifically, we can pass in "self.Inbox" as the
generator with is activated and read from in line "foo".
Not only this, this is actually closer to the way that grep works when you do
this:
* tail -f /var/log/messages |grep -
Specifically, the grep will indeed read normally until it's source is
exhausted and wait until there's more work to do, or until it's told to stop.
This particular issue was really my sticking point in writing decorators, and
so without further ado, here's a simple decorator/generator based pipeline
written using the new decorators.
They're currently highly experimental, and located here:
http://code.go
[kamaelia-list] Re: Porting Axon Architecture to D
On Tuesday 29 September 2009 18:48:49 erisian wrote:
> I've modified my axon port so that it schedules according to a rather
> well-known message passing kernel policy:
>
> 1. The sender transmits to the receiver via a FIFO buffer.
> 2. The sender only blocks if it tries to send on a full FIFO
> 3. The receiver blocks if it reads from an empty FIFO
> 4. Transmitter and receiver can both check the state of the FIFO
> before attempting a send/receive.
> 5. The receiver can override the full buffer behavior by:
> a) asserting the "ready" boolean property before yielding, which
> will allow the sender to continue sending even on full FIFO
> b) by overriding the "ready" boolean with a "ready" boolean
> function which can take mitigating action to recover from the full
> state. Some examples might be dynamically increasing it's maximum
> FIFO size or correct from a data overrun condition by clearing out the
> FIFO and resetting the pipeline state etc
> c) by overriding the "ready" with a function and setting the size
> to 0 or 1 so as to simulate an actual callback or interrupt function.
> Though I haven't tried this yet, it was interesting that it just seems
> to naturally fall out of the code structure. (One of the reasons I
> ported axon was to avoid having to deal with nasty callback
> structures.)
>
> I might be over-architecting in feature 5, but it looked like a
> natural interrupt-like capability for out-of-band processing.
This sounds like the same decision making process we went through with Axon.
With ours we decided to set a maximum "pipe-width" for the link. That is
whilst we implemented our FIFO using a plain old python list, we recognised
that it was possible that such lists could grow without bounds in some
scenarios, cause a memory leak and go bang.
Since storage is logically owned by the recipient (inboxes are real storage,
outboxes are proxies), this means that a reciever gets to be able to change
that maximum size.
Now, in practice, it was unclear what to set that limit to. Default to 1?
Default to 10? Unlimited? Setting it too low would cause un-necessary
processing. Setting it too high could cause resource explosion.
In practice, we decided to try leaving it to unlimited and see what problems
occurred. We've found that in practice that this has been less of an issue
than expected.
Like yourself though a "Box" does have a callback. This is primarily useful in
allowing the storage of a piece of code to wake a component when something
happens. This isn't a callback that a user usually interacts with, and is
used primarily by the system to optimise CPU usage.
Probably the only other difference though is items 2 & 3. In those scenarios,
we don't block, we simply fail gracefully. As a result our code structure
tends to be more line:
for data in self.Inbox("someinbox")
doSomething(data)
for data in self.Inbox("someotherinbox")
doSomething(data)
if not self.anyReady():
self.pause()
yield 1
This is largely because you don't have the equivalent of a stackless or
greenlet .switch statement in plain cpython. The ability to be woken only
when there's data to process though means this isn't as bad as it may appear
at first glance. (If you're sleeping waiting for data on an inbox and you're
only woken when data is put in it, that's kinda what you're after :-)
> I'm writing the unit tests right now and might do one refactoring to
> clean up single responsibility and encapsulation issues.
Sounds good.
> As soon as I get something cleaned up a bit with unit tests I'll post.
If you need somewhere to post to, the offer of space in /trunk/Sketches in
kamaelia's SVN tree still stands :-)
Best Regards,
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamaelia@googlegroups.com
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~--~~~~--~~--~--~---
[kamaelia-list] Re: Porting Axon Architecture to D
On Wednesday 30 September 2009 20:56:59 erisian wrote: > Now that I have the basic message passing and blocking policy working, > I'm investigating how to integrate my little lib with asynchronous > events in a (relatively) portable manner. Hopefully what follows here is useful. I'm summarising the sorts of things / thinking we followed with kamaelia. > One of the most interesting and portable solutions seems to be > 'libev'. It's a high-performance c-library that allows you to > integrate asynchronous timer, device and other events into your > application. It integrates with the operating system's select/poll > subsystem and allows a lot of customization including the integration > of co-routine subsystems (very similar to axon). > > Part of the trick is figuring out how to intersperse co-routine yields > with async polls so that the whole system only blocks when all of the > co-routines are WAITING, but then wakes up the co-routine scheduler > when an async event puts any co-routine back into run mode. That has > to be balanced with periodic checks with the async status so that > components that require async feeds don't starve. This is similar to one of the original use cases of Axon - which was to build a network server system. As may expect this developed relatively organically at first. Rather than just answer "this is how we do it", I'll explain how we got there, which is perhaps more interesting/useful. I don't know if it's a good fit for you, but hopefully useful. My initial approach was to create what we called a "PrimaryListenerSocket" (PLS) component, which listened on a port, and used to select to check for new connections. It would accept them and create a basic connected socket adapter (CSA). At that time the CSA was much more tightly bound to the protocol than it is today, and the select loop was tied up in the PLS. Naturally at the same time we built a basic component for handling client side aspects of the server. At that point in time, due to limited functionality in python I asked a q on comp.lang.python about how best to go about things: You'll be able to see the outline of that code still in kamaelia today. The next step was seeking to build out a proxy. My aim here was to evolve out the necessary code from actual need, rather than from theory. Effectively like TDD or BDD. The issue that arose from the above very simple design was the fact that running 2 servers, or running clients and servers in the same code would result in multiple concurrent calls to select. (ie select in one, then another, then another) That's clearly a bad design, but was expected. The design of a proxy led to this issue: * How does a TCPClient component know of the availability of something that does select? * How does a TCPServer reuse someone else's select? After all, we didn't want to have components with information hardwired if possible. (Open to the idea, but wanted to avoid it if possible) This led to the realisation that Select itself was a thing that: * Needed to be a component * Needed to be findable in the wider system Like many of the metaphors in kamaelia which are based on the real world, I asked the question "How do I find such a thing?". To her credit, the answer to this came from my _mum_ of all places. (Not because my mum knows how to develop software, but because I was using organisations as a starting metaphor, and those are familiar) I explained the problem, and she simply turned round and said "you'd ask someone wouldn't you?". The sort of person I'd ask at work is called a team assistant, and I figured that this would be useful for tracking what we decided to call services, and would effectively co-ordinate things and it was a friday, this caused the creation of the co-ordinating assistant tracker (aka CAT :-). (To my mind this is an advantage - shifting the problem domain to a more human one made it possible for a non-programmer to help solve what is essentially a concurrency issue, without locking being part of the solution...) This caused the creation of a fundamental concept in kamaelia - a service. A service is a tuple consisting of a (component, inboxname) pair. This allows someone to create a linkage to that and send messages to it. This resulted in refactoring the TCP subsystem being changed to the structure we have today, visually described here: * http://www.kamaelia.org/Docs/NotationForVisualisingAxon Textually though: * TCPServer component. Creates a listening socket. Asks for access to a selector service. If this doesn't exist, one is created along the way, and returned. The TCPServer then sends a message to the Selector saying "wake me when there's something for me to do". After this, when the TCPServer gets a message saying there's activity on the socket, this means there's a connection waiting so it accepts the connection. It creates a CSA and passes the CSA
[kamaelia-list] Re: Porting Axon Architecture to D
On Tuesday 29 September 2009 18:48:49 erisian wrote: > I love replying to my own posts :-) I know the feeling. For what it's worth, it is very welcome that you are taking the time to post this to the list. Like everyone I've got a limited set of time to reply, and not waiting is a good policy :-) I *am* watching with a lot of interest though. I've not used D though, so for me this is as much a learning exercise as it is to try and help sanity check ideas :-) (Currently reading through the mail :-) Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Fwd: Reminder: PyCon talk proposals due in 2 weeks
Hi Gloria, On Thursday 24 September 2009 22:02:31 Gloria wrote: > Hi Michael, > > I am doing something at PyArkansas. I was asked to present an > intermediate level view of existing tools and conventions. PyAR seems > to have a heavy Twisted influence, not sure why, but I wanted to show > alternatives. Do you have a simple Kamaelia example to be used with > 2.6, that makes use of multiprocessing, for interprocess > communication? It could be something as simple as the backplane chat > example. But I want to be able to distribute it on people's laptops. > > if you could give me some non-HTTP base snippets/ideas, I would > appreciate that! I won't have a huge amount of time to focus on > concurrency tools, but I do want to mention Kamaelia, and show > something small and very useful, not tied to PyGame or vorbis (so that > people can install it easily), not HTTP based (so people don't mistake > it for yet another web server). I'll have a think about what I can do to help. The bulletin board example from the tutorial I gave earlier in the year is perhaps interesting - though I admit it doesn't using multiprocessing. I'd appreciate anyone throwing out a bunch of random ideas to bounce off each other and see what comes out of the mix though. If I've googled this right, this is the 14th November? Which gives us a fair amount of time to come up with something neat and useful? I agree that not focussing on HTTP is a good idea for the reason you suggest :) Many thanks for including Kamaelia :-) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: SDL etc.
On Friday 18 September 2009 00:08:49 Edward Hartley wrote: > hi > I'm trying to do a bit of catch up here so could someone clarify if > the whiteboard etc. Is currently working or not and if this is due to > third party libraries for example SDL. The whiteboarding code is working fine at the moment. If anything the code has recently (mid July) been consolidated here: http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Apps/Whiteboard As a seperately packageable & installable thing. I've just noticed that the tar ball that gets generated by DistBuild/BuildRelease.sh wasn't uploaded, so I've just done that: * http://www.kamaelia.org/release/Kamaelia-Whiteboard-1.0.0.tar.gz This reduces the installation steps to: * Install python * Install pygame * Install pyspeex, if you want audio shared between machines * Install the tar ball Kamaelia-Whiteboard-1.0.0.tar.gz The install steps for that tar ball are the obvious: ~> tar zxvf Kamaelia-Whiteboard-1.0.0.tar.gz ~> cd Kamaelia-Whiteboard-1.0.0 ~> sudo python setup.py install (Perhaps worth noting - app specific installers do not currently check what versions of axon/kamaelia are installed. Probably should (patches welcome)) > I'm asking this because I've > been having a lot of success lately with cocos2d in it's objective-c > incarnation. In turn in it's python incarnation it is built on pyglet > which is SFAICS a much lighter weight OpenGL wrapper than SDL. Interesting. Incidentally the opengl components use PyOpenGL and allow the pygame based components to run on Open GL surfaces. (PyOpneGL provides a display manager service in the same way as the pygame one) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Fwd: Reminder: PyCon talk proposals due in 2 weeks
For those that are able to go. I'd like to, but can't. (I looked at the costs, and I just can't justify it) If anyone happens to be in the US and near it, more than willing to help out with any materials etc. Michael. -- Forwarded Message -- Subject: Re: Reminder: PyCon talk proposals due in 2 weeks PyCon proposal deadline is Oct. 1. Ten days! PyCon 2010, Atlanta, GA, USA core conference days Feb. 19-21 http://us.pycon.org/2010/conference/proposals/ Remember, the Call for Proposals is a diversity issue. Giving a talk at PyCon is one of the many ways new people can take places of prominence in the Python community. This is a concrete measure we can take, now. Please do this: 1. Consider proposing a talk yourself. 2. Pass the notice along. 3. Is there anyone you should personally encourage to propose a talk? Rack your brains; give them a nudge; call on some of us to help do the nudging, if you like. Many people - for gender reasons, culture reasons, or whatever - may hesitate to put themselves forward to speak. They may need a push from you to get over misplaced modesty, shyness, an "impostor syndrome" feeling, or just nervousness. Some may (falsely) believe that PyCon will come asking them. (PyCon does specifically invite a handful of speakers each year, chosen from among a few of the conference's favorite many-time veteran speakers - a tiny fraction of those who end up giving talks. Overall, though, PyCon realizes it's not nearly smart enough to find the people who should speak, so we let them come to us.) PyCon always gets a fine crop of talk proposals, but this year we're working hard to spread the call for proposals far and wide, because we want to make sure we don't fall into a rut of using the same speakers year after year. We want to make sure the group of people giving presentations spreads as Python itself spreads. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Kamaelia Diversity Statement
Hi, I'm not sure our community is big enough for such a beast, but I'm always in favour of such things, partly because I have a non-traditional background in some respects. I hope that anyone who's been involved with kamaelia at any point in time has felt that they're welcome, and if they haven't my apologies for that, I can be socially inept at times. Anyway, personally I'm willing to work with anyone who's willing to move the project forward, and after seeing lots of handwringing on the python-diversity mailing list (including some which made me want to be elsewhere for a while), a statement was posted there which I quite liked: > >""" > >The Python community is a diverse, global community - and both it > >and the Python Software foundation welcomes and encourages > >participation by all people. Being a diverse community means we're > >based on mutual respect, tolerance, and encouragement. We > >continually strive to help each other live up to these ideals. > >""" with an addition by someone else > I think "strives to be a diverse, global community" would be more > accurate than "is" one. Anyway, I've drafted this as the kamaelia diversity statement: The Kamaelia community strives to be a diverse community, welcoming and encouraging participation by all people. This community is based on mutual respect, tolerance, and encouragement. We continually strive to help each other live up to and protect these ideals, and invite you to join us. I think that sums up everything that I'd want to say in such a statement. (It is what I'd like to see, even though I recognise I can be inept at times like many of us :-) It's also short & positive without being patronising, and quite frankly the code in the kamaelia *is* from a diverse collection of people, even if not all of them are around at the moment :) Assuming no major objections, I'll figure out somewhere sensible to put this - the website needs a revamp anyway. If someone does have major objections, please discuss it with me off-list - or preferably on #kamaelia on IRC. Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Porting Axon Architecture to D
Hi Erisian,
Let me just say that I think this is awesome, and I'm very glad to hear that
Kamaelia's been helpful to you here.
I've always viewed the current implementation of Kamaelia/Axon to be just
that - an implementation of the ideas, with the idea of remainable tractable
& applicable in other languages, so I think this is really awesome :)
On Wednesday 16 September 2009 00:26:48 erisian wrote:
> Michael and friends...
>
> I've been a fan of Kamaelia for a couple years now and when I needed
> to come up with a componentized, flow-based development environment I
> modeled it after Axon for two reasons:
>
> 1. The excellent level of tutorial documentation
> 2. The straightforward architectural approach used by Axon.
>
> I actually just followed the "write your own axon" tutorial and
> implemented it in D. Of course, implementing an axon-like subsystem
> in a statically typed language was, well, challenging.
I can understand that! There's a deliberately very naive C++ implementation in
the repository here...
http://tinyurl.com/n3gsnz
... so I can understand the challenges involved in doing something more
useful :-) (incidentally this was also written based on following the
mini-axon tutorial :-)
> I now have MicroProcess, MicroScheduler, Inbox, Outbox and Component
> working. It's pretty simple right now, but I've run a basic test that
> creates three components (an even integer producer, an odd integer
> producer and an integer printer consumer), links them up and cycles
> the scheduler through about 1000 yield events.
>
> It works like a charm and it's incredibly fast.
That's really cool to here.
> This was all done using free tools (SciTE editor, llvm ldc d compiler,
> D language and the Tango Foundation Class Library).
I've not used/written any D nor really aware of it's language specific
foibles, but I *am* aware of what LLVM is, so I suspect that this could
actually really fly.
> Yields in D are implemented using the Tango "Fiber" class. I believe
> the assembler created generates setjmp/longjmps.
That would make sense. As you may see from the C++ version I've hacked up
before this essentially uses a class attribute to control syntactic sugar for
a switch statement (builds on duffs device...), which can under the hood be
optimised in a similar way.
> I don't think I could have done it easily in other languages static OO
> languages.
I'd agree with this, having tried it - it's possible though :-)
There is also a Java mini-axon floating out there, which was doable, but my
understanding was that aspects of Java did indeed make it harder.
> One of the reasons I used D was that it compiles all the
> way down to the metal and its support for generic programming using D
> templates are second to none. Some of it's introspection features
> were inspired by Python, Scheme and Lua.
I guess I ought to take a better look at D then :)
> Here's the current simple definition of Component:
>
> class Component : MicroProcess
> {
> Inbox!(SIGNAL) controlBox;
> LinkedList!(IntrFace) inboxes;
> LinkedList!(IntrFace) outboxes;
>
> this(char[] _name) {
> super(_name);
> inboxes = new LinkedList!(IntrFace);
> outboxes = new LinkedList!(IntrFace);
> controlBox = new Inbox!(SIGNAL);
> inboxes.append(controlBox);
> }
>
> // same as in parent
> void run() {
> Fiber.yield();
> }
>
> }
>
> Just thought you might like to know. Hopefully I can get it to alpha
> release quality in the near future.
If you would like to use the kamaelia code repository to work on this, you are
very welcome to do so.
To help you decide if that makes sense for you, the three places there I'd
normally suggested would be:
/trunk/Sketches/{your initials}/
... for development/playing around.
/branches/private_{your initials}_anything
... for feature branches (if appropriate).
and...
/trunk/Code/D/
For the specific implementation. (mirroring the idea behind
/trunk/Code/Python, /trunk/Code/CPP, /trunk/Code/Ruby - even if
the only practical version at present is the python version)
If that's helpful, please let me know. If you use elsewhere, please do let us
know, I'd like to link to your code :)
> If I can make it relatively compatible with Axon, it may be straightforward
> to port Kamaelia components. Eventually I hope to integrate it with GTK.
This sounds really good, and I'm more than willing to do what I can to support
you with this :-)
> If you have any questions, feel free to email me.
>
> have a good one,
You too! Many thanks for getting in touch like this - it's very much
appreciated.
Best Regards,
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To
[kamaelia-list] Re: HTTPClient through Proxy Server?
Hi!,
On Thursday 27 August 2009 15:43:16 erikcw wrote:
> Hi,
>
> I can't seem to figure out how to make the HTTPClient work with a
> proxy server. My existing code uses urllib2 which works well with the
> Proxy -- I'd keep using it in Kamaeila, but if I'm not mistaken it
> will block and break concurrency (right?)
The HTTPClient component doesn't really handle proxies, which is why you're
not able to get that working.
In an Axon.Component.component, grabbing something using urllib2 would indeed
block the system, however, if you use a threaded component it won't.
As a quick example that will play nicely.
class UrlLib2Client(Axon.ThreadedComponent.threadedcomponent):
def main(self):
while not self.dataReady("control"):
for url in self.Inbox("inbox"):
X = urllib2.open(url)
page = X.read()
X.close()
self.send(page, "outbox")
self.pause()
self.send(self.recv("control"), "signal")
(Admitedly I've not tested this, but you get the idea :-)
The component above sits there waiting for URLs on its main inbox "inbox", and
sends complete pages out its outbox "outbox". Similarly it also honours
shutdown messages in that any message sent to its control inbox will cause it
to shutdown and also pass on the shutdown message to the next component.
Ideally HTTPClient would be extended to support proxies, but this is a
pragmatic solution, which will work.
Generally speaking, if you have any operations which you need to handle which
are blocking, and you can't transform them to non-blocking, that's really the
purpose behind the threaded components. It's possible at some point the two
component sorts may be unified and threading changed to a decorator which
delcares the component to be blocking, but kamaelia's dev tends to be driven
by pragmatic needs :)
>From a development perspective regarding threaded components
* Their main() function is just a normal function, not a generator
* self.pause() in threaded components actually contains a micro-sleep
inside it, and can also be used like this:
* self.pause(some_sleep_time)
Which is an interruptable sleep, unlike time.sleep(some_sleep_time)
which is not.
* Global services shouldn't (at present) be accessed directly from inside
one (I can expand on this, but this would be rare inside such a
component anyway - you're more likely to use a backplane)
* You continue to use the normal inbox/outbox API
* You use them in pipelines, graphlines, linkages etc in exactly the same
way as normal components.
* The inbox/outboxes are implemented behind the API differently however
using threadsafe queues.
... ie more or less what you'd want :-)
If you have any more q's please do say. Also, it'd be really interesting to
hear what you're building or playing with :-)
Best Regards,
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamaelia@googlegroups.com
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~--~~~~--~~--~--~---
[kamaelia-list] Pypes meet Axon ?
Hi,
Just saw pypes pass through the PyPI RSS feed, and think it's really
interesting/cool. Congratulations on getting a release out the door :-)
For the reason why I find it interesting, I'll let code do the talking, with
some things I find interesting parallels... :-)
pype.component.py
class Component(object): # stackless.tasklet based
Axon.Component.py
class component(microprocess): # generator based
(we also have thread based)
pypes.component.py
def __init__(self):
self.inputs = {'in' : [None, 'Default input port'] }
self.outputs = {'out': [None, 'Default output port']}
self._parameters = {}
Axon.Component.py
Inboxes = {"inbox" : "Default inbox for bulk data. Used in a pipeline
much like stdin",
"control" : "Secondary inbox often used for signals. The
closest analogy is unix signals"
}
Outboxes = {"outbox": "Default data out outbox, used in a pipeline much
like stdout",
"signal": "The default signal based outbox - kinda like
stderr,but more for sending singal type
signals",
}
def __init__(self, *args, **argd):
"""(subclass always calls this via something like this:
super(component, self).__init__()
self.__dict__.update(argd)
self.inboxes = dict()
self.outboxes = dict()
for boxname in self.Inboxes:
self.inboxes[boxname] = makeInbox(notify=self.unpause)
for boxname in self.Outboxes:
self.outboxes[boxname] = makeOutbox(notify=self.unpause)
pypes.component.py
def recv(self, port):
Axon.Component.py
def recv(self,boxname="inbox"):
pypes.component.py
def send(self, port, data):
Axon.Component.py
def send(self,message, boxname="outbox"):
pypes.component.py
def recvall(self, port):
Axon.Component.py
def Inbox(self, boxname="inbox"):
You'll see further equivalences between Axon.Box.py and pype.py
Clearly we have some similar thinking here, and I must also say that your
visual design programme is very shiny, and similar to our Compose tool, which
is currently bust - so yours is by definition far nicer at the moment :-)
(Though it shares code with our pygame based run time visualiser which you
can see described here: http://www.kamaelia.org/AxonVisualiser )
Anyway some differences:
-> You're using stackless tasklets & multiprocessing
<- We're using standard generators, threads & pprocessing
-> You appear to deny cyclic graphs. (I could be very wrong here though :)
<- Many of our systems contain cycles.
-> You appear to have been aware of J P Morrisons work before starting
<- I wasn't . (starting points for me were occam, unix, hardware, etc)
Interesting to me:
* I've wanted to switch from pprocessing to multiprocessing
* I've been curious about supporting more forms of component,
including stackless tasklets and greenlets for a while :-)
* You also have a very shiny web based system editor, whereas our
compose tool (which uses pygame & tkinter) is currently out of
action :-)
ie figuring out some way of interoperating with Pypes strikes me as a really
nice thing to do :) (The biggest "issue" I can see there is the switch around
of port/box name & data really)
Anyway, I just wanted to say that pypes looks really neat & interesting, and
congratulations on getting a release out! :-)
Good luck & have fun :)
:-)
Michael.
--
http://yeoldeclue.com/blog
http://twitter.com/kamaelian
http://www.kamaelia.org/Home
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamaelia@googlegroups.com
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en
-~--~~~~--~~--~--~---
[kamaelia-list] Re: ThreadedChassis
On Friday 03 July 2009 21:07:13 Michael Sparks wrote: > But the first step is to get a threaded chassis working. > > As noted, an optimisation is to "simply" convert the component from one > type to another, but that has slightly more risk involved, and I think > starting with something constrained like this makes the most sense. I > suspect though that for somethings though actually /converting/ the > component from generator to threaded would make the most sense. > > It'll probably force a small rethink of some areas, but this is probably > the best initial solution to the problem "How do I use all my CPUs" since > it would provide a natural path for migrating from generators to threads, > and on a GIL-less implementation of python (IronPython, Jython, etc), it > would allow the use of multiple CPUs relatively cleanly. For what it's worth, I think this is now a far far less theoretical question than it was just 4 weeks ago. The reason I say that is due to the various reports from people on the list saying that Kamaelia "kinda" works on Jython before then, and the fact that Sylvain's XMPP code now runs on top of Jython on top of Kamaelia. We've been chatting on IRC about that, and then I realised that one way for Sylvain to get better performance out of the system wouldn't be to optimise the code, but to change Kamaelia slightly to implement the threaded chassis, which then allows more performance by just adding CPUs to the box... Which is a really neat/promising thought. At the moment I'm working on reviewing, refactoring, merging or rejecting code in /branches which been there for a while, but would really welcome any thoughts, suggestions, code sketches etc working towards getting that working. Otherwise I know what my next goal is :-) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] One outbox -> Multiple Inboxes ( Re: Simplifying Components )
Hi Nathan,
I'm going to chop this into 2 threads if that's OK, to make it more
manageable. I'll chop the outbox -> inboxes & mascot stuff into here to leave
the syntactic suggestions in a separate thread, so that they're easier to
follow.
On Monday 27 July 2009 05:46:48 Nathan Davis wrote:
> Thanks for the response, Michael. First off, I think it would be helpful
> to explain where I'm coming from. One of my major interests is image
> processing, and I have a modest library of routines to perform various
> tasks (enhancement, segmention, labelling, etc.). (It's publicly available
> at https://launchpad.net/pimp if anyone's interested.) What I would like
> to have is a way to "wire" these routines together.
I see the link :-)
> I currently have a system based on generators that can be used to chain
> processors together into a pipeline. This works surprisingly well, but has
> the significant shortcoming that it can not represent arbitrary graphs --
> it's pretty much restricted to pipelines.
Yep. You can munge pure generators in other ways, but it gets messy when
you do, IMO. (processing flow gets tied up with dataflow, which can lead to
subtle bugs)
> Most of the time, a DAG would probably be sufficient. Some cases may call
> for feedback loops, however.
Indeed.
> It's also worth noting that a single output may be connected to more than
> one input. I bring this up mainly because doing this in Kamaelia seems to
> require the use of Backplanes.
Ah. Yes. The situation where you send a single image down two different
processing things before later combining them. There's lots of technotes at
work based around this sort of thing with that sort of diagram :-)
Incidentally, the reason for one outbox -> one inbox rule in kamaelia is based
around making the system more deterministic.
Backplanes are useful for this sort of thing, but aren't the only mechanism
for handling this. Also I don't think current code is the final example of
this sort of thing either!
Current options for one outbox -> multiple inboxes are found here (in rough
order that they were written) :
1 Kamaelia.Util.Fanout : Fanout
2 Kamaelia.Util.Splitter : Splitter
3 Kamaelia.Util.Splitter : PlugSplitter & Plug
4 Backplane
4 actually uses 3 underneath the hood, and 2 is really a simplified version of
3. There was an older version, which is still in Sketches that looks like
this:
class Splitter(Axon.Component.component):
def __init__(self, outboxes):
print self.__class__.Outboxes
self.Outboxes = dict(self.__class__.Outboxes)
self.Outboxes.update(outboxes)
super(Splitter, self).__init__()
def main(self):
while 1:
yield 1
self.pause()
if self.dataReady("inbox"):
data = self.recv("inbox")
for box in self.Outboxes:
self.send(data, box)
Which is a bit hacky, but works. (in here: http://tinyurl.com/kjl6x3 )
Also more recently I was creating a load balancing based splitter here:
* http://tinyurl.com/ml56tz
(Uses a round robin balancing scheme - very hacky, but what was needed)
Fanout for example is used like this:
Graphline( source = MyDataSource(...),
split = Fanout(["toConsole","toFile"]),
file= SimpleFileWriter(filename="outfile"),
console = ConsoleEchoer(),
linkages = {
("source","outbox") : ("split","inbox"),
("split","toConsole") : ("console","inbox"),
("split","toFile"): ("file","inbox"),
}
).run()
Or like this:
X = Graphline(
newCat = newCat,
rotator = loopingCounter(rotation_speed),
translation = cartesianPingPong(position,screensize[0],
screensize[1],border),
scaler = bouncingFloat(scale_speed),
imaging = continuousIdentity(cat),
shutdown_fanout = Fanout(["rotator","translation","scaler",
"imaging","self_shutdown"]),
linkages = {
("rotator","outbox" ) : ("newCat", "rotator"),
("translation","outbox" ) : ("newCat", "translation"),
("scaler","outbox" ) : ("newCat", "scaler"),
("imaging","outbox" ) : ("newCat", "imaging"),
("newCat", "signal" ): ("shutdown_fanout", "inbox"),
("shutdown_fanout", "rotator") : ("rotator", "control"),
("shutdown_fanout", "translation") : ("translation", "control"),
("shutdown_fanout", "scaler") : ("scaler", "control"),
("shutdown_fanout", "imaging") : ("imaging", "control"),
("shutdown_fanout", "self_shutdown") :
("shutdown_fanout", "control"),
}
).activate()
The problem I think you'll have with this is the complexity of the graphline.
Uses of the splitter or plug splitter mentioned above was reall
[kamaelia-list] What's left to refactor/fix/review/merge (maybe) from private_JMB_MPS_Wsgi_Detangled
Hi, Since it may help, this is what's left to review, fix, refactor etc before possibly merging the rest of that branch I'm going through: Index: Apps/Kamaelia-Publish/Gateway/scripts/__init__.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/scripts/http.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/scripts/interface.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/scripts/BoxManager.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/scripts/main.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/scripts/jabber.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/make-unix.sh# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/prepare.sh# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/data/kp.ini# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/data/kpuser/urls.ini# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/data/kpuser/kp.log# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/util/kpgate.db# KAMPUBLISH Index: Apps/Kamaelia-Publish/Gateway/zipheader.unix# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/scripts/transactions.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/scripts/__init__.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/scripts/main.py# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/make-unix.sh# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/prepare.sh# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/data/kp.ini# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/data/kpuser/urls.ini# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/data/kpuser/kp.log# KAMPUBLISH Index: Apps/Kamaelia-Publish/Peer/zipheader.unix# KAMPUBLISH Index: Kamaelia/Kamaelia/IPC.py # KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Gateway/consts.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Gateway/translator.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Gateway/__init__.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Gateway/JIDLookup.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Gateway/UserDatabase.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/__init__.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Peer/translator.py# KAMPUBLISH Index: Kamaelia/Kamaelia/Apps/Publish/Peer/__init__.py# KAMPUBLISH Index: Kamaelia/setup.py # KAMPUBLISH Index: Axon/Axon/Ipc.py # KAMPUBLISH Index: urls # KAMPUBLISH This should be interesting because this needs to work with Sylvain's XMPP code. For those unaware this work is actually a very late merging exercise of Jason's summer of code work last summer. Aside from anything else, what this does is allow people to host webservers behind NAT'ing firewalls and serve it to friends via a gateway. The HTTP requests are encoded & encapsulated inside XMPP messages. ie a user starts up their webserver, and it connects to an XMPP service, and it registers with the gateway service. The gateway service recieves HTTP requests for the user's website,which are then encoded suitable for sending inside an XMPP message to the webserver running on the user's machine. The user's webserver decapsulates this, does the expected work, and then takes the result web response and encodes that suitable for sending over XMPP back to the web gateway. The web gateway decapsulates the web page and sends that back to the user. Since that's a lot to test in one go, that's obviously the reason why I'm taking this bite-size approach to carving up the branch for testing, review, refactoring, repackaging before merge. Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Kamaelia-0.9.7.0-rc1.tar.gz uploaded to /release
Hi, I've uploaded rc1 of Kamaelia-0.9.7.0 - ie this months monthly snapshot to: * http://www.kamaelia.org/release/MonthlyReleases/ Initial list of delta's from 0.9.6.0 - files changed or added: Index: Apps/GSOC_JMB/App/.keep Index: Apps/GSOC_JMB/App/simple_wsgi_example.py Index: Apps/GSOC_JMB/App/PersonalWsgiServer/plugins/WsgiApps/__init__.py Index: Apps/GSOC_JMB/App/PersonalWsgiServer/plugins/WsgiApps/django_app.py Index: Apps/GSOC_JMB/App/PersonalWsgiServer/make-unix.sh Index: Apps/GSOC_JMB/App/PersonalWsgiServer/prepare.sh Index: Apps/GSOC_JMB/App/PersonalWsgiServer/scripts/__init__.py Index: Apps/GSOC_JMB/App/PersonalWsgiServer/scripts/main.py Index: Apps/GSOC_JMB/App/PersonalWsgiServer/data/kp.ini Index: Apps/GSOC_JMB/App/PersonalWsgiServer/data/kpuser/urls.ini Index: Apps/GSOC_JMB/App/PersonalWsgiServer/data/kpuser/www/index.html Index: Apps/GSOC_JMB/App/PersonalWsgiServer/data/kpuser/kp.log Index: Apps/GSOC_JMB/App/PersonalWsgiServer/zipheader.unix Index: Apps/GSOC_JMB/App/wsgi_example.py Index: Apps/GSOC_JMB/DistBuild/BuildRelease.sh Index: Apps/GSOC_JMB/DistBuild/CleanBuild.sh Index: Apps/GSOC_JMB/DistBuild/setup.py.src Index: Apps/GSOC_JMB/DistBuild/upload.sh Index: Apps/GSOC_JMB/Config/.keep Index: Apps/GSOC_JMB/Doc/README Index: Apps/GSOC_JMB/MANIFEST.in Index: Apps/Europython09/App/BB14.py Index: Apps/Europython09/App/cat.png Index: Apps/Europython09/App/BB2.py Index: Apps/Europython09/App/Video-2.py Index: Apps/Europython09/App/Video-4.py Index: Apps/Europython09/App/BB4.py Index: Apps/Europython09/App/SimpleDiracPlayer.py Index: Apps/Europython09/App/BB6.py Index: Apps/Europython09/App/users.passwd Index: Apps/Europython09/App/.keep Index: Apps/Europython09/App/BB8.py Index: Apps/Europython09/App/MiniAxon-0.py Index: Apps/Europython09/App/messages/1 Index: Apps/Europython09/App/messages/2 Index: Apps/Europython09/App/messages/3 Index: Apps/Europython09/App/messages/4 Index: Apps/Europython09/App/messages/.meta Index: Apps/Europython09/App/prompt.py Index: Apps/Europython09/App/MiniAxon-2.py Index: Apps/Europython09/App/simplegame-2.py Index: Apps/Europython09/App/MiniAxon-4.py Index: Apps/Europython09/App/mouse.png Index: Apps/Europython09/App/BB11.py Index: Apps/Europython09/App/BB13.py Index: Apps/Europython09/App/BB15.py Index: Apps/Europython09/App/Video-1.py Index: Apps/Europython09/App/BB1.py Index: Apps/Europython09/App/Video-3.py Index: Apps/Europython09/App/BB3.py Index: Apps/Europython09/App/Video-5.py Index: Apps/Europython09/App/BB5.py Index: Apps/Europython09/App/BB7.py Index: Apps/Europython09/App/BB9.py Index: Apps/Europython09/App/MiniAxon-1.py Index: Apps/Europython09/App/Ulysses Index: Apps/Europython09/App/simplegame-1.py Index: Apps/Europython09/App/MiniAxon-3.py Index: Apps/Europython09/App/BB10.py Index: Apps/Europython09/App/MiniAxon-5.py Index: Apps/Europython09/App/BB12.py Index: Apps/Europython09/DistBuild/.keep Index: Apps/Europython09/DistBuild/BuildRelease.sh Index: Apps/Europython09/DistBuild/CleanBuild.sh Index: Apps/Europython09/DistBuild/setup.py.src Index: Apps/Europython09/DistBuild/upload.sh Index: Apps/Europython09/Config/.keep Index: Apps/Europython09/Config/README Index: Apps/Europython09/Doc/.keep Index: Apps/Europython09/Doc/README Index: Apps/Europython09/Extras/SimpleDiracPlayer.py Index: Apps/Europython09/Extras/users.passwd Index: Apps/Europython09/Extras/WebcamBasedDiracVideoRecorder.py Index: Apps/Europython09/Extras/TranscodeAndUploadToBlipTV.py Index: Apps/Europython09/Extras/messages/1 Index: Apps/Europython09/Extras/messages/2 Index: Apps/Europython09/Extras/messages/3 Index: Apps/Europython09/Extras/messages/4 Index: Apps/Europython09/Extras/messages/.meta Index: Apps/Europython09/Extras/BulletinBoardExample.py Index: Apps/Europython09/MANIFEST.in Index: Apps/GSOCPaint/App/KamPaint.py Index: Apps/GSOCPaint/App/icons/MenuFileSaveIcon.png Index: Apps/GSOCPaint/App/icons/MinusButtonIcon.png Index: Apps/GSOCPaint/App/icons/RedGreenIcon.png Index: Apps/GSOCPaint/App/icons/EllipseToolIcon.png Index: Apps/GSOCPaint/App/icons/MenuLayersMoveLayerDownIcon.png Index: Apps/GSOCPaint/App/icons/LineCapButton.ArrowFilled.Start.png Index: Apps/GSOCPaint/App/icons/GreenBlueIcon.png Index: Apps/GSOCPaint/App/icons/PaintBucketIcon.png Index: Apps/GSOCPaint/App/icons/ColorPickerToolIcon.png Index: Apps/GSOCPaint/App/icons/ToolConfigStrip.SelectionCombineMode.Replace.png Index: Apps/GSOCPaint/App/icons/LineCapButton.ArrowFilled.End.png Index: Apps/GSOCPaint/App/icons/PaintBrushToolIcon.png Index: Apps/GSOCPaint/App/icons/MenuLayersMoveLayerUpIcon.png Index: Apps/GSOCPaint/App/icons/PlusButtonIcon.png Index: Apps/GSOCPaint/App/icons/EraserToolIcon.png Index: Apps/GSOCPaint/App/icons/RedBlueIcon.png Index: Apps/GSOCPaint/DistBuild/Build.Paint.sh Index: Apps/GSOCPaint/DistBuild/CleanBuild.sh Index: Apps/GSOCPaint/DistBuild/setup.py.src Index: Apps/GSOCPaint/DistBuild/upload.sh Index: Apps/GSOCPaint/Doc/Overview_and_todolist Index: Apps/GSOCPaint/Doc/REA
[kamaelia-list] Merge of branch: private_JMB_MPS_WebServer - A Personal WSGI Server application
Hi, Merging more of Jason's project. Again a fair amount of restructuring and repackaging has happened here. I'm also rapidly coming to the conclusion that packaging applications in the first instance as Kamaelia.Apps rather than targeting the main namespace is a good idea, largely because it simplifies the bar to merge due to the barrier being "does this work for this application" rather than "does this work for multiple apps, and can we have more than one of these running at once?" What is it? It's fundamentally a Personal WSGI Server. ie something the average user can run if it's installed on their system, change some config files in their home directory and serve content in an adhoc way. However that content can also be arbitrary (within some limits) WSGI applications as well. If you build the customised distribution, and then install from that, you'll find that inside /usr/local/bin you have an extra thing to run called personal_wsgi_server. It's this that the user can run. The first time you run it, it prompts you with a question regarding whether you want to copy/install the default configuration. If you answer "y", and return this is what you see: ~> personal_wsgi_server It does not appear that Kamaelia WebServe has been installed. Would you like to do so now? [y/n]y INFO/kamaelia.web_common.autoinstall: Kamaelia WebServe is now done installing. INFO/kamaelia.WebServe.main: Serving on port 8080 This creates a kp.ini file which configures the server. It also creates a kpuser directory which contains stuff for the server at runtime, for example ~/kpuser/www/index.html On http://127.0.0.1:8080/ you get the output from a WSGI variable test. http://127.0.0.1:8080/static/ is the doc root for ~/kpuser/www/ , and displays the content of index.html ( as you might expect) Status == Merged onto trunk, but is not "finished" IMO yet. This ought to move its configuration file into ~/.kamaelia/jmb/pws.ini, and change kpuser to something more like PWS/ or MyPWSContent/ or something. Other than that there's been a bunch of bug fixes to get this working, as well as namespace changes to make it "fit" better with Kamaelia's directory structure. There's also been some bugfix work around bugs in specific version of python (specifically a bug in 2.5.1 regarding a conflict of zipfile & tarfiles). I'll note though that this application does fulfill Jason's original goal of having a simple to install server which a user can configure, since if this is "built" for the right OS, then actually just copying the personal_wsgi_server file to a machine is sufficient for the application to run. I don't think I like everything under the hood of this[1], but the result is pretty neat. It's potentially a nice way of locally testing & serving wsgi apps. [1] I don't think any two developers would ever agree on style It would be particularly interesting to be able to provide a hosting environment for running Google App Engine apps locally using this, since it appears to provide a good base for that. I don't know how practical that is though. Playing with this = In order to play with this, download the specialised distribution from here: http://www.kamaelia.org/release/Kamaelia-GSOC-JMB-0.1.0.tar.gz Install: # tar zxvf Kamaelia-GSOC-JMB-0.1.0.tar.gz # cd Kamaelia-GSOC-JMB-0.1.0/ # sudo python setup.py install Then run the server: # personal_wsgi_server It does not appear that Kamaelia WebServe has been installed. Would you like to do so now? [y/n] y INFO/kamaelia.web_common.autoinstall: Kamaelia WebServe is now done installing. INFO/kamaelia.WebServe.main: Serving on port 8080 You can then point your browser at http://127.0.0.1:8080/ http://127.0.0.1:8080/static/index.html And change configuration in ~/kp.ini and inside ~/kpuser/ As I say, these directory names probably ought to change to play nicer with the system. Details of Merge Index: Apps/GSOC_JMB/DistBuild/ * All the common packaging stuff. Index: Apps/GSOC_JMB/MANIFEST.in * Changed to pull in all the files inside App, including the personal wsgi server Index: Kamaelia/setup.py * Changed to pull in JMB's common files for his web server code. Index: Kamaelia/Kamaelia/Apps/JMB/Common/ * Collection of common core python files used/core to JMB's web server apps. Index: Apps/GSOC_JMB/App/PersonalWsgiServer/plugins/WsgiApps/django_app.py * Not clear on the purpose of this. Docs say it doesn't actually work. If that's the case, why have it here? Index: Apps/GSOC_JMB/App/PersonalWsgiServer/scripts * Directory containing the actual application Index: Apps/GSOC_JMB/App/PersonalWsgiServer/scripts/main.py * The actual main.py file for the application. Gets embedded elsewhere but this is the main code core. Index: Apps/GSOC_JMB/App/PersonalWsgiServer/make-unix.sh * Makes a stand alone unix "executable" of the personal wsgi server
[kamaelia-list] Re: Merge of Branch: private_JMB_MPS_CoreChanges_2
Hi Nathan, On Saturday 25 July 2009 22:06:29 Nathan Davis wrote: > FYI, if you haven't already, you may want to look into using SVK to handle > your branches. SVK is a distributed VCS that works well Subversion > repositories (it uses Subversion's filesystem component). Its main > advantage is that you can merge two branches by just telling it what > branches to merge ... it figures out the exact revisions to merge > automatically. I'm not certain, but I believe it can even handle cases > where some changes have already been merged. I'm aware of SVK, but hadn't considered it in this case. I think however, that in this case it wouldn't be quite what I needed. Rather than containing one set of changes really, the branch I'm merging really contains about 5 or 6 things which can be distinctly mergeable. The fact that it also contains code from another branch already merged adds an extra layer on top complicating things. In practice what I'm needing to do is to take that composite branch and split it into discrete reviewable chunks, such that they can be cleanly merged. Part of the issue there is that there's changes to code outside Kamaelia/Apps which is very application specific, which is problematic. Basically you can take this as meaning that the branch was proposed as being ready for merge when in fact it wasn't in it's current form. The way I'm essentially refactoring each discrete chunk before merge is to recognise that the code I'm merging IS good though in it's application domain, and shifting code into it's own Apps namespace relevant to the application. That then allows it to be mergeable. If the branch had been mergeable as is, despite the mixed branch issue, then SVK certainly would've been useful, but unfortunately it wasn't really quite that simple. All that said, using SVK to create the detangled version may have been a good option, one that I'll bear in mind in future :-) > It integrates really well with Subversion (you basically just point svk to > you subversion repository and any svk updates also update Subversion). I > highly recommend it to anyone who needs to do a significant amount of > merging with SVN. > Note that the smerge command is probably what you want for merging. I'll take a look - anything that helps with merging is IMO a good thing. > Also, I think the Repository page (http://www.kamaelia.org/Repository) has > a mistake in it: svn co http://kamaelia.googlecode.com/svn kamaelia-trunk > should be > svn co http://kamaelia.googlecode.com/svn/trunk kamaelia-trunk Good catch! I'll fix that. > Let me know if you want some help getting started with SVK. I'm not an > expert, but should be able to point you in the right direction. Much appreciated - I may well take you up on that. :) > Hope this is helpful, It is indeed - thanks ! Regards, Michael -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Merge of branch private_JMB_MPS_WSGI
Hi, I've taken the WSGI specific changes from Jason's private_JMB_MPS_Wsgi_Detangled branch, and copied them into this branch - private_JMB_MPS_WSGI. That branch has been restructured somewhat and merged. After reviewing them, there's a lot of code in there which is actually app specific - for example custom routing which most people using WSGI wouldn't use; wsgi values called kp.* etc, - so I've decided to shift out all this code into two places. Specifically a Kamaelia.Apps.JMB namespace to hold all the core code relatively unchanged - but so that it will play nicely with the rest of the system. Also, Apps/GSOC_JMB has been created to package up Jason's code cleanly, along with applications using the app specific wsgi routing included. Detailed description of what's been merged. Apps/GSOC_JMB/App/ * Holds the 2 examples of how to use JMB's WSGI code. This contains some app specific things, and hence is seperated out as an app. The generic code will get merged onto core after refactoring/simplification. * Examples: Apps/GSOC_JMB/App/simple_wsgi_example.py * Hosts a simple wsgi application. Apps/GSOC_JMB/App/wsgi_example.py * Hosts 2 simple wsgi applications based on an "url list". Apps/GSOC_JMB/DistBuild/ Distribution/release build scripts: * BuildRelease.sh * CleanBuild.sh * setup.py.src * upload.sh Apps/GSOC_JMB/Config/ Config directory for future use. Apps/GSOC_JMB/Doc Docs directory. Docs need writing. Apps/GSOC_JMB/MANIFEST.in * Standard packaging file. Kamaelia/Kamaelia/Apps/JMB/__init__.py Base directory for JMB's project(s) Kamaelia/Kamaelia/Apps/JMB/WSGI/__init__.py Base directory for JMB's WSGI related code. Kamaelia/Kamaelia/Apps/JMB/WSGI/Apps/ Base directory of App specific wsgi adaptors to work with JMB's wsgi server. Current example Apps/adaptors: * Simple.py * CpyApp.py * PasteApp.py * Test.py * Moin.py * ErrorHandler.py * Static.py Kamaelia/Kamaelia/Apps/JMB/WSGI/_WSGIHandler.py File containing the *actual* WSGI handling code (NB: MPS is Baffled as to why this isn't exposed as a regular component, and it's due to the multiple levels of indirection in the code that this is in the apps directory, as well as the app specific ( kp.* ) wsgi application values.) Kamaelia/Kamaelia/Apps/JMB/WSGI/_factory.py Factory system for spawning WSGI applications based on specific routing. Problem is that is in itself really a function of wsgi, and there's much better things out there for doing this, so this is really an app specific router, rather than general router. It's interesting, but I can see some serious issues with it myself. Kamaelia/setup.py Changed to pull in the Apps.JMB namespaces as well. Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Merge of Branch: private_JMB_MPS_CoreChanges_2
Hi, I'm slowly extracting and cleaning up code changes from Jason's work last summer. I'm doing this by extracting related changes out into seperate branches after having created a detangled/cleaned up version of Jason's branch. (For some reason Jason's branch contained significant amounts of code already merged onto trunk from private_MPS_Scratch from some time ago, seriously complicating the merge) Anyway, private_JMB_MPS_CoreChanges_2 has been brought down to *just* the HTTP Server specific changes which were written by Jason (the rest appear to be from my old scratch branch, and already merged). There's a fair amount of work (fixmes) added to this branch, but overall the code change is a net improvment, so I'm merging the branch. Details: Branch name: private_JMB_MPS_CoreChanges_2 Branch version: r6326 Detailed changelog/notes for future work: Index: Kamaelia/Kamaelia/Support/Protocol/HTTP.py Collection of utilities to simplify working with the HTTP components Adds in a concept of "request translators". Request translators take the raw output from the HTTP Subsystem that you're expected to process, and convert it to a form suitable for other forms of processors. For example one request translator takes the HTTP request and converts it to a WSGI environ, suitable for passing to a WSGI app. Functions added: * ReqTranslatorFactory * WSGILikeTranslator * ConvertHeaders * PopURI * PopWsgiURI * PopKamaeliaURI * checkSlashes Prefabs (maybe) added: * HTTPProtocol * requestHandlers Adds in "pop" functions which appear very badly named. FIXES Needed: lots of Rest/doc fixes, better explanations/doc strings. Potentially need to move the components/prefabs out to the main namespace. "Pop" concept needs reworking. Argument names to functions could do with being changed to something human readable. HTTPProtocol needs renaming potentially. HTTPServer/getHttpServer name is confusing since it actually returns a protocol handler. requestHandlers almost certainly in the wrong place. _Purpose_ of checkSlashes is unclear. Index: Kamaelia/Kamaelia/Protocol/HTTP/ErrorPages.py * Documentation extended slightly * Added in 502 error handling * Changed ErrorPageHandler to be a OneShot handler Index: Kamaelia/Kamaelia/Protocol/HTTP/Handlers/Minimal.py * Changed in a variety of ways, unclear if overall better. Possibly is, but a number of further improvements can be made as noted in fixmes. * FIXMES: Factory function looks nasty, redundant and badly named. However it is used by other code, so will be left for a short while before being eliminated. Should use inheritable defaults instead. Pathname handling of files looks like it needs work. Index: Kamaelia/Kamaelia/Protocol/HTTP/HTTPParser.py * Adds in non-query-uri and query into the http request object. This looks & feels wrong, and needs review. Left in for the moment pragmatically because other parts of the system rely on this being there. Index: Kamaelia/Kamaelia/Protocol/HTTP/HTTPServer.py * Example added at the end of the file, which has been fixed by MPS to actually work. Index: Kamaelia/Examples/TCP_Systems/HTTP/KamaeliaWebServer.py * Error handling fixed, now uses OneShot and a custom createRequestHandler. (Should probably not use a custom one, but it does...) Index: Kamaelia/Examples/TCP_Systems/HTTP/cookbook_1.py * Changed to use new(er) facilities added into the HTTP subsystem. Index: Kamaelia/Examples/TCP_Systems/HTTP/cookbook_2.py * Changed to use new(er) facilities added into the HTTP subsystem. * Also, reverted non-discussed changes which are inappropriate changes IMO. * Adds doc strings to a bunch of stuff too. (For those curious, this needs this level of review before merge because the changes are in core code, rather than in application specific code, and thereby needed a wider view being taken before being proposed as "ready for merge". Based on the amount of work this has taken, the branch was not ready for merge) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Simplifying Components
On Wednesday 22 July 2009 20:55:20 Nathan Davis wrote: > I just started using Kamaelia a little. First off, let me just say that I > like the idea of wiring components together very much. That model works > very well for certain types of applications, and the closer your code gets > to that model the easier it is to use. That's very nice to hear :-) > So far, Kamaelia is the closest to my ideal for this domain. Cool :-) Let's see if we can get it closer. > That being said, I think there is a lot of room for improvement. There wouldn't be any checkins ever again if it was perfect ;-) FWIW, I also think that there's a long way to go, but based on experience gained with the current system now, I feel it's a solid base to build on. (which kinda surprises me :-) > I'm sure that's a big shocker to everyone, given Kamaelia's relatively young > age ;-). :-) > In particular, I find coding and wiring components to be cumbersome. There's a reason behind this. Again, it's not shocking reasons, but there are reasons. The key two however are these: "Explicit is better than implicit" & "In the face of ambiguity, refuse the temptation to guess." In order to explore the most sensible route for making coding and wiring components less cumbersome, it's useful to cover some background on Kamaelia and those reasons/design forces. Let me be clear on one thing first though - I very much welcome this email as an opening point for a discussion. I think it's a nice starting point, and I view this sort of discussion as pretty vital for the future of a project. After all unless people like yourself find the project *usable* and *nice* to work with, it will be an evolutionary dead end. (same goes for other types of people too of course :) Beyond that, I also tend to be pragmatic. I don't believe that anyone can see the future, beyond the obvious approach of trying to build it, and then seeing what works. For example, in Kamaelia things that follow that model is the co-ordinating assistant tracker is nice, but the STM code is nicer, and should replace the fomer. Implementations that have been played with are the most compelling arguments that can be made. Projects need to evolve or they become irrelevant. Given that, it's useful to outline the reasons/design forces. * Originally it was unclear if the model would be a good idea or not. * It was unclear if there was a need for inboxes & outboxes, or if "boxes" alone would suffice. It turns out that from an implementation perspective "boxes" is sufficient, but that the abstractions of inboxes & outboxes helps, a lot. (directly helps with late wiring, protecting youself from potentially shooting yourself in the foot, unless you really want to (passthrough linkages) etc) * It was unclear whether python would be a sensible choice in the short/medium term. * It is unclear if python (espeically cpython) is a sensible choice in the long term. (10 year time frame) Now those are all forces on the project, but not reasons for existence of the project. Also, since you've started using, I'm assuming you've noticed the 3 way split of the project: 1 Axon, core tools 2 Kamaelia, big old bucket of components for building new apps 3 Kamaelia based apps, a growing collection of slowly becoming more formalised applications. Now, if the focus of the project had always been 1,2,3 and the focus had been, "I want to build 1,2,3 that is the perfect 1,2,3 for python", then the project would look very different. However, the focus has always been 3,2,1, with the original application being developed (or question asked) was "How can we build a system that allows us to organically create and play with highly scalable network systems, in the least constraining manner, which results in easily maintainable systems". The other less obvious consequence of this is that this means the system needs to be accessible to the vast majority of developers, crucially including those starting their careers. The obvious model to build on is the sort of model legions of sys admins round the world are happy with, hence inbox/outbox (stdin/out), component (vs process), the tracker (unix environment variables), and to try and anthropomorphise the model since any model you can explain with little robots or act out with people is easier (IMO) to pick up. Pragmatically, you have to pick a language, but beyond that we didn't want to be tied forever to that language, and it needed to support multiple modes of concurrency. (This is similar to not wanting to be tied to any particular operating system just because a given concurrency model on that OS is good or cheap. Threads have very different scaling properties on different OS's and different OS versions after all). This means that when a decision has been taken on a particular kind of syntactic sugar, careful thought has been taken "can this model be implemented in ano
[kamaelia-list] Re: Europython Kamaelia Tutorial Notes, slides.
On Wednesday 15 July 2009 17:23:57 chumpalump wrote: > For the last several days I've been reading, eating, and sleeping with > "Kamaelia: Pragmatic Concurrency". These 82 pages, along with a lot > of code reading, have been amazingly useful. Seeing my rabbitmq > messages delivered to a backplane was quite a beautiful moment. > > Nothing is so nice as great documentation paired with great code! > > Thank you very much, > Dave Thank you for your kind words & you're welcome :-) It's also very nice to hear of people picking up the tools we created and hooking it up to something new and interesting :-) Best Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Bidirectional Json-Rpc Server
On Wednesday 15 July 2009 03:04:20 Rasjid Wilcox wrote: > Due to other work commitments it will be a couple of weeks before I > get to do much with this. No problem :-) Michael -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Re: Bidirectional Json-Rpc Server
On Saturday 11 July 2009 19:48:58 Michael Sparks wrote: > Excellent, what I'll do then is merge this branch onto trunk, and I suggest > that you work on your changes on a branch Just as a note, I've done this. In case you've not used svn to create branches before, the way to do this is as follows. Assuming you have a full checkout of the codebase via: svn co https://kamaelia.googlecode.com/svn kamaelia You create a branch as follows: cd kamaelia/branches svn cp ../trunk/Code/Python private_RW_ svn ci private_RW_ The last line is important, since at a later point in time it allows you to do this: svn log --stop-on-copy To find out all the changes. Or more specifically: svn log --stop-on-copy|egrep '^r[0-9]+'|tail -1 eg: ~/code.google/kamaelia/branches/private_MPS_RW_JsonRpc> svn log --stop-on-copy|egrep '^r[0-9]+'|tail -1 r6204 | sparks.m | 2009-07-07 22:52:07 +0100 (Tue, 07 Jul 2009) | 1 line To give you the first revision for the branch. (ie the svn ci above). In this example "r6204" This then allows someone reviewing the branch to see all the changes on that branch simply as follows: cd svn diff -:HEAD . eg: cd private_MPS_RW_JsonRpc svn diff -r6204:HEAD . This also then allows someone reviewing your code (or you if there's general agreement) to merge the changes on /trunk as follows: cd trunk/Code/Python svn update svn merge -:HEAD ../../../branches/ . Incidentally, this also means that renaming a branch is a pain in SVN, since it breaks this work flow. The other final request I'd make as a result of this, is that checkin messages mention why a change was made. However, once again, please note that the namespaces I mentioned I view as "owned" by yourself, because you're the one building & using the application :-) It's just that doing these things makes it easier for people to work with you. It does also mean you can have a working branch for your code at any point in time and not be blocked whilst waiting for a merge onto /trunk. If you already know all the above, cool, if you have any q's fire away :) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Distributed data processing
On Saturday 11 July 2009 01:15:08 manimal45 wrote: > I read about Kamaelia investigation to be used for data modeling. > I'm working on data migration projects where we have to handle tons of > data, with performance issues as each single rows and fields for an > entire database have to be read and transformed. > i believe it's somehow the same issue one can face when dealing with > massive medical data for research subjects. Quite probably. Kamaelia should be well suited to this sort of task. Kamaelia is at it's core a data flow system really, and many of the earliest data processing systems were inherently data flow. The best way to find out of course though is to try it, so I find it interesting that you're trying it! :) > I already posted a couple of questions some months ago when I first > discovered Kamaelia (which i find to be so great, and I've converted > some java friends to python and kamaelia). Cool - that's really nice to hear. It's always nice to hear from anyone using Kamaelia. Incidentally, since you posted to the list last time, I've created and given a tutorial on kamaelia, which you can find here: * http://www.kamaelia.org/PragmaticConcurrency It includes a number of step by step examples aimed at understanding the core, building components, and larger systems. The PDF is linked near the bottom of that page. > I came up with the idea to use Kamaelia to distribute data and queries > across some nodes. That explains the questions :-) > Summer student worked with me and we've managed to have something > running with sqlite nodes. I'd be very interested to see this, if you're able to share. If you can't, no problem :-) > We have some naive concepts : > 1) data are distributed physically across nodes, without any key/range > partitioning > 2) user send queries to proxies > 3) proxies redirects queries to nodes > 4) nodes requests missing data across the system (missing data can > arise when joining tables together, and we've defined some tags to > define parent and child tables), this is the most complex part of the > system: > i) bloom filters are computed on column sets of a join > ii) bloom are sent to all other nodes > iii) matching data are computed on each node and sent back to > requesting node > iv) bloom are stored so that bloom on a column set are never > computed twice This sounds like an optimised form of map-reduce. I like the use of bloom filters in this scenario, since if I understand what you're doing correctly, you do not need to store a central index and do not have an indexing algorithm based on the number of nodes. (I've come across them before) That would suggest that you can just add more nodes with data to your system without any reindexing, etc. > We do not use Json to send message across network, rather we just > cpicke python native dictionaries. > I don't know if it's the best idea I had, but it's rather simple. I'm a great believer of picking something simple and moving forward with that. > This system works quite well as we deal with dead data (no writes nor > update on source data, just writes on target tables which we create on > the fly). > There's a strong overhead on first queries as bloom are computed and > data are sent and received. > But after first N queries, data are "self balancing", and there's no > more data transport so that we can scale up to full parallelism. This makes a lot of sense. If you want an example of something in a different domain that uses a similar technique, this also happens in clusters of squid http web caches. (they call their bloom filters cache digest) > The big issue here is sqlite : > 1) no type support > 2) no transaction support (==> single user system) > 3) some obscur bugs (ex: sqlite sometimes raises an Exception > named "NotAnError", which is very hard to understand, don't you > think? ;)). I've not used sql-lite so probably can't speak to that. That *is* a rather odd error though :) > As Python clearly lacks a good database API, we cannot move easily to > another database. > Dealing with Oracle, MySQL, Postgres from Python is not so easy, and > DB API is never implemented the same way(!), which makes switching > from a RDMS to another a pain. This is something that has always surprised me about python. In the perl world there is one obvious way to do databases - using DBI::DBD, even though perl's motto is "there's more than one way to do it". Whereas in python there are multiple competing DB API's, despite the motto "there should be one obvious way to do it". > JDBC use would be so nice. I can understand that motivation. > So we tried to use Kamaleia within Jython. > We've managed to import Axon + Kamaelia modules "as is", and it > surprisingly worked ... at some level. > I can state a 99% Axon compatibility with Jython. This is interesting, and nice to hear. If you have any patches, or fixes for that final 1% please post them to the l
[kamaelia-list] Re: Bidirectional Json-Rpc Server
On Friday 10 July 2009 14:15:30 Rasjid Wilcox wrote: > Thanks Michael for setting this up. I'll set myself up a code.google > developer id over the weekend and get back to you with it. > > Just so people know, I've got some significant changes in mind, so the > current code is more to get a feel for things than how I think it will > look in the long term. I have another week or two of urgent > end-of-financial year work to get done, and then I will put some more > time into moving it forward. Excellent, what I'll do then is merge this branch onto trunk, and I suggest that you work on your changes on a branch, and I'm happy to review your branches before merge. I'll explain later how this works, but for now please work on the assumption that you "own" the following namespaces: /trunk/Sketches/RW/ /branches/private_RW_* /trunk/Code/Python/Apps/RW /trunk/Code/Python/Apps/JsonRPC /trunk/Code/Python/Kamaelia/Kamaelia/Apps/JsonRPC The only caveat I would have about the last two, is please consider how your project can integrate with the core, as you go on. However, the aim of this structure is to prevent you being blocked by other people, so common sense prevails :-) The overall idea behind this structure btw is that it provides you somewhere to play, somewhere to put stuff up for review, somewhere for packaging to be dealt with, and somewhere for components to be shared & co-ordinated. That last one also means that as time goes on, components which have more general utility can slowly migrate out of Kamaelia.Apps into somewhere else more logical in the hierarchy. Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Bidirectional Json-Rpc Server
Hi All, On Wednesday 22 April 2009 16:12:28 Rasjid wrote: > I'm currently working on (and mostly > finished) a fully bi-directional fully asynchronous json-rpc server > and client written with Kamaelia, Since this conversation, Rasjid and I continued this discussion off list, and over a longer period than I think either of us realised, Rasjid contributed this code back to the project - which is extremely welcome - thank you! The original code/code layout is here: http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/Contrib/jsonrpc/ However, in order to make it more generally useful faster/easier, I've created the namespace Kamaelia.Apps.JsonRPC to be owned by this project. In practical terms, I've created a branch for work to be able to continue on this called private_MPS_RW_JsonRpc Inside /branches/private_MPS_RW_JsonRpc/Apps, a new Apps directory has been created called JsonRPC The structure inside /branches/private_MPS_RW_JsonRpc/Apps/JsonRPC is this: . ./App ./App/server-test.py ./App/client-test.py ./Doc ./Doc/README # needs writing! :-) ./Config ./jsonrpc.py ./MANIFEST.in ./DistBuild ./DistBuild/upload.sh ./DistBuild/setup.py.src ./DistBuild/BuildRelease.sh ./DistBuild/CleanBuild.sh In order to build a distribution, you do this: ~> cd DistBuild/ ~> ./BuildRelease.sh Building the Kamaelia JSON RPC Server Distribution -- Copying current Axon Copying current Kamaelia Copying authors & copying files Creating setup.py file running sdist warning: sdist: standard file not found: should have one of README, README.txt reading manifest template 'MANIFEST.in' warning: no files found matching 'CHANGELOG' writing manifest file 'MANIFEST' This leaves a tarball in dist ~> cd ../dist/ ~> ls Kamaelia-JsonRPC-0.0.1.tar.gz ~> tar zxf Kamaelia-JsonRPC-0.0.1.tar.gz You can find this tarball here: http://www.kamaelia.org/release/Kamaelia-JsonRPC-0.0.1.tar.gz To install: ~> cd Kamaelia-JsonRPC-0.0.1/ ~> sudo python setup.py install --force This installs: * Rasjid's components * jsonrpc * Axon & Kamaelia There's an issue with mainline Axon.Handle at present which I need to track down (nothing to do with Rasjid's code, except it uses Axon.Handle), but once that's resolved this should just work fine. If you let me have your code.google developer id, I can let you "own" these two parts of the tree: /trunk/Code/Python/Kamaelia/Kamaelia/Apps/JsonRPC /trunk/Code/Python/Apps/JsonRPC/ Which you can then use as you would normally, except you can build your own specific distributions, whilst also simplifying the use of your code inside other Kamaelia systems. Assuming you're happy with this Rasjid, I'll merge this onto /trunk - since it allows you the freedom to continue developing this without being constrained by the main project. I think this then enables me to get out of the way, and to be able to help where needed :) Once again, many thanks - this is a really cool contribution :) Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Videos: Europython Kamaelia Tutorial
Hi, These videos are now up on blip.tv. Oddly there's a sync issue with the flash transcode on blip.tv (or it could just be playback here), which isn't in the original files or the ones I transcoded down to(also on blip.tv). But it's all there. It's 4 video files, uneditted: * http://blip.tv/file/2323925 * http://blip.tv/file/2327817 * http://blip.tv/file/2328583 * http://blip.tv/file/2328810 Based on watching these, I'll probably re-record some of these as standalone screencasts, since I've noticed I talk *a lot* quicker in a presentation than I do normally :-) (probably nerves) Incidentally, it was very hot weather that morning! Comments on the content, format, delivery, etc welcome. I think that another day, it would be better to take things at a slower overall pace, but with the same amount of stuff. Feels like a lot for a morning, but not for a day. But then having nice handouts, for people to come back and read makes a difference there I think. Regards, Michael. -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] Europython Kamaelia Tutorial Notes, slides.
Hi, The Kamaelia tutorial at europython has been and gone, as has europython itself. I've uploaded all the materials, and they're now linked here: * http://www.kamaelia.org/PragmaticConcurrency The videos are being transcoded, uploaded etc, and will be available here: * http://europython09.blip.tv/ I've had lots of feedback from the session, primarily of the kind that whilst it was fast, it was paced well, and most people walked away with something that worked for them. The slides had more on them than I'd normally put, but many were skipped because I'd generally work through their contents on a python command line, or in a text editor instead - which worked quite well. Also, I've had lots of feedback on the extensive notes/material I provided people with, which seemed particularly appreciated - using lulu.com for printed materials seems to have been a good idea :-) (More time for proof reading would've been a good idea it seems too, but hey :) There's also a specialised europython distribution as well: * http://www.kamaelia.org/release/Kamaelia-Europython09-1.0.0.tar.gz As well as Axon/Kamaelia 0.9.6.0 this contains the following worked examples: * MiniAxon from generators through to full system including file reader & multicast sender. * A pygame.camera based V4L2 video recording application. Records dirac encoded video - building up from just a basic (non-kamaelia) webcam app. * The bulletin board example, for all files. You can see the logical progression through this like this: clear; cat BB1.py; ./prompt.py; export J=1; for i in `seq 2 15`; do clear; diff -u BB$J.py BB$i.py; J=$i; ./prompt.py; done Any feedback & suggestions for improvement welcome. Otherwise, please enjoy & I hope they're useful :-) Regards, Michael -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
[kamaelia-list] ThreadedChassis
Hi, Given Pipeline and ProcessPipeline, the rather obvious question was asked at Europython "where's ThreadedPipeline? ". Now I know there's issues in doing this, but I think it's worth considering. So, hypothetically, suppose we create ThreadedChassis, which is defined as follows: * It takes a single, non-activated component as it's sole argument. We assume for the moment this is a standard generator component. * It subclasses threadedadaptivecommscomponent * It creates inboxes and outboxes mirroring the interface of the wrapped component, forwarding data to/from it. * Specifically, it also runs the wrapped generator inside the thread. Any new components the generator starts should also be started in the same thread as the wrapped generator. * This implies that the threaded chassis is itself able to act as or owns a scheduler. (This is doable, since a scheduler is a microprocess) * It would require an audit of components to check that they use .activate() The first question that springs to mind is "does this seem feasible", to which I think the answer is yes. We could require components to placed inside a threaded chassis to set a flag stating that they don't change any *class* state, unless it's protected by an STM store. The second question that springs to mind is "where can this go horribly wrong". To my mind, areas include: * Accidental interception of messages to/from a subcomponent that's using a service. * Service discovery & usage * Service advertising. The other point is "is actually wrapping it appropriate" or "should we do something more like @staticmethod or @classmethod" which actually transforms the component? (ie in the manner of a class decorator) Clearly such a beast then allows: Pipeline( ThreadedChassis( A() ), ThreadedChassis( B() ), ThreadedChassis( C() ), ThreadedChassis( D() ), ) Which would then obviously also allow: * ThreadedPipeline( A(),B(), C(), D() ) But the first step is to get a threaded chassis working. As noted, an optimisation is to "simply" convert the component from one type to another, but that has slightly more risk involved, and I think starting with something constrained like this makes the most sense. I suspect though that for somethings though actually /converting/ the component from generator to threaded would make the most sense. It'll probably force a small rethink of some areas, but this is probably the best initial solution to the problem "How do I use all my CPUs" since it would provide a natural path for migrating from generators to threads, and on a GIL-less implementation of python (IronPython, Jython, etc), it would allow the use of multiple CPUs relatively cleanly. If you can spot any problems with this I'd be interested. I don't think there's any fundamental problems here, aside from the usual issues around shared values, but a second set of eyes would be appreciated. I think getting this done, and then someone wrapping py.execnet would probably result in a rather compelling system. Regards, Michael -- http://yeoldeclue.com/blog http://twitter.com/kamaelian http://www.kamaelia.org/Home --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "kamaelia" group. To post to this group, send email to kamaelia@googlegroups.com To unsubscribe from this group, send email to kamaelia+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/kamaelia?hl=en -~--~~~~--~~--~--~---
