Re: Comparison operators in Python
On Jun 2, 1:44 am, harrismh777 harrismh...@charter.net wrote: .. Just another example (excluding print 1/2 and unicode) where 3.x seems to be completely compatible with 2.x/ (tongue-in-cheek) One of the key purposes of the 3.x line of code is to get rid of warts in the language. As a result, if someone is relying on warts, then their code will break when changing from 2.x to 3.x. IMO, this is actually a good thing since it encourages the reduction in warty code. (People who want to use 2.x and 3.x can either use 2to3 and maintain 2to3-able code or write code that works in both 2.x and 3.x - which is eminently doable) (do Brits say tongue-in-cheek?) Yes. Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Extending classes __init__behavior for newbies
It can be broken if someone tries to use the class as is - that is treating the class as a model - to drive a display of the ship. If it was written using super() then that wouldn't be a problem. For example, I could write a display mixin that I'd like to use like this: class VisibleShip(ship, sprite): ... and class FasterVisibleShip(faster_ship, sprite): ... But because neither ship nor faster_ship call their super class correctly, this won't work. To turn this into something concrete, consider the following code: class mixin(object): The key benefit of this mixin is that it tracks instances created and also allows subclasses to have inheritable default values. objects = [] def __init__(self, **kwargs): super(mixin, self).__init__() self.__dict__.update(**kwargs) self.objects.append(self) Using that code, I can create my ship model as a mixin: class ship(mixin): x = 0 y = 0 step = 1 angle = 0 name = def update(self): self.x = self.x + math.cos(math.radians(self.angle)) * self.step self.y = self.y + math.sin(math.radians(self.angle)) * self.step As well as the faster ship: class fastership(ship): speed = 1 def update(self): self.x = self.x + math.cos(math.radians(self.angle)) * self.step * self.speed self.y = self.y + math.sin(math.radians(self.angle)) * self.step * self.speed I can then create a sprite mixin: class sprite(mixin): x = 0 y = 0 image = sprite.png display = None def __init__(self, **kwargs): super(sprite, self).__init__(**kwargs) self._image = pygame.image.load(self.image) print Image Loaded, self.__class__ def render(self,surface): # assume surface is something like a pygame surface surface.blit(self._image, (self.x, self.y)) And also for debug purposes a debug mixin: import time class debug(mixin): debug_file = debug.log debug_handle = None def __init__(self, **kwargs): print Creation arguments, self.__class__, kwargs super(debug, self).__init__(**kwargs) @classmethod def dump_state(klass,self): if klass.debug_handle is None: klass.debug_handle = open(self.debug_file, w) klass.debug_handle.write(str(time.time()) + : + str(klass) + : + str(self.__dict__)+\n\n) klass.debug_handle.flush() Using these we can create visible ships which can also send debug information to a file when they're pinged: class VisibleShip(ship, sprite, debug): x = 300 y = 300 image = ship.png class FasterVisibleShip(fastership, sprite, debug): x = 400 y = 400 image = fastship.png And use them like this: import pygame pygame.init() display = pygame.display.set_mode((800, 600), 0) ship_one = VisibleShip(step=1) ship_two = FasterVisibleShip(angle = 60) t = time.time() while time.time()-t 1 : display.fill((0,0,0)) for sprite in mixin.objects: sprite.render(display) sprite.dump_state(sprite) sprite.update() pygame.display.flip() time.sleep(0.05) Now, you could do this the other way I mentioned by by using kwargs, and super in this way you can get the various classes to play nicely with each other, which means you can have clearer code. After all, our ship / fastership have now become focussed on the code you want them to model - not on faff with superclasses: class ship(mixin): x = 0 y = 0 step = 1 angle = 0 name = def update(self): self.x = self.x + math.cos(math.radians(self.angle)) * self.step self.y = self.y + math.sin(math.radians(self.angle)) * self.step class fastership(ship): speed = 1 def update(self): self.x = self.x + math.cos(math.radians(self.angle)) * self.step * self.speed self.y = self.y + math.sin(math.radians(self.angle)) * self.step * self.speed But not only that they play nicely with debug code and also display code, and you get defaults you can inherit at the same time too. If you wrote ship / fastership with an explicit init and didn't do the super() call - eg like this: class ship(object): def __init__(self,x=0,y=0,step=1,angle=0, name=''): self.x = x self.y = y self.step = step self.angle = angle self.name = name def update(self): self.x = self.x + math.cos(math.radians(self.angle)) * self.step self.y = self.y + math.sin(math.radians(self.angle)) * self.step class fastership(ship): def __init__(self,speed=1): ship.__init__(self,x=0,y=0,step=1,angle=0, name='')
Re: Extending classes __init__behavior for newbies
On Feb 14, 7:15 pm, rantingrick rantingr...@gmail.com wrote: On Feb 14, 11:55 am, Michael Sparks spark...@gmail.com wrote: It can be broken if someone tries to use the class as is - that is treating the class as a model - to drive a display of the ship. If it was written using super() then that wouldn't be a problem. For example, I could write a display mixin that I'd like to use like this: [snip: miles of code not formatted for 79 chars!] Core code recopied into a pastebin instead, if that's your only problem. * http://pastebin.com/nuaafqSE I hardly call 87 lines of code miles of code though myself. I call it a relatively trivial example aimed at showing the benefit of using super() in your code rather than hard coding brittle fragility into your code. That was a nice long winded example of how to break code by using it in an improper manner. No, it was a thought out example of how writing code slightly differently from how you do now can result in something more practical, reusable, extendable, and maintainable. I was hoping to educate you as to why one approach *might* be considered better than another. Deriding is as you do completely misses the point, sadly. Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Namespaces
On Jan 21, 10:39 am, sl33k_ ahsanbag...@gmail.com wrote: What is namespace? And what is built-in namespace? tl;dr - Namespaces are sets that contain names. You can think of namespaces as being /like/ boxes. A namespace is therefore an organisational tool, forming a similar purpose to human names surnames - to identify the right value. (eg Sparks is a namespace, Smith is another.) The built-in namespace contains all the values which python understands which you _don't_ define that don't have dots in. (eg int, True, None) Looking at this in more detail... We can create a simple namespace using an empty class Family: class Family(object): pass Sparks = Family() Smith = Family() Now clearly Sparks is a name, and Smith is a name. Those names are defined to be two different Family objects/values. (I'm going to deliberately sidestep which namespace Sparks and Smith sit inside for the moment.) The neat trick is that namespaces are values themselves. In fact the really neat trick is that every value contains a namespace. How do I define a name inside a namespace? Suppose I want to define the name Michael as a person inside the Sparks namespace, I can do that like this: class Person(object): pass Sparks.Michael = Person() I can then define the name Michael inside the Smith namespace as well: Smith.Michael = Person() As you can see, I can now refer to two different values with the same name - Michael. This may look a little like sophistry, so let's suppose the Person we're referring to as Sparks.Michael has an height of 180cm, and a favourite colour of green, and Smith.Michael has a height of 120cm and a favourite colour of 120. In both cases, it makes sense for use to name the height value height, and name the favourite colour value as favourite_colour. If we did this though ... height = 180 favourite_colour = green height = 120 favourite_colour = purple .. python would only remember the most recent value of each. By recognising that every value is a namespace too, we can define those names inside their namespace. Sparks.Michael.height = 180 Sparks.Michael.favourite_colour = green Smith.Michael.height = 120 Smith.Michael.favourite_colour = purple Now the question that might arise is this: Given I can rewrite the examples above like this... class Family(object): pass class Person(object): pass Sparks = Family() Smith = Family() Sparks_Michael = Person() Smith_Michael = Person() Sparks_Michael_height = 180 Sparks_Michael_favourite_colour = green Smith_Michael_height = 120 Smith_Michael_favourite_colour = purple ... how is this different from before? Well in this latter version we're not using namespaces to organise our names. This means that if I want to write a function that prints a person's height and favourite colour, it has to look like this: def describe_person(height, favourite_colour): print The person is, height, cm tall print Their favourite colour is, favourite_colour Then if I want to use this, I have to do this: describe_person(Sparks_Michael_height, Sparks_Michael_favourite_colour) describe_person(Smith_Michael_height, Smith_Michael_favourite_colour) That's quite messy. What does it look like for the namespace version? def describe_person(somePerson): print The person is, somePerson.height, cm tall print Their favourite colour is, somePerson.favourite_colour describe_person(Sparks.Michael) describe_person(Smith.Michael) describe_person now expects to recieve a single value. Inside that value's namespace it expects to find the values height and colour, and just uses them. As a result, when we use it, rather than passing in each low level attribute (height, colour) we can work at a more convenient level of working with People, and the higher level code becomes clearer. Not only this, if we decide to add an another name to both People ... Sparks.Michael.Pythonista = True Sparks.Michael.Pythonista = False ... we can change describe_person to use this: def describe_person(somePerson): print The person is, somePerson.height, cm tall print Their favourite colour is, somePerson.favourite_colour if somePerson.Pythonista: print And they like python! else: print They don't know python Then our code for describing them remains the same: describe_person(Sparks.Michael) describe_person(Smith.Michael) So far so good I hope. Namespaces can contain code as well as basic values. This means we can have ... tiggles = Cat() rover = Dog() jemima = Duck() tiggles.name = tiggles rover.name = rover jemima.name = jemima ... and we can get them all to have some behaviour called make_noise defined by the call to Cat(), Dog(), Duck() inside their namespace, which allows us to write: tiggles.make_noise() Meow! rover.make_noise() Woof! jemima.make_noise() Quack! And again that means we can do things like: def describe_animal(animal): print animal.name, goes, animal.make_noise() And use it like this: describe_animal(tiggles)
ANN: Kamaelia 1.0.12.0 Released
Hi, I'm happy to announce Kamaelia's 4th release of 2010: 1.0.12.0 (Y.Y.M.r) Kamaelia is a component system based around unix-like concurrency/composition pipelining. There's a strong focus on networked multimedia systems. Kamaelia's license changed earlier this year to the Apache 2.0 License. The release is divided up as follows: * Axon - the core component framework. Provides safe and secure message based concurrency composition using generators as limited co-routines, threads, experimental process based support, and (simplified) software transactional memory. Includes examples. * Kamaelia - A large Ol' Bucket of components, both application specific and generic. Components vary from network systems, through digital tv, graphics, visualisation, data processing etc. These reflect the work and systems that Kamaelia has been used to build. Includes examples. * Apps - A collection of applications built using Kamaelia. Whilst Kamaelia includes a collection of examples, these are either releases of internal apps or exemplars created by contributors. * Bindings - a collection of bindings we maintain as part of Kamaelia, including things like DVB bindings. (Bindings recently changed over to using Cython to make life simpler) Website: http://www.kamaelia.org/Home.html Source: http://code.google.com/p/kamaelia Tutorial: http://www.kamaelia.org/PragmaticConcurrency.html Detail of changes: http://groups.google.com/group/kamaelia/browse_frm/thread/db45646ce1790233 Download: http://www.kamaelia.org/release/MonthlyReleases/Kamaelia-1.0.12.0.tar.gz Overview of Changes in this release: * This rolls up (primarily) 3 application and examples branches. The core functionality for these, as ever, is in the main Kamaelia.Apps namespace, meaning these applications and examples are designed for inclusion or extraction into other applications relatively easily. As a result they act as exemplars for things like 3D visualisation, video and audio communications, twitter mining, database interaction and analysis and django integration. They're also useful (and used) as standalone apps in their own right. * Examples (and application components) added for using the 3D graph visualisation (PyOpenGL based) - one based on visualising collaborations, another based on viewed FOAF networks. * Whiteboard application extended such that: * It supports multiway video comms as well as multiway audio comms. * Adds support for decks (collections of slides which can be downloaded, saved, loaded, emailed, encrypted, etc) * Removes pymedia dependency * Change audio over to us PyAlsaAudio directly. * Adds support for calibrated touch screen displays to Pygame Display. - For example large digital whiteboards in addition to existing tablets etc. * Adds in a Social Bookmarking system that does the following: * Harvests a semantic web/RDF data store for realtime search terms (relating to live television broadcast) * Uses these search terms to search twitter, to identify conversations around the semantic web data. * Takes the resulting tweets, and stores them in a DB * Analyses the tweets (including fixing language for analysis using NLTK) for a variety of aspects, storing these in the DB * Presents the results (graphs of buzz/popularity around the content) * Additionally the system attempts to identify particularly interesting/notable moments based on audience conversations, and provides links back to the actual broadcast programmes. * Additionally provides an API for data, generates word clouds etc. * Front end uses Django and web graph APIs to presnet data. Mailing list: http://groups.google.com/group/kamaelia Have fun :-) Michael Sparks -- http://twitter.com/kamaelian http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/
Re: Nested Scopes unintended behaviour ?
On Mar 17, 8:29 pm, Terry Reedy tjre...@udel.edu wrote:
On 3/17/2010 11:44 AM, Emile van Sebille wrote:
On 3/17/2010 8:16 AM Michael Sparks said...
Hi,
Is the following behaviour expected ?
In short, yes. Assignment within a function forces the variable to
locals.
In 3.x, one can declare names to be nonlocal (ie, local to some outer
function, as opposed to local to the current function or module global).
In your case,
nonlocal on
in your inner swtchfun function would give the behavior you wanted.
Ah, excellent. That makes python closures work more like I'd expect
them to. (A colleague had written the swtchfun I posted, whereas the
generator form was the version I wrote, and he was puzzled as to why
it didn't work as he expected. When I saw it I also couldn't see why.
After hearing it's expected behaviour in 2.6 it's clear that assigning
a name to a value declares the variable to be local, and that unlike
much of python (but like yield) this appears based on static analysis
of the function declaration, rather than dynamic. This does also make
sense since it prevents a name switching scope in a function, and a
nonlocal keyword also makes sense as a result.
Thanks to Emile for pointing out you can also do this in 2.6:
def Toggler(F, B):
print F(Hello)
print F(Hello)
print F(Hello)
print F(Hello)
print F(Hello)
def Switcher(A,B):
enclose={on : True}
def swtchfun(msg, enclose=enclose):
if enclose[on]:
enclose[on] = False
print Switched to A
return A
else:
enclose[on] = True
print Switched to B
return B
#
return Toggler(swtchfun,True)
Switcher(1,2)
I think personally I'd use the generator form myself, since I think
it's clearer (more clearly loops between the two), but this may be a
useful thing to know occasionally.
Cheers :-)
Michael.
--
http://mail.python.org/mailman/listinfo/python-list
Nested Scopes unintended behaviour ?
Hi, Is the following behaviour expected ? Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] on linux2 Type help, copyright, credits or license for more information. def Toggler(F, B): ... print F(Hello) ... print F(Hello) ... print F(Hello) ... print F(Hello) ... print F(Hello) ... def Switcher(A,B): ... on=True ... def swtchfun(msg): ... on_ = on ... if on: ... on = False ... print Switched to A ... return A ... else: ... print Switched to B ... return B ... # ... return Toggler(swtchfun,True) ... Switcher(1,2) Traceback (most recent call last): File stdin, line 1, in module File stdin, line 13, in Switcher File stdin, line 2, in Toggler File stdin, line 4, in swtchfun UnboundLocalError: local variable 'on' referenced before assignment The reason I ask is because logically it makes sense. The on_ = on statement should resolve on to be the value on in the enclosing scope, however it appears that the on = False statement is taking priority. The reason I say this is because if you remove the on = False line you get the expected name resolution: def Toggler(F, B): ... print F(Hello) ... print F(Hello) ... print F(Hello) ... print F(Hello) ... print F(Hello) ... def Switcher(A,B): ... on=True ... def swtchfun(msg): ... on_ = on ... if on: ... print Switched to A ... return A ... else: ... print Switched to B ... return B ... # ... return Toggler(swtchfun,True) ... Switcher(1,2) Switched to A 1 ie it looks like python is not looking at the expected scope in the first instance. To me it looks like a bug, but I can also see a rationale where it's considered a feature (because the on is on the left hand side resolving the value to a local, rather than a value in an enclosed scope) I know that you can work around this as follows: Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] on linux2 Type help, copyright, credits or license for more information. def Toggler(F, B): ... print F(Hello) ... print F(Hello) ... print F(Hello) ... print F(Hello) ... def Switcher(A,B): ... def switchgen(): ... while True: ... yield A ... yield B ... G = switchgen() ... def swtchfun(msg): ... return G.next() ... # ... return Toggler(swtchfun,True) ... Switcher(1,2) 1 2 1 2 But I'm curious as to whether the nested scope issue above is considered a bug or a feature, so I can deal with it appropriately. Any comments welcome :-) Regards, Michael. -- http://yeoldeclue.com/blog http://www.kamaelia.org/Home.html http://twitter.com/kamaelian -- http://mail.python.org/mailman/listinfo/python-list
Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.
On Feb 18, 4:15 pm, Steve Howell showel...@yahoo.com wrote:
...
def print_numbers()
[1, 2, 3, 4, 5, 6].map { |n|
[n * n, n * n * n]
}.reject { |square, cube|
square == 25 || cube == 64
}.map { |square, cube|
cube
}.each { |n|
puts n
}
end
This strikes me as a terrible example. For example, this is
significantly clearer:
def print_numbers()
for n in [1,2,3,4,5,6]:
square, cube = n * n, n * n * n
if square != 25 and cube != 64:
print n
I /can/ see arguments for ruby style blocks in python, but not for
this sort of thing, or lisp style quoted expressions[1]. ie I can see
situations where you have more complex code in real life where they
will definitely simplify things.
[1] This is perhaps more appropriate because '(a b c) is equivalent
to (quote a b c), and quote a b c can be viewed as close to
python's expression lambda: a b c
However, I can also see that in simple situations - such as the
example you post - they will have a tendency to make code
significantly less clear/direct.
I suppose, if I have a choice between something (hard being possible
simple code looking simple) and (hard things being simpler simple
things looking harder), I'd probably personally choose the former.
This is not because I don't like hard things being simple, but because
I think that simple things are more common and making them look harder
is a mistake.
I'm well aware that's opinion however,
Regards,
Michael.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Modifying Class Object
Hi Alf,
On Feb 12, 8:22 pm, Alf P. Steinbach al...@start.no wrote:
Thanks for the effort at non-flaming discussion, it *is*
appreciated.
I would appreciate it if you tried to be non-flaming yourself,
since you can see I am not flaming you.
I was seeking to educate you on a simple matter which you seem
to be misunderstanding in python. This will be my second and
last attempt to do so since you chose to be inflammatory in your
response. (In case you do not understand what I find inflammatory,
I will discuss that at the end)
Please note below I may use CAPS occasionally. Whilst usually
taken as shouting, I mean them as BOLD. Please be aware of this :-)
My reason for choosing to do reply is for the simple reason
that recently I had similar discussions with a colleague who was
likewise stuck talking about implementation aspects (call by
reference/value rather than call with object/sharing).
Before I start, note we're talking about semantics, not
implementation. That distinction is very important.
Yes.
[ inflammatory comment snipped]
Good - common ground - a good starting point.
Now, if I define a language, this has 3 main parts:
* Syntax
* Semantics
* Implementation
The syntax of python is simply and shortly defined in a machine
readable format, and is therefore not interesting to discuss
here.
The implementation of python is similarly well defined. There are
multiple such implementations, one of which is CPython.
However, all those references to implementation aspects,
persisting
[ inflammatory comment snipped]
In theory when talking about a language, we do not need to talk
about implementation details. However when using a language,
implementation details do matter as well.
That's why people talk about implementation aspects. When talking
about how the language is defined, you need to talk about how the
language is defined. It's not defined in terms of Java pointers or
references. It's defined in terms of objects and names. (Or objects
and labels)
The exception to this is with pure functional language. In a pure
functional language I do not care about implementation details,
since they are outside the scope of the language.
It is worth noting that python and functional languages share a
common ethos - though with different conclusions - that optimising
for the programmers expression of the problem rather than for the
machine *matters*.
If you miss this, you will be stuck misunderstanding python,
pretty much forever. If you (the reader, not necessarily Alf)
understand this, good. If you don't, you need to re-read this
and really understand it.
(please bear in mind when I say YOU in that paragraph, I
mean whomever is reading this, not anyone specific)
Let's get down to brass tacks.
In python, the LANGUAGE, there are no pointers or references,
much like in SML, haskell and friends there are no pointers or
references. I'm using SML here as an example, because it is
conceptually close to python in terms to some aspects of
evaluation and how it deals with names and values. (There
are many differences as well, but we're dealing with calling,
names values, in which they are close enough)
Taking an example from SML:
structure Stk =
struct
exception EmptyStack;
datatype 'x stack = EmptyStack | push of ('x * 'x stack);
fun pop(push(x,y)) = y | pop EmptyStack = raise EmptyStack;
fun top(push(x,y)) = x | top EmptyStack = raise EmptyStack;
end;
This may be used, say from the interactive prompt, as follows:
val x = EmptyStack; (* 1 *)
val 'x = x; (* 2 *)
val y = push(5,'x); (* 3 *)
val z = push(4,y); (* 4 *)
top z; (* 5 *)
Now, in this example, in SML, there are only names and values.
Unlike python, all values are immutable, and theoretically, no
sequencing of statements.
Now line 1 takes the EmptyStack value, and the name x is bound
to it. Line 2 takes that same EmptyStack value, and the name 'x
is also bound to it.
There are no pointers in SML, just names and values.
Like python, SML has aliases. 'x for example is an alias for x.
However that is just a symbolic name for the object itself.
When line 3 is evaluated, the value push(5, 'x) is bound to the
name y. Note - push(5, 'x) is in itself a value in SML, because
it has been defined as such in the Datatype definition in the
structure definition Stk.
When we reach line 5, the function top is called with the value
that z is bound to. Not a reference. Not a pointer. The actual
value. In this case z's value is push(4,push(5,EmptyStack)).
Thus the SML runtime evaluates the expression
top push(4,push(5,EmptyStack))
And returns the value 4.
In python, I don't have such richness of ability to define values,
but I have these types play with:
* numbers - 1.0 , 1, 1+2i, 0x11 0777
* strings - hello world
* lists - [hello, world ]
* tuples - (hello, world)
* dicts - {hello : world }
All of these types are defined in python as python objects. How
they are
Re: Modifying Class Object
Hi Alf, Before I start, note we're talking about semantics, not implementation. That distinction is very important. On Feb 11, 4:49 am, Alf P. Steinbach al...@start.no wrote: *The* standard general language independent definition? [ of pointer ] Yes. As defined where? For example, as I used as reference in my first posting, the Java language spec. But it has nothing specifically to do with Java. It is a basic concept in computer science, that (most) CS students learn in their *first year*. E.g. quote src=http://cslibrary.stanford.edu/106/; A pointer stores a reference to something. Unfortunately there is no fixed term for the thing that the pointer points to, and across different computer languages there is a wide variety of things that pointers point to. We use the term pointee for the thing that the pointer points to, and we stick to the basic properties of the pointer/pointee relationship which are true in all languages. The term reference means pretty much the same thing as pointer -- reference implies a more high-level discussion, while pointer implies the traditional compiled language implementation of pointers as addresses. For the basic pointer/pointee rules covered here, the terms are effectively equivalent. /quote This is where you have gone wrong. You have taken a first year undergraduate academic generalisation and assumed that it applies to The World. In theory, there is no difference between practice and theory, but in practice there is (so the saying goes). The World however has another place for defining terms. That place is of highly varying quality, but generally a better place to correct semantics of terms. Who knows, eventually there may be a single commonly accepted viewpoint. (Which would bring a whole new level of pedantry of course )-: I am referring to Wikipedia here. (this is a vague attempt at humour, rather than an attempt to patronise which it may also come over as) Let's look at the tip of the iceberg for that: In computer science, a pointer is a programming language data type whose value refers directly to (or points to) another value stored elsewhere in the computer memory using its address. http://en.wikipedia.org/wiki/Pointer_%28computing%29 Similarly for Call by Value: (which *is* a loaded term) In call-by-value, the argument expression is evaluated, and the resulting value is bound to the corresponding variable in the function (frequently by copying the value into a new memory region). If the function or procedure is able to assign values to its parameters, only its local copy is assigned — that is, anything passed into a function call is unchanged in the caller's scope when the function returns. http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_value Call by Reference: (again, loaded term): In call-by-reference evaluation (also referred to as pass-by- reference), a function receives an implicit reference to the argument, rather than a copy of its value. This typically means that the function can modify the argument- something that will be seen by its caller. http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_reference Call by Sharing: (far less common term) Also known as call by object or call by object-sharing is an evaluation strategy first named by Barbara Liskov et al. for the language CLU in 1974[1]. It is used by languages such as Python[2], Iota, Java (for object references)[3], Ruby, Scheme, OCaml, AppleScript, and many other languages. The semantics of call-by-sharing differ from call-by-reference in that assignments to function arguments within the function aren't visible to the caller (unlike by-reference semantics). http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_sharing As you can see, there are generally accepted terms and definitions here, and python is accepted as falling not into the value or reference camp, along with some other languages. Understanding why IMO comes back to some basic aspects of python which I believe trip up experienced developers. This claim is based on talking to someone experienced in coding for a couple of decades, who has done a CS degree (like me), and just didn't understand why I would use python. I spent an couple of _days_ explaining this, along with python's evaluation model, and at the end we ended up where we started: * Python is a language which is focussed on programmer performance, not machine performance, because relative to programmer cost, machines are cheap. Therefore you focus your language to optimise for the programmer, not the machine. In this case, let's drop back to the word pointer which I can understand that you like. Indeed, it took me a fair while to let go of the word when talking about python, but you do have to. Why? Well, assume this definition isn't bad: In computer science, a pointer is
Re: 2.6, 3.0, and truly independent intepreters
Glenn Linderman wrote: so a 3rd party library might be called to decompress the stream into a set of independently allocated chunks, each containing one frame (each possibly consisting of several allocations of memory for associated metadata) that is independent of other frames We use a combination of a dictionary + RGB data for this purpose. Using a dictionary works out pretty nicely for the metadata, and obviously one attribute holds the frame data as a binary blob. http://www.kamaelia.org/Components/pydoc/Kamaelia.Codec.YUV4MPEG gives some idea structure and usage. The example given there is this: Pipeline( RateControlledFileReader(video.dirac,readmode=bytes, ...), DiracDecoder(), FrameToYUV4MPEG(), SimpleFileWriter(output.yuv4mpeg) ).run() Now all of those components are generator components. That's useful since: a) we can structure the code to show what it does more clearly, and it still run efficiently inside a single process b) We can change this over to using multiple processes trivially: ProcessPipeline( RateControlledFileReader(video.dirac,readmode=bytes, ...), DiracDecoder(), FrameToYUV4MPEG(), SimpleFileWriter(output.yuv4mpeg) ).run() This version uses multiple processes (under the hood using Paul Boddies pprocess library, since this support predates the multiprocessing module support in python). The big issue with *this* version however is that due to pprocess (and friends) pickling data to be sent across OS pipes, the data throughput on this would be lowsy. Specifically in this example, if we could change it such that the high level API was this: ProcessPipeline( RateControlledFileReader(video.dirac,readmode=bytes, ...), DiracDecoder(), FrameToYUV4MPEG(), SimpleFileWriter(output.yuv4mpeg) use_shared_memory_IPC = True, ).run() That would be pretty useful, for some hopefully obvious reasons. I suppose ideally we'd just use shared_memory_IPC for everything and just go back to this: ProcessPipeline( RateControlledFileReader(video.dirac,readmode=bytes, ...), DiracDecoder(), FrameToYUV4MPEG(), SimpleFileWriter(output.yuv4mpeg) ).run() But essentially for us, this is an optimisation problem, not a how do I even begin to use this problem. Since it is an optimisation problem, it also strikes me as reasonable to consider it OK to special purpose and specialise such links until you get an approach that's reasonable for general purpose data. In theory, poshmodule.sourceforge.net, with a bit of TLC would be a good candidate or good candidate starting point for that optimisation work (since it does work in Linux, contrary to a reply in the thread - I've not tested it under windows :). If someone's interested in building that, then someone redoing our MiniAxon tutorial using processes shared memory IPC rather than generators would be a relatively gentle/structured approach to dealing with this: * http://www.kamaelia.org/MiniAxon/ The reason I suggest that is because any time we think about fiddling and creating a new optimisation approach or concurrency approach, we tend to build a MiniAxon prototype to flesh out the various issues involved. Michael -- http://www.kamaelia.org/Home -- http://mail.python.org/mailman/listinfo/python-list
Re: Python suitable for Midi ?
Protocol wrote: Is Python suitable for building a multi-track midi sequencer (with a gui), that would run on windows / mac ? I fail to find sufficient information on this, being a newbie and all. We had a Google Summer of Code student working on this sort of thing this year (This clearly puts a bounds on experience btw). He got a fair way along with this, but didn't get his code to a mergeable state (mainly in the application files rather than in the components which are largely mergeable). Essentially it generated a bunch of components which he was using in his application, which are naturally reusable. I've pinged him since the end of GSOC, and had a response that essentially says sorry, too busy, maybe next semester. We'll see on that (he was having fun, so I suspect he really does want to come back to it, but hasn't had time). In the meantime, the answer is, yes, you can do this in Python. His code is here, if it's useful to you: * http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/JT * ( svn co http://kamaelia.googlecode.com/svn/trunk/Sketches/JT ) - requires Kamaelia 0.6.0 btw Top level file/application file is this: http://kamaelia.googlecode.com/svn/trunk/Sketches/JT/Jam/application/trunk/jam (The reason I don't feel it's mergeable as is, is because it could with a code clean up IMO, but it's pretty good - to say the least - considering the GSOC timeframe - also it shows it is doable, and if you're reusing his components, then it should be very doable) It is likely though that his component library: http://kamaelia.googlecode.com/svn/trunk/Sketches/JT/Jam/library/trunk/Kamaelia/Apps/Jam/Protocol/ will get merged before christmas though, since that code is a lot cleaner. In case you're wondering, he was aiming to build something a bit like this, but using Pygame as the interface, and it allows multiple users to run their local version, connect to each other and do shared jamming - hence the interesting subdirectory is Jam As well as Midi it also supports OSC. Underneath it all, he was using the python rtmidi bindings, and pyosc bindings to talk Midi and Osc. (Most of his quagmire in the last bit of GSOC was caused by audio under linux, which doesn't sound relevant to you) Furthermore, i found references on Python not being really able of multi-threading, that further adds to the confusion. Kamaelia's component model effectively gives you concurrency for free, since you build systems out of components that talk to each other. I haven't attached the introspector to Jam, but I suspect it's embarrassingly parallel. He didn't have to worry about that though :-) If you're curious about the model, this tutorial is specifically targetted at new developers: http://www.kamaelia.org/MiniAxon/ It was originally written for someone (specific) who had learnt python the previous week, had little programming experience, and we needed to get them up to speed quickly and gently. It's been slightly generalised since, but is a nice introduction to the ideas. (Under the hood Kamaelia is significantly more optimised than that example, but the components you create for that tutorial system work with the full/real world system) Regards, Michael -- http://www.kamaelia.org/Home -- http://mail.python.org/mailman/listinfo/python-list
Re: 2.6, 3.0, and truly independent intepreters
Philip Semanchuk wrote: On Oct 25, 2008, at 7:53 AM, Michael Sparks wrote: Glenn Linderman wrote: In the module multiprocessing environment could you not use shared memory, then, for the large shared data items? If the poshmodule had a bit of TLC, it would be extremely useful for this,... http://poshmodule.sourceforge.net/ Last time I checked that was Windows-only. Has that changed? I've only tested it under Linux where it worked, but does clearly need a bit of work :) The only IPC modules for Unix that I'm aware of are one which I adopted (for System V semaphores shared memory) and one which I wrote (for POSIX semaphores shared memory). http://NikitaTheSpider.com/python/shm/ http://semanchuk.com/philip/posix_ipc/ I'll take a look at those - poshmodule does need a bit of TLC and doesn't appear to be maintained. If anyone wants to wrap POSH cleverness around them, go for it! If not, maybe I'll make the time someday. I personally don't have the time do do this, but I'd be very interested in hearing someone building an up-to-date version. (Indeed, something like this would be extremely useful for everyone to have in the standard library now that the multiprocessing library is in the standard library) Michael. -- http://www.kamaelia.org/Home -- http://mail.python.org/mailman/listinfo/python-list
Re: Do I need a lock here?
jasiu85 wrote: Do I need a lock to protect the COMMON_DICT dictionary? AFAIK bytecode operations are atomic and in each thread there's only one crucial bytecode op: STORE_NAME in the first thread and LOAD_NAME in the second one. So I suspect that everything will work just fine. Am I right? Will it crash your system? Probably not. Will you end up with correct values in your dict? Probably not, since you can still end up with race hazards. If you want to share data that way, you may want to look at Kamaelia's STM[1] model, which is described on this page: * http://www.kamaelia.org/STM You can created stores - which are /essentially/ (key value) threadsafe stores which enable you to figure out when things go wrong, so you can do something about it ! :) A short example: from Axon.STM import Store S = Store() # Shared store (like your COMMON_DICT) D = S.using(account_one, account_two, myaccount) D[account_one].set(50) D[account_two].set(100) D.commit()# This will fail if any of the values have changed S.dump() D = S.using(account_one, account_two, myaccount) D[myaccount].set(D[account_one].value+D[account_two].value) D[account_one].set(0) D[account_two].set(0) D.commit()# This will fail if any of the values have changed S.dump() If this looks familiar, it's the same basic idiom as version control. The nice thing of course, is in this version you don't need to worry about locks, but just be aware that the .commit() may fail, and you may need to retry. Two examples - one using bare threads, one using Kamaelia components are here: * http://kamaelia.googlecode.com/svn/trunk/Code/Python/Axon/Examples/STM/ Crucially whilst the above says this will fail, the examples in that directory handle the failures correctly :) This code is included in the latest Kamaelia release, described here: * http://www.kamaelia.org/GetKamaelia However you can also use the standalone tarball attached to that page. [1] ie a minimal software transactional memory implementation. The point of these things really is to allow you to detect when something has gone wrong (eg and update fails) and to redo the thing that led to the update. It's conceptually very similar to version control for variables. I'm tempted to replace using with checkout to mirror commit. Michael. -- http://www.kamaelia.org/GetKamaelia -- http://mail.python.org/mailman/listinfo/python-list
Re: 2.6, 3.0, and truly independent intepreters
Hi Andy, Andy wrote: However, we require true thread/interpreter independence so python 2 has been frustrating at time, to say the least. Please don't start with but really, python supports multiple interpreters because I've been there many many times with people. And, yes, I'm aware of the multiprocessing module added in 2.6, but that stuff isn't lightweight and isn't suitable at all for many environments (including ours). This is a very conflicting set of statements and whilst you appear to be extremely clear on what you want here, and why multiprocessing, and associated techniques are not appropriate, this does sound very conflicting. I'm guessing I'm not the only person who finds this a little odd. Based on the size of the thread, having read it all, I'm guessing also that you're not going to have an immediate solution but a work around. However, also based on reading it, I think it's a usecase that would be generally useful in embedding python. So, I'll give it a stab as to what I think you're after. The scenario as I understand it is this: * You have an application written in C,C++ or similar. * You've been providing users the ability to script it or customise it in some fashion using scripts. Based on the conversation: * This worked well, and you really liked the results, but... * You only had one interpreter embedded in the system * You were allowing users to use multiple scripts Suddenly you go from: Single script, single memory space. To multiple scripts, unconstrained shared shared memory space. That then causes pain for you and your users. So as a result, you decided to look for this scenario: * A mechanism that allows each script to think it's the only script running on the python interpreter. * But to still have only one embedded instance of the interpreter. * With the primary motivation to eliminate the unconstrained shared memory causing breakage to your software. So, whilst the multiprocessing module gives you this: * With the primary motivation to eliminate the unconstrained shared memory causing breakage to your software. It's (for whatever reason) too heavyweight for you, due to the multiprocess usage. At a guess the reason for this is because you allow the user to run lots of these little scripts. Essentially what this means is that you want green processes. One workaround of achieving that may be to find a way to force threads in python to ONLY be allowed access to (and only update) thread local values, rather than default to shared values. The reason I say that, is because the closest you get to green processes in python at the moment is /inside/ a python generator. It's nowhere near the level you want, but it's what made me think of the idea of green processes. Specifically if you have the canonical example of a python generator: def fib(): a,b = 1,1 while 1: a,b = b, a+b yield 1 Then no matter how many times I run that, the values are local, and can't impact each other. Now clearly this isn't what you want, but on some level it's *similar*. You want to be able to do: run(this_script) and then when (this_script) is running only use a local environment. Now, if you could change the threading API, such that there was a means of forcing all value lookups to look in thread local store before looking outside the thread local store [1], then this would give you a much greater level of safety. [1] I don't know if there is or isn't I've not been sufficiently interested to look... I suspect that this would also be a very nice easy win for many multi-threaded applications as well, reducing accidental data sharing. Indeed, reversing things such that rather than doing this: myLocal = threading.local() myLocal.X = 5 Allowing a thread to force the default to be the other way round: systemGlobals = threading.globals() systemGlobals = 5 Would make a big difference. Furthermore, it would also mean that the following: import MyModule from MyOtherModule import whizzy thing I don't know if such a change would be sufficient to stop the python interpreter going bang for extension modules though :-) I suspect also that this change, whilst potentially fraught with difficulties, would be incredibly useful in python implementations that are GIL-free (such as Jython or IronPython) Now, this for me is entirely theoretical because I don't know much about python's threading implementation (because I've never needed to), but it does seem to me to be the easier win than looking for truly independent interpreters... It would also be more generally useful, since it would make accidental sharing of data (which is where threads really hurt people most) much harder. Since it was raised in the thread, I'd like to say use Kamaelia, but your usecase is slightly different as I understand it. You want to take existing stuff that won't be written in any particular way, to
Re: 2.6, 3.0, and truly independent intepreters
Andy O'Meara wrote: Yeah, that's the idea--let the highest levels run and coordinate the show. Yes, this works really well in python and it's lots of fun. We've found so far you need at minimum the following parts to a co-ordination little language: Pipeline Graphline Carousel Seq OneShot PureTransformer TPipe Filter Backplane PublishTo SubscribeTo The interesting thing to me about this is in most systems these would be patterns of behaviour in activities, whereas in python/kamaelia these are concrete things you can drop things into. As you'd expect this all becomes highly declarative. In practice the world is slightly messier than a theoretical document would like to suggest, primarily because if you consider things like pygame, sometimes you have only have a resource instantiated once in a single process. So you do need a mechanism for advertising services inside a process and looking those up. (The Backplane idea though helps with wrapping those up a lot I admit, for certain sorts of service :) And sometimes you do need to just share data, and when you do that's when STM is useful. But concurrent python systems are fun to build :-) Michael. -- http://www.kamaelia.org/GetKamaelia -- http://mail.python.org/mailman/listinfo/python-list
Re: 2.6, 3.0, and truly independent intepreters
Glenn Linderman wrote: In the module multiprocessing environment could you not use shared memory, then, for the large shared data items? If the poshmodule had a bit of TLC, it would be extremely useful for this, since it does (surprisingly) still work with python 2.5, but does need a bit of TLC to make it usable. http://poshmodule.sourceforge.net/ Michael -- http://www.kamaelia.org/GetKamaelia -- http://mail.python.org/mailman/listinfo/python-list
Re: 2.6, 3.0, and truly independent intepreters
Andy O'Meara wrote: basically, it seems that we're talking about the embarrassingly parallel scenario raised in that paper We build applications in Kamaelia and then discover afterwards that they're embarrassingly parallel and just work. (we have an introspector that can look inside running systems and show us the structure that's going on - very useful for debugging) My current favourite example of this is a tool created to teaching small children to read and write: http://www.kamaelia.org/SpeakAndWrite Uses gesture recognition and speech synthesis, has a top level view of around 15 concurrent components, with signficant numbers of nested ones. (OK, that's not embarrasingly parallel since it's only around 50 things, but the whiteboard with around 200 concurrent things, is) The trick is to stop viewing concurrency as the problem, but to find a way to use it as a tool for making it easier to write code. That program was a 10 hour or so hack. You end up focussing on the problem you want to solve, and naturally gain a concurrent friendly system. Everything else (GIL's, shared memory etc) then just becomes an optimisation problem - something only to be done if you need it. My previous favourite examples were based around digital TV, or user generated content transcode pipelines. My reason for preferring the speak and write at the moment is because its a problem you wouldn't normally think of as benefitting from concurrency, when in this case it benefitted by being made easier to write in the first place. Regards, Michael -- http://www.kamaelia.org/GetKamaelia -- http://mail.python.org/mailman/listinfo/python-list
Re: 2.6, 3.0, and truly independent intepreters
Jesse Noller wrote: http://www.kamaelia.org/Home Thanks for the mention :) I don't think it's a good fit for the original poster's question, but a solution to the original poster's question would be generally useful IMO, _especially_ on python implementations without a GIL (where threads are the more natural approach to using multiple processes multiple processors). The approach I think would be useful would perhaps by allowing python to have some concept of green processes - that is threads that can only see thread local values or they search/update thread local space before checking globals, ie flipping X = threading.local() X.foo = bar To something like: X = greenprocesses.shared() X.foo = bar Or even just changing the search for values from: * Search local context * Search global context To: * Search thread local context * Search local context * Search global context Would probably be quite handy, and eliminate whole classes of bugs for people using threads. (Probably introduce all sorts of new ones of course, but perhaps easier to isolate ones) However, I suspect this is also *a lot* easier to say than to implement :-) (that said, I did hack on the python internals once (cf pep 318) so it might be quite pleasant to try) It's also independent of any discussions regarding the GIL of course since it would just make life generally safer for people. BTW, regarding Kamaelia - regarding something you said on your blog - whilst the components list on /Components looks like a large amount of extra stuff you have to comprehend to use, you don't. (The interdependency between components is actually very low.) The core that someone needs to understand is the contents of this: http://www.kamaelia.org/MiniAxon/ Which is sufficient to get someone started. (based on testing with a couple of dozen novice developers now :) If someone doesn't want to rewrite their app to be kamaelia based, they can cherry pick stuff, by running kamaelia's scheduler in the background and using components in a file-handle like fashion: * http://www.kamaelia.org/AxonHandle The reason /Components contains all those things isn't because we're trying to make it into a swiss army knife, it's because it's been useful in domains that have generated those components which are generally reusable :-) Michael. -- http://www.kamaelia.org/GetKamaelia -- http://mail.python.org/mailman/listinfo/python-list
ANN: Kamaelia 0.6.0, Axon 1.6.0 - New Release
/MulticoreExample
- Largely boils down to put Process in front of Pipeline to
make all the subcomponents of the pipeline run in seperate
processes
- Practical benefit for pygame components - it allows multiwindow
pygame apps.
* Inheritable default values for component initialisers.
- The core aim of this is to allow declarative config for
systems rather than something less clear.
- This allows you to turn this sort of code:
def ReusableSocketAddrServer(port=100,
protocol=EchoProtocol):
return ServerCore(protocol=protocol,
port=port,
socketOptions=(socket.SOL_SOCKET,
socket.SO_REUSEADDR, 1))
- Into this:
class ReusableSocketAttrServer(ServerCore):
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
- Full discussion here:
http://www.kamaelia.org/InheritableDefaultValues
* Added in Handle support. This provides two key pieces of
functionality:
- The ability to run Kamaelia systems in the background, via:
from Axon.background import background
background().start()
- The ability to wrap Kamaelia components or systems in a Handle
for use in non-Kamaelia systems.
from Axon.Handle import Handle
from Kamaelia.Internet.TCPClient import TCPClient
conn = Handler(TCPClient(www.kamaelia.org, 80)).activate()
conn.put(GET / HTTP/1.0\r\n, inbox)
conn.put(Host: www.kamaelia.org\r\n, inbox)
conn.put(\r\n, inbox)
- More detail:
http://www.kamaelia.org/AxonHandle
* Simplified system shutdown. If you want to close down an entire
Kamaelia based
system rapidly from inside a component, just do this:
- self.scheduler.stop()
- This puts the scheduler into a shutting down mode in which:
- It calls the .stop() method on all the things it's running
- It then shuts down.
- The Kamaelia.Internet components take this as an opportunity
to close all the connections they have open cleanly for example.
- Example usage: http://tinyurl.com/AxonShutdown
* Support for WaitComplete extended, allowing better handling of
more complex protocols which are not stateless in a debuggable
fashion. It also simplifies working with Pygame, etc.
- http://www.kamaelia.org/WaitComplete
* As well as unpausing a component when a message is delivered to
the component, it gets unpaused when a message is taken from its
outbox. This allows better synchronous behaviour for rate limited
in/out-boxes.
Overview of Changes to Kamaelia itself
==
Key changes to Kamaelia itself:
* Creation of the Kamaelia.Apps namespace
* Shifting of the core code for Kamaelia tools into Kamaelia.Apps
* Significant numbers of new components
* Significant number of bugfixes
* SimpleServer code changed to ServerCore, representing it's more
general structure.
In this release there is a slew of extra components and bug fixes, a
variety of new tools - from video shot change detection, through to
SMTP greylisting, but also perhaps the biggest extra: Multiprocess
hence multicore support (experimental at this stage, but so far so
good :) )
Since even the summary is not a short list of changes, I've left that
summary until after my .sig below.
Platforms
=
Kamaelia has been used successfully under both Linux, Windows and
Mac OS X. (mostly developed/tested under Linux Mac OS X)
Where can I get it? Docs?
===
Download:
http://www.kamaelia.org/GetKamaelia
http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz
Docs:
http://www.kamaelia.org/Docs/Axon/Axon
http://www.kamaelia.org/Components
http://www.kamaelia.org/Cookbook
http://www.kamaelia.org/MiniAxon
Presentations:
http://www.slideshare.net/kamaelian
Get involved:
http://www.kamaelia.org/Developers/
http://groups.google.com/group/kamaelia *CHANGED MAILING LIST*
http://code.google.com/p/kamaelia/
Licensing
=
Kamaelia is released under the Mozilla tri-license scheme
(MPL1.1/GPL2.0/LGPL2.1). See http://www.kamaelia.org/Licensing
Finally, many thanks to everyone who's contributed to this release.
Best Regards,
Michael.
--
Michael Sparks, Senior Research Engineer, BBC Research
[EMAIL PROTECTED], Kamaelia Project Lead,
http://www.kamaelia.org/GetKamaelia
BBC Manchester
New files in Kamaelia.*
===
This represents significant amounts of new components and new abilities
added into Kamaelia.
Kamaelia/
Chassis/Seq.py
Codec/
WAV.py YUV4MPEG.py
Device/DVB/
SoftDemux.py
Parse/
{ lots of components for working with DVB information
ANN: Kamaelia 0.6.0, Axon 1.6.0 - New Release
- Largely boils down to put Process in front of Pipeline to
make all the subcomponents of the pipeline run in seperate
processes
- Practical benefit for pygame components - it allows multiwindow
pygame apps.
* Inheritable default values for component initialisers.
- The core aim of this is to allow declarative config for
systems rather than something less clear.
- This allows you to turn this sort of code:
def ReusableSocketAddrServer(port=100,
protocol=EchoProtocol):
return ServerCore(protocol=protocol,
port=port,
socketOptions=(socket.SOL_SOCKET,
socket.SO_REUSEADDR, 1))
- Into this:
class ReusableSocketAttrServer(ServerCore):
socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
- Full discussion here:
http://www.kamaelia.org/InheritableDefaultValues
* Added in Handle support. This provides two key pieces of
functionality:
- The ability to run Kamaelia systems in the background, via:
from Axon.background import background
background().start()
- The ability to wrap Kamaelia components or systems in a Handle
for use in non-Kamaelia systems.
from Axon.Handle import Handle
from Kamaelia.Internet.TCPClient import TCPClient
conn = Handler(TCPClient(www.kamaelia.org, 80)).activate()
conn.put(GET / HTTP/1.0\r\n, inbox)
conn.put(Host: www.kamaelia.org\r\n, inbox)
conn.put(\r\n, inbox)
- More detail:
http://www.kamaelia.org/AxonHandle
* Simplified system shutdown. If you want to close down an entire
Kamaelia based
system rapidly from inside a component, just do this:
- self.scheduler.stop()
- This puts the scheduler into a shutting down mode in which:
- It calls the .stop() method on all the things it's running
- It then shuts down.
- The Kamaelia.Internet components take this as an opportunity
to close all the connections they have open cleanly for example.
- Example usage: http://tinyurl.com/AxonShutdown
* Support for WaitComplete extended, allowing better handling of
more complex protocols which are not stateless in a debuggable
fashion. It also simplifies working with Pygame, etc.
- http://www.kamaelia.org/WaitComplete
* As well as unpausing a component when a message is delivered to
the component, it gets unpaused when a message is taken from its
outbox. This allows better synchronous behaviour for rate limited
in/out-boxes.
Overview of Changes to Kamaelia itself
==
Key changes to Kamaelia itself:
* Creation of the Kamaelia.Apps namespace
* Shifting of the core code for Kamaelia tools into Kamaelia.Apps
* Significant numbers of new components
* Significant number of bugfixes
* SimpleServer code changed to ServerCore, representing it's more
general structure.
In this release there is a slew of extra components and bug fixes, a
variety of new tools - from video shot change detection, through to
SMTP greylisting, but also perhaps the biggest extra: Multiprocess
hence multicore support (experimental at this stage, but so far so
good :) )
Since even the summary is not a short list of changes, I've left that
summary until after my .sig below.
Platforms
=
Kamaelia has been used successfully under both Linux, Windows and
Mac OS X. (mostly developed/tested under Linux Mac OS X)
Where can I get it? Docs?
===
Download:
http://www.kamaelia.org/GetKamaelia
http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz
Docs:
http://www.kamaelia.org/Docs/Axon/Axon
http://www.kamaelia.org/Components
http://www.kamaelia.org/Cookbook
http://www.kamaelia.org/MiniAxon
Presentations:
http://www.slideshare.net/kamaelian
Get involved:
http://www.kamaelia.org/Developers/
http://groups.google.com/group/kamaelia *CHANGED MAILING LIST*
http://code.google.com/p/kamaelia/
Licensing
=
Kamaelia is released under the Mozilla tri-license scheme
(MPL1.1/GPL2.0/LGPL2.1). See http://www.kamaelia.org/Licensing
Finally, many thanks to everyone who's contributed to this release.
Best Regards,
Michael.
--
Michael Sparks, Senior Research Engineer, BBC Research
[EMAIL PROTECTED], Kamaelia Project Lead,
http://www.kamaelia.org/GetKamaelia
BBC Manchester
New files in Kamaelia.*
===
This represents significant amounts of new components and new abilities
added into Kamaelia.
Kamaelia/
Chassis/Seq.py
Codec/
WAV.py YUV4MPEG.py
Device/DVB/
SoftDemux.py
Parse/
{ lots of components for working with DVB information
tables
Re: Best way to spawn process on back end computer
sophie_newbie wrote: Hi, I'm running a python cgi script on a frontend web server and I want it to spawn another script (that takes a long time to run) on a backend number crunching server thats connected to the same network. What do you think is the best way to do this? I have a few ideas but I'm sure there is a best way to go about this. Best is always subjective - what sort of interaction are you after? Web request to trigger an activity (fire and forget) Web request to trigger an activity, which produces output suitable for something else? Web request to trigger an activity, which produces output after a while that you want to send back over a later web request? Web request to trigger an activity, which produces output after a while that you want to send back over a later web request, and you want to easily tie the two things together? Web request to trigger an activity, which produces output after a while that you want to send back over a later web request, and you want those two events to look like part of one operation? Picking a concrete example... Or something a bit more like: * User uploads an image * Gets converted to a bunch of standard sizes, and placed into a queue for moderation and then later use? Or perhaps: * User uploads a video * That gets transcoded to a different video format ? (eg video contribution transcoded to flash video?) In the latter two cases, this is something I've needed to do, so I created a simple WSGI filter that dumps the uploaded images/videos into a standard file location, and then had a script that watches that standard location for new images videos, and does those conversions and transcodes. Strictly speaking to be clean about it, you need to dump the file to a temporary location whilst you're grabbing it from the network and when it's been written to disk rename it. I've no idea if this maps to your problem at all. (since I can think of a few scenarios that match your description). If it does match, then this code may be handy: http://code.google.com/p/kamaelia/source/browse/branches/private_MPS_Participate/Apps/FileProcessor/App/BatchFileProcessor.py If you want to use that code, let me know and I'll package it up. There's an older version here: * http://www.kamaelia.org/release/Kamaelia-FileProcessor-0.1.0.tar.gz Regards, Michael. -- http://www.kamaelia.org/Home -- http://mail.python.org/mailman/listinfo/python-list
Re: [Kamaelia] TCPClient: How to sense connection failure?
Bjoern Schliessmann wrote: Whoops, the TCP client does in fact quit if the server closes connection :) Great - so it wasn't a problem with the TCPClient after all :-) For some reason, my Listener doesn't quit. I thought it's sufficient to exit the main method in some way to quit a component? That's what I do using break in the 'if self.dataReady(control)' part of main. It is sufficient, and running with Kamaelia from /trunk, your listener does indeed shutdown correctly - so it looks like it's a bug in that release of Kamaelia. That's my bad really because we haven't done a full release in quite some time. Looking at the old code, it appears that in that code the TCPClient wasn't forwarding shutdown messages correctly, so there *was* a bug, but there doesn't seem to be now. My suggestion for the moment would be to use the code on /trunk since this is stable at present (development happens on branches rather than /trunk) and that I really ought to sort out the next release - I really hadn't realised just how long it had been since the last release! steps: ~ svn co https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk kamaelia-trunk ~ cd kamaelia-trunk ~/kamaelia-trunk cd Code/Python/Axon ~/kamaelia-trunk/Code/Python/Axon sudo python setup.py install ~/kamaelia-trunk/Code/Python/Axon cd ../Kamaelia ~/kamaelia-trunk/Code/Python/Kamaelia sudo python setup.py install I've tested using this, and your code works as you'd expect. Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: [Kamaelia] TCPClient: How to sense connection failure?
Bjoern Schliessmann wrote: Hello, I'm currently trying to implement a simulation program with Kamaelia and need a reliable TCP connection to a data server. The behaviour you're seeing sounds odd (which is hopefully encouraging :-), but it's not clear from the description whether its a bug in your code or Kamaelia. One question I really have as a result is what version are you using? Current release version 0.5.0, the version on /trunk, or the bleeding edge version on /branches/private_MPS_Scratch. (I'm using the latter to run my greylisting server - as are a few others). All that said though, looking at the differences between versions, I'm not convinced they're large enough to show the problem you're seeing. I'm not about to rule out a bug I don't know about though :-) From Twisted, I know that a method is called if the connection fails by whatever reason. I tried to get the same results with Kamaelia's TCPClient component. If I start up the component and try to connect to a closed TCP port it fails and sends a message out of the signal box, that's okay. If you'd prefer more information in that message, please let me know. (all components send out a message when they shutdown. For things that send data out as one of their primary actions send out a producerFinished message) But if the connection attempt succeeds and, after some time, the server drops the connection with full TCP handshake (FIN, FIN+ACK, ACK), the component just hangs and does nothing. Is this by design, or could there be an error in my setup? It sounds like an error in your setup... but I hate saying that. (Doesn't tell me or you what it is, and doesn't help change things to discourage or detect mistakes in usage) When the server drops the connection in my setups, the client disconnects cleanly when the server dies, with the client code looking like this: self.send(producerFinished(self,self.howDied), signal) Meaning you get a message telling you how the component shutdown as well as the fact it shutdown. (If howDied is None, it's just a normal shutdown - ie as a result of being told to shut down) The function where this is managed is runClient in the class Kamaelia.Internet.TCPClient.TCPClient (fully qualified name) The actual socket connections are handled by a class called ConnectedSocketAdapter which manages all logic of checking for errors, remote shutdown etc. That works the same for both servers and clients so breakage in clients would show up as breakage in servers as well, which would be particularly bad. Also, how long does a TCPClient component live -- or how can/should I terminate it explicitly? I'm afraid that if I create a ReconnectingTCPClient component, it could eat up memory over long runtime with hanging TCPClients. That shouldn't be an issue (I hate the word should), but you can do this using a carousel component. (I ought to write that as an example of how to use the Carousel component) In the meantime - whilst I check to see if there's a bug I didn't know about, the following 2 cookbook entries may be of use: * http://kamaelia.sourceforge.net/Cookbook/TCPSystems * http://kamaelia.sourceforge.net/Cookbook/Carousels - allows you to make something that exits reusable. It's a little awkward to get your head around, but is quite useful when you do. (I've heard of others using Carousel TCPClient to make a reconnecting TCPClient in the past) All that said, I'm not going to rule out a bug and look into it. (if you have a simple example you find fails, please forward it to me :) *thinks* The following code may also be useful when debugging: from Kamaelia.Chassis.Pipeline import Pipeline class PeriodicWakeup(Axon.ThreadedComponent.threadedcomponent): interval = 300 def main(self): while 1: time.sleep(self.interval) self.send(tick, outbox) class WakeableIntrospector(Axon.Component.component): def main(self): while 1: Q = [ q.name for q in self.scheduler.listAllThreads() ] Q.sort() print *debug* THREADS+ str(Q) self.scheduler.debuggingon = False yield 1 while not self.dataReady(inbox): self.pause() yield 1 while self.dataReady(inbox): self.recv(inbox) Pipeline( PeriodicWakeup(), WakeableIntrospector(), ).activate() If you put this code somewhere before your run call, you'll get periodic output to tell you what's running. When debugging manually I'd drop the interval to 3-10 seconds or so. I use 300 for a server. Now, off to see if I can reproduce your problem... :) Regards, Michael. -- http://kamaelia.sourceforge.net/Home http://yeoldclue.com/blog -- http://mail.python.org/mailman/listinfo/python-list
Re: cloud computing (and python)?
Aaron Watters wrote: (from a gmail account) So cloud computing is java diskless workstations warmed over but less flexible? I'm having trouble understanding why people would want to buy in to this. Why do you like gmail - since you appear to use it? (I can think of several possibilities) The reason I ask Gmail is a an example of computing in the cloud. Specifically it's an application in the cloud. You get several classes of things in the cloud - one possible break up: * Applications - gmail, amazon, hotmail, facebook widgets, writely, blogger, flickr, etc. * Components - YUI, EC2, S3 * Frameworks - open social, facebook etc. Each has benefits. Some examples: * gmail, hotmail, yahoomail - spam filtering, access your mail anywhere. You rent the application by paying with attention (or paying money - I think hotmail still do that) * S3 - scalable storage in the cloud WITH scalable serving. The trade off here is how much does it cost you to run a colo box or dedicated server vs how much to rent the space. You rent capacity on demand. (a bit like why buy storage at a self-storage place rather than buy a garage? - there are good reasons both ways round :-) * EC2 - Similar, but to do with computing capacity. EC2 S3 allow you to scale for example in line _and in time_ with the size of your userbase - assuming your business model (if you have one :-) matches * open social, facebook - rather than build your own social graph, you can attach yourself to an existing one to simplify take-up. I must admit I feel a hint of amusement though at your comment above, when it's sent from precisely the sort of setup you appear bemused by - since you appear to have already bought into it without realising ! :-D Have fun :-) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: fiber(cooperative multi-threading)
Bjoern Schliessmann wrote: Michael Sparks wrote: All that said, my personal primary aim for kamaelia is to try and make it into a general toolkit for making concurrency easy natural (as well as efficient) to work with. If full blown coroutines turn out to be part of that c'est le vie :-) I must admit I mostly didn't follow this thread, but it sparked my interest in Kamaelia. I already did the MiniAxon tutorial and I plan to try out Kamaelia with a project I had discussed here a while ago (which I had to suspend until recently because of not enough spare time; it's about simulating complex relais circuits). Please continue the great work. Many thanks for your kind words - the work is continuing :-) Also, I'd be interested in hearing how your project gets on - it sounds like the sort of thing that Kamaelia should be able to help with. (If it doesn't/can't, then it's a bug IMO :) Regards merry christmas everyone, Björn P.S.: In the MiniAxon tutorial, I noticed a formatting problem and a bunch of typos on one page but the feedback link doesn't work; are you interested in a list? I'm always interested in feedback! The fact the feedback link doesn't work for you is particularly useful - I'll look into that! Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: fiber(cooperative multi-threading)
Duncan Booth wrote: There are also problems where full blown coroutines are appropriate. The example I quoted earlier of turning a parser from one which generates a lot of callbacks to one which 'yields' tokens is the usual example given. For completeness, I looked at the other half of the thread at the expat parser, and decided to write that in the style we use for Kamaelia - it ends up looking like this: import xml.parsers.expat import Axon from Kamaelia.Chassis.Pipeline import Pipeline from Kamaelia.Util.Console import ConsoleEchoer class Parser(Axon.ThreadedComponent.threadedcomponent): data = h1 Default /h1 # Can be overridden by kwargs as normal def start_element(self, name, attrs): self.send((START, name, attrs), outbox) def end_element(self, name): self.send((END, name), outbox) def char_data(self, data): data = data.strip() self.send((DATA, data), outbox) def main(self): p = xml.parsers.expat.ParserCreate() p.StartElementHandler = self.start_element p.EndElementHandler = self.end_element p.CharacterDataHandler = self.char_data p.Parse(self.data, 1) self.send(Axon.Ipc.producerFinished(), signal) Pipeline( Parser(data=bodyh1Hello/h1 world pWoo/p/body), ConsoleEchoer(), ).run() You'll note we don't use generators for Parser in this context. This also isn't 100% identical to form you use since we don't turn this into an iterator (obviously :). We do also use 1 more thread than the greenlet approach though. Pipeline ConsoleEchoer are generator based though, as is the scheduler that runs them :-) Have fun :-) Michael. -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Developers/ -- http://mail.python.org/mailman/listinfo/python-list
ANN: Axon.STM 1.0.1 (release) Minimalistic Software Transactional Memory (with examples)
: X = self.getforks() time.sleep(0.2) self.releaseforks(X) time.sleep(0.3+random.random()) def getforks(self): gotforks = False while not gotforks: try: X = self.store.using(*self.forks) if all([ X[fork].value for fork in self.forks], None): for fork in self.forks: X[fork].value = self.name X.commit() gotforks = True else: time.sleep(random.random()) except Axon.STM.ConcurrentUpdate: time.sleep(random.random()) print Got forks!, self.name, self.forks return X def releaseforks(self,X): print releasing forks, self.name for fork in self.forks: X[fork].value = None X.commit() S = Store() N = 5 for i in range(1,N): Philosopher(store=S,forks=[fork.%d % i ,fork.%d % (i+1)]).activate() Philosopher(store=S,forks=[fork.%d % N ,fork.%d % 1]).run() Feedback Feedback is very welcome, preferably via email to the Kamaelia List * [EMAIL PROTECTED] Feedback especially regarding bugs and logical errors is particularly welcome. (hopefully there aren't any - but it's always hard to spot your own) Thanks == Many thanks to Fuzzyman, Duncan Booth, John J Lee Sylvain Hellegouarch for feedback whilst I was prototyping this. Further thanks go to Richard Taylor for detailed feedback and discussion regarding locking and for pointing me at MASCOT which made me think of doing the dining philosophers this way :-) Future == This will be merged onto the mainline of Kamaelia with some auxillary functions , as another feather aimed at making concurrency easy to work with :-) Best Regards, Michael. -- Michael Sparks, Kamaelia Project http://kamaelia.sourceforge.net/Developers/ http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations.html
ANN: Axon.STM 1.0.1 (release) Minimalistic Software Transactional Memory (with examples)
: X = self.getforks() time.sleep(0.2) self.releaseforks(X) time.sleep(0.3+random.random()) def getforks(self): gotforks = False while not gotforks: try: X = self.store.using(*self.forks) if all([ X[fork].value for fork in self.forks], None): for fork in self.forks: X[fork].value = self.name X.commit() gotforks = True else: time.sleep(random.random()) except Axon.STM.ConcurrentUpdate: time.sleep(random.random()) print Got forks!, self.name, self.forks return X def releaseforks(self,X): print releasing forks, self.name for fork in self.forks: X[fork].value = None X.commit() S = Store() N = 5 for i in range(1,N): Philosopher(store=S,forks=[fork.%d % i ,fork.%d % (i+1)]).activate() Philosopher(store=S,forks=[fork.%d % N ,fork.%d % 1]).run() Feedback Feedback is very welcome, preferably via email to the Kamaelia List * [EMAIL PROTECTED] Feedback especially regarding bugs and logical errors is particularly welcome. (hopefully there aren't any - but it's always hard to spot your own) Thanks == Many thanks to Fuzzyman, Duncan Booth, John J Lee Sylvain Hellegouarch for feedback whilst I was prototyping this. Further thanks go to Richard Taylor for detailed feedback and discussion regarding locking and for pointing me at MASCOT which made me think of doing the dining philosophers this way :-) Future == This will be merged onto the mainline of Kamaelia with some auxillary functions , as another feather aimed at making concurrency easy to work with :-) Best Regards, Michael. -- Michael Sparks, Kamaelia Project http://kamaelia.sourceforge.net/Developers/ http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-list
Re: fiber(cooperative multi-threading)
Duncan Booth wrote: [ snip sections about why the single layer aspect can be helpful ] I'm happy with generators as they are: they're a great solution to a problem that most of us didn't realise we had until the solution came along. That doesn't mean that I wouldn't also like to have a separate coroutine library, but I would use it to replace situations that might otherwise have to use threads, not to replace generators. Oh agreed. For that, it would probably be nice to see greenlets (or something similar) make it into the standard library, but I suspect that's too much of a jump at the moment. I think for me, what we tend to do is prototype something as a stub program, and either turn that into a threaded component or a generator component. If I was to do something halfway I'd probably look at greenlets at the moment, though so far we've not found a use for that. (which is relatively suprising) Any time it's looked useful, generally it's actually meant spin this control out to another smaller component. That's generally then resulted in that spun out component being reusable. For example, probable border case for coroutines is in this: http://kamaelia.svn.sourceforge.net/viewvc/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py?revision=3684view=markup We do do something that implies full coroutine support would be useful - specifically the calls: yield WaitComplete(self.getline(), tag=_getline1) ... yield WaitComplete(self.getline(), tag=getline2) To: 31 def getline(self): 32 control_message = 33 while 1: 34 while not self.anyReady(): 35 self.pause(); 36 yield 1 37 while self.dataReady(control): 38 control_message = self.recv(control) 39 if isinstance(control_message, socketShutdown): 40 self.client_connected = False 41 if self.dataReady(inbox): 42 self.logging_recv_connection() 43 return 44 else: 45 if not self.client_connected : 46 self.breakConnection = True 47 return 48 yield 1 (This is of course faking full co-routines by getting the caller to call something else temporarily) The thing is, if we split that out to an external component, there's potential benefits. *shrug* IMO, code should be written for clarity first, so simply being able to have this as pure co-routines may be useful. However, being able to tag the calls like this: yield WaitComplete(self.getline(), tag=_getline1) ... yield WaitComplete(self.getline(), tag=getline2) Meant that I could run an introspector: 506 Pipeline( 507 PeriodicWakeup(), 508 WakeableIntrospector(), 509 ).activate() And was able to see that some clients were misbehaving at different time points, but also seeing clearly able to see what state the component was waiting/hanging in when dealing with broken clients, and able to add in timeouts trivally. cf: *debug* 'THREADS'['Axon.Microprocess.microprocess_7068_getline1', 'Kamaelia.Chassis.Graphline.Graphline_7062', 'Kamaelia.Chassis.Pipeline.Pipeline_7', ... snip ... 'Kamaelia.Internet.ConnectedSocketAdapter.ConnectedSocketAdapter_7059', 'Kamaelia.Internet.Selector.Selector_11', 'Kamaelia.Internet.TimeOutCSA.ActivityMonitor_7060', 'Kamaelia.Internet.TimeOutCSA.ResettableSender_7061', '__main__.GreylistServer_8', '__main__.PeriodicWakeup_5', '__main__.TCPS_10', '__main__.WakeableIntrospector_6'] Now you could of course design a coroutine system for python to do that and I'd hope if anyone did, they'd think of it being useful to trace the where a co-routine is currently stuck :-) All that said, my personal primary aim for kamaelia is to try and make it into a general toolkit for making concurrency easy natural (as well as efficient) to work with. If full blown coroutines turn out to be part of that c'est le vie :-) I think it's a bit like the arguments around lambda being limited to a single expresson vs a throwaway def. Both are useful, being stuck with one or the other is more limiting than is perhaps ideal. Thanks you. I really do intend most of my posts to be helpful. Likewise :-) Have fun :-) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: fiber(cooperative multi-threading)
Duncan Booth wrote: Unfortunately generators only save a single level of stack-frame, so they are not really a replacement for fibers/coroutines. The OP should perhaps look at Stackless Python or Greenlets. See On the surface of things, the single level aspect *LOOKS* like a problem, but in fact is actually really useful. The reason is because it encourages a generator to be relatively simple and focused encouraging reuse. cf the components listed here: * http://kamaelia.sourceforge.net/Components Also, because they're simple, you can link every component in that list either directly with all the others or via a trivial filtering component. Also the fact that you have to build your own scheduler means you can do the equivalent of saying to the scheduler Don't run me next, run this next. This gives you pretty much the full power of co-routines but allows for very heavy reusability and independence of testing. An example application that makes moderate use of this bounce back coroutines is a greylisting server you can find here: * http://kamaelia.sourceforge.net/KamaeliaGrey I'd (almost) argue that the single level aspect of generators is perhaps their best feature. Michael. -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Developers/ -- http://mail.python.org/mailman/listinfo/python-list
Re: fiber(cooperative multi-threading)
Hi, It just works, but using native Python threads for non-preemptive threading is not cost-effective. Python has generator instead but it seemed to be very restricted for general scripting. I wish I could write nested (generator) functions easily at least. Is there any plan of implementing real (lightweight) fiber in Python? Please take a look at Kamaelia. Generators work extremely well, and are extremely useful to work with: * http://kamaelia.sourceforge.net/Home (I need to update/improve the website...) If you *must* have nested generators (it's rarely useful) you can do that, but just need to have your thing running the generators understand a don't run me next, run this next message. Our message is called WaitComplete, and you can see it used here (in a non-trivial useful system): https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py (I could point at trivial examples, but prefer something real) The shortest overall description is here: * http://kamaelia.sourceforge.net/t/TN-LightTechnicalIntroToKamaelia.pdf Cookbook examples: * http://kamaelia.sourceforge.net/Cookbook List of dozens of reusable (all with each other) components: * http://kamaelia.sourceforge.net/Components How to build your own (non-optimised) core: * http://kamaelia.sourceforge.net/MiniAxon/ Ruby (mini) version: (for comparison :-) https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk/Code/Ruby/miniaxon.rb Experimental process based (as well as thread generator based) version: http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpostnodeid=1196129474 (Going to move to process based components for static segmentation of an application across process boundaries to give explicit multicore support. May move to automatic distribution at some point, but static is an easy win as the above example shows :-) It's not as lightweight as stackless python's microthreads, but it's pretty close. The scaling style is pretty similar as well - cf: http://www.rhonabwy.com/wp/2007/11/13/generator-based-concurrent-programming-in-python/ I also blogged about that here: http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpostnodeid=1195688924 (Someone curious about comparing Kamaelia to stackless. Interesting to see the same scaling curve, but unsurprisingly stackless wins :-) Kamaelia however works with standard python from version 2.2.late onwards :-) ) Interestingly recently discovered that Kamaelia's been slowly reinventing a system the UK's MOD have been using for around 30 years to make concurrent systems easier to build (mascot) :-) Merry Christmas :-) Michael -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Developers/ -- http://mail.python.org/mailman/listinfo/python-list
Re: fiber(cooperative multi-threading)
Duncan Booth wrote: Michael Sparks [EMAIL PROTECTED] wrote: Duncan Booth wrote: Unfortunately generators only save a single level of stack-frame, so they are not really a replacement for fibers/coroutines. The OP should perhaps look at Stackless Python or Greenlets. See On the surface of things, the single level aspect LOOKS like a problem, but in fact is actually really useful. The reason is because it encourages a generator to be relatively simple and focused encouraging reuse. Ah, perhaps Python should similarly limit function call nesting to one level so as to keep things simple and encourage reuse. Bit of a grumpy response? ... and a straw man argument ? Generators can be used to build co-operative multitasking (eg WSGI is implicitly co-op multitasking), whereas function calls are used to build sequential code. I agree for sequential code generators being single level can be rather limiting. However, the thread is about co-operative multitasking. Decades of experience has shown that the best way to build concurrent systems (MASCOT[1], Unix, MPI, Occam, Erlang, even VHDL some mashups) is to have small things loosely connected. Generators encourage making little things which then encourages connecting them together. It seems, to me, to help. (Which is all I said really) [1] Only recently heard about MASCOT - best reference is here: http://async.org.uk/Hugo.Simpson/ (MASCOT is over 30 years old... (fun thing of the week that I've been pointed at for me :-) ) It's the same basic ethos that makes Unix pipelines useful, and is why WSGI is useful. Simple transformational things which work best when they're pipelined together. The fact that you can't trivially nest generators in a way that feels nice[2] IMO naturally encourages making small things, which end up being reusable. [2] I don't personally feel for i in X(): yield i is particularly nice. To be clear - I REALLY didn't like the fact that generators were single layer when I first saw them - it seemed a huge limitation. (Indeed as huge a limitation as only having single level function calls, or only single layer of nesting for namespaces, etc)... I even asked a question [3] on that point before choosing python to write Kamaelia in... The fact that it's been a benefit rather than a problem rather surprised me. [3] http://groups.google.com/group/comp.lang.python/tree/browse_frm/thread/fcd2709952d23e34/8a0b9ba0e3beb108 That's rather different from the recursion limit (which is a limit on function call depth) that really *has* been a pain at times. (Something which tail recursion would really help with) Incidentally, you may note that you helped me back then (which I'm thankful for :-), so you can kinda view this as me reporting back actually, it's turned out to be helpful rather than a pain :-) Took a while to figure that out though ;) Your mileage may vary :) Incidentally, it was probably your response (message 7) in that thread that was responsible for me deciding to see where things could go by trying python :-) Merry Christmas, Michael. (hoping that's a suitably positive response :-) -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Developers/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Passing by reference
[EMAIL PROTECTED] wrote: ... the first element of the list to which x refers is a reference to the new string and back outside foo, the first element of the list to which x refers will be a reference to the new string. I'd rephrase that as: * Both the global context and the inside of foo see the same list * They can therefore both update the list * If a new string is put in the first element of the list, the can both see the same new string. Right? You know you can get python to answer your question - yes? Might be slightly more illuminating than twisting round english... :-) OK, you're passing in a string in a list. You have 2 obvious ways of doing that - either as an argument: def foo(y): y[0] += other print id(y[0]) ... or as a global: (which of course you wouldn't do :-) def bar(): global x x[0] += another print id(x[0]) So let's see what happens. x = [some string] # create container with string x[0] # Check that looks good it does 'some string' id(x[0]) # What's the id of that string?? 3082578144L foo(x)# OK, foo thinks the new string has the following id 3082534160 x[0] # Yep, our x[0] has updated, as expected. 'some string other' id(x[0]) # Not only that the string has the same id. 3082534160L bar() # Update the global var, next line is new id 3082543416 x[0] # Check the value's updated as expected 'some string other another' id(x[0]) # Note that the id is the same as the output from bar 3082543416L Does that perhaps answer your question more precisely ? Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: state machine and a global variable
[EMAIL PROTECTED] wrote: Basically, I agree that often the local state is much more useful. It just seems to me that for some application it's an overkill. Like say, for Turtle [1] (no jokes, please :) or PostScript [2]. Sounds also a bit similar to what happens under the hood in Open GL and some other systems which revolve around display lists. The approach generally taken there is to recognise that you often have a current context which is being operated on. A basic version of this might look like this: file: statemachine_user.py #!/usr/bin/python from statemachine import * context = statemachine() context2 = statemachine() set_context(context) set_state(3) print get_state() set_context(context2) set_state(1) print get_state() set_context(context) print get_state() file: statemachine.py #!/usr/bin/python class statemachine(object): def __init__(self): self.state = None def set_state(self, value): self.state = value def get_state(self): return self.state context = statemachine() def set_context(somecontext): global context context = somecontext def set_state(value): context.set_state(value) def get_state(): return context.get_state() A more interesting example which probably relates closer to your example, and also can be quite useful for interpretting little languages is where the states that get stored are matrices representing transforms and are used to put objects into a 3D space. You might in that circumstance want to use your a statemachine more like this: (this code is untested) class statemachine(object): default = None def __init__(self, **argd): self.__dict__.update(**argd) self.state = self.default def set_state(self, value): self.state = value def get_state(self): return self.state context = [ statemachine() ] # default context def push_context(somecontext): global context context.append( somecontext ) def pop_context(somecontext): global context return context.pop( -1 ) def set_state(value): context[-1].set_state(value) def get_state(): return context[-1].get_state() def get_allstates(): return [ x.get_state() for x in context ] This isn't really quite what the do, but gives a different possible perspective. Michael. -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Developers/ -- http://mail.python.org/mailman/listinfo/python-list
ANN: Axon.STM 1.0.0 (beta) Minimalistic Software Transactional Memory
Hi, I've packaged up the minimal STM discussed over the past couple of days as a standalone package which I've now uploaded Getting it == You can download a beta test version here: http://thwackety.com/Axon.STM-1.0.0.tar.gz Previewing it = You can look at the sourcecode online here: https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Bindings/STM/Axon/STM.py Installing it = ~ tar zxf Axon.STM-1.0.0.tar.gz ~ cd Axon.STM-1.0.0/ ~ sudo python setup.py install What IS it? === Software Transactional Memory (STM) is a technique for allowing multiple threads to share data in such a way that they know when something has gone wrong. It's been used in databases (just called transactions there really) for some time and is also very similar to version control. Indeed, you can think of STM as being like variable level version control. Note: Because this is NOT intended to be persistent, this is not an ACID store because it doesn't support the D - durability across a crash. (after all, we don't save the state to disk) (The other aspects atomicity, consistency isolation are supported though) I've written this to allow a part of Kamaelia to share manage a dictionary of atomic values between threads simply, and as a result this code is also going into mainline Kamaelia. (Specifically into Axon Kamaelia's core) However STM is something that should hopefully be of use to others doing concurrent *things* whether or not they're using kamaelia, hence this stand alone release. This stand alone release should *not* be used alongside mainline Axon yet. (Well you can, as long as you reinstall your Axon over the top, but that's icky :-) Why is it useful? = [ please skip this (or correct me :) if you understand concurrency already :-) ] Why do you need it? Well, in normal code, Global variables are generally shunned because it can make your code a pain to work with and a pain to be certain if it works properly. Even with linear code, you can have 2 bits of code manipulating a structure in surprising ways - but the results are repeatable. Not-properly-managed-shared-data is to threaded systems as not-properly-managed-globals are to normal code. (This code is one way of helping manage shared data) Well, with code where you have multiple threads active, having shared data is like an even nastier version of globals. Why? Well, when you have 2 (or more) running in parallel, the results of breakage can become hard to repeat as two pieces of code race to update values. With STM you make it explicit what the values are you want to update, and only once you're happy with the updates do you publish them back to the shared storage. The neat thing is, if someone else changed things since you last looked, you get told (your commit fails), and you have to redo the work. This may sound like extra work (you have to be prepared to redo the work), but it's nicer than your code breaking :-) The way you get that message is the .commit raises a ConcurrentUpdate exception. Also, it's designed to work happily in code that requires non-blocking usage - which means you may also get a BusyRetry exception under load. If you do, you should as the exception suggests retry the action that you just tried. (With or without restarting the transaction) Apologies if that sounds too noddy :) Docs for it === http://kamaelia.sourceforge.net/STM Using It # Initialising a Store from Axon.STM import Store S = Store() # Single values greeting = S.usevar(hello) print repr(greeting.value) greeting.set(Hello World) greeting.commit() S.dump() # Groups of values D = S.using(account_one, account_two, myaccount) D[account_one].set(50) D[account_two].set(100) D.commit() S.dump() D = S.using(account_one, account_two, myaccount) D[myaccount].set(D[account_one].value+D[account_two].value) D[account_one].set(0) D[account_two].set(0) D.commit() S.dump() License === Take your pick of MPL V1.1, GPL 2.0, LGPL 2.1 :-) Feedback Feedback is very welcome, preferably via email to the Kamaelia List * [EMAIL PROTECTED] Feedback especially regarding bugs and logical errors is particularly welcome. (hopefully there aren't any - but it's always hard to spot your own) Thanks == Many thanks to Fuzzyman, Duncan Booth, John J Lee Sylvain Hellegouarch for feedback whilst I was prototyping this. Best Regards, Michael. -- Michael Sparks, Kamaelia Project http://kamaelia.sourceforge.net/Developers/ http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations.html
Re: Minimalistic Software Transactional Memory
Duncan Booth wrote: Michael Sparks [EMAIL PROTECTED] wrote: I'm interested in writing a simple, minimalistic, non persistent (at this stage) software transactional memory (STM) module. The idea being it should be possible to write such a beast in a way that can be made threadsafe fair easily. For those who don't know, STM is a really fancy way of saying variables with version control (as far as I can tell :-) designed to enable threadsafe shared data. I'm starting with the caveat here that the following code is almost certainly not threadsafe (not put any real thought into that as yet), and I'm interested in any feedback on the following: * Does the API look simple enough? * Are there any glaring mistakes in the code ? (It's always harder to see your own bugs) * What key areas appear least threadsafe, and any general suggestions around that. If I get no feedback I hope this is of interest. Since these things get archived, if you're reading this a month, 6 months, a year or more from now, I'll still be interested in feedback... Unless you really are desperate to reinvent the wheel, have you looked at ZODB? https://launchpad.net/zodb ZODB gives you the transactional model you want. It also gives you persistence, but if you don't want that you can simply connect to a non- persistent store. That seems somewhat overkill for my needs. ZODB's distribution is 3.3MB in size, whereas the system I want a minimalistic, non-persistant[1] transactional memory for is 345K in size. (The Axon part of kamaelia) I'll take a look though. [1] I said at this stage because its something I *may* want in future for some possible usecases, but generally I'm not really very interested in persistence. (At least not where I'm putting this :-) Context: I want thing that use the following class threadsafe, and can think of a few ways of doing this, and STM seems one of the more nicer ways of doing so. (it's clients are currently just generators which makes it safe at the moment) https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Axon/Axon/CoordinatingAssistantTracker.py It's used to provide an ENVironment like facility (similar to os.environ) for generator based Kamaelia components. Thanks for the feedback :-) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Minimalistic Software Transactional Memory
Fuzzyman wrote: STM seems more in keeping with Kamaelia being generally lock-free. STM isn't lock free - it just abstracts the locks away from the 'user'. You still need to lock around committing the transaction. I perhaps phrased what I meant too tersely. Kamaelia isn't lock free either. It's generally lock free. User code however, doesn't ever see locks... (Though threaded components do communicate over thread safe Queues which contain locks internally. Generator components don't use locking) I suppose the reason why I like STM (conceptually) is because it appears to follow the same sort of idea - the user of the code doesn't have to worry about locks - the just have to worry about whether they're using the system correctly or not. :-) Based on this, I think I'm right in thinking then that given the code I posted, that at minimum I need locking here, since it is the critical section. def set(self, key, value): # Attempt to acquire lock if not (self.store[key].version value.version): # Also assuming we have lock self.store[key] = Value(value.version+1, copy.deepcopy(value.value), self, key) value.version= value.version+1 # Release lock else: # Also do this if we can't get access to the lock raise ConcurrentUpdate I know I then have a choice of locking the value (ie a lock specific to self.store[key]) or the self.store as a whole. Depends on how antisocial you want the locking to be I suppose :-) Other threads accessing the data during the transaction should see the old values until the commit is successful. The implementation I'm suggesting means that other threads would see the old values until the try to commit their changes. (I'm after something simple since I'm expecting to deal with simple cases) Many thanks :) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Minimalistic Software Transactional Memory
John J. Lee wrote: Durus might be worth a look too (though I doubt it's suitable for your situation): http://www.mems-exchange.org/software/durus/ The link to their paper about it seems to be broken, but I think it was based somewhat on ZODB, but is simpler (67k tarball :-). Much appreciated. I've downloaded this and it looks more suitable, however it still looks like overkill... After all, I'm just after a simple system for concurrent update of in-memory shared values by multiple threads - how hard can that be ?[1] ;-) :-) [1] Yes, yes, for those who don't know me, I know, I know... :-) Thanks to Fuzzyman's comments and a code review on IRC I think I've got what I think is a minimally sufficient system - though I'll extend to include Fuzzyman's suggestion regarding concurrent update of multiple independent values :-) Code is here for those curious/wanting something similar/similarly lightweight: https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk/Sketches/MPS/Experiments/NewSTM.py I'll package it up independently of the rest of Kamaelia as well probably once I've made the addition noted above :-) Regards, Michael -- http://mail.python.org/mailman/listinfo/python-list
ANN: Axon.STM 1.0.0 (beta) Minimalistic Software Transactional Memory
Hi, I've packaged up the minimal STM discussed over the past couple of days as a standalone package which I've now uploaded Getting it == You can download a beta test version here: http://thwackety.com/Axon.STM-1.0.0.tar.gz Previewing it = You can look at the sourcecode online here: https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Bindings/STM/Axon/STM.py Installing it = ~ tar zxf Axon.STM-1.0.0.tar.gz ~ cd Axon.STM-1.0.0/ ~ sudo python setup.py install What IS it? === Software Transactional Memory (STM) is a technique for allowing multiple threads to share data in such a way that they know when something has gone wrong. It's been used in databases (just called transactions there really) for some time and is also very similar to version control. Indeed, you can think of STM as being like variable level version control. Note: Because this is NOT intended to be persistent, this is not an ACID store because it doesn't support the D - durability across a crash. (after all, we don't save the state to disk) (The other aspects atomicity, consistency isolation are supported though) I've written this to allow a part of Kamaelia to share manage a dictionary of atomic values between threads simply, and as a result this code is also going into mainline Kamaelia. (Specifically into Axon Kamaelia's core) However STM is something that should hopefully be of use to others doing concurrent *things* whether or not they're using kamaelia, hence this stand alone release. This stand alone release should *not* be used alongside mainline Axon yet. (Well you can, as long as you reinstall your Axon over the top, but that's icky :-) Why is it useful? = [ please skip this (or correct me :) if you understand concurrency already :-) ] Why do you need it? Well, in normal code, Global variables are generally shunned because it can make your code a pain to work with and a pain to be certain if it works properly. Even with linear code, you can have 2 bits of code manipulating a structure in surprising ways - but the results are repeatable. Not-properly-managed-shared-data is to threaded systems as not-properly-managed-globals are to normal code. (This code is one way of helping manage shared data) Well, with code where you have multiple threads active, having shared data is like an even nastier version of globals. Why? Well, when you have 2 (or more) running in parallel, the results of breakage can become hard to repeat as two pieces of code race to update values. With STM you make it explicit what the values are you want to update, and only once you're happy with the updates do you publish them back to the shared storage. The neat thing is, if someone else changed things since you last looked, you get told (your commit fails), and you have to redo the work. This may sound like extra work (you have to be prepared to redo the work), but it's nicer than your code breaking :-) The way you get that message is the .commit raises a ConcurrentUpdate exception. Also, it's designed to work happily in code that requires non-blocking usage - which means you may also get a BusyRetry exception under load. If you do, you should as the exception suggests retry the action that you just tried. (With or without restarting the transaction) Apologies if that sounds too noddy :) Docs for it === http://kamaelia.sourceforge.net/STM Using It # Initialising a Store from Axon.STM import Store S = Store() # Single values greeting = S.usevar(hello) print repr(greeting.value) greeting.set(Hello World) greeting.commit() S.dump() # Groups of values D = S.using(account_one, account_two, myaccount) D[account_one].set(50) D[account_two].set(100) D.commit() S.dump() D = S.using(account_one, account_two, myaccount) D[myaccount].set(D[account_one].value+D[account_two].value) D[account_one].set(0) D[account_two].set(0) D.commit() S.dump() License === Take your pick of MPL V1.1, GPL 2.0, LGPL 2.1 :-) Feedback Feedback is very welcome, preferably via email to the Kamaelia List * [EMAIL PROTECTED] Feedback especially regarding bugs and logical errors is particularly welcome. (hopefully there aren't any - but it's always hard to spot your own) Thanks == Many thanks to Fuzzyman, Duncan Booth, John J Lee Sylvain Hellegouarch for feedback whilst I was prototyping this. Best Regards, Michael. -- Michael Sparks, Kamaelia Project http://kamaelia.sourceforge.net/Developers/ http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-list
Minimalistic Software Transactional Memory
Hi,
I'm interested in writing a simple, minimalistic, non persistent (at this
stage) software transactional memory (STM) module. The idea being it should
be possible to write such a beast in a way that can be made threadsafe fair
easily.
For those who don't know, STM is a really fancy way of saying variables
with version control (as far as I can tell :-) designed to enable threadsafe
shared data.
I'm starting with the caveat here that the following code is almost
certainly not threadsafe (not put any real thought into that as yet),
and I'm interested in any feedback on the following:
* Does the API look simple enough?
* Are there any glaring mistakes in the code ? (It's always harder to see
your own bugs)
* What key areas appear least threadsafe, and any general suggestions
around that.
If I get no feedback I hope this is of interest. Since these things get
archived, if you're reading this a month, 6 months, a year or more from
now, I'll still be interested in feedback...
OK, API.
First of all we need to initialise the store:
S = Store()
We then want to get a value from the store such that we can use the value,
and do stuff with it:
greeting = S.using(hello)
Access the value:
print repr(greeting.value)
Update the value:
greeting.set(Hello World)
Commit the value back to the store:
greeting.commit()
If you have concurrent updates of the same value, the following exception
gets thrown:
ConcurrentUpdate
cf:
S = Store()
greeting = S.using(hello)
par = S.using(hello)
greeting.set(Hello World)
par.set(Woo)
greeting.commit()
par.commit()
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 12, in commit
File stdin, line 11, in set
__main__.ConcurrentUpdate
That's pretty much the simplest API I can come up with. (I've tried a few
others but didn't really like them)
The way this works is we have a Store that manages Values. (perhaps a better
name may be variables to avoid clashing with pythons parlance of labels and
values?)
Anyhow, you can ask the store for Value, which it will give you. The Value
knows what it's called and where it's from, so when you tell it to commit,
it can try to do so. The little detail here is you get a /copy/ of the
Value not the stored Value. (This is to avoid accidental concurrent update
of the same actual object)
As a result, Store looks like this:
class Store(object):
def __init__(self):
self.store = {}
def get(self, key):
return self.store[key].clone()
def set(self, key, value):
if not (self.store[key].version value.version):
self.store[key] = Value(value.version+1, value.value, self, key)
value.version= value.version+1
else:
raise ConcurrentUpdate
def using(self, key):
try:
return self.get(key)
except KeyError:
self.store[key] = Value(0, None,self,key)
return self.get(key)
def dump(self):
for k in self.store:
print k, :, self.store[k]
You'll note that the set method is the greatest candidate for any possible
race hazard here - though I can see a possible boundary issue in using.
(I think :-)
Otherwise I think the above code is relatively straightforward. You'll note
that this API allows this:
greeting.set(Hello)
greeting.commit()
greeting.set(Hello World)
greeting.commit()
greeting.set(Hello World. Game)
greeting.commit()
greeting.set(Hello World. Game Over)
greeting.commit()
The other class is value that looks like this:
class Value(object):
def __init__(self, version, value,store,key):
self.version = version
self.value = value
self.store = store
self.key = key
def __repr__(self):
return Value+repr((self.version,self.value,self.store,self.key))
def set(self, value):
self.value = value
def commit(self):
self.store.set(self.key, self)
def clone(self):
return Value(self.version, self.value,self.store,self.key)
To me this looks like a pretty complete minimalistic thing, which does seem
to work OK, but I'm interested in the three points I mention above - if
anyone is willing to comment - specifcally:
* Does the API look simple enough?
* Are there any glaring mistakes in the code ? (It's always harder to see
your own bugs)
* What key areas appear least threadsafe, and any general suggestions
around that.
Full code below.
Many thanks for any comments in advance,
Michael
--
Michael Sparks, Kamaelia Project Lead
http://kamaelia.sourceforge.net/Developers/
http://yeoldeclue.com/blog
#!/usr/bin/python
class ConcurrentUpdate(Exception): pass
class Value(object):
def __init__(self, version, value,store,key):
self.version = version
self.value = value
self.store = store
self.key = key
def
Re: Minimalistic Software Transactional Memory
to be able to detect failure) It's probably useful to know for the more general approach you suggest. Thanks! Michael. -- Michael Sparks, Kamaelia Project Lead http://kamaelia.sourceforge.net/Developers/ http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-list
Multiple Windows in Pygame, using true concurrency in python via Kamaelia
Hi, Just thought some people may be interested to hear that I've recently been looking at adding true concurrency into Kamaelia, by using Paul Boddie's pprocess as the core mechanism to allow us to run multiple Kamaelia systems in the same app. (Since we have thread based, and co-operative generator based components, process based is the next logical step) [1] http://kamaelia.sourceforge.net/Home [2] http://pypi.python.org/pypi/pprocess Anyhow, first proof of concept is to take a _relatively_ non-trivial app and a library hostile to being run in a single process and run it across multiple processes trivially. (I can't test multicpu since I don't have a multicpu machine, but this should work fine with a modern OS) So the app I've chosen is a simple pygame based drawing/doodling app, and to get it to launch 8 pygame windows from 1 app. Since neither SDL nor pygame directly support this it struck me as not just a visible proof of concept, but a useful one too (since pygame rocks! :-) Anyhow, I've written up the code here and posted a screenshot here: http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpostnodeid=1196129474 You'll note 8 pygame windows ... :-) The original base test is here: http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpostnodeid=1196029230 For those curious who don't like clicking on links, the code needed to allow the 8 pygame windows looks like this: class SecondProcessBasedComponent(SimplestProcessComponent): def main(self): from Kamaelia.UI.Pygame.Display import PygameDisplay from Kamaelia.UI.Pygame.MagnaDoodle import MagnaDoodle X=PygameDisplay(width=200,height=200).activate() PygameDisplay.setDisplayService(X) MagnaDoodle().run() yield 1 exchange = SecondProcessBasedComponent().activate() R = [] for _ in xrange(7): R.append(SecondProcessBasedComponent().activate()) So far, so good :-) Incidentally, yes the project is alive and well, but a resource crunch at work means I'm working on this only on my own time at the mo, which means things like releases are having a longer cycle between them than I'd like. Some recent developments include a greylisting proxy server, a ER modelling tool and a (very) simple pygame based logo like tool for teaching programming to small children. The webserver is aiming for WSGI compliance, and the project also had a small number of students working over the summer, assisting with some useful things like AIM/IRC integration, the ability to use Kamaelia systems in non-Kamaelia based scripts/systems, and looking at a sub-component model. (ie non-concurrent systems) Michael. -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Home -- http://mail.python.org/mailman/listinfo/python-list
Greylisting with Kamaelia (was Re: Python North-West meeting - 6 november 18.30)
Giacomo Lacava wrote: New meeting of the Python North-West UK community! This month's talk is: - Michael Sparks on Greylisting with Kamaelia - Just a small note that the slides from this are now up here: http://www.slideshare.net/kamaelian/kamaelia-grey With the main page on the Kamaelia site for it here: http://kamaelia.sourceforge.net/KamaeliaGrey (Which has the slideshare embedded as well for those who prefer one URL :-) One para summary: This presentation was given at Python North West. It explains a complete Kamaelia application for greylisting which was written specifically to eliminate my personal spam problem. It walks through the code as well (though that's best looked at with the code side by side!) Michael. -- http://yeoldeclue.com/blog http://kamaelia.sourceforge.net/Developers/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Python component model
what python already provides you) However you might find that and some of the other things on the following link interesting: http://www2.lifl.fr/~marvie/software.html It's worth bearing in mind though that your description above is one approach for component based design. A survey of different approaches which you might find useful: Thanks for all the link regarding kamaelia. Probably went a bit overboard there :) However I do agree that a visual system is something important, since not everyone thinks the same way. (I can talk about our system till the cows come home, show people code, but when I show them the visual builder, everyone seems to understand). Regards, Michael. -- Michael Sparks, Kamaelia Dust Puppy http://kamaelia.sf.net/ http://yeoldeclue.com/blog -- http://mail.python.org/mailman/listinfo/python-list
Re: Help me use my Dual Core CPU!
Paul Rubin wrote: Ramon Diaz-Uriarte [EMAIL PROTECTED] writes: You might also want to check http://www.lindaspaces.com/products/NWS_overview.html by the guys who invented Linda. Cool, I guess. (The Oz language/Mozart system is a good example of a different and very neat approach to concurrency; somewhat similar Python solutions can be found at Kamaelia and Candygram. Links and other stuff at: I looked at these. Oz/Mozart is a whole nother language, worth examining for its ideas, but the implementation is quite slow. Kamaelia doesn't attempt concurrency at all. Its main idea is to use generators to simulate microthreads. Regarding Kamaelia, that's not been the case for over a year now. We've had threaded components as well as generator based ones since around last July, however their API stablised properly about 4 months back. If you use C extensions that release the GIL and are using an OS that puts threads on different CPUs then you have genuine concurrency. (those are albeit some big caveats, but not uncommon ones in python). Also integrating things as a sub process is as simple instantiating a component that talks to the subprocess over stdin/out to the inbox/outbox model of Kamaelia and then just using it. Something concrete this is useful for: mencoder_options = -ovc lavc -oac mp3lame -ffourcc DX50 -lavcopts acodec=mp3:vbitrate=200:abitrate=128 -vf scale=320:-2 - ...# assume 'encodingfile' is defined above Pipeline( DVB_TuneToChannel(channel=BBC ONE,fromDemuxer=MUX1), UnixProcess(mencoder -o +encodingfile+ +mencoder_options) ).run() On a dual CPU machine that code does indeed both use CPUs (as you'd want and expect). Also whilst we haven't had the chance to implement OS level process based components, that doesn't mean to say we're not interested in them, it's just that 2 people have to focus on something so we've been focussed on building things using the system rather than fleshing out the concurrently. To say we don't attempt implies that we don't want to go down these routes of adding in genuine concurrency. (Which is really why I'm replying - that's not the case - I do want to go down these routes, and it's more man-hours than desire that are the issue). Personally, I'm very much in the camp that says shared data is invariably a bad idea unless you really know what you're doing (largely because it's the most common source of bugs for people where they're trying to do more than one thing at a time). People also generally appear to find writing threadsafe code very hard. (not everyone, just the people who aren't at the top end of the bell curve for writing code that does more than one thing at a time) This is why Kamaelia is message based (ie it's a concious choice in favour), except for certain types of data (where we have a linda-esque type system for more systemic information). The reason for this is to help the average programmer from shooting himself in his own foot (with a 6 CPU-barrelled shotgun :-). In terms of how this is *implemented* however, we have zero copying of data (except to/from threads at the moment) and so data is shared directly, but in a location the user of the system thinks its natural to have handoff to someone else. This approach we find tends to encourage arbitration of access to shared resources, which IMO is a good (defensive) approach to avoiding the problems people have with shared resources. But if it turns out our approach sucks for the average programmer, then that's a bug, so we'd have to work to fix it. And if new approaches are better, we'd welcome implementations since not all problems are screws and not all tools are hammers :-) (as a result I'd also welcome people saying what sucks and why, but preferably based on the system as it is today, not as it was :) Have fun :) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Invitation: Kamaelia Open Space, Brussels, Sept 22 2006
Hello, I'd like to invite you to our first Kamaelia Open Space event. Our theme is Making Software like Lego through intuitive useful concurrency. Perhaps you want to learn to use Kamaelia, or you're already using it, or you're simply interested in code reuse or concurrency being actually useful in everyday code rather than theory. If these match your interests, please come along. WHEN The day after Euro OSCON, and just before Bar Camp Brussels: * Friday 22nd September 2006 - 11am - 5pm WHERE * FOAM Offices, Brussels, Belgium * Koolmijnenkaai 30-34, 1080 Brussels, Belgium * http://fo.am/contact.html * http://tinyurl.com/hsarl (link showing location on multimap) WHAT http://kamaelia.sourceforge.net/Home Kamaelia started as an applied research project, at BBC Research, and has an underlying goal of making software systems easier to maintain and create through the use of intuitive and safe concurrency, in the form of components, whilst maintaining performance. It's being fleshed out by creating systems useful in the real world. This approach turns software into systems similar to a number of systems people may be familiar with : Unix pipes (except non-linear is practical), CSP, hardware (especially HDL's), Occam, K'nex and Lego. Components send messages out via outboxes, receive messages on inbox, have zero copy delivery and messages may be any python object including a component. Practical systems using Kamaelia developed to-date include time-shifting digital TV and a variety of network systems, however components exist simplifying the use of audio, video, pygame Open GL. PURPOSE The purpose of the event is similar to a python sprint. Our aims for this event are to spread what we've learnt with the wider community - for the simple reason we're finding it works for us, and hope it does for you too. This includes - but not exclusively: * To assist people to get started using Kamaelia :-) * Help you copy or tailor our approach to your systems. (eg making our system or approach usable in twisted). * To investigate areas where we can flesh Kamaelia out (eg we have a webserver than can run in a similar manner to seaside, how would you like that fleshed out) * We'd like to assist others using other languages take the lessons we've learnt and apply them in their preferred language. (especially Ruby and C++) * Help attendees integrate their projects with Kamaelia encouraging reuse between projects, bosting the system for everyone (eg integrate pygtk, wxwidgets, pyqt, or other projects you may use or contribute to, building on experience from integrating tk, pygame and open gl) * To share the work done by students during Google's Summer of Code * How to apply this to teaching children to create these systems effectively. (We've noticed that pre-university users of Kamaelia tend to achieve the most using it) * Use of Kamaelia for art and entertainment At the other extreme, we could also like to use the time to share information (if people are interested) on how some of the practical systems we use work since they are likely to be useful to others. Two interesting areas: * Distributed whiteboarding including audio. This is served in an effectively peer to peer manner - extending this to include grid setup (and/or DHT search) would be useful. This is potentially interesting beyond simple whiteboarding since this really forms a simple distributed events backplane * Practical flexible timeshifting either entire channels, and multiplexes or based on programme names Nascent areas perhaps of interest which are include in Kamaelia include a handwriting recogniser (currently at stroke/letter recognition level), graphical systems creation, and a basic open GL toolkit. FORMAT The above list of suggestions for discussion is just that, suggestions. The specific agenda for the event will be decided on the day by those who attend. Since much of what's to be discussed will be new, this will be a cross between a python sprint and open space in format. THANKS TO Finally, many thanks to the very kind people at FOAM - http://fo.am/ for hosting this event, it is very much appreciated. WHAT NEXT If you're interested in coming, please email your interest either to me at: [EMAIL PROTECTED] Or to the kamaelia mailing list: [EMAIL PROTECTED] If you know someone who you think is interested in the theme, could benefit from coming - for example someone interested in making practical concurrency safer and easier to use in future - please don't hesitate to forward this invitation to them. Hope to see you there! Michael -- Michael Sparks, Senior Research Engineer, BBC Research, Technology Group [EMAIL PROTECTED], Kamaelia Project Lead, http://kamaelia.sf.net/ This message may contain personal views which are not the views of the BBC -- http://mail.python.org/mailman/listinfo
Re: beta.python.org content
Kay Schluehr wrote: The new Python site is incredibly boring. Sorry to say this. The old site is/was amateurish but engaged. Now after ~15 years of existence Pythons looks like it wants to be popular among directors of a german job centers. It aims to do everything right but what could be said worse? The text on the beginners page tries to argue with the potential users in a pointless monologue. Who wants to read this text? Who wants to be convinced that Python is *not* slow? Do you stop beating your wife? And where is fun, irony and black humour? Why Python? Python in industry - I see chimneys of 19th century factories, proletarian heroes as well as futuristic hybrid robots superseeding humanity. Python community - a dutch grand-family photo from the beginning of the 20ths century - some ( or all? ) of the members are accidentally looking like Guido, Python in science - snake-like RNA strand. It need not be like this but I wonder about the total lack of personality. +1 This doesn't show the vibrance of to me somewhere like Europython. It doesn't show Oh, and this is incredibly, cool, fun and useful. That to me is what both http://www.turbogears.org/ and http://www.rubyonrails.org/ do show. RonR has 4 very simple engaging points: * Get Excited * Get Started * Get Better * Get Involved Blindly copying something else is rarely IMO a good idea, but having NO personality simply turns people off. I'm not sure how to fix this, but I'd suspect starting with the visual humour style in python might be a good place to start... Put another way, I'd expect *python.com* [1] to look like the new site, and *python.org* to look like something that, well, represents some of the utter (very cool, very diverse) madness of the people involved with python. [1] NB: I **really** wouldn't go to python.com, I REALLY wasn't expecting that (REALLY) Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP Totally my opinion and no-one elses and expecially not the views of the BBC (!) -- http://mail.python.org/mailman/listinfo/python-list
Python Meet Manchester 8th Feb, Lass O'Gowrie
Hi, This is to announce what looks like it might be the first Python Meet Manchester - should be fun! It's happening NEXT week. * Where: Lass O'Gowrie Pub. Directions: http://tinyurl.com/cp3kv * When: 7pm onwards, Wed 8th Feb If you've been to one in London you know pretty much what to expect - a bunch of geeks in a pub talking, well, about python and stuff they find interesting as well as demos and informal talks. If you've used/ written/ found something you think's cool and want to talk about it show it off please do. As it's the first time we've no idea if the Lass O'Gowrie will be ideal or not, and if it isn't there's always the possibility of a Python Crawl to solve that :-) Sign up page (not obligatory, but would be nice if people did - it at least has my ugly mug on as a point of reference for people :-) here: * http://tinyurl.com/a9cry Current topics people have offered up for discussion: * Turbo Gears * Kamaelia (Similar, but not identical, problem space to twisted for people who don't know what it is) * How to write Pythonically. (I'm convinced you need to be dutch for that one myself ;-) Also if you're vaguely interested in python, but are interested in other things like RoR, please come along - the London meets are interesting because of the eclectic mix of people/interests :-) Likewise if there's anything you'd be interested in seeing/hearing about, please come and chat about it :) Please forward this around! See you there! Michael. -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations.html
Re: - E04 - Leadership! Google, Guido van Rossum, PSF
On Wednesday 28 Dec 2005 17:58:33, Robert Kern wrote: ... Sorry to reply to the thread so late in the day, but I noticed (via QOTW :-( ) that Anton got worked up at me suggesting that congratulating someone with a new job was a nice idea (surprised me too - all the Google employees I've met have been very nice people), read the thread (got sad) and then saw this: Who is the man? If Google were to hire you with no experience, would you then have worked for the man? If you want to understand the reference, the most fun recent explanation I've seen is in School of Rock. Jack Black's character explains it far better than I ever could :) Regards, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Python on GP2X (Linux Based Handheld Console)
Hi, I hadn't seen any announcements regarding this, but there's a little device recently released called a GP2X which is a small dual CPU (2x200Mhz) device which runs Linux. Anyway, I thought there might be someone in here interested to hear that python AND pygame have both been ported to it already (not by me). I've also ported some of our code to it (the bouncing cats demo I've shown at a couple of python conferences), and found it really nice to work with so far. The cats bounce at an acceptable (to me) framerate :-) Link for python: http://gp2x-emulation.dcemu.co.uk/python.shtml Link for device: http://www.gp2x.com/product/product.asp Just thought I'd post about this, since I thought someone might find it useful :) Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message contains personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
ANN: Kamaelia 0.3.0 released!
Kamaelia 0.3.0 has been released! Introduction Kamaelia is a networking/communications infrastructure for innovative multimedia systems. Kamaelia uses a component architecture designed to simplify creation and testing of new protocols and large scale media delivery systems. A subset of the system has been tested on series 60 phones. General feedback is welcome either directly, mailing lists or via IRC (#kamaelia, freenode). People are also more than welcome to use the system and suggest improvements not directly related to our specific goals, because we recognise the system can be used in more areas, not just networked multimedia. A diversity of systems built improves the system (for all users of course!) as a whole. What's New Changed? = *NOTE* Kamaelia 0.3.0 requires Axon-1.1.2 to run for some newer components (which has also just been released). Full release notes and change log: * http://kamaelia.sourceforge.net/Kamaelia-0.3.0-ReleaseNotes.html New Examples - 7 new examples including: * Simple reliable multicast based streamer using Ogg Vorbis. * Dirac Player * Dirac encode playback. * Simple bouncing images game. Designed for very small children who are amused by things take beep and react when you press left/right mouse buttons. * Simple example showing how to use the ticker (First developed for showing subtitles). * Demonstration system showing how to use the new software chassis facility in the context of multiple chassis. New Tools, Notable Additions * Visual tool for building Kamaelia pipelines * Tk Support * Video encode, decode and playback. (dirac) New Packages Subsystems These names should provide you with a flavour of the new subsystems that have been added: * Kamaelia.Codec * Kamaelia.Chassis * Kamaelia.File * Kamaelia.UI.Tk * Kamaelia.Internet.Simulate Other Highlights * Software chassis (software backplane will be coming in Kamaelia-NEXT) * Tk integration. (The pipeline builder is a nice example of a tool this enables) * Dirac encoded video decoders and encoders * Support for video playback. (dirac YUV) * Variety of pygame based components, including * Tools for greater control over the pygame surface managed environment * Tools for building simple games. (controlling sprite behaviour for example) * Much richer tools for file reading and writing * Includes re-usable file readers. * More utilities for message filters and splitting of messages * Basic tools for simuluating error conditions and failure rates for delivery of messages (Sufficient for simulating an unstable underlying internet infrastructure). What is Kamaelia? = The project aims to make it easy to build networked multimedia systems (eg audio, video, interactive systems). The result is systems which are naturally componentised. Also, the resulting systems are /naturally concurrent/ allowing quick and fast reuse in the same way as Unix pipelines do. It is designed as a practical toolkit, such that you can build systems such as: * Ogg Vorbis streaming server/client systems (via vorbissimple) * Create Video players streaming systems (for dirac). * With subtitles. * Simple network aware games (via pygame) * Quickly build TCP Multicast based network servers and clients * Presentation tools * A networked audio mixer matrix (think multiple audio sources over network connections mixed and sent on to multiple locations with different mixes) * Look at graph topologies customise the rules of display particle types. Mix and match all of the above. You can also do a lot of this *visually* using the new PipeBuilder application in Tools. Essentially if the system you want to build involves audio or moving pictures, and you want to be able to make the system network aware, then this should be quick and easy to do using Kamaelia. (If it isn't, then a) it's a bug b) needs improving :-) It runs on Linux, Window, Mac OS X with a subset running on Series 60 phones. The basic underlying metaphor of a component us like an office worker with inboxes and outboxes, with deliveries occuring between desks, offices, and depts. The component can thus do work anyway it likes but only communicates with these inboxes and outboxes. Like office workers, components run in parallel, and to achieve this are generally implemented using python generators, but can also used threads. The rationale behind the project is to provide a toolkit enabling the development of new protocols, including streaming, for large scale media delivery. The license essentially allows use in proprietary systems without change, but all changes to the system itself must be shared. Oh, and due to things like the visual editor, the use of pygame in a lot of examples, the use of dirac vorbis, it's quite a
Re: Crypto.Cipher.ARC4, bust or me doing something wrong?
Michael J. Fromberger wrote: ... Since ARC4 is a stream cipher, the keystream changes over time -- with ARC4, after each character enciphered. To decrypt successfully, you need to make sure the decrypting keystream exactly matches the encrypting one. ... from Crypto.Cipher import ARC4 as cipher enc = cipher.new(abcdefgh) dec = cipher.new(abcdefgh) x = enc.encrypt(This is some random text) x \x05o\xd5XH|\xa4\xfc\xf7z\xecd\xe92\xfb\x05rR'\xbf\xc0F\xfc\xde y = dec.decrypt(x) y 'This is some random text' enc.decrypt(x) 'M|[bI\x1ciG6A]\x13Hz\xb0\x19\xca\xf1-\x9a\x1a2\x9e%' I hope this helps clear up your confusion. Hi Michael, Thanks for this, much appreciated. Michael -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Crypto.Cipher.ARC4, bust or me doing something wrong?
Paul Rubin wrote: Michael Sparks [EMAIL PROTECTED] writes: I'm looking at using this library and to familiarise myself writing small tests with each of the ciphers. When I hit Crypto.Cipher.ARC4 I've found that I can't get it to decode what it encodes. This might be a case of PEBKAC, but I'm trying the following: You have to reinitialize the cipher state for decryption, as someone else explained. You also have to make sure that keys are unique and independent for separate messages. Hmm... Thanks for this. For most applications you probably want to think about adding authentication. Indeed - though I'm working bottom up. How a key gets transfered from a to b safely will be another step. Another one will be how to trust that exchange, for how long, and how much, etc. I'm well aware that this is a well trod area though, hence why I'm working bottom up. In general you shouldn't use arc4 unless you know what you're doing. Having looked at how it works from a user perspective, it's fairly inappropriate anyway, due to wanting to work over unreliable channels anyway. What's the application? Components for secure communications and identity confirmation. Use cases: * Signing content multicast by the BBC to indicate that it came from the BBC. (ohh, so many issues :-) * Signing content captured on a mobile device such that we know that it came from that mobile. Specific high level use case of that is to be able to accept contributions from people from arbitrary devices and know who they came from. Perhaps embedded in the essence. * Related to that would be the ability to tag content with rights [1] information, and to know it's not been tampered with. [1] eg Who created it. When? Has it been published/broadcast or not? When? Have additional rights over and above fair use/dealing been granted (eg creative commons attribution license). * Similarly people want the ability to protect data in transit between trusted end points. Some people using the system would probably be looking to build restrictions management as well, but that's largely beyond the scope of our system. Indeed restrictions management would require breaking our system. For these use cases to work, encryption digest tools are clearly an option, and hence having components that support encryption and digest are (to say the least) useful. They're obviously not the only techniques though. Rather than re-inventing wheels I thought I'd pick a library sit down and see how pycrypt's meant to be used before actually going anyway. (Amongst other reasons, this is why I suspected me, rather than the library :-) Given the pycrypt library docs page starts off with: This documentation assumes you have some basic knowledge about the Python language, but not necessarily about cryptography. I do for example know enough about cryptography to know that me devising my own approach is foolhardy. Thus the statement above appealed :) As a result I decided to sit down and learn how to use this to form some basic components for encryption/decryption. Given another part early in the docs was A central goal of the author's has been to provide a simple, consistent interface for similar classes of algorithms. I decided to start off with sketches implementing minimal examples for each digest and cipher. I'm now working through the Public Key examples. FWIW, I'm well aware how easy it is to get cipher/digest/etc based security/id systems wrong. I'm really starting with pycrypt because it looked simple enough, low level enough and self contained enough to act as a base for working with existing more complex systems. I supsect we'll end up looking at or wrapping some other library instead/as well, but it struck me as a nice starting point. Anyway, once I've gone through all of the existing digests/ciphers/PK ciphers, I'll post the snippets up on our site as raw examples for pycrypto, which will hopefully be a) correct usage b) be useful to others. Thanks for the comments, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Free seminar on domain-specific modeling
Martijn Iseger wrote: ... I believe the point being made by the organization is that during computing history the most successful shifts in productivity were achieved by similar shifts in raising the abstraction level on which developers specify solutions. The alternate point is that during computing history, many, many, many promises were made for many, many, many, technologies based on the same principle of raising the abstraction level. Many, many, many of those technologies promised much and failed to deliver on their claims when used beyond the people inventing/using those technologies. Furthermore, virtually all of them get marketed as being the next big thing that Will Change The World. As a result anyone marketing an idea in this way meets skepticism. From my perspective your site talks a lot about general ideas but has little on specifics. One thing is relatively clear - your approach appears to include a graphical approach to systems building. Personally I suspect that the fact people are able to engage other parts of their brain when building these systems beyond linguistic is the real reason you see benefits, rather than actually the specific thing that led to the visual approach being possible. Maybe your approach will change the world. Maybe it won't. If it's better and it does, good. If it's not better and it does, that's a lot of effort for no gain. Unfortunately that latter point is a common result. (On a sad note it looks like you're reinvented how hardware is designed and made, but not made the intuitive leap :-/ ) Best Regards, Michael. -- This message (and any attachments) contains personal views which are not the views of the BBC. -- http://mail.python.org/mailman/listinfo/python-list
Re: Crypto.Cipher.ARC4, bust or me doing something wrong?
Paul Rubin wrote: Michael Sparks [EMAIL PROTECTED] writes: Rather than re-inventing wheels I thought I'd pick a library sit down and see how pycrypt's meant to be used before actually going anyway. (Amongst other reasons, this is why I suspected me, rather than the library :-) Pycrypt doesn't operate at anything like the level you need. It just gives you low level cipher primitives. You need higher level protocols. Agreed. As I say I'm building up systems slowly on this front, and part of the aim will be to show how outright dumb it is to build these things without a decent understanding of the underlying tech. Sadly there's many people (where I work) who don't understand how easy it is to get these things wrong, and unfortunately practical demonstration is one of the few languages people listen to :-/ This is where I border on being unprofessional so I'll be quiet at that stage :-(( On a more positive note, as I say I picked something simple initially, and have no intention at this stage of it being the primary approach. FWIW, I'm well aware how easy it is to get cipher/digest/etc based security/id systems wrong. I'm really starting with pycrypt because it looked simple enough, low level enough and self contained enough to act as a base for working with existing more complex systems. Do yourself a favor and stick to something standard like TLS, rather than cook up your own protocol. There are some Python wrappers for OpenSSL or GNU TLS, for example. If those operated over multicast then they'd be an option... However... Incidentally our primary use is to indicate *integrity* of data not to prevent content being viewed. It's also obviously susceptible to attack in many ways. Again, I'd say more offline about this - Usenet is just the wrong place for it. Anyway, once I've gone through all of the existing digests/ciphers/PK ciphers, I'll post the snippets up on our site as raw examples for pycrypto, which will hopefully be a) correct usage b) be useful to others. You really need to know a lot more than it sounds like you know, *shrug* It's not a matter of what I don't know it's a matter of showing how things can be used, and I want to use it (internally) to demonstrate what goes wrong when people try to design their own systems naively. One way to learn is to build. to have any chance of getting fancy protocol designs correct. FWIW, I have little intention of inventing something new (in terms of protocols) /if/ possible. What these components will be used for is to demonstrate how they work inside, and what goes wrong if people try to reinvent wheels. Quite frankly we *should* be using a crypto professional, however that's currently not a realistic option :-((( As a result currently it's up to me to demonstrate here the risks involved in creating your own system. http://www.cs.ucdavis.edu/~rogaway/classes/227/spring05/book/main.pdf is a textbook that will show you how to do this, or at least give you an idea of what you're dealing with. Watch out, it is rather theoretical. That's fine/great :) - thanks for the reference :-) FWIW, I *do* have a good idea of what I'm dealing with, //which is why// I'm pessimistic about getting it right. I might have more experience with these things than those I'm working with, however I am very aware of my (current :) limits. Hopefully I'll be able to use these tools educate those around me why rolling your own is an idea fraught with issues. Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Crypto.Cipher.ARC4, bust or me doing something wrong?
Hi, I suspect this is a bug with AMK's Crypto package from http://www.amk.ca/python/code/crypto , but want to check to see if I'm being dumb before posting a bug report. I'm looking at using this library and to familiarise myself writing small tests with each of the ciphers. When I hit Crypto.Cipher.ARC4 I've found that I can't get it to decode what it encodes. This might be a case of PEBKAC, but I'm trying the following: from Crypto.Cipher import ARC4 as cipher key = obj = cipher.new(key) obj.encrypt(This is some random text) ')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX' X=_ X ')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX' obj.decrypt(X) '\x87\xe1\x83\xc1\x93\xdb\xed\x93U\xe4_\x92}\x9f\xdb\x84Y\xa3\xd4b\x9eHu~' Clearly this decode doesn't match the encode. Me being dumb or bug? Any comments welcome :) Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Crypto.Cipher.ARC4, bust or me doing something wrong?
Jp Calderone wrote: On Tue, 20 Sep 2005 16:08:19 +0100, Michael Sparks [EMAIL PROTECTED] wrote: Hi, I suspect this is a bug with AMK's Crypto package from http://www.amk.ca/python/code/crypto , but want to check to see if I'm being dumb before posting a bug report. I'm looking at using this library and to familiarise myself writing small tests with each of the ciphers. When I hit Crypto.Cipher.ARC4 I've found that I can't get it to decode what it encodes. This might be a case of PEBKAC, but I'm trying the following: from Crypto.Cipher import ARC4 as cipher key = obj = cipher.new(key) obj.encrypt(This is some random text) ')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX' X=_ X ')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX' obj.decrypt(X) '\x87\xe1\x83\xc1\x93\xdb\xed\x93U\xe4_\x92}\x9f\xdb\x84Y\xa3\xd4b\x9eHu~' Clearly this decode doesn't match the encode. Me being dumb or bug? Any comments welcome :) You need two ARC4 instances. Performing any operation alters the internal state (as it is a stream cipher), which is why your bytes did not come out intact. Ahh. That makes perfect sense. (I thought it must be me missing something) Many thanks! :-) Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Release of Shed Skin 0.0.2: Easy Windows/OSX Installation
Mark Dufour wrote: Shed Skin is an experimental Python-to-C++ compiler. Along with GNU/Linux, version 0.0.2 should now also install easily under Windows 2000/XP and OSX. Please give it a try and let me know if there are still some problems. ss.py writes a make file, but unfortunately doesn't detect correctly whether you need libdl linked in or not. As a result the generated Makefile for the test includes this for me: MAINOBJ=test.o test: $(MAINOBJ) /home/zathras/Documents/shedskin-0.0.2/libss.a $(CC) $(CCFLAGS) $(MAINOBJ) /home/zathras/Documents/shedskin-0.0.2/libss.a -lgc -o test Whereas it would need to be the following because I'm under linux: MAINOBJ=test.o test: $(MAINOBJ) /home/michaels/Documents/shedskin-0.0.2/libss.a $(CC) $(CCFLAGS) $(MAINOBJ) /home/michaels/Documents/shedskin-0.0.2/libss.a -lgc -ldl -o test (The reason I'm not suggesting just throwing in -ldl is because that will then break on other platforms... You need to detect whether it's needed or not for the platform it's currently compiling for) Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Software bugs aren't inevitable
Giles Brown wrote: Michael Sparks wrote: The problem that these sorts of approaches don't address is the simple fact that simple creating a formal spec and implementing it, even if you manage to create a way of automating the test suite from the spec *doesn't guarantee that it will do the right thing*. snip As a result I'd say that the subject Software bugs aren't inevitable is not true. I think you can argue (I would) that any behaviour that is in the specification this isn't right is not a software bug, but a specification error. To a user there is no difference. If the software doesn't do what they wanted it to do/asked for, then to *them* the person who is accepting the software it's a bug. It doesn't matter to them what caused it - be it a buffer overflow, a misunderstanding of a language feature or an incorrectly written formal spec, it's still a bug to them. I'm actually a big fan of formal specification (specifically VDM), but that doesn't stop me realising that it's not a cure-all, and it's also not an excuse - if code doesn't do what it was asked to do, it's bust. Regards, MIchael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Roguelike programmers needed
Robert Kern wrote: Thomas Jollans wrote: what exactly is RPG/roguelike etc ? (what debian package provides an example?) Google is your friend. Often a fair answer, but I'd suggest that the question was fair, especially given the OP was seeking help :-) After all, I read the subject line and simply assumed they were after programmers with roguish qualities. Perhaps to work in the newly formed IT division of the Crimson Permanent Assurance Company. After all, don't forget - #It's Fun to Charter, and Accountant...# ;-) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Software bugs aren't inevitable
Paddy wrote: A work colleague circulated this interesting article about reducing software bugs by orders of magnitude: The problem that these sorts of approaches don't address is the simple fact that simple creating a formal spec and implementing it, even if you manage to create a way of automating the test suite from the spec *doesn't guarantee that it will do the right thing*. The hidden assumption is that the formal spec will be correct. If it isn't then you have the same problems as before. Sure you might be able to reduce the number of bugs, but you can be certain of one thing, given: * Customer has a need * Customer communicates this to Supplier * Supplier translates needs to spec * Supplier translates spec back to english * Customer agrees spec This involves human communication, and misunderstandings happen all the time then. Sure it can be easier to detect errors at this stage, but you can't guarantee that it will do so. You can almost guarantee though that people will misunderstand each other from time to time. A purely engineering approach to dealing with correctness cannot guarantee that misunderstandings won't occur. Those misunderstandings translate into bugs, no matter what approach you use. It's why I find XP and agile approaches interesting, they're often more than just engineering. As a result I'd say that the subject Software bugs aren't inevitable is not true. Regards, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Software bugs aren't inevitable
Steven D'Aprano wrote: On Thu, 15 Sep 2005 18:07:28 +0100, phil hunt wrote: On Thu, 15 Sep 2005 21:56:06 +1000, Steven D'Aprano [EMAIL PROTECTED] wrote: Are you saying that the recursion done by serious languages is a fake? That it is actually implemented behind the scenes by iteration? It seems to me that if recursion and iteration produce the exact same machine code, the argument for preferring recursion over iteration is gutted. It seems to me that if a high level language and assembler produce the exact same machine code, the argument for preferring high level languages is gutted. Do you see the fallacy in your statement now? Ah, yes, you got me on that one. But there is a difference: writing assembly is *hard*, which is why we prefer not to do it. Are you suggesting that functional programming is significantly easier to do than declarative? But there is a difference: writing assembly is *hard*, which is why we prefer not to do it. Are you suggesting that object oriented programming is significantly easier to do than old style imperative? (sorry, couldn't resist) FWIW, IMO once you've learnt functional programming's idioms it certainly can be easier and more natural. The problem is the tools that make things like recursion efficient aren't available normally in mainstream languages meaning that most people simply don't get the practice. Essentially it's about expressiveness. Think of it this way - we normally write left to write, however some languages read up and down. Neither is inherently better or easier than the other, but for some things *may* be more expressive. If you think about it being about choosing the most clear/expressive way to describe an algorithm, the argument may become clearer. After all, the recursive definition of some things is clearer than the non-recursive. Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: global interpreter lock
Stephen Thorne wrote: On 15/09/05, Michael Sparks [EMAIL PROTECTED] wrote: At the moment, one option that springs to mind is this: yield WaitDataAvailable(inbox) Twisted supports this. help(twisted.internet.defer.waitForDeferred) Thanks for this. I'll take a look and either we'll use that or we'll use something that maps cleanly. (Reason for pause is because running on mobiles is important to us.) Thanks for the example too :) Best Regards, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: global interpreter lock
Michele Simionato wrote:
It looks like I am reinventing Twisted and/or Kamaelia.
If it's /fun/ , is that a problem ? ;) (Interesting implementation BTW :)
FWIW, I've about a year ago it wasn't clear if we would be able to release
our stuff, so as part of a presentation I included a minimalistic decorator
based version of our system that has some similarities to yours. (The idea
was then at least the ideas had been shared, if not the main code - which
had approval)
Posted below in case it's of interest:
import copy
def wrapgenerator(bases=object, **attrs):
def decorate(func):
class statefulgenerator(bases):
__doc__ = func.__doc__
def __init__(self,*args):
super(statefulgenerator, self) __init__(*args)
self.func=func(self,*args)
for k in attrs.keys():
self.__dict__[k] = copy.deepcopy(attrs[k])
self.next=self.__iter__().next
def __iter__(self): return iter(self.func)
return statefulgenerator
return decorate
class com(object):
def __init__(_, *args):
# Default queues
_.queues = {inbox:[],control:[],
outbox:[], signal:[]}
def send(_,box,obj): _.queues[box].append(obj)
def dataReady(_,box): return len(_.queues[box])0
def recv(_, box): # NB. Exceptions aren't caught
X=_.queues[box][0]
del _.queues[box][0]
return X
A sample component written using this approach then looks like this:
@wrapgenerator(com)
def forwarder(self):
Simple data forwarding generator
while 1:
if self.dataReady(inbox):
self.send(outbox,self.recv(inbox))
elif self.dataReady(control):
if self.recv(control) == shutdown:
break
yield 1
self.send(signal,shutdown)
yield 0
Since we're not actualy using this approach, there's likely to be border
issues here. I'm not actually sure I like this particular approach, but it
was an interesting experiment.
Best Regards,
Michael.
--
http://mail.python.org/mailman/listinfo/python-list
Re: global interpreter lock
Paul Rubin wrote: ... I don't see how generators substitute for microthreads. In your example from another post: I've done some digging and found what you mean by microthreads - specifically I suspect you're referring to the microthreads package for stackless? (I tend to view an activated generator as having a thread of control, and since it's not a true thread, but is similar, I tend to view that as a microthread. However your term and mine don't co-incide, and it appears to cause confusion, so I'll switch my definition to match yours, given the microthreads package, etc) You're right, generators aren't a substitue for microthreads. However I do see them as being a useful alternative to microthreads. Indeed the fact that you're limited to a single stack frame I think has actually helped our architecture. The reason I say this is because it naturally encourages small components which are highly focussed in what they do. For example, when I was originally looking at how to wrap network handling up, it was logical to want to do this: [ writing something probably implementable using greenlets, but definitely pseudocode ] @Nestedgenerator def runProtocol(...) while: data = get_data_from_connection( ... ) # Assume non-blocking socket def get_data_from_connection(...) try: data = sock.recv() return data except ... : Yield(WaitSocketDataReady(sock)) except ... : return failure Of something - you get the idea (the above code is naff, but that's because it's late here) - the operation that would block normally you yield inside until given a message. The thing about this is that we wouldn't have resulted in the structure we do have - which is to have components for dealing with connected sockets, listening sockets and so on. We've been able to reuse the connected socket code between systems much more cleanly that we would have done (I suspect) than if we'd been able to nest yields (as I once asked about here) or have true co-routines. At some point it would be interesing to rewrite our entire system based on greenlets and see if that works out with more or less reuse. (And more or less ability to make code more parallel or not) [re-arranging order slightly of comments ] class encoder(component): def __init__(self, **args): self.encoder = unbreakable_encryption.encoder(**args) def main(self): while 1: if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) yield 1 ... In that particular example, the yield is only at the end, so the generator isn't doing anything that an ordinary function closure couldn't: def main(self): def run_event(): if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) return run_event Indeed, in particular we can currently rewrite that particular example as: class encoder(component): def __init__(self, **args): self.encoder = unbreakable_encryption.encoder(**args) def mainLoop(self): if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) return 1 That's a bad example though. A more useful example is probably something more like this: class Multicast_sender(Axon.Component.component): def __init__(self, local_addr, local_port, remote_addr, remote_port): super(Multicast_sender, self).__init__() self.local_addr = local_addr self.local_port = local_port self.remote_addr = remote_addr self.remote_port = remote_port def main(self): sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP) sock.bind((self.local_addr,self.local_port)) sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10) while 1: if self.dataReady(inbox): data = self.recv() l = sock.sendto(data, (self.remote_addr,self.remote_port) ); yield 1 With a bit of fun with decorators, that can actually be collapsed into something more like: @component def Multicast_sender(self, local_addr, local_port, remote_addr, remote_port): sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP) sock.bind((self.local_addr,self.local_port)) sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10) while 1: if self.dataReady(inbox): data = self.recv() l = sock.sendto(data, (self.remote_addr,self.remote_port) ); yield 1 You've got the main method creating a generator that has its own event loop that yields after each event it processes.
Re: global interpreter lock
[ Second time lucky... ] Paul Rubin wrote: ... I don't see how generators substitute for microthreads. In your example from another post: I've done some digging and found what you mean by microthreads - specifically I suspect you're referring to the microthreads package for stackless? (I tend to view an activated generator as having a thread of control, and since it's not a true thread, but is similar, I tend to view that as a microthread. However your term and mine don't co-incide, and it appears to cause confusion, so I'll switch my definition to match yours, given the microthreads package, etc) The reason I say this is because it naturally encourages small components which are highly focussed in what they do. For example, when I was originally looking at how to wrap network handling up, it was logical to want to do this: [ writing something probably implementable using greenlets, but definitely pseudocode ] @Nestedgenerator def runProtocol(...) while: data = get_data_from_connection( ... ) # Assume non-blocking socket def get_data_from_connection(...) try: data = sock.recv() return data except ... : Yield(WaitSocketDataReady(sock)) except ... : return failure Of something - you get the idea (the above code is naff, but that's because it's late here) - the operation that would block normally you yield inside until given a message. The thing about this is that we wouldn't have resulted in the structure we do have - which is to have components for dealing with connected sockets, listening sockets and so on. We've been able to reuse the connected socket code between systems much more cleanly that we would have done (I suspect) than if we'd been able to nest yields (as I once asked about here) or have true co-routines. At some point it would be interesing to rewrite our entire system based on greenlets and see if that works out with more or less reuse. (And more or less ability to make code more parallel or not) [re-arranging order slightly of comments ] class encoder(component): def __init__(self, **args): self.encoder = unbreakable_encryption.encoder(**args) def main(self): while 1: if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) yield 1 ... In that particular example, the yield is only at the end, so the generator isn't doing anything that an ordinary function closure couldn't: def main(self): def run_event(): if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) return run_event Indeed, in particular we can currently rewrite that particular example as: class encoder(component): def __init__(self, **args): self.encoder = unbreakable_encryption.encoder(**args) def mainLoop(self): if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) return 1 And that will work today. (We have a 3 callback form available for people who aren't very au fait with generators, or are just more comfortable with callbacks) That's a bad example though. A more useful example is probably something more like this: (changed example from accidental early post) ... center = list(self.rect.center) self.image = self.original current = self.image scale = 1.0 angle = 1 pos = center while 1: self.image = current if self.dataReady(imaging): self.image = self.recv(imaging) current = self.image if self.dataReady(scaler): # Scaling scale = self.recv(scaler) w,h = self.image.get_size() self.image = pygame.transform.scale(self.image, (w*scale, h*scale)) if self.dataReady(rotator): angle = self.recv(rotator) # Rotation self.image = pygame.transform.rotate(self.image, angle) if self.dataReady(translation): # Translation pos = self.recv(translation) self.rect = self.image.get_rect() self.rect.center = pos yield 1 (this code is from Kamaelia.UI.Pygame.BasicSprite) Can it be transformed to something event based? Yes of course. Is it clear what's happening though? I would say yes. Currently we encourage the user to look to see if data is ready before taking it, simply because it's the simplest interface that we can guarantee consistency with. For example, currently the exception based equivalent would be: try: pos = self.recv(translation) except IndexError: pass Which
Python Bindings for Dirac, (sorta announcement)
Hi, We're in the process of creating python bindings for Dirac. We currently have /decode/ of dirac functioning nicely, so I've packaged up the bindings separately from the rest of the Kamaelia project for those that are interested and would want a play. (Encoding will naturally follow next) To build the bindings, you need: * Pyrex (0.9.3) installed: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ * Dirac (0.5.3) installed, from source: http://dirac.sf.net/ (Also, includes should be living in /usr/local/include/dirac) A sample video player, entirely written in python/python extensions (eg pygame) for playing back dirac is in ... /Code/Python/Kamaelia/Examples/example10 ... in Kamaelia's tree in the CVS head. (I'd point at view CVS, but it hasn't caught up yet) What is Dirac? == Not my project, but colleagues', so description from project page: Dirac is a general-purpose video codec aimed at resolutions from QCIF (180x144) to HDTV (1920x1080) progressive or interlaced. It uses wavelets, motion compensation and arithmetic coding and aims to be competitive with other state of the art codecs. How Do I Get Video To Decode? = The easiest way to get some dirac video is to use ffmpeg to convert a file to yuv and then the dirac encoder. eg: # ffmpeg -i file_from_digital_camera.avi rawvideo.yuv ... Stream #0.0: Video: mjpeg, 320x240, 16.10 fps ... Useful line from ffmpeg's output pulled out. (in this case the avi file above came from a cheap digital camera's video mode) # dirac_encoder -qf 5 -width 320 -height 240 -fr 16 rawvideo myvideo The resulting myvideo.drc that the above line would create is what the decoder can handle. The -width -height flags should be obvious, and came from the output from ffmpeg. The -fr flag is the frame rate (fps), again from ffmpeg. The -qf is a quality factor. Bigger is better quality, but bigger also requires beefier systems for decoding dirac. The above settings result in something decodable using dirac using a 1.6Ghz machine. If you have a beefier system you can use a higher quality factor, if you have a less beefy system, a lower quality factor is preferable. Caveats === Finally, I'm mentioning this for people who'd want to play/tinker - it's not really suitable for (say) someone wanting to convert all their home videos to yet, largely because dirac's bitstream isn't yet finalised. (If you don't understand this, this might be a hint to stay away, if you do understand this you'll hopefully understand why I'm giving this warning) Also, the API and usage are also subject to change. For reference an example file encoded with the above settings: * 2,694,380 bytes before * 293,579 bytes after To my untrained eye there's not a /huge/ amount of difference in quality, though there *is* a drop in quality (since that is a rather drastic drop). That's ~ 10 seconds of video. Right now the easiest way to play with this is to do a cvs checkout of Kamaelia's CVS head, install and play with example10 and tinker with the components used, but the bindings aren't tied to Kamaelia and could be used with other things. (We just happened to write them because we need them) If anyone's interesting in integrating these with other python libraries we'd be interested in chatting :-) The reason I say this is sorta an announcement is because I'm really curious as to whether people think the API is sensible or not, but I'll skip spammin the group with those details and ask anyone prepared to take a look download the bindings from: http://tinyurl.com/arbk2 . Best Regards, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: First release of Shed Skin, a Python-to-C++ compiler.
Mark Dufour wrote:
With this initial release, I hope to attract other people to help me
locate remaining problems,
Well, you did say you want help with locating problems. One problem with
this is it doesn't build...
If I try and build (following your instructions), I get presented with a
whole slew of build errors - knock on errors from the first few:
In file included from builtin_.cpp:1:
builtin_.hpp:4:29: gc/gc_allocator.h: No such file or directory
builtin_.hpp:5:23: gc/gc_cpp.h: No such file or directory
In file included from builtin_.cpp:1:
builtin_.hpp:89: error: syntax error before `{' token
builtin_.hpp:93: error: virtual outside class declaration
Which C++ libraries are you dependent on? (Stating this would be really
useful, along with specific versions and if possible where you got them :)
For reference, I'm building this on SuSE 9.3, under which I also have
boehm-gc-3.3.5-5 installed. I suspect you're using the same gc library
(having downloaded libgc from sourceforge and finding the includes don't
match the above include names) but a different version. For reference this
version/distribution of boehm-gc has the following file structure:
/usr/include/gc.h
/usr/include/gc_backptr.h
/usr/include/gc_config_macros.h
/usr/include/gc_cpp.h
/usr/include/gc_local_alloc.h
/usr/include/gc_pthread_redirects.h
/usr/lib/libgc.a
/usr/lib/libgc.la
/usr/lib/libgc.so
/usr/lib/libgc.so.1
/usr/lib/libgc.so.1.0.1
It's specifically the gc_cpp.h file that makes me suspect it's the same gc.
Regards,
Michael.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python versus Perl
Terry Reedy wrote: [...] I am being picky because various people have claimed that Python suffers in popularity because it is known as an 'interpreted language'. So maybe advocates should be more careful than we have been to not reinforce the misunderstanding. I sometimes wonder if it might help people understand the situation if people described as interpreted in the same way Java is (However I think that risks confusing things since python doesn't generally come with a JIT subsystem, yet). That said, if you do describe it that way, it'd be more accurate to describe the python binary as a compiler/runtime rather than interpreter since it'd be more accurate. After all: $ python somefile.py Is very close to being the same as: $ javac somefile.java $ java somefile.class It strikes me as ironic that python would probably gain more credibility with some circles if it had two binaries like this, even though it'd be a step backwards from a usability perspective :-) Personally I agree that any language that is described as interpreted has an image issue. However I'm not sure who's problem that is - some people claim it's Python's problem, however personally I'd view as a problem for the people who buy into interpretted bad, compiled good argument. After all, they're the ones limiting themselves, and missing out on a whole class of languages (of which python is just one of course) ! Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: First release of Shed Skin, a Python-to-C++ compiler.
Paul Boddie wrote: Michael Sparks wrote: Well, you did say you want help with locating problems. One problem with this is it doesn't build... I found that I needed both the libgc and libgc-dev packages for my Kubuntu distribution - installing them fixed the include issues that you observed - and it does appear to be the Boehm-Demers-Weiser GC library, yes. The only other issue I observed was the importing of the profile and pstats modules which don't exist on my system, but those imports seemed to be redundant and could be commented out anyway. Mark's also let me know this. Part of the problem is the version in SuSE 9.3 of the GC used is ancient - it should be version 6.5 onwards. Also for people compiling from source you (at minimum) should be using the configure line along the lines of: ./configure --enable-cplusplus If you don't, you get build problems because one of the needed libraries isn't built by default. I started off with the obvious hello world type program : print GAME OVER Which compiled cleanly and worked as expected. I then read Mark's short paper linked from his blog Efficient Implementation of Modern Imperative Languages; Application to Python, and got concerned by the comments: We have decided not to investigate two types of features: [...snip...]; and those features that may be turned off without affecting correct programs, e.g. array bounds checking, and exceptions That set some alarm bells ringing, largely because LBYL being deprecated by many people in favour of exceptions based code. (And more to the point, widely used as a result) As a result, I tried a trivial, but obvious program that should have clear behaviour: x = [] print GAME OVER x.append(5) print x[0] try: print x[1] print This should never be seen... except IndexError: print It's OK, we caught it... This compiles, but unfortunately has the following behaviour: GAME OVER 5 0 This should never be seen... It's OK, we caught it... Obviously, neither the 0 nor the message following should have been displayed. It's a pity that this assumption was made, but given the short time the project's been going I can understand it, hopefully Mark will continue towards greater python compliance :) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Video display, frame rate 640x480 @ 30fps achievable?
Guenter wrote: Michael Sparks schrieb: Yes. Co-incidentally we've been looking at video playback this week as well. We've been using Pygame with an Overlay surface, and it works fairly well. I guess Pygame was more suitable overall for your application? It's not really that, it's just that it was the easiest most obvious (and we've found portable) approach to getting the video onscreen. You create an overlay object and simply throw decoded YUV data at in and don't worry too much. We can generally mix and match UI's happily at the moment with our work, so we can have tkinter and pygame side by side for example. I would just be interested whether you have considered using PyMedia? We did look at it, but found sufficient numbers of problems to decide that it might be worth coming back to at a later point in time. Also we kept hitting issues on 64-bit machines which didn't really help. API changes also invalidated some of the examples causing confusion (I wasn't the person directly working on it). That said I've heard enough people say enough good things about it to suggest that don't take our experience as the best example - I suspect it's far from the best example. (I wish we had more time to try and help fix it!) (We've decided to wrap dirac instead for now) Initially we're testing with simple IYUV raw video data, and it's a good idea to use a modern video card supported by your OS, but other than that we've not had problems. If you're interested in code, let us know :-) Thanks for the offer. If it is getting serious and I need some jump start I might come back to you about the code. No problem. Michael -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: Video display, frame rate 640x480 @ 30fps achievable?
Guenter wrote: I need to develop an application that displays video 640x480 16-bit per pixel with 30 fps it is possible to achieve that frame rate and still have some resources for other processing left? Yes. Co-incidentally we've been looking at video playback this week as well. We've been using Pygame with an Overlay surface, and it works fairly well. Initially we're testing with simple IYUV raw video data, and it's a good idea to use a modern video card supported by your OS, but other than that we've not had problems. If you're interested in code, let us know :-) If you end up using the framebuffer device in linux under xorg's X11 (not exactly an ideal setup for video playback anyway!) there is a little oddity that we found, due to SDL underneath, but we've been able to work around that. The system I'm working on normally has that issue, and as a result isn't accelerated and is a 1.6Ghz machine, but can do standard defintion video (720x576) playback happily at 30fps. Normally though most people can offload this to their graphics card so there shouldn't be issues on your 1Ghz machine. Clearly if you're interested in driving that you need to decode some data source to IYUV, but that's a different issue. People have embedded pygame in wxPython before (I don't know how though, I've only seen screenshots of the integration) so although you said wxPython and might've discounted pygame based on that, it might still be suitable for you. Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: reading the last line of a file
Xah Lee wrote: isn't there a way to implement tail in python with the same class of performance? how's tail implemented?: Those crazy open source developers have an implementation here: http://cvs.sourceforge.net/viewcvs.py/mkcdrec/mkcdrec/busybox-0.60.5/Attic/tail.c?rev=1.1view=markup It's not documented. It's pretty short though, I'm sure you won't have any problems strikeripping it to pieces/strike understanding it. If it is a problem, a more documented version here (however I'm sorry, this also by those open source/free software people you love to attack): http://www.koders.com/c/fid8DEE98A42C35A1346FA89C328CC3BF94E25CF377.aspx If you want something not by open source crazies (as you like to refer to us), you may prefer this link: http://minnie.tuhs.org/UnixTree/V7/usr/src/cmd/tail.c.html However that's even less documented (despite being not open source, just visible), and more obfuscated. As a compromise, you could always look at plan 9's implementation here: * http://swtch.com/usr/local/plan9/src/cmd/tail.c If you're after alternatives, you could always try here: (where all these links came from) http://www.google.com/search?q=tail.c Fairly informative set of links provided by that highly non-obvious search Hope that helps, Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python versus Perl
Dieter Vanderelst wrote: Dear all, I'm currently comparing Python versus Perl to use in a project that involved a lot of text processing. I'm trying to determine what the most efficient language would be for our purposes. I have to admit that, although I'm very familiar with Python, I'm complete Perl noob (and I hope to stay one) which is reflected in my questions. I know that the web offers a lot of resources on Python/Perl differences. But I couldn't find a satisfying answer to my questions: 1 - How does the speed of execution of Perl compares to that of Python? Much of a muchness in my experience.(Qualitative, not quantative) 2 - Regular Expressions are a valuable tool in text processing. I have noticed that Regular Expressions are executed very fast in Python. Does anybody know whether Python executes RE faster than Perl does? 3 - In my opinion Python is very well suited for text processing. Does Perl have any advantages over Python in the field of textprocessing (like a larger standard library maybe). These two are related. If you're writing code and you expect to be using *a lot* of regular expression [*] type code then you may find perl more convenient. [*] That /might/ suggest you're taking the wrong approach mind you... Python, for me, tends to be more readable, both immediately after writing and if I go back to a year later - for maintenance, extension etc. Personally I like both languages for day in day out use, but these days tend to choose python if I think I'm likely to want to modify or extend the code. With the exception being where I'm doing heavy text processing work that I think will be more maintainable in perl, or I'm really sure I won't have to maintain it. (eg quick and dirty scripts) One side effect of perl usage though is that due to them being easy to use and convenient, they can get over used. (Rather than thinking what's the best way of solving this problem, people can end up thinking What regular expression can solve this problem - which isn't ideal) Your comment I'm complete Perl noob (and I hope to stay one) would suggest to me that if you really feel that way, stay that way :-) (Though personally I do like learning new programming languages, since you get more idioms and tools under your belt that way.) I hope somebody can answer my questions. Of course, every remark and tip on Python/Perl in texprocessing is most welcome. In case you're not aware there's the book Text Processing in Python by David Mertz, which is available online in a free as in beer form which might be of use if you decide python instead of perl. Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: dual processor
Jorgen Grahn wrote: On Tue, 06 Sep 2005 08:57:14 +0100, Michael Sparks [EMAIL PROTECTED] wrote: ... Are you so sure? I suspect this is due to you being used to writing code that is designed for a single CPU system. What if you're basic model of system creation changed to include system composition as well as function calls? Then each part of the system you compose can potentially run on a different CPU. Take the following for example: ... It probably looks strange, but it's really just a logical extension of the Unix command line's pipelines to allow multiple pipelines. Similarly, from a unix command line perspective, the following will automatically take advantage of all the CPU's I have available: (find |while read i; do md5sum $i; done|cut -b-32) 2/dev/null |sort And a) most unix sys admins I know find that easy (probably the above laughable) b) given a multiprocessor system will probably try to maximise pipelining c) I see no reason why sys admins should be the only people writing programs who use concurrency without thinking about it :-) Nitpick: not all Unix users are sysadmins ;-) Some Unix sysadmins actually have real users, and the clued users use the same tools. I used the 'make -j3' example elsewhere in the thread (I hadn't read this posting when I responded there). I simply picked a group that do this often :-) The example pipeline I gave above is I admit a particularly dire one. Things like the following are far more silly: # rm file; fortune | tee file | wc | cat - file 3 16 110 Bubble Memory, n.: A derogatory term, usually referring to a person's intelligence. See also vacuum tube. And # (rm file; (while [ ! -s file ]; do echo /dev/null; done; cat file |wc) fortune | tee file) 2/dev/null Yea, though I walk through the valley of the shadow of APL, I shall fear no evil, for I can string six primitive monadic and dyadic operators together. -- Steve Higgins # 4 31 171 It seems to me that there must be a flaw in your arguments, but I can't seem to find it ;-) Sorry, but that's probably the funniest thing I've read all day :-) Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: dual processor
Thomas Bellman wrote: Michael Sparks [EMAIL PROTECTED] writes: Similarly, from a unix command line perspective, the following will automatically take advantage of all the CPU's I have available: (find |while read i; do md5sum $i; done|cut -b-32) 2/dev/null |sort No, it won't. At the most, it will use four CPU:s for user code. OK, maybe I should've been more precise. That said, the largest machine I could potentially get access relatively easily to would be a quad CPU machine so if I wanted to be be pedantic, regarding *I* have available the idea stands. (Note I didn't say take /best/ advantage - that would require rewriting all the indvidual parts of the pipeline above to be structured in a similar manner or some other parallel approach) You've essentially re-iterated my point though - that it naturally sorts itself out, and does the best fit it can, which is better than none (despite this being a naff example - as I mentioned). Worst case, yes, everything serialises itself. Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: dual processor
Jeremy Jones wrote:
Michael Sparks wrote:
Steve Jorgensen wrote:
On 05 Sep 2005 10:29:48 GMT, Nick Craig-Wood [EMAIL PROTECTED] wrote:
Jeremy Jones [EMAIL PROTECTED] wrote:
One Python process will only saturate one CPU (at a time) because
of the GIL (global interpreter lock).
I'm hoping python won't always be like this.
I don't get that. Python was never designed to be a high performance
language, so why add complexity to its implementation by giving it
high-performance capabilities like SMP?
It depends on personal perspective.
Unot totally. It depends on what you're doing.
Yes, it does. Hence why I said personal perspective.
Sort of. Steve
brings up an interesting argument of making the language do some of your
thinking for you. Maybe I'll address that momentarily
Personally I think that the language and tools will have to help. I'm
working on the latter, I hope the GIL goes away to help with the former,
but does so in an intelligent manner. Why am I working on the latter?
I work with naturally concurrent systems all the time, and they're
concurrent not for performance reasons, but simply because that's what they
are. And as a result I want concurrency easy to deal with, and efficiency as
a secondary concern. However in that scenario, having multiple CPU's not
being utilised sensibly *is* a concern to me.
I'm not saying I wish the GIL would stay around. I wish it would go.
As the price of computers goes down, the number of CPUs per computer
goes up, and the price per CPU in a single system goes down, the ability
to utilize a bunch of CPUs is going to become more important.
And maybe
Steve's magical thinking programming language will have a ton of merit.
I see no reason to use such derisory tones, though I'm sure you didn't mean
it that way. (I can see you mean it as extreme skepticism though :-)
If in a few years time we all have
machines with multiple cores (eg the CELL with effective 9 CPUs on a chip,
albeit 8 more specialised ones), would you prefer that your code *could*
utilise your hardware sensibly rather than not.
But let me play devil's advocate for
a sec. Let's say we *could* fully utilize a multi CPU today with
Python.
...
I would almost bet money that the majority of code would
not be helped by that at all.
Are you so sure? I suspect this is due to you being used to writing code
that is designed for a single CPU system. What if you're basic model of
system creation changed to include system composition as well as
function calls? Then each part of the system you compose can potentially
run on a different CPU. Take the following for example:
(sorry for the length, I prefer real examples :-)
Graphline(
EXIT = ExceptionRaiser(FORCED SYSTEM QUIT),
MOUSE = Multiclick(caption=,
position=(0,0),
transparent=True,
msgs = [ , NEXT, FIRST, PREV, PREV,NEXT ],
size=(1024,768)),
KEYS = KeyEvent(outboxes = { slidecontrol : Normal place for message,
shutdown : Place to send some shutdown
messages,
trace : Place for trace messages to go,
},
key_events = {112: (PREV, slidecontrol),
110: (NEXT,slidecontrol),
113: (QUIT, shutdown),
}),
SPLITTER = Splitter(outboxes = {totimer : For sending copies of key
events to the timer,
tochooser : This is the primary
location for key events,
}),
TIMER = TimeRepeatMessage(NEXT,3),
FILES = Chooser(items = files, loop=True),
DISPLAY = Image(size=(1024,768),
position=(0,0),
maxpect=(1024,768) ),
linkages = {
(TIMER, outbox) : (FILES, inbox),
(MOUSE, outbox) : (SPLITTER, inbox),
(KEYS, slidecontrol) : (SPLITTER, inbox),
(SPLITTER, tochooser) : (FILES, inbox),
(SPLITTER, totimer) : (TIMER, reset),
(KEYS, shutdown) : (EXIT, inbox),
(FILES, outbox) : (DISPLAY, inbox),
}
).run()
What does that do? Its a slideshow program for display pictures. There's a
small amount of setup before this (identifying files for display, imports,
etc),
but that's by far the bulk of the system.
That's pure python code (aside from pygame), and the majority of the code
is written single threaded, with very little concern for concurrency. However
the code above will naturally sit on a 7 CPU system and use all 7 CPUs (when
we're done). Currently however we use generators to limit the overhead in
a single CPU system, though if the GIL was eliminated sensibly, using threads
would allow the same code above to run on a multi-CPU system efficientally.
It probably looks strange, but it's really just
Re: dual processor
Steve Jorgensen wrote: On 05 Sep 2005 10:29:48 GMT, Nick Craig-Wood [EMAIL PROTECTED] wrote: Jeremy Jones [EMAIL PROTECTED] wrote: One Python process will only saturate one CPU (at a time) because of the GIL (global interpreter lock). I'm hoping python won't always be like this. I don't get that. Python was never designed to be a high performance language, so why add complexity to its implementation by giving it high-performance capabilities like SMP? It depends on personal perspective. If in a few years time we all have machines with multiple cores (eg the CELL with effective 9 CPUs on a chip, albeit 8 more specialised ones), would you prefer that your code *could* utilise your hardware sensibly rather than not. Or put another way - would you prefer to write your code mainly in a language like python, or mainly in a language like C or Java? If python, it's worth worrying about! If it was python (or similar) you might only have to worry about concurrency issues. If it's a language like C you might have to worry about memory management, typing AND concurrency (oh my!). (Let alone C++'s TMP :-) Regards, Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: OpenSource documentation problems
Paul Rubin wrote: Michael Sparks [EMAIL PROTECTED] writes: I've submitted a number of doc bugs to sourceforge and the ones that are simple errors and omissions do get fixed. Cool. Better than nothing, but it's only one class of problem, and maybe the easiest kind to report. [lots and lots of stuff snipped] All very interesting, and will take me a while to digest. I must admit I find the whole set of suggestions/observations taken together to be very interesting/useful and I'll try and take them on board with my own projects first. Many thanks for the detailed response, it's very much appreciated. Regards, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: OpenSource documentation problems
Hi Paul, Paul Rubin wrote: Michael Sparks [EMAIL PROTECTED] writes: [[[ some random stuff, /intended/ at supporting people who have contributed docs, rather than saying people who offer constructive suggestions are bad. Possibly badly written. ]]] I've submitted a number of doc bugs to sourceforge and the ones that are simple errors and omissions do get fixed. Cool. However, your suggestion doesn't constitute good Python advocacy, if that's what you were attempting. Nope. I rarely attempt advocacy because I'd suck at it if I tried (worse than my docs :), and also I'm not a fan of advocacy anyway - I prefer to make up my own mind rather than be told. When I use a tool like Python, I like to think I'm working on one project at a time, that project being whatever I'm trying to develop. Telling me that whenever I use Python, I should really consider myself to be working on two projects at once I didn't mean it that way (I probably put it over the wrong way). What I did mean is that when someone in here says someone here should make the docs better, is naturally including themselves in the set of people who can write docs. If they then attack people who /can/ write docs they're attacking themselves. Stating that something needs writing or updating I'd agree is good. People whinging because no-one has written it, or constantly dissing docs without helping improve them strikes me as unfair. To my mind yourself, Bryan, Steve, Terry and others in the thread contribute amazingly (and I wish I had more time to do so), by simply being nice and helpful in *here*. [ O'Reilly, and friends, books as a docs for open source ] If those docs aren't free to redistribute, then that's not an open-source solution at all; if anything, it's an anti-solution, since it decreases the motivation to write good free docs. OK, it's not the best solution in the world I'll agree, but my point was in general very few people like writing docs even when paid, let alone when not. As a result it's not that suprising IMO that if paid you end up with more docs. Even weirder is the amount of gratis (free as in beer) but unfree (as in freedom) docs out there for Python. I'm not aware of those, so I don't feel able to comment sensibly. The best way to repay those who have spent the time writing what can be viewed as a first draft is to provide corrections. If you (generic, not just you personally) don't have the time, effort, inclination or effort to summarise and feedback a documentation patch *why should anyone else* ? I usually do report doc bugs, but my frustration (and I think Bryan's) is that there are so many bugs in the first place. It means that the authors are not applying high enough quality standards to their own work before releasing it. That applies to Python's code as well as its docs. It's not crap, Or maybe they're just doing their best? It might seem silly but when I do write docs, personally it takes me around 4-10 times longer to write the same number of lines as documentation than as code, whilst trying to maintain a similar quality. I think thats on a good day. but it's not at the level that those of us from the non-Windows world like to (realistically or otherwise) expect. FWIW, I'm not part of the windows crowd. Ceased using windows regularly around Windows 3.1, and have periodically gone back (95, 98, ME, 2000, XP) to see whether it's improved, and come to the conclusion for me it's a downgrade in the way I work. It is sometimes useful for playing DVDs on long journeys though since the power management works better IME. [ specific example of bad or missing docs snipped ] I agree there's always valid criticisms. I think there is an attitude problem in the central Python development community, which is to expect external volunteers to do stuff with no cajoling and no guidance. That just doesn't work very well. I have no idea what to do about that. Giving T-Shirts to people who write a decent tutorial that covers an entire library module? (Semi-serious if there's a way of making that work) I was the first FSF staff programmer on the GNU project and we spent a LOT of our time coordinating volunteers and maintaining lists of tasks to recruit people to do, and generally trying to make stuff happen according to what we saw as the project's priorities, as opposed to simply passively waiting for code and doc contributions to come to us fully done. We also saw doing gap-filling and grunt-work that didn't excite volunteers to be an important part of our purpose as paid staff: if somebody had to do it and no one volunteered, then the somebody was us. This might sound really dumb, but I'd be genuinely interested to hear more about how you organised this, how you managed to motivate people to do these things, what rewards the people got for doing this. (Working on the assumption that rewards range from simple thanks upwards!) it's ironic
Re: OpenSource documentation problems
A.M. Kuchling wrote: On 1 Sep 2005 05:04:33 -0700, Paul Boddie [EMAIL PROTECTED] wrote: Please note that I'm not labelling you as a troll. No, he's simply barking mad. I was amused by a rec.arts.sf.written discussion [1] where Lee complains that Jonathan Swift (1667-1745)'s writing was unclear in style; apparently he's not aware that conventions and styles change over time. --amk [1] http://groups.google.com/group/alt.usage.english/msg/0ec9871395fc90d3 Oddly that makes me wonder if Xah Lee is a some kind of bot - given it follows a similar fashion: oh, this writing is [criticism] of [ specific] the allow that should be allow the notion that. But the whole thing is written in a confounded way. Specific grammar correction and note that the grammar subsystem detected too many errors. Better: Grammar correction suggestion. Not actually any clearer to many people. more better with logicality: Similar bad parsing. A person would translate the old style english into a more modern style if what they're doing is criticising old english. The whole thing just smacks to me of some text analyser. The final paragraph contains some standard writing advice and cliches: * to write clearly, the first thing is to think clearly * get rid of big words and use short sentences. It'd certainly explain the online tourettes syndrome of swearing all the way through a post - the text generator getting stuck on a particular word sets... Not-really-seriously :-) (Though usenet would be an interesting, if antisocial and annoying place to test such a beast :-) Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: OpenSource documentation problems
Bryan Olson wrote: A plausible theory. I have some possibly-illustrative examples of what I ran into within the last few weeks. hypothetical Did you take what you learnt, and use that to create better documentation to be posted on python's SF project as a patch? /hypothetical (Not aimed at you, just a preface, and you also provided the right starter point to my main comment :) Whatever else one says about open-source documentation, keeping it current is a major unsolved problem. I agree. However, the reason for this is very simple: most people who write open source software (especially so AFAICT with python) do so on their own time. In my experience its very difficult to get professional programmers when they're being paid to write good documentation, and this is /despite/ the possible alternatives. Add in the fact that part of the problem is that ability to write good code doesn't always equate to the ability to write good documentation. Also add in that the desire to write code isn't always accompanied by the desire to write documentation. Taking all that into account if it's difficult to get people who are paid to do these things to write decent documentation (which is why people hire tech writers), I find it suprising that people expect open source developers to spend their own time producing high quality documentation, which may in fact be a skill some of them lack. Ironically though I do think the problem of documentation has been half solved in the open source world, and it really boils down (IMO) to companies like O'Reilly who appear to try and target popular not-best-documented open source projects and provide the motivation that results in good documentation. As things go, python is pretty well documented IMO. Sure there's weak areas, but considering this is all stuff people have done largely on their own time, and anyone who's written docs knows it's not fun. The best way to repay those who have spent the time writing what can be viewed as a first draft is to provide corrections. If you (generic, not just you personally) don't have the time, effort, inclination or effort to summarise and feedback a documentation patch *why should anyone else* ? When people complain /in here/ about the documentation not being perfect for python I personally find it sad and ironic. It's sad because it says to the person who spent their time (when they could be doing something else) that they wasted it, the docs are worthless etc - when they clearly *haven't* wasted their time and the docs are worthwhile. It's ironic though because they're criticising themselves. After all they understand how the docs can be better and the docs are shared. Unless they contribute back they're actually attacking themselves as much as the person who wastes their time writing worthless docs. (Again, not specifically aimed at you, but unless you've actually contributed back the results of your findings, you have to realise that any reasons or excuses that apply to you may also apply to anyone/everyone in the group and it's therefore suprising we have any docs ! :-) Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Release of PyPy 0.7.0
Michael Hudson wrote: ... The chance of any random module you have written being rpython is more or less zero, so it's not _that_ interesting for you to try to compile them with PyPy. I know - the code I use contains LOTS of generators for example, which obviously don't fit the requirements :) (It's not difficult to convert code from being a generator though :-) No, you're still operating at the wrong level here (very easily done). This is the _translated PyPy_ interpreting pystone. If you run a _translated pystone_ you'll (hopefully) get a different, faster answer. Actually what I was aiming to compare was speed of pystone running on top of CPython vs speed of pystone on top of translated pypy. ie running pystone on top of something native that can execute python code :-) In expected order of execution speed: interpreted pypy interpreting pystone translated pypy interpreting pystone cpython interpreting pystone translated pystone Damn, I'm now going to have to sit down and find out if that's true :-) *shucks* :-) Anyway, whether it's sensible or not I'm going to play with translating some of my modules :) Whatever floats your boat :) Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Release of PyPy 0.7.0
Carl Friedrich Bolz wrote: [[... great news ...]] Would it be useful for people to start trying out their modules/code to see if they work with this release, and whether they can likewise be translated using the C/LLVM backends, or would you say this is too early? (I'm more thinking in terms of it providing real world usecases in the hope of finding things that don't work - rather than anything else) Either way, this looks like a great milestone - congratulations to the entire team. (I remember PyPy being met with skepticism as to whether it could even be done! :-) Best Regards, Michael -- http://mail.python.org/mailman/listinfo/python-list
Re: Release of PyPy 0.7.0
Valentino Volonghi aka Dialtone wrote: Michael Sparks [EMAIL PROTECTED] wrote: Would it be useful for people to start trying out their modules/code to see if they work with this release, and whether they can likewise be translated using the C/LLVM backends, or would you say this is too early? (I'm more thinking in terms of it providing real world usecases in the hope of finding things that don't work - rather than anything else) This is not how it works. I beg to differ - it is how it can work (just not the default or currently recommended). Currently the biggest, most interesting example is this: (from the getting started page, compiling pypy) cd pypy/translator/goal python translate_pypy.py Which has just finished executing on my machine and took about 3 hours to complete (partly because the cc1 process reached ~780MB in terms of memory footprint and my machine has 512Mb). However it goes on to talk about compiling other things: You can also use the translate_pypy.py script to try out several smaller programs, e.g. a slightly changed version of Pystone: cd pypy/translator/goal python translate_pypy.py targetrpystone Which is pretty cool of course. For those of interest running pystone with the pypy compiled native binary has the following results for pystones on my machine: [EMAIL PROTECTED]:~/pypy-0.7.0/pypy/translator/goal ./pypy-c debug: entry point starting debug: argv - ./pypy-c debug: importing code debug: calling code.interact() Python 2.4.1 (pypy 0.7.0 build) on linux2 Type help, copyright, credits or license for more information. (InteractiveConsole) from test import pystone pystone.main(1000) Pystone(1.1) time for 1000 passes = 13.97 This machine benchmarks at 71.582 pystones/second The same results for CPython: [EMAIL PROTECTED]:~/pypy-0.7.0/pypy/translator/goal python Python 2.4 (#1, Mar 22 2005, 21:42:42) [GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2 Type help, copyright, credits or license for more information. from test import pystone pystone.main() Pystone(1.1) time for 5 passes = 1.58 This machine benchmarks at 31645.6 pystones/second Obviously therefore anyone seeking to translate their existing code from python to an executable directly using pypy would not be doing it for performance reasons (again, something I'm aware of watching the updates come out and having run svn checkouts at previous times). (Factor of ~450 though is pretty d*^%n cool though :) I'm also well aware of the difference between python and rpython :-) Pypy doesn't translate your code to C/LLVM. Currently PyPy can only translate a pretty simple subset of python called RPython which has a very C-like syntax (but without memory management code). This is needed to allow type inference inside the interpreter code. The code in your application is application code and can be whatever you want, you may try to translate it to C/LLVM but it won't be that good of course because the annotator is not that intelligent. *Shrug* Seemed pretty good at annotating and compiling a module longer than many of mine at Europython, and then compiled using the LLVM backed. :) Anyway, whether it's sensible or not I'm going to play with translating some of my modules :) Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: aproximate a number
billiejoex wrote: Hi all. I'd need to aproximate a given float number into the next (int) bigger one. Because of my bad english I try to explain it with some example: 5.7 -- 6 52.987 -- 53 3.34 -- 4 2.1 -- 3 What about 2.0? By your spec that should be rounded to 3 - is that what you intend? If you do, you can simply do this: def approx(x): return int(x+1.0) Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Network performance
Roland Hedberg wrote: What the protocol has to accomplish is extremely simple; the client sends a number of lines (actually a RDF) and the server accepts or rejects the packet. That's all ! ... Now, presently I'm using ( why so is a long history which I will not go into here) and that is a bit to slow for my needs. On my present setup [Twisted and XMLRPC ]it takes close to 200 ms to perform one transfer. But since that includes setting up and tearing down the connection I thought I'd try doing something simpler so I wrote a server and a client using socket. I was surprised to find that the performance was equal to what Twisted/XMLRPC did. Around 200 ms per set, not significantly less. That should tell you two things: * Twisted/XMLRPC is as efficient as you can hand craft. (which is a good use reason for using it). * That what you're measuring is overhead - and most likely of setup. I'd measure the ping time between your two hosts. Why? TCP is not the fastest protocol in the world. If you're measuring the whole connection, you're dealing with a number of TCP messages: * Client - SYN * Server - SYN ACK * Client - ACK * Client - DATA (Assume data fits inside one message) * Server - ACK * Client - FIN * Server - FIN ACK * Server - FIN * Client - FIN ACK (Based on Fig 18.3, Steven TCP/IP Ill, Vol 2) That's 9 round trips. 200ms / 9 == 22.2 ms *IF* your ping time is close to this, then what you're really measuring is TCP setup/teardown latency, not python latency. (*IF* you're on a LAN and the latency is around that I'd suggest that you may have discovered a local network problem) If your ping time is significantly lower - eg you're running on localhost - I'd suggest you translate your code to C (and/or post your code), and repeat the same tests. So far you haven't shown that python is slow here, just that something that you're measuring causes it to be 200ms whether you use Twisted or not. (So you HAVE shown that twisted isn't slow :-) Another issue might be how you're measuring the latency - especially whether if you're including startup of python in the 200ms or not! Regards, Michael. -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: global interpreter lock
Paul Rubin wrote: Mike Meyer [EMAIL PROTECTED] writes: Even simpler to program in is the model used by Erlang. It's more CSP than threading, though, as it doesn't have shared memory as part of the model. But if you can use the simpler model to solve your problem - you probably should. Well, ok, the Python equivalent would be wrapping every shareable object in its own thread, that communicates with other threads through Queues. This is how some Pythonistas suggest writing practically all multi-threaded Python code. It does a reasonable job of avoiding synchronization headaches and it's not that hard to code that way. But I think to do it on Erlang's scale, Python needs user-level microthreads and not just OS threads. You've just described Kamaelia* BTW, except substitute micro-thread with generator :-) (Also we call the queues outboxes and inboxes, and the combination of a generator in a class with inboxes and outboxes components) * http://kamaelia.sf.net/ For those who really want threads as well, theres a threaded component based class that uses Queues instead :) Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Kamaelia 0.2.0 released!
Terry Reedy wrote: Michael Sparks [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] def updater(interval, message): t = time.time(): while 1: if time.time() - t interval: print message yield None # add this Yes. (I can't believe I missed that out! :) I'd normally use yield 1, but that's force of habit. I tend to use a true value to indicate a never ending generator - partly because we use a false value to indicate clean shutdown of a component to our scheduler. Thanks :) Michael -- [EMAIL PROTECTED], http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This message (and any attachments) may contain personal views which are not the views of the BBC unless specifically stated. -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Kamaelia 0.2.0 released!
I've reordered the q's slightly to avoid repetition... Also by answering this question first, it may put the rest of the answer into context better. phil hunt wrote: At what stage of completion is it? This is something we deliberately try to reflect in the version number. Yes, you can build network servers and new protocols relatively simply at the moment. Yes, you can integrate with pygame in a limited useful fashion at present. Yes we have audio playback. However we don't have yet... (some examples) * Decent GUI integration yet. * /Full/ pygame integration. * Nice integration with pymedia * Direct support for Dirac. Which aside from other things means you can't build (say) a video SMIL playback system trivially, yet. As a result that's why the version number is 0.2 - whilst you /can/ do a lot, there's a lot more to do. Clearly that also naturally implies that we don't expect any end user to be looking at the site. (The low version number will normally scare them away) The project aims to make it simple to build networked multimedia systems (eg audio, video, interactive systems), There's plenty of software that facilitates networking, for example Python already has software for tcp and http clients/servers, and for xmlrpc remote procedure calls. There is indeed. So what does Kamaelia do that's extra? I imagine it's to to with streaming large amounts of data. For example, a streaming video or audio player. Or VoIP, perhaps. It's designed to make bolting things together to make these sorts of system simpler and easier. At the same time it's designed to encourage writing code in a way that makes it simpler. The natural side effect of this is the system might make it easier to take advantage of multiple CPU systems as they come online, since it makes a system naturally concurrent. As the original announcement said Kamaelia is designed as a testbed. And by testbed I mean testbed as it testing out new ideas, see if they work and see if they pan out. (Not as in a testing suite) Probably the best way of describing the difference is this... After my talk about Kamaelia at Europython, I had an long chat with Tommi Virtinan about integration between Kamaelia and Twisted. I haven't had a chance to follow up with him yet regarding how this would work, though I have set a line in the sand aiming to have easy integration between Kamaelia and Twisted before Kamaelia hits version 1.0.0. The impression I got from Tommi was that he was most interested in the communications aspect - the fact we can bolt together systems in a manner directly akin to Unix pipelines, though I suspect he found the graphines aspect more interesting. Or as someone asking a similar question at Open Tech exclaimed after I finally managed to explain it better to them Ooooh - you're trying to make concurrency EASY!. OK, so what do the components in the pipelines do? What sort of data do they hold? Unix pipelines act on ascii files; I assume you are do this on audio and visual data. What langauage will the ele,ments in thne pipelines be written it? I assume some will be in C/C++ for speed. Components are object instances of python classes. The data passed between components are object instances. Clearly these python classes can be written in python, C, C++, pyrex etc. Currently all of Kamaelia's components are python based. Some existing components make calls into some related C libraries via python bindings.. An example of writing a component using Pyrex can be found here: * http://kamaelia.sourceforge.net/PyrexComponents.html It is designed as a practical toolkit, such that you can build systems such as: When you say you who do you mean? Generally I expect the readership of c.l.p/python-list@python.org to be programmers. Python is generally easy to pick up and having asked someone who's not done much programming beforehand (beyond a small amount of VB and Access), and is pre-university to use the system to build a simple streaming system prototyping visualising PVR content on a mobile (and watching them succeed), they seem relatively reasonable examples. At some point, the ability to allow non-programmers to bolt together Kamaelia systems would be desirable, but a first step is making it simpler for programmers to bolt together systems. We currently have an interactive visualisation tool(*), and the logical extension of that is a tool that allows systems to be bolted together without knowing any code. (*) http://kamaelia.sourceforge.net/AxonVisualiser.html It'd be an interesting side project for someone to take forward, and might be low hanging fruit in terms of projects. (Especially if viewed initially as a tool for assisting development, rather than replacing development) Is the audience programmers or less technical people? A project that allows non-technical people to build complex network applications is an ambitious one, but not impossible (I'd find it very impressive and very
Re: ANN: Kamaelia 0.2.0 released!
phil hunt wrote: On Wed, 03 Aug 2005 16:57:34 +0100, Michael Sparks [EMAIL PROTECTED] wrote: Is the audience programmers or less technical people? A project that allows non-technical people to build complex network applications is an ambitious one, but not ... It's a little ambitious at this stage, yes. But it couldbe there eventually? Could? Yes. Will? Can't say. I can agree it would be nice, and given time/resources (or someone sufficiently interested externally) then it may happen. (IMO there's no real reason that it couldn't happen aside from time/effort/resources) What sort of servers and clients? Whatever you feel like. If you want a server to split and serve audio, you could do that. This is streaming audio, right? For non-streaming I can just use an ftp or http server. There's more to network servers and clients than just audio video, or unidirectional download. For example the visualisation/introspection tool is an example of a client server system. The server is the visualisation tool. It listens on a specified port waiting for a connection. The client connects and sends data to it about the internal structure, and the server displays this. * Quickly build Multicast based network servers and clients Serving what? Could I use it, for example, to build an n-player encrypted VoIP server to allow people to do conference calls over the Internet? You could do that probably. (Though we don't have a component for audio capture (though a read file adaptor reading from /dev/audio might work depending on your platform I suppose) and audio encoding at the moment, so those would probably be the core components to integrate. That's a slightly worrying answer for me, worrying because it seems I've misunderstood the nature of the project. I assumed that components for audio capture, and related activities, would be at the heart of the project. *Our* main interest, at the moment, is in /delivery/ of content. Dealing with capture would be largely re-inventing wheels before we know whether the framework is a suitable framework. We are looking at making it possible to use pymedia for dealing with capture/encoding/decoding. There's a number of things along the way we need to deal with this, but we're not starting from the perspective of capture. (After all for capture we can generally look at using dedicated encoder hardware that will often spit out it's encoded information in the form of a network connection. As a result capture and encoding hasn't been a priority as yet. Sometimes looking into a project from the outside I can appreciate that certain decisions might look strange, but consider that you don't need to worry about capture in order (I mean proper encryption here, the sort GCHQ or the NSA can't break) I'd be impressed if that could be written, using anything really. (Can't implies never) What -- good encryption? That's pretty much a well-known technique these days (unless the NSA has some *very* advanced hardware in their basement, which I strongly suspect they don't). You said *can't*. That says to me cannot ever be broken. If you have a large number of listeners, as your statement implied, that implies decryptable by many listeners - you then just need one compromised listener (essentially you're asking for the equivalent of implementing a DRM system that the NSA couldn't break...). If you can provide me with a library that you can guarantee that it will satisfy the following properties: encoded = F(data) and a piece of client code that can do this: decoded = G(encoded) Then yes, that can be wrapped. That's trivial in fact: ---(start)--- from magic import unbreakable_encryption class encoder(component): def __init__(self, **args): self.encoder = unbreakable_encryption.encoder(**args) def main(self): while 1: if self.dataReady(inbox): data = self.recv(inbox) encoded = self.encoder.encode(data) self.send(encoded, outbox) yield 1 class decoder(component): def __init__(self, **args): self.decoder = unbreakable_encryption.decoder(**args) def main(self): while 1: if self.dataReady(inbox): data = self.recv(inbox) decoded = self.decoder.decode(data) self.send(decoded, outbox) yield 1 ---(end)--- If you believe you can implement FG for general use such that FG can //never// be decrypted by anyone other than authorised recipients, I suggest you cease this conversation - you have some highly marketable technology. The basic underlying metaphor of a component us like an office worker with inboxes and outboxes, with deliveries occuring between desks, That metaphor brings up an image (at least to me) that the sorts of data that can be communicated are things like documents, spreadsheets, business graphs, memos. They could indeed. The underlying framework doesn't differentiate between data nor have
Re: ANN: Kamaelia 0.2.0 released!
phil hunt wrote: On Wed, 03 Aug 2005 16:57:34 +0100, Michael Sparks [EMAIL PROTECTED] wrote: ... Which aside from other things means you can't build (say) a video SMIL playback system trivially, yet. Isn't SMIL something that's goinhg to go into web browsers? In which case, you'd presumably not want to build one yourself, I imagine? I was using SMIL as an example to be illustrative of a class of networked applications that require visual support. A more direct example would be MHEG (not mpeg) decoding and display systems, which have similarities to SMIL systems in terms of capabilities required. It's designed to make bolting things together to make these sorts of system simpler and easier. What you say bolting things together do you mean writing Python code? Yes. Or will there be some other way? A clicky pointy approach would be nice. (There's two sorts of interface in the world IMO - tappity tappity and clicky pointy - if being equally disdainful :) Unix shell systems are very tappity tappity, and whilst I can do tappity tappity, and have been doing tappity tappity for my entiring working life, I do prefer clicky pointy systems.) Though building systems using AR Toolkit type systems would be the ideal. ( http://www.hitl.washington.edu/people/poup/research/ar.htm#artoolkit ) That's probably a couple of years off though (at least). What I have in mind is something like a project I worked on some time ago, where a program could be written by drawing boxes on a GUI application, and drawing lines (representing data flow) to connect the boxes. So one half of a VoIP application might look like: +--+ ++ +---+ | listen | | convert to | | volume| | on udp || audio || control + | | port 600 | | stream | | output| +--+ ++ +---+ From example 4: pipeline( Multicast_transceiver(0.0.0.0, 1600, 224.168.2.9, 0), detuple(1), SRM_Receiver(), detuple(1), VorbisDecode(), AOAudioPlaybackAdaptor(), ).run() Join multicast group 224.168.2.9, listen for packets on port 1600, throw away information relating to who sent it, pass it through a simple reliable multicast reordering subsystem, throw away the annotation, decode the vorbis, and playback. The visual representation of that isn't dissimilar to your diagram above. With something like this novel multimedia apps could be prototyped quickly (though making them into useful apps would take extra work -- in the case of a VoIP application you'd want a phonebook, for example). Agreed. At the same time it's designed to encourage writing code in a way that makes it simpler. Examples would be useful here. Both of what it can do now, and what it will be able to do eventually. Examples of what can be done now are here: * http://kamaelia.sourceforge.net/KamaeliaStatus.html I'll think about the latter part of that. (I personally loath technologies that promise much and deliver little. Hence why I like to focus on what we /can/ do) The natural side effect of this is the system might make it easier to take advantage of multiple CPU systems as they come online, since it makes a system naturally concurrent. As the original announcement said Kamaelia is designed as a testbed. And by testbed I mean testbed as it testing out new ideas, see if they work and see if they pan out. (Not as in a testing suite) So what it will eventually do is not cast in stone? The high level goals are to solve the problems in the Challenges document. We're not particularly wedded to any one particular approach, and Kamaelia can/should be moulded to solve those problems, not the other way round. By definition along the way a variety of components will be generated, which can obviously be put together in an arbitrary fashion, assuming type adaptors as necessary, to form new interesting systems. This ability to explore new/interesting systems quickly is something generally of value to an RD dept, so there's some people at BBC RD interested in using Kamaelia for all sorts of interesting projects. (which I can't discuss here) Components are object instances of python classes. The data passed between components are object instances. What sort of objects? Python objects. Numbers, strings, chunks of audio in a string, chunks of audio in a binary object wrapped by a python class, etc. Precisely as I said. For example, if an application is audio straming, what sort of objects will be passed? That depends on the level to which you dive inside the audio being streamed. If you're trying to be a content agnostic streamer then you don't care what's inside the byte stream (from, say, a file) just what the bit rate is you're supposed to send at. (for that you can just chop up the file and send it chunks at a time) If however you're dealing with variable bit rate data you may wish to look inside the wrapper to figure
RE: ANN: Kamaelia 0.2.0 released!
Tim Golden wrote: I just wanted to say that I find the ideas behind Kamaelia interesting, and I only wish I had an application for it! Because I'm not especially into media-streaming, I'm more interested in it from the point of view of the generator-based architecture. It's nice to know that people find it interesting even without a direct need. (Indeed I suspect this is the case with many open source projects.) One thing we are learning as we go along is how to use generators to refactor out commonly used code out of loops like this. One example is status updating: import time def long_running_thing(foo): t = time.time()pplication for it! Because I'm not especially into media-streaming, I'm more interested in it from the point of view of the generator-based architecture. It's nice to know that people find it interesting even without a direct need. (Indeed I suspect this is the case with many open source projects.) One thing we are learning as we go along is how to use generators to refactor out commonly used code out of loops like this. One example is status updating: import time def long_running_thing(foo): t = time.time() while running: do something long running if time.time() - t update_interval: print update_message An alternative to this is this: def updater(interval, message): t = time.time(): while 1: if time.time() - t interval: print message def long_running_thing(foo): myUpdater = updater(5, Still running) while running: do something long running myUpdater.next() You can do similar things for frame rates and so on. Obviously in our system we generally go one step further and translate the resulting generators into components, but sometimes simple conversion to generators alone can make the code much clearer. Tim Golden wrote: What prompted me to write now was in appreciation of Michael's taking the effort to recast his words into a form which might well be understood better by others (including me) when he might well have simply huffed a bit and told the other poster to read the words on the site and stop complaining about the HTML! I tend to generally think that for each poster to a newsgroup there's often several lurkers who share the same opinion, and given text is a very harsh medium, I try to make allowances for that. Answering questions can sometimes also help others to assist with documentation patches of course. Technical writing not necessarily being the same skill set as code writing or architecture design after all :-) I must admit I do remember the earlier days of usenet when people would ask a question, get several answers, and the original poster would summarise the responses (something that doesn't happen as often these days unfortunately, possibly due to services like google news making it easier to go back and see the discussion). phil hunt wrote: (And on a more general level to improve the average quality of documentation for open source projects. Something which is bvery often deficient.) You have to remember as well that most open source projects rely on volunteers, and anyone can write docs. If you want to improve the average quality of docs for open source projects there is only one way to do this - to be prepared to write the documentation yourself or pay someone to do it. (In our case documentation is driven by internal need, which doesn't always match the needs of people externally. You might not've seen the white paper giving a detailed overview of the project from a technical perspective here: http://tinyurl.com/8543e) Tim Golden wrote: Thanks, Michael. Hope the project continues apace and if I can ever find a need for it, I'll know where to look. Thanks for your kind words. Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Advanced concurrancy
Peter Tillotson wrote: I've not yet had a chance to try some examples, but i've looked through the documentation. It feels quite familiar, It hopefully should. The approach is based essentially on an asynchronous hardware approach, on the recognition that the fundamental reason that hangs together is because it can be reasoned about in a similar manner to CSP. (That naturally leads to similarities of course with Occam) but i'd say that it is closer to Jade, the fipa (federation of intelligent physical agents) compliant agent framework Thanks - I've not come across those - I'll have to take a look at that in detail. (With a cursory look, they sounds similar to an application area a colleague is interested in investigating) than CSP or pi calculus. I like the behaviour (component microthread) model, but the advantage of CSP / pi calculus is that the resulting distributed system remains open to mathematical analysis. Indeed. However, by trying to have the same basic limitations in the system, Kamaelia should be amenable to the sort of approaches taken by hardware verification systems. It's possibly worth mentioning that there is one person at BBC RD who is extremely interested in applying CSP style analysis techniques to Kamaelia systems whom I sure would be interested in chatting to you. (Indeed the similarities between Kamaelia and CSP is biggest reason for his interest :-) The idea of a verification system for software that is actually in use for real world systems would be extremely nice/useful to have (though not simple to achieve). FWIW, I am interested in hearing suggestions on changing the system to make it more amenable to mathematical analysis. (Largely because I'm a fan of formal methods where possible, but pragmatically view TDD as the current most practical way of getting towards the levels of reliabiliy aimed for by formal methods) For concurency its is the best framework i've seen :-) Thanks! Have you come across the pylinda tuplespace implementation. It might be well worth a look. I have, and I liked what I saw :) I might be barking up the wrong tree - but it seems to me that there could be considerable overlap between tuplespaces and mailboxes, Probably the closer match is the co-ordinating assistant tracker (I can see what you mean though). That provides a basic global key/value store/retrieve service, which in a concurrent system is probably most akin to a Linda tuplespace. In a biological system it's more akin to the hormonal system which is more of the model I like to think of as to /when/ to use that part of the system. (Evolution having more 'experience' than me on how to build a massively concurrent system after all !!) It's generally only used at present for one of two purposes: * Finding services that already exist (so that we don't try to open two displays for example) * Stats collection. Using a Linda tuplespace there would be an interesting next move, however we generally only extend the system based on specific need rather than exploring architecture (Which I'm sure you understand!). though you did mention that you were moving towards twisted as the underlying platform for the future. Perhaps not underlying, but definitely interoperating with (It seems rather wasteful to reinvent wheels). I'm quite interested in the mini version and also using the modules as mobile code rather than installing it formally. I'll document it slightly better and post up on the website in the next 48 hours or so. In the meantime, the optimisation test which includes a slightly modified mini-axon can be found here: http://cvs.sourceforge.net/viewcvs.py/kamaelia/Sketches/OptimisationTest/ Specifically the file simplegame.py. This includes a cutdown version of Axon that's about 100 lines long (lines 41 - 144 specifically), with the important classes being: class microprocess(object): class newComponent(object): class scheduler(microprocess): class component(microprocess): class send_one_component(component): def linkage(source,sink): I'll probably zip the Axon directory and distribute the zip with the code, adding the zip to the python path dynamically. Sounds interesting. If you go ahead with this we'd be interested in hearing how you get on. Best Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
ANN: Kamaelia 0.2.0 has been released!
Kamaelia 0.2.0 has been released! What is it? === Kamaelia is a collection of Axon components designed for network protocol experimentation in a single threaded, select based environment. Axon components are python generators are augmented by inbox and outbox queues (lists) for communication in a communicating sequential processes (CSP) like fashion. The architecture is specifically designed to try and simplify the process of designing and experimenting with new network protocols in real environments. More background on the motivations behind Kamaelia can be found here: http://kamaelia.sourceforge.net/Challenges/ The focus of this release adds in support for introspection, pygame based interfaces, 4 new examples using these, visualisation tools, as well as syntactic sugar to make building Kamaelia systems simpler. (Specifically Graphline and pipeline systems. This build upon the existing base allowing TCP and Multicast based client/server systems. Other additions and changes include updated examples, variety of bugfixes in existing components (some pre-emptively discovered by introspection), and a variety of utility components. It is now also possible to write components using threading as the concurrency model rather than generators - allowing integration with thread/blocking only based systems. The system is known to work under Linux, Mac OS X, Windows and a subset has been tested on Series 60 mobiles. General feedback is welcome either directly, mailing lists or via the project weblog which is here: * http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi What's new in version 0.2.0 ? = Lots! Full release notes can be found here: * http://kamaelia.sourceforge.net/Kamaelia-0.2.0-ReleaseNotes.html Editted highlights... Debian Packages! Many thanks are due to Gintautas Miliauskas, Programmers of Vilnius, for assistance in building Debian packages. (The current packages are based on his, and any errors are likely to be mine, not his) These have been tested successfully on Ubuntu 5.04. 4 new examples have been added showing of various new subsystem: * Example 5 : This creates a simple streaming system, and looks inside to see what components are running/active, and passes the resulting information an Axon Visualiser. * Example 6 : This is a simple/generic topology visualisation server. Accepts the following commands over the network ADD NODE id label auto - ADD LINK id id DEL NODE id DEL ALL As this stands this is pretty useful, but that's pretty much everything it does like this. * Example 7 : This shows how the visualisation subsystem can be extended to work in different ways. What this does by default when run is randomly create new nodes and new linkages quite quickly, allowing you to see how the system works. * Example 8 : Sample slideshow/presentation tool. Unlike traditional slideshow/presentation tools, you can modify this to run arbitrary components. An example of how this can work is provided - allowing stepping through some graph visualisations along with the presentation. A Tools directoy has been added with the following tools: * Axon Shell. (Requires IPython) Implements a simple command line shell which allows experimentation with Axon systems - the shell runs a scheduler as a background thread. For a tutorial of use, see: * http://kamaelia.sourceforge.net/AxonShell.html * Axon Visualiser. Implements a simple tool for looking inside (quite literally) running Axon/Kamaelia systems. This allows a very different style of debugging and can be extremely useful. Tutorial on its way! Graphlines and Pipelines These are probably the most useful additions to Kamaelia since 0.1.2. They are essentially syntactic sugar for building and working with systems of components, but make building interesting systems rapidly out of pre-existing components fun and easy. The pipelines follow the same sort of model as the Unix pipeline. Graphlines are something new, and like pipelines and all linkages may take any data along them. Please take a look at the release notes for a graphline example. A couple of simple pipelines looks like this: pipeline( ReadFileAdaptor(file_to_stream, readmode=bitrate, bitrate=40, chunkrate=50), blockise(), # Ensure chunks small enough for multicasting! Multicast_transceiver(0.0.0.0, 0, 224.168.2.9, 1600), ).activate() pipeline( Multicast_transceiver(0.0.0.0, 1600, 224.168.2.9, 0), detuple(1), VorbisDecode(), AOAudioPlaybackAdaptor(), ).run() A selection of other subsystems have been added - targeted at visualisation of Kamaelia (and other graph structured) systems using pygame. The layout mechanism is a simple physics engine. Key packages of note added: Kamaelia.UI, Kamaelia.UI.Pygame,
Re: ANN: Kamaelia 0.2.0 released!
Phil Hunt wrote: Kamaelia seems it might be an interesting project. However, I don't think the project is well served by this announcement -- which I find vague and hard to understand. Which is a shame, because it means that other people probably don't understand it very well either, which means less people will use it. It is a shame, and thanks for mentioning this. Let me have another go :-) (ripping to shreds welcome :) OK, here's a better go. (It's always difficult to think where to pitch this sort of thing) ---START--- The project aims to make it simple to build networked multimedia systems (eg audio, video, interactive systems), which are naturally componentised and naturally concurrent allowing quick and fast reuse in the same way as Unix pipelines do. It is designed as a practical toolkit, such that you can build systems such as: * Ogg Vorbis streaming server/client systems (via vorbissimple) * Simple network aware games (via pygame) * Quickly build TCP based network servers and clients * Quickly build Multicast based network servers and clients It runs on Linux, Window, Mac OS X with a subset running on Series 60 phones. The basic underlying metaphor of a component us like an office worker with inboxes and outboxes, with deliveries occuring between desks, offices, and depts. The component can thus do work anyway it likes but only communicates with these inboxes and outboxes. Like office workers, components run in parallel, and to achieve this are generally implemented using python generators, but can also used threads. The rationale behind the project is to provide a toolkit enabling the development of new protocols, including streaming, for large scale media delivery. The license essentially allows use in proprietary systems without change, but all changes to the system itself must be shared. ---END--- Is that clearer ? A short summary of all that could be: Kamaelia is a networking/communications infrastructure for innovative multimedia systems. Kamaelia uses a component architecture designed to simplify creation and testing of new protocols and large scale media delivery systems. Hopefully that's clearer than: Kamaelia is a collection of Axon components designed for network protocol experimentation in a single threaded, select based environment. Axon components are python generators are augmented by inbox and outbox queues (lists) for communication in a communicating sequential processes (CSP) like fashion. [ which you noted I really have very little idea what this means. ] The information I can guess out of this is: Kamaelia is a library for creating network protocols that sit on top of tcp and/or udp. Given that it's a BBC project, I imagine we're talking about protocols for transferring large amount of data, e.g. sound or motion pictures. Indeed. However, if you're having to guess I'm not doing the best job (despite trying!) at explaining this. More background on the motivations behind Kamaelia can be found here: http://kamaelia.sourceforge.net/Challenges/ There's something interesting here: In Building Public Value, the BBC has committed to the delivery of BBC content to the British Public over this multitude of systems, in a way that is enjoyable, accessible and **repurposable** by the British Public in the way they choose. (my emphasis) This is a laudable aim, but if the BBC want to do this, the first step they could make is to open up their content under a Creative Commons licence. not such a wise thing for me to reply to, but please read the disclaimer at the bottom, these are personal opinions only I would suggest that you forward this comment directly to the team involved in working on the Creative Archive. You can do this via this URL: * http://creativearchive.bbc.co.uk/ This is unfortunately not as simple as just putting stuff up, and any comments you have, I'm sure would be welcome. However, the Creative Archive hasn't yet been put to a Public Value Test (including impact on industry), and rights is a much more problematic issue than you might expect. (eg background music on the radio in eastenders, posters on people's walls...) There's also http://www.bbc.co.uk/info/contactus/form.shtml which might be more appropriate. On a positive note, the Creative Archive has more organisations involved than just the BBC, it inlucde the BFI , Channel 4, Open University Teachers'TV, so even if these things take longer than some people might like they do seem to be moving. Public organisations are weighed down whether the should do something, as well as whether they can. BBC RD's role in this is to make these decisions possible. They could start with their news, current affairs and documentaries. BBC Backstage is a start here - at least with some of the online content. (http://backstage.bbc.co.uk) But the download only staryed up for a week, and they didn't put the files under a CC license, so they could be
ANN: Kamaelia 0.2.0 released!
Kamaelia 0.2.0 has been released! What is it? === Kamaelia is a collection of Axon components designed for network protocol experimentation in a single threaded, select based environment. Axon components are python generators are augmented by inbox and outbox queues (lists) for communication in a communicating sequential processes (CSP) like fashion. The architecture is specifically designed to try and simplify the process of designing and experimenting with new network protocols in real environments. More background on the motivations behind Kamaelia can be found here: http://kamaelia.sourceforge.net/Challenges/ The focus of this release adds in support for introspection, pygame based interfaces, 4 new examples using these, visualisation tools, as well as syntactic sugar to make building Kamaelia systems simpler. (Specifically Graphline and pipeline systems. This build upon the existing base allowing TCP and Multicast based client/server systems. Other additions and changes include updated examples, variety of bugfixes in existing components (some pre-emptively discovered by introspection), and a variety of utility components. It is now also possible to write components using threading as the concurrency model rather than generators - allowing integration with thread/blocking only based systems. The system is known to work under Linux, Mac OS X, Windows and a subset has been tested on Series 60 mobiles. General feedback is welcome either directly, mailing lists or via the project weblog which is here: * http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi What's new in version 0.2.0 ? = Lots! Full release notes can be found here: * http://kamaelia.sourceforge.net/Kamaelia-0.2.0-ReleaseNotes.html Editted highlights... Debian Packages! Many thanks are due to Gintautas Miliauskas, Programmers of Vilnius, for assistance in building Debian packages. (The current packages are based on his, and any errors are likely to be mine, not his) These have been tested successfully on Ubuntu 5.04. 4 new examples have been added showing of various new subsystem: * Example 5 : This creates a simple streaming system, and looks inside to see what components are running/active, and passes the resulting information an Axon Visualiser. * Example 6 : This is a simple/generic topology visualisation server. Accepts the following commands over the network ADD NODE id label auto - ADD LINK id id DEL NODE id DEL ALL As this stands this is pretty useful, but that's pretty much everything it does like this. * Example 7 : This shows how the visualisation subsystem can be extended to work in different ways. What this does by default when run is randomly create new nodes and new linkages quite quickly, allowing you to see how the system works. * Example 8 : Sample slideshow/presentation tool. Unlike traditional slideshow/presentation tools, you can modify this to run arbitrary components. An example of how this can work is provided - allowing stepping through some graph visualisations along with the presentation. A Tools directoy has been added with the following tools: * Axon Shell. (Requires IPython) Implements a simple command line shell which allows experimentation with Axon systems - the shell runs a scheduler as a background thread. For a tutorial of use, see: * http://kamaelia.sourceforge.net/AxonShell.html * Axon Visualiser. Implements a simple tool for looking inside (quite literally) running Axon/Kamaelia systems. This allows a very different style of debugging and can be extremely useful. Tutorial on its way! Graphlines and Pipelines These are probably the most useful additions to Kamaelia since 0.1.2. They are essentially syntactic sugar for building and working with systems of components, but make building interesting systems rapidly out of pre-existing components fun and easy. The pipelines follow the same sort of model as the Unix pipeline. Graphlines are something new, and like pipelines and all linkages may take any data along them. Please take a look at the release notes for a graphline example. A couple of simple pipelines looks like this: pipeline( ReadFileAdaptor(file_to_stream, readmode=bitrate, bitrate=40, chunkrate=50), blockise(), # Ensure chunks small enough for multicasting! Multicast_transceiver(0.0.0.0, 0, 224.168.2.9, 1600), ).activate() pipeline( Multicast_transceiver(0.0.0.0, 1600, 224.168.2.9, 0), detuple(1), VorbisDecode(), AOAudioPlaybackAdaptor(), ).run() A selection of other subsystems have been added - targeted at visualisation of Kamaelia (and other graph structured) systems using pygame. The layout mechanism is a simple physics engine. Key packages of note added: Kamaelia.UI, Kamaelia.UI.Pygame,
Re: Python-cgi web or Perl-cgi script doubt
praba kar wrote: I want to know difference between Python-cgi and Perl-cgi and also I want to which one is efficient from the performance. Possibly the most important difference between the two is when you're using JUST cgi - ie no mod_perl, no mod_python, etc. With python, if your cgi is structured to use more than one file, then your programs can be compiled and stored on disk (compiled to something called bytecode). This can just be loaded and executed immediately on subsequent runs. With perl, the program needs to be loaded and recompiled with each run. For small CGI systems this difference is negligible, but for large CGI based systems (eg when using a large framework), these differences can become very important - if you do not have the option to use mod_perl/mod_python or similar. However, many people running large web systems do tend to use mod_perl and friends when using perl which discounts this argument pretty much completely. That said, I'd suggest looking to see *what* you want to do, look at what *tools* are available and *then* decide which *language* to use. Personally I find python nicer for this sort of things these days - larger because it naturally encourages better strutcure. (And this is after having using perl day-in, day-out for 5 years, and still liking perl :-) Hope that's of some use, Michael. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python language ver 2.4 , development platform 0.4
Dark Cowherd wrote: The Python language is at ver 2.4 and a thing of beauty. As a development environment IMHO it is probably 0.4 Have you considered looking at any of the commercial IDEs? Personally I *like* command line based systems, but I do know many people who swear by GUI based IDEs. If you have, what did want to see but didn't see ? (System usability from my perspective starts with developers, since whilst beauty is skin deep, ugly goes to the bone) Most of these IDEs tend to integrate with one of the more common GUI toolkits as well, which /may/ be more TOOWTDI from your perspective. As for batteries included, I suspect it depends on what you expect as batteries :-) Small projects, utilities which I am going to code myself I will continue to use Python because I love it. Nice to hear. I hope to learn Python well enough to be able to contribute to some projects to achieve this. But I still see that as six to nine months away. I'm sure such contributions would be welcome - especially if you're scratching an itch to help resolve the things you currently suggest as deficiencies :-) Regards, Michael. -- http://mail.python.org/mailman/listinfo/python-list
