Re: Comparison operators in Python

[Michael Sparks]

On Jun 2, 1:44 am, harrismh777  wrote:
..
>     Just another example (excluding  print  1/2  and  unicode) where 3.x
> seems to be completely compatible with 2.x/   (tongue-in-cheek)

One of the key purposes of the 3.x line of code is to get rid of warts
in the language. As a result, if someone is relying on warts, then
their code will break when changing from 2.x to 3.x.

IMO, this is actually a good thing since it encourages the reduction
in warty code. (People who want to use 2.x and 3.x can either use 2to3
and maintain 2to3-able code or write code that works in both 2.x and
3.x - which is eminently doable)

> (do Brits say tongue-in-cheek?)

Yes.


Michael.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Extending classes __init__behavior for newbies

[Michael Sparks]

On Feb 14, 7:15 pm, rantingrick  wrote:
> On Feb 14, 11:55 am, Michael Sparks  wrote:
>
> > It can be broken if someone tries to use the class as is - that is
> > treating the class as a model - to drive a display of the ship. If
> > it was written using super() then that wouldn't be a problem.
>
> > For example, I could write a display mixin that I'd like to use like
> > this:
>
> [snip: miles of code not formatted for 79 chars!]

Core code recopied into a pastebin instead, if that's your only
problem.
 * http://pastebin.com/nuaafqSE

I hardly call 87 lines of code "miles of code" though myself.  I call
it a relatively trivial example aimed at showing the benefit of using
super() in your code rather than hard coding brittle fragility into
your code.

> That was a nice long winded example of how to break code by using it
> in an improper manner.

No, it was a thought out example of how writing code slightly
differently from how you do now can result in something more
practical, reusable, extendable, and maintainable. I was hoping to
educate you as to why one approach *might* be considered better than
another.

Deriding is as you do completely misses the point, sadly.


Michael.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Extending classes __init__behavior for newbies

[Michael Sparks]

It can be broken if someone tries to use the class as is - that is
treating the
class as a model - to drive a display of the ship. If it was written
using super()
then that wouldn't be a problem.

For example, I could write a display mixin that I'd like to use like
this:

class VisibleShip(ship, sprite):
   ...

and
class FasterVisibleShip(faster_ship, sprite):
   ...


But because neither ship nor faster_ship call their super class
correctly,
this won't work.

To turn this into something concrete, consider the following code:

class mixin(object):
"""
The key benefit of this mixin is that it tracks instances created
and
also allows subclasses to have inheritable default values.
"""
objects = []
def __init__(self, **kwargs):
super(mixin, self).__init__()
self.__dict__.update(**kwargs)
self.objects.append(self)

Using that code, I can create my ship model as a mixin:

class ship(mixin):
x = 0
y = 0
step = 1
angle = 0
name = ""
def update(self):
self.x = self.x + math.cos(math.radians(self.angle)) *
self.step
self.y = self.y + math.sin(math.radians(self.angle)) *
self.step

As well as the faster ship:

class fastership(ship):
speed = 1
def update(self):
self.x = self.x + math.cos(math.radians(self.angle)) *
self.step * self.speed
self.y = self.y + math.sin(math.radians(self.angle)) *
self.step * self.speed

I can then create a sprite mixin:

class sprite(mixin):
x = 0
y = 0
image = "sprite.png"
display = None
def __init__(self, **kwargs):
super(sprite, self).__init__(**kwargs)
self._image = pygame.image.load(self.image)
print "Image Loaded", self.__class__

def render(self,surface):
# assume surface is something like a pygame surface
surface.blit(self._image, (self.x, self.y))

And also for debug purposes a debug mixin:
import time
class debug(mixin):
debug_file = "debug.log"
debug_handle = None
def __init__(self, **kwargs):
print "Creation arguments", self.__class__, kwargs
super(debug, self).__init__(**kwargs)

@classmethod
def dump_state(klass,self):
if klass.debug_handle is None:
klass.debug_handle = open(self.debug_file, "w")
klass.debug_handle.write(str(time.time()) + ":" +
str(klass) + ":" + str(self.__dict__)+"\n\n")
klass.debug_handle.flush()

Using these we can create visible ships which can also send debug
information to a file when they're pinged:
class VisibleShip(ship, sprite, debug):
x = 300
y = 300
image = "ship.png"

class FasterVisibleShip(fastership, sprite, debug):
x = 400
y = 400
image = "fastship.png"

And use them like this:

import pygame
pygame.init()

display = pygame.display.set_mode((800, 600), 0)

ship_one = VisibleShip(step=1)
ship_two = FasterVisibleShip(angle = 60)

t = time.time()
while time.time()-t <1 :
display.fill((0,0,0))
for sprite in mixin.objects:
sprite.render(display)
sprite.dump_state(sprite)
sprite.update()
pygame.display.flip()
time.sleep(0.05)

Now, you could do this the other way I mentioned by by using kwargs,
and super in this way
you can get the various classes to play nicely with each other, which
means you can have
clearer code.

After all, our ship / fastership have now become focussed on the code
you want them to model - not
on faff with superclasses:

class ship(mixin):
x = 0
y = 0
step = 1
angle = 0
name = ""
def update(self):
self.x = self.x + math.cos(math.radians(self.angle)) *
self.step
self.y = self.y + math.sin(math.radians(self.angle)) *
self.step

class fastership(ship):
speed = 1
def update(self):
self.x = self.x + math.cos(math.radians(self.angle)) *
self.step * self.speed
self.y = self.y + math.sin(math.radians(self.angle)) *
self.step * self.speed

But not only that they play nicely with debug code and also display
code, and you get
defaults you can inherit at the same time too. If you wrote ship /
fastership with an
explicit init and didn't do the super() call - eg like this:

class ship(object):
def __init__(self,x=0,y=0,step=1,angle=0, name=''):
self.x = x
self.y = y
self.step = step
self.angle = angle
self.name = name
def update(self):
self.x = self.x + math.cos(math.radians(self.angle)) *
self.step
self.y = self.y + math.sin(math.radians(self.angle)) *
self.step

class fastership(ship):
def __init__(self,speed=1):
ship.__init__(self,x=0,y=0,step=1

Re: Namespaces

[Michael Sparks]

On Jan 21, 10:39 am, sl33k_  wrote:
> What is namespace? And what is built-in namespace?

tl;dr - Namespaces are sets that contain names. You can think of
namespaces as being /like/ boxes. A namespace is therefore an
organisational tool, forming a similar purpose to human names &
surnames - to identify the right value. (eg "Sparks" is a namespace,
"Smith" is another.) The built-in namespace contains all the values
which python understands which you _don't_ define that don't have dots
in. (eg "int", "True", "None")


Looking at this in more detail...


We can create a simple namespace using an empty class Family:

class Family(object):
   pass

Sparks = Family()
Smith = Family()

Now clearly Sparks is a name, and Smith is a name. Those names are
defined to be two different Family objects/values. (I'm going to
deliberately sidestep which namespace "Sparks" and "Smith" sit inside
for the moment.)

The neat trick is that namespaces are values themselves.

In fact the really neat trick is that every value contains a
namespace.

How do I define a name inside a namespace? Suppose I want to define
the name "Michael" as a person inside the Sparks namespace, I can do
that like this:

class Person(object):
   pass

Sparks.Michael = Person()

I can then define the name Michael inside the Smith namespace as well:

Smith.Michael = Person()

As you can see, I can now refer to two different values with the same
name -  "Michael".

This may look a little like sophistry, so let's suppose the Person
we're referring to as Sparks.Michael has an height of 180cm, and a
favourite colour of green, and Smith.Michael has a height of 120cm and
a favourite colour of 120.

In both cases, it makes sense for use to name the height value
"height", and name the favourite colour value as "favourite_colour".
If we did this though ...

height = 180
favourite_colour = "green"
height = 120
favourite_colour = "purple"

.. python would only remember the most recent value of each. By
recognising that every value is a namespace too, we can define those
names inside their namespace.

Sparks.Michael.height = 180
Sparks.Michael.favourite_colour = "green"
Smith.Michael.height = 120
Smith.Michael.favourite_colour = "purple"

Now the question that might arise is this: Given I can rewrite the
examples above like this...

class Family(object):
   pass

class Person(object):
   pass

Sparks = Family()
Smith = Family()
Sparks_Michael = Person()
Smith_Michael = Person()
Sparks_Michael_height = 180
Sparks_Michael_favourite_colour = "green"
Smith_Michael_height = 120
Smith_Michael_favourite_colour = "purple"

... how is this different from before?

Well in this latter version we're not using namespaces to organise our
names. This means that if I want to write a function that prints a
person's height and favourite colour, it has to look like this:

def describe_person(height, favourite_colour):
   print "The person is", height, "cm tall"
   print "Their favourite colour is", favourite_colour

Then if I want to use this, I have to do this:

describe_person(Sparks_Michael_height,
Sparks_Michael_favourite_colour)
describe_person(Smith_Michael_height, Smith_Michael_favourite_colour)

That's quite messy.

What does it look like for the namespace version?

def describe_person(somePerson):
   print "The person is", somePerson.height, "cm tall"
   print "Their favourite colour is", somePerson.favourite_colour

describe_person(Sparks.Michael)
describe_person(Smith.Michael)

describe_person now expects to recieve a single value. Inside that
value's namespace it expects to find the values "height" and "colour",
and just uses them.

As a result, when we use it, rather than passing in each low level
attribute (height, colour) we can work at a more convenient level of
working with People, and the higher level code becomes clearer.

Not only this, if we decide to add an another name to both People ...

Sparks.Michael.Pythonista = True
Sparks.Michael.Pythonista = False

... we can change describe_person to use this:


def describe_person(somePerson):
   print "The person is", somePerson.height, "cm tall"
   print "Their favourite colour is", somePerson.favourite_colour
   if somePerson.Pythonista:
  print "And they like python!"
   else:
  print "They don't know python"

Then our code for describing them remains the same:

describe_person(Sparks.Michael)
describe_person(Smith.Michael)

So far so good I hope.

Namespaces can contain code as well as basic values.

This means we can have ...

tiggles = Cat()
rover = Dog()
jemima = Duck()

tiggles.name = "tiggles"
rover.name = "rover"
jemima.name = "jemima"

... and we can get them all to have some behaviour called "make_noise"
defined by the call to Cat(), Dog(), Duck() inside their namespace,
which allows us to write:

>>> tiggles.make_noise()
Meow!
>>> rover.make_noise()
Woof!
>>> jemima.make_noise()
Quack!

And again that means we can do things like:

def describe_animal(animal):
print animal.name, "goes", animal.ma

Re: Nested Scopes unintended behaviour ?

[Michael Sparks]

On Mar 17, 8:29 pm, Terry Reedy  wrote:
> On 3/17/2010 11:44 AM, Emile van Sebille wrote:
>
> > On 3/17/2010 8:16 AM Michael Sparks said...
> >> Hi,
>
> >> Is the following behaviour expected ?
>
> > In short, yes. Assignment within a function forces the variable to
> > locals.
>
> In 3.x, one can declare names to be nonlocal (ie, local to some outer
> function, as opposed to local to the current function or module global).
> In your case,
>    nonlocal on
> in your inner swtchfun function would give the behavior you wanted.

Ah, excellent. That makes python closures work more like I'd expect
them to. (A colleague had written the swtchfun I posted, whereas the
generator form was the version I wrote, and he was puzzled as to why
it didn't work as he expected. When I saw it I also couldn't see why.

After hearing it's expected behaviour in 2.6 it's clear that assigning
a name to a value declares the variable to be local, and that unlike
much of python (but like yield) this appears based on static analysis
of the function declaration, rather than dynamic. This does also make
sense since it prevents a name "switching scope" in a function, and a
"nonlocal" keyword also makes sense as a result.

Thanks to Emile for pointing out you can also do this in 2.6:
def Toggler(F, B):
print F("Hello")
print F("Hello")
print F("Hello")
print F("Hello")
print F("Hello")

def Switcher(A,B):
enclose={"on" : True}
def swtchfun(msg, enclose=enclose):
if enclose["on"]:
enclose["on"] = False
print "Switched to A"
return A
else:
enclose["on"] = True
print "Switched to B"
return B
#
return Toggler(swtchfun,True)

Switcher(1,2)


I think personally I'd use the generator form myself, since I think
it's clearer (more clearly loops between the two), but this may be a
useful thing to know occasionally.

Cheers :-)


Michael.
-- 
http://mail.python.org/mailman/listinfo/python-list

Nested Scopes unintended behaviour ?

[Michael Sparks]

Hi,


Is the following behaviour expected ?

Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def Toggler(F, B):
... print F("Hello")
... print F("Hello")
... print F("Hello")
... print F("Hello")
... print F("Hello")
...
>>> def Switcher(A,B):
... on=True
... def swtchfun(msg):
... on_ = on
... if on:
... on = False
... print "Switched to A"
... return A
... else:
... print "Switched to B"
... return B
... #
... return Toggler(swtchfun,True)
...
>>> Switcher(1,2)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 13, in Switcher
  File "", line 2, in Toggler
  File "", line 4, in swtchfun
UnboundLocalError: local variable 'on' referenced before assignment

The reason I ask is because logically it makes sense. The on_ = on
statement should resolve "on" to be the value on in the enclosing
scope, however it appears that the "on = False" statement is taking
priority. The reason I say this is because if you remove the "on =
False" line you get the expected name resolution:

>>> def Toggler(F, B):
... print F("Hello")
... print F("Hello")
... print F("Hello")
... print F("Hello")
... print F("Hello")
...
>>> def Switcher(A,B):
... on=True
... def swtchfun(msg):
... on_ = on
... if on:
... print "Switched to A"
... return A
... else:
... print "Switched to B"
... return B
... #
... return Toggler(swtchfun,True)
...
>>> Switcher(1,2)
Switched to A
1

ie it looks like python is not looking at the expected scope in the
first instance.

To me it looks like a bug, but I can also see a rationale where it's
considered a feature (because the "on" is on the left hand side
resolving the value to a local, rather than a value in an enclosed
scope)

I know that you can work around this as follows:
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def Toggler(F, B):
... print F("Hello")
... print F("Hello")
... print F("Hello")
... print F("Hello")
...
>>> def Switcher(A,B):
... def switchgen():
... while True:
... yield A
... yield B
... G = switchgen()
... def swtchfun(msg):
... return G.next()
... #
... return Toggler(swtchfun,True)
...
>>>
>>> Switcher(1,2)
1
2
1
2

But I'm curious as to whether the nested scope issue above is
considered a bug or a feature, so I can deal with it appropriately.

Any comments welcome :-)

Regards,


Michael.
--
http://yeoldeclue.com/blog
http://www.kamaelia.org/Home.html
http://twitter.com/kamaelian



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Interesting talk on Python vs. Ruby and how he would like Python to have just a bit more syntactic flexibility.

[Michael Sparks]

On Feb 18, 4:15 pm, Steve Howell  wrote:
...
>     def print_numbers()
>         [1, 2, 3, 4, 5, 6].map { |n|
>             [n * n, n * n * n]
>         }.reject { |square, cube|
>             square == 25 || cube == 64
>         }.map { |square, cube|
>             cube
>         }.each { |n|
>             puts n
>         }
>     end

This strikes me as a terrible example. For example, this is
significantly clearer:
def print_numbers()
for n in [1,2,3,4,5,6]:
square, cube = n * n, n * n * n
if square != 25 and cube != 64:
print n

I /can/ see arguments for ruby style blocks in python, but not for
this sort of thing, or lisp style quoted expressions[1]. ie I can see
situations where you have more complex code in real life where they
will definitely simplify things.

[1] This is perhaps more appropriate because '(a b c) is equivalent
to (quote a b c), and quote a b c can be viewed as close to
python's expression "lambda: a b c"

However, I can also see that in simple situations - such as the
example you post - they will have a tendency to make code
significantly less clear/direct.

I suppose, if I have a choice between something (hard being possible &
simple code looking simple) and (hard things being simpler & simple
things looking harder), I'd probably personally choose the former.
This is not because I don't like hard things being simple, but because
I think that simple things are more common and making them look harder
is a mistake.

I'm well aware that's opinion however,

Regards,


Michael.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Modifying Class Object

[Michael Sparks]

Hi Alf,


On Feb 12, 8:22 pm, "Alf P. Steinbach"  wrote:
> Thanks for the effort at non-flaming discussion, it *is*
> appreciated.

I would appreciate it if you tried to be non-flaming yourself,
since you can see I am not flaming you.

I was seeking to educate you on a simple matter which you seem
to be misunderstanding in python. This will be my second and
last attempt to do so since you chose to be inflammatory in your
response. (In case you do not understand what I find inflammatory,
I will discuss that at the end)

Please note below I may use CAPS occasionally. Whilst usually
taken as shouting, I mean them as BOLD. Please be aware of this :-)

My reason for choosing to do reply is for the simple reason
that recently I had similar discussions with a colleague who was
likewise stuck talking about implementation aspects (call by
reference/value rather than call with object/sharing).

> > Before I start, note we're talking about semantics, not
> > implementation. That distinction is very important.
>
> Yes.
[ inflammatory comment snipped]

Good - common ground - a good starting point.

Now, if I define a language, this has 3 main parts:
   * Syntax
   * Semantics
   * Implementation

The syntax of python is simply and shortly defined in a machine
readable format, and is therefore not interesting to discuss
here.

The implementation of python is similarly well defined. There are
multiple such implementations, one of which is CPython.

> However, all those references to implementation aspects,
> persisting
[ inflammatory comment snipped]

In theory when talking about a language, we do not need to talk
about implementation details. However when using a language,
implementation details do matter as well.

That's why people talk about implementation aspects. When talking
about how the language is defined, you need to talk about how the
language is defined. It's not defined in terms of Java pointers or
references. It's defined in terms of objects and names. (Or objects
and labels)

The exception to this is with pure functional language. In a pure
functional language I do not care about implementation details,
since they are outside the scope of the language.

It is worth noting that python and functional languages share a
common ethos - though with different conclusions - that optimising
for the programmers expression of the problem rather than for the
machine *matters*.

If you miss this, you will be stuck misunderstanding python,
pretty much forever. If you (the reader, not necessarily Alf)
understand this, good. If you don't, you need to re-read this
and really understand it.

(please bear in mind when I say "YOU" in that paragraph, I
mean "whomever is reading this", not anyone specific)

Let's get down to brass tacks.

In python, the LANGUAGE, there are no pointers or references,
much like in SML, haskell and friends there are no pointers or
references.  I'm using SML here as an example, because it is
conceptually close to python in terms to some aspects of
evaluation and how it deals with names and values. (There
are many differences as well, but we're dealing with calling,
names & values, in which they are close enough)

Taking an example from SML:

structure Stk =
struct
  exception EmptyStack;
  datatype 'x stack = EmptyStack | push of ('x * 'x stack);
  fun pop(push(x,y)) = y | pop EmptyStack = raise EmptyStack;
  fun top(push(x,y)) = x | top EmptyStack = raise EmptyStack;
end;

This may be used, say from the interactive prompt, as follows:

   val x = EmptyStack;  (* 1 *)
   val 'x = x;  (* 2 *)
   val y = push(5,'x);  (* 3 *)
   val z = push(4,y);   (* 4 *)
   top z;   (* 5 *)

Now, in this example, in SML, there are only names and values.
Unlike python, all values are immutable, and theoretically, no
sequencing of statements.

Now line 1 takes the EmptyStack value, and the name x is bound
to it. Line 2 takes that same EmptyStack value, and the name 'x
is also bound to it.

There are no pointers in SML, just names and values.

Like python, SML has aliases. 'x for example is an alias for x.
However that is just a symbolic name for the object itself.

When line 3 is evaluated, the value push(5, 'x) is bound to the
name y. Note - push(5, 'x) is in itself a value in SML, because
it has been defined as such in the Datatype definition in the
structure definition Stk.

When we reach line 5, the function top is called with the value
that z is bound to. Not a reference. Not a pointer. The actual
value. In this case z's value is push(4,push(5,EmptyStack)).

Thus the SML runtime evaluates the expression
   top push(4,push(5,EmptyStack))
And returns the value 4.

In python, I don't have such richness of ability to define values,
but I have these types play with:

   * numbers - 1.0 , 1, 1+2i, 0x11 0777
   * strings - "hello" "world"
   * lists - ["hello", "world" ]
   * tuples - ("hello", "world")
   * dicts - {"hello" : "world" }

All of these types are defined in python as python objects. Ho

Re: Modifying Class Object

[Michael Sparks]

Hi Alf,


Before I start, note we're talking about semantics, not
implementation. That distinction is very important.

On Feb 11, 4:49 am, "Alf P. Steinbach"  wrote:
> > *The* standard general language independent definition?

[ of pointer ]

> Yes.
>
> > As defined where?
>
> For example, as I used as reference in my first posting, the Java language 
> spec.
>   But it has nothing specifically to do with Java. It is a basic concept in
> computer science, that (most) CS students learn in their *first year*.
>
> E.g.
>
> http://cslibrary.stanford.edu/106/";>
> A pointer stores a reference to something. Unfortunately there is no fixed 
> term
> for the thing that the pointer points to, and across different computer
> languages there is a wide variety of things that pointers point to. We use the
> term pointee for the thing that the pointer points to, and we stick to the 
> basic
> properties of the pointer/pointee relationship which are true in all 
> languages.
> The term "reference" means pretty much the same thing as "pointer" --
> "reference" implies a more high-level discussion, while "pointer" implies the
> traditional compiled language implementation of pointers as addresses. For the
> basic pointer/pointee rules covered here, the terms are effectively 
> equivalent.
> 

This is where you have gone wrong. You have taken a first year
undergraduate
academic generalisation and assumed that it applies to The World. In
theory,
there is no difference between practice and theory, but in practice
there is
(so the saying goes).

The World however has another place for defining terms. That place is
of highly
varying quality, but generally a better place to correct semantics of
terms.
Who knows, eventually there may be a single commonly accepted
viewpoint. (Which
would bring a whole new level of pedantry of course )-:

I am referring to Wikipedia here. (this is a vague attempt at humour,
rather
than an attempt to patronise which it may also come over as)

Let's look at the tip of the iceberg for that:

   "In computer science, a pointer is a programming language data type
whose value refers directly to (or "points to") another value
stored
elsewhere in the computer memory using its address."

   http://en.wikipedia.org/wiki/Pointer_%28computing%29

Similarly for Call by Value: (which *is* a loaded term)

In call-by-value, the argument expression is evaluated, and the
resulting value is bound to the corresponding variable in the
function (frequently by copying the value into a new memory
region).
If the function or procedure is able to assign values to its
parameters, only its local copy is assigned — that is, anything
passed into a function call is unchanged in the caller's scope
when the function returns.

  http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_value

Call by Reference: (again, loaded term):

   In call-by-reference evaluation (also referred to as pass-by-
reference),
   a function receives an implicit reference to the argument, rather
than
   a copy of its value. This typically means that the function can
modify
   the argument- something that will be seen by its caller.

  http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_reference

Call by Sharing: (far less common term)
   Also known as "call by object" or "call by object-sharing" is an
   evaluation strategy first named by Barbara Liskov et al. for the
   language CLU in 1974[1]. It is used by languages such as Python[2],
   Iota, Java (for object references)[3], Ruby, Scheme, OCaml,
   AppleScript, and many other languages.

   The semantics of call-by-sharing differ from call-by-reference in
   that assignments to function arguments within the function aren't
   visible to the caller (unlike by-reference semantics).

   http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_sharing

As you can see, there are generally accepted terms and definitions
here,
and python is accepted as falling not into the value or reference
camp,
along with some other languages.

Understanding why IMO comes back to some basic aspects of python which
I believe trip up experienced developers. This claim is based on
talking
to someone experienced in coding for a couple of decades, who has done
a
CS degree (like me), and just didn't understand why I would use
python.

I spent an couple of _days_ explaining this, along with python's
evaluation
model, and at the end we ended up where we started:

   * Python is a language which is focussed on programmer performance,
 not machine performance, because relative to programmer cost,
 machines are cheap. Therefore you focus your language to optimise
 for the programmer, not the machine.

In this case, let's drop back to the word "pointer" which I can
understand
that you like. Indeed, it took me a fair while to let go of the word
when
talking about python, but you do have to.

Why?

Well, assume this definition isn't bad:

   "In computer 

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Philip Semanchuk wrote:
> On Oct 25, 2008, at 7:53 AM, Michael Sparks wrote:
>> Glenn Linderman wrote:
>>> In the module multiprocessing environment could you not use shared
>>> memory, then, for the large shared data items?
>>
>> If the poshmodule had a bit of TLC, it would be extremely useful for
>> this,... http://poshmodule.sourceforge.net/
> 
> Last time I checked that was Windows-only. Has that changed?

I've only tested it under Linux where it worked, but does clearly need a bit
of work :)

> The only IPC modules for Unix that I'm aware of are one which I
> adopted (for System V semaphores & shared memory) and one which I
> wrote (for POSIX semaphores & shared memory).
> 
> http://NikitaTheSpider.com/python/shm/
> http://semanchuk.com/philip/posix_ipc/

I'll take a look at those - poshmodule does need a bit of TLC and doesn't
appear to be maintained.

> If anyone wants to wrap POSH cleverness around them, go for it! If
> not, maybe I'll make the time someday.

I personally don't have the time do do this, but I'd be very interested in
hearing someone building an up-to-date version. (Indeed, something like
this would be extremely useful for everyone to have in the standard library
now that the multiprocessing library is in the standard library)


Michael.
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python suitable for Midi ?

[Michael Sparks]

Protocol wrote:

> Is Python suitable for building a multi-track midi sequencer (with a
> gui), that would run on windows / mac ? I fail to find sufficient
> information on this, being a newbie and all. 

We had a Google Summer of Code student working on this sort of thing this
year (This clearly puts a bounds on experience btw). He got a fair way along
with this, but didn't get his code to a mergeable state (mainly in the
application files rather than in the components which are largely
mergeable).

Essentially it generated a bunch of components which he was using in his
application, which are naturally reusable.

I've pinged him since the end of GSOC, and had a response that essentially
says "sorry, too busy, maybe next semester". We'll see on that (he was
having fun, so I suspect he really does want to come back to it, but hasn't
had time). In the meantime, the answer is, yes, you can do this in Python.
His code is here, if it's useful to you:

   * http://code.google.com/p/kamaelia/source/browse/trunk/Sketches/JT
   * ( svn co http://kamaelia.googlecode.com/svn/trunk/Sketches/JT )
  - requires Kamaelia 0.6.0 btw

Top level file/application file is this:

http://kamaelia.googlecode.com/svn/trunk/Sketches/JT/Jam/application/trunk/jam

(The reason I don't feel it's mergeable as is, is because it could with a
code clean up IMO, but it's pretty good - to say the least - considering
the GSOC timeframe - also it shows it is doable, and if you're reusing his
components, then it should be very doable)

It is likely though that his component library:
http://kamaelia.googlecode.com/svn/trunk/Sketches/JT/Jam/library/trunk/Kamaelia/Apps/Jam/Protocol/

will get merged before christmas though, since that code is a lot cleaner.

In case you're wondering, he was aiming to build something a bit like this,
but using Pygame as the interface, and it allows multiple users to run
their local version, connect to each other and do shared "jamming" - hence
the interesting subdirectory is "Jam"

As well as Midi it also supports OSC.

Underneath it all, he was using the python "rtmidi" bindings, and pyosc
bindings to talk Midi and Osc. (Most of his quagmire in the last bit of
GSOC was caused by audio under linux, which doesn't sound relevant to you)

> Furthermore, i found references on Python not being really able of
> multi-threading, that further adds to the confusion.

Kamaelia's component model effectively gives you concurrency for free,
since you build systems out of components that talk to each other. I
haven't attached the introspector to Jam, but I suspect it's embarrassingly
parallel. He didn't have to worry about that though :-)

If you're curious about the model, this tutorial is specifically targetted
at new developers:
http://www.kamaelia.org/MiniAxon/

It was originally written for someone (specific) who had learnt python the
previous week, had little programming experience, and we needed to get them
up to speed quickly and gently. It's been slightly generalised since, but
is a nice introduction to the ideas. (Under the hood Kamaelia is
significantly more optimised than that example, but the components you
create for that tutorial system work with the full/real world system)

Regards,


Michael
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Glenn Linderman wrote:

> so a 3rd party library might be called to decompress the stream into a
> set of independently allocated chunks, each containing one frame (each
> possibly consisting of several allocations of memory for associated
> metadata) that is independent of other frames

We use a combination of a dictionary + RGB data for this purpose. Using a
dictionary works out pretty nicely for the metadata, and obviously one
attribute holds the frame data as a binary blob.

http://www.kamaelia.org/Components/pydoc/Kamaelia.Codec.YUV4MPEG gives some
idea structure and usage. The example given there is this:

Pipeline( RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
).run()

Now all of those components are generator components.

That's useful since:
   a) we can structure the code to show what it does more clearly, and it
  still run efficiently inside a single process
   b) We can change this over to using multiple processes trivially:

ProcessPipeline(
  RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
).run()

This version uses multiple processes (under the hood using Paul Boddies
pprocess library, since this support predates the multiprocessing module
support in python).

The big issue with *this* version however is that due to pprocess (and
friends) pickling data to be sent across OS pipes, the data throughput on
this would be lowsy. Specifically in this example, if we could change it
such that the high level API was this:

ProcessPipeline(
  RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
use_shared_memory_IPC = True,
).run()

That would be pretty useful, for some hopefully obvious reasons. I suppose
ideally we'd just use shared_memory_IPC for everything and just go back to
this:

ProcessPipeline(
  RateControlledFileReader("video.dirac",readmode="bytes", ...),
  DiracDecoder(),
  FrameToYUV4MPEG(),
  SimpleFileWriter("output.yuv4mpeg")
).run()

But essentially for us, this is an optimisation problem, not a "how do I
even begin to use this" problem. Since it is an optimisation problem, it
also strikes me as reasonable to consider it OK to special purpose and
specialise such links until you get an approach that's reasonable for
general purpose data.

In theory, poshmodule.sourceforge.net, with a bit of TLC would be a good
candidate or good candidate starting point for that optimisation work
(since it does work in Linux, contrary to a reply in the thread - I've not
tested it under windows :).

If someone's interested in building that, then someone redoing our MiniAxon
tutorial using processes & shared memory IPC rather than generators would
be a relatively gentle/structured approach to dealing with this:

   * http://www.kamaelia.org/MiniAxon/

The reason I suggest that is because any time we think about fiddling and
creating a new optimisation approach or concurrency approach, we tend to
build a MiniAxon prototype to flesh out the various issues involved.


Michael
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

Re: Do I need a lock here?

[Michael Sparks]

jasiu85 wrote:

> Do I need a lock to protect the COMMON_DICT dictionary? AFAIK bytecode
> operations are atomic and in each thread there's only one crucial
> bytecode op: STORE_NAME in the first thread and LOAD_NAME in the
> second one. So I suspect that everything will work just fine. Am I
> right?

Will it crash your system? Probably not. Will you end up with correct values
in your dict? Probably not, since you can still end up with race hazards.

If you want to share data that way, you may want to look at Kamaelia's
STM[1] model, which is described on this page:

   * http://www.kamaelia.org/STM

You can created stores - which are /essentially/ (key value) threadsafe
stores which enable you to figure out when things go wrong, so you can do
something about it ! :)

A short example:

from Axon.STM import Store

S = Store() # Shared store (like your COMMON_DICT)

D = S.using("account_one", "account_two", "myaccount")
D["account_one"].set(50)
D["account_two"].set(100)
D.commit()# This will fail if any of the values have changed
S.dump()

D = S.using("account_one", "account_two", "myaccount")
D["myaccount"].set(D["account_one"].value+D["account_two"].value)
D["account_one"].set(0)
D["account_two"].set(0)
D.commit()# This will fail if any of the values have changed
S.dump()

If this looks familiar, it's the same basic idiom as version control. The
nice thing of course, is in this version you don't need to worry about
locks, but just be aware that the .commit() may fail, and you may need to
retry.

Two examples - one using bare threads, one using Kamaelia components are
here:
   * http://kamaelia.googlecode.com/svn/trunk/Code/Python/Axon/Examples/STM/

Crucially whilst the above says "this will fail", the examples in that
directory handle the failures correctly :)

This code is included in the latest Kamaelia release, described here:
   * http://www.kamaelia.org/GetKamaelia

However you can also use the standalone tarball attached to that page.

[1] ie a minimal software transactional memory implementation. The point of
these things really is to allow you to detect when something has gone
wrong (eg and update fails) and to redo the thing that led to the
update. It's conceptually very similar to version control for variables.
I'm tempted to replace "using" with "checkout" to mirror "commit".



Michael.
-- 
http://www.kamaelia.org/GetKamaelia

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Jesse Noller wrote:

> http://www.kamaelia.org/Home

Thanks for the mention :)

I don't think it's a good fit for the original poster's question, but a
solution to the original poster's question would be generally useful IMO,
_especially_ on python implementations without a GIL (where threads are the
more natural approach to using multiple processes & multiple processors).

The approach I think would be useful would perhaps by allowing python to
have some concept of "green processes" - that is threads that can only see
thread local values or they search/update thread local space before
checking globals, ie flipping

   X = threading.local()
   X.foo = "bar"

To something like:
   X = greenprocesses.shared()
   X.foo = "bar"

Or even just changing the search for values from:
   * Search local context
   * Search global context

To:
   * Search thread local context
   * Search local context
   * Search global context

Would probably be quite handy, and eliminate whole classes of bugs for
people using threads. (Probably introduce all sorts of new ones of course,
but perhaps easier to isolate ones)

However, I suspect this is also *a lot* easier to say than to implement :-)

(that said, I did hack on the python internals once (cf pep 318) so it might
be quite pleasant to try)

It's also independent of any discussions regarding the GIL of course since
it would just make life generally safer for people.

BTW, regarding Kamaelia - regarding something you said on your blog - whilst
the components list on /Components looks like a large amount of extra stuff
you have to comprehend to use, you don't. (The interdependency between
components is actually very low.)

The core that someone needs to understand is the contents of this:
http://www.kamaelia.org/MiniAxon/

Which is sufficient to get someone started. (based on testing with a couple
of dozen novice developers now :)

If someone doesn't want to rewrite their app to be kamaelia based, they can
cherry pick stuff, by running kamaelia's scheduler in the background and
using components in a file-handle like fashion:
* http://www.kamaelia.org/AxonHandle

The reason /Components contains all those things isn't because we're trying
to make it into a swiss army knife, it's because it's been useful in
domains that have generated those components which are generally
reusable :-)



Michael.
--
http://www.kamaelia.org/GetKamaelia

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Andy O'Meara wrote:

> basically, it seems that we're talking about the
> "embarrassingly parallel" scenario raised in that paper

We build applications in Kamaelia and then discover afterwards that they're
embarrassingly parallel and just work. (we have an introspector that can
look inside running systems and show us the structure that's going on -
very useful for debugging)

My current favourite example of this is a tool created to teaching small
children to read and write:
   http://www.kamaelia.org/SpeakAndWrite

Uses gesture recognition and speech synthesis, has a top level view of
around 15 concurrent components, with signficant numbers of nested ones.

(OK, that's not embarrasingly parallel since it's only around 50 things, but
the whiteboard with around 200 concurrent things, is)

The trick is to stop viewing concurrency as the problem, but to find a way
to use it as a tool for making it easier to write code. That program was a
10 hour or so hack. You end up focussing on the problem you want to solve,
and naturally gain a concurrent friendly system.

Everything else (GIL's, shared memory etc) then "just" becomes an
optimisation problem - something only to be done if you need it.

My previous favourite examples were based around digital TV, or user
generated content transcode pipelines.

My reason for preferring the speak and write at the moment is because its a
problem you wouldn't normally think of as benefitting from concurrency,
when in this case it benefitted by being made easier to write in the first
place.

Regards,



Michael
--
http://www.kamaelia.org/GetKamaelia

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Glenn Linderman wrote:

> In the module multiprocessing environment could you not use shared
> memory, then, for the large shared data items?

If the poshmodule had a bit of TLC, it would be extremely useful for this,
since it does (surprisingly) still work with python 2.5, but does need a
bit of TLC to make it usable.

http://poshmodule.sourceforge.net/


Michael
--
http://www.kamaelia.org/GetKamaelia
--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Andy O'Meara wrote:

> Yeah, that's the idea--let the highest levels run and coordinate the
> show.

Yes, this works really well in python and it's lots of fun. We've found so
far you need at minimum the following parts to a co-ordination little
language:

Pipeline
Graphline
Carousel
Seq
OneShot
PureTransformer
TPipe
Filter
Backplane
PublishTo
SubscribeTo

The interesting thing to me about this is in most systems these would be
patterns of behaviour in activities, whereas in python/kamaelia these are
concrete things you can drop things into. As you'd expect this all becomes
highly declarative.

In practice the world is slightly messier than a theoretical document would
like to suggest, primarily because if you consider things like pygame,
sometimes you have only have a resource instantiated once in a single
process. So you do need a mechanism for advertising services inside a
process and looking those up. (The Backplane idea though helps with
wrapping those up a lot I admit, for certain sorts of service :)

And sometimes you do need to just share data, and when you do that's when
STM is useful.

But concurrent python systems are fun to build :-)


Michael.
-- 
http://www.kamaelia.org/GetKamaelia

--
http://mail.python.org/mailman/listinfo/python-list

Re: 2.6, 3.0, and truly independent intepreters

[Michael Sparks]

Hi Andy,


Andy wrote:

> However, we require true thread/interpreter
> independence so python 2 has been frustrating at time, to say the
> least.  Please don't start with "but really, python supports multiple
> interpreters" because I've been there many many times with people.
> And, yes, I'm aware of the multiprocessing module added in 2.6, but
> that stuff isn't lightweight and isn't suitable at all for many
> environments (including ours).

This is a very conflicting set of statements and whilst you appear to be
extremely clear on what you want here, and why multiprocessing, and
associated techniques are not appropriate, this does sound very
conflicting. I'm guessing I'm not the only person who finds this a
little odd.

Based on the size of the thread, having read it all, I'm guessing also
that you're not going to have an immediate solution but a work around.
However, also based on reading it, I think it's a usecase that would be
generally useful in embedding python.

So, I'll give it a stab as to what I think you're after.

The scenario as I understand it is this:
* You have an application written in C,C++ or similar.
* You've been providing users the ability to script it or customise it
  in some fashion using scripts.

Based on the conversation:
* This worked well, and you really liked the results, but...
* You only had one interpreter embedded in the system
* You were allowing users to use multiple scripts

Suddenly you go from: Single script, single memory space.
To multiple scripts, unconstrained shared shared memory space.

That then causes pain for you and your users. So as a result, you decided to
look for this scenario:
* A mechanism that allows each script to think it's the only script
  running on the python interpreter.
* But to still have only one embedded instance of the interpreter.
* With the primary motivation to eliminate the unconstrained shared
  memory causing breakage to your software.

So, whilst the multiprocessing module gives you this:
* With the primary motivation to eliminate the unconstrained shared
  memory causing breakage to your software.

It's (for whatever reason) too heavyweight for you, due to the multiprocess
usage. At a guess the reason for this is because you allow the user to run
lots of these little scripts.

Essentially what this means is that you want "green processes".

One workaround of achieving that may be to find a way to force threads in
python to ONLY be allowed access to (and only update) thread local values,
rather than default to shared values.

The reason I say that, is because the closest you get to green processes in
python at the moment is /inside/ a python generator. It's nowhere near the
level you want, but it's what made me think of the idea of green processes.

Specifically if you have the canonical example of a python generator:

def fib():
a,b = 1,1
while 1:
a,b = b, a+b
yield 1

Then no matter how many times I run that, the values are local, and can't
impact each other. Now clearly this isn't what you want, but on some level
it's *similar*.

You want to be able to do:
run(this_script)

and then when (this_script) is running only use a local environment.

Now, if you could change the threading API, such that there was a means of
forcing all value lookups to look in thread local store before looking
outside the thread local store [1], then this would give you a much greater
level of safety.

[1] I don't know if there is or isn't I've not been sufficiently interested
to look...

I suspect that this would also be a very nice easy win for many
multi-threaded applications as well, reducing accidental data sharing.

Indeed, reversing things such that rather than doing this:
   myLocal = threading.local()
   myLocal.X = 5

Allowing a thread to force the default to be the other way round:
   systemGlobals = threading.globals()
   systemGlobals = 5

Would make a big difference. Furthermore, it would also mean that the
following:
   import MyModule
   from MyOtherModule import whizzy thing

I don't know if such a change would be sufficient to stop the python
interpreter going bang for extension modules though :-)

I suspect also that this change, whilst potentially fraught with
difficulties, would be incredibly useful in python implementations
that are GIL-free (such as Jython or IronPython)

Now, this for me is entirely theoretical because I don't know much about
python's threading implementation (because I've never needed to), but it
does seem to me to be the easier win than looking for truly independent
interpreters...

It would also be more generally useful, since it would make accidental
sharing of data (which is where threads really hurt people most) much
harder.

Since it was raised in the thread, I'd like to say "use Kamaelia", but your
usecase is slightly different as I understand it. You want to take existing
stuff that won't be written in any particular w

Re: Best way to spawn process on back end computer

[Michael Sparks]

sophie_newbie wrote:

> Hi,
> 
> I'm running a python cgi script on a frontend web server and I want it
> to spawn another script (that takes a long time to run) on a backend
> number crunching server thats connected to the same network. What do
> you think is the best way to do this? I have a few ideas but I'm sure
> there is a "best" way to go about this.

Best is always subjective - what sort of interaction are you after?

Web request to trigger an activity (fire and forget)

Web request to trigger an activity, which produces output suitable for
something else?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want to easily tie
the two things together?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want those two
events to look like part of one operation?

Picking a concrete example...

Or something a bit more like:
   * User uploads an image
   * Gets converted to a bunch of standard sizes, and placed into a queue
 for moderation and then later use?

Or perhaps:
   * User uploads a video
   * That gets transcoded to a different video format ? (eg video
 contribution transcoded to flash video?)

In the latter two cases, this is something I've needed to do, so I created a
simple WSGI filter that dumps the uploaded images/videos into a standard
file location, and then had a script that watches that standard location
for new images & videos, and does those conversions and transcodes.

Strictly speaking to be "clean" about it, you need to dump the file to a
temporary location whilst you're grabbing it from the network and when it's
been written to disk rename it.

I've no idea if this maps to your problem at all. (since I can think of a
few scenarios that match your description).

If it does match, then this code may be handy:
http://code.google.com/p/kamaelia/source/browse/branches/private_MPS_Participate/Apps/FileProcessor/App/BatchFileProcessor.py

If you want to use that code, let me know and I'll package it up. There's an
older version here:
   * http://www.kamaelia.org/release/Kamaelia-FileProcessor-0.1.0.tar.gz

Regards,


Michael.
--
http://www.kamaelia.org/Home

--
http://mail.python.org/mailman/listinfo/python-list

ANN: Kamaelia 0.6.0, Axon 1.6.0 - New Release

[Michael Sparks]

ents.

   * Experimental multicore support
       - http://www.kamaelia.org/MulticoreExample
       - Largely boils down to put "Process" in front of "Pipeline" to
         make all the subcomponents of the pipeline run in seperate
         processes  
       - Practical benefit for pygame components - it allows multiwindow
         pygame apps.

   * Inheritable default values for component initialisers.
       - The core aim of this is to allow declarative config for
         systems rather than something less clear.
       - This allows you to turn this sort of code:
            def ReusableSocketAddrServer(port=100,
                                   protocol=EchoProtocol):
                return ServerCore(protocol=protocol,
                                  port=port,
                                  socketOptions=(socket.SOL_SOCKET,
                                                 socket.SO_REUSEADDR, 1))
       - Into this:
            class ReusableSocketAttrServer(ServerCore):
                socketOptions=(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

       - Full discussion here:
         http://www.kamaelia.org/InheritableDefaultValues

   * Added in "Handle" support. This provides two key pieces of
     functionality:
     - The ability to run Kamaelia systems in the background, via:
       from Axon.background import background
       background().start()

     - The ability to wrap Kamaelia components or systems in a Handle
       for use in non-Kamaelia systems.
       from Axon.Handle import Handle
       from Kamaelia.Internet.TCPClient import TCPClient

       conn = Handler(TCPClient("www.kamaelia.org", 80)).activate()
       conn.put("GET / HTTP/1.0\r\n", "inbox")
       conn.put("Host: www.kamaelia.org\r\n", "inbox")
       conn.put("\r\n", "inbox")

     - More detail:
       http://www.kamaelia.org/AxonHandle

   * Simplified system shutdown. If you want to close down an entire
Kamaelia based
     system rapidly from inside a component, just do this:

       - self.scheduler.stop()

       - This puts the scheduler into a "shutting down mode" in which:
           - It calls the .stop() method on all the things it's running
           - It then shuts down.

       - The Kamaelia.Internet components take this as an opportunity
         to close all the connections they have open cleanly for example.

       - Example usage: http://tinyurl.com/AxonShutdown

   * Support for WaitComplete extended, allowing better handling of
     more complex protocols which are not stateless in a debuggable
     fashion. It also simplifies working with Pygame, etc.

       - http://www.kamaelia.org/WaitComplete

   * As well as unpausing a component when a message is delivered to
     the component, it gets unpaused when a message is taken from its
     outbox. This allows better synchronous behaviour for rate limited
     in/out-boxes.

Overview of Changes to Kamaelia itself
==
Key changes to Kamaelia itself:
   * Creation of the Kamaelia.Apps namespace
   * Shifting of the core code for Kamaelia tools into Kamaelia.Apps
   * Significant numbers of new components
   * Significant number of bugfixes
   * SimpleServer code changed to "ServerCore", representing it's more
     general structure.

In this release there is a slew of extra components and bug fixes, a
variety of new tools - from video shot change detection, through to
SMTP greylisting, but also perhaps the biggest extra: Multiprocess &
hence multicore support (experimental at this stage, but so far so
good :) )

Since even the summary is not a short list of changes, I've left that
summary until after my .sig below.

Platforms
=

Kamaelia has been used successfully under both Linux, Windows and
Mac OS X. (mostly developed/tested under Linux & Mac OS X)

Where can I get it? & Docs?
===

Download:
    http://www.kamaelia.org/GetKamaelia
    http://www.kamaelia.org/release/Kamaelia-0.6.0.tar.gz

Docs:
    http://www.kamaelia.org/Docs/Axon/Axon
    http://www.kamaelia.org/Components
    http://www.kamaelia.org/Cookbook
    http://www.kamaelia.org/MiniAxon

Presentations:
    http://www.slideshare.net/kamaelian

Get involved:
    http://www.kamaelia.org/Developers/
    http://groups.google.com/group/kamaelia *CHANGED MAILING LIST*
    http://code.google.com/p/kamaelia/

Licensing
=

Kamaelia is released under the Mozilla tri-license scheme
(MPL1.1/GPL2.0/LGPL2.1). See http://www.kamaelia.org/Licensing

Finally, many thanks to everyone who's contributed to this release.

Best Regards,


Michael.
 --
Michael Sparks, Senior Research Engineer, BBC Research
[EMAIL PROTECTED], Kamaelia Project Lead, 
http://www.kamaelia.org/GetKamaelia
BBC Manchester


New files in Kamaelia.*
===
This represents significant amounts of 

Re: [Kamaelia] TCPClient: How to sense connection failure?

[Michael Sparks]

Bjoern Schliessmann wrote:

> Whoops, the TCP client does in fact quit if the server closes
> connection :) 

Great - so it wasn't a problem with the TCPClient after all :-)

> For some reason, my Listener doesn't quit. I thought 
> it's sufficient to exit the main method in some way to quit a
> component? That's what I do using "break" in the
> 'if self.dataReady("control")' part of main.

It is sufficient, and running with Kamaelia from /trunk, your listener does
indeed shutdown correctly - so it looks like it's a bug in that release of
Kamaelia.

That's my bad really because we haven't done a full release in quite some
time. 

Looking at the old code, it appears that in that code the TCPClient wasn't
forwarding shutdown messages correctly, so there *was* a bug, but there
doesn't seem to be now.

My suggestion for the moment would be to use the code on /trunk since this
is stable at present (development happens on branches rather than /trunk)
and that I really ought to sort out the next release - I really hadn't
realised just how long it had been since the last release!

steps:

~> svn co https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk
kamaelia-trunk
~> cd kamaelia-trunk
~/kamaelia-trunk> cd Code/Python/Axon
~/kamaelia-trunk/Code/Python/Axon> sudo python setup.py install
~/kamaelia-trunk/Code/Python/Axon> cd ../Kamaelia
~/kamaelia-trunk/Code/Python/Kamaelia> sudo python setup.py install

I've tested using this, and your code works as you'd expect.

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: [Kamaelia] TCPClient: How to sense connection failure?

[Michael Sparks]

Bjoern Schliessmann wrote:
> Hello,
> 
> I'm currently trying to implement a simulation program with Kamaelia
> and need a reliable TCP connection to a data server.

The behaviour you're seeing sounds odd (which is hopefully encouraging :-),
but it's not clear from the description whether its a bug in your code or
Kamaelia. One question I really have as a result is what version are you
using?

Current release version 0.5.0, the version on /trunk, or the bleeding edge
version on /branches/private_MPS_Scratch. (I'm using the latter to run my
greylisting server - as are a few others).

All that said though, looking at the differences between versions, I'm not
convinced they're large enough to show the problem you're seeing.

I'm not about to rule out a bug I don't know about though :-)

> From Twisted, I know that a method is called if the connection fails
> by whatever reason. I tried to get the same results with Kamaelia's
> TCPClient component. If I start up the component and try to connect
> to a closed TCP port it fails and sends a message out of the signal
> box, that's okay.

If you'd prefer more information in that message, please let me know.
(all components send out a message when they shutdown. For things that
send  data out as one of their primary actions send out a producerFinished
message)

> But if the connection attempt succeeds and, after some time, the
> server drops the connection with full TCP handshake (FIN, FIN+ACK,
> ACK), the component just hangs and does nothing. Is this by design,
> or could there be an error in my setup?

It sounds like an error in your setup... but I hate saying that. (Doesn't
tell me or you what it is, and doesn't help change things to discourage or
detect mistakes in usage)

When the server drops the connection in my setups, the client disconnects
cleanly when the server dies, with the client code looking like this:
  self.send(producerFinished(self,self.howDied), "signal")

Meaning you get a message telling you how the component shutdown as well as
the fact it shutdown. (If "howDied" is None, it's just a normal shutdown -
ie as a result of being told to shut down)

The function where this is managed is runClient in the class 
   Kamaelia.Internet.TCPClient.TCPClient
(fully qualified name)

The actual socket connections are handled by a class called
ConnectedSocketAdapter which manages all logic of checking for
errors, remote shutdown etc. That works the same for both servers
and clients so breakage in clients would show up as breakage in servers
as well, which would be particularly bad. 

> Also, how long does a TCPClient component live -- or how can/should
> I terminate it explicitly? I'm afraid that if I create
> a "ReconnectingTCPClient" component, it could eat up memory over
> long runtime with hanging TCPClients.

That shouldn't be an issue (I hate the word "should"), but you can do this
using a carousel component. (I ought to write that as an example of how to
use the Carousel component)

In the meantime - whilst I check to see if there's a bug I didn't know
about, the following 2 cookbook entries may be of use:
   * http://kamaelia.sourceforge.net/Cookbook/TCPSystems
   * http://kamaelia.sourceforge.net/Cookbook/Carousels - allows you to make
 something that exits reusable. It's a little awkward to get your head
 around, but is quite useful when you do. (I've heard of others using
 Carousel & TCPClient to make a reconnecting TCPClient in the past)

All that said, I'm not going to rule out a bug and look into it. (if you
have a simple example you find fails, please forward it to me :)

*thinks*

The following code may also be useful when debugging:

from Kamaelia.Chassis.Pipeline import Pipeline

class PeriodicWakeup(Axon.ThreadedComponent.threadedcomponent):
interval = 300
def main(self):
while 1:
time.sleep(self.interval)
self.send("tick", "outbox")

class WakeableIntrospector(Axon.Component.component):
def main(self):
while 1:
Q = [ q.name for q in self.scheduler.listAllThreads() ]
Q.sort()
print "*debug* THREADS"+ str(Q)
self.scheduler.debuggingon = False
yield 1
while not self.dataReady("inbox"):
self.pause()
yield 1
while self.dataReady("inbox"):
self.recv("inbox")

Pipeline(
PeriodicWakeup(),
WakeableIntrospector(),
).activate()

If you put this code somewhere before your "run" call, you'll get periodic
output to tell you what's running. When debugging manually I'd drop the
interval to 3-10 seconds or so. I use 300 for a server.

Now, off to see if I can reproduce your problem... :)

Regards,


Michael.
--
http://kamaelia.sourceforge.net/Home
http://yeoldclue.com/blog

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: cloud computing (and python)?

[Michael Sparks]

Aaron Watters wrote: (from a gmail account)
> So cloud computing is java diskless workstations warmed over but less
> flexible?
> 
> I'm having trouble understanding why people would want
> to buy in to this.

Why do you like gmail - since you appear to use it? (I can think of several
possibilities) The reason I ask Gmail is a an example of computing in the
cloud. Specifically it's an application in the cloud.

You get several classes of things "in the cloud" - one possible break up:
   * Applications - gmail, amazon, hotmail, facebook widgets, writely,
 blogger, flickr, etc.
   * Components - YUI, EC2, S3
   * Frameworks - open social, facebook 

etc. Each has benefits. Some examples:
   * gmail, hotmail, yahoomail - spam filtering, access your mail anywhere.
 You rent the application by paying with attention (or paying money - I
 think hotmail still do that)
   * S3 - scalable storage in the cloud WITH scalable serving. The trade off
 here is "how much does it cost you to run a colo box or dedicated
 server" vs "how much to rent the space". You rent capacity on demand.
 (a bit like "why buy storage at a self-storage place rather than buy a
 garage?" - there are good reasons both ways round :-)
   * EC2 - Similar, but to do with computing capacity.
 EC2 & S3 allow you to scale for example in line _and in time_ with the
 size of your userbase - assuming your business model (if you have
 one :-) matches
   * open social, facebook - rather than build your own social graph, you
 can attach yourself to an existing one to simplify take-up.

I must admit I feel a hint of amusement though at your comment above, when
it's sent from precisely the sort of setup you appear bemused by - since
you appear to have already bought into it without realising ! :-D

Have fun :-)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fiber(cooperative multi-threading)

[Michael Sparks]

Duncan Booth wrote:

> There are also problems where full blown coroutines are appropriate. The
> example I quoted earlier of turning a parser from one which generates a
> lot of callbacks to one which 'yields' tokens is the usual example given.

For completeness, I looked at the other half of the thread at the expat
parser, and decided to write that in the style we use for Kamaelia - it
ends up looking like this:

import xml.parsers.expat
import Axon
from Kamaelia.Chassis.Pipeline import Pipeline
from Kamaelia.Util.Console import ConsoleEchoer

class Parser(Axon.ThreadedComponent.threadedcomponent):
data = " Default "  # Can be overridden by kwargs as normal

def start_element(self, name, attrs):
self.send(("START", name, attrs), "outbox")

def end_element(self, name):
self.send(("END", name), "outbox")

def char_data(self, data):
data = data.strip()
self.send(("DATA", data), "outbox")

def main(self):
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = self.start_element
p.EndElementHandler = self.end_element
p.CharacterDataHandler = self.char_data
p.Parse(self.data, 1)
self.send(Axon.Ipc.producerFinished(), "signal")

Pipeline(
Parser(data="Hello world Woo"),
ConsoleEchoer(),
).run()

You'll note we don't use generators for "Parser" in this context. This also
isn't 100% identical to form you use since we don't turn this into an
iterator (obviously :).

We do also use 1 more thread than the greenlet approach though. Pipeline
& ConsoleEchoer are generator based though, as is the scheduler that
runs them :-)

Have fun :-)


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fiber(cooperative multi-threading)

[Michael Sparks]

Bjoern Schliessmann wrote:

> Michael Sparks wrote:
> 
>> All that said, my personal primary aim for kamaelia is to try and
>> make it into a general toolkit for making concurrency easy &
>> natural (as well as efficient) to work with. If full blown
>> coroutines turn out to be part of that c'est le vie :-)
> 
> I must admit I mostly didn't follow this thread, but it sparked my
> interest in Kamaelia. I already did the MiniAxon tutorial and I
> plan to try out Kamaelia with a project I had discussed here a
> while ago (which I had to suspend until recently because of not
> enough spare time; it's about simulating complex relais circuits).
> Please continue the great work.

Many thanks for your kind words - the work is continuing :-)

Also, I'd be interested in hearing how your project gets on - it sounds
like the sort of thing that Kamaelia should be able to help with. (If it
doesn't/can't, then it's a bug IMO :)


> Regards & merry christmas everyone,
> 
> 
> Björn
> 
> P.S.: In the MiniAxon tutorial, I noticed a formatting problem and a
> bunch of typos on one page but the feedback link doesn't work; are
> you interested in a list?

I'm always interested in feedback! The fact the feedback link doesn't work
for you is particularly useful - I'll look into that!

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fiber(cooperative multi-threading)

[Michael Sparks]

Duncan Booth wrote:
[ snip sections about why the single layer aspect can be helpful ]

> I'm happy with generators as they are: they're a great solution to a
> problem that most of us didn't realise we had until the solution came
> along. That doesn't mean that I wouldn't also like to have a separate
> coroutine library, but I would use it to replace situations that might
> otherwise have to use threads, not to replace generators.

Oh agreed. For that, it would probably be nice to see greenlets (or
something similar) make it into the standard library, but I suspect
that's too much of a jump at the moment.

I think for me, what we tend to do is prototype something as a stub
program, and either turn that into a threaded component or a generator
component. If I was to do something halfway I'd probably look at greenlets
at the moment, though so far we've not found a use for that. (which is
relatively suprising)

Any time it's looked useful, generally it's actually meant "spin this
control out to another smaller component". That's generally then resulted
in that spun out component being reusable.

For example, probable border case for coroutines is in this:
http://kamaelia.svn.sourceforge.net/viewvc/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py?revision=3684&view=markup

We do do something that implies full coroutine support would be useful -
specifically the calls:
 yield WaitComplete(self.getline(), tag="_getline1")
...
 yield WaitComplete(self.getline(), tag="getline2")

To:
   31 def getline(self):
   32 control_message = ""
   33 while 1:
   34 while not self.anyReady():
   35 self.pause();
   36 yield 1
   37 while self.dataReady("control"):
   38 control_message = self.recv("control")
   39 if isinstance(control_message, socketShutdown):
   40 self.client_connected = False
   41 if self.dataReady("inbox"):
   42 self.logging_recv_connection()
   43 return
   44 else:
   45 if not self.client_connected :
   46 self.breakConnection = True
   47 return
   48 yield 1

(This is of course faking full co-routines by getting the caller to call
something else temporarily)

The thing is, if we split that out to an external component, there's
potential benefits.

*shrug* IMO, code should be written for clarity first, so simply being able
to have this as pure co-routines may be useful. However, being able to tag
the calls like this:
 yield WaitComplete(self.getline(), tag="_getline1")
...
 yield WaitComplete(self.getline(), tag="getline2")

Meant that I could run an introspector:
  506 Pipeline(
  507 PeriodicWakeup(),
  508 WakeableIntrospector(),
  509 ).activate()

And was able to see that some clients were misbehaving at different time
points, but also seeing clearly able to see what state the component was
waiting/hanging in when dealing with broken clients, and able to add in
timeouts trivally. cf:

*debug* 'THREADS'['Axon.Microprocess.microprocess_7068_getline1', 
'Kamaelia.Chassis.Graphline.Graphline_7062',
'Kamaelia.Chassis.Pipeline.Pipeline_7',
... snip ...
'Kamaelia.Internet.ConnectedSocketAdapter.ConnectedSocketAdapter_7059', 
'Kamaelia.Internet.Selector.Selector_11',
'Kamaelia.Internet.TimeOutCSA.ActivityMonitor_7060',
'Kamaelia.Internet.TimeOutCSA.ResettableSender_7061',
'__main__.GreylistServer_8',
'__main__.PeriodicWakeup_5',
'__main__.TCPS_10',
'__main__.WakeableIntrospector_6']

Now you could of course design a coroutine system for python to do that
and I'd hope if anyone did, they'd think of it being useful to trace the
where a co-routine is currently stuck :-)

All that said, my personal primary aim for kamaelia is to try and make it
into a general toolkit for making concurrency easy & natural (as well
as efficient) to work with. If full blown coroutines turn out to be part
of that c'est le vie :-)

I think it's a bit like the arguments around lambda being limited to a
single expresson vs a throwaway def. Both are useful, being stuck with one
or the other is more limiting than is perhaps ideal.

> Thanks you. I really do intend most of my posts to be helpful.

Likewise :-)

Have fun :-)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: Axon.STM 1.0.1 (release) Minimalistic Software Transactional Memory (with examples)

[Michael Sparks]

mport Store
import random

def all(aList, value):
for i in aList:
if value != i:
return False
return True

class Philosopher(Axon.ThreadedComponent.threadedcomponent):
forks = ["fork.1", "fork.2"] # default for testing :-)
def main(self): # start here :-)
while 1:
X = self.getforks()
time.sleep(0.2)
self.releaseforks(X)
time.sleep(0.3+random.random())

def getforks(self):
gotforks = False
while not gotforks:
try:
X = self.store.using(*self.forks)
if all([ X[fork].value for fork in self.forks], None):
for fork in self.forks:
X[fork].value = self.name
X.commit()
gotforks = True
else:
time.sleep(random.random())
except Axon.STM.ConcurrentUpdate:
time.sleep(random.random())
print "Got forks!", self.name, self.forks
return X

def releaseforks(self,X):
print "releasing forks", self.name
for fork in self.forks:
X[fork].value = None
X.commit()

S = Store()
N = 5
for i in range(1,N):
Philosopher(store=S,forks=["fork.%d" % i ,"fork.%d" % (i+1)]).activate()

Philosopher(store=S,forks=["fork.%d" % N ,"fork.%d" % 1]).run()

Feedback

Feedback is very welcome, preferably via email to the Kamaelia List
    * [EMAIL PROTECTED]

Feedback especially regarding bugs and logical errors is particularly
welcome. (hopefully there aren't any - but it's always hard to spot your
own)

Thanks
==
Many thanks to Fuzzyman, Duncan Booth, John J Lee & Sylvain Hellegouarch for
feedback whilst I was prototyping this.

Further thanks go to Richard Taylor for detailed feedback and discussion
regarding locking and for pointing me at MASCOT which made me think of
doing the dining philosophers this way :-)

Future
==
This will be merged onto the mainline of Kamaelia with some auxillary
functions , as another feather aimed at making concurrency easy to
work with :-)

Best Regards,


Michael.
--
Michael Sparks, Kamaelia Project
http://kamaelia.sourceforge.net/Developers/
http://yeoldeclue.com/blog

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fiber(cooperative multi-threading)

[Michael Sparks]

Duncan Booth wrote:

> Michael Sparks <[EMAIL PROTECTED]> wrote:
> 
>> Duncan Booth wrote:
>> 
>>> Unfortunately generators only save a single level of stack-frame, so
>>> they are not really a replacement for fibers/coroutines. The OP
>>> should perhaps look at Stackless Python or Greenlets. See
>> 
>> On the surface of things, the single level aspect LOOKS like a
>> problem, but in fact is actually really useful. The reason is because
>> it encourages a generator to be relatively simple and focused
>> encouraging reuse.
>> 
> 
> Ah, perhaps Python should similarly limit function call nesting to one
> level so as to keep things simple and encourage reuse.

Bit of a grumpy response?  ... and a straw man argument ? Generators can be
used to build co-operative multitasking (eg WSGI is implicitly co-op
multitasking), whereas function calls are used to build sequential code. I
agree for sequential code generators being single level can be rather
limiting. However, the thread is about co-operative multitasking. 

Decades of experience has shown that the best way to build concurrent
systems (MASCOT[1], Unix, MPI, Occam, Erlang, even VHDL & some mashups)
is to have small things loosely connected. Generators encourage making
little things which then encourages connecting them together. It seems,
to me, to help. (Which is all I said really)

[1] Only recently heard about MASCOT - best reference is here:
http://async.org.uk/Hugo.Simpson/
(MASCOT is over 30 years old... (fun thing of the week that I've been
 pointed at for me :-) )

It's the same basic ethos that makes Unix pipelines useful, and is why
WSGI is useful. Simple transformational things which work best when they're
pipelined together. The fact that you can't trivially nest generators in a
way that feels nice[2] IMO naturally encourages making small things, which
end up being reusable. 

   [2] I don't personally feel for i in X(): yield i is particularly nice.

To be clear - I REALLY didn't like the fact that generators were single
layer when I first saw them - it seemed a huge limitation. (Indeed as huge
a limitation as only having single level function calls, or only single
layer of nesting for namespaces, etc)... I even asked a question [3] on
that point before choosing python to write Kamaelia in... The fact that
it's been a benefit rather than a problem rather surprised me.

[3]
http://groups.google.com/group/comp.lang.python/tree/browse_frm/thread/fcd2709952d23e34/8a0b9ba0e3beb108

That's rather different from the recursion limit (which is a limit on
function call depth) that really *has* been a pain at times. (Something
which tail recursion would really help with)

Incidentally, you may note that you helped me back then (which I'm thankful
for :-), so you can kinda view this as me reporting back "actually, it's
turned out to be helpful rather than a pain" :-) Took a while to figure
that out though ;)

Your mileage may vary :)

Incidentally, it was probably your response (message 7) in that thread
that was responsible for me deciding to see where things could go by
trying python :-)

Merry Christmas,


Michael. (hoping that's a suitably positive response :-)
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fiber(cooperative multi-threading)

[Michael Sparks]

Hi,


> It just works, but using native Python threads for non-preemptive
> threading is not cost-effective. Python has generator instead but it
> seemed to be very restricted for general scripting. I wish I could
> write nested (generator) functions easily at least.
> 
> Is there any plan of implementing real (lightweight) fiber in Python?

Please take a look at Kamaelia. Generators work extremely well, and are
extremely useful to work with:

   * http://kamaelia.sourceforge.net/Home

(I need to update/improve the website...)

If you *must* have nested generators (it's rarely useful) you can do that,
but just need to have your "thing running the generators" understand
a "don't run me next, run this next" message. Our message is called
WaitComplete, and you can see it used here (in a non-trivial useful
system):

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py

(I could point at trivial examples, but prefer something real)

The shortest overall description is here:
   * http://kamaelia.sourceforge.net/t/TN-LightTechnicalIntroToKamaelia.pdf

Cookbook examples:
   * http://kamaelia.sourceforge.net/Cookbook

List of dozens of reusable (all with each other) components:
   * http://kamaelia.sourceforge.net/Components

How to build your own (non-optimised) core:
   * http://kamaelia.sourceforge.net/MiniAxon/

Ruby (mini) version:  (for comparison :-)
https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk/Code/Ruby/miniaxon.rb

Experimental process based (as well as thread & generator based) version:

http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1196129474

(Going to move to process based components for static segmentation of an
application across process boundaries to give explicit multicore support.
May move to automatic distribution at some point, but static is an easy win
as the above example shows :-)

It's not as lightweight as stackless python's microthreads, but it's pretty
close. The scaling style is pretty similar as well - cf:

http://www.rhonabwy.com/wp/2007/11/13/generator-based-concurrent-programming-in-python/
I also blogged about that here:
http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1195688924

(Someone curious about comparing Kamaelia to stackless. Interesting to see
the same scaling curve, but unsurprisingly stackless wins :-) Kamaelia
however works with standard python from version 2.2.late onwards :-) )

Interestingly recently discovered that Kamaelia's been slowly reinventing a
system the UK's MOD have been using for around 30 years to make concurrent
systems easier to build (mascot) :-)

Merry Christmas :-)


Michael
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: fiber(cooperative multi-threading)

[Michael Sparks]

Duncan Booth wrote:

> Unfortunately generators only save a single level of stack-frame, so they
> are not really a replacement for fibers/coroutines. The OP should perhaps
> look at Stackless Python or Greenlets. See

On the surface of things, the single level aspect *LOOKS* like a problem,
but in fact is actually really useful. The reason is because it encourages
a generator to be relatively simple and focused encouraging reuse.

cf the components listed here:
   * http://kamaelia.sourceforge.net/Components

Also, because they're simple, you can link every component in that list
either directly with all the others or via a trivial filtering component.

Also the fact that you have to build your own scheduler means you can do the
equivalent of saying to the scheduler "Don't run me next, run this next".
This gives you pretty much the full power of co-routines but allows for
very heavy reusability and independence of testing. 

An example application that makes moderate use of this "bounce back"
coroutines is a greylisting server you can find here:
   * http://kamaelia.sourceforge.net/KamaeliaGrey

I'd (almost) argue that the single level aspect of generators is perhaps
their best feature.


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Passing by reference

[Michael Sparks]

[EMAIL PROTECTED] wrote:

> ... the first element of the list to which x refers is a reference to
> the new string and back outside foo, the first element of the list to
> which x refers will be a reference to the new string.

I'd rephrase that as:
   * Both the global context and the inside of foo see the same list
   * They can therefore both update the list
   * If a new string is put in the first element of the list, the can
 both see the same new string.

> Right?

You know you can get python to answer your question - yes? Might be slightly
more illuminating than twisting round english... :-)

OK, you're passing in a string in a list. You have 2 obvious ways of doing
that - either as an argument:

def foo(y):
y[0] += " other"
print id(y[0])

... or as a global: (which of course you wouldn't do :-)

def bar():
global x
x[0] += " another"
print id(x[0])

So let's see what happens.

>>> x = ["some string"]  # create container with string
>>> x[0]  # Check that looks good & it does
'some string'
>>> id(x[0])  # What's the id of that string??
3082578144L
>>> foo(x)# OK, foo thinks the new string has the following id
3082534160
>>> x[0]  # Yep, our x[0] has updated, as expected.
'some string other'
>>> id(x[0])  # Not only that the string has the same id.
3082534160L
>>> bar() # Update the global var, next line is new id
3082543416
>>> x[0]  # Check the value's updated as expected
'some string other another'
>>> id(x[0])  # Note that the id is the same as the output from bar
3082543416L

Does that perhaps answer your question more precisely ?


Michael.




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: state machine and a global variable

[Michael Sparks]

[EMAIL PROTECTED] wrote:

> Basically, I agree that often the local state is much more useful. It
> just seems to me that for some application it's an overkill. Like say,
> for Turtle [1] (no jokes, please :) or PostScript [2].

Sounds also a bit similar to what happens under the hood in Open GL and some
other systems which revolve around display lists. The approach generally
taken there is to recognise that you often have a "current" context which
is being operated on.

A basic version of this might look like this:

file: statemachine_user.py

#!/usr/bin/python

from statemachine import *

context = statemachine()
context2 = statemachine()

set_context(context)
set_state(3)
print get_state()

set_context(context2)
set_state(1)
print get_state()

set_context(context)
print get_state()


file: statemachine.py

#!/usr/bin/python

class statemachine(object):
def __init__(self):
self.state = None
def set_state(self, value):
self.state = value
def get_state(self):
return self.state

context = statemachine()

def set_context(somecontext):
global context
context = somecontext

def set_state(value):
context.set_state(value)

def get_state():
return context.get_state()

A more interesting example which probably relates closer to your example,
and also can be quite useful for interpretting little languages is where
the "states" that get stored are matrices representing transforms and are
used to put objects into a 3D space. 

You might in that circumstance want to use your a statemachine more like
this: (this code is untested)

class statemachine(object):
default = None
def __init__(self, **argd):
self.__dict__.update(**argd)
self.state = self.default
def set_state(self, value):
self.state = value
def get_state(self):
return self.state

context = [ statemachine() ] # default context

def push_context(somecontext):
global context
context.append( somecontext )

def pop_context(somecontext):
global context
return context.pop( -1 )

def set_state(value): context[-1].set_state(value)

def get_state(): return context[-1].get_state()

def get_allstates(): return [ x.get_state() for x in context ]

This isn't really quite what the do, but gives a different possible
perspective.


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/

-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: Axon.STM 1.0.0 (beta) Minimalistic Software Transactional Memory

[Michael Sparks]

Hi,


I've packaged up the minimal STM discussed over the past couple of days as a
standalone package which I've now uploaded

Getting it
==
You can download a beta test version here:
http://thwackety.com/Axon.STM-1.0.0.tar.gz

Previewing it
=
You can look at the sourcecode online here:

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Bindings/STM/Axon/STM.py

Installing it
=
~ > tar zxf Axon.STM-1.0.0.tar.gz
~ > cd Axon.STM-1.0.0/
~ > sudo python setup.py install

What IS it?
===
Software Transactional Memory (STM) is a technique for allowing multiple
threads to share data in such a way that they know when something has gone
wrong. It's been used in databases (just called transactions there really)
for some time and is also very similar to version control. Indeed, you can
think of STM as being like variable level version control.

Note: Because this is NOT intended to be persistent, this is not an ACID
store because it doesn't support the D - durability across a crash. (after
all, we don't save the state to disk) (The other aspects atomicity,
consistency & isolation are supported though)

I've written this to allow a part of Kamaelia to share & manage a dictionary
of atomic values between threads simply, and as a result this code is also
going into mainline Kamaelia. (Specifically into Axon Kamaelia's core)

However STM is something that should hopefully be of use to others doing
concurrent *things* whether or not they're using kamaelia, hence this stand
alone release.

This stand alone release should *not* be used alongside mainline Axon yet.
(Well you can, as long as you reinstall your Axon over the top, but that's
icky :-)

Why is it useful?
=
[ please skip this (or correct me :) if you understand concurrency
  already :-) ]

Why do you need it? Well, in normal code, Global variables are generally
shunned because it can make your code a pain to work with and a pain to be
certain if it works properly. Even with linear code, you can have 2 bits of
code manipulating a structure in surprising ways - but the results are
repeatable. Not-properly-managed-shared-data is to threaded systems as
not-properly-managed-globals are to normal code. (This code is one way of
helping manage shared data)

Well, with code where you have multiple threads active, having shared data
is like an even nastier version of globals. Why? Well, when you have 2 (or
more) running in parallel, the results of breakage can become hard to
repeat as two pieces of code "race" to update values.

With STM you make it explicit what the values are you want to update, and
only once you're happy with the updates do you publish them back to the
shared storage. The neat thing is, if someone else changed things since you
last looked, you get told (your commit fails), and you have to redo the
work. This may sound like extra work (you have to be prepared to redo the
work), but it's nicer than your code breaking :-)

The way you get that message is the .commit raises a ConcurrentUpdate
exception.

Also, it's designed to work happily in code that requires non-blocking
usage - which means you may also get a "BusyRetry" exception under load. If
you do, you should as the exception suggests retry the action that you just
tried. (With or without restarting the transaction)

Apologies if that sounds too noddy :)

Docs for it
===
http://kamaelia.sourceforge.net/STM

Using It


# Initialising a Store
from Axon.STM import Store

S = Store()

# Single values
greeting = S.usevar("hello")
print repr(greeting.value)
greeting.set("Hello World")
greeting.commit()
S.dump()

# Groups of values
D = S.using("account_one", "account_two", "myaccount")
D["account_one"].set(50)
D["account_two"].set(100)
D.commit()
S.dump()

D = S.using("account_one", "account_two", "myaccount")
D["myaccount"].set(D["account_one"].value+D["account_two"].value)
D["account_one"].set(0)
D["account_two"].set(0)
D.commit()
S.dump()

License
===
Take your pick of MPL V1.1, GPL 2.0, LGPL 2.1 :-)

Feedback

Feedback is very welcome, preferably via email to the Kamaelia List
* [EMAIL PROTECTED]

Feedback especially regarding bugs and logical errors is particularly
welcome. (hopefully there aren't any - but it's always hard to spot your
own)

Thanks
==
Many thanks to Fuzzyman, Duncan Booth, John J Lee & Sylvain Hellegouarch for
feedback whilst I was prototyping this.

Best Regards,


Michael.
--
Michael Sparks, Kamaelia Project
http://kamaelia.sourceforge.net/Developers/
http://yeoldeclue.com/blog

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Minimalistic Software Transactional Memory

[Michael Sparks]

John J. Lee wrote:

> Durus might be worth a look too (though I doubt it's suitable for your
> situation):
> 
> http://www.mems-exchange.org/software/durus/
> 
> The link to their paper about it seems to be broken, but I think it
> was based somewhat on ZODB, but is simpler (67k tarball :-).

Much appreciated. I've downloaded this and it looks more suitable,
however it still looks like overkill... After all, I'm just after a
simple system for concurrent update of in-memory shared values by
multiple threads - how hard can that be ?[1] ;-) :-)

 [1] Yes, yes, for those who don't know me, I know, I know... :-)

Thanks to Fuzzyman's comments and a code review on IRC I think I've got what
I think is a minimally sufficient system - though I'll extend to include
Fuzzyman's suggestion regarding concurrent update of multiple independent
values :-)

Code is here for those curious/wanting something similar/similarly
lightweight:

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/trunk/Sketches/MPS/Experiments/NewSTM.py

I'll package it up independently of the rest of Kamaelia as well probably
once I've made the addition noted above :-)

Regards,


Michael
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Minimalistic Software Transactional Memory

[Michael Sparks]

Fuzzyman wrote:

>> STM seems more in
>> keeping with Kamaelia being generally lock-free.
> 
> STM isn't lock free - it just abstracts the locks away from the
> 'user'. You still need to lock around committing the transaction.
> 

I perhaps phrased what I meant too tersely.

Kamaelia isn't lock free either. It's generally lock free. User code
however, doesn't ever see locks... (Though threaded components do
communicate over thread safe Queues which contain locks internally.
Generator components don't use locking)

I suppose the reason why I like STM (conceptually) is because it appears to
follow the same sort of idea - the user of the code doesn't have to worry
about locks - the just have to worry about whether they're using the system
correctly or not. :-)

Based on this, I think I'm right in thinking then that given the code I
posted, that at minimum I need locking here, since it is the critical
section.

def set(self, key, value):
# Attempt to acquire lock
if not (self.store[key].version > value.version):
# Also assuming we have lock
self.store[key] = Value(value.version+1,
copy.deepcopy(value.value), self, key)
value.version= value.version+1
# Release lock
else:
# Also do this if we can't get access to the lock
raise ConcurrentUpdate

I know I then have a choice of locking the value (ie a lock specific to
self.store[key]) or the self.store as a whole. Depends on how antisocial
you want the locking to be I suppose :-)

> Other threads accessing the data during the transaction should see the
> old values until the commit is successful.

The implementation I'm suggesting means that other threads would see the
old values until the try to commit their changes. (I'm after something
simple since I'm expecting to deal with simple cases)

Many thanks :)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Minimalistic Software Transactional Memory

[Michael Sparks]

Duncan Booth wrote:

> Michael Sparks <[EMAIL PROTECTED]> wrote:
> 
>> I'm interested in writing a simple, minimalistic, non persistent (at
>> this stage) software transactional memory (STM) module. The idea being
>> it should be possible to write such a beast in a way that can be made
>> threadsafe fair easily.
>> 
>> For those who don't know, STM is a really fancy way of saying
>> variables with version control (as far as I can tell :-) designed to
>> enable threadsafe shared data.
>> 
>> I'm starting with the caveat here that the following code is almost
>> certainly not threadsafe (not put any real thought into that as yet),
>> and I'm interested in any feedback on the following:
>> 
>> * Does the API look simple enough?
>> * Are there any glaring mistakes in the code ? (It's always harder
>> to see
>> your own bugs)
>> * What key areas appear least threadsafe, and any general
>> suggestions
>> around that.
>> 
>> If I get no feedback I hope this is of interest. Since these things
>> get archived, if you're reading this a month, 6 months, a year or more
>> from now, I'll still be interested in feedback...
> 
> Unless you really are desperate to reinvent the wheel, have you looked at
> ZODB? https://launchpad.net/zodb
> 
> ZODB gives you the transactional model you want. It also gives you
> persistence, but if you don't want that you can simply connect to a non-
> persistent store.

That seems somewhat overkill for my needs. ZODB's distribution is 3.3MB in
size, whereas the system I want a minimalistic, non-persistant[1]
transactional memory for is 345K in size. (The Axon part of kamaelia)
I'll take a look though.

[1] I said "at this stage" because its something I *may* want in future
for some possible usecases, but generally I'm not really very interested
in persistence. (At least not where I'm putting this :-)

Context: I want thing that use the following class threadsafe, and can think
of a few ways of doing this, and STM seems one of the more "nicer" ways of
doing so. (it's clients are currently just generators which makes it safe at
the moment)

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Axon/Axon/CoordinatingAssistantTracker.py

It's used to provide an ENVironment like facility (similar to os.environ)
for generator based Kamaelia components.

Thanks for the feedback :-)


Michael.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Minimalistic Software Transactional Memory

[Michael Sparks]

 100
E.set(Y)
E.commit() # Second
S.dump()

X["account_two"] = 0
D.set(X)
D.commit()  # Third
S.dump()
print "Committed", D.value["myaccount"]

Fails as follows:

accounts : Value(1, {'account_two': 100, 'myaccount': 0, 'account_one': 50})
accounts : Value(2, {'account_two': 100, 'myaccount': -100, 'account_one':
100})
Traceback (most recent call last):
  File "./NewSTM.py", line 70, in 
D.commit()  # Third
  File "./NewSTM.py", line 20, in commit
self.store.set(self.key, self)
  File "./NewSTM.py", line 37, in set
raise ConcurrentUpdate
__main__.ConcurrentUpdate

(which is of course the error wanted - since we want to be able to detect
failure)

It's probably useful to know for the more general approach you suggest.

Thanks!


Michael.
--
Michael Sparks, Kamaelia Project Lead
http://kamaelia.sourceforge.net/Developers/
http://yeoldeclue.com/blog

-- 
http://mail.python.org/mailman/listinfo/python-list

Minimalistic Software Transactional Memory

[Michael Sparks]

Hi,


I'm interested in writing a simple, minimalistic, non persistent (at this
stage) software transactional memory (STM) module. The idea being it should
be possible to write such a beast in a way that can be made threadsafe fair
easily.

For those who don't know, STM is a really fancy way of saying variables
with version control (as far as I can tell :-) designed to enable threadsafe
shared data.

I'm starting with the caveat here that the following code is almost
certainly not threadsafe (not put any real thought into that as yet),
and I'm interested in any feedback on the following:

   * Does the API look simple enough?
   * Are there any glaring mistakes in the code ? (It's always harder to see
 your own bugs)
   * What key areas appear least threadsafe, and any general suggestions
 around that.

If I get no feedback I hope this is of interest. Since these things get
archived, if you're reading this a month, 6 months, a year or more from
now, I'll still be interested in feedback...

OK, API.

First of all we need to initialise the store:

S = Store()

We then want to get a value from the store such that we can use the value,
and do stuff with it:

greeting = S.using("hello")

Access the value:

print repr(greeting.value)

Update the value:

greeting.set("Hello World")

Commit the value back to the store:

greeting.commit()

If you have concurrent updates of the same value, the following exception
gets thrown:
ConcurrentUpdate

cf:
>>> S = Store()
>>> greeting = S.using("hello")
>>> par = S.using("hello")
>>> greeting.set("Hello World")
>>> par.set("Woo")
>>> greeting.commit()
>>> par.commit()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 12, in commit
  File "", line 11, in set
__main__.ConcurrentUpdate

That's pretty much the simplest API I can come up with. (I've tried a few
others but didn't really like them)

The way this works is we have a Store that manages Values. (perhaps a better
name may be variables to avoid clashing with pythons parlance of labels and
values?) 

Anyhow, you can ask the store for Value, which it will give you. The Value
knows what it's called and where it's from, so when you tell it to commit,
it can try to do so. The little detail here is you get a /copy/ of the
Value not the stored Value. (This is to avoid accidental concurrent update
of the same actual object)

As a result, Store looks like this:

class Store(object):
def __init__(self):
self.store = {}

def get(self, key):
return self.store[key].clone()

def set(self, key, value):
if not (self.store[key].version > value.version):
self.store[key] = Value(value.version+1, value.value, self, key)
value.version= value.version+1
else:
raise ConcurrentUpdate

def using(self, key):
try:
return self.get(key)
except KeyError:
self.store[key] = Value(0, None,self,key)
return self.get(key)

def dump(self):
for k in self.store:
print k, ":", self.store[k]

You'll note that the set method is the greatest candidate for any possible
race hazard here - though I can see a possible boundary issue in "using".
(I think :-)

Otherwise I think the above code is relatively straightforward. You'll note
that this API allows this:

greeting.set("Hello")
greeting.commit()
greeting.set("Hello World")
greeting.commit()
greeting.set("Hello World. Game")
greeting.commit()
greeting.set("Hello World. Game Over")
greeting.commit()

The other class is value that looks like this:

class Value(object):
def __init__(self, version, value,store,key):
self.version = version
self.value = value
self.store = store
self.key = key

def __repr__(self):
return "Value"+repr((self.version,self.value,self.store,self.key))

def set(self, value):
self.value = value

def commit(self):
self.store.set(self.key, self)

def clone(self):
return Value(self.version, self.value,self.store,self.key)

To me this looks like a pretty complete minimalistic thing, which does seem
to work OK, but I'm interested in the three points I mention above - if
anyone is willing to comment - specifcally:

   * Does the API look simple enough?
   * Are there any glaring mistakes in the code ? (It's always harder to see
 your own bugs)
   * What key areas appear least threadsafe, and any general suggestions
 around that.

Full code below.

Many thanks for any comments in advance,


Michael
--
Michael Sparks, Kam

Multiple Windows in Pygame, using true concurrency in python via Kamaelia

[Michael Sparks]

Hi,


Just thought some people may be interested to hear that I've recently been
looking at adding true concurrency into Kamaelia, by using Paul Boddie's
pprocess as the core mechanism to allow us to run multiple Kamaelia systems
in the same app. (Since we have thread based, and co-operative generator
based components, process based is the next logical step)

   [1] http://kamaelia.sourceforge.net/Home
   [2] http://pypi.python.org/pypi/pprocess

Anyhow, first proof of concept is to take a _relatively_ non-trivial app and
a library hostile to being run in a single process and run it across
multiple processes trivially. (I can't test multicpu since I don't have a
multicpu machine, but this should work fine with a modern OS)

So the app I've chosen is a simple pygame based drawing/doodling app, and to
get it to launch 8 pygame windows from 1 app. Since neither SDL nor pygame
directly support this it struck me as not just a visible proof of concept,
but a useful one too (since pygame rocks! :-)

Anyhow, I've written up the code here and posted a screenshot here:
   http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1196129474

You'll note 8 pygame windows ... :-)

The original base test is here:
   http://yeoldeclue.com/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1196029230

For those curious who don't like clicking on links, the code needed to allow
the 8 pygame windows looks like this:

  class SecondProcessBasedComponent(SimplestProcessComponent):
def main(self):
from Kamaelia.UI.Pygame.Display import PygameDisplay
from Kamaelia.UI.Pygame.MagnaDoodle import MagnaDoodle

X=PygameDisplay(width=200,height=200).activate()
PygameDisplay.setDisplayService(X)
MagnaDoodle().run()
yield 1

  exchange = SecondProcessBasedComponent().activate()
  R = []
  for _ in xrange(7):
 R.append(SecondProcessBasedComponent().activate())

So far, so good :-)

Incidentally, yes the project is alive and well, but a resource crunch
at work means I'm working on this only on my own time at the mo, which
means things like releases are having a longer cycle between them than
I'd like. Some recent developments include a greylisting proxy server,
a ER modelling tool and a (very) simple pygame based logo like tool for
teaching programming to small children. The webserver is aiming for WSGI
compliance, and the project also had a small number of students working
over the summer, assisting with some useful things like AIM/IRC integration,
the ability to use Kamaelia systems in non-Kamaelia based scripts/systems,
and looking at a sub-component model. (ie non-concurrent systems)


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Home

-- 
http://mail.python.org/mailman/listinfo/python-list

Greylisting with Kamaelia (was Re: Python North-West meeting - 6 november 18.30)

[Michael Sparks]

Giacomo Lacava wrote:

> New meeting of the Python North-West UK community!
> 
> This month's talk is:
> - Michael Sparks on "Greylisting with Kamaelia" -

Just a small note that the slides from this are now up here:
   http://www.slideshare.net/kamaelian/kamaelia-grey

With the main page on the Kamaelia site for it here:
http://kamaelia.sourceforge.net/KamaeliaGrey

(Which has the slideshare embedded as well for those who prefer one URL :-)

One para summary:

This presentation was given at Python North West. It explains a complete
Kamaelia application for greylisting which was written specifically to
eliminate my personal spam problem. It walks through the code as well
(though that's best looked at with the code side by side!)


Michael.
--
http://yeoldeclue.com/blog
http://kamaelia.sourceforge.net/Developers/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python component model

[Michael Sparks]

 in
> .Net a control is a component with other added qualities. So while one
> could build components which are not controls, it is necessary to add
> functionality to the base level idea of a component in order to create a
> control.

You *may* also want to take a look at picolo as well then - but as far
as I'm
aware it's not actually *used* by anyone. It is in some respects more
like
the kind of component model you describe here. (I personally didn't
find much
useful about their proposal that goes beyond what python already
provides
you) However you might find that and some of the other things on the
following
link interesting: http://www2.lifl.fr/~marvie/software.html

It's worth bearing in mind though that your description above is one
approach
for component based design. A survey of different approaches which you
might find useful:

> Thanks for all the link regarding kamaelia.

Probably went a bit overboard there :)

However I do agree that a visual system is something important, since
not everyone thinks the same way. (I can talk about our system till the
cows come home, show people code, but when I show them the visual
builder, everyone seems to understand).

Regards,


Michael.
--
Michael Sparks, Kamaelia Dust Puppy
http://kamaelia.sf.net/
http://yeoldeclue.com/blog

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Help me use my Dual Core CPU!

[Michael Sparks]

Paul Rubin wrote:
> "Ramon Diaz-Uriarte" <[EMAIL PROTECTED]> writes:
> > You might also want to check
> > http://www.lindaspaces.com/products/NWS_overview.html
> > by the guys who "invented" Linda.
>
> Cool, I guess.
>
> > (The Oz language/Mozart system is a good example of a different and
> > very neat approach to concurrency; somewhat similar Python solutions
> > can be found at Kamaelia and Candygram. Links and other stuff at:
>
> I looked at these.  Oz/Mozart is a whole nother language, worth
> examining for its ideas, but the implementation is quite slow.
> Kamaelia doesn't attempt concurrency at all.  Its main idea is to use
> generators to simulate microthreads.

Regarding Kamaelia, that's not been the case for over a year now.

We've had threaded components as well as generator based ones since
around last July, however their API stablised properly about 4 months
back. If you use C extensions that release the GIL and are using an OS
that puts threads on different CPUs then you have genuine concurrency.
(those are albeit some big caveats, but not uncommon ones in python).

Also integrating things as a sub process is as simple instantiating a
component that talks to the subprocess over stdin/out to the
inbox/outbox model of Kamaelia and then just using it. Something
concrete this is useful for:
mencoder_options = "-ovc lavc -oac mp3lame -ffourcc DX50 -lavcopts
acodec=mp3:vbitrate=200:abitrate=128 -vf scale=320:-2 -"
...#  assume 'encodingfile' is defined above
Pipeline( DVB_TuneToChannel(channel="BBC ONE",fromDemuxer="MUX1"),
  UnixProcess("mencoder -o "+encodingfile+"
"+mencoder_options)
).run()

On a dual CPU machine that code does indeed both use CPUs (as you'd
want and expect).

Also whilst we haven't had the chance to implement OS level process
based components, that doesn't mean to say we're not interested in
them, it's just that 2 people have to focus on something so we've been
focussed on building things using the system rather than fleshing out
the concurrently. To say we don't attempt implies that we don't want to
go down these routes of adding in genuine concurrency. (Which is really
why I'm replying - that's not the case - I do want to go down these
routes, and it's more man-hours than desire that are the issue).

Personally, I'm very much in the camp that says "shared data is
invariably a bad idea unless you really know what you're doing"
(largely because it's the most common source of bugs for people where
they're trying to do more than one thing at a time). People also
generally appear to find writing threadsafe code very hard. (not
everyone, just the people who aren't at the top end of the bell curve
for writing code that does more than one thing at a time)

This is why Kamaelia is message based (ie it's a concious choice in
favour), except for certain types of data (where we have a linda-esque
type system for more systemic information). The reason for this is to
help the average programmer from shooting himself in his own foot (with
a 6 CPU-barrelled shotgun :-).

In terms of how this is *implemented* however, we have zero copying of
data (except to/from threads at the moment) and so data is shared
directly, but in a location the user of the system thinks its natural
to have handoff to someone else. This approach we find tends to
encourage arbitration of access to shared resources, which IMO is a
good (defensive) approach to avoiding the problems people have with
shared resources.

But if it turns out our approach sucks for the average programmer, then
that's a bug, so we'd have to work to fix it. And if new approaches are
better, we'd welcome implementations since not all problems are screws
and not all tools are hammers :-) (as a result I'd also welcome people
saying what sucks and why, but preferably based on the system as it is
today, not as it was :)

Have fun :)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Invitation: Kamaelia Open Space, Brussels, Sept 22 2006

[Michael Sparks]

Hello,


I'd like to invite you to our first Kamaelia Open Space event.

Our theme is "Making Software like Lego through intuitive useful
concurrency".

Perhaps you want to learn to use Kamaelia, or you're already using it,
or you're simply interested in code reuse or concurrency being actually
useful in everyday code rather than theory. If these match your interests,
please come along.

WHEN

The day after Euro OSCON, and just before Bar Camp Brussels:
   * Friday 22nd September 2006 - 11am - 5pm

WHERE
   * FOAM Offices, Brussels, Belgium
   * Koolmijnenkaai 30-34, 1080 Brussels, Belgium
   * http://fo.am/contact.html
   * http://tinyurl.com/hsarl (link showing location on multimap)

WHAT

http://kamaelia.sourceforge.net/Home

Kamaelia started as an applied research project, at BBC Research, and
has an underlying goal of making software systems easier to maintain
and create through the use of intuitive and safe concurrency, in the
form of components, whilst maintaining performance. It's being fleshed
out by creating systems useful in the real world.

This approach turns software into systems similar to a number of
systems people may be familiar with : Unix pipes (except non-linear
is practical), CSP, hardware (especially HDL's), Occam, K'nex and Lego.
Components send messages out via outboxes, receive messages on inbox,
have zero copy delivery and messages may be any python object including
a component.

Practical systems using Kamaelia developed to-date include time-shifting
digital TV and a variety of network systems, however components exist
simplifying the use of audio, video, pygame & Open GL.

PURPOSE

The purpose of the event is similar to a python sprint. Our aims for this
event are to spread what we've learnt with the wider community - for the
simple reason we're finding it works for us, and hope it does for you too.
This includes - but not exclusively:

   * To assist people to get started using Kamaelia :-)

   * Help you copy or tailor our approach to your systems.
 (eg making our system or approach usable in twisted).

   * To investigate areas where we can flesh Kamaelia out 
 (eg we have a webserver than can run in a similar manner to
 seaside, how would you like that fleshed out)

   * We'd like to assist others using other languages take the lessons
 we've learnt and apply them in their preferred language.
 (especially Ruby and C++)

   * Help attendees integrate their projects with Kamaelia encouraging
 reuse between projects, bosting the system for everyone
 (eg integrate pygtk, wxwidgets, pyqt, or other projects you may
 use or contribute to, building on experience from integrating tk,
 pygame and open gl)

   * To share the work done by students during Google's Summer of Code

   * How to apply this to teaching children to create these systems
 effectively. (We've noticed that pre-university users of Kamaelia tend
 to achieve the most using it)

   * Use of Kamaelia for art and entertainment 

At the other extreme, we could also like to use the time to share
information (if people are interested) on how some of the practical
systems we use work since they are likely to be useful to others.
Two interesting areas:

   * Distributed whiteboarding including audio. This is served in an
 effectively peer to peer manner - extending this to include grid
 setup (and/or DHT search) would be useful. This is potentially
 interesting beyond simple whiteboarding since this really forms a
 simple distributed events backplane

   * Practical flexible timeshifting either entire channels, and
 multiplexes or based on programme names

Nascent areas perhaps of interest which are include in Kamaelia include
a handwriting recogniser (currently at stroke/letter recognition level),
graphical systems creation, and a basic open GL toolkit.

FORMAT

The above list of suggestions for discussion is just that, suggestions.
The specific agenda for the event will be decided on the day by those
who attend.

Since much of what's to be discussed will be new, this will be a cross
between a python sprint and open space in format.

THANKS TO

Finally, many thanks to the very kind people at FOAM - http://fo.am/ for
hosting this event, it is very much appreciated.

WHAT NEXT

If you're interested in coming, please email your interest either to me at:
[EMAIL PROTECTED]

Or to the kamaelia mailing list:
[EMAIL PROTECTED]

If you know someone who you think is interested in the theme, could
benefit from coming - for example someone interested in making practical
concurrency safer and easier to use in future - please don't hesitate
to forward this invitation to them.

Hope to see you there!


Michael
--
Michael Sparks, Senior Research Engineer, BBC Research, Technology Group
[EMAIL PROTECTED], Kamaelia Project Lead, http://kamaelia.sf.net/

This message may cont

Re: beta.python.org content

[Michael Sparks]

Kay Schluehr wrote:

> The new Python site is incredibly boring. Sorry to say this. The old
> site is/was amateurish but engaged. Now after ~15 years of existence
> Pythons looks like it wants to be popular among directors of a german
> job centers. It aims to do everything right but what could be said
> worse? The text on the beginners page tries to argue with the
> potential users in a pointless monologue. Who wants to read this text?
> Who wants to be convinced that Python is *not* slow? "Do you stop
> beating your wife?" And where is fun, irony and black humour? Why
> Python? "Python in industry" - I see chimneys of 19th century
> factories, proletarian heroes as well as futuristic hybrid robots
> superseeding humanity. "Python community" - a dutch grand-family photo
> from the beginning of the 20ths century - some ( or all? ) of the
> members are accidentally looking like Guido, "Python in science" -
> snake-like RNA strand. It need not be like this but I wonder about the
> total lack of personality.

+1

This doesn't show the vibrance of to me somewhere like Europython.
It doesn't show "Oh, and this is incredibly, cool, fun and useful". That
to me is what both http://www.turbogears.org/ and
http://www.rubyonrails.org/ do show.

RonR has 4 very simple engaging points:
   * Get Excited
   * Get Started
   * Get Better
   * Get Involved

Blindly copying something else is rarely IMO a good idea, but having
NO personality simply turns people off. I'm not sure how to fix this,
but I'd suspect starting with the visual humour style in python might
be a good place to start...

Put another way, I'd expect *python.com* [1] to look like the new site,
and *python.org* to look like something that, well, represents some of
the utter (very cool, very diverse) madness of the people involved with
python.

   [1] NB: I **really** wouldn't go to python.com, I REALLY wasn't
expecting that (REALLY)


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

Totally my opinion and no-one elses and expecially not
the views of the BBC (!)

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: - E04 - Leadership! Google, Guido van Rossum, PSF

[Michael Sparks]

On Wednesday 28 Dec 2005 17:58:33, Robert Kern wrote:
>
...
Sorry to reply to the thread so late in the day, but I noticed (via
QOTW :-( ) that Anton got worked up at me suggesting that congratulating
someone with a new job was a nice idea (surprised me too - all the
Google employees I've met have been very nice people), read the
thread (got sad) and then saw this:

> Who is "the man"? If Google were to hire you with no experience, would 
> you then have "worked for the man"?

If you want to understand the reference, the most fun recent
explanation I've seen is in "School of Rock". Jack Black's character
explains it far better than I ever could :)

Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Python on GP2X (Linux Based Handheld Console)

[Michael Sparks]

Hi,


I hadn't seen any announcements regarding this, but there's a little
device recently released called a GP2X which is a small dual CPU
(2x200Mhz) device which runs Linux.

Anyway, I thought there might be someone in here interested to hear
that python AND pygame have both been ported to it already (not by
me). I've also ported some of our code to it (the bouncing cats demo
I've shown at a couple of python conferences), and found it really nice
to work with so far. The cats bounce at an acceptable (to me)
framerate :-)

Link for python:
http://gp2x-emulation.dcemu.co.uk/python.shtml

Link for device:
http://www.gp2x.com/product/product.asp

Just thought I'd post about this, since I thought someone might find it
useful :)


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message contains personal views which are not the views of the
BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: Kamaelia 0.3.0 released!

[Michael Sparks]

Kamaelia 0.3.0 has been released!

Introduction


Kamaelia is a networking/communications infrastructure for innovative
multimedia systems. Kamaelia uses a component architecture designed to
simplify creation and testing of new protocols and large scale media
delivery systems. A subset of the system has been tested on series 60
phones.

General feedback is welcome either directly, mailing lists or via IRC
(#kamaelia, freenode). People are also more than welcome to use the
system and suggest improvements not directly related to our specific
goals, because we recognise the system can be used in more areas, not
just networked multimedia. A diversity of systems built improves the
system (for all users of course!) as a whole.


What's New & Changed?
=

*NOTE* Kamaelia 0.3.0 requires Axon-1.1.2 to run for some newer
components
  (which has also just been released).

Full release notes and change log:
* http://kamaelia.sourceforge.net/Kamaelia-0.3.0-ReleaseNotes.html

New Examples - 7 new examples including:
* Simple reliable multicast based streamer using Ogg Vorbis.
* Dirac Player
* Dirac encode & playback.
* Simple bouncing images game. Designed for very small children who
  are amused by things take beep and react when you press left/right
  mouse buttons.
* Simple example showing how to use the ticker (First developed for
  showing subtitles).
* Demonstration system showing how to use the new software chassis
  facility in the context of multiple chassis. 

New Tools, Notable Additions

* Visual tool for building Kamaelia pipelines
* Tk Support
* Video encode, decode and playback. (dirac)

New Packages & Subsystems

These names should provide you with a flavour of the new subsystems that
have been added:

* Kamaelia.Codec
* Kamaelia.Chassis
* Kamaelia.File
* Kamaelia.UI.Tk
* Kamaelia.Internet.Simulate

Other Highlights

* Software chassis (software backplane will be coming in
Kamaelia-NEXT)
* Tk integration. (The pipeline builder is a nice example of a tool
this
  enables)
* Dirac encoded video decoders and encoders
* Support for video playback. (dirac & YUV)
* Variety of pygame based components, including
* Tools for greater control over the pygame surface managed
environment
* Tools for building simple games. (controlling sprite behaviour for
  example)
* Much richer tools for file reading and writing
* Includes re-usable file readers.
* More utilities for message filters and splitting of messages
* Basic tools for simuluating error conditions and failure rates
  for delivery of messages (Sufficient for simulating an unstable
  underlying internet infrastructure).


What is Kamaelia?
=

The project aims to make it easy to build networked multimedia
systems (eg audio, video, interactive systems). The result is systems
which are naturally componentised. Also, the resulting systems are
/naturally concurrent/ allowing quick and fast reuse in the same way
as Unix pipelines do. 

It is designed as a practical toolkit, such that you can build systems
such as:
   * Ogg Vorbis streaming server/client systems (via vorbissimple)
   * Create Video players & streaming systems (for dirac).
   * With subtitles.
   * Simple network aware games (via pygame)
   * Quickly build TCP & Multicast based network servers and clients
   * Presentation tools
   * A networked audio mixer matrix (think multiple audio sources over
 network connections mixed and sent on to multiple locations with
 different mixes)
   * Look at graph topologies & customise the rules of display &
 particle types.
 Mix and match all of the above.

You can also do a lot of this *visually* using the new PipeBuilder
application in Tools.

Essentially if the system you want to build involves audio or moving
pictures, and you want to be able to make the system network aware,
then this should be quick and easy to do using Kamaelia. (If it isn't,
then a) it's a bug b) needs improving :-)

It runs on Linux, Window, Mac OS X with a subset running on Series 60
phones.

The basic underlying metaphor of a component us like an office worker
with inboxes and outboxes, with deliveries occuring between desks,
offices, and depts. The component can thus do work anyway it likes but
only communicates with these inboxes and outboxes. Like office workers,
components run in parallel, and to achieve this are generally
implemented using python generators, but can also used threads.

The rationale behind the project is to provide a toolkit enabling the
development of new protocols, including streaming, for large scale
media delivery. The license essentially allows use in proprietary
systems without change, but all changes to the system itself must be
shared.

Oh, and due to things like the visual editor, the use of pygame in
a lot of examples, the use of dirac & vorbis, it's qu

Re: Crypto.Cipher.ARC4, bust or me doing something wrong?

[Michael Sparks]

Paul Rubin wrote:
> Michael Sparks <[EMAIL PROTECTED]> writes:
>> Rather than re-inventing wheels I thought I'd pick a library sit down
>> and see how pycrypt's meant to be used before actually going anyway.
>> (Amongst other reasons, this is why I suspected me, rather than the
>> library :-)
> 
> Pycrypt doesn't operate at anything like the level you need.  It just
> gives you low level cipher primitives.  You need higher level protocols.

Agreed. As I say I'm building up systems slowly on this front, and part of
the aim will be to show how outright dumb it is to build these things
without a decent understanding of the underlying tech.

Sadly there's many people (where I work) who don't understand how easy
it is to get these things wrong, and unfortunately practical demonstration
is one of the few languages people listen to :-/

This is where I border on being unprofessional so I'll be quiet at that
stage :-((

On a more positive note, as I say I picked something simple initially, and
have no intention at this stage of it being the primary approach.

>> FWIW, I'm well aware how easy it is to get cipher/digest/etc based
>> security/id systems wrong. I'm really starting with pycrypt because it
>> looked simple enough, low level enough and self contained enough to
>> act as a base for working with existing more complex systems.
> 
> Do yourself a favor and stick to something standard like TLS, rather
> than cook up your own protocol.  There are some Python wrappers for
> OpenSSL or GNU TLS, for example.

If those operated over multicast then they'd be an option... However...

Incidentally our primary use is to indicate *integrity* of data not to
prevent content being viewed. It's also obviously susceptible to attack
in many ways. Again, I'd say more offline about this - Usenet is just the
wrong place for it.

>> Anyway, once I've gone through all of the existing digests/ciphers/PK
>> ciphers, I'll post the snippets up on our site as raw examples for
>> pycrypto, which will hopefully be a) correct usage b) be useful to
>> others.
> 

> You really need to know a lot more than it sounds like you know, 

*shrug* It's not a matter of what I don't know it's a matter of showing how
things can be used, and I want to use it (internally) to demonstrate what
goes wrong when people try to design their own systems naively.

One way to learn is to build. 

> to have any chance of getting fancy protocol designs correct.

FWIW, I have little intention of inventing something new (in terms of
protocols) /if/ possible. What these components will be used for is to
demonstrate how they work inside, and what goes wrong if people try to
reinvent wheels.

Quite frankly we *should* be using a crypto professional, however that's
currently not a realistic option :-((( As a result currently it's up to me
to demonstrate here the risks involved in creating your own system.

>  http://www.cs.ucdavis.edu/~rogaway/classes/227/spring05/book/main.pdf
> 
> is a textbook that will show you how to do this, or at least give you
> an idea of what you're dealing with.  Watch out, it is rather theoretical.

That's fine/great :) - thanks for the reference :-)

FWIW, I *do* have a good idea of what I'm dealing with, //which is why// I'm
pessimistic about getting it right. I might have more experience with these
things than those I'm working with, however I am very aware of my
(current :) limits. Hopefully I'll be able to use these tools educate those
around me why rolling your own is an idea fraught with issues.

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Free seminar on domain-specific modeling

[Michael Sparks]

Martijn Iseger wrote:
...
> I believe the point being made by the organization is that during
> computing history the most successful shifts in productivity were
> achieved by similar shifts in raising the abstraction level on which
> developers specify solutions.

The alternate point is that during computing history, many, many, many
promises were made for many, many, many, technologies based on the
same principle of raising the abstraction level. Many, many, many of
those technologies promised much and failed to deliver on their claims
when used beyond the people inventing/using those technologies.

Furthermore, virtually all of them get marketed as being the next big
thing that Will Change The World. As a result anyone marketing an idea
in this way meets skepticism. From my perspective your site talks a lot
about general ideas but has little on specifics.

One thing is relatively clear - your approach appears to include a
graphical approach to systems building. Personally I suspect that the
fact people are able to engage other parts of their brain when building
these systems beyond linguistic is the real reason you see benefits,
rather than actually the specific thing that led to the visual approach
being possible.

Maybe your approach will change the world. Maybe it won't. If it's
better and it does, good. If it's not better and it does, that's a lot
of effort for no gain. Unfortunately that latter point is a common
result.

(On a sad note it looks like you're reinvented how hardware is designed
and made, but not made the intuitive leap :-/ )

Best Regards,


Michael.
-- 
This message (and any attachments) contains personal views
which are not the views of the BBC.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Crypto.Cipher.ARC4, bust or me doing something wrong?

[Michael Sparks]

Paul Rubin wrote:

> Michael Sparks <[EMAIL PROTECTED]> writes:
>> I'm looking at using this library and to familiarise myself writing
>> small tests with each of the ciphers. When I hit Crypto.Cipher.ARC4
>> I've found that I can't get it to decode what it encodes. This might
>> be a case of PEBKAC, but I'm trying the following:
> 
> You have to reinitialize the cipher state for decryption, as someone
> else explained.  You also have to make sure that keys are unique and
> independent for separate messages.  

Hmm... Thanks for this.

> For most applications you probably 
> want to think about adding authentication.  

Indeed - though I'm working bottom up. How a key gets transfered
from a to b safely will be another step. Another one will be how to
trust that exchange, for how long, and how much, etc. I'm well aware
that this is a well trod area though, hence why I'm working bottom up.

> In general you shouldn't 
> use arc4 unless you know what you're doing. 

Having looked at how it works from a user perspective, it's fairly
inappropriate anyway, due to wanting to work over unreliable
channels anyway.

> What's the application? 

Components for secure communications and identity confirmation.
Use cases:
   * Signing content multicast by the BBC to indicate that it came from
  the BBC. (ohh, so many issues :-)
   * Signing content captured on a mobile device such that we know that
  it came from that mobile. Specific high level use case of that is
  to be able to accept contributions from people from arbitrary
  devices and know who they came from. Perhaps embedded in the
  essence.
   * Related to that would be the ability to tag content with rights [1]
  information, and to know it's not been tampered with.

[1] eg Who created it. When? Has it been published/broadcast or not?
When? Have additional rights over and above fair use/dealing
been granted (eg creative commons attribution license).

   * Similarly people want the ability to protect data in transit
 between trusted end points. 

Some people using the system would probably be looking to build
restrictions management as well, but that's largely beyond the scope
of our system. Indeed restrictions management would require breaking
our system.

For these use cases to work, encryption & digest tools are clearly an
option, and hence having components that support encryption and
digest are (to say the least) useful. They're obviously not the only
techniques though.

Rather than re-inventing wheels I thought I'd pick a library sit down
and see how pycrypt's meant to be used before actually going anyway.
(Amongst other reasons, this is why I suspected me, rather than the
library :-)

Given the pycrypt library docs page starts off with:
   "This documentation assumes you have some basic knowledge about
the Python language, but not necessarily about cryptography."

I do for example know enough about cryptography to know that me
devising my own approach is foolhardy. Thus the statement above
appealed :)

As a result I decided to sit down and learn how to use this to form some
basic components for encryption/decryption. Given another part early in
the docs was """A central goal of the author's has been to provide a
simple, consistent interface for similar classes of algorithms.""" I
decided to start off with "sketches"  implementing minimal examples for
each digest and cipher. I'm now working through the Public Key
examples.

FWIW, I'm well aware how easy it is to get cipher/digest/etc based
security/id systems wrong. I'm really starting with pycrypt because it
looked simple enough, low level enough and self contained enough to
act as a base for working with existing more complex systems. 

I supsect we'll end up looking at or wrapping some other library
instead/as well, but it struck me as a nice starting point.

Anyway, once I've gone through all of the existing digests/ciphers/PK
ciphers, I'll post the snippets up on our site as raw examples for
pycrypto, which will hopefully be a) correct usage b) be useful to
others. 

Thanks for the comments,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Crypto.Cipher.ARC4, bust or me doing something wrong?

[Michael Sparks]

Michael J. Fromberger wrote:
...
> Since ARC4 is a stream cipher, the keystream changes over time -- with
> ARC4, after each character enciphered.  To decrypt successfully, you
> need to make sure the decrypting keystream exactly matches the
> encrypting one.
...
 from Crypto.Cipher import ARC4 as cipher
 enc = cipher.new("abcdefgh")
 dec = cipher.new("abcdefgh")
 x = enc.encrypt("This is some random text")
 x
> "\x05o\xd5XH|\xa4\xfc\xf7z\xecd\xe92\xfb\x05rR'\xbf\xc0F\xfc\xde"
 y = dec.decrypt(x)
 y
> 'This is some random text'
 enc.decrypt(x)
> 'M|[bI\x1ciG6A]\x13Hz\xb0\x19\xca\xf1-\x9a\x1a2\x9e%'
> 
> I hope this helps clear up your confusion.

Hi Michael,


Thanks for this, much appreciated. 


Michael
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Release of Shed Skin 0.0.2: Easy Windows/OSX Installation

[Michael Sparks]

Mark Dufour wrote:
> Shed Skin is an experimental Python-to-C++ compiler. Along with
> GNU/Linux, version 0.0.2 should now also install easily under Windows
> 2000/XP and OSX. Please give it a try and let me know if there are
> still some problems.

ss.py writes a make file, but unfortunately doesn't detect correctly whether
you need libdl linked in or not. As a result the generated Makefile for the
test includes this for me:

MAINOBJ=test.o
test:   $(MAINOBJ) /home/zathras/Documents/shedskin-0.0.2/libss.a
$(CC) $(CCFLAGS)
$(MAINOBJ) /home/zathras/Documents/shedskin-0.0.2/libss.a -lgc -o test

Whereas it would need to be the following because I'm under linux:

MAINOBJ=test.o
test:   $(MAINOBJ) /home/michaels/Documents/shedskin-0.0.2/libss.a
$(CC) $(CCFLAGS)
$(MAINOBJ) /home/michaels/Documents/shedskin-0.0.2/libss.a -lgc -ldl -o
test

(The reason I'm not suggesting just throwing in -ldl is because that will
then break on other platforms... You need to detect whether it's needed or
not for the platform it's currently compiling for)

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Crypto.Cipher.ARC4, bust or me doing something wrong?

[Michael Sparks]

Jp Calderone wrote:

> On Tue, 20 Sep 2005 16:08:19 +0100, Michael Sparks <[EMAIL PROTECTED]>
> wrote:
>>Hi,
>>
>>
>>I suspect this is a bug with AMK's Crypto package from
>>http://www.amk.ca/python/code/crypto , but want to
>>check to see if I'm being dumb before posting a bug
>>report.
>>
>>I'm looking at using this library and to familiarise myself writing
>>small tests with each of the ciphers. When I hit Crypto.Cipher.ARC4 I've
>>found that I can't get it to decode what it encodes. This might be a
>>case of PEBKAC, but I'm trying the following:
>>
>>>>> from Crypto.Cipher import ARC4 as cipher
>>>>> key = ""
>>>>> obj = cipher.new(key)
>>>>> obj.encrypt("This is some random text")
>>')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX'
>>>>> X=_
>>>>> X
>>')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX'
>>>>> obj.decrypt(X)
>>'\x87\xe1\x83\xc1\x93\xdb\xed\x93U\xe4_\x92}\x9f\xdb\x84Y\xa3\xd4b\x9eHu~'
>>
>>Clearly this decode doesn't match the encode. Me being dumb or bug?
>>
>>Any comments welcome :)
>>
> 
> You need two ARC4 instances.  Performing any operation alters the internal
> state (as it is a stream cipher), which is why your bytes did not come out
> intact.

Ahh. That makes perfect sense. (I thought it must be me missing something)

Many thanks! 

:-)

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Crypto.Cipher.ARC4, bust or me doing something wrong?

[Michael Sparks]

Hi,


I suspect this is a bug with AMK's Crypto package from
http://www.amk.ca/python/code/crypto , but want to
check to see if I'm being dumb before posting a bug
report.

I'm looking at using this library and to familiarise myself writing
small tests with each of the ciphers. When I hit Crypto.Cipher.ARC4 I've
found that I can't get it to decode what it encodes. This might be a
case of PEBKAC, but I'm trying the following:

>>> from Crypto.Cipher import ARC4 as cipher
>>> key = ""
>>> obj = cipher.new(key)
>>> obj.encrypt("This is some random text")
')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX'
>>> X=_
>>> X
')f\xd4\xf6\xa6Lm\x9a%}\x8a\x95\x8ef\x00\xd6:\x12\x00!\xf3k\xafX'
>>> obj.decrypt(X)
'\x87\xe1\x83\xc1\x93\xdb\xed\x93U\xe4_\x92}\x9f\xdb\x84Y\xa3\xd4b\x9eHu~'

Clearly this decode doesn't match the encode. Me being dumb or bug?

Any comments welcome :)


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Software bugs aren't inevitable

[Michael Sparks]

Giles Brown wrote:
> Michael Sparks wrote:
>> The problem that these sorts of approaches don't address is the simple
>> fact that simple creating a formal spec and implementing it, even if
>> you manage to create a way of automating the test suite from the spec
>> *doesn't guarantee that it will do the right thing*.
> 
>> As a result I'd say that the subject "Software bugs aren't inevitable"
>> is not true.
> 
> I think you can argue (I would) that any behaviour that is in the
> specification this "isn't right" is not a software bug, but a
> specification error.  

To a user there is no difference. If the software doesn't do what they
wanted it to do/asked for, then to *them* the person who is accepting
the software it's a bug. It doesn't matter to them what caused it - be it a
buffer overflow, a misunderstanding of a language feature or an incorrectly
written formal spec, it's still a bug to them.

I'm actually a big fan of formal specification (specifically VDM), but
that doesn't stop me realising that it's not a cure-all, and it's also not
an excuse - if code doesn't do what it was asked to do, it's bust.

Regards,


MIchael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Roguelike programmers needed

[Michael Sparks]

Robert Kern wrote:

> Thomas Jollans wrote:
>> what exactly is RPG/roguelike etc ? (what debian package provides an
>> example?)
> 
> Google is your friend.

Often a fair answer, but I'd suggest that the question was fair, especially
given the OP was seeking help :-)

After all, I read the subject line and simply assumed they were after
programmers with roguish qualities. Perhaps to work in the newly formed IT
division of the Crimson Permanent Assurance Company. After all, don't
forget - #It's Fun to Charter, and Accountant...#

;-)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Software bugs aren't inevitable

[Michael Sparks]

Steven D'Aprano wrote:

> On Thu, 15 Sep 2005 18:07:28 +0100, phil hunt wrote:
>> On Thu, 15 Sep 2005 21:56:06 +1000, Steven D'Aprano
>> <[EMAIL PROTECTED]> wrote:
>>>
>>>Are you saying that the recursion done by "serious" languages is a fake?
>>>That it is actually implemented behind the scenes by iteration?
>>>
>>>It seems to me that if recursion and iteration produce "the exact same
>>>machine code", the argument for preferring recursion over iteration is
>>>gutted.
>> 
>> It seems to me that if a high level language and assembler produce
>> "the exact same machine code", the argument for preferring high
>> level languages is gutted.
>> 
>> Do you see the fallacy in your statement now?
> 
> Ah, yes, you got me on that one.
> 
> But there is a difference: writing assembly is *hard*, which is why we
> prefer not to do it. Are you suggesting that functional programming is
> significantly easier to do than declarative?
> 

But there is a difference: writing assembly is *hard*, which is why we
prefer not to do it. Are you suggesting that object oriented programming is
significantly easier to do than old style imperative?

(sorry, couldn't resist)

FWIW, IMO once you've learnt functional programming's idioms it certainly
can be easier and more natural. The problem is the tools that make things
like recursion efficient aren't available normally in mainstream languages
meaning that most people simply don't get the practice.

Essentially it's about expressiveness.

Think of it this way - we normally write left to write, however some
languages read up and down. Neither is inherently better or easier
than the other, but for some things *may* be more expressive. 

If you think about it being about choosing the most clear/expressive way to
describe an algorithm, the argument may become clearer. After all, the
recursive definition of some things is clearer than the non-recursive.


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Software bugs aren't inevitable

[Michael Sparks]

Paddy wrote:

> A work colleague circulated this interesting article about reducing
> software bugs by orders of magnitude:

The problem that these sorts of approaches don't address is the simple
fact that simple creating a formal spec and implementing it, even if
you manage to create a way of automating the test suite from the spec
*doesn't guarantee that it will do the right thing*.

The hidden assumption is that the formal spec will be correct. If it
isn't then you have the same problems as before. Sure you might be able
to reduce the number of bugs, but you can be certain of one thing,
given:
   * Customer has a need
   * Customer communicates this to Supplier
   * Supplier translates needs to spec
   * Supplier translates spec back to english
   * Customer agrees spec

This involves human communication, and misunderstandings happen all
the time then.  Sure it can be easier to detect errors at this stage,
but you can't guarantee that it will do so. You can almost guarantee
though that people will misunderstand each other from time to time.
A purely engineering approach to dealing with correctness cannot
guarantee that misunderstandings won't occur. Those misunderstandings
translate into bugs, no matter what approach you use.

It's why I find XP and agile approaches interesting, they're often more
than just engineering.

As a result I'd say that the subject "Software bugs aren't inevitable"
is not true.

Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Michele Simionato wrote:

> It looks like I am reinventing Twisted and/or Kamaelia.

If it's /fun/ , is that a problem ? ;) (Interesting implementation BTW :)

FWIW, I've about a year ago it wasn't clear if we would be able to release
our stuff, so as part of a presentation I included a minimalistic decorator
based version of our system that has some similarities to yours. (The idea
was then at least the ideas had been shared, if not the main code - which
had approval)

Posted below in case it's of interest:

import copy
def wrapgenerator(bases=object, **attrs):
   def decorate(func):
   class statefulgenerator(bases):
  __doc__ = func.__doc__
  def __init__(self,*args):
   super(statefulgenerator, self) __init__(*args)
   self.func=func(self,*args) 
   for k in attrs.keys():
 self.__dict__[k] = copy.deepcopy(attrs[k])
   self.next=self.__iter__().next
  def __iter__(self): return iter(self.func)
   return statefulgenerator
   return decorate

class com(object):
  def __init__(_, *args):
 # Default queues
 _.queues = {"inbox":[],"control":[],
 "outbox":[], "signal":[]}
  def send(_,box,obj): _.queues[box].append(obj)
  def dataReady(_,box): return len(_.queues[box])>0
  def recv(_, box): # NB. Exceptions aren't caught
X=_.queues[box][0]
del _.queues[box][0]
return X


A sample component written using this approach then looks like this:

@wrapgenerator(com)
def forwarder(self):
   "Simple data forwarding generator"
   while 1:
  if self.dataReady("inbox"):
 self.send("outbox",self.recv("inbox"))
  elif self.dataReady("control"):
 if self.recv("control") == "shutdown":
break
  yield 1
   self.send("signal","shutdown")
   yield 0

Since we're not actualy using this approach, there's likely to be border
issues here. I'm not actually sure I like this particular approach, but it
was an interesting experiment.

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Stephen Thorne wrote:

> On 15/09/05, Michael Sparks <[EMAIL PROTECTED]> wrote:
>> At the moment, one option that springs to mind is this:
>> yield WaitDataAvailable("inbox")
> 
> Twisted supports this.
> 
> help("twisted.internet.defer.waitForDeferred")

Thanks for this. I'll take a look and either we'll use that or we'll use
something that maps cleanly. (Reason for pause is because running on
mobiles is important to us.) Thanks for the example too :)

Best Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/ 
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

[ Second time lucky... ]
Paul Rubin wrote:
...
> I don't see how generators substitute for microthreads.  In your example
> from another post:

I've done some digging and found what you mean by microthreads -
specifically I suspect you're referring to the microthreads package for
stackless? (I tend to view an activated generator as having a thread of
control, and since it's not a true thread, but is similar, I tend to view
that as a microthread. However your term and mine don't co-incide, and it
appears to cause confusion, so I'll switch my definition to match yours,
given the microthreads package, etc)

The reason I say this is because it naturally encourages small components
which are highly focussed in what they do. For example, when I was
originally looking at how to wrap network handling up, it was logical to
want to do this: 

[ writing something probably implementable using greenlets, but definitely
  pseudocode ]

@Nestedgenerator
def runProtocol(...)
   while:
   data = get_data_from_connection( ... )

# Assume non-blocking socket
def get_data_from_connection(...)
try:
data = sock.recv()
return data
except ... :
Yield(WaitSocketDataReady(sock))
except ... :
return failure

Of something - you get the idea (the above code is naff, but that's because
it's late here) - the operation that would block normally you yield inside
until given a message.

The thing about this is that we wouldn't have resulted in the structure we
do have - which is to have components for dealing with connected sockets,
listening sockets and so on. We've been able to reuse the connected socket
code between systems much more cleanly that we would have done (I
suspect) than if we'd been able to nest yields (as I once asked about here)
or have true co-routines.

At some point it would be interesing to rewrite our entire system based on
greenlets and see if that works out with more or less reuse. (And more or
less ability to make code more parallel or not)

[re-arranging order slightly of comments ]
>class encoder(component):
>   def __init__(self, **args):
>   self.encoder = unbreakable_encryption.encoder(**args)
>   def main(self):
>  while 1:
>  if self.dataReady("inbox"):
> data = self.recv("inbox")
> encoded = self.encoder.encode(data)
> self.send(encoded, "outbox")
>  yield 1
> 
...
> In that particular example, the yield is only at the end, so the
> generator isn't doing anything that an ordinary function closure
> couldn't:
> 
>def main(self):
>def run_event():
>if self.dataReady("inbox"):
>   data = self.recv("inbox")
>   encoded = self.encoder.encode(data)
>   self.send(encoded, "outbox")
>return run_event


Indeed, in particular we can currently rewrite that particular example as:

class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def mainLoop(self):
  if self.dataReady("inbox"):
 data = self.recv("inbox")
 encoded = self.encoder.encode(data)
 self.send(encoded, "outbox")
  return 1

And that will work today. (We have a 3 callback form available for people
who aren't very au fait with generators, or are just more comfortable with
callbacks)


That's a bad example though. A more useful example is probably something
more like this: (changed example from accidental early post)
...
  center = list(self.rect.center)
  self.image = self.original
  current = self.image
  scale = 1.0
  angle = 1
  pos = center
  while 1:
self.image = current
if self.dataReady("imaging"):
   self.image = self.recv("imaging")
   current = self.image
if self.dataReady("scaler"):
   # Scaling
   scale = self.recv("scaler")
w,h = self.image.get_size()
self.image = pygame.transform.scale(self.image, (w*scale,
h*scale))
if self.dataReady("rotator"):
   angle = self.recv("rotator")
   # Rotation
self.image = pygame.transform.rotate(self.image, angle)
if self.dataReady("translation"):
   # Translation
   pos = self.recv("translation")
self.rect = self.image.get_rect()
self.rect.center = pos
yield 1


(this code is from Kamaelia.UI.Pygame.BasicSprite)

Can it be transformed to something event based? Yes of course. Is it clear
what's happening though? I would say yes. Currently we encourage the user
to look to see if data is ready before taking it, simply because it's the
simplest interface that we can guarantee consistency with.

For example, currently the exception based equivalent would be:
try:
pos = 

Re: global interpreter lock

[Michael Sparks]

arrgh... hit wrong keystroke which caused an early send before I'd finished
typing... (skip the message I'm replying to hear for a minute please :-)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Paul Rubin wrote:
...
> I don't see how generators substitute for microthreads.  In your example
> from another post:

I've done some digging and found what you mean by microthreads -
specifically I suspect you're referring to the microthreads package for
stackless? (I tend to view an activated generator as having a thread of
control, and since it's not a true thread, but is similar, I tend to view
that as a microthread. However your term and mine don't co-incide, and it
appears to cause confusion, so I'll switch my definition to match yours,
given the microthreads package, etc)

You're right, generators aren't a substitue for microthreads. However I do
see them as being a useful alternative to microthreads. Indeed the fact
that you're limited to a single stack frame I think has actually helped our
architecture.

The reason I say this is because it naturally encourages small components
which are highly focussed in what they do. For example, when I was
originally looking at how to wrap network handling up, it was logical to
want to do this: 

[ writing something probably implementable using greenlets, but definitely
  pseudocode ]

@Nestedgenerator
def runProtocol(...)
   while:
   data = get_data_from_connection( ... )

# Assume non-blocking socket
def get_data_from_connection(...)
try:
data = sock.recv()
return data
except ... :
Yield(WaitSocketDataReady(sock))
except ... :
return failure

Of something - you get the idea (the above code is naff, but that's because
it's late here) - the operation that would block normally you yield inside
until given a message.

The thing about this is that we wouldn't have resulted in the structure we
do have - which is to have components for dealing with connected sockets,
listening sockets and so on. We've been able to reuse the connected socket
code between systems much more cleanly that we would have done (I
suspect) than if we'd been able to nest yields (as I once asked about here)
or have true co-routines.

At some point it would be interesing to rewrite our entire system based on
greenlets and see if that works out with more or less reuse. (And more or
less ability to make code more parallel or not)


[re-arranging order slightly of comments ]
>class encoder(component):
>   def __init__(self, **args):
>   self.encoder = unbreakable_encryption.encoder(**args)
>   def main(self):
>  while 1:
>  if self.dataReady("inbox"):
> data = self.recv("inbox")
> encoded = self.encoder.encode(data)
> self.send(encoded, "outbox")
>  yield 1
> 
...
> In that particular example, the yield is only at the end, so the
> generator isn't doing anything that an ordinary function closure
> couldn't:
> 
>def main(self):
>def run_event():
>if self.dataReady("inbox"):
>   data = self.recv("inbox")
>   encoded = self.encoder.encode(data)
>   self.send(encoded, "outbox")
>return run_event

Indeed, in particular we can currently rewrite that particular example as:

class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def mainLoop(self):
  if self.dataReady("inbox"):
 data = self.recv("inbox")
 encoded = self.encoder.encode(data)
 self.send(encoded, "outbox")
  return 1

That's a bad example though. A more useful example is probably something
more like this:

class Multicast_sender(Axon.Component.component):
   def __init__(self, local_addr, local_port, remote_addr, remote_port):
   super(Multicast_sender, self).__init__()
   self.local_addr = local_addr
   self.local_port = local_port
   self.remote_addr = remote_addr
   self.remote_port = remote_port

   def main(self):
   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM,
socket.IPPROTO_UDP)
   sock.bind((self.local_addr,self.local_port))
   sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10)
   while 1:
  if self.dataReady("inbox"):
 data = self.recv()
 l = sock.sendto(data, (self.remote_addr,self.remote_port) );
  yield 1

With a bit of fun with decorators, that can actually be collapsed into
something more like:

@component
def Multicast_sender(self, local_addr, local_port, remote_addr,
remote_port):
   sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM,
socket.IPPROTO_UDP)
   sock.bind((self.local_addr,self.local_port))
   sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 10)
   while 1:
  if self.dataReady("inbox"):
 data = self.recv()
 l = sock.sendto(data, (self.remote_addr,self.remote_port) );
  yield 1





> You've got the "main" method creating a generator that has its own
> even

Python Bindings for Dirac, (sorta announcement)

[Michael Sparks]

Hi,


We're in the process of creating python bindings for Dirac. We currently
have /decode/ of dirac functioning nicely, so I've packaged up the bindings
separately from the rest of the Kamaelia project for those that are
interested and would want a play. (Encoding will naturally follow next)

To build the bindings, you need:
   * Pyrex (0.9.3) installed:
  http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/
   * Dirac (0.5.3) installed, from source:
  http://dirac.sf.net/
  (Also, includes should be living in /usr/local/include/dirac)

A sample video player, entirely written in python/python extensions 
(eg pygame) for playing back dirac is in ...
   /Code/Python/Kamaelia/Examples/example10
... in Kamaelia's tree in the CVS head. (I'd point at view CVS, but it
hasn't caught up yet)

What is Dirac?
==
Not my project, but colleagues', so description from project page:

"""Dirac is a general-purpose video codec aimed at resolutions from QCIF
(180x144) to HDTV (1920x1080) progressive or interlaced. It uses wavelets,
motion compensation and arithmetic coding and aims to be competitive with
other state of the art codecs."""

How Do I Get Video To Decode?
=

The easiest way to get some dirac video is to use ffmpeg to convert a file
to yuv and then the dirac encoder. eg:

# ffmpeg -i file_from_digital_camera.avi rawvideo.yuv
...
  Stream #0.0: Video: mjpeg, 320x240, 16.10 fps
...

Useful line from ffmpeg's output pulled out. (in this case the avi file
above came from a cheap digital camera's video mode)

# dirac_encoder -qf 5 -width 320 -height 240 -fr 16 rawvideo myvideo

The resulting myvideo.drc that the above line would create is what the
decoder can handle. The -width & -height flags should be obvious, and
came from the output from ffmpeg. The -fr flag is the frame rate (fps),
again from ffmpeg. The -qf is a "quality factor". 

Bigger is better quality, but bigger also requires beefier systems for
decoding dirac.

The above settings result in something decodable using dirac using a 1.6Ghz
machine. If you have a beefier system you can use a higher quality factor,
if you have a less beefy system, a lower quality factor is preferable.

Caveats
===

Finally, I'm mentioning this for people who'd want to play/tinker - it's not
really suitable for (say) someone wanting to convert all their home
videos to yet, largely because dirac's bitstream isn't yet finalised. (If
you don't understand this, this might be a hint to stay away, if you do
understand this you'll hopefully understand why I'm giving this warning)
Also, the API and usage are also subject to change.

For reference an example file encoded with the above settings:
   * 2,694,380 bytes before
   * 293,579 bytes after

To my untrained eye there's not a /huge/ amount of difference in quality,
though there *is* a drop in quality (since that is a rather drastic drop). 
That's ~ 10 seconds of video.

Right now the easiest way to play with this is to do a cvs checkout of
Kamaelia's CVS head, install and play with example10 and tinker with
the components used, but the bindings aren't tied to Kamaelia and could
be used with other things. (We just happened to write them because we
need them)

If anyone's interesting in integrating these with other python libraries
we'd be interested in chatting :-)

The reason I say this is "sorta" an announcement is because I'm really
curious as to whether people think the API is sensible or not, but I'll
skip spammin the group with those details and ask anyone prepared to
take a look download the bindings from: http://tinyurl.com/arbk2 . 

Best Regards,


Michael.
--
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: First release of Shed Skin, a Python-to-C++ compiler.

[Michael Sparks]

Paul Boddie wrote:

> Michael Sparks wrote:
>> Well, you did say you want help with locating problems. One problem with
>> this is it doesn't build...
> 
> I found that I needed both the libgc and libgc-dev packages for my
> Kubuntu distribution - installing them fixed the include issues that
> you observed - and it does appear to be the Boehm-Demers-Weiser GC
> library, yes. The only other issue I observed was the importing of the
> profile and pstats modules which don't exist on my system, but those
> imports seemed to be redundant and could be commented out anyway.

Mark's also let me know this. Part of the problem is the version in SuSE 9.3
of the GC used is ancient - it should be version 6.5 onwards. Also for
people compiling from source you (at minimum) should be using the
configure line along the lines of:

 ./configure  --enable-cplusplus

If you don't, you get build problems because one of the needed libraries
isn't built by default. 

I started off with the obvious "hello world" type program :


print "GAME OVER"


Which compiled cleanly and worked as expected. I then read Mark's short
paper linked from his blog "Efficient Implementation of Modern Imperative
Languages; Application to Python", and got concerned by the comments:

"""We have decided not to investigate two types of features: [...snip...];
and those features that may be turned off without affecting correct
programs, e.g. array bounds checking, and exceptions"""

That set some alarm bells ringing, largely because LBYL being deprecated by
many people in favour of exceptions based code. (And more to the point,
widely used as a result)

As a result, I tried a trivial, but obvious program that should have clear
behaviour:


x = []
print "GAME OVER"

x.append(5)
print x[0]
try:
   print x[1]
   print "This should never be seen..."
except IndexError:
   print "It's OK, we caught it..."


This compiles, but unfortunately has the following behaviour:

GAME OVER
5
0
This should never be seen...
It's OK, we caught it...

Obviously, neither the 0 nor the message following should have been
displayed. It's a pity that this assumption was made, but given the short
time the project's been going I can understand it, hopefully Mark will
continue towards greater python compliance :)



Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python versus Perl

[Michael Sparks]

Terry Reedy wrote:
[...]
> I am being picky because various people have claimed that Python suffers
> in popularity because it is known as an 'interpreted language'.  So maybe
> advocates should be more careful than we have been to not reinforce the
> misunderstanding.

I sometimes wonder if it might help people understand the situation if
people described as "interpreted in the same way Java is" (However I think
that risks confusing things since python doesn't generally come with a JIT
subsystem, yet).

That said, if you do describe it that way, it'd be more accurate to describe
the python binary as a compiler/runtime rather than interpreter since it'd
be more accurate.

After all:
   $ python somefile.py

Is very close to being the same as:
   $ javac somefile.java
   $ java somefile.class

It strikes me as ironic that python would probably gain more credibility
with some circles if it had two binaries like this, even though it'd be a
step backwards from a usability perspective :-)

Personally I agree that any language that is described as interpreted has an
image issue. However I'm not sure who's problem that is - some people claim
it's "Python's problem", however personally I'd view as a problem for the
people who buy into "interpretted bad, compiled good" argument. After all,
they're the ones limiting themselves, and missing out on a whole class of
languages (of which python is just one of course) !


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: First release of Shed Skin, a Python-to-C++ compiler.

[Michael Sparks]

Mark Dufour wrote:

> With this initial release, I hope to attract other people to help me
> locate remaining problems, 

Well, you did say you want help with locating problems. One problem with
this is it doesn't build...


If I try and build (following your instructions), I get presented with a
whole slew of build errors - knock on errors from the first few:

In file included from builtin_.cpp:1:
builtin_.hpp:4:29: gc/gc_allocator.h: No such file or directory
builtin_.hpp:5:23: gc/gc_cpp.h: No such file or directory
In file included from builtin_.cpp:1:
builtin_.hpp:89: error: syntax error before `{' token
builtin_.hpp:93: error: virtual outside class declaration

Which C++ libraries are you dependent on? (Stating this would be really
useful, along with specific versions and if possible where you got them :)

For reference, I'm building this on SuSE 9.3, under which I also have
boehm-gc-3.3.5-5 installed. I suspect you're using the same gc library
(having downloaded libgc from sourceforge and finding the includes don't
match the above include names) but a different version. For reference this
version/distribution of boehm-gc has the following file structure:

/usr/include/gc.h
/usr/include/gc_backptr.h
/usr/include/gc_config_macros.h
/usr/include/gc_cpp.h
/usr/include/gc_local_alloc.h
/usr/include/gc_pthread_redirects.h
/usr/lib/libgc.a
/usr/lib/libgc.la
/usr/lib/libgc.so
/usr/lib/libgc.so.1
/usr/lib/libgc.so.1.0.1

It's specifically the gc_cpp.h file that makes me suspect it's the same gc.

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Video display, frame rate 640x480 @ 30fps achievable?

[Michael Sparks]

Guenter wrote:
> Michael Sparks schrieb:
>> Yes.
>>
>> Co-incidentally we've been looking at video playback this week as
>> well. We've been using Pygame with an Overlay surface, and it works
>> fairly well.
>
> I guess Pygame was more suitable overall for your application? 

It's not really that, it's just that it was the easiest most obvious
(and we've found portable) approach to getting the video onscreen.
You create an overlay object and simply throw decoded YUV data at
in and don't worry too much.

We can generally mix and match UI's happily at the moment with our
work, so we can have tkinter and pygame side by side for example.

> I would just be interested whether you have considered using
> PyMedia? 

We did look at it, but found sufficient numbers of problems to decide
that it might be worth coming back to at a later point in time. Also
we kept hitting issues on 64-bit machines which didn't really help. API
changes also invalidated some of the examples causing confusion (I
wasn't the person directly working on it). 

That said I've heard enough people say enough good things about it to
suggest that don't take our experience as the best example - I suspect
it's far from the best example. (I wish we had more time to try and
help fix it!)

(We've decided to wrap dirac instead for now)

>> Initially we're testing with simple IYUV raw video data, and it's a
>> good idea to use a modern video card supported by your OS, but other
>> than that we've not had problems. If you're interested in code, let
>> us know :-)
>>
> Thanks for the offer. If it is getting serious and I need some jump
> start I might come back to you about the code.

No problem.


Michael
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: reading the last line of a file

[Michael Sparks]

Xah Lee wrote:
> isn't there a way to implement tail in python with the same class of
> performance?
>
> how's tail implemented?:

Those crazy open source developers have an implementation here:
http://cvs.sourceforge.net/viewcvs.py/mkcdrec/mkcdrec/busybox-0.60.5/Attic/tail.c?rev=1.1&view=markup

It's not documented. It's pretty short though, I'm sure you won't have any
problems ripping it to pieces understanding it.

If it is a problem, a more documented version here (however I'm sorry, this
also by those open source/free software people you love to attack):

http://www.koders.com/c/fid8DEE98A42C35A1346FA89C328CC3BF94E25CF377.aspx

If you want something not by open source crazies (as you like to refer to
us), you may prefer this link:
   http://minnie.tuhs.org/UnixTree/V7/usr/src/cmd/tail.c.html

However that's even less documented (despite being not open source, just
"visible"), and more obfuscated.

As a compromise, you could always look at plan 9's implementation here:
   * http://swtch.com/usr/local/plan9/src/cmd/tail.c

If you're after alternatives, you could always try here: (where all these
links came from)
http://www.google.com/search?q=tail.c

Fairly informative set of links provided by that highly non-obvious
search

Hope that helps,

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Video display, frame rate 640x480 @ 30fps achievable?

[Michael Sparks]

Guenter wrote:

> I need to develop an application that displays video 640x480 16-bit per
> pixel with 30 fps  it is possible to achieve that frame rate and still
> have some resources for other processing left? 

Yes.

Co-incidentally we've been looking at video playback this week as well. 
We've been using Pygame with an Overlay surface, and it works fairly well.
Initially we're testing with simple IYUV raw video data, and it's a good 
idea to use a modern video card supported by your OS, but other than that 
we've not had problems. If you're interested in code, let us know :-)

If you end up using the framebuffer device in linux under xorg's X11 (not
exactly an ideal setup for video playback anyway!) there is a little oddity
that we found, due to SDL underneath, but we've been able to work
around that. The system I'm working on normally has that issue, and as
a result isn't accelerated and is a 1.6Ghz machine, but can do standard
defintion video (720x576) playback happily at >30fps. Normally though
most people can offload this to their graphics card so there shouldn't
be issues on your 1Ghz machine.

Clearly if you're interested in driving that you need to decode some data
source to IYUV, but that's a different issue. 

People have embedded pygame in wxPython before (I don't know how
though, I've only seen screenshots of the integration) so although you said
wxPython and might've discounted pygame based on that, it might still be
suitable for you. 

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dual processor

[Michael Sparks]

Thomas Bellman wrote:
> Michael Sparks <[EMAIL PROTECTED]> writes:
>> Similarly, from
>> a unix command line perspective, the following will automatically take
>> advantage of all the CPU's I have available:
>>(find |while read i; do md5sum $i; done|cut -b-32) 2>/dev/null |sort
> 
> No, it won't.  At the most, it will use four CPU:s for user code.

OK, maybe I should've been more precise. That said, the largest machine I
could potentially get access relatively easily to would be a quad CPU
machine so if I wanted to be be pedantic, regarding "*I* have available"
the idea stands. (Note I didn't say take /best/ advantage - that would
require rewriting all the indvidual parts of the pipeline above to be
structured in a similar manner or some other parallel approach)

You've essentially re-iterated my point though - that it naturally sorts
itself out, and does the best fit it can, which is better than none
(despite this being a naff example - as I mentioned).  Worst case, yes,
everything serialises itself.


Michael.


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dual processor

[Michael Sparks]

Jorgen Grahn wrote:

> On Tue, 06 Sep 2005 08:57:14 +0100, Michael Sparks <[EMAIL PROTECTED]>
> wrote: ...
>> Are you so sure? I suspect this is due to you being used to writing code
>> that is designed for a single CPU system. What if you're basic model of
>> system creation changed to include system composition as well as
>> function calls? Then each part of the system you compose can potentially
>> run on a different CPU. Take the following for example:
> ...
>> It probably looks strange, but it's really just a logical extension of
>> the Unix command line's pipelines to allow multiple pipelines. Similarly,
>> from a unix command line perspective, the following will automatically
>> take advantage of all the CPU's I have available:
>>
>>(find |while read i; do md5sum $i; done|cut -b-32) 2>/dev/null |sort
>>
>> And a) most unix sys admins I know find that easy (probably the above
>> laughable) b) given a multiprocessor system will probably try to maximise
>> pipelining c) I see no reason why sys admins should be the only people
>> writing programs who use concurrency without thinking about it :-)
> 
> Nitpick: not all Unix users are sysadmins ;-) Some Unix sysadmins actually
> have real users, and the clued users use the same tools. I used the 'make
> -j3' example elsewhere in the thread (I hadn't read this posting when I
> responded there).

I simply picked a group that do this often :-) The example pipeline I gave
above is I admit a particularly dire one. Things like the following are far
more silly:

# rm file; fortune | tee file | wc | cat - file
  3  16 110
Bubble Memory, n.:
A derogatory term, usually referring to a person's
intelligence.  See also "vacuum tube".

And

# (rm file; (while [ ! -s file ]; do echo >/dev/null; done; cat file |wc) & 
fortune | tee file) 2>/dev/null
Yea, though I walk through the valley of the shadow of APL, I shall
fear no evil, for I can string six primitive monadic and dyadic
operators together.
-- Steve Higgins
#   4  31 171

> It seems to me that there must be a flaw in your arguments, but I can't
> seem to find it ;-)

Sorry, but that's probably the funniest thing I've read all day :-)

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python versus Perl

[Michael Sparks]

Dieter Vanderelst wrote:

> Dear all,
> 
> I'm currently comparing Python versus Perl to use in a project that
> involved a lot of text processing. I'm trying to determine what the
> most efficient language would be for our purposes. I have to admit
> that, although I'm very familiar with Python, I'm complete Perl noob
> (and I hope to stay one) which is reflected in my questions.
> 
> I know that the web offers a lot of resources on Python/Perl
> differences. But I couldn't find a satisfying answer to my questions:
> 
> 1 - How does the speed of execution of Perl compares to that of
> Python?

Much of a muchness in my experience.(Qualitative, not quantative)

> 2 - Regular Expressions are a valuable tool in text processing. I have
> noticed that Regular Expressions are executed very fast in Python.
> Does anybody know whether Python executes RE faster than Perl does?
> 3 - In my opinion Python is very well suited for text processing. Does
> Perl have any advantages over Python in the field of textprocessing
> (like a larger standard library maybe).

These two are related. If you're writing code and you expect to be
using *a lot* of regular expression [*] type code then you may find perl
more convenient.

   [*] That /might/ suggest you're taking the wrong approach mind you...

Python, for me, tends to be more readable, both immediately after
writing and if I go back to a year later - for maintenance, extension
etc.

Personally I like both languages for day in day out use, but these days
tend to choose python if I think I'm likely to want to modify or extend
the code. With the exception being where I'm doing heavy text
processing work that I think will be more maintainable in perl, or I'm
really sure I won't have to maintain it. (eg quick and dirty scripts)

One side effect of perl usage though is that due to them being easy to
use and convenient, they can get over used. (Rather than thinking
"what's the best way of solving this problem", people can end up
thinking "What regular expression can solve this problem" - which isn't
ideal)

Your comment """I'm complete Perl noob (and I hope to stay one) """
would suggest to me that if you really feel that way, stay that way :-)
(Though personally I do like learning new programming languages, since
you get more idioms and tools under your belt that way.)

> I hope somebody can answer my questions. Of course, every remark and
> tip on Python/Perl in texprocessing is most welcome.

In case you're not aware there's the book "Text Processing in Python" by
David Mertz, which is available online in a "free as in beer" form
which might be of use if you decide python instead of perl.


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: dual processor

[Michael Sparks]

Jeremy Jones wrote:

> Michael Sparks wrote:
>>Steve Jorgensen wrote:
>>>On 05 Sep 2005 10:29:48 GMT, Nick Craig-Wood <[EMAIL PROTECTED]> wrote:
>>>
>>>
>>>
>>>>Jeremy Jones <[EMAIL PROTECTED]> wrote:
>>>>> One Python process will only saturate one CPU (at a time) because
>>>>> of the GIL (global interpreter lock).
>>>>I'm hoping python won't always be like this.

>>>I don't get that.  Python was never designed to be a high performance
>>>language, so why add complexity to its implementation by giving it
>>>high-performance capabilities like SMP?

>>It depends on personal perspective.
>>
> Unot totally.  It depends on what you're doing.  

Yes, it does. Hence why I said personal perspective. 

>  Sort of.  Steve
> brings up an interesting argument of making the language do some of your
> thinking for you.  Maybe I'll address that momentarily

Personally I think that the language and tools will have to help. I'm
working on the latter, I hope the GIL goes away to help with the former, 
but does so in an intelligent manner. Why am I working on the latter? 

I work with naturally concurrent systems all the time, and they're
concurrent not for performance reasons, but simply because that's what they
are. And as a result I want concurrency easy to deal with, and efficiency as
a secondary concern. However in that scenario, having multiple CPU's not
being utilised sensibly *is* a concern to me.

> I'm not saying I wish the GIL would stay around.  I wish it would go.
> As the price of computers goes down, the number of CPUs per computer
> goes up, and the price per CPU in a single system goes down, the ability
> to utilize a bunch of CPUs is going to become more important.  

> And maybe  
> Steve's magical thinking programming language will have a ton of merit.

I see no reason to use such derisory tones, though I'm sure you didn't mean
it that way. (I can see you mean it as extreme skepticism though :-)

>>If in a few years time we all have
>>machines with multiple cores (eg the CELL with effective 9 CPUs on a chip,
>>albeit 8 more specialised ones), would you prefer that your code *could*
>>utilise your hardware sensibly rather than not.
>>  
>>

> But let me play devil's advocate for
> a sec.  Let's say we *could* fully utilize a multi CPU today with
> Python. 
...
> I would almost bet money that the majority of code would 
> not be helped by that at all.  

Are you so sure? I suspect this is due to you being used to writing code
that is designed for a single CPU system. What if you're basic model of
system creation changed to include system composition as well as
function calls? Then each part of the system you compose can potentially
run on a different CPU. Take the following for example:

(sorry for the length, I prefer real examples :-)

Graphline(
 EXIT = ExceptionRaiser("FORCED SYSTEM QUIT"),
 MOUSE = Multiclick(caption="",
position=(0,0),
transparent=True,
msgs = [ "", "NEXT", "FIRST", "PREV", "PREV","NEXT" ],
size=(1024,768)),
 KEYS = KeyEvent(outboxes = { "slidecontrol" : "Normal place for message",
  "shutdown" : "Place to send some shutdown 
messages",
  "trace" : "Place for trace messages to go",
},
 key_events = {112: ("PREV", "slidecontrol"),
   110: ("NEXT","slidecontrol"),
   113: ("QUIT", "shutdown"),
  }),
 SPLITTER = Splitter(outboxes = {"totimer" : "For sending copies of key 
events to the timer",
 "tochooser" : "This is the primary 
location for key events",
}),
 TIMER = TimeRepeatMessage("NEXT",3),
 FILES = Chooser(items = files, loop=True),
 DISPLAY = Image(size=(1024,768),
 position=(0,0),
 maxpect=(1024,768) ),
 linkages = {
("TIMER", "outbox") : ("FILES", "inbox"),

("MOUSE", "outbox") : ("SPLITTER", "inbox"),
("KEYS", "slidecontrol") : ("SPLITTER", "inbox"),
("SPLITTER", "tochooser") : ("FILES&quo

Re: dual processor

[Michael Sparks]

Steve Jorgensen wrote:

> On 05 Sep 2005 10:29:48 GMT, Nick Craig-Wood <[EMAIL PROTECTED]> wrote:
> 
>>Jeremy Jones <[EMAIL PROTECTED]> wrote:
>>>  One Python process will only saturate one CPU (at a time) because
>>>  of the GIL (global interpreter lock).
>>
>>I'm hoping python won't always be like this.
> 
> I don't get that.  Python was never designed to be a high performance
> language, so why add complexity to its implementation by giving it
> high-performance capabilities like SMP? 

It depends on personal perspective. If in a few years time we all have
machines with multiple cores (eg the CELL with effective 9 CPUs on a chip,
albeit 8 more specialised ones), would you prefer that your code *could*
utilise your hardware sensibly rather than not.

Or put another way - would you prefer to write your code mainly in a
language like python, or mainly in a language like C or Java? If python,
it's worth worrying about!

If it was python (or similar) you might "only" have to worry about
concurrency issues. If it's a language like C you might have to worry
about  memory management, typing AND concurrency (oh my!).
(Let alone C++'s TMP :-)

Regards,


Michael
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: OpenSource documentation problems

[Michael Sparks]

Paul Rubin wrote:

> Michael Sparks <[EMAIL PROTECTED]> writes:
>> > I've submitted a number of doc bugs to sourceforge and the ones
>> > that are simple errors and omissions do get fixed.
>> 
>> Cool.
> 
> Better than nothing, but it's only one class of problem, and maybe the
> easiest kind to report.

[lots and lots of stuff snipped]

All very interesting, and will take me a while to digest. I must admit I
find the whole set of suggestions/observations taken together to be
very interesting/useful and I'll try and take them on board with my own
projects first.

Many thanks for the detailed response, it's very much appreciated.

Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: OpenSource documentation problems

[Michael Sparks]

A.M. Kuchling wrote:

> On 1 Sep 2005 05:04:33 -0700,
> Paul Boddie <[EMAIL PROTECTED]> wrote:
>> Please note that I'm not labelling you as a troll.
> 
> No, he's simply barking mad.  I was amused by a rec.arts.sf.written
> discussion [1] where Lee complains that Jonathan Swift (1667-1745)'s
> writing was unclear in style; apparently he's not aware that conventions
> and styles change over time.
> 
> --amk
> 
> [1] http://groups.google.com/group/alt.usage.english/msg/0ec9871395fc90d3

Oddly that makes me wonder if Xah Lee is a some kind of bot - given it
follows a similar fashion:
 
oh, this writing is [criticism] of [ specific]

"the "allow that" should be "allow the notion that". But the 
 whole thing is written in a confounded way. "

Specific grammar correction and note that the grammar subsystem detected too
many errors.

Better: Grammar correction suggestion. Not actually any clearer to many
people. 

more better with logicality: Similar bad parsing. A person would translate
the old style english into a more modern style if what they're doing is
criticising old english.

The whole thing just smacks to me of some text analyser. The final paragraph
contains some standard writing advice and cliches:
   * to write clearly, the first thing is to think clearly
   * get rid of big words and use  short sentences.

It'd certainly explain the online tourettes syndrome of swearing all the way
through a post - the text generator getting stuck on a particular word
sets...

Not-really-seriously :-) 
(Though usenet would be an interesting, if antisocial and annoying place to
test such a beast :-)


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: OpenSource documentation problems

[Michael Sparks]

Hi Paul,


Paul Rubin wrote:
> Michael Sparks <[EMAIL PROTECTED]> writes:
[[[ some random stuff, /intended/ at supporting people who have contributed
  docs, rather than saying people who offer constructive suggestions are
  bad. Possibly badly written. ]]]
> I've submitted a number of doc bugs to sourceforge and the ones that
> are simple errors and omissions do get fixed.  

Cool. 

> However, your 
> suggestion doesn't constitute good Python advocacy, if that's what you
> were attempting.  

Nope. I rarely attempt advocacy because I'd suck at it if I tried (worse
than my docs :), and also I'm not a fan of advocacy anyway - I prefer to
make up my own mind rather than be told. 

> When I use a tool like Python, I like to think I'm 
> working on one project at a time, that project being whatever I'm
> trying to develop.  

> Telling me that whenever I use Python, I should 
> really consider myself to be working on two projects at once

I didn't mean it that way (I probably put it over the wrong way). What I did
mean is that when someone in here says "someone here should make the docs
better", is naturally including themselves in the set of people who can
write docs. If they then attack people who /can/ write docs they're
attacking themselves. 

Stating that something needs writing or updating I'd agree is good. People
whinging because no-one has written it, or constantly dissing docs without
helping improve them strikes me as unfair.

To my mind yourself, Bryan, Steve, Terry and others in the thread contribute
amazingly (and I wish I had more time to do so), by simply being nice and
helpful in *here*.

>> [ O'Reilly, and friends, books as a docs for open source ]
> 
> If those docs aren't free to redistribute, then that's not an
> open-source solution at all; if anything, it's an anti-solution, since
> it decreases the motivation to write good free docs.

OK, it's not the best solution in the world I'll agree, but my point was in
general very few people like writing docs even when paid, let alone when
not. As a result it's not that suprising IMO that if paid you end up with
more docs. 

> Even weirder is the amount of gratis (free as in beer) but unfree (as
> in freedom) docs out there for Python.  

I'm not aware of those, so I don't feel able to comment sensibly.

>> The best way to repay those who have spent the time writing what can
>> be viewed as a first draft is to provide corrections. If you
>> (generic, not just you personally) don't have the time, effort,
>> inclination or effort to summarise and feedback a documentation
>> patch *why should anyone else* ?
> 
> I usually do report doc bugs, but my frustration (and I think Bryan's)
> is that there are so many bugs in the first place.  It means that the
> authors are not applying high enough quality standards to their own
> work before releasing it.  That applies to Python's code as well as
> its docs.  It's not crap,

Or maybe they're just doing their best? It might seem silly but when I do
write docs, personally it takes me around 4-10 times longer to write the
same number of lines as documentation than as code, whilst trying to
maintain a similar quality. I think thats on a good day.

> but it's not at the level that those of us 
> from the non-Windows world like to (realistically or otherwise) expect.

FWIW, I'm not part of the windows crowd. Ceased using windows regularly
around Windows 3.1, and have periodically gone back (95, 98, ME, 2000, XP)
to see whether it's improved, and come to the conclusion for me it's a
downgrade in the way I work. It is sometimes useful for playing DVDs on
long journeys though since the power management works better IME.

[ specific example of bad or missing docs snipped ]

I agree there's always valid criticisms.

> I think there is an attitude problem in the central Python development
> community, which is to expect external volunteers to do stuff with no
> cajoling and no guidance.  That just doesn't work very well.  

I have no idea what to do about that. Giving T-Shirts to people who write a
decent tutorial that covers an entire library module? (Semi-serious if
there's a way of making that work)

> I was the first FSF staff programmer on the GNU project and we spent a LOT
> of our time coordinating volunteers and maintaining lists of tasks to
> recruit people to do, and generally trying to make stuff happen
> according to what we saw as the project's priorities, as opposed to
> simply passively waiting for code and doc contributions to come to us
> fully done.  We also saw doing gap-filling and grunt-work that didn't
> excite volunteers to be an important part of our purpose as paid
&g

Re: OpenSource documentation problems

[Michael Sparks]

Bryan Olson wrote:

> A plausible theory. I have some possibly-illustrative examples
> of what I ran into within the last few weeks.

 Did you take what you learnt, and use that to create better
documentation to be posted on python's SF project as a patch?
 

(Not aimed at you, just a preface, and you also provided the right "starter
point" to my main comment :)

> Whatever else one says about open-source documentation, keeping
> it current is a major unsolved problem.

I agree. However, the reason for this is very simple: most people who write
open source software (especially so AFAICT with python) do so on their own
time. In my experience its very difficult to get professional programmers
when they're being paid to write good documentation, and this is /despite/
the possible alternatives.

Add in the fact that part of the problem is that ability to write good code
doesn't always equate to the ability to write good documentation. Also add
in that the desire to write code isn't always accompanied by the desire to
write documentation.

Taking all that into account if it's difficult to get people who are paid
to do these things to write decent documentation (which is why people
hire tech writers), I find it suprising that people expect open source
developers to spend their own time producing high quality documentation,
which may in fact be a skill some of them lack.

Ironically though I do think the problem of documentation has been
half solved in the open source world, and it really boils down (IMO)
to companies like O'Reilly who appear to try and target popular
not-best-documented open source projects and provide the
motivation that results in good documentation.

As things go, python is pretty well documented IMO. Sure there's weak
areas, but considering this is all stuff people have done largely on their
own time, and anyone who's written docs knows it's not fun. The best way to
repay those who have spent the time writing what can be viewed as a first
draft is to provide corrections. If you (generic, not just you personally)
don't have the time, effort, inclination or effort to summarise and
feedback a documentation patch *why should anyone else* ?

When people complain /in here/ about the documentation not being perfect for
python I personally find it sad and ironic. It's sad because it says to the
person who spent their time (when they could be doing something else) that
they wasted it, the docs are worthless etc - when they clearly *haven't*
wasted their time and the docs are worthwhile. It's ironic though because
they're criticising themselves. After all they understand how the docs can
be better and the docs are shared. Unless they contribute back they're
actually attacking themselves as much as the person who "wastes" their time
writing "worthless" docs.

(Again, not specifically aimed at you, but unless you've actually
contributed back the results of your findings, you have to realise that any
reasons or excuses that apply to you may also apply to anyone/everyone in
the group and it's therefore suprising we have any docs ! :-)

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Release of PyPy 0.7.0

[Michael Sparks]

Michael Hudson wrote:
...
> The chance of any random module you have written being rpython is more
> or less zero, so it's not _that_ interesting for you to try to compile
> them with PyPy.

I know - the code I use contains LOTS of generators for example, which
obviously don't fit the requirements :) (It's not difficult to convert code
from being a generator though :-)

> No, you're still operating at the wrong level here (very easily done).
> This is the _translated PyPy_ interpreting pystone.  If you run a
> _translated pystone_ you'll (hopefully) get a different, faster
> answer.

Actually what I was aiming to compare was speed of pystone running on top of
CPython vs speed of pystone on top of translated pypy. ie running pystone
on top of something native that can execute python code :-)

> In expected order of execution speed:
> 
> interpreted pypy interpreting pystone
> translated pypy interpreting pystone
> cpython interpreting pystone
> translated pystone

Damn, I'm now going to have to sit down and find out if that's true :-)

*shucks* :-)

>> Anyway, whether it's sensible or not I'm going to play with translating
>> some of my modules :)
> 
> Whatever floats your boat :)

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: aproximate a number

[Michael Sparks]

billiejoex wrote:

> Hi all. I'd need to aproximate a given float number into the next (int)
> bigger one. Because of my bad english I try to explain it with some
> example:
> 
> 5.7 --> 6
> 52.987 --> 53
> 3.34 --> 4
> 2.1 --> 3

What about 2.0? By your spec that should be rounded to 3 - is that what you
intend?

If you do, you can simply do this:

def approx(x):
return int(x+1.0)

Regards,


Michael.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Release of PyPy 0.7.0

[Michael Sparks]

Valentino Volonghi aka Dialtone wrote:

> Michael Sparks <[EMAIL PROTECTED]> wrote:
> 
>> Would it be useful for people to start trying out their modules/code to
>> see if they work with this release, and whether they can likewise be
>> translated using the C/LLVM backends, or would you say this is too early?
>> (I'm more thinking in terms of it providing real world usecases in the
>> hope of finding things that don't work - rather than anything else)
> 
> This is not how it works. 

I beg to differ - it is how it can work (just not the default or currently
recommended). Currently the biggest, most interesting example is this:
(from the getting started page, compiling pypy)

cd pypy/translator/goal
python translate_pypy.py

Which has just finished executing on my machine and took about 3 hours to
complete (partly because the cc1 process reached ~780MB in terms of memory
footprint and my machine has 512Mb). However it goes on to talk about
compiling other things:

"""You can also use the translate_pypy.py script to try out several smaller
programs, e.g. a slightly changed version of Pystone:
cd pypy/translator/goal
python translate_pypy.py targetrpystone
"""

Which is pretty cool of course. For those of interest running pystone with
the pypy compiled native binary has the following results for pystones on
my machine:

[EMAIL PROTECTED]:~/pypy-0.7.0/pypy/translator/goal> ./pypy-c
debug: entry point starting
debug:  argv -> ./pypy-c
debug: importing code
debug: calling code.interact()
Python 2.4.1 (pypy 0.7.0 build) on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>> from test import pystone
>>>> pystone.main(1000)
Pystone(1.1) time for 1000 passes = 13.97
This machine benchmarks at 71.582 pystones/second
>>>>

The same results for CPython:
[EMAIL PROTECTED]:~/pypy-0.7.0/pypy/translator/goal> python
Python 2.4 (#1, Mar 22 2005, 21:42:42)
[GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import pystone
>>> pystone.main()
Pystone(1.1) time for 5 passes = 1.58
This machine benchmarks at 31645.6 pystones/second

Obviously therefore anyone seeking to translate their existing code from
python to an executable directly using pypy would not be doing it for
performance reasons (again, something I'm aware of watching the
updates come out and having run svn checkouts at previous times).

(Factor of ~450 though is pretty d*^%n cool though :)

I'm also well aware of the difference between python and rpython :-)

> Pypy doesn't translate your code to C/LLVM.

> Currently PyPy can only translate a pretty simple subset of python
> called RPython which has a very C-like syntax (but without memory
> management code). This is needed to allow type inference inside the
> interpreter code.
>
> The code in your application is application code and can be whatever you
> want, you may try to translate it to C/LLVM but it won't be that good of
> course because the annotator is not that intelligent.

*Shrug* Seemed pretty good at annotating and compiling a module longer than
many of mine at Europython, and then compiled using the LLVM backed.  :)

Anyway, whether it's sensible or not I'm going to play with translating some
of my modules :)

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Release of PyPy 0.7.0

[Michael Sparks]

Carl Friedrich Bolz wrote:
[[... great news ...]]

Would it be useful for people to start trying out their modules/code to see
if they work with this release, and whether they can likewise be translated
using the C/LLVM backends, or would you say this is too early? (I'm more
thinking in terms of it providing real world usecases in the hope of
finding things that don't work - rather than anything else)

Either way, this looks like a great milestone - congratulations to the
entire team. (I remember PyPy being met with skepticism as to whether
it could even be done! :-)

Best Regards,


Michael

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Socket Troubles

[Michael Sparks]

Chris Spencer wrote:
> My code's ... at http://deadbeefbabe.org/paste/1525/0
...
> I've written a simple class to manage P2P socket connections. However,
> whenever I try to receive data, the socket raises an exception with the
> error message  (11, 'Resource temporarily unavailable').

At one point in your code you do this: 
self._socket.setblocking(0)

This says "if we can't recieve or send data without blocking, fail rather 
than succeed". One of the failure modes is to return error 11. This is 
infact normally defined as:
#define EAGAIN  11  /* Try again */

What this means is "sorry, I couldn't do this without blocking right now, 
try again very shortly!".

Looking at your code it continually loops (in BaseConnectionHandler_recv) 
receiving data - whether or not there's any data to receive:
def _recv(self):
while self.running:
try:
data = self._socket.recv(4096)
if not len(data):
time.sleep(0.1)
continue
except Exception, e:
log('Recieve failed for handler',self.address,'because of',e)

break

Since you never actually check to see if the socket is ready to give you
data, and you've set it non-blocking, seeing lots of EAGAIN errors is
pretty much what you'd expect to see. (Simply sleeping is not sufficient!)

I suppose the short answer though really is this: you set the socket
non-blocking, you should therefore expect to see failures telling you
to try again, and follow their advice! :)

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Network performance

[Michael Sparks]

Roland Hedberg wrote:
> What the protocol has to accomplish is extremely simple; the client
> sends a number of lines (actually a RDF) and the server accepts or
> rejects the packet. That's all !
...
> Now, presently I'm using  ( why so is a long
> history which I will not go into here) and that is a bit to slow for
> my needs. On my present setup [Twisted and XMLRPC ]it takes close to
> 200 ms to perform one transfer. But since that includes setting up and
> tearing  down the connection I thought I'd try doing something simpler
> so I wrote a server and a client using socket.

> I was surprised to find that the performance was equal to what
> Twisted/XMLRPC did. Around 200 ms per set, not significantly less.

That should tell you two things:
   * Twisted/XMLRPC is as efficient as you can hand craft. (which is a
 good use reason for using it).
   * That what you're measuring is overhead - and most likely of setup.

I'd measure the ping time between your two hosts. Why? TCP is not the
fastest protocol in the world. If you're measuring the whole
connection, you're dealing with a number of TCP messages:
   * Client -> SYN
   * Server -> SYN ACK
   * Client -> ACK
   * Client -> DATA (Assume data fits inside one message)
   * Server -> ACK
   * Client -> FIN
   * Server -> FIN ACK
   * Server -> FIN
   * Client -> FIN ACK
(Based on Fig 18.3, Steven TCP/IP Ill, Vol 2)

That's 9 round trips. 200ms / 9 == 22.2 ms

*IF* your ping time is close to this, then what you're really measuring
is TCP setup/teardown latency, not python latency. (*IF* you're on a
LAN and the latency is around that I'd suggest that you may have
discovered a local network problem) 

If your ping time is significantly lower -  eg you're running on
localhost - I'd suggest you translate your code to C (and/or post
your code), and repeat the same tests. So far you haven't shown
that python is slow here, just that something that you're measuring
causes it to be 200ms whether you use Twisted or not. (So you HAVE
shown that twisted isn't slow :-)

Another issue might be how you're measuring the latency - especially
whether if you're including startup of python in the 200ms or not!

Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: global interpreter lock

[Michael Sparks]

Paul Rubin wrote:
> Mike Meyer <[EMAIL PROTECTED]> writes:
>> Even simpler to program in is the model used by Erlang. It's more CSP
>> than threading, though, as it doesn't have shared memory as part of
>> the model. But if you can use the simpler model to solve your problem
>> - you probably should.
> 
> Well, ok, the Python equivalent would be wrapping every shareable
> object in its own thread, that communicates with other threads through
> Queues.  This is how some Pythonistas suggest writing practically all
> multi-threaded Python code.  It does a reasonable job of avoiding
> synchronization headaches and it's not that hard to code that way.
> 
> But I think to do it on Erlang's scale, Python needs user-level
> microthreads and not just OS threads.  

You've just described Kamaelia* BTW, except substitute micro-thread
with generator :-) (Also we call the queues outboxes and inboxes, and 
the combination of a generator in a class with inboxes and outboxes
components)
   * http://kamaelia.sf.net/

For those who really want threads as well, theres a threaded component based
class that uses Queues instead :)

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

Terry Reedy wrote:
> "Michael Sparks" <[EMAIL PROTECTED]> wrote in message
> news:[EMAIL PROTECTED]

>> def updater(interval, message):
>>   t = time.time():
>>   while 1:
>>  if time.time() - t > interval:
>> print message
> 
>yield None # add this 


Yes. (I can't believe I missed that out! :) I'd normally use yield 1,
but that's force of habit. I tend to use a true value to indicate a
never ending generator - partly because we use a false value to
indicate clean shutdown of a component to our scheduler.

Thanks :)


Michael
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Advanced concurrancy

[Michael Sparks]

Peter Tillotson wrote:

> I've not yet had a chance to try some examples, but i've looked through
> the documentation. It feels quite familiar, 

It hopefully should. The approach is based essentially on an asynchronous
hardware approach, on the recognition that the fundamental reason that
hangs together is because it can be reasoned about in a similar manner to
CSP. (That naturally leads to similarities of course with Occam)

> but i'd say that it is closer to Jade, the fipa (federation of intelligent
> physical agents) compliant agent framework 

Thanks - I've not come across those - I'll have to take a look at that in
detail. (With a cursory look, they sounds similar to an application area a
colleague is interested in investigating)

> than CSP or pi calculus.  I like the behaviour 
> (component microthread) model,  but the advantage of CSP / pi calculus is 
> that the resulting distributed system remains open to mathematical
> analysis.

Indeed. However, by trying to have the same basic limitations in the system, 
Kamaelia should be amenable to the sort of approaches taken by hardware
verification systems. It's possibly worth mentioning that there is one
person at BBC R&D who is extremely interested in applying CSP style
analysis techniques to Kamaelia systems whom I sure would be interested in
chatting to you. (Indeed the similarities between Kamaelia and CSP is
biggest reason for his interest :-)
 
The idea of a verification system for software that is actually in use for
real world systems would be extremely nice/useful to have (though not
simple to achieve).

FWIW, I am interested in hearing suggestions on changing the system to make
it more amenable to mathematical analysis. (Largely because I'm a fan of
formal methods where possible, but pragmatically view TDD as the current
most practical way of getting towards the levels of reliabiliy aimed for by
formal methods)

> For concurency its is the best framework i've seen :-) 

Thanks!

> Have you come across the pylinda tuplespace implementation. It might be
> well worth a  look. 

I have, and I liked what I saw :)

> I might be barking up the wrong tree - but it seems to me that 
> there could be considerable overlap between tuplespaces and mailboxes,

Probably the closer match is the co-ordinating assistant tracker (I can
see what you mean though). That provides a basic global key/value
store/retrieve service, which in a concurrent system is probably most
akin to a Linda tuplespace. In a biological system it's more akin to the
hormonal system which is more of the model I like to think of as to /when/
to use that part of the system. (Evolution having more 'experience' than
me on how to build a massively concurrent system after all !!)

It's generally only used at present for one of two purposes:
   * Finding services that already exist (so that we don't try to open two
 displays for example)
   * Stats collection.

Using a Linda tuplespace there would be an interesting next move, however
we generally only extend the system based on specific need rather than
exploring architecture (Which I'm sure you understand!).

> though you did mention that you were moving towards twisted as the
> underlying platform for the future.

Perhaps not underlying, but definitely interoperating with (It seems rather
wasteful to reinvent wheels).

> I'm quite interested in the mini version and also using the modules as
> mobile code rather than installing it formally. 

I'll document it slightly better and post up on the website in the next 48
hours or so. In the meantime, the optimisation test which includes a
slightly modified mini-axon can be found here:

   http://cvs.sourceforge.net/viewcvs.py/kamaelia/Sketches/OptimisationTest/

Specifically the file "simplegame.py". This includes a cutdown version of
Axon that's about 100 lines long (lines 41 - 144 specifically), with the
important classes being:

  class microprocess(object):
  class newComponent(object):
  class scheduler(microprocess):
  class component(microprocess):
  class send_one_component(component):
  def linkage(source,sink):

> I'll probably zip the Axon directory and distribute the zip with the code,
> adding the zip to  the python path dynamically.

Sounds interesting. If you go ahead with this we'd be interested in hearing
how you get on.

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

Tim Golden wrote:
> I just wanted to say that I find the ideas behind Kamaelia
> interesting, and I only wish I had an application for it!
> Because I'm not especially into media-streaming, I'm more
> interested in it from the point of view of the generator-based
> architecture.

It's nice to know that people find it interesting even without a direct
need. (Indeed I suspect this is the case with many open source projects.)

One thing we are learning as we go along is how to use generators to
refactor out commonly used code out of loops like this. One example is
status updating:

import time
def long_running_thing(foo):
   t = time.time()pplication for it!
> Because I'm not especially into media-streaming, I'm more
> interested in it from the point of view of the generator-based
> architecture.

It's nice to know that people find it interesting even without a direct
need. (Indeed I suspect this is the case with many open source projects.)

One thing we are learning as we go along is how to use generators to
refactor out commonly used code out of loops like this. One example is
status updating:

import time
def long_running_thing(foo):
   t = time.time()
   while running:
   << do something long running>>
  if time.time() - t > update_interval:
 print update_message

An alternative to this is this:

def updater(interval, message):
   t = time.time():
   while 1:
  if time.time() - t > interval:
 print message

def long_running_thing(foo):
   myUpdater = updater(5, "Still running")
   while running:
   << do something long running>>
   myUpdater.next()

You can do similar things for frame rates and so on. Obviously in our system
we generally go one step further and translate the resulting generators into
components, but sometimes simple conversion to generators alone
can make the code much clearer.

Tim Golden wrote:
> What prompted me to write now was in appreciation of Michael's
> taking the effort to recast his words into a form which might
> well be understood better by others (including me) when he
> might well have simply huffed a bit and told the other poster
> to read the words on the site and stop complaining about the HTML!

I tend to generally think that for each poster to a newsgroup there's often
several lurkers who share the same opinion, and given text is a very harsh
medium, I try to make allowances for that. 

Answering questions can sometimes also help others to assist with
documentation patches of course. Technical writing not necessarily being
the same skill set as code writing or architecture design after all :-)

I must admit I do remember the earlier days of usenet when people would ask
a question, get several answers, and the original poster would summarise
the responses (something that doesn't happen as often these days
unfortunately, possibly due to services like google news making it easier
to go back and see the discussion).

phil hunt wrote:
> (And on a more general level to improve the average quality of 
> documentation for open source projects. Something which is bvery 
> often deficient.)

You have to remember as well that most open source projects rely on
volunteers, and anyone can write docs. If you want to improve the average
quality of docs for open source projects there is only one way to do this -
to be prepared to write the documentation yourself or pay someone to do
it. (In our case documentation is driven by internal need, which doesn't
always match the needs of people externally. You might not've seen the
white paper giving a detailed overview of the project from a technical
perspective here: http://tinyurl.com/8543e)

Tim Golden wrote:
> Thanks, Michael. Hope the project continues apace and if I
> can ever find a need for it, I'll know where to look.

Thanks for your kind words.

Best Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

phil hunt wrote:
> On Wed, 03 Aug 2005 16:57:34 +0100, Michael Sparks <[EMAIL PROTECTED]>
> wrote: 
...
>>Which aside from other things means you can't build (say) a video
>>& SMIL playback system trivially, yet.
> 
> Isn't SMIL something that's goinhg to go into web browsers? In which
> case, you'd presumably not want to build one yourself, I imagine?

I was using SMIL as an example to be illustrative of a class of networked
applications that require visual support. A more direct example
would be MHEG (not mpeg) decoding and display systems, which
have similarities to SMIL systems in terms of capabilities required.

>>It's designed to make bolting things together to make these sorts of
>>system simpler and easier.
> 
> What you say "bolting things together" do you mean writing Python
> code? 

Yes.

> Or will there be some other way? 

A clicky pointy approach would be nice. (There's two sorts of interface in
the world IMO - "tappity tappity" and "clicky pointy" - if being equally
disdainful :) Unix shell systems are very "tappity tappity", and whilst I
can do "tappity tappity", and have been doing "tappity tappity" for my
entiring working life, I do prefer "clicky pointy" systems.)

Though building systems using AR Toolkit type systems would be the ideal.
( http://www.hitl.washington.edu/people/poup/research/ar.htm#artoolkit )
That's probably a couple of years off though (at least).

> What I have in mind is something like a project I worked on some
> time ago, where a program could be written by drawing boxes on a GUI
> application, and drawing lines (representing data flow) to connect
> the boxes. So one half of a VoIP application might look like:
> 
>   +--+ ++ +---+
>   | listen   | | convert to | | volume|
>   | on udp   |>| audio  |>| control + |
>   | port 600 | | stream | | output|
>   +--+ ++ +---+

>From example 4:

pipeline(
Multicast_transceiver("0.0.0.0", 1600, "224.168.2.9", 0),
detuple(1),
SRM_Receiver(),
detuple(1),
VorbisDecode(),
AOAudioPlaybackAdaptor(),
).run()

Join multicast group "224.168.2.9", listen for packets on port 1600, throw
away information relating to who sent it, pass it through a simple reliable
multicast reordering subsystem, throw away the annotation, decode the
vorbis, and playback.

The visual representation of that isn't dissimilar to your diagram above.

> With something like this novel multimedia apps could be prototyped
> quickly (though making them into useful apps would take extra work
> -- in the case of a VoIP application you'd want a phonebook, for
> example).

Agreed.

>>At the same time it's designed to encourage
>>writing code in a way that makes it simpler.
> 
> Examples would be useful here. Both of what it can do now, and what
> it will be able to do eventually.

Examples of what can be done now are here:
   * http://kamaelia.sourceforge.net/KamaeliaStatus.html

I'll think about the latter part of that. (I personally loath technologies
that promise much and deliver little. Hence why I like to focus on what
we /can/ do)

>>The natural side effect of
>>this is the  system might make it easier to take advantage of multiple
>>CPU systems as they come online, since it makes a system naturally
>>concurrent. As the original announcement said "Kamaelia is designed as
>>a testbed". And by testbed I mean testbed as it testing out new ideas,
>>see if they work and see if they pan out. (Not as in a testing suite)
> 
> So what it will eventually do is not cast in stone?

The high level goals are to solve the problems in the Challenges document.
We're not particularly wedded to any one particular approach, and Kamaelia
can/should be moulded to solve those problems, not the other way round.

By definition along the way a variety of components will be generated, which
can obviously be put together in an arbitrary fashion, assuming type
adaptors as necessary, to form new interesting systems.

This ability to explore new/interesting systems quickly is something
generally of value to an R&D dept, so there's some people at BBC R&D
interested in using Kamaelia for all sorts of interesting projects. (which I
can't discuss here)

>>Components are object instances of python classes. The data passed
>>between components are object instances.
> 
> What sort of objects? 

Python objects. Numbers, strings, chunks of audio in a string, chunks of
audio in a binary object wrapped by a python class, etc. Precisely as I
said.

> For example, if an application is audio 

Re: ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

phil hunt wrote:
> On Wed, 03 Aug 2005 16:57:34 +0100, Michael Sparks <[EMAIL PROTECTED]> 
wrote:
>>> Is the audience programmers or
>>> less technical people? A project that allows non-technical people
>>> to build complex network applications is an ambitious one, but not
...
>>It's a little ambitious at this stage, yes.

> But it couldbe there eventually?

Could? Yes. Will? Can't say. I can agree it would be nice, and given
time/resources (or someone sufficiently interested externally) then it
may happen. (IMO there's no real reason that it couldn't happen aside
from time/effort/resources)

>>> What sort of servers and clients?
>>Whatever you feel like. If you want a server to split and serve audio,
>>you could do that.
> This is streaming audio, right? For non-streaming I can just use an
> ftp or http server.

There's more to network servers and clients than just audio & video, or
unidirectional download.

For example the visualisation/introspection tool is an example of a
client server system. The server is the visualisation tool. It listens on a
specified port waiting for a connection. The client connects and sends
data to it about the internal structure, and the server displays this.

>>>>   * Quickly build Multicast based network servers and clients
>>> Serving what? Could I use it, for example, to build an n-player
>>> encrypted  VoIP server to allow people to do conference calls over
>>> the Internet?
>>
>>You could do that probably. (Though we don't have a component
>>for audio capture (though a read file adaptor reading from /dev/audio
>>might work depending on your platform I suppose) and audio
>>encoding at the moment, so those would probably be the core
>>components to integrate.
> 
> That's a slightly worrying answer for me, worrying because it seems
> I've misunderstood the nature of the project. I assumed that
> components for audio capture, and related activities, would be at
> the heart of the project.

*Our* main interest, at the moment, is in /delivery/ of content. 

Dealing with capture would be largely re-inventing wheels before we know
whether the framework is a suitable framework. We are looking at making it
possible to use pymedia for dealing with capture/encoding/decoding.

There's a number of things along the way we need to deal with this, but
we're not starting from the perspective of capture. 

(After all for capture we can generally look at using dedicated encoder
hardware that will often spit out it's encoded information in the form of a
network connection. As a result capture and encoding hasn't been a priority
as yet. Sometimes looking into a project from the outside I can appreciate
that certain decisions might look strange, but consider that you don't need
to worry about capture in order 

>>> (I mean proper encryption here, the sort GCHQ or the NSA can't break)
>>
>>I'd be impressed if that could be written, using anything really. (Can't
>>implies never)
> 
> What -- good encryption? That's pretty much a well-known technique
> these days (unless the NSA has some *very* advanced hardware in
> their basement, which I strongly suspect they don't).

You said *can't*. That says to me cannot ever be broken. If you have a large
number of listeners, as your statement implied, that implies decryptable by
many listeners - you then just need one compromised listener (essentially
you're asking for the equivalent of implementing a DRM system that the NSA
couldn't break...).

If you can provide me with a library that you can guarantee that it will
satisfy the following properties:

   encoded = F(data)

and a piece of client code that can do this:
   decoded = G(encoded)

Then yes, that can be wrapped. That's trivial in fact:
---(start)---
from magic import unbreakable_encryption

class encoder(component):
   def __init__(self, **args):
   self.encoder = unbreakable_encryption.encoder(**args)
   def main(self):
  while 1:
  if self.dataReady("inbox"):
 data = self.recv("inbox")
 encoded = self.encoder.encode(data)
 self.send(encoded, "outbox")
  yield 1

class decoder(component):
   def __init__(self, **args):
   self.decoder = unbreakable_encryption.decoder(**args)
   def main(self):
  while 1:
  if self.dataReady("inbox"):
 data = self.recv("inbox")
 decoded = self.decoder.decode(data)
 self.send(decoded, "outbox")
  yield 1
---(end)---

If you believe you can implement F&G for general use such that F&G
can //never// be decrypted by anyone other than authorised recipients, I
suggest

Re: ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

I've reordered the q's slightly to avoid repetition... Also by answering
this question first, it may put the rest of the answer into context
better.

phil hunt wrote:

> At what stage of completion is it? 

This is something we deliberately try to reflect in the version number.
Yes, you can build network servers and new protocols relatively simply
at the moment. Yes, you can integrate with pygame in a limited useful
fashion at present. Yes we have audio playback.

However we don't have yet... (some examples)
   * Decent GUI integration yet. 
   * /Full/ pygame integration. 
   * Nice integration with pymedia
   * Direct support for Dirac.
Which aside from other things means you can't build (say) a video
& SMIL playback system trivially, yet.

As a result that's why the version number is 0.2 - whilst you /can/ do a
lot, there's a lot more to do. Clearly that also naturally implies that
we don't expect any end user to be looking at the site. (The low
version number will normally scare them away)

>>The project aims to make it simple to build networked multimedia
>>systems (eg audio, video, interactive systems),
> 
> There's plenty of software that facilitates networking, for example
> Python already has software for tcp and http clients/servers, and
> for xmlrpc remote procedure calls.

There is indeed.

> So what does Kamaelia do that's extra? I imagine it's to to with
> streaming large amounts of data. For example, a streaming video or
> audio player. Or VoIP, perhaps.

It's designed to make bolting things together to make these sorts of
system simpler and easier. At the same time it's designed to encourage
writing code in a way that makes it simpler. The natural side effect of
this is the  system might make it easier to take advantage of multiple
CPU systems as they come online, since it makes a system naturally
concurrent. As the original announcement said "Kamaelia is designed as
a testbed". And by testbed I mean testbed as it testing out new ideas,
see if they work and see if they pan out. (Not as in a testing suite)

Probably the best way of describing the difference is this... After my
talk about Kamaelia at Europython, I had an long chat with Tommi
Virtinan about integration between Kamaelia and Twisted. I haven't had
a chance to follow up with him yet regarding how this would work, though
I have set a line in the sand aiming to have easy integration between
Kamaelia and Twisted before Kamaelia hits version 1.0.0. The impression
I got from Tommi was that he was most interested in the communications
aspect - the fact we can bolt together systems in a manner directly
akin to Unix pipelines, though I suspect he found the graphines aspect
more interesting. 

Or as someone asking a similar question at Open Tech exclaimed after
I finally managed to explain it better to them "Ooooh - you're trying
to make concurrency EASY!".

> OK, so what do the components in the pipelines do?  What sort of data
> do they hold? Unix pipelines act on ascii files; I assume you are
> do this on audio and visual data. What langauage will the ele,ments
> in thne pipelines be written it? I assume some will be in C/C++ for
> speed.

Components are object instances of python classes. The data passed
between components are object instances. Clearly these python classes
can be written in python, C, C++, pyrex etc. Currently all of Kamaelia's
components are python based. Some existing components make calls
into some related C libraries via python bindings.. An example of
writing a component using Pyrex can be found here:
   * http://kamaelia.sourceforge.net/PyrexComponents.html

>>It is designed as a practical toolkit, such that you can build systems
>>such as:
> When you say "you" who do you mean? 

Generally I expect the readership of c.l.p/python-list@python.org to be
programmers. Python is generally easy to pick up and having asked
someone who's not done much programming beforehand (beyond
a small amount of VB and Access), and is pre-university to use the
system to build a simple streaming system prototyping visualising PVR
content on a mobile (and watching them succeed), they seem
relatively reasonable examples.

At some point, the ability to allow non-programmers to bolt together
Kamaelia systems would be desirable, but a first step is making it
simpler for programmers to bolt together systems. We currently have an
interactive visualisation tool(*), and the logical extension of that is
a tool that allows systems to be bolted together without knowing any
code.
   (*) http://kamaelia.sourceforge.net/AxonVisualiser.html

It'd be an interesting side project for someone to take forward, and
might be low hanging fruit in terms of projects. (Especially if viewed
initially as a tool for assisting development, rather than replacing
development)

> Is the audience programmers or
> less technical people? A project that allows non-technical people
> to build complex network applications is an ambitious one, but not
> impossible (I'd find 

Re: ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

Phil Hunt wrote:

> Kamaelia seems it might be an interesting project. However, I don't
> think the project is well served by this announcement -- which I 
> find vague and hard to understand. Which is a shame, because it
> means that other people probably don't understand it very well
> either, which means less people will use it.

It is a shame, and thanks for mentioning this. Let me have another
go :-) (ripping to shreds welcome :)

OK, here's a better go. (It's always difficult to think where to pitch
this sort of thing)
---START---
The project aims to make it simple to build networked multimedia
systems (eg audio, video, interactive systems), which are naturally
componentised and naturally concurrent allowing quick and fast reuse in
the same way as Unix pipelines do. 

It is designed as a practical toolkit, such that you can build systems
such as:
   * Ogg Vorbis streaming server/client systems (via vorbissimple)
   * Simple network aware games (via pygame)
   * Quickly build TCP based network servers and clients
   * Quickly build Multicast based network servers and clients

It runs on Linux, Window, Mac OS X with a subset running on Series 60
phones.

The basic underlying metaphor of a component us like an office worker
with inboxes and outboxes, with deliveries occuring between desks,
offices, and depts. The component can thus do work anyway it likes but
only communicates with these inboxes and outboxes. Like office workers,
components run in parallel, and to achieve this are generally
implemented using python generators, but can also used threads.

The rationale behind the project is to provide a toolkit enabling the
development of new protocols, including streaming, for large scale
media delivery. The license essentially allows use in proprietary
systems without change, but all changes to the system itself must be
shared.
---END---

Is that clearer ? 

A short summary of all that could be:

"""Kamaelia is a networking/communications infrastructure for innovative
multimedia systems. Kamaelia uses a component architecture designed to
simplify creation and testing of new protocols and large scale media
delivery systems."""

Hopefully that's clearer than:

>>Kamaelia is a collection of Axon components designed for network
>>protocol experimentation in a single threaded, select based
>>environment. Axon components are python generators are augmented by
>>inbox and outbox queues (lists) for communication in a communicating
>>sequential processes (CSP) like fashion.

[ which you noted "I really have very little idea what this means." ]

> The information I can guess out of this is: "Kamaelia is a library
> for creating network protocols that sit on top of tcp and/or udp".
> Given that it's a BBC project, I imagine we're talking about
> protocols for transferring large amount of data, e.g. sound or
> motion pictures.

Indeed. However, if you're having to guess I'm not doing the best job
(despite trying!) at explaining this.

>>More background on the motivations behind Kamaelia
>>can be found here: http://kamaelia.sourceforge.net/Challenges/
> 
> There's something interesting here:
> 
>   In Building Public Value, the BBC has committed to the delivery of
>   BBC content to the British Public over this multitude of systems,
>   in a way that is enjoyable, accessible and **repurposable** by the
>   British Public in the way they choose.
> 
> (my emphasis)
> 
> This is a laudable aim, but if the BBC want to do this, the first
> step they could make is to open up their content under a Creative
> Commons licence.



I would suggest that you forward this comment directly to the team
involved in working on the Creative Archive. You can do this via this
URL:
   * http://creativearchive.bbc.co.uk/

This is unfortunately not as simple as "just putting stuff up", and any
comments you have, I'm sure would be welcome. However, the Creative
Archive hasn't yet been put to a Public Value Test (including impact on
industry), and rights is a much more problematic issue than you might
expect. (eg background music on the radio in eastenders, posters on
people's walls...)

There's also http://www.bbc.co.uk/info/contactus/form.shtml which might
be more appropriate.

On a positive note, the Creative Archive has more organisations involved
than just the BBC, it inlucde the BFI , Channel 4, Open University &
Teachers'TV, so even if these things take longer than some people might
like they do seem to be moving. Public organisations are weighed down
whether the should do something, as well as whether they can.

BBC R&D's role in this is to make these decisions possible.

> They could start with their news, current affairs 
> and documentaries. 

BBC Backstage is a start here - at least with some of the online
content. (http://backstage.bbc.co.uk)

> But the download only staryed up for a week, and they didn't put the
> files under a CC license, so they could be redistributed. Why not? It
> would have cost them no more to do so than

ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

Kamaelia 0.2.0 has been released!

What is it?
===
Kamaelia is a collection of Axon components designed for network
protocol experimentation in a single threaded, select based environment.
Axon components are python generators are augmented by inbox and outbox
queues (lists) for communication in a communicating sequential processes
(CSP) like fashion.

The architecture is specifically designed to try and simplify the
process of designing and experimenting with new network protocols in
real environments. More background on the motivations behind Kamaelia
can be found here: http://kamaelia.sourceforge.net/Challenges/

The focus of this release adds in support for introspection, pygame based
interfaces, 4 new examples using these, visualisation tools, as well as
syntactic sugar to make building Kamaelia systems simpler. (Specifically
Graphline and pipeline systems. This build upon the existing base allowing
TCP and Multicast based client/server systems.

Other additions and changes include updated examples, variety of bugfixes
in existing components (some pre-emptively discovered by introspection), and
a variety of utility components. It is now also possible to write components
using threading as the concurrency model rather than generators - allowing
integration with thread/blocking only based systems.

The system is known to work under Linux, Mac OS X, Windows and a subset
has been tested on Series 60 mobiles.

General feedback is welcome either directly, mailing lists or via the
project weblog which is here:
* http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi

What's new in version 0.2.0 ?
=

Lots! Full release notes can be found here:
   * http://kamaelia.sourceforge.net/Kamaelia-0.2.0-ReleaseNotes.html

Editted highlights...

Debian Packages! Many thanks are due to Gintautas Miliauskas, Programmers
of Vilnius, for assistance in building Debian packages. (The current
packages are based on his, and any errors are likely to be mine, not
his) These have been tested successfully on Ubuntu 5.04.

4 new examples have been added showing of various new subsystem:
   * Example 5 : This creates a simple streaming system, and looks inside
 to see what components are running/active, and passes the resulting
 information an Axon Visualiser.

   * Example 6 : This is a simple/generic topology visualisation server.
 Accepts the following commands over the network
ADD NODE id label auto -
ADD LINK id id
DEL NODE id
DEL ALL
 As this stands this is pretty useful, but that's pretty much everything
 it does like this.

   * Example 7 : This shows how the visualisation subsystem can be extended
 to work in different ways. What this does by default when run is
 randomly create new nodes and new linkages quite quickly, allowing
 you to see how the system works.

   * Example 8 : Sample slideshow/presentation tool. Unlike traditional
 slideshow/presentation tools, you can modify this to run arbitrary
 components. An example of how this can work is provided - allowing
 stepping through some graph visualisations along with the presentation.

A Tools directoy has been added with the following tools:
   * Axon Shell. (Requires IPython) Implements a simple command line shell
 which allows experimentation with Axon systems - the shell runs a
 scheduler as a background thread. For a tutorial of use, see:
   * http://kamaelia.sourceforge.net/AxonShell.html
   * Axon Visualiser. Implements a simple tool for looking inside (quite
 literally) running Axon/Kamaelia systems. This allows a very different
 style of debugging and can be extremely useful. Tutorial on its way!

Graphlines and Pipelines

  These are probably the most useful additions to Kamaelia since 0.1.2.
  They are essentially syntactic sugar for building and working with
  systems of components, but make building interesting systems rapidly
  out of pre-existing components fun and easy. The pipelines follow the
  same sort of model as the Unix pipeline. Graphlines are something new,
  and like pipelines and all linkages may take any data along them.

  Please take a look at the release notes for a graphline example.

  A couple of simple pipelines looks like this:

   pipeline(
  ReadFileAdaptor(file_to_stream, readmode="bitrate", bitrate=40,
  chunkrate=50),
  blockise(), # Ensure chunks small enough for multicasting!
  Multicast_transceiver("0.0.0.0", 0, "224.168.2.9", 1600),
   ).activate()
   pipeline(
  Multicast_transceiver("0.0.0.0", 1600, "224.168.2.9", 0),
  detuple(1),
  VorbisDecode(),
  AOAudioPlaybackAdaptor(),
  ).run()

A selection of other subsystems have been added - targeted at visualisation
of Kamaelia (and other graph structured) systems using pygame. The layout
mechanism is a simple physics engine. 

Key packages of note added: Kamaelia.UI, Kamaelia.UI.Pygame,
Ka

Re: Python language ver 2.4 , development platform 0.4

[Michael Sparks]

Dark Cowherd wrote:

> The Python language is at ver 2.4 and a thing of beauty. As a
> development environment IMHO it is probably 0.4

Have you considered looking at any of the commercial IDEs? Personally I
*like* command line based systems, but I do know many people who swear
by GUI based IDEs. If you have, what did want to see but didn't see ?
(System usability from my perspective starts with developers, since whilst
beauty is skin deep, ugly goes to the bone)

Most of these IDEs tend to integrate with one of the more common GUI
toolkits as well, which /may/ be more TOOWTDI from your perspective. As for
batteries included, I suspect it depends on what you expect as
batteries :-)

> Small projects, utilities which I am going to code myself I will
> continue to use Python because I love it.

Nice to hear.

> I hope to learn Python well enough to be able to contribute to some
> projects to achieve this. But I still see that as six to nine months
> away.

I'm sure such contributions would be welcome - especially if you're 
scratching an itch to help resolve the things you currently suggest as 
deficiencies :-)

Regards,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python-cgi web or Perl-cgi script doubt

[Michael Sparks]

praba kar wrote:
>I want to know difference between
> Python-cgi and Perl-cgi and also I want
> to which one is efficient from the performance.

Possibly the most important difference between the two is when you're using
JUST cgi - ie no mod_perl, no mod_python, etc. With python, if your cgi is
structured to use more than one file, then your programs can be compiled
and stored on disk (compiled to something called bytecode). This can just
be loaded and executed immediately on subsequent runs.

With perl, the program needs to be loaded and recompiled with each run.

For small CGI systems this difference is negligible, but for large CGI based
systems (eg when using a large framework), these differences can become
very important - if you do not have the option to use mod_perl/mod_python
or similar.

However, many people running large web systems do tend to use mod_perl and
friends when using perl which discounts this argument pretty much
completely.

That said, I'd suggest looking to see *what* you want to do, look at what
*tools* are available and *then* decide which *language* to use. Personally
I find python nicer for this sort of things these days - larger because it
naturally encourages better strutcure. (And this is after having using perl
day-in, day-out for 5 years, and still liking perl :-)

Hope that's of some use,


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Advanced concurrancy

[Michael Sparks]

from simple audio clients/servers through to presentation
tools. That presentation can be downloaded from here:
   * http://kamaelia.sourceforge.net/Kamaelia_OT2005.tgz

Both the Europython & Open tech bundles actually include the core
concurrency systems (called Axon), and the component modules (Kamaelia) in
the bundle, in addition to the slides and install docs. The presentation
tools included in the bundles are written in python using Kamaelia.

The project website is here: http://kamaelia.sourceforge.net/

> I'm trying to develop distributed / Grid computing modules based on
> python. The aim is to be able to use barriers for synchronisation and
> channels for communication between processes running on a single box.
> Then the jump to multiple processes on multiple boxes and eventually to
> MPI implementations. Hopefully, each jump should not be that big a leap.
> 
> Of course it would be nice if there was a robust way of managing
> concurrency in python aswell ;-)

It's getting there - the recent PEP regarding the ability to cause a
generator to have an exception thrown inside it provides very similar to
Unix signals as provided via "kill", which will be nice when 2.5 is
released.

Hopefully you find the system useful,

Best Regards,


Michael.
--
Michael Sparks, Senior R&D Engineer, Digital Media Group
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This e-mail may contain personal views which are not the views of the BBC.

-- 
http://mail.python.org/mailman/listinfo/python-list

Kamaelia Talk at Open Tech 2005, anyone for sprinting?

[Michael Sparks]

Hi,


Apologies first to those outside the UK... Open Tech 2005* is a follow
on from previous years' NotCon events which are community driven
low cost events by geeks & developers for geeks & developers. (Much
like Pycon & Europython but much more general in nature)

Website: http://www.ukuug.org/events/opentech2005/
Where/When: Hammersmith, London, UK, July 23rd

The reason I'm posting about it is because I'm going to be talking
about Kamaelia there. Unlike Python UK & Europython this is going to
be a 15 minute talk, so I'll be focussing on what we've been doing
with Kamaelia, what you can do with Kamaelia, and so on rather
than going through internals.

In addition since my talk on Kamaelia is before lunch, I'd really like
to help people get started using Kamaelia (this really the point of
giving talks!), so I'm going to be running a small ad-hoc/mini sprint
that afternoon. 

There's a variety of possible sprintable things ranging from how to
put together simple Kamaelia systems through to using Kamaelia for
simplifying networking on Nokia mobiles through to using Kamaelia
for writing games (since we have some nice Pygame based components
now). 

I'll probably be sprinting on converting a proof of concept P2P
swarming algorithm into a protocol for creating TCP server swarms.
This is for joining together multicast islands to make it so that
internet broadcasting, rather than narrowcasting, can become
a reality for the BBC - which has clear benefits for everyone!
(Even though the internet is not TV :)

Kamaelia is designed to be able to picked up relatively quickly/easily,
but having a working knowledge of python is probably a necessity.

If you're interested, please let me know! (Either before, or at Open
Tech :)

Best Regards,


Michael.
-- 
[EMAIL PROTECTED], http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python concurrent tasking frameworks?

[Michael Sparks]

Chris Stiles wrote:

> I was wondering if anyone had a list of the various (presuming more than
> one)
> Python tasking frameworks?  I know of Twisted already, but I'm really
> looking
> for something along the lines of a task/thread pool type of arrangement. 

The Kamaelia framework might or might not be what you're looking for. It's
not really thread pool based, but rather based around the idea of
communicating generators (which as used as components) to form
interesting systems. (Though you can also use threads if you want)

Communications happens in a manner very similar to unix pipelines, but
since it's a single system data is passed (essentially) by reference rather
than serialised to go over a unix pipe, and also rather than a single 1D
pipe, you can form arbitrary "graphlines".

A simple TCP based vorbis streaming client can look like this:

pipeline(TCPClient("127.0.0.1",clientServerTestPort),
 VorbisDecode(),
 AOAudioPlaybackAdaptor()
).run()

(Assuming necessary imports!)

This says take the data the TCPClient sends out and pass to the
VorbisDecode, and pass its output to the AOAudioPlaybackAdaptor.
The corresponding server accompanies this as example3 in the
distribution.

A simple (useful) example of a "graphline", looks like this:

Graphline(
 CHOOSER = Chooser(items = files),
 IMAGE = Image(size=(800,600), position=(8,48)),
 NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
 PREVIOUS = Button(caption="Previous", msg="PREV",position=(8,8)),
 FIRST = Button(caption="First", msg="FIRST",position=(256,8)),
 LAST = Button(caption="Last", msg="LAST",position=(320,8)),
 linkages = {
("NEXT","outbox") : ("CHOOSER","inbox"),
("PREVIOUS","outbox") : ("CHOOSER","inbox"),
("FIRST","outbox") : ("CHOOSER","inbox"),
("LAST","outbox") : ("CHOOSER","inbox"),
("CHOOSER","outbox") : ("IMAGE","inbox"),
 }
).run()

This forms a primitive slideshow - in which a "chooser" gets a list of
filenames, which depending on which button is pressed (next, previous,
first, last) sends filename to an Image display component for display.

(Again assuming a little bit of setup)

I gave a talk about Kamaelia at Europython, using Kamaelia, so if you feel
like grabbing a copy of the presentation which also includes everything you
need to get started, please look here:
   * http://tinyurl.com/cnk9c

That includes a more complex slideshow, though you need more than one
mouse button for it to be useful. (It can show graph/network topologies
dynamically, and can be more non-linear in terms of presentation)

There's a tutorial on creating components here:
   * http://tinyurl.com/dp8n7

A longer detailed introduction document here:
   * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

And the main website/docs here:
   * http://kamaelia.sourceforge.net/
   * http://kamaelia.sourceforge.net/Docs/

Depending on your focus Kamaelia contains tools for building TCP &
Multicast servers and clients relatively easily and is designed to naturally
encourage the use of concurrency easily and scalably. There's also a growing
collection of components for more aural/visual work with a simple Pygame
based "window manager" (really stretching things there, surface manager
might be more appropriate) for components, and audio decode/playback.

It isn't as mature as Twisted though, so that should affect your viewpoint!
(Had some good chats at Europython though about how we can go about
integration with twisted!)

 I suppose it should be possible to have pools of existing
components for wiring into connections on demand though, which
might be interesting. 

Best Regards,


Michael.
--
http://kamaelia.sourceforge.net/

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Assigning to None

[Michael Sparks]

Mike Meyer wrote:
> Peter Hansen <[EMAIL PROTECTED]> writes:
>> Mike Meyer wrote:
>>> Yes. I once grabbed an old program that did assignments to None. But
>>> that's always been a bad idea.
>> What was the use case!?
> 
> Unpacking a tuple. Something like this:
> 
>   (foo, bar, None) = gen_tuple(stuff)

I tend to do:
   (foo, bar, _,_,_) = gen_tuple(stuff)

In cases where I decide I just want the bits I don't want. Aside from
anything else, if someone is testing things in the interactive prompt, it's
clear that the value is ephemeral, the fact it isn't name emphasises it's
unimportance and the repetition (in the above case) really makes it clear.
It also _looks_ like the value is thrown away as well.

And it doesn't clutter up your namespace...


Michael.

-- 
http://mail.python.org/mailman/listinfo/python-list