Hi,
On Saturday 28 August 2010 22:30:16 Gloria W wrote:
> If I send it a message via a SimpleServer, can components be added after
> run() is executed?
Fundamentally, yes. The longer answer to this was delayed. The reason it was
delayed is because I started writing something about using the visualiser
which can be useful for explaining this better, and then discovering a latent
bug in the TCPServer code.
OK, I didn't know that *then* but I do now.
Anyway, I'll be posting that later today, along with an explanation of *how* I
found the bug, since I think that's perhaps as much, if not more useful. In
particular, I've enhanced on part of linkage tracing, and also more in depth
system tracing.
Fundamentally though the answer to the above question is "yes".
> Right, I have my own components with which I did this, and it makes
> sense.
Indeed.
> I end up converting things that I can post to this list to
> generic examples which will not violate the NDA I signed when I started
> this work.
I guessed that might be the case. Violating NDA's falls into my list of bad
ideas, so I approve :-) (Not that approval is required, but hey :-)
> Instead of doing this from this point forward, I'm first
> developing the generic weather handling model, then rewriting it to meet
> the needs of the specific project.
Sounds sensible.
> The ConditionalTransformer rocks. Passing function factories to filter
> out messages is a beautiful implementation.
:-)
> > However, if I did want access to instance storage space, or to add in my
> > own defaults, I can do that too. For example, to change the above
> > "Divisible3" example into something generic, I want to shift the divisor
> > value out into the
> >
> > class definition:
> > class Divisible(ConditionalTransformer):
> > divisor = 4
> >
> > def condition(self, message): # condition callback with state
> >
> > return (message % self.divisor) == 0
> >
> > def process(self, message): # process callback with state
> >
> > return "Divisible by "+str(self.divisor) + " ! " +
> > str(message)
> >
> > And then I can use this as follows:
> > Divisible(TPP, "FOUR").activate()
>
> Right, all without having to call super(). This is one of my favorite
> features of Python.
Indeed, it's a very nice aspect of python.
> > Finally, I since "divisor = 4" acts as a Kamaelia inheritable default
> > value, I
> >
> > can override it in other instances:
> > Divisible(TPP, "FIVE", divisor = 5).run()
>
> Ah! Oh wow...
> > Then define 2 inheritable default values, which subclasses can override
> > explicitly.
> >
> > condition=None
> > process=None
> >
> > Then we declare our __init__ method signature. As usual we include **argd
> > to enable someone subclassing us to also define inheritable defaults.
> >
> > def __init__(self, processor=None, tag=None, condition=None,
process=None,**argd):
> This is really slick. I like it, but is it considered bad practice,
> because the inheritable values are implicit?
It's debateable.
The normal approach we've been using looks like this:
class ServerCore(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent):
port = 1601
protocol = None
socketOptions=None
TCPS=TCPServer
def __init__(self, **argd):
super(ServerCore, self).__init__(**argd)
This then allows override elsewhere.
However, the __init__ method now has entirely implicit arguments, which is a
little ugly. I'd personally wondered whether this would be viewed as pythonic
or unpythonic by someone experienced, but figured that the **argd (or
**kwargs) form was so useful that utility overrode things.
ie in this case pragmatism beat purity.
At Europython Raymond Hettinger, with David Jones was running a code clinic
where people could bring up there code for critique, etc. After seeing the
format, I figured why not and thought it worth seeing what Raymond would think
of it.
We went through all sorts, and if I was the sort of person to pull quotes out
of context, there'd be some really amazingly cool ones I could put here :-)
(The videos are on europythonvideos.blip.tv)
Anyway, it transpired that he thought that this:
class ServerCore(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent):
port = 1601
protocol = None
socketOptions=None
TCPS=TCPServer
def __init__(self, **argd):
super(ServerCore, self).__init__(**argd)
Was pretty close to the best thing he'd seen for this and managing defaults -
since that's effectively done here:
class component(microprocess):
def __init__(self, *args, **argd):
super(component, self).__init__()
self.__dict__.update(argd)
In particular, the reason being that the **argd form plays particularly well
with super() which in turn is a good combination of things in a library.
However, he also noted that you're then left with this signature:
def __init__(self, **argd):
Which is pretty ugly, and could be improved. (Hence pretty close, but not
quite ideal) In part it also means that you always have to use the keyword
form of things like this, which is a benefit in some respects, but a cost for
things where the arguments would be obvious.
He then pointed out though that you have access to locals() as well which
inside the body of a function gives you access to the arguments' values as
passed in.
As a result, you can do this:
class ServerCore(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent):
port = 1601
protocol = None
socketOptions=None
TCPS=TCPServer
def __init__(self, **argd):
super(ServerCore, self).__init__(**argd)
OR this:
class ServerCore2(Axon.AdaptiveCommsComponent.AdaptiveCommsComponent):
port = 1601
protocol = None
socketOptions=None
TCPS=TCPServer
def __init__(self, port=None, protocol=None, socketOptions=None,
TCPS=None, **argd):
[ argd.__setitem__(k,i) for k,i in locals().iteritems() if (k != \
"self")
and (i!=None)]
super(ServerCore, self).__init__(**argd)
(I guess the list comprehension could be pulled out as well)
The advantage of the latter becomes clear when I show examples of usage...
ServerCore(port=1601, protocol=EchoProtocol)
ServerCore2(1601, EchoProtocol)
... which is that of compactness at the cost of clarity of override.
Either way though, based on how excited Raymond got, rather than repulsed, and
how much he waxed lyrical about it being a good approach, I'll simply say that
it seems a good idea :-)
In particular, if you skip down to the class GreylistServer in this file:
* http://bit.ly/ajjHne
You'll see how the config file is used to configure every aspect of the
server, right down even inside the TCPServer code to override how connections
are handled.
It's that sort of utility that inheritable default values enables, and as you
say it's a toss up between explicitness/implicitness vs flexibility. Each app
will have it's own call on that.
> This is great, but it looks like it would cause a newbie a bad headache. :)
Quite possibly. In which case they don't have to use it :-)
Effectively it's an extra tool for dealing with awkward cases as cleanly as
possible, rather than trying to make clean cases awkward \o/ :)
> > This does two things: firstly it is an optimisation. Secondly it means
> > that we can just use the callback as process(value) rather than the
> > slightly more obfuscated (self.process)(value).
>
> The small, embedded Python tricks here are awesome.
:-)
Force of habit I guess :)
Incidentally, my primary reason for doing this:
process = self.process
...
while not self.dataReady("control"):
for message in self.Inbox("inbox"):
self.send(process(message), "outbox")
Is not for performance, but for clarity. I simply view this:
self.send(process(message), "outbox")
to be clearer than this:
self.send((self.process)(message), "outbox")
It also happens to be quicker, which is a bonus, but not the point :)
> > And the upshot of this is that you can have a TaggingPluggableProcessor
> > which you can add more ConditionalTransformers to after the fact, by
> > having them simply send an appropriately formed message to the
> > TaggingPluggableProcessor.
> >
> > You'll note however that this process is beginning to look alot more like
> > a backplace and a conditional subscriber... So there may actually be
> > some mileage in looking at that as a next possible transform. Or maybe
> > not.
>
> The backplane code I have works really well for me, with the exception
> of being able to add components after I run().
I'll cover that in the mail I refer to above.
> > Designing things such that the mechanics of this are handled by a base
> > class - and the actual work happens in a subclass can also lead to
> > clearer separation of concerns. For example ConditionalTransformer
> > itself is very clear in that it's purpose is to set up a means of wiring
> > the component in, whereas "Divisible" is very clearly a processor which
> > processes stuff based on a condition, but still uses a given source.
>
> Yes, it seems like the same subdivision found in map-reduce algorithms.
I hadn't thought of it that way, but you're right.
> > Whether that's your style is another matter, but it's one that makes
> > sense to me.
> >
> > OK, hopefully that's made how you would add additional processors after
> > the fact clearer. Regarding the larger design questions I'll leave those
> > for tomorrow. (It's late here :-)
>
> Understood. Again, thanks for your time and energy. This is really awesome.
As noted above, apologies for the delay. In writing the larger design question
thing I came across a bug that took a little while to stomp, and was quite an
important one to stomp on. (Doesn't show up for certain sorts of server, but
does for others)
> > Quickly skimming your example though, I *don't* think you need multiple
> > processes to achieve what you're after. You can after all have as many
> > pipelines active at any instant as you wish.
>
> I am still wrestling with these issues:
>
> After run() is executed, I will need to change/kill off/start
> processors/groups of processors, based on changing weather conditions.
> Do I shut down and reconstruct the whole series of pipelines to handle
> this? This is why I was tempted to try a NACK based UDP backplane with
> different processes.
OK, I'll look at that at the same time. There's a bunch of choices you could
make - from message based through shutting down things and restarting them,
which are probably equally valid depending on what you want to do. (Even in a
weather thingy :-)
> > ... and yes, this is very much a place for thinking out loud :-)
>
> That's what keeps me coming back :)
Cool :-)
Michael.
--
You received this message because you are subscribed to the Google Groups
"kamaelia" group.
To post to this group, send email to kamae...@googlegroups.com.
To unsubscribe from this group, send email to
kamaelia+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/kamaelia?hl=en.
