Hi William,
On Thu, Apr 23, 2009 at 17:10 -0400, William McVey wrote:
> On Wed, 2009-04-22 at 10:54 -0400, William McVey wrote:
> > I'm using py.execnet with the SshGateway and I'd like to fire off
> > logging messages that get sent over the ssh connection. I was wondering
> > if someone has already coded up a logging handler for this.
I think it is a fine usecase.
> After a bit of frustration, I have remote logging over an execnet
> channel working, with no changes to the py.execnet code itself to
> support this. I had to resort to a nasty trick in order to get the
> channel to send and receive the logging.LogRecord objects. I think this
> could be made a lot cleaner if py.execnet were tweaked to allow defining
> a dictionary of namespaces to use when deserializing objects coming over
> a channel. If there is any interest in this by the py lib maintainers,
> I'll go ahead and fork the code using bitbucket, add this capability in
> (probably as a new method on the channel object) and issue a pull
> notification.
> Anyway, attached is a simple demo script that shows the ExecNetHandler
> logging handler in action. I'd love to hear feedback on this.
you could put the "remote_code" into a module and pass the
imported module to remote_exec - this allows to keep syntax
coloring and also allows to more easily unittest it. Maybe
that would alleviate some of the frustration? Also you could
maybe just send a dictionary of instance values and call a
Logrecord object with it as **kwargs.
As you say yourself, your code works on the presumtion that
py.execnet uses repr/eval for sending and receiving of
objects. That may change soon and would break your code.
A more general solution for sending non-marshallable objects
is to have some pickling layer on top of current channels.
For use with distributed testing, i've implemented
a "PickleChannel", see here:
http://bitbucket.org/hpk42/py-trunk/src/c90f1bf8bac4/py/test/dist/mypickle.py
with unit and functional tests here:
http://bitbucket.org/hpk42/py-trunk/src/c90f1bf8bac4/py/test/dist/testing/test_mypickle.py
This code also uses an additional concept "ImmutablePickle"
which restricts object pickling in the following way: If
Process A sends an object OBJ to process B and process B sends
it back (either directly or as an attribute of another object)
then A will get a reference to OBJ and not a fresh copy.
Usually pickle would create a copy in process A. I've found
this preserving of object identity useful for distributing
testing tasks.
Obviously one could also use plain pickle, maybe by
introducing a pickling option to remote_exec() that could have
different values like e.g. "ipickle", "pickle", "repr" etc.
If you are up to coding this in a fork i'd be happy to review
it. As a first step, having "pickle" and a default that
reflects the current situation would be fine and should help
your use case. I'd probably keep the layering as it probably
keeps code changes to a minimum.
cheers,
holger
> -- William
> import py, time, logging
>
> remote_code = """
> import logging
>
> class FakeLogRecord(object):
> "Fakes a LogRecord's repr for passing over an execnet channel"
> def __init__(self, log):
> self.log = log
>
> def __repr__(self):
> l = self.log
> # Serialization of objects going over a channel happens by calling
> # repr() on the object. Deserialization happens with an
> # eval(mesg, {}). With a minimal namespace, I need to handle the
> # importing of the module myself as an expression encoded in the repr
> # of the object. Currently, I ignore specified traceback objects
> # attached to a LogRecord via exc_info=True.
> return "__import__('logging').LogRecord(%r, %r, %r, %r, %r, %r, ())"
> % (
> l.name, l.levelno, l.pathname, l.lineno, l.msg, l.args)
>
>
> class ExecNetHandler(logging.Handler):
> "Send logging messages over an py.execnet connection"
>
> def __init__(self, channel):
> logging.Handler.__init__(self)
> self.channel = channel
>
> def emit(self, record):
> obj = FakeLogRecord(record)
> self.channel.send(obj)
>
> # A testrun
> log = logging.getLogger("remote")
> log.setLevel(logging.DEBUG)
> log.addHandler(ExecNetHandler(channel))
>
> log.debug("Starting up")
> channel.send("Look ma, I'm home")
> log.info("I'm running on the remote side")
> try:
> raise RuntimeError("just an exception")
> except:
> log.error("Somebody set up us the bomb", exc_info=True)
> log.debug("Closing down")
> """
>
> logging.basicConfig(format="%(name)s: %(levelname)s: %(message)s")
> remote_log = logging.getLogger("remote")
> handler = logging.StreamHandler()
> handler.setLevel(logging.INFO)
> remote_log.addHandler(handler)
>
>
> gw = py.execnet.PopenGateway()
> channel = gw.remote_exec(remote_code)
>
> time.sleep(2)
> for mesg in channel:
> if isinstance(mesg, logging.LogRecord):
> remote_log.handle(mesg)
> continue
> print "From the remote side:", `mesg`
>
> _______________________________________________
> py-dev mailing list
> [email protected]
> http://codespeak.net/mailman/listinfo/py-dev
--
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu
_______________________________________________
py-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/py-dev
