On Wed, Jul 22, 2009 at 10:11 PM, ghtdak <gl...@tarbox.org> wrote:
---------- Forwarded message ----------
From: Dorian Raymer <deldo...@gmail.com>
Date: Jul 22, 8:00 pm
Subject: notebook rewrite
To: sage-devel
Hi Glenn,
On Wed, Jul 22, 2009 at 4:41 PM, ghtdak <gl...@tarbox.org> wrote:
> Of course, once you put an event loop in the sub-process, much
of
> pexpect probably becomes unnecessary. Seems to me that Twisted
could
> do the spawning of the twisted sub-process and just shuffle
strings
> across. I don't really know pexpect but my guess is the things
it
> handles are also handled by twisted (although I don't actually
know).
> Its doubtful that there is sufficient complexity that pickle
couldn't
> handle the marshalling of data across the pipe.
The essence of what you have described is architecture of
Codenode.
With out
getting too much into a side by side comparison, this is how it
works
using
our terminology:
An Engine is a computation process (like what you call a Sage sub-
process).
An Engine contains two parts: First part, an object representing
the
Python
(or Sage) interpreter<http://github.com/codenode/codenode/blob/
8747088439efce423cefd680eab2...>,
with methods like evaluate, and tab complete. Second part, a
simple
RPC
server<http://github.com/codenode/codenode/blob/
8747088439efce423cefd680eab2...>that
adapts the interpreter object to a network transport; the current
working implementation is an XML-RPC server.
Great. This is where I was going to eventually head. In truth, what
we really want is a "Sage Service" which can lie under any number of
transports. XML-RPC is certainly consistent with that approach
The Interpreter object knows nothing about the RPC server, and the
RPC
server only has to map it's rpc methods to the interpreter object
methods.
Exactly
A Backend server creates and manages Engines.
The Backend server is a Twisted event loop.
It spawns, monitors, and terminates Engine Processes.
It relays all notebook AJAX requests to the appropriate engine via
an
XML-RPC client (which is non-blocking).
Again, exactly. Any IO should be non-blocking including the period
between a request and a response. This can be painful to implement
which is why Twisted introduces the deferred for a one time response
and more general callbacks when any number of responses (or
invocations in the case of peer-to-peer) may happen. None of this is
trivial and they don't call it Twisted for nothing, but once you
understand the foundation and rationale, it becomes pretty obvious.
Since the Backend is a Twisted event loop, it handles all process
spawning/monitoring, AJAX requests, and XML-RPC requests totally
asynchronously.
It has no dependencies on the interpreter libraries that it runs,
and
does
not need to understand the data going in and out of the engines.
This is where things are sometimes layered. For example, one might
chose to put authentication / encryption in this layer. Additionally,
pre and post processing can be added here leaving the "engines"
generic. But this is outside the scope of the discussion...
Have a look at all the pieces currently work
here<http://github.com/codenode/codenode/tree/
8747088439efce423cefd680eab2...>
.
I am currently improving this Backend concept and have a
completely
new
iteration of code<http://github.com/deldotdr/codenode/tree/
8b7658ebc51e7ab005e950f373a1...>(still
in development) that should be a bit more digestible (it's
cleaner,
among other improvements).
Ahhhh... Git :-)... Oh.. I mean HERETIC!!!!
> Once this change was made, you'd have a full infrastructure with
which
> to build much more flexible applications yet still have the
notebook
> interface. This would also facilitate building distributed
> computation engines, data collectors etc.
Yes! Exactly. Combined with the third element (called the Frontend
web
application server who's most important job is storing users
notebook
data
in a database), you have distributed notebook computation. We have
successfully done deployments where the Frontend runs on one
server,
and the
Backend on another server, usually in another City!
When the next piece of development is finished, a Frontend will be
capable
of supporting multiple Backends, again, no matter where they are
on
the
network.
Well, once you're serializing the data, it all comes down to physics.
Typically, LAN comms failure modes are less severe than WAN and often
involve a bunch more code to handle this wackiness... but this is also
painful stuff better left for you younger / more energetic folk... Too
many years spent on network failure issues myself... but its
fascinating and painful, two of my (former) favorites.
> Furthermore, as things evolved, truly dynamic AJAX could be
built
> because the underlying Sage process could be asynchronously
receiving
> data, talking with other Sage processes, periodically polling
other
> servers (e.g. yahoo finance)
This is interesting, and sounds like something different than just
a
"Computation Engine".
I interpret this as a formalization of some kind of information
base
into a
service accessible by the the notebook Engine processes.
Well, I could be, but that wasn't the intent of this section. All I
was saying is that blocking is bad. I usually also say that Threads
are dangerous / hard... when one introduces Threads to get around
unnecessary blocking, we approach insane.
In my case, I have a need for the sage process to access other
"stuff" (technical term) but am unable to do it in the "main" sage
notebook thread because its sucking on a dry pipe from the twisted
server most of its life... so I had to do some tricky threaded
nonsense to get around it.
The simple limit of this is the formalization of how a processes
OS
Environment Variables are set. For example, I could want to set up
a
environment supporting scientific analysis, so I would want to
configure
numpy and scipy to be in my Engine process python path.
This environment could also facilitate access to some kind of data
set, like
one produced by an Astrophysicist's galaxy simulation. The data
set
could be
a file, a set of files, etc. Or, in the large limit, and coming
back
to your
example, it could be a network service explicitly supported by the
Backend,
with accessibility permissions from each engine process explicitly
configurable.
Wow, you college kids sure do get some learnin' :-)
I'm sure all this is right but I was attacking a much simpler first
step. I currently have most of what I'm proposing built for my uses
and the technical configuration details are somewhat orthogonal to the
fundamental axioms... the most basic being "blocking is bad"
I'm not sure it makes sense, architecturally, for an Engine
process to
run
an event loop, because it's principal purpose is to execute
arbitrary
code,
and that must be generally assumed to be a blocking procedure.
Maybe
you can
elaborate more on this idea; it's very interesting.
Ahhhh.... so lets pull on this one a bit.
First, why not. In the case I believe you're making, you're talking
about compute bound. If you're compute bound, you're bound. If you
have a lot of compute bound yet need interactive response, then you're
into multiple threads either through multi-process integration or
multi-threading. The former being more generally termed "message
passing" and the latter being "shared memory" architecture.
But, when not computing and is blocking on a read, it is an event loop
but masked to accept a single event. The simplest event loop example
is probably the unix select() statement where a set of file
descriptors are being listened to along with a timeout... when the
select returns its because something was received (or a write
operation completed) or a timeout occurred. Most of the higher layers
in an event loop are simply mechanisms to handle what can often become
very nasty code.
Sucking on a single pipe is a select statement with only one file
descriptor. But its still an event loop.
On the other hand, if one were to write a bunch of code to do request-
response processing, with an arbitrary number of requesters talking to
a bunch of different object instances in a process, it can get pretty
ugly. The deferred manages much of this plumbing although one needs
to become comfortable with getting a "promise" for a result at some
point in the future (which is what other asynchronous systems
sometimes call their deferreds)
But, once you get over that part, its trivial to have hundreds or
thousands of conversations going on with all the complexity being
hidden inside the reactor and the state maintained between the various
target objects and information in the deferred... all the while
avoiding any need for thread synchronization and its dual, the lack of
required synchronization causing much hair loss in us older folk.
Now, in the case of the sage notebook, the idea is, basically, that
there is state within the sage process which comes from more than one
source during the session. If I were to make an outgoing request, I
wouldn't be able to simultaneously wait for the response and wait for
requests from the browser simultaneously. My choice then becomes
block on the response which hangs the notebook, or block on the
notebook hoping that I get the thread of control back and whatever
response I got is in the buffer.
In the more general case, if I were receiving an outside feed (say
financial tick data) I would need an entirely separate process which
would then be polled if I wanted to use the notebook / sage for
exploitation. This is a bunch of extra and unnecessary work.
Then we get into a couple of areas that are very much related. The
first is distributed computing. Ideally, my "primary" sage process
should be able to farm out work and manage the overall set of
computations while I'm watching / doing other stuff in the notebook.
I shouldn't need to separate out a "coordinator" and shove state
around which naturally lives there.
Additionally, with AJAX, the browser code itself may be
asynchronous... meaning the sage notebook might send stuff up based on
timed events, or external communications from data sources or
indicating that a set of farmed out processes have completed.
In any of these cases, the critical element for implementation is the
event loop which serves as the management system to facilitate the
required multiplexing.
Lastly, none of this costs anything in terms of performance or code.
Twisted is already part of sage and it has a huge amount of
functionality supporting just about anything a normal human would
need. In fact, I used twisted in the sage notebook to do the things I
describe above although I needed to put twisted in a separate thread
and manage the threading issues with synchronization etc. It wasn't
all that bad but was unnecessary and is likely to bite me one of these
days with an intermittent / non-reproduce-able bug as threading tends
to do.
More generally, Sage needs a real architecture for distributed
computing which would facilitate combining multiple processes with
varying capabilities fairly easily. This is relatively simple to
accomplish given the tools available but some of the mechanisms
currently in place are somewhat dense... in particular the pexpect
code and the strings of python inside the python code... and the
special case handling etc... I have looked a couplea times and believe
that trying to decode its functionality is pretty hard so sorta needed
william / mabshoff to pull it off. Unfortunately, as I said earlier,
we never did get around to it.
But, since you're poking the beast anyways, we might as well slay the
dragon while we're there.
-glenn
-Dorian
> It seems this would be a fairly easy task and have substantial
> payoff. I would have done it myself, but got bogged down when
looking
> through the pexpect code and the strings shoved over the pipe.
I
> figured that a few hours with the author would save a lot of
time but
> the runaway stack interfered.
> -glenn
> > William
> > > -glenn
--
Glenn H. Tarbox, PhD || 206-274-6919
http://www.tarbox.org
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to
sage-devel-unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
