Re: [Web-SIG] Web Site Process Bus

[Robert Brewer]

Phillip J. Eby wrote:
> At 02:17 PM 6/25/2007 -0700, Robert Brewer wrote:
> > Phillip J. Eby wrote:
> > > At 01:51 PM 6/25/2007 -0700, Robert Brewer wrote:
> > > For example, if an error occurs, isn't that an indication that the
> > > component is broken?  Masking the failure just makes it 
> > > less likely the component will get fixed in a timely manner.
> >
> > Yes, the component is broken. However, at runtime, breakage in a
> > CherryPy component shouldn't keep a Quixote component from, say,
> > correctly freeing its DB connections.
> 
> In theory that makes sense, but in practice if you're using 
> priorities because there is a dependency sequence involved, then you 
> now have a new problem, since a component you're relying on having 
> started or stopped first, is now violating its invariants.
> 
> I'm not opposed to logging or catching errors, but I am opposed (in 
> the absence of more specific evidence) to allowing callbacks to 
> propagate unhandled exceptions in the spec, or encouraging event 
> senders to make heroic efforts in the face of unhandled 
> exceptions.  Trying to recover from brokenness is generally not very 
> likely to result in *less* breaking, IMO.

I agree with that in the general case, and specifically for site
startup, which is prone to dependencies and priorities. The specific
case where I feel we need different behavior is when trying to shut down
a site, which rarely involves dependencies and priorities, but can often
lead to increasing damage if an early component errors and remaining
resource cleanup routines are not allowed to run. Maybe we should just
special-case the latter and let the rest fail fast.

> > > Meanwhile, if you get a start call, you must be starting, 
> > > right?  So why worry about the state?  It'd be simpler to
> > > just use "before/during/after" messages the way Twisted does.
> > > Your "block" example could be replaced by waiting for the "after" 
> > > message of the desired state, for example.
> >
> > That's a possible way to go. My intention was to support both
> > 1) examination of the state by external components (for
> > operations other than 'block'--progress meters spring to mind)
> > and 2) restrict some state transitions if necessary; for example,
> > make bus.start() do nothing (or block) unless the state is
"STOPPED".
> 
> Progress meters can be handled by callbacks, too.

With sufficent complexity, yes.

> As for the restrictions, who benefits?  ISTM that components need
> to manage their own lifecycles anyway and should be idempotent
> with respect to repeated transitions.

Good point. Requiring idempotent operations would allow fun sequences
like:

>>> bus.exit()
>>> bus.exit()
>>> help!
SyntaxError: invalid syntax
>>> os.unlink(errant_lock_file)
>>> bus.exit()
...process finally exits...

That would also allow someone to call bus.exit() in the middle of
another thread executing bus.start()...if each component manages its own
state, that would minimize shutdown errors like closing DB connections
that were never opened.

Hmm...


Robert Brewer
System Architect
Amor Ministries
[EMAIL PROTECTED]
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Chris McDonough]

On Jun 26, 2007, at 5:07 PM, Robert Brewer wrote:
>> I think I'm mostly confused by the name "process bus" because it
>> seems like the primary use case for something like this is where all
>> of the applications share the same process space
>
> I don't see why it should be limited by that. The primary use case is
> anywhere site components and application components are interacting,
> that could benefit from a shared understanding (and control) of the
> state of the site. To me, that requires a common set of messages, but
> the transport mechanism for those messages should be flexible so that
> it's useful in both multithread and multiprocess architectures.

Thank you.  I see.  This is a little too abstract for me to get my  
brain around, but I'll continue listening and maybe I'll get  
religion. ;-)

- C



___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Robert Brewer]

Chris McDonough wrote:
> On Jun 26, 2007, at 2:39 PM, Robert Brewer wrote:
> 
> > Chris McDonough wrote:
> >> There are also non-webbish processes like postgres, mysql, 
> >> etc. that
> >> need to be treated as "part of the application".
> >>
> >> I handle this currently by running all of the processes 
> >> related to a
> >> specific project under a process controller (which happens to be
> >> implemented in Python, but that's besides the point, see http://
> >> www.plope.com/software/supervisor2/).  The process controller is
> >> responsible for execing the child processes upon its own startup.
> >> It is also responsible for restarting children if they die,
> >> capturing their output (if any), and allowing sufficiently
> >> privileged users to start and stop each one independently.
> >> The only promise a subprocess must make to be managed is that
> >> it must be possible to start the process "in the foreground"
> >> (not under its down daemon manager).
> >>
> >> If a "process bus" is implemented I suspect it should be 
> >> implemented at this kind of level.
> >
> > Ah, but there's the rub: we all have different ideas about how to
> > *implement* IPC and control.
> 
> I'm confused by this in your earlier message, describing example  
> scenarios:
> 
> """
> If I'm primarily a Zope user instead, I might start my website with
> zdaemon. This would work exactly like the above, but the Bus object
> would be instantiated and started by the zdaemon package. If I'm using
> Graham's new mod_wsgi with Apache, I might expect it to create and
> control the Bus.
> """
> 
> I think I'm mostly confused by the name "process bus" because it  
> seems like the primary use case for something like this is where all  
> of the applications share the same process space

I don't see why it should be limited by that. The primary use case is
anywhere site components and application components are interacting,
that could benefit from a shared understanding (and control) of the
state of the site. To me, that requires a common set of messages, but
the transport mechanism for those messages should be flexible so that
it's useful in both multithread and multiprocess architectures.

> ...and are all written in Python.  Am I right?

That's the initial target market, yes. But I think we can design the
messaging spec to be useful with non-Python application components.

> If so, maybe a different name is in order?   
> "Application Bus"?  Or even "WSGI Bus", if its presumed that all of  
> the applications will be WSGI applications?

Sure, "application bus" is fine, although it's just the other side of
the same coin: "applications" on one side, "site" on the other.

I wouldn't want this to be "WSGI Bus", simply because there's no benefit
for that relationship; the two specs should be useful independently of
each other. In particular, we should be able to design a site-messaging
bus which works for WSGI 1.0, 1.1, 2.0, and whatever might obsolete the
current WSGI in the future.

> I'm confused because zdaemon is a generic process controller, it  
> knows nothing in particular about the application running under it  
> except that it's a UNIX process. It could start postgres instead of  
> Zope if you configured it to.

Sorry, I was really thinking of zopectl when I wrote that. I'm not sure
how zdaemon itself would fit into this whole scheme--it's the process
which zdaemon invokes that should be directly involved. If that process
is a good provider of bus-aware services, you might not need zdaemon
anymore.

> If zdaemon creates a Bus object,
> nothing will be able to send messages to the bus except zdaemon  
> itself, and there can't be any useful listeners because it doesn't  
> share the same process space as its child.

If you use the example Bus implementation I posted, then yes. That's why
I'm pitching WSPBus as a spec, not an implementation. Multiprocess
controllers could implement the bus using any of various forms of IPC;
they just need to arrange for each application component to get a Bus
object that, behind the scenes, is specific to the chosen method of IPC.

So, yes, interprocess communication is more complicated than
intraprocess. But that's true whether you standardize on a bus spec or
not.


Robert Brewer
System Architect
Amor Ministries
[EMAIL PROTECTED]
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Alan Kennedy]

[Graham Dumpleton]
 > First comment is about WSGI applications somehow themselves using
 > SIGTERM etc as triggers for things they want to do. For Apache at
 > least, allowing any part of a hosted Python application to register
 > its own signal handlers is a big no no. This is because Apache itself
 > uses a whole range of signals to manage such tasks as shutting down
 > sub processes or signaling worker and/or listener threads within a
 > process that its time to wakeup or shutdown. If a WSGI application
 > starts registering signal handlers it can as a result stop Apache from
 > even being able to process requests. In mod_wsgi I have had to
 > specifically take steps to prevent applications breaking things in
 > this way by replacing signal.signal() on creation of an interpreter.
 > Instead I log a warning that the signal registration has been ignored
 > and otherwise do nothing. This was simply the safest thing to do.
 >
 > Thus I believe a clear statement should be made that UNIX signals are
 > off limits to WSGI applications or components.

 From a jython POV, I agree with this statement; signals don't even 
exist on java/jython (although some JVMs have non-standard extensions 
for signals).

Thus, any "standard" involving signals would not be implementable on 
jython, and I guess ironpython too.

[Graham Dumpleton]
 > Anyway, just wanted to make it absolutely clear that I don't believe a
 > hosted WSGI application and associated framework has any business
 > taking direct interest in low level UNIX signals.

Agreed.

Regards,

Alan.
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Chris McDonough]

On Jun 26, 2007, at 2:39 PM, Robert Brewer wrote:

> Chris McDonough wrote:
>> There are also non-webbish processes like postgres, mysql, etc. that
>> need to be treated as "part of the application".
>>
>> I handle this currently by running all of the processes related to a
>> specific project under a process controller (which happens to be
>> implemented in Python, but that's besides the point, see http://
>> www.plope.com/software/supervisor2/).  The process controller is
>> responsible for execing the child processes upon its own startup.
>> It is also responsible for restarting children if they die,
>> capturing their output (if any), and allowing sufficiently
>> privileged users to start and stop each one independently.
>> The only promise a subprocess must make to be managed is that
>> it must be possible to start the process "in the foreground"
>> (not under its down daemon manager).
>>
>> If a "process bus" is implemented I suspect it should be implemented
>> at this kind of level.
>
> Ah, but there's the rub: we all have different ideas about how to
> *implement* IPC and control.

I'm confused by this in your earlier message, describing example  
scenarios:

"""
If I'm primarily a Zope user instead, I might start my website with
zdaemon. This would work exactly like the above, but the Bus object
would be instantiated and started by the zdaemon package. If I'm using
Graham's new mod_wsgi with Apache, I might expect it to create and
control the Bus.
"""

I'm confused because zdaemon is a generic process controller, it  
knows nothing in particular about the application running under it  
except that it's a UNIX process.  It could start postgres instead of  
Zope if you configured it to.  If zdaemon creates a Bus object,  
nothing will be able to send messages to the bus except zdaemon  
itself, and there can't be any useful listeners because it doesn't  
share the same process space as its child.

I think I'm mostly confused by the name "process bus" because it  
seems like the primary use case for something like this is where all  
of the applications share the same process space and are all written  
in Python.  Am I right?  If so, maybe a different name is in order?   
"Application Bus"?  Or even "WSGI Bus", if its presumed that all of  
the applications will be WSGI applications?

- C

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Robert Brewer]

Chris McDonough wrote:
> There are also non-webbish processes like postgres, mysql, etc. that  
> need to be treated as "part of the application".
> 
> I handle this currently by running all of the processes related to a  
> specific project under a process controller (which happens to be  
> implemented in Python, but that's besides the point, see http:// 
> www.plope.com/software/supervisor2/).  The process controller is  
> responsible for execing the child processes upon its own startup.
> It is also responsible for restarting children if they die,
> capturing their output (if any), and allowing sufficiently
> privileged users to start and stop each one independently.
> The only promise a subprocess must make to be managed is that
> it must be possible to start the process "in the foreground"
> (not under its down daemon manager).
> 
> If a "process bus" is implemented I suspect it should be implemented  
> at this kind of level.

Ah, but there's the rub: we all have different ideas about how to
*implement* IPC and control. Which is why the WSPBus I outlined says
nothing about the mechanisms of message transport, RPC/IPC, or process
or thread boundaries. Instead, it defines the messages themselves, a set
of states for a given site, and a singleton message broker and state
machine. That's it.

If we can get that one piece into place, I think it can be a focal point
for interop between 1) application components, 2) HTTP servers, and 3)
"process controllers" (great term, I think I'll use it from now on). We
can achieve that without specifying how any process controller is
implemented, I think. It's difficult to discuss and reason about,
because CherryPy, Apache, Django, Zope, etcetera etcetera all provide
all three. A common bus should make it easier to decouple, say,
CherryPy's app+server from its process controller, allowing more people
to try out supervisor2 more easily.

> "Actions" could be registered for a specific subprocess types
> to send some input to a pipe file descriptor, send a signal to
> the process, etc.

Yes, and a supervisor2-wspb package could provide a Bus which does that,
then hand a reference to it to each child's components. The "reference"
in this case would be a child-side proxy object, which knows how to send
signals back to the parent process. But the children don't need to know
those transport details--all they have to do is call bus.subscribe,
bus.publish('log'), etc.

> It would also be possible to create some sort of dependency
> map between processes in a configuration, that relate the
> actions of one process to another (restart process A if
> process B is restarted, send a signal S to process C if
> signal T is sent to process D, etc).

Dependencies are a layer that can be built on top of the basic bus.
Since it's the process controller that's calling bus.start, stop and
restart, there's nothing about the WSPBus that stops supervisor2 from
handling dependency graphs on its own. If process A has to *know* that
process B has been restarted, that's a problem (which could be addressed
via custom bus channels), but if only the process controller has to
know, then there's no need to add that to the bus spec, IMO.


Robert Brewer
System Architect
Amor Ministries
[EMAIL PROTECTED]
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

[Web-SIG] Paste releases

[Ian Bicking]

I've just made releases for Paste (core/Script/Deploy).  Quick summary:

Paste core:

* The paste.httpserver has neat new threadpool stuff to improve the
reliability of your threaded server.  Kill threads, see wedged threads,
restart the process, and more.  This has been in progress for some time
(and has held up releases), but after having it in production for a
month or so (from trunk -- it's okay for *me* to deploy from trunk ;)
I'm feeling fairly comfortable about it.

* Save HTTP proxy information, so you can in theory write HTTP proxies
with paste.httpserver

* More methods in TestApp, for your RESTful testing pleasure, like
app.put() and app.delete().

* paste.util.template has a few more constructs.

* Some request (object/function) bugs fixed.

In Paste Deploy:

* If you've had problems using filter-with and setup-app together (or
with appconfig() or any other problem with filter-with), it should be
better.

Paste Script only has one boring little change for logging.

easy_install -U to upgrade.  See the website for slightly more complete
news:
   http://pythonpaste.org/news.html
   http://pythonpaste.org/script/news.html
   http://pythonpaste.org/deploy/news.html


-- 
Ian Bicking | [EMAIL PROTECTED] | http://blog.ianbicking.org
 | Write code, do good | http://topp.openplans.org/careers

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Chris McDonough]

On Jun 26, 2007, at 1:04 AM, Graham Dumpleton wrote:
> In Apache changing the certificates would need a complete restart of
> everything. Because the  child processes aren't privileged they would
> not be able to trigger the main server to do so. This actually gets to
> one of my reservations about some of the stuff being discussed. That
> is, that the WSGI applications should even have any ability to control
> the underlying web server. In a shared web hosting environment using
> Apache, allowing such control is not practical as you don't want
> arbitrary user doing things to the server. If you are running Apache
> as a dedicated server for a single application that is a different
> matter however. Thus some aspects of what can be done by via the bus
> would  have to be controllable dependent on the environment in which
> one is running.
>
> At least with Apache, even initiating this sort of stuff from inside
> of a WSGI application may not make a great deal of sense even then. It
> would be far easier and preferable in Apache to use a suexec CGI
> script to accept the upload of the SSL certificate and then trigger a
> restart of Apache. So in the end the bus concept may be great for pure
> Python system, but not so sure about a complicated mixed code system
> like Apache, especially where there may be better ways of handling it
> through other features of Apache.

There are also non-webbish processes like postgres, mysql, etc. that  
need to be treated as "part of the application".

I handle this currently by running all of the processes related to a  
specific project under a process controller (which happens to be  
implemented in Python, but that's besides the point, see http:// 
www.plope.com/software/supervisor2/).  The process controller is  
responsible for execing the child processes upon its own startup.  It  
is also responsible for restarting children if they die, capturing  
their output (if any), and allowing sufficiently privileged users to  
start and stop each one independently.  The only promise a subprocess  
must make to be managed is that it must be possible to start the  
process "in the foreground" (not under its down daemon manager).

If a "process bus" is implemented I suspect it should be implemented  
at this kind of level.  "Actions" could be registered for a specific  
subprocess types to send some input to a pipe file descriptor, send a  
signal to the process, etc.  It would also be possible to create some  
sort of dependency map between processes in a configuration, that  
relate the actions of one process to another (restart process A if  
process B is restarted, send a signal S to process C if signal T is  
sent to process D, etc).

- C

___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com

Re: [Web-SIG] Web Site Process Bus

[Robert Brewer]

Graham, thanks for this and the previous post. I've had these issues in
mind while designing the current bus, but you raised them far more
eloquently than I could have.

Graham Dumpleton wrote:
> The only way we may know is to start stepping through specific uses
> cases one at a time, not even worrying about the mechanisms of how the
> bus may actually work, and discuss what would the intention of each
> be.
>
> > My particular use case for keeping SystemExit around is 
> > I have an app that allows the user to upload a new SSL
> > certificate.  Without a restart, or perhaps, given a WSPBus,
> > just a drop to IDLE state, the new SSL certificate would
> > not be applied to new incoming connections.  The thread that
> > puts the new SSL Certificate in place needs to be able to
> > tell the entire server to reload.
> 
> In Apache changing the certificates would need a complete restart of
> everything. Because the child processes aren't privileged they would
> not be able to trigger the main server to do so. This actually gets to
> one of my reservations about some of the stuff being discussed. That
> is, that the WSGI applications should even have any ability to control
> the underlying web server. In a shared web hosting environment using
> Apache, allowing such control is not practical as you don't want
> arbitrary user doing things to the server. If you are running Apache
> as a dedicated server for a single application that is a different
> matter however. Thus some aspects of what can be done by via the bus
> would  have to be controllable dependent on the environment in which
> one is running.

Right. My expectation was that Apache interfaces (like mod_wsgi and
modpython_gateway) would supply custom Bus objects which deny certain
behaviors (like calling bus state-transition methods from a WSGI
application). I think there's room for Apache site admins to choose
whether applications are allowed to do dangerous things, much like how
AllowOverride works for .htaccess.

> At least with Apache, even initiating this sort of stuff from inside
> of a WSGI application may not make a great deal of sense even then. It
> would be far easier and preferable in Apache to use a suexec CGI
> script to accept the upload of the SSL certificate and then trigger a
> restart of Apache.

That's not contrary to the bus concept. If there's a preferred way of
doing things, then a function can be written to do that, supplied by the
site interface, and be subscribed to the appropriate channel.

> So in the end the bus concept may be great for pure
> Python system, but not so sure about a complicated mixed code system
> like Apache, especially where there may be better ways of handling it
> through other features of Apache.

Cannot those "other features" be comoponentized? The only thing the Bus
tries to do is make a common interface for such behaviors--if Apache has
native methods to achieve the desired behavior, then great! Wrapping
them in bus listeners (and subscribing a safe set of them by default)
allows deployers who aren't familiar with Apache to get their site up
and running faster. However, mod_wsgi can still use Apache directives
for attaching/detaching the listeners if it likes, providing a more
Apache-like look-and-feel.


Robert Brewer
System Architect
Amor Ministries
[EMAIL PROTECTED]
___
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: 
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com