Roger, On Thu, Sep 23, 2010 at 6:46 PM, Roger Hoover <[email protected]>wrote:
> Hi Bruce, > > On Thu, Sep 23, 2010 at 1:41 PM, Bruce Frederiksen <[email protected]>wrote: > >> I'm new to supervisor and would like a capability that takes the >> fcgi-program feature one more step. >> >> What I want is for supervisord to create a socket (like it does for >> fcgi-program), but then also do the accept on it. Each time a connection >> comes in, spawn the child process and hand-off the connection to the new >> child. From that point forward, supervisord is not involved in the >> communication between the connecting client and the spawned server. >> > > Do you mean always spawn a new process per connection? If so, that's how > CGI works so you should just use that instead of FCGI. If you mean > dynamically spawning a limited number of FCGI processes on demand, then I > think an nginx_plugin is the way to go. See A below. > I'm not running http servers here. So they are not running behind nginx or apache. Actually, these are one level removed from that. What we have is apache -> front-end-server -> back-end-server. I'm talking about the connections between the front-end-servers and the back-end-servers (which run on different boxes). This connection is over an intranet and does not use the http protocol. These connections are long-lived, but I want to give the front-end-servers the ability to control the number of back-end-server processes depending on work load. So what I want is to simply connect to the back-end box to spawn a back-end-server process if the work load goes up. Then closing the connection will terminate the process, if the work load later goes down. All I need is a process manager that baby-sits these back-end-processes. That's where supervisor comes in... > >> The reason I want this is to give the client control over how many server >> processes are spawned. Supervisord would create the socket when it starts, >> but not spawn any of these type of processes until connections come in. >> This is a case where I am writing both the client and server, and the server >> is only visible on the internal network. >> >> A. Does anybody know if something like this already exists? >> > > Can what you need be accomplished with this nginx plugin? > http://labs.frickle.com/nginx_ngx_supervisord/ > > I haven't actually tried it yet but it looks like you can configure it to > spawn fcgi-programs on demand > > upstream backend { > server 127.0.0.1:8000; > server 127.0.0.1:8001; > supervisord 127.0.0.1:9001 admin:super; > fair; > } > > server { > location / { > proxy_pass http://backend; > } > } > > >> B. For those "in the know" on the internal architecture of supervisor, >> what pointers do you have on how to implement this? >> > > I think this would be pretty hard to do it in the supervisord process. You > have to jump two hurdles. > a) You'd have to figure out a reliable way to know when an FCGI socket > needs to be cleaned up. If you search recent posts to this lists, you'll > see that the currently implementation keeps a count of how many FCGI child > processes there are and closes the socket when the count hits zero and > creates it when the count goes above zero. I won't go into the details here > as to why but this problem would have to be overcome if you want to create > the socket before any FCGI children have been spawned. > This problem is due to the different usage of how fcgi-program works. Because supervisord is not doing the accept, when there are no child processes there is nothing to do the accept and you have a problem. Hence the need to close the socket in supervisord so that clients get an immediate "server refusing connection" error. But I want supervisord to always have the socket open and always be doing an accept on it. Thus clients will never be turned away. Another difference is that in the fcgi-program case, supervisord hands off the listening socket to each child, which each accept their own connections. But in my case, supervisord only hands off the new connection sockets. The child processes do not get the listening socket, they only get the one new connection socket. The child process never deals with the listening socket, and supervisord never deals with the new connection sockets (except to hand them off to the new child process). Thus, there is a cleaner separation of responsibilities. b) You can't do any blocking network calls so you'd have to plugin into > the main select() loop. I'm not sure how hard or easy that will be. Then I > think it would have to work something like this: you'd have to create the > socket with the O_NONBLOCK flag set until the first connection comes and set > it back to blocking before handing it off to the first child process. Keep > calling accept() and checking for EWOULDBLOCK or EAGAIN until it's ready. > Then you spawn the appropriate FCGI process to handle it. > Overall, it does not look that complicated. It looks like supervisord has the architecture already in place to support this. Another question, should I be working on trunk, or do the project owners want to do an svn branch for this? Thank you for your response! -Bruce
_______________________________________________ Supervisor-users mailing list [email protected] http://lists.supervisord.org/mailman/listinfo/supervisor-users
