Re: Should fastcgi be a proxy backend?
On 3/7/06, Brian Candler <[EMAIL PROTECTED]> wrote: > I'm not sure what you mean there, in particular what you mean by 'assumes > that you can make multiple connections to back end fastcgi processes' > > What I'm familiar with is apache 1.x with mod_fcgi. In that case, the > typical fastcgi program does indeed handle a single request at once, but you > have a pool of them, analagous to httpd with its pool of worker processes. > > Multiple front-end processes can open sockets to the pool, and each > connection is assigned to a different worker. In this regard, a fastcgi > process is just like a httpd process. I don't see why mod_proxy_foo can't > open multiple fastcgi connections to the pool, in the same way as it could > open multiple http connections to an apache 1.x server. > > (You could think of the fastcgi protocol as just a bastardised form of HTTP. > I wonder why they didn't just use HTTP as the protocol, and add some > X-CGI-Environment: headers or somesuch) Oh, believe me, I'm fully aware of how fastcgi typically works ;-) The problem is meshing that concept with how mod_proxy works. When you give ProxyPass a URL it really wants a host + port combo, you can't (now anyway) have N different backend processes (listening on different ports) all associated with the same URL. To do that you need to wrap them up in a balancer group and use mod_proxy_balancer. Balancer the has the problem that the worker processes don't coordinate things with each other, so even if you're droping the connection to the back end process as soon as you're done with it you can still get into situations where there are free back end processes, but you're sitting there waiting because you tried to connect to one that's already in use. > > Second, mod_proxy_balancer doesn't (seem to) have any real mechanism > > for adding back end processes on the fly, which is something that > > would be really nice to be able to do. I'd eventually love to be able > > to tell mod_proxy_fcgi that it should start up N back end processes at > > startup, and create up to M more if needed at any given time. > > Processes should be able to be killed off if they become nonresponsive > > (or probably after processing a certain number of requests) > > ... sounds very similar to httpd worker process management (for non-threaded > workers) > > > , and they > > should NOT be bound up to a single httpd worker process. > > In that case, is the underlying problem that mod_proxy_foo shouldn't really > hold open a *persistent* connection to the fastcgi worker pool, otherwise it > will tie up a fastcgi worker without good reason, preventing it from doing > work for someone else? Just keeping it from having a persistant connection helps, but it's not sufficient to solve the entire problem. > > So is there some reason I'm missing that justifies staying within the > > proxy framework > > Maybe. You might want to consider the case where the fastcgi server is a > *remote* pool of workers, where the fastcgi messages are sent over a TCP/IP > socket, rather than a local Unix domain socket. In that case, some remote > process is responsible for managing the pool, and this is arguably very > similar to the proxy case. That's actually the case we support now. There is no support for unix domain sockets yet, it's all TCP. > OTOH, the typical approach when using such a remote pool is to have a > different port number for each fastcgi application, since I'm not sure that > the fastcgi protocol itself has some way of passing down a URL or partial > URL which could identify the particular worker of interest. If it did, a > single process listening on a single socket could manage a number of > different applications, each with a different pool of workers. In any case, > though, it probably needs a configured list of applications, as it will need > some parameters for each one (e.g. minimum and maximum size of pool, as you > state) That's what you'd be doing with the current state of things, you use mod_proxy_balancer to wrap up the different back end processes. Alternatively, you can just use the new -N flag I gave to fcgistarter, which lets you start N workers that all listen on the same port, then your problem has sort of gone away. Unfortunately, at this point you don't really have a good solution for managing the processes, since you can't easily start new ones (fcgistarter would have to persist for that) and you can't easily (from mod_proxy_fcgi's point of view) tell the difference between the various back end processes because they all listen on the same port, so when one of them times out you can't say "kill that off and start a new one" because you don't know which one to kill off. Anyway, as you can see there are a number of issues at this point. -garrett
Re: Should fastcgi be a proxy backend?
On Sun, Mar 05, 2006 at 03:06:09PM -0800, Garrett Rooney wrote: > First of all, mod_proxy_balancer really assumes that you can make > multiple connections to back end fastcgi processes at once. This may > be true for some things that speak fastcgi (python programs that use > flup to do it sure look like they'd work for that sort of thing, but I > haven't really tried it yet), but in general the vast majority of > fastcgi programs are single threaded, non-asynchronous, and are > designed to process exactly one connection at a time. This is sort of > a problem because mod_proxy_balancer doesn't actually have any > mechanism for coordinating between the various httpd processes about > who is using what backend process. I'm not sure what you mean there, in particular what you mean by 'assumes that you can make multiple connections to back end fastcgi processes' What I'm familiar with is apache 1.x with mod_fcgi. In that case, the typical fastcgi program does indeed handle a single request at once, but you have a pool of them, analagous to httpd with its pool of worker processes. Multiple front-end processes can open sockets to the pool, and each connection is assigned to a different worker. In this regard, a fastcgi process is just like a httpd process. I don't see why mod_proxy_foo can't open multiple fastcgi connections to the pool, in the same way as it could open multiple http connections to an apache 1.x server. (You could think of the fastcgi protocol as just a bastardised form of HTTP. I wonder why they didn't just use HTTP as the protocol, and add some X-CGI-Environment: headers or somesuch) > Second, mod_proxy_balancer doesn't (seem to) have any real mechanism > for adding back end processes on the fly, which is something that > would be really nice to be able to do. I'd eventually love to be able > to tell mod_proxy_fcgi that it should start up N back end processes at > startup, and create up to M more if needed at any given time. > Processes should be able to be killed off if they become nonresponsive > (or probably after processing a certain number of requests) ... sounds very similar to httpd worker process management (for non-threaded workers) > , and they > should NOT be bound up to a single httpd worker process. In that case, is the underlying problem that mod_proxy_foo shouldn't really hold open a *persistent* connection to the fastcgi worker pool, otherwise it will tie up a fastcgi worker without good reason, preventing it from doing work for someone else? > So is there some reason I'm missing that justifies staying within the > proxy framework Maybe. You might want to consider the case where the fastcgi server is a *remote* pool of workers, where the fastcgi messages are sent over a TCP/IP socket, rather than a local Unix domain socket. In that case, some remote process is responsible for managing the pool, and this is arguably very similar to the proxy case. OTOH, the typical approach when using such a remote pool is to have a different port number for each fastcgi application, since I'm not sure that the fastcgi protocol itself has some way of passing down a URL or partial URL which could identify the particular worker of interest. If it did, a single process listening on a single socket could manage a number of different applications, each with a different pool of workers. In any case, though, it probably needs a configured list of applications, as it will need some parameters for each one (e.g. minimum and maximum size of pool, as you state) Just a few thoughts... Brian.
Re: Should fastcgi be a proxy backend?
Garrett Rooney wrote: On 3/6/06, William A. Rowe, Jr. <[EMAIL PROTECTED]> wrote: Garrett Rooney wrote: On 3/6/06, William A. Rowe, Jr. <[EMAIL PROTECTED]> wrote: Jim Jagielski wrote: See, the issue for fastcgi isn't controlling persistence, persistent connections are fine as long as you're actually making use of the backend process, the problem is avoiding having more than one connection to a backend process that simply cannot handle multiple concurrent connections. This seems to be a problem unique (so far anyway) to fastcgi. So the issue is that mod_proxy_fastcgi needs to create a pool of single process workers, and ensure that each has only one concurrent request, right? That's an issue for the proxy_fastcgi module, to mutex them all. The problem is that with the way mod_proxy currently works there isn't any way to do that, at least as far as I can tell. It seems like it will require us to move away from having mod_proxy manage the back end connections, and if we do that then we're back to the "what exactly is the advantage to using mod_proxy again?" question. Hmmm... it really doesn't seem that it's mod_proxies' responsibility to decide to make only one request to a backend at a time. But I agree with you that it's valuable to a -subset- of proxy backends to have some generic request pool, 1-at-a-time service. Perhaps we also drop in mod_proxy_dequeue or something like that to provide such a service, and put a generic create-workers code at the parent level? The module would still be responsible to implement the create-worker callback from this generic service, but it would then be possible for mod_proxy_dequeue to manage such a pool. If you want this to be robust, it would be necessary for mod_proxy_dequeue probably to run in a child of the parent process to handle all this various delegation, replace lost children, and handle things like long-lost-child has exited/returned. A 'do me next' pipe back to the parent would be necessary to serialize the requests and hand them back a worker. Bill
Re: Should fastcgi be a proxy backend?
On 3/6/06, William A. Rowe, Jr. <[EMAIL PROTECTED]> wrote: > Garrett Rooney wrote: > > On 3/6/06, William A. Rowe, Jr. <[EMAIL PROTECTED]> wrote: > > > >>Jim Jagielski wrote: > > > > See, the issue for fastcgi isn't controlling persistence, persistent > > connections are fine as long as you're actually making use of the > > backend process, the problem is avoiding having more than one > > connection to a backend process that simply cannot handle multiple > > concurrent connections. > > > > This seems to be a problem unique (so far anyway) to fastcgi. > > So the issue is that mod_proxy_fastcgi needs to create a pool of single > process workers, and ensure that each has only one concurrent request, > right? That's an issue for the proxy_fastcgi module, to mutex them all. The problem is that with the way mod_proxy currently works there isn't any way to do that, at least as far as I can tell. It seems like it will require us to move away from having mod_proxy manage the back end connections, and if we do that then we're back to the "what exactly is the advantage to using mod_proxy again?" question. -garrett
Re: Should fastcgi be a proxy backend?
Garrett Rooney wrote: On 3/6/06, William A. Rowe, Jr. <[EMAIL PROTECTED]> wrote: Jim Jagielski wrote: See, the issue for fastcgi isn't controlling persistence, persistent connections are fine as long as you're actually making use of the backend process, the problem is avoiding having more than one connection to a backend process that simply cannot handle multiple concurrent connections. This seems to be a problem unique (so far anyway) to fastcgi. So the issue is that mod_proxy_fastcgi needs to create a pool of single process workers, and ensure that each has only one concurrent request, right? That's an issue for the proxy_fastcgi module, to mutex them all. In the ASP.NET module, we create threads on the fly. But this is why I suggest being able to describe worker pools for a backend. Then again the backend worker pool directives probably make just as much sense as module-specific directives . Bill
Re: Should fastcgi be a proxy backend?
On 3/6/06, Plüm, Rüdiger, VIS <[EMAIL PROTECTED]> wrote: > > von Garrett Rooney > > > > Exactly, the pool of available backends needs to be managed globally, > > which we don't currently have and it's not clear if that ability would > > be useful outside of fastcgi. > > But as connection pools are per worker and not per cluster > this problem should also appear in the unbalanced environment. Oh sure, right now what we've got only kinda sorta works, if you don't put it under load. I've only done very limited testing, and it certainly seems logical that we'd see this problem even without balancer in the equation. -garrett
AW: Should fastcgi be a proxy backend?
> von Garrett Rooney > > Exactly, the pool of available backends needs to be managed globally, > which we don't currently have and it's not clear if that ability would > be useful outside of fastcgi. But as connection pools are per worker and not per cluster this problem should also appear in the unbalanced environment. Regards Rüdiger
Re: Should fastcgi be a proxy backend?
On 3/6/06, Jim Jagielski <[EMAIL PROTECTED]> wrote: > Garrett Rooney wrote: > > > > On 3/6/06, Jim Jagielski <[EMAIL PROTECTED]> wrote: > > > I think the whole issue revolves around whether the balancer > > > should, or should not, pre-open connections and manage them > > > internally, or whether it should be one-shot. The real > > > power is being able to load balance, and implement > > > that in a central location. > > > > > > So it seems to me that some sort of Balancer member > > > option that determines whether or not the connection > > > is "persistent" or not would alleviate some of > > > the issues you raise. > > > > We actually have a way to do that, it's the close_on_recycle flag, and > > I had to turn it on in order to get anything approaching reliability > > for fastcgi. > > > > That's how we do it *internally* and not what I meant. I meant > more a parameter to be set in httpd.conf... Sure, we could add a parameter to control that behavior, but it still won't solve the real underlying problem, you'll still get collisions under load, they just won't be as common. -garrett
Re: Should fastcgi be a proxy backend?
On 3/6/06, Plüm, Rüdiger, VIS <[EMAIL PROTECTED]> wrote: > > > > -Ursprüngliche Nachricht- > > Von: [EMAIL PROTECTED] > > > > We actually have a way to do that, it's the close_on_recycle flag, and > > I had to turn it on in order to get anything approaching reliability > > for fastcgi. The problem with just using that is that without some > > coordination between worker processes you're still going to end up > > with collisions where more than one connection is made to a given > > fastcgi process, and the majority of those don't know how to handle > > I think the problem is that we only manage connection pools that are local > to the httpd processes. Exactly, the pool of available backends needs to be managed globally, which we don't currently have and it's not clear if that ability would be useful outside of fastcgi. -garrett
Re: Should fastcgi be a proxy backend?
Garrett Rooney wrote: > > On 3/6/06, Jim Jagielski <[EMAIL PROTECTED]> wrote: > > I think the whole issue revolves around whether the balancer > > should, or should not, pre-open connections and manage them > > internally, or whether it should be one-shot. The real > > power is being able to load balance, and implement > > that in a central location. > > > > So it seems to me that some sort of Balancer member > > option that determines whether or not the connection > > is "persistent" or not would alleviate some of > > the issues you raise. > > We actually have a way to do that, it's the close_on_recycle flag, and > I had to turn it on in order to get anything approaching reliability > for fastcgi. > That's how we do it *internally* and not what I meant. I meant more a parameter to be set in httpd.conf... -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "If you can dodge a wrench, you can dodge a ball."
AW: Should fastcgi be a proxy backend?
> -Ursprüngliche Nachricht- > Von: [EMAIL PROTECTED] > > We actually have a way to do that, it's the close_on_recycle flag, and > I had to turn it on in order to get anything approaching reliability > for fastcgi. The problem with just using that is that without some > coordination between worker processes you're still going to end up > with collisions where more than one connection is made to a given > fastcgi process, and the majority of those don't know how to handle I think the problem is that we only manage connection pools that are local to the httpd processes. Regards Rüdiger
Re: Should fastcgi be a proxy backend?
I seem to recall that there was something about that with the old mod_fastcgi module... Garrett Rooney wrote: > > On 3/6/06, Sascha Schumann <[EMAIL PROTECTED]> wrote: > > > > Also, we tend to run most of our fastcgi's using a domain socket. I'= > m > > > > sure others do that as well. > > > > > > > > > > Isn't that very unreliable? > > > > Why should Unix domain sockets be unreliable? > > Yeah, that's my question as well. Quite a few people seem to use them... > > -garrett > -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "If you can dodge a wrench, you can dodge a ball."
AW: Should fastcgi be a proxy backend?
> -Ursprüngliche Nachricht- > Von: [EMAIL PROTECTED] > > > > > > Isn't that very unreliable? > > > > Why should Unix domain sockets be unreliable? > > Yeah, that's my question as well. Quite a few people seem to > use them... Maybe he is working on an upatched Solaris 10 ;-). Regards Rüdiger
Re: Should fastcgi be a proxy backend?
On 3/6/06, Sascha Schumann <[EMAIL PROTECTED]> wrote: > > > Also, we tend to run most of our fastcgi's using a domain socket. I'm > > > sure others do that as well. > > > > > > > Isn't that very unreliable? > > Why should Unix domain sockets be unreliable? Yeah, that's my question as well. Quite a few people seem to use them... -garrett
Re: Should fastcgi be a proxy backend?
On 3/6/06, William A. Rowe, Jr. <[EMAIL PROTECTED]> wrote: > Jim Jagielski wrote: > > I think the whole issue revolves around whether the balancer > > should, or should not, pre-open connections and manage them > > internally, or whether it should be one-shot. The real > > power is being able to load balance, and implement > > that in a central location. > > > > So it seems to me that some sort of Balancer member > > option that determines whether or not the connection > > is "persistent" or not would alleviate some of > > the issues you raise. > > That would be the ideal model for any remoted ASP.NET container as well. > Some persistance flag to indicate that a backed should be persistant, > and pooled, and the pool constraints (for this client side, not the actual > backend's true constraints), would be ideal. See, the issue for fastcgi isn't controlling persistence, persistent connections are fine as long as you're actually making use of the backend process, the problem is avoiding having more than one connection to a backend process that simply cannot handle multiple concurrent connections. This seems to be a problem unique (so far anyway) to fastcgi. -garrett
Re: Should fastcgi be a proxy backend?
On 3/6/06, Brian Akins <[EMAIL PROTECTED]> wrote: > Garrett Rooney wrote: > [snip] > > > Also, we tend to run most of our fastcgi's using a domain socket. I'm > sure others do that as well. True, but that's actually fairly simple to implement. I've got a scheme for making that work under proxy already, just haven't actually implemented it yet. -garrett
Re: Should fastcgi be a proxy backend?
On 3/6/06, Jim Jagielski <[EMAIL PROTECTED]> wrote: > I think the whole issue revolves around whether the balancer > should, or should not, pre-open connections and manage them > internally, or whether it should be one-shot. The real > power is being able to load balance, and implement > that in a central location. > > So it seems to me that some sort of Balancer member > option that determines whether or not the connection > is "persistent" or not would alleviate some of > the issues you raise. We actually have a way to do that, it's the close_on_recycle flag, and I had to turn it on in order to get anything approaching reliability for fastcgi. The problem with just using that is that without some coordination between worker processes you're still going to end up with collisions where more than one connection is made to a given fastcgi process, and the majority of those don't know how to handle more than one connection at a time, so requests will simply hang. -garrett
Re: Should fastcgi be a proxy backend?
Jim Jagielski wrote: I think the whole issue revolves around whether the balancer should, or should not, pre-open connections and manage them internally, or whether it should be one-shot. The real power is being able to load balance, and implement that in a central location. So it seems to me that some sort of Balancer member option that determines whether or not the connection is "persistent" or not would alleviate some of the issues you raise. That would be the ideal model for any remoted ASP.NET container as well. Some persistance flag to indicate that a backed should be persistant, and pooled, and the pool constraints (for this client side, not the actual backend's true constraints), would be ideal. Bill
Re: Should fastcgi be a proxy backend?
It's getting there, yes :) Brian Akins wrote: > > Jim Jagielski wrote: > > > So it seems to me that some sort of Balancer member > > option that determines whether or not the connection > > is "persistent" or not would alleviate some of > > the issues you raise. > > > Also, if the persistent connections we actually persistent... Is this > fixed in trunk? > > > -- > Brian Akins > Lead Systems Engineer > CNN Internet Technologies > -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "If you can dodge a wrench, you can dodge a ball."
Re: Should fastcgi be a proxy backend?
> > Also, we tend to run most of our fastcgi's using a domain socket. I'm > > sure others do that as well. > > > > Isn't that very unreliable? Why should Unix domain sockets be unreliable? - Sascha
Re: Should fastcgi be a proxy backend?
Brian Akins wrote: > > Garrett Rooney wrote: > [snip] > > > Also, we tend to run most of our fastcgi's using a domain socket. I'm > sure others do that as well. > Isn't that very unreliable? -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "If you can dodge a wrench, you can dodge a ball."
Re: Should fastcgi be a proxy backend?
Jim Jagielski wrote: So it seems to me that some sort of Balancer member option that determines whether or not the connection is "persistent" or not would alleviate some of the issues you raise. Also, if the persistent connections we actually persistent... Is this fixed in trunk? -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Should fastcgi be a proxy backend?
Garrett Rooney wrote: [snip] Also, we tend to run most of our fastcgi's using a domain socket. I'm sure others do that as well. -- Brian Akins Lead Systems Engineer CNN Internet Technologies
Re: Should fastcgi be a proxy backend?
I think the whole issue revolves around whether the balancer should, or should not, pre-open connections and manage them internally, or whether it should be one-shot. The real power is being able to load balance, and implement that in a central location. So it seems to me that some sort of Balancer member option that determines whether or not the connection is "persistent" or not would alleviate some of the issues you raise.
Should fastcgi be a proxy backend?
So, predictably, now that we've gotten mod_proxy_fcgi to the point where it's actualy able to run real applications I'm starting to question some basic assumptions we made when we started out along this course. The general idea was that we want to be able to get content from some fastcgi processes. That seems pretty similar to what mod_proxy_http does with other http servers, and mod_proxy_ajp with java app servers, and heck, since we're probably going to have lots of back end fastcgi processes it sure is cool that we've got that mod_proxy_balancer stuff to handle that part of the equation. It sure seems like a good idea, doesn't it? And at first glance it is, I mean it basically works, I can set up a balancer group with a bunch of back end fastcgi processes that I started up with the new fcgistarter program, and it'll pretty much do what we want. But there are some issues looming on the horizon. First of all, mod_proxy_balancer really assumes that you can make multiple connections to back end fastcgi processes at once. This may be true for some things that speak fastcgi (python programs that use flup to do it sure look like they'd work for that sort of thing, but I haven't really tried it yet), but in general the vast majority of fastcgi programs are single threaded, non-asynchronous, and are designed to process exactly one connection at a time. This is sort of a problem because mod_proxy_balancer doesn't actually have any mechanism for coordinating between the various httpd processes about who is using what backend process. Second, mod_proxy_balancer doesn't (seem to) have any real mechanism for adding back end processes on the fly, which is something that would be really nice to be able to do. I'd eventually love to be able to tell mod_proxy_fcgi that it should start up N back end processes at startup, and create up to M more if needed at any given time. Processes should be able to be killed off if they become nonresponsive (or probably after processing a certain number of requests), and they should NOT be bound up to a single httpd worker process. This all means that some kind of mechanism for coordinating access to and creation of back end processes needs to be created, and as it moves on it starts to feel less and less like this sort of functionality is generically useful to other back end fastcgi processes. Maybe I'm wrong about that though. Oh, and in order to do any of the really cool stuff we'll also have to rework the way mod_proxy handles arguments that are given to ProxyPass statements, so that they can be passed down to something other than either mod_proxy or mod_proxy_balancer. And even after we do that, we'll still be stuck in this situation where you end up with like a bazillion options on the end of each fastcgi ProxyPass, when really we'd want them to be per-balancer or per-directory or something like that. It just feels kinda clunky. Finally, I have to say that I'm starting to wonder what we're actually getting out of using the proxy framework for this. I mean all it's doing is creating some sockets for us, all the other stuff I just talked about pretty much needs to be implemented itself, and it's questionable whether any of it would be useful for something other than the fastcgi code. So is there some reason I'm missing that justifies staying within the proxy framework, cause I'm really tempted to just create a handler module that reuses most of the mod_proxy_fcgi code, since it sure feels like it'd be easier to write this stuff if I didn't have to shoehorn it into mod_proxy. -garrett
