Re: ping for http in mod_proxy
On Fri, Feb 15, 2008 at 8:26 AM, Jim Jagielski <[EMAIL PROTECTED]> wrote: > Not quite... there's nothing in providers, per se, that limit > how they are *used* to be oneshot... Just ask Brad and aaa :) +1. I personally find the vtable interface easier to deal with - you have a 'set' of functions that all go together in one structure - whereas for a hook, everything must be individual and one-off. *shrug* -- justin
Re: ping for http in mod_proxy
On 2/15/08 11:03 AM, "Plüm, Rüdiger, VF-Group" <[EMAIL PROTECTED]> wrote: > My main point is that I want to avoid > using both hook and provider if not really needed, as it Agreed. Was just stating my preference. As long as it's "easy" to use, I have no strong feelings either way. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
On Feb 15, 2008, at 10:43 AM, Akins, Brian wrote: On 2/15/08 8:13 AM, "Plüm, Rüdiger, VF-Group" <[EMAIL PROTECTED] > wrote: Any specific reason why we need to add an hook here and why this cannot be done by the existing provider (interface). I am scared of adding another level of indirection here if it is not really needed and things can be already done with the existing infrastructure. I like hooks bcs providers are "one-shot." I use both, but find my self using hooks more and more. A good example is the discussion around having stacked providers in mod_cache. If it were a hook, you'd already have that... Providers are good when you will have one, and only one, "thing" that needs to munge/manipulate/compute like database stuff. With the proxy stuff, it looks like we want "n" things to be able to manipulate the data. Once you make the leap from 1 to 2, might as well make it a hook. Not quite... there's nothing in providers, per se, that limit how they are *used* to be oneshot... Just ask Brad and aaa :)
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Akins, Brian > Gesendet: Freitag, 15. Februar 2008 16:44 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > On 2/15/08 8:13 AM, "Plüm, Rüdiger, VF-Group" > <[EMAIL PROTECTED]> > wrote: > > > > Any specific reason why we need to add an hook here and why > this cannot be > > done by the existing provider (interface). I am scared of > adding another > > level of indirection here if it is not really needed and > things can be already > > done with the existing infrastructure. > > > I like hooks bcs providers are "one-shot." I use both, but > find my self > using hooks more and more. > > A good example is the discussion around having stacked providers in > mod_cache. If it were a hook, you'd already have that... > > Providers are good when you will have one, and only one, > "thing" that needs > to munge/manipulate/compute like database stuff. With the > proxy stuff, it > looks like we want "n" things to be able to manipulate the > data. Once you > make the leap from 1 to 2, might as well make it a hook. But in the Auth framework we also work with providers and may call more then one provider per request to do authn / authz. So I guess this will be also doable. My main point is that I want to avoid using both hook and provider if not really needed, as it - creates unneeded overhead which lowers performance - makes it harder to extend things with own modules as things are more complex - make debugging harder So as far as possible for solving the problem in a flexible manner I would like to stick to KISS. Regards Rüdiger
Re: ping for http in mod_proxy
On 2/15/08 8:13 AM, "Plüm, Rüdiger, VF-Group" <[EMAIL PROTECTED]> wrote: > Any specific reason why we need to add an hook here and why this cannot be > done by the existing provider (interface). I am scared of adding another > level of indirection here if it is not really needed and things can be already > done with the existing infrastructure. I like hooks bcs providers are "one-shot." I use both, but find my self using hooks more and more. A good example is the discussion around having stacked providers in mod_cache. If it were a hook, you'd already have that... Providers are good when you will have one, and only one, "thing" that needs to munge/manipulate/compute like database stuff. With the proxy stuff, it looks like we want "n" things to be able to manipulate the data. Once you make the leap from 1 to 2, might as well make it a hook. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
On Feb 15, 2008, at 8:13 AM, Plüm, Rüdiger, VF-Group wrote: -Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Freitag, 15. Februar 2008 14:05 An: dev@httpd.apache.org Betreff: Re: ping for http in mod_proxy Well, right now all it does is say "give me the best worker" and leaves it up to the selected provider on "what that means", so it queries a provider, not a hook. Of course, we could create an actual hook which then queries providers, to allow people to extend the selection even more... Any specific reason why we need to add an hook here and why this cannot be done by the existing provider (interface). I am scared of adding another level of indirection here if it is not really needed and things can be already done with the existing infrastructure. No, no reason. In fact, I think the provider interface works relatively well. I do mention 2 existing hooks which should be changed (RUN_FIRST to RUN_ALL) which would allow even more coolness... I was just thinking that allowing a hook would allow people to add things without even using providers, which some people may like. Plus, it seemed an easy accommodation for people who "wanted" it (I wasn't one).
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Jim Jagielski > Gesendet: Freitag, 15. Februar 2008 14:05 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > > On Feb 14, 2008, at 6:41 PM, Graham Leggett wrote: > > > Akins, Brian wrote: > > > >> Yeah that was my thought. I guess you pass around the array of > >> servers. > >> Just remove (or mark as N/A) from the list and/or reorder it. At > >> the end, > >> core proxy picks whatever is in index 0 (possibly walking > the list > >> in case > >> of connection error or something). > > > > Essentially that is what proxy does now (or did before the > balancer > > stuff arrived) - it loops through everything that came back > from the > > DNS round robin query, trying to connect in turn until it gets > > something other than bad gateway. > > > > Rather than just the simplistic round robin DNS query, it should > > query a hook (in which is embedded as the sensible default, a > > simplistic round robin DNS query). > > > > Well, right now all it does is say "give me the best worker" and > leaves it up to the selected provider on "what that means", so it > queries a provider, not a hook. Of course, we could create an actual > hook which then queries providers, to allow people to extend > the selection even more... Any specific reason why we need to add an hook here and why this cannot be done by the existing provider (interface). I am scared of adding another level of indirection here if it is not really needed and things can be already done with the existing infrastructure. Regards Rüdiger
Re: ping for http in mod_proxy
On Feb 14, 2008, at 6:41 PM, Graham Leggett wrote: Akins, Brian wrote: Yeah that was my thought. I guess you pass around the array of servers. Just remove (or mark as N/A) from the list and/or reorder it. At the end, core proxy picks whatever is in index 0 (possibly walking the list in case of connection error or something). Essentially that is what proxy does now (or did before the balancer stuff arrived) - it loops through everything that came back from the DNS round robin query, trying to connect in turn until it gets something other than bad gateway. Rather than just the simplistic round robin DNS query, it should query a hook (in which is embedded as the sensible default, a simplistic round robin DNS query). Well, right now all it does is say "give me the best worker" and leaves it up to the selected provider on "what that means", so it queries a provider, not a hook. Of course, we could create an actual hook which then queries providers, to allow people to extend the selection even more...
Re: ping for http in mod_proxy
Akins, Brian wrote: Yeah that was my thought. I guess you pass around the array of servers. Just remove (or mark as N/A) from the list and/or reorder it. At the end, core proxy picks whatever is in index 0 (possibly walking the list in case of connection error or something). Essentially that is what proxy does now (or did before the balancer stuff arrived) - it loops through everything that came back from the DNS round robin query, trying to connect in turn until it gets something other than bad gateway. Rather than just the simplistic round robin DNS query, it should query a hook (in which is embedded as the sensible default, a simplistic round robin DNS query). Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: ping for http in mod_proxy
On 2/14/08 9:54 AM, "Graham Leggett" <[EMAIL PROTECTED]> wrote: > In theory, you should be able to stack the providers, so that a balancer > module could return the list of servers to try in the right order, and > then another module could further reduce that list down to servers that > are actually up. Yeah that was my thought. I guess you pass around the array of servers. Just remove (or mark as N/A) from the list and/or reorder it. At the end, core proxy picks whatever is in index 0 (possibly walking the list in case of connection error or something). -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
Akins, Brian wrote: Someone could write a spread based module for origin status (or mysql, or memcache, or...) if the interface was well defined and "clean." The way balancer is so hooked into proxy makes it hard to write a replacement without hacking "core" proxy. In proxy it could be as simple a call as: apr_status_t ap_get_origin_server(request_rec *r, proxy_origin_t **o) And that figures out all the balancer and sticky stuff. That was the idea. In theory, you should be able to stack the providers, so that a balancer module could return the list of servers to try in the right order, and then another module could further reduce that list down to servers that are actually up. Smells like a filter actually. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: ping for http in mod_proxy
On Feb 14, 2008, at 6:33 AM, Akins, Brian wrote: On 2/13/08 12:50 PM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: That was the other option as well... some sort of hearbeat loop which updates worker status. Still, we get into the issue with how much of "how proxy connects to and communicates with the backend" to honor or work around. An external process (using serf maybe) would be easy. Just have some of the worker stats in shared memory. The healthchecker writes status to it, and httpd reads from it. Yep... the process would fork off at Apache start time and update it's own current health as well as update its own knowledge of the health of the external backends... Not convinced that multicasting is right for this due to the types of architectures web servers usually find themselves in :)
Re: ping for http in mod_proxy
On Feb 13, 2008, at 6:01 PM, Graham Leggett wrote: Jim Jagielski wrote: Ideally, it would be nice if we had better insight on the actual health of the backends than a simple "do they respond to OPTIONS * and how long does it take", but that's pretty much all we can do unless go full-on multicasting of info ala mod_backhand... At least the balancer is setup to allow for load balancing based on that, if the submodule/provider actually existed :) Is there anything stopping us going the multicasting route, say by adding a hook or hooks of some kind to proxy that keeps track of known server states? A dedicated module could then have it's job as handling the multicasting and letting the rest of the server know if a server is on or off the air, in realtime. The proxy can then just say "which of these servers should I connect to?", and the hook will return the suggested ones in the given order. Well, we almost do that now... :) As a dedicated module, people who don't need it can turn it off or not build it.
Re: ping for http in mod_proxy
On 2/14/08 6:44 AM, "Plüm, Rüdiger, VF-Group" <[EMAIL PROTECTED]> wrote: > 1. We currently have no mechanism in place that "simulates" these kind of >failures we experience ourselves with the backend for the client. Returning >a 500 or 503 does not cause the client to repeat the request. IMHO we would >need to cut off the network connection without sending anything to trigger >this behaviour in a well designed client. Hrm.. Seems like the HTTP client should "just handle" this case. > 2. Clients are only allowed to resend the request automatically, if the > request >was idempotent. Clients are not allowed to do so with non-idempotent > requests >like POST's without user intervention. So by probing the keepalive > connection >before sending the request we want to reduce these cases. >From real world experience, I can say that we have rarely every had an issue with POSTS. The active health-checking we do is based on how our load balancers do it. They (the load balancers) can occasionally send requests to a down server for a few seconds if it goes down in between healthchecks. Sound bad in theory, but in reality, it has never been a real issue. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Akins, Brian [mailto:[EMAIL PROTECTED] > Gesendet: Donnerstag, 14. Februar 2008 12:37 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > On 2/13/08 1:09 PM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > > > And again, we're basically doubling traffic and adding > > overheard (more overhead than AJP's cping/cpong) at which > > point I go back into wondering whether this sort of > > implementation makes sense at all... > > > So is the main issue we are trying to solve is that we have a > keepalive > connection to an origin but it does away (closes) before we > make or next > request and this causes an "error." (Sorry if I'm behind on > the thread.) Basicly yes. > Shouldn't the HTTP client handle all of this? I know that > libcurl handles > this situations very well - it just tries to reconnect. How does serf > handle it? There are two problems here: 1. We currently have no mechanism in place that "simulates" these kind of failures we experience ourselves with the backend for the client. Returning a 500 or 503 does not cause the client to repeat the request. IMHO we would need to cut off the network connection without sending anything to trigger this behaviour in a well designed client. 2. Clients are only allowed to resend the request automatically, if the request was idempotent. Clients are not allowed to do so with non-idempotent requests like POST's without user intervention. So by probing the keepalive connection before sending the request we want to reduce these cases. Regards Rüdiger
Re: ping for http in mod_proxy
On 2/13/08 6:01 PM, "Graham Leggett" <[EMAIL PROTECTED]> wrote: > Is there anything stopping us going the multicasting route, say by > adding a hook or hooks of some kind to proxy that keeps track of known > server states? Multicasting doesn't work well for us, for example, because servers are spread across several different vlan's that explicitly don't allow multicast between them. If it were "hookable" or used providers, that would be fine. I think sticky sessions should be hookable/provider as well. Someone could write a spread based module for origin status (or mysql, or memcache, or...) if the interface was well defined and "clean." The way balancer is so hooked into proxy makes it hard to write a replacement without hacking "core" proxy. In proxy it could be as simple a call as: apr_status_t ap_get_origin_server(request_rec *r, proxy_origin_t **o) And that figures out all the balancer and sticky stuff. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
On 2/13/08 2:10 PM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > The latter is relatively easy to do with the current > impl... Maybe I'll drop the ping idea and work on this ;) +1 ;) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
On 2/13/08 1:09 PM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > And again, we're basically doubling traffic and adding > overheard (more overhead than AJP's cping/cpong) at which > point I go back into wondering whether this sort of > implementation makes sense at all... So is the main issue we are trying to solve is that we have a keepalive connection to an origin but it does away (closes) before we make or next request and this causes an "error." (Sorry if I'm behind on the thread.) Shouldn't the HTTP client handle all of this? I know that libcurl handles this situations very well - it just tries to reconnect. How does serf handle it? -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
On 2/13/08 12:50 PM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > That was the other option as well... some sort of hearbeat > loop which updates worker status. Still, we get into the issue > with how much of "how proxy connects to and communicates with > the backend" to honor or work around. An external process (using serf maybe) would be easy. Just have some of the worker stats in shared memory. The healthchecker writes status to it, and httpd reads from it. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
Graham Leggett wrote: Jim Jagielski wrote: Ideally, it would be nice if we had better insight on the actual health of the backends than a simple "do they respond to OPTIONS * and how long does it take", but that's pretty much all we can do unless go full-on multicasting of info ala mod_backhand... At least the balancer is setup to allow for load balancing based on that, if the submodule/provider actually existed :) Is there anything stopping us going the multicasting route, say by adding a hook or hooks of some kind to proxy that keeps track of known server states? A dedicated module could then have it's job as handling the multicasting and letting the rest of the server know if a server is on or off the air, in realtime. The proxy can then just say "which of these servers should I connect to?", and the hook will return the suggested ones in the given order. As a dedicated module, people who don't need it can turn it off or not build it. this is an idea I've been sitting on for a couple of years now. Tomcat's cluster can also multicast more than just that the node is alive, it can broadcast, what applications (URL contexts) are available, how many threads are in use, what the server load is etc. It can even broadcast automount points, or when applications are being taken offline, while the server is still up. Filip
Re: ping for http in mod_proxy
Jim Jagielski wrote: Ideally, it would be nice if we had better insight on the actual health of the backends than a simple "do they respond to OPTIONS * and how long does it take", but that's pretty much all we can do unless go full-on multicasting of info ala mod_backhand... At least the balancer is setup to allow for load balancing based on that, if the submodule/provider actually existed :) Is there anything stopping us going the multicasting route, say by adding a hook or hooks of some kind to proxy that keeps track of known server states? A dedicated module could then have it's job as handling the multicasting and letting the rest of the server know if a server is on or off the air, in realtime. The proxy can then just say "which of these servers should I connect to?", and the hook will return the suggested ones in the given order. As a dedicated module, people who don't need it can turn it off or not build it. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: ping for http in mod_proxy
On Feb 13, 2008, at 1:49 PM, Mads Toftum wrote: On Wed, Feb 13, 2008 at 12:22:43PM -0500, Akins, Brian wrote: Would it be more useful to have active healthchecking to backend servers? Ie, periodically hit a url on each origin and mark them up/down based on response. Only send traffic to up servers. I think mod_backhand does something similar but much more complex (?). Yes. Being able to check a URL seems more useful than plain ping. Even more useful would be recording the time a backend takes to answer a request and using that to distribute requests. The latter is relatively easy to do with the current impl... Maybe I'll drop the ping idea and work on this ;)
Re: ping for http in mod_proxy
On Wed, Feb 13, 2008 at 12:22:43PM -0500, Akins, Brian wrote: > Would it be more useful to have active healthchecking to backend servers? > Ie, periodically hit a url on each origin and mark them up/down based on > response. Only send traffic to up servers. I think mod_backhand does > something similar but much more complex (?). > Yes. Being able to check a URL seems more useful than plain ping. Even more useful would be recording the time a backend takes to answer a request and using that to distribute requests. vh Mads Toftum -- http://soulfood.dk
Re: ping for http in mod_proxy
On Feb 13, 2008, at 1:09 PM, Plüm, Rüdiger, VF-Group wrote: Right. Furthermore I guess we could create a generic module that needs a URL configured for a HEAD request that only replies 200 if the backend can still handle more requests. If it does not respond or with a different code this backend would be dead. I guess this module needs to fork a separate process / thread to do this checking periodically and modify the worker status accordingly. So much interesting work and so few time :-(. Or send a custom request header that has Apache respond with a custom response header that contains specific health info...
Re: ping for http in mod_proxy
On Feb 13, 2008, at 1:04 PM, Plüm, Rüdiger, VF-Group wrote: Sorry for my I-want-it-all-at-once approach :-). But this leaves the problems (most notably PR 37770) open for SSL backend connection (which would be a pity). So IMHO the socket approach would be only a first step. No, I want it all too... The problem is that you then start going beyond a simply yes/no implementation to something more robust and information-filled, but also with more overhead...
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Jim Jagielski > Gesendet: Mittwoch, 13. Februar 2008 19:00 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > > On Feb 13, 2008, at 12:41 PM, Plüm, Rüdiger, VF-Group wrote: > > > > Agreed, but I doubt that it is possible with a reasonable amout of > > health > > check frequency to find out before the first real request falls > > through, > > provided that your health checks are designed to only fail if the > > backend > > is down. If your health checks are smarter and notice that the > > backend will > > fail soon (e.g. because it reached 98% or 99% percent of its > > capacity) then > > this is a different story and can be very useful. > > Regarding the reactivation of a failed backend I fully > agree: Active > > health > > checks would be an asset here instead of just retrying failed > > backends periodically > > with real requests. > > > > Ideally, it would be nice if we had better insight on the > actual health of the backends than a simple "do they respond > to OPTIONS * and how long does it take", but that's pretty > much all we can do unless go full-on multicasting of info > ala mod_backhand... At least the balancer is setup to allow > for load balancing based on that, if the submodule/provider > actually existed :) Right. Furthermore I guess we could create a generic module that needs a URL configured for a HEAD request that only replies 200 if the backend can still handle more requests. If it does not respond or with a different code this backend would be dead. I guess this module needs to fork a separate process / thread to do this checking periodically and modify the worker status accordingly. So much interesting work and so few time :-(. Regards Rüdiger
Re: ping for http in mod_proxy
On Feb 13, 2008, at 12:59 PM, Plüm, Rüdiger, VF-Group wrote: We will never be able to completely avoid race conditions... whether keepalives are in place or not. But at least the one that comes from the keepalive timer expiring on the backend at the same time I sent the request to it. If the backend answered to OPTIONS * on a keepalive connection it is reasonable to assume that the keepalive timeout will not expire until the real request hit the backend. Regarding other races I agree, but I think the above one is the one that is hit most frequently. Ahh... but that goes back to one of my original questions: do we honor force-proxy-request-1.0 and/or proxy-nokeepalive for example or not... I can see reasons for both :) And again, we're basically doubling traffic and adding overheard (more overhead than AJP's cping/cpong) at which point I go back into wondering whether this sort of implementation makes sense at all...
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Jim Jagielski [mailto:[EMAIL PROTECTED] > Gesendet: Mittwoch, 13. Februar 2008 18:55 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > > On Feb 13, 2008, at 12:23 PM, Plüm, Rüdiger, VF-Group wrote: > > > > > > >> -Ursprüngliche Nachricht- > >> Von: Jim Jagielski > >> Gesendet: Mittwoch, 13. Februar 2008 17:07 > >> An: dev@httpd.apache.org > >> Betreff: ping for http in mod_proxy > >> > >> I've started looking at adding "ping" support for > >> mod_proxy_http to complement whats in mod_proxy_ajp... > >> The idea is to send a simple OPTIONS * to the backend > >> and hope for a reply. > >> > >> The rub is that I've been working on 2 separate > >> implementations: one talks direct to the socket and the > >> other basically tries to do what ap_proxy_http_request() > >> and ap_proxy_http_process_response() does, as far > >> as using brigades, etc... The former is much further > >> along, btw. > > > > We cannot use the socket approach since we need to be able to > > handle SSL backend connections and this requires the brigades - > > filters > > approach. > > The initial approach was to not honor SSL connections and > just ignore that... The idea that we really want a fast and > dirty "Are you there" flag, so to avoid as much overhead > as possible... Sorry for my I-want-it-all-at-once approach :-). But this leaves the problems (most notably PR 37770) open for SSL backend connection (which would be a pity). So IMHO the socket approach would be only a first step. Regards Rüdiger
Re: ping for http in mod_proxy
On Feb 13, 2008, at 12:41 PM, Plüm, Rüdiger, VF-Group wrote: Agreed, but I doubt that it is possible with a reasonable amout of health check frequency to find out before the first real request falls through, provided that your health checks are designed to only fail if the backend is down. If your health checks are smarter and notice that the backend will fail soon (e.g. because it reached 98% or 99% percent of its capacity) then this is a different story and can be very useful. Regarding the reactivation of a failed backend I fully agree: Active health checks would be an asset here instead of just retrying failed backends periodically with real requests. Ideally, it would be nice if we had better insight on the actual health of the backends than a simple "do they respond to OPTIONS * and how long does it take", but that's pretty much all we can do unless go full-on multicasting of info ala mod_backhand... At least the balancer is setup to allow for load balancing based on that, if the submodule/provider actually existed :)
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Jim Jagielski > Gesendet: Mittwoch, 13. Februar 2008 18:52 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > > On Feb 13, 2008, at 12:27 PM, Plüm, Rüdiger, VF-Group wrote: > > > > > > >> -Ursprüngliche Nachricht- > >> Von: Akins, Brian > >> Gesendet: Mittwoch, 13. Februar 2008 18:23 > >> An: dev@httpd.apache.org > >> Betreff: Re: ping for http in mod_proxy > >> > >> On 2/13/08 11:07 AM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > >> > >>> I've started looking at adding "ping" support for > >>> mod_proxy_http to complement whats in mod_proxy_ajp... > >>> The idea is to send a simple OPTIONS * to the backend > >>> and hope for a reply. > >> > >> > >> Would it be more useful to have active healthchecking to > >> backend servers? > > > > This does not help with race conditions on HTTP keepalive > connections. > > We will never be able to completely avoid race conditions... > whether keepalives are in place or not. But at least the one that comes from the keepalive timer expiring on the backend at the same time I sent the request to it. If the backend answered to OPTIONS * on a keepalive connection it is reasonable to assume that the keepalive timeout will not expire until the real request hit the backend. Regarding other races I agree, but I think the above one is the one that is hit most frequently. Regards Rüdiger
Re: ping for http in mod_proxy
On Feb 13, 2008, at 12:23 PM, Plüm, Rüdiger, VF-Group wrote: -Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Mittwoch, 13. Februar 2008 17:07 An: dev@httpd.apache.org Betreff: ping for http in mod_proxy I've started looking at adding "ping" support for mod_proxy_http to complement whats in mod_proxy_ajp... The idea is to send a simple OPTIONS * to the backend and hope for a reply. The rub is that I've been working on 2 separate implementations: one talks direct to the socket and the other basically tries to do what ap_proxy_http_request() and ap_proxy_http_process_response() does, as far as using brigades, etc... The former is much further along, btw. We cannot use the socket approach since we need to be able to handle SSL backend connections and this requires the brigades - filters approach. The initial approach was to not honor SSL connections and just ignore that... The idea that we really want a fast and dirty "Are you there" flag, so to avoid as much overhead as possible...
Re: ping for http in mod_proxy
On Feb 13, 2008, at 12:27 PM, Plüm, Rüdiger, VF-Group wrote: -Ursprüngliche Nachricht- Von: Akins, Brian Gesendet: Mittwoch, 13. Februar 2008 18:23 An: dev@httpd.apache.org Betreff: Re: ping for http in mod_proxy On 2/13/08 11:07 AM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: I've started looking at adding "ping" support for mod_proxy_http to complement whats in mod_proxy_ajp... The idea is to send a simple OPTIONS * to the backend and hope for a reply. Would it be more useful to have active healthchecking to backend servers? This does not help with race conditions on HTTP keepalive connections. We will never be able to completely avoid race conditions... whether keepalives are in place or not.
Re: ping for http in mod_proxy
On Feb 13, 2008, at 12:22 PM, Akins, Brian wrote: On 2/13/08 11:07 AM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: I've started looking at adding "ping" support for mod_proxy_http to complement whats in mod_proxy_ajp... The idea is to send a simple OPTIONS * to the backend and hope for a reply. Would it be more useful to have active healthchecking to backend servers? Ie, periodically hit a url on each origin and mark them up/down based on response. Only send traffic to up servers. I think mod_backhand does something similar but much more complex (?). I had started looking to add this to balancer, but I have no time. That was the other option as well... some sort of hearbeat loop which updates worker status. Still, we get into the issue with how much of "how proxy connects to and communicates with the backend" to honor or work around.
Re: ping for http in mod_proxy
On 2/13/08 12:41 PM, "Plüm, Rüdiger, VF-Group" <[EMAIL PROTECTED]> wrote: > If your health checks are smarter and notice that the backend will > fail soon (e.g. because it reached 98% or 99% percent of its capacity) then > this is a different story and can be very useful. Correct. Perhaps a weighted round-robin that is based on response time would be fairly easy to code... (Says the guy with no time.) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Akins, Brian > Gesendet: Mittwoch, 13. Februar 2008 18:34 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > On 2/13/08 12:27 PM, "Plüm, Rüdiger, VF-Group" > <[EMAIL PROTECTED]> > wrote: > > > This does not help with race conditions on HTTP keepalive > connections. > > Nevertheless active healthchecking could be useful. But on > a busy site > > I guess a real request will notice before the healthcheck > that one backend > > is down or the frequence of health checks needs to be insane. > > > On a busy site you want to know if a server is down before > you send a few > thousand requests to it and want to know that it's up as soon as it's > available so you can send a few thousand requests to it. Agreed, but I doubt that it is possible with a reasonable amout of health check frequency to find out before the first real request falls through, provided that your health checks are designed to only fail if the backend is down. If your health checks are smarter and notice that the backend will fail soon (e.g. because it reached 98% or 99% percent of its capacity) then this is a different story and can be very useful. Regarding the reactivation of a failed backend I fully agree: Active health checks would be an asset here instead of just retrying failed backends periodically with real requests. Regards Rüdiger
Re: ping for http in mod_proxy
On 2/13/08 12:27 PM, "Plüm, Rüdiger, VF-Group" <[EMAIL PROTECTED]> wrote: > This does not help with race conditions on HTTP keepalive connections. > Nevertheless active healthchecking could be useful. But on a busy site > I guess a real request will notice before the healthcheck that one backend > is down or the frequence of health checks needs to be insane. On a busy site you want to know if a server is down before you send a few thousand requests to it and want to know that it's up as soon as it's available so you can send a few thousand requests to it. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Akins, Brian > Gesendet: Mittwoch, 13. Februar 2008 18:23 > An: dev@httpd.apache.org > Betreff: Re: ping for http in mod_proxy > > On 2/13/08 11:07 AM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > > > I've started looking at adding "ping" support for > > mod_proxy_http to complement whats in mod_proxy_ajp... > > The idea is to send a simple OPTIONS * to the backend > > and hope for a reply. > > > Would it be more useful to have active healthchecking to > backend servers? This does not help with race conditions on HTTP keepalive connections. Nevertheless active healthchecking could be useful. But on a busy site I guess a real request will notice before the healthcheck that one backend is down or the frequence of health checks needs to be insane. Regards Rüdiger
re: ping for http in mod_proxy
> -Ursprüngliche Nachricht- > Von: Jim Jagielski > Gesendet: Mittwoch, 13. Februar 2008 17:07 > An: dev@httpd.apache.org > Betreff: ping for http in mod_proxy > > I've started looking at adding "ping" support for > mod_proxy_http to complement whats in mod_proxy_ajp... > The idea is to send a simple OPTIONS * to the backend > and hope for a reply. > > The rub is that I've been working on 2 separate > implementations: one talks direct to the socket and the > other basically tries to do what ap_proxy_http_request() > and ap_proxy_http_process_response() does, as far > as using brigades, etc... The former is much further > along, btw. We cannot use the socket approach since we need to be able to handle SSL backend connections and this requires the brigades - filters approach. > > In any case, both implementations can have what I can > envision as some "weird effects" as far as how they > should actually work in the real world. For example, > should it honor force-proxy-request-1.0 and/or > proxy-nokeepalive, and, if so, how does that affect I think the ping is of limited use for HTTP/1.0 / non keepalive requests. I see it mainly as a measure to avoid race conditions with keepalive connections that get closed just in the moment as the request is being sent. The only benefit I would see for HTTP/1.0 / non keepalive requests would be that a successful connect call (we always create a new connections in this case) does not guarantee us that you get *any* response (in a timely manner). This could be because the connect is queued up on the backend in the connection backlog or in an application specific request queue and will be never processed. > the subsequent "real" request... In other words, > it's not as clean as AJP which has a "real" ping. > And all this is wondering whether or not it's > really useful in the 1st place... > > So before I go any further, any comments or ideas > on whether is is even a Good Idea or not... I'm starting > to have 2nd thoughts :) > It is a good idea for certain purposes and I really appreciate that you work on this. This issue is still on my when-I-got-time-TODO list. If not done until then, maybe something for the hackathon to work on. Regards Rüdiger
Re: ping for http in mod_proxy
On 2/13/08 11:07 AM, "Jim Jagielski" <[EMAIL PROTECTED]> wrote: > I've started looking at adding "ping" support for > mod_proxy_http to complement whats in mod_proxy_ajp... > The idea is to send a simple OPTIONS * to the backend > and hope for a reply. Would it be more useful to have active healthchecking to backend servers? Ie, periodically hit a url on each origin and mark them up/down based on response. Only send traffic to up servers. I think mod_backhand does something similar but much more complex (?). I had started looking to add this to balancer, but I have no time. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
ping for http in mod_proxy
I've started looking at adding "ping" support for mod_proxy_http to complement whats in mod_proxy_ajp... The idea is to send a simple OPTIONS * to the backend and hope for a reply. The rub is that I've been working on 2 separate implementations: one talks direct to the socket and the other basically tries to do what ap_proxy_http_request() and ap_proxy_http_process_response() does, as far as using brigades, etc... The former is much further along, btw. In any case, both implementations can have what I can envision as some "weird effects" as far as how they should actually work in the real world. For example, should it honor force-proxy-request-1.0 and/or proxy-nokeepalive, and, if so, how does that affect the subsequent "real" request... In other words, it's not as clean as AJP which has a "real" ping. And all this is wondering whether or not it's really useful in the 1st place... So before I go any further, any comments or ideas on whether is is even a Good Idea or not... I'm starting to have 2nd thoughts :)
