Re: anybody know how to health check a twisted xmlrpc server?
Hi Krzysztof, On Sun, Jan 03, 2010 at 02:19:24AM +0100, Krzysztof Ol?dzki wrote: > >I would not be surprized if > >he already had some pieces of code in good shape. > > Not this time. ;) hehe :-) > The code I have currently is only a fast-hack to help my "IIS > colleagues" by extracting IIS "extended http codes" provided in the > html, so they are able to track malfunctions faster. However, I believe > it would be quite easy to make it more generic, mainly by adding a > proper configuration interface and allowing to fail a check, not only by > a http response code, but also by a matched/unmatched regex (slower) or > by a found/missing string (faster). In fact I don't care much if regex tests are slower, because people are not forced to use them. If people want scalable fast tests, they have to run an agent on each server (inetd script). They want regex or string matching for simplicity, which is acceptable as long as they don't have too many servers. > I think I should be able to publish a first RFC patch at the beginning > of the next week (~11.01.2009), so after 2-3 interactions it should be > ready to be merged. If you are still going to accept such changes after > the 1.4dev5, it should get into the 1.4 release. That would be very cool. Yes, such changes will be accepted after dev5. I think we'll be able to merge the generic stickiness framework too, and that should be OK for a new release. Next versions will be dedicated to chase bugs and add minor features. Best regards, Willy
Re: anybody know how to health check a twisted xmlrpc server?
On 2009-12-24 10:22, Willy Tarreau wrote: [ forgot to CC Krzysztof ] On Wed, Dec 23, 2009 at 11:27 PM, Willy Tarreau wrote: On Wed, Dec 23, 2009 at 10:52:52PM -0800, David Birdsong wrote: this can be dropped for now. it turns out that twisted doesn't actually return error codes when you instruct it to. it puts the error codes in the xml response for code to deal with but slaps a http 200 in the headers. so we really need a working regex-based health check :-/ Willy i've had coworkers complain about health checks only looking at http response codes. i'm undecided. i think it's reasonable for proxies/load balancers to work at the protocol level, but i've definitely been hamstrung by it when working with libraries that don't always provide access to the protocol levels without combing through and learning the entire library. Other people need to ensure that a specific tag is present (or is not present) in a given page. For instance, it can make sense to kill a server which reports SQL errors in the HTML code. how hard would that be to add? It's not that hard and it has already been done once, but the code contained various issues and bugs which need to be fixed. After some review, it appeared that it would be easier to re-implement it (possibly keeping some parts of the patch) than auditing it completely and fixing the remaining issues. If you want to work on something like that, please first check with Krzysztof (CCed), because he regularly works on stats and checks and I would not be surprized if he already had some pieces of code in good shape. Not this time. ;) The code I have currently is only a fast-hack to help my "IIS colleagues" by extracting IIS "extended http codes" provided in the html, so they are able to track malfunctions faster. However, I believe it would be quite easy to make it more generic, mainly by adding a proper configuration interface and allowing to fail a check, not only by a http response code, but also by a matched/unmatched regex (slower) or by a found/missing string (faster). I think I should be able to publish a first RFC patch at the beginning of the next week (~11.01.2009), so after 2-3 interactions it should be ready to be merged. If you are still going to accept such changes after the 1.4dev5, it should get into the 1.4 release. Best regards, Krzysztof Olędzki
Re: anybody know how to health check a twisted xmlrpc server?
[ forgot to CC Krzysztof ] > On Wed, Dec 23, 2009 at 11:27 PM, Willy Tarreau wrote: > > On Wed, Dec 23, 2009 at 10:52:52PM -0800, David Birdsong wrote: > >> this can be dropped for now. it turns out that twisted doesn't > >> actually return error codes when you instruct it to. it puts the > >> error codes in the xml response for code to deal with but slaps a http > >> 200 in the headers. > > > > so we really need a working regex-based health check :-/ > > > > Willy > > i've had coworkers complain about health checks only looking at http > response codes. > > i'm undecided. i think it's reasonable for proxies/load balancers to > work at the protocol level, but i've definitely been hamstrung by it > when working with libraries that don't always provide access to the > protocol levels without combing through and learning the entire > library. Other people need to ensure that a specific tag is present (or is not present) in a given page. For instance, it can make sense to kill a server which reports SQL errors in the HTML code. > how hard would that be to add? It's not that hard and it has already been done once, but the code contained various issues and bugs which need to be fixed. After some review, it appeared that it would be easier to re-implement it (possibly keeping some parts of the patch) than auditing it completely and fixing the remaining issues. If you want to work on something like that, please first check with Krzysztof (CCed), because he regularly works on stats and checks and I would not be surprized if he already had some pieces of code in good shape. > i guess there's not much overhead > involved seeing any sane admin runs health checks at most a handful of > times in a 3-5 second period. You could even do that dozens of times a second (provided you don't have thousands of servers). Regards, Willy
Re: anybody know how to health check a twisted xmlrpc server?
On Thu, Dec 24, 2009 at 12:05:54AM -0800, David Birdsong wrote: > On Wed, Dec 23, 2009 at 11:27 PM, Willy Tarreau wrote: > > On Wed, Dec 23, 2009 at 10:52:52PM -0800, David Birdsong wrote: > >> this can be dropped for now. it turns out that twisted doesn't > >> actually return error codes when you instruct it to. it puts the > >> error codes in the xml response for code to deal with but slaps a http > >> 200 in the headers. > > > > so we really need a working regex-based health check :-/ > > > > Willy > > i've had coworkers complain about health checks only looking at http > response codes. > > i'm undecided. i think it's reasonable for proxies/load balancers to > work at the protocol level, but i've definitely been hamstrung by it > when working with libraries that don't always provide access to the > protocol levels without combing through and learning the entire > library. Other people need to ensure that a specific tag is present (or is not present) in a given page. For instance, it can make sense to kill a server which reports SQL errors in the HTML code. > how hard would that be to add? It's not that hard and it has already been done once, but the code contained various issues and bugs which need to be fixed. After some review, it appeared that it would be easier to re-implement it (possibly keeping some parts of the patch) than auditing it completely and fixing the remaining issues. If you want to work on something like that, please first check with Krzysztof (CCed), because he regularly works on stats and checks and I would not be surprized if he already had some pieces of code in good shape. > i guess there's not much overhead > involved seeing any sane admin runs health checks at most a handful of > times in a 3-5 second period. You could even do that dozens of times a second (provided you don't have thousands of servers). Regards, Willy
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 11:27 PM, Willy Tarreau wrote: > On Wed, Dec 23, 2009 at 10:52:52PM -0800, David Birdsong wrote: >> this can be dropped for now. it turns out that twisted doesn't >> actually return error codes when you instruct it to. it puts the >> error codes in the xml response for code to deal with but slaps a http >> 200 in the headers. > > so we really need a working regex-based health check :-/ > > Willy i've had coworkers complain about health checks only looking at http response codes. i'm undecided. i think it's reasonable for proxies/load balancers to work at the protocol level, but i've definitely been hamstrung by it when working with libraries that don't always provide access to the protocol levels without combing through and learning the entire library. how hard would that be to add? i guess there's not much overhead involved seeing any sane admin runs health checks at most a handful of times in a 3-5 second period. > >
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 10:52:52PM -0800, David Birdsong wrote: > this can be dropped for now. it turns out that twisted doesn't > actually return error codes when you instruct it to. it puts the > error codes in the xml response for code to deal with but slaps a http > 200 in the headers. so we really need a working regex-based health check :-/ Willy
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 10:40:17PM -0800, David Birdsong wrote: > On Wed, Dec 23, 2009 at 10:07 PM, David Birdsong > wrote: > > On Wed, Dec 23, 2009 at 9:57 PM, Willy Tarreau wrote: > >> On Wed, Dec 23, 2009 at 09:48:15PM -0800, David Birdsong wrote: > >>> i see that POST is a valid httpchk parameter, but without ripping > >>> twisted apart, i'm not sure how it could allow an empty POST through > >>> to code that could return a 200 or 204. > >>> > >>> has anybody already gone through this? > >> > >> Never tested, but right now you can still post contents and headers, > >> though it's a ugly hack. It consists in appending the headers and > >> data after the HTTP version : > >> > >> option httpchk POST /something HTTP/1.0\r\nContent-length:\ > >> 7\r\n\r\nfoo=bar > >> > > i was wondering if contents were allowed after the headers. i'll give > > it a shot. > > > > ugly, and so far not working: > option httpchk POST /db_api/lucene_check.php HTTP/1.0\r\n\r\n version="1.0" encoding="iso-8859-1"?> > query > kafjkdfajdfkjfkajdf > > > if that line wrapped, i have it all on one line in the config. But you forgot to escape all spaces. The option takes only 3 args max, so the rest is dropped. You should also escape the quotes. I'm just realizing that it should be easier to quote the whole block than escaping all spaces BTW. Regards, Willy
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 10:40 PM, David Birdsong wrote: > On Wed, Dec 23, 2009 at 10:07 PM, David Birdsong > wrote: >> On Wed, Dec 23, 2009 at 9:57 PM, Willy Tarreau wrote: >>> On Wed, Dec 23, 2009 at 09:48:15PM -0800, David Birdsong wrote: i see that POST is a valid httpchk parameter, but without ripping twisted apart, i'm not sure how it could allow an empty POST through to code that could return a 200 or 204. has anybody already gone through this? >>> >>> Never tested, but right now you can still post contents and headers, >>> though it's a ugly hack. It consists in appending the headers and >>> data after the HTTP version : >>> >>> option httpchk POST /something HTTP/1.0\r\nContent-length:\ >>> 7\r\n\r\nfoo=bar >>> >> i was wondering if contents were allowed after the headers. i'll give >> it a shot. >> > > ugly, and so far not working: > option httpchk POST /db_api/lucene_check.php HTTP/1.0\r\n\r\n version="1.0" encoding="iso-8859-1"?> > query > kafjkdfajdfkjfkajdf > > > if that line wrapped, i have it all on one line in the config. > > tcpdump shows only this much coming through: > > POST /db_api/lucene_check.php HTTP/1.0 > > > > > > > HTTP/1.0 200 OK > this can be dropped for now. it turns out that twisted doesn't actually return error codes when you instruct it to. it puts the error codes in the xml response for code to deal with but slaps a http 200 in the headers. >>> I told you it was ugly :-) >>> >>> You may also have to specify a Host: header depending on your >>> service, and maybe even a content-type. Find about it by trial >>> and error. Please post your findings on the list, as I'm sure >>> other people might be interested too. >>> >>> Regards, >>> Willy >>> >>> >> >
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 10:07 PM, David Birdsong wrote: > On Wed, Dec 23, 2009 at 9:57 PM, Willy Tarreau wrote: >> On Wed, Dec 23, 2009 at 09:48:15PM -0800, David Birdsong wrote: >>> i see that POST is a valid httpchk parameter, but without ripping >>> twisted apart, i'm not sure how it could allow an empty POST through >>> to code that could return a 200 or 204. >>> >>> has anybody already gone through this? >> >> Never tested, but right now you can still post contents and headers, >> though it's a ugly hack. It consists in appending the headers and >> data after the HTTP version : >> >> option httpchk POST /something HTTP/1.0\r\nContent-length:\ >> 7\r\n\r\nfoo=bar >> > i was wondering if contents were allowed after the headers. i'll give > it a shot. > ugly, and so far not working: option httpchk POST /db_api/lucene_check.php HTTP/1.0\r\n\r\n query kafjkdfajdfkjfkajdf if that line wrapped, i have it all on one line in the config. tcpdump shows only this much coming through: POST /db_api/lucene_check.php HTTP/1.0 >> I told you it was ugly :-) >> >> You may also have to specify a Host: header depending on your >> service, and maybe even a content-type. Find about it by trial >> and error. Please post your findings on the list, as I'm sure >> other people might be interested too. >> >> Regards, >> Willy >> >> >
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 9:57 PM, Willy Tarreau wrote: > On Wed, Dec 23, 2009 at 09:48:15PM -0800, David Birdsong wrote: >> i see that POST is a valid httpchk parameter, but without ripping >> twisted apart, i'm not sure how it could allow an empty POST through >> to code that could return a 200 or 204. >> >> has anybody already gone through this? > > Never tested, but right now you can still post contents and headers, > though it's a ugly hack. It consists in appending the headers and > data after the HTTP version : > > option httpchk POST /something HTTP/1.0\r\nContent-length:\ > 7\r\n\r\nfoo=bar > i was wondering if contents were allowed after the headers. i'll give it a shot. > I told you it was ugly :-) > > You may also have to specify a Host: header depending on your > service, and maybe even a content-type. Find about it by trial > and error. Please post your findings on the list, as I'm sure > other people might be interested too. > > Regards, > Willy > >
Re: anybody know how to health check a twisted xmlrpc server?
On Wed, Dec 23, 2009 at 09:48:15PM -0800, David Birdsong wrote: > i see that POST is a valid httpchk parameter, but without ripping > twisted apart, i'm not sure how it could allow an empty POST through > to code that could return a 200 or 204. > > has anybody already gone through this? Never tested, but right now you can still post contents and headers, though it's a ugly hack. It consists in appending the headers and data after the HTTP version : option httpchk POST /something HTTP/1.0\r\nContent-length:\ 7\r\n\r\nfoo=bar I told you it was ugly :-) You may also have to specify a Host: header depending on your service, and maybe even a content-type. Find about it by trial and error. Please post your findings on the list, as I'm sure other people might be interested too. Regards, Willy
