Hacking in an SLA for proxied requests in mod_proxy_http
Hey all,
I wanted to enforce an SLA on certain http requests to apache.
Essentially provide a the external client users an guarantee that a
valid response will be given within XXms and all errors are
suppressed. This is for an ReST API that returns JSON or XML data.
Ronald Park attempted to do something similar for Apache 2.0 in Feb
2008 here: http://bit.ly/HglyS
The issues with making apache do this are as follows:
* ProxyTimeout is global or scoped to the particular vhost
* ErrorDocuments still return the error code (503, 404, etc)
* No way to tie ErrorDocuments and ProxyTimeouts to particular requests.
I followed Ronald's lead and reimplemented in apache 2.2 with a few
new additions.
The below example is a rewrite rule that makes no changes to the URL
itself for a JS API presumably returning data in JSON.
RewriteRule ^/api/(.*).js\?*(.*)$ http://backendproxy/api/$1.js?$2
[P,QSA,E=proxy-timeout:900ms,E=error-suppress:true,E=error-headers:/errorapi.js.HTTP,E=error-document:/errorapi.js]
With the SLA enforcement modifications enabled, the URL will return
data from the backend system within 900ms or a timeout occurs. At this
point apache will stop waiting for the backend response and serve back
the static files /errorapi.js.HTTP as HTTP headers and /errorapi.js as
contents.
$cat /var/www/html/errorapi.js.HTTP
Status: 204
Content-type: application/javascript
$cat /var/www/html/errorapi.js
var xxx_api_data={data:[]}; /* ERR */
There are four environment variables the SLA hack looks for:
* proxy-timeout: - time in seconds or milliseconds to wait until timing out
* error-suppress: - true/false switch on suppressing all non 2xx
errors from the backend.
* error-headers: - file of syntax correct HTTP headers to return
to the client
* error-document: - file of content body to be returned to the client
Leaving off the proxy-timeout will only suppress errors from the
backend after the global timeout occurs. Leaving off
error-suppress:true will ensure that the 5xx timeout error from
mod_proxy_http is returned intact to the client.
Code here: http://github.com/nealrichter/mod_proxy_http_sla/tree
Diff here: http://bit.ly/UdAHB
Blog Post:
http://aicoder.blogspot.com/2009/07/hacking-apaches-modproxyhttp-to-enforce.html
Please examine and comment.. it's on production servers handling
approx of 50Million API hits per day.. no issues thus far. I'd like
some feedback if I'm resetting the timeout back to the previous one on
the socket correctly and if there is a better way to do this in the
code. I essentially used Ronald's code plus some stuff from
mod_asis.
Thanks - Neal Richter
Re: Module that forward requests
Hi Eric. NOOWWW I understood !!! But happen a error !! I posted this error. Help me Thank you Ricardo Eric Covener wrote: > > On Mon, Jul 13, 2009 at 10:06 AM, ricardo13 > wrote: It's difficult. I'd suggest finding a way to use existing modules. >>> >>> Use or modify, that is. >>> Can my module to use mod_proxy for forward requests ? How do I do it ? > > I'm not convinced you need a module. mod_rewrite already illustrates > this with the P flag. > > -- > Eric Covener > cove...@gmail.com > > -- View this message in context: http://www.nabble.com/Module-that-forward-requests-tp24461810p24486953.html Sent from the Apache HTTP Server - Module Writers mailing list archive at Nabble.com.
[warn] worker http://ip_maquina/ already used by another worker
Hi, I have a webcluster and I'm prioritize the requests. I modify request_rec adding a field called prior and conditional test ( IF() ) in mod_rewrite for classify. All time that start APACHE show the follow: # sudo apachectl -k start [warn] worker http://ip_maquina/ already used by another worker What's this ?? Thank you. Ricardo -- View this message in context: http://www.nabble.com/-warn--worker-http%3A--ip_maquina--already-used-by-another-worker-tp24486952p24486952.html Sent from the Apache HTTP Server - Dev mailing list archive at Nabble.com.
Re: mod_deflate DoS using HEAD
Roy T. Fielding wrote: > On Jul 14, 2009, at 10:02 AM, Nick Kew wrote: >> That, on the other hand, stands. In the case of an HTTP/1.0 >> request, we'd be closing the connection to signal end-of-response. > > Not on a HEAD request. But on the GET request with the deflate filter installed, we would not send a C-L but would close the connection upon transmission of the body, so we can drop the pretense of querying the true C-L.
Re: mod_deflate DoS using HEAD
On Jul 14, 2009, at 10:02 AM, Nick Kew wrote: That, on the other hand, stands. In the case of an HTTP/1.0 request, we'd be closing the connection to signal end-of-response. Not on a HEAD request. Roy
Re: mod_deflate DoS using HEAD
Nick Kew wrote: The content-length could've been set anyway - the simplest case being a static file that's been "stat"ed. Have we definitely unset it? D'oh. Of course we have. Is this really an optimization? Sounds like correctness :) And do we want to also validate that Accept-Encoding: chunked is present? No, deflate doesn't imply chunked encoding. That, on the other hand, stands. In the case of an HTTP/1.0 request, we'd be closing the connection to signal end-of-response. -- Nick Kew
Re: mod_deflate DoS using HEAD
William A. Rowe, Jr. wrote:
Plüm, Rüdiger, VF-Group wrote:
+/*
+ * Optimization: If we are a HEAD request and bytes_sent is not zero
+ * it means that we have passed the content-length filter once and
+ * have more data to sent. This means that the content-length filter
+ * could not determine our content-length for the response to the
+ * HEAD request anyway (the associated GET request would deliver the
+ * body in chunked encoding) and we can stop compressing.
+ */
The content-length could've been set anyway - the simplest case being
a static file that's been "stat"ed. Have we definitely unset it?
Is this really an optimization? Sounds like correctness :) And do we want
to also validate that Accept-Encoding: chunked is present?
No, deflate doesn't imply chunked encoding.
+if (r->header_only && r->bytes_sent) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f->next, bb);
+}
Other than comments above - +1!
Other than comments above, +0.5. The other +half is still
contemplating giving control to the sysop - c.f. my previous
mail on the subject.
--
Nick Kew
Re: FTP open questions
On Jul 13, 2009, at 8:28 PM, William A. Rowe, Jr. wrote: Just finished the last showstopper. I would be happy to advance this to release / general availability vote with the next release, if we can determine just a few oddball issue resolutions. Jim and I have already gone ahead and moved many internal interfaces out of the private headers, which was my motivation for holding off a year ago. Should we advertise the commands we have not implemented, or remove them? +1 for keeping as is. Should we alert the user to the ServerAdmin address in the HELP contents? +1 Right now HELP offers up; 214-The following commands are recognized (* =>'s unimplemented). FEATTYPERMD QUITRNTOPORT *MODEAPPE *ALLOSTORPWD*STOU *REINAUTHMDTMSYST XMKD *SITEXCWDPASSPASVDELE *ACCTEPRT SIZEXRMDNOOPLISTRESTPBSZXCUPNLST *SMNTXPWDABORPROTHELPCDUP *STRURNFR MKD*STATRETRCWD EPSVUSER 214 Direct comments to [no address given] Just for reference, three popular linux servers respond with no unimplemented features, one offers "Direct comments to" admin address, one offers the website address of it's project, and one just ends with the result "Help OK" The admin has little control over which commands are supported (although which commands are -allowed- is another matter entirely :) So it just strikes me as odd to direct comments with respect to the HELP inquiry.
Re: Apache delivers PHP source code for vim backup files
On Tue, Jul 14, 2009 at 11:54 AM, Jens Winter wrote: > Hi, > > I wonder which rules are used to decide if a file is processed by PHP. For > example x.php, x.php.bak and x.php.x~ are all processed, but x.php~ is not > (at least by default). This could be an issue if you use vim or similar > editors to edit the config files of e.g. WordPress or MediaWiki (containing > DB passwords) directly in the server directory (which you shouldn't do, but > we all know that some people will do anyway...). > > So if so many filename schemes result in processing the PHP code, why are > these critical files delivered as source code (again talking about default > behavior)? Directives that accept "filename extensions" treat "foo.bar.baz" as having two distinct extensions. http://httpd.apache.org/docs/2.2/mod/mod_mime.html http://httpd.apache.org/userslist.html -- Eric Covener cove...@gmail.com
Re: mod_deflate DoS using HEAD
Plüm, Rüdiger, VF-Group wrote:
>
> +/*
> + * Optimization: If we are a HEAD request and bytes_sent is not zero
> + * it means that we have passed the content-length filter once and
> + * have more data to sent. This means that the content-length filter
> + * could not determine our content-length for the response to the
> + * HEAD request anyway (the associated GET request would deliver the
> + * body in chunked encoding) and we can stop compressing.
> + */
Is this really an optimization? Sounds like correctness :) And do we want
to also validate that Accept-Encoding: chunked is present?
> +if (r->header_only && r->bytes_sent) {
> +ap_remove_output_filter(f);
> +return ap_pass_brigade(f->next, bb);
> +}
Other than comments above - +1!
RE: mod_deflate DoS using HEAD
> -Original Message-
> From: William A. Rowe, Jr.
> Sent: Montag, 13. Juli 2009 23:58
> To: dev@httpd.apache.org
> Subject: Re: mod_deflate DoS using HEAD
>
> Nick Kew wrote:
> > Eric Covener wrote:
> >
> >> /* For a 304 response, only change the headers */
> >> -if (r->status == HTTP_NOT_MODIFIED) {
> >> +if (r->status == HTTP_NOT_MODIFIED || r->header_only) {
> >
> > Technically speaking, screws up the protocol.
> >
> > IMHO it would be acceptable provided:
> > (a) it's an option for the admin, rather than enforced
> > (b) it's documented
> > (c) the headers are correct: either Content-Encoding is
> > unset (uncompressed response) or Content-Length is
> > unset. Probably the former.
>
> Agreed. It's not a DoS. If the admin wants to conserve CPU
> resources, they must either;
>
> * cache the deflated pages (avoid user-agent header if there
>are multiples, which reminds me we need a module to unset the
>accept deflate trigger on non-compliant browsers running
>very-first in the quick_handler.)
>
> * create gzip'ed content, navigate the choice of content through
>multiviews.
>
> * do not do server-side deflation (it is expensive).
>
All very true. But how about the following patch. It should do no
harm and should solve the issue in at least some cases (I think
in most cases):
Index: modules/filters/mod_deflate.c
===
--- modules/filters/mod_deflate.c (revision 793927)
+++ modules/filters/mod_deflate.c (working copy)
@@ -629,6 +629,19 @@
apr_bucket *b;
apr_size_t len;
+/*
+ * Optimization: If we are a HEAD request and bytes_sent is not zero
+ * it means that we have passed the content-length filter once and
+ * have more data to sent. This means that the content-length filter
+ * could not determine our content-length for the response to the
+ * HEAD request anyway (the associated GET request would deliver the
+ * body in chunked encoding) and we can stop compressing.
+ */
+if (r->header_only && r->bytes_sent) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f->next, bb);
+}
+
e = APR_BRIGADE_FIRST(bb);
if (APR_BUCKET_IS_EOS(e)) {
Regards
Rüdiger
Apache delivers PHP source code for vim backup files
Hi, I wonder which rules are used to decide if a file is processed by PHP. For example x.php, x.php.bak and x.php.x~ are all processed, but x.php~ is not (at least by default). This could be an issue if you use vim or similar editors to edit the config files of e.g. WordPress or MediaWiki (containing DB passwords) directly in the server directory (which you shouldn't do, but we all know that some people will do anyway...). So if so many filename schemes result in processing the PHP code, why are these critical files delivered as source code (again talking about default behavior)? Jens
Re: FTP open questions
On Jul 14, 2009, at 8:02 AM, Jorge Schrauwen wrote: On Tue, Jul 14, 2009 at 2:28 AM, William A. Rowe, Should we advertise the commands we have not implemented, or remove them? Yes, It's always useful to know for a more advance user. +1 on keeping them. Should we alert the user to the ServerAdmin address in the HELP contents? Maybe reuse ServerTokens for this? Elegant! S. -- Sander Temme scte...@apache.org PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF smime.p7s Description: S/MIME cryptographic signature
Re: FTP open questions
On Tue, Jul 14, 2009 at 2:28 AM, William A. Rowe, Jr. wrote: > Just finished the last showstopper. I would be happy to advance this > to release / general availability vote with the next release, if we can > determine just a few oddball issue resolutions. Jim and I have already > gone ahead and moved many internal interfaces out of the private headers, > which was my motivation for holding off a year ago. > > Should we advertise the commands we have not implemented, or remove them? Yes, It's always useful to know for a more advance user. > > Should we alert the user to the ServerAdmin address in the HELP contents? Maybe reuse ServerTokens for this? Full/OS => serveradmin + unimplemented commands Minimal => unimplemented commands Minor => only list implemented commands Major => only list allowed commands Prod => no help? That way the server admin still has a say in it? Not sure the extra coding is worth it though. > > Right now HELP offers up; > > 214-The following commands are recognized (* =>'s unimplemented). > FEAT TYPE RMD QUIT RNTO PORT *MODE APPE > *ALLO STOR PWD *STOU *REIN AUTH MDTM SYST > XMKD *SITE XCWD PASS PASV DELE *ACCT EPRT > SIZE XRMD NOOP LIST REST PBSZ XCUP NLST > *SMNT XPWD ABOR PROT HELP CDUP *STRU RNFR > MKD *STAT RETR CWD EPSV USER > 214 Direct comments to [no address given] > > Just for reference, three popular linux servers respond with no > unimplemented features, one offers "Direct comments to" admin address, > one offers the website address of it's project, and one just ends with > the result "Help OK" > > The admin has little control over which commands are supported (although > which commands are -allowed- is another matter entirely :) So it just > strikes me as odd to direct comments with respect to the HELP inquiry. > >
Re: AuthBasicProvider failover and mod_authnz_ldap
Plüm, Rüdiger, VF-Group wrote: > Haven't thought this through but from a first glance it makes sense that > the next provider can continue if the first one had a fatal error. +1. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
RE: AuthBasicProvider failover and mod_authnz_ldap
> -Original Message- > From: Eric Covener > Sent: Montag, 13. Juli 2009 23:31 > To: dev@httpd.apache.org > Subject: AuthBasicProvider failover and mod_authnz_ldap > > PR#47521 points out that when mod_authnz_ldap has some fatal LDAP > connectivity error, it doesn't allow other AuthBasicProviders to have > a shot at checking the userid. > > It seems like the normal use case for two providers is when there are > two disjoint user repositories, and we only move on to search the > second when the user of interest isn't found in the first. > > For LDAP, should we treat a failure to even search the database this > same way, allowing it to move onto other providers > (AUTH_USER_NOT_FOUND vs. AUTH_GENERAL_ERROR)? It seems to me that the > LDAP backends often have poor reliability and lots of use cases would > want the 2nd provider for emergencies, at little expense (hypothetical > attacker that took out your LDAP servers, and compromised e.g. > AuthUserFile). > > Thoughts? Haven't thought this through but from a first glance it makes sense that the next provider can continue if the first one had a fatal error. Regards Rüdiger
