Re: re-do of proxy request body handling - ready for review
Justin Erenkrantz wrote: I don't see any way to implement that cleanly and without lots of undue complexity. Many dragons lay in that direction. When I put together the initial framework of mod_cache, solving this problem was one of my goals. How do we know when another worker has already started to fetch a page? Because there is an (incomplete) entry in the cache. How do we even know if the response is even cacheable at all? RFC2616. How do we know when the content is completed? Because of a flag in the cache entry telling us. For example, if the response is chunked, there is no way to know what the final length is ahead of time. We have no need to know. The "in progress" cache flag is only going to be marked as "complete" when the request is complete. If that request was chunked, streamed, whatever makes no difference. If we're still waiting for the initial response (i.e. request has already been issued but no data received back yet), then we don't know if the origin server will tack on a Cache-Control: no-store or Vary or there is some other server-driven reason that it won't be cached or acceptable to this client. As the cache was designed to cache multiple variants of the same URL, Vary should not be a problem. If we are still waiting for the initial response, then we have no cache object yet - the race condition is still there, but a few orders of magnitude shorter in duration. Additionally, with this strategy, if the first client to request a page is on a slow link, then other clients who are on faster links will be stalled while the cached content is stored and then served. If this is happening now then it's a design flaw in mod_cache. Cache should fill as fast as the sender will go, and the client should be able to read as slow as it likes. This is important to ensure backend servers are not left hanging around waiting for slow frontend clients. The downside of stalling in the hope that we'll be able to actually serve from our cache because another process has made the same request seems much worse to me than our current approach. We could end up making the client wait an indefinite amount of time for little advantage. There have been bugs outstanding in mod_proxy v1.3 complaining about this issue - the advantage to fixing this is real. The downside of the current approach is that we introduce no performance penalty to the users at the expense of additional bandwidth towards the origin server: we essentially act as if there was no cache present at all. But we introduce a performance penalty to the backend server, which must now handle load spikes from clients. This problem can (and has been reported in the past to) have a significant impact on big sites. I would rather focus on getting mod_cache reliable than rewriting it all over again to minimize a relatively rare issue. If it's that much of a problem, many pre-caching/priming strategies are also available. -- justin Nobody is expecting a rewrite of the cache, and this issue is definitely not rare. I'll start looking at this when I finished getting the LDAP stuff done. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: mod_status and Req
On Wed, Feb 02, 2005 at 11:11:15PM -0500, Stas Bekman wrote: > thanks Jim. and 3 +1s were added already. if it makes into 2.0.53 that > would be great. If no one has beaten me to it, I plan to merge all changes with 3 +1s in STATUS on Friday morning before I lay down the 2.0.53 tag. -- justin
Re: mod_status and Req
Jim Jagielski wrote: Stas Bekman wrote: Jim Jagielski wrote: Justin Erenkrantz wrote: --On Wednesday, February 2, 2005 1:43 PM -0500 Jim Jagielski <[EMAIL PROTECTED]> wrote: Any reason why we don't enable reporting of Req? I have a 2.1 patch ready to go, but I don't know why we don't do this I have no earthly idea what you are talking about. =) Can you please provide some more details? -- justin Already committed (love CTR). Basically, the 'Req' value displayed by mod_status is always 0 since we never update the required worker fields that note start and stop processing time. Whoah! wait long enough and the work will be done by someone else :) I was just about to get this done :) Jim, what would it take to backport it to 2.0? I see that 149550 changes ap_time_process_request API, but it's an internal API, no? Piece o' cake. I'll add it to 2.0's STATUS. Yeah, the ap_time_process_request API changed (for the better, IMO, since the worker "info" should be opaque(sp?)), but it's just internal. At least, we don't declare it as public :) thanks Jim. and 3 +1s were added already. if it makes into 2.0.53 that would be great. -- __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_status and Req
Stas Bekman wrote: > > Jim Jagielski wrote: > > Justin Erenkrantz wrote: > > > >>--On Wednesday, February 2, 2005 1:43 PM -0500 Jim Jagielski > >><[EMAIL PROTECTED]> wrote: > >> > >> > >>>Any reason why we don't enable reporting of Req? I have > >>>a 2.1 patch ready to go, but I don't know why we don't > >>>do this > >> > >>I have no earthly idea what you are talking about. =) > >> > >>Can you please provide some more details? -- justin > > > Already committed (love CTR). Basically, the 'Req' value displayed by > > mod_status is always 0 since we never update the required worker fields > > that note start and stop processing time. > > Whoah! wait long enough and the work will be done by someone else :) I was > just about to get this done :) > > Jim, what would it take to backport it to 2.0? I see that 149550 changes > ap_time_process_request API, but it's an internal API, no? > Piece o' cake. I'll add it to 2.0's STATUS. Yeah, the ap_time_process_request API changed (for the better, IMO, since the worker "info" should be opaque(sp?)), but it's just internal. At least, we don't declare it as public :) -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "There 10 types of people: those who read binary and everyone else."
2.1: release showstopper: * the edge connection filter cannot be removed
httpd-2.1/STATUS: RELEASE SHOWSTOPPERS: * the edge connection filter cannot be removed http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=105366252619530&w=2 jerenkrantz asks: Why should this block a release? because it requires a rewrite of the filters stack implementation (you have suggested that) and once 2.2 is released you can't do that anymore. -- __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_status and Req
Jim Jagielski wrote: Justin Erenkrantz wrote: --On Wednesday, February 2, 2005 1:43 PM -0500 Jim Jagielski <[EMAIL PROTECTED]> wrote: Any reason why we don't enable reporting of Req? I have a 2.1 patch ready to go, but I don't know why we don't do this I have no earthly idea what you are talking about. =) Can you please provide some more details? -- justin Already committed (love CTR). Basically, the 'Req' value displayed by mod_status is always 0 since we never update the required worker fields that note start and stop processing time. Whoah! wait long enough and the work will be done by someone else :) I was just about to get this done :) Jim, what would it take to backport it to 2.0? I see that 149550 changes ap_time_process_request API, but it's an internal API, no? -- __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_status and Req
Justin Erenkrantz wrote: > > --On Wednesday, February 2, 2005 1:43 PM -0500 Jim Jagielski > <[EMAIL PROTECTED]> wrote: > > > Any reason why we don't enable reporting of Req? I have > > a 2.1 patch ready to go, but I don't know why we don't > > do this > > I have no earthly idea what you are talking about. =) > > Can you please provide some more details? -- justin > Already committed (love CTR). Basically, the 'Req' value displayed by mod_status is always 0 since we never update the required worker fields that note start and stop processing time. -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "There 10 types of people: those who read binary and everyone else."
Re: re-do of proxy request body handling - ready for review
--On Wednesday, February 2, 2005 8:40 AM -0500 Jeff Trawick <[EMAIL PROTECTED]> wrote: Please review the proxy-reqbody branch for proposed improvements to 2.1-dev. There is a 2.0.x equivalent of the patch at http://httpd.apache.org/~trawick/20proxyreqbody.txt. To help people review Jeff's changes: svn diff -r 124193:HEAD http://svn.apache.org/repos/asf/httpd/httpd/branches/proxy-reqbody/modules/proxy/proxy_http.c Anyhow, only one really minor nit: in spool_reqbody_cl, I think it'd be good to add a comment that we're intentionally leaving the first 16k (or so) of content in our memory buffer. That took me a little bit to catch on to. +1 to merge into 2.1. Looks good. =) -- justin
Re: svn commit: r149548 - httpd/httpd/branches/2.0.x/STATUS
At 03:28 PM 2/2/2005, Jeff Trawick wrote: >On Wed, 02 Feb 2005 19:17:03 -, [EMAIL PROTECTED] ><[EMAIL PROTECTED]> wrote: >> Author: stoddard >> Date: Wed Feb 2 11:17:02 2005 >> New Revision: 149548 >> >> URL: http://svn.apache.org/viewcvs?view=rev&rev=149548 >> Log: >> yeow, Bill why didn't you propose this earlier? > >*that's* the one I was telling you and Allan about a couple of weeks ago ;) Actually, I have an answer to Bill's question :) The httpd-2.0 flavor of the mpm worker threads doesn't correspond, it seems that httpd-2.1 has enjoyed some significant refactoring and cleanup, already.
Re: re-do of proxy request body handling - ready for review
--On Wednesday, February 2, 2005 11:38 PM +0200 Graham Leggett <[EMAIL PROTECTED]> wrote: If mod_cache was taught to serve a being-cached URL directly from the cache (shadowing the real download), there would be no need for parallel connections to the backend server while the file is being cached, and no load spike. I don't see any way to implement that cleanly and without lots of undue complexity. Many dragons lay in that direction. How do we know when another worker has already started to fetch a page? How do we even know if the response is even cacheable at all? How do we know when the content is completed? For example, if the response is chunked, there is no way to know what the final length is ahead of time. If we're still waiting for the initial response (i.e. request has already been issued but no data received back yet), then we don't know if the origin server will tack on a Cache-Control: no-store or Vary or there is some other server-driven reason that it won't be cached or acceptable to this client. Additionally, with this strategy, if the first client to request a page is on a slow link, then other clients who are on faster links will be stalled while the cached content is stored and then served. The downside of stalling in the hope that we'll be able to actually serve from our cache because another process has made the same request seems much worse to me than our current approach. We could end up making the client wait an indefinite amount of time for little advantage. The downside of the current approach is that we introduce no performance penalty to the users at the expense of additional bandwidth towards the origin server: we essentially act as if there was no cache present at all. I'm also unsure that this strategy would mesh well with mod_disk_cache. I think an entirely new and different provider would have to be written (assuming we could surmount the above challenges, which I believe are much harder than they look). mod_disk_cache deliberately doesn't use shared memory because it introduces unnecessary complexity to the code. mod_disk_cache also delays any indication that it has started to fetch the page until content has been received. In fact, the way mod_disk_cache works right now is we have an acceptable race condition in that the last one to finish will store the data and overwrite all the instances that came before. I would rather focus on getting mod_cache reliable than rewriting it all over again to minimize a relatively rare issue. If it's that much of a problem, many pre-caching/priming strategies are also available. -- justin
RE: re-do of proxy request body handling - ready for review
-Original Message- From: Graham Leggett [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 02, 2005 10:39 PM To: dev@httpd.apache.org Subject: Re: re-do of proxy request body handling - ready for review Justin Erenkrantz wrote: > I don't understand the purpose of serving incomplete files from a cache. > Can you please elaborate on what you think mod_cache should do? -- > justin Since the early days of mod_proxy, there has been a race condition in the caching code which has been carried over to mod_cache. After an URL has been invalidated, and before the new version of that URL has been downloaded completely from the backend server, any attempts by other clients to fetch the same URL are passed to the backend server directly. This results in the load spike on the backend server briefly while the replacement cache file is downloaded. If mod_cache was taught to serve a being-cached URL directly from the cache (shadowing the real download), there would be no need for parallel connections to the backend server while the file is being cached, and no load spike. Regards, Graham --
Re: re-do of proxy request body handling - ready for review
Justin Erenkrantz wrote: I don't understand the purpose of serving incomplete files from a cache. Can you please elaborate on what you think mod_cache should do? -- justin Since the early days of mod_proxy, there has been a race condition in the caching code which has been carried over to mod_cache. After an URL has been invalidated, and before the new version of that URL has been downloaded completely from the backend server, any attempts by other clients to fetch the same URL are passed to the backend server directly. This results in the load spike on the backend server briefly while the replacement cache file is downloaded. If mod_cache was taught to serve a being-cached URL directly from the cache (shadowing the real download), there would be no need for parallel connections to the backend server while the file is being cached, and no load spike. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: svn commit: r149548 - httpd/httpd/branches/2.0.x/STATUS
On Wed, 02 Feb 2005 19:17:03 -, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > Author: stoddard > Date: Wed Feb 2 11:17:02 2005 > New Revision: 149548 > > URL: http://svn.apache.org/viewcvs?view=rev&rev=149548 > Log: > yeow, Bill why didn't you propose this earlier? *that's* the one I was telling you and Allan about a couple of weeks ago ;)
Re: [PATCH] get a pointer to the raw cert from mod_ssl
William A. Rowe, Jr. wrote:
At 04:17 AM 2/2/2005, David Reid wrote:
Basically this allows us to gain access to the actual cert structure.
Agreed that raw cert isn't that useful, and somewhat frightens
me in the environment table. The PEM or DER formats would be
generally useful. Unpacking the extended X509 attributes
might be even more useful.
Bill
This is the patch that provides me with the functionality I need. It's
generalised to a high degree and provides an easy way to get access to
extension data. It keeps the details hidden within mod_ssl where they
belong.
david
Index: ssl_private.h
===
--- ssl_private.h (revision 123890)
+++ ssl_private.h (working copy)
@@ -633,6 +633,7 @@
/* Variables */
void ssl_var_register(void);
char*ssl_var_lookup(apr_pool_t *, server_rec *, conn_rec *,
request_rec *, char *);
+char*ssl_ext_lookup(apr_pool_t *, conn_rec *, char *, char *);
void ssl_var_log_config_register(apr_pool_t *p);
#define APR_SHM_MAXSIZE (64 * 1024 * 1024)
Index: ssl_engine_vars.c
===
--- ssl_engine_vars.c (revision 123890)
+++ ssl_engine_vars.c (working copy)
@@ -60,6 +60,7 @@
{
APR_REGISTER_OPTIONAL_FN(ssl_is_https);
APR_REGISTER_OPTIONAL_FN(ssl_var_lookup);
+APR_REGISTER_OPTIONAL_FN(ssl_ext_lookup);
return;
}
@@ -654,6 +655,68 @@
return result;
}
+/* This function must remain safe to use for a non-SSL connection.
+ * This function is essentially the same as ssl_var_lookup but it looks
ONLY
+ * in the X509_EXTENSION structures. It looks for a matching OID and
(optionally)
+ * a value to match.
+ */
+char *ssl_ext_lookup(apr_pool_t *p, conn_rec *c, char *oid, char *var)
+{
+SSLConnRec *sslconn = myConnConfig(c);
+char *result = NULL;
+SSL *ssl = sslconn->ssl;
+X509 *xs = NULL;
+int count = 0, j;
+size_t oidlen, varlen = 0;
+
+if (!oid)
+return result;
+
+oidlen = strlen(oid);
+if (var)
+varlen = strlen(var);
+
+/*
+ * When no pool is given try to find one
+ */
+if (p == NULL) {
+if (c != NULL)
+p = c->pool;
+}
+if (!p)
+return result;
+
+/* We ONLY operate on peer certificates */
+if (! (xs = SSL_get_peer_certificate(ssl)))
+return result;
+
+if ((count = X509_get_ext_count(xs)) == 0)
+return result;
+
+for (j = 0; j < count; j++) {
+X509_EXTENSION *ext = X509_get_ext(xs, j);
+char *this_oid = alloca(oidlen + 1);
+if (OBJ_obj2txt(this_oid, oidlen + 1, ext->object, 1) == oidlen &&
+strncmp(this_oid, oid, oidlen) == 0) {
+/* We have found a matching OID.
+ * If we have been asked to match a value, then we check it
here,
+ * if we've just been asked to find an instance of the OID
+ * then we simply return the value.
+ */
+char *str = ASN1_STRING_data((ASN1_STRING *)ext->value);
+if (var) {
+if (strncmp(str, var, varlen) == 0)
+result = apr_pstrdup(p, str);
+} else
+result = apr_pstrdup(p, str);
+
+if (result)
+break;
+}
+}
+return result;
+}
+
/* _
**
** SSL Extension to mod_log_config
Index: mod_ssl.h
===
--- mod_ssl.h (revision 123890)
+++ mod_ssl.h (working copy)
@@ -26,6 +26,11 @@
conn_rec *, request_rec *,
char *));
+/* The ssl_ext_lookup() optional function retrieves SSL environment
+ * variables. */
+APR_DECLARE_OPTIONAL_FN(char *, ssl_ext_lookup,
+(apr_pool_t *, conn_rec *, char *, char *));
+
/* An optional function which returns non-zero if the given connection
* is using SSL/TLS. */
APR_DECLARE_OPTIONAL_FN(int, ssl_is_https, (conn_rec *));
Re: re-do of proxy request body handling - ready for review
--On Wednesday, February 2, 2005 8:49 PM +0200 Graham Leggett <[EMAIL PROTECTED]> wrote: Mod_cache already supports the concept of spooling files to disk (or memory, or shared memory), and can be taught how to serve an incompletely downloaded file to other clients (apparently it cannot at the moment...?). I don't understand the purpose of serving incomplete files from a cache. Can you please elaborate on what you think mod_cache should do? -- justin
Re: mod_status and Req
--On Wednesday, February 2, 2005 1:43 PM -0500 Jim Jagielski <[EMAIL PROTECTED]> wrote: Any reason why we don't enable reporting of Req? I have a 2.1 patch ready to go, but I don't know why we don't do this I have no earthly idea what you are talking about. =) Can you please provide some more details? -- justin
Re: LDAPTrustedMode has the wrong scope...
You read my mind. I'm all over it. :) Brad >>> Graham Leggett <[EMAIL PROTECTED]> Wednesday, February 02, 2005 12:13:56 PM >>> Brad Nicholes wrote: >I have got something that almost works now. The problem that I am > running into is that util_ldap_connection_find() doesn't know the > difference between APR_LDAP_NONE(the secure mode was never set so use > the default) vs. APR_LDAP_NONE (NONE is what I really want). Seems like > we need another flag meaning UNSET. Hmmm... maybe if we teach util_ldap_connection_find() to accept "-1", which means "none of NONE|SSL|TLS"...? The -1 can be passed if AuthLDAPURL had one parameter, and NONE|SSL|TLS can be passed if AuthLDAPURL has two parameters? Regards, Graham --
Re: LDAPTrustedMode has the wrong scope...
Brad Nicholes wrote: I have got something that almost works now. The problem that I am running into is that util_ldap_connection_find() doesn't know the difference between APR_LDAP_NONE(the secure mode was never set so use the default) vs. APR_LDAP_NONE (NONE is what I really want). Seems like we need another flag meaning UNSET. Hmmm... maybe if we teach util_ldap_connection_find() to accept "-1", which means "none of NONE|SSL|TLS"...? The -1 can be passed if AuthLDAPURL had one parameter, and NONE|SSL|TLS can be passed if AuthLDAPURL has two parameters? Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: LDAPTrustedMode has the wrong scope...
I have got something that almost works now. The problem that I am running into is that util_ldap_connection_find() doesn't know the difference between APR_LDAP_NONE(the secure mode was never set so use the default) vs. APR_LDAP_NONE (NONE is what I really want). Seems like we need another flag meaning UNSET. Brad >>> [EMAIL PROTECTED] Wednesday, February 02, 2005 9:38:59 AM >>> Brad Nicholes wrote: > +1, allowing mod_authnz_ldap to override the default makes a lot more > sense. Unless you are already working on a patch, I will try to put > something together today. But after today I will be offline for the > next two days. I will only have a chance to work on this tomorrow :( Regards, Graham --
Re: Multi-threaded proxy? was Re: re-do of proxy request bodyhandling - ready for review
On Wed, Feb 02, 2005 at 08:45:53PM +0200, Graham Leggett wrote: > Peter J. Cranstone wrote: > > >Who is trying to serve up 2GB files? > > Redhat's DVD images (just one example). Ahhh, and I was just about to pipe-up :) But for the record, ftp.heanet.ie currently servers 23 distinct > 2Gb files an average of about 800 times daily, we only have 3 > 4Gb files, and these are very rarely downloaded, a few times a week each maybe. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: re-do of proxy request body handling - ready for review
Ronald Park wrote: In this picture: C <--> A <--> P C, the client, makes a request to A which proxies it to P. A & P can do their exchange independant of C & A. If A-P can be done fast, but C-A is slow, then P's wasting resource, no? If you configured mod_cache, my understanding is that P would serve to A as much as A was willing to handle, while A could serve C as slowly as it liked, freeing up P. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: re-do of proxy request body handling - ready for review
Ronald Park wrote: I was recently considering a similar patch for mod_proxy along the lines of spool_reqbody_cl() method but it would go one step further: spawning off a thread to asynchronously read the request into a temp file (or files) while the initial thread would continue to stream the io_bufsize chunks down the filter chain. This would 'untie' the original client and the proxy server in cases where they ran at different speeds (more a problem for *large* proxy files where one side or the other could be tied up waiting for the slower side for long periods of time... and poor Apache caught in the middle). I hadn't gotten too far along with my patch but with this, it's about 90% of the way done. :) This is a job for mod_cache, not mod_proxy. Mod_cache already supports the concept of spooling files to disk (or memory, or shared memory), and can be taught how to serve an incompletely downloaded file to other clients (apparently it cannot at the moment...?). Adding this capability to proxy is making the proxy code unnecessarily complex, and leaves this real problem unsolved for other parts of httpd, like CGIs. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Multi-threaded proxy? was Re: re-do of proxy request bodyhandling - ready for review
Peter J. Cranstone wrote: Who is trying to serve up 2GB files? Redhat's DVD images (just one example). Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: re-do of proxy request body handling - ready for review
True, the client and the proxying server are tied up intrinsically. I used the wrong wording to name who would benefit. I meant that the server providing the proxied content (the 'other' server; the one not directly talking to the client) could be done and on it's way doing other work while the first two finish their dance. In this picture: C <--> A <--> P C, the client, makes a request to A which proxies it to P. A & P can do their exchange independant of C & A. If A-P can be done fast, but C-A is slow, then P's wasting resource, no? Ron On Wed, 2005-02-02 at 13:26 -0500, Jeff Trawick wrote: > On Wed, 02 Feb 2005 12:32:39 -0500, Ronald Park <[EMAIL PROTECTED]> wrote: > > I was recently considering a similar patch for mod_proxy along the lines > > of spool_reqbody_cl() method but it would go one step further: spawning > > off a thread to asynchronously read the request into a temp file (or > > files) while the initial thread would continue to stream the io_bufsize > > chunks down the filter chain. > > stray thoughts... > > one thread per proxy request seems pretty heavy weight... perhaps > cool for small number of clients, perhaps a gratuitous use of > resources on a busy server... > > and if threads are shared, then generic event handling apparatus to be > used for other non-proxy code seems more appropriate > > ... > > >This would 'untie' the original client > >and the proxy server in cases where they ran at different speeds (more > >a problem for *large* proxy files where one side or the other could be > >tied up waiting for the slower side for long periods of time... and > >poor Apache caught in the middle). > > the client and proxy server are necessarily tied up until origin > server reads the request and writes the response... why do we care if > they are tied up writing the request out vs. waiting for the response? > it seems like we could burn a lot of resources but get only marginal > response time improvement as payback -- Ronald Park <[EMAIL PROTECTED]>
mod_status and Req
Any reason why we don't enable reporting of Req? I have a 2.1 patch ready to go, but I don't know why we don't do this -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ "There 10 types of people: those who read binary and everyone else."
RE: Multi-threaded proxy? was Re: re-do of proxy request bodyhandling - ready for review
Ron, Who is trying to serve up 2GB files? Peter J. Cranstone -Original Message- From: Ronald Park [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 02, 2005 11:24 AM To: dev@httpd.apache.org Subject: Re: Multi-threaded proxy? was Re: re-do of proxy request bodyhandling - ready for review Imagine, just as a wild theoretical scenario (:D), that you have the following setup: Apache -> (proxy) -> Squid -> (cache miss) -> Apache -> (docroot) Where the back-end Apache serves up large files (in the 2G range) (and, yes, there are far more files that can be effectively cached). Now imagine you have thousands of clients trying to get those files some of which have very slow connections. And also imagine that their are more front-end Apache instances than back-ends. The backend Apache could quickly delivery the file through to the frontend Apache's mod_proxy if it wasn't held up by waiting for each chunk to be spoonfed over to the slow client. Even for relatively good clients, it's likely a number of them are going to tie up a thread in the back-end for longer than it would if the front-end gobbled up the proxy response faster. The problem with the 'gobble up the whole proxy response' all at once though is that for these huge files, the original client might not get any response for a noticable amount of time. Further, if an impatient client, it might give up and reissue the request again, tying up another set of threads (and internal bandwidth). :( Ron On Wed, 2005-02-02 at 18:51 +0100, Mladen Turk wrote: > Paul Querna wrote: > > > > One thought I have been tossing around for a long time is tying the > > proxy code into the Event MPM. Instead of a thread blocking while it > > waits for a backend reply, the backend server's FD would be added to the > > Event Thread, and then when the reply is ready, any available worker > > thread would handle it, like they do new requests. > > > > This would work well for backend servers that might take a second or two > > for a reply, but it does add at least 3 context switches. (in some use > > cases this would work great, in others, it would hurt performance...) > > > > I don't think it would give any benefit. Well perhaps only on > forward proxies it could spare some keep-alive connections. > > Regards, > Mladen. -- Ronald Park <[EMAIL PROTECTED]>
Re: re-do of proxy request body handling - ready for review
On Wed, 02 Feb 2005 12:32:39 -0500, Ronald Park <[EMAIL PROTECTED]> wrote: > I was recently considering a similar patch for mod_proxy along the lines > of spool_reqbody_cl() method but it would go one step further: spawning > off a thread to asynchronously read the request into a temp file (or > files) while the initial thread would continue to stream the io_bufsize > chunks down the filter chain. stray thoughts... one thread per proxy request seems pretty heavy weight... perhaps cool for small number of clients, perhaps a gratuitous use of resources on a busy server... and if threads are shared, then generic event handling apparatus to be used for other non-proxy code seems more appropriate ... >This would 'untie' the original client >and the proxy server in cases where they ran at different speeds (more >a problem for *large* proxy files where one side or the other could be >tied up waiting for the slower side for long periods of time... and >poor Apache caught in the middle). the client and proxy server are necessarily tied up until origin server reads the request and writes the response... why do we care if they are tied up writing the request out vs. waiting for the response? it seems like we could burn a lot of resources but get only marginal response time improvement as payback
Re: Multi-threaded proxy? was Re: re-do of proxy request body handling - ready for review
Imagine, just as a wild theoretical scenario (:D), that you have the following setup: Apache -> (proxy) -> Squid -> (cache miss) -> Apache -> (docroot) Where the back-end Apache serves up large files (in the 2G range) (and, yes, there are far more files that can be effectively cached). Now imagine you have thousands of clients trying to get those files some of which have very slow connections. And also imagine that their are more front-end Apache instances than back-ends. The backend Apache could quickly delivery the file through to the frontend Apache's mod_proxy if it wasn't held up by waiting for each chunk to be spoonfed over to the slow client. Even for relatively good clients, it's likely a number of them are going to tie up a thread in the back-end for longer than it would if the front-end gobbled up the proxy response faster. The problem with the 'gobble up the whole proxy response' all at once though is that for these huge files, the original client might not get any response for a noticable amount of time. Further, if an impatient client, it might give up and reissue the request again, tying up another set of threads (and internal bandwidth). :( Ron On Wed, 2005-02-02 at 18:51 +0100, Mladen Turk wrote: > Paul Querna wrote: > > > > One thought I have been tossing around for a long time is tying the > > proxy code into the Event MPM. Instead of a thread blocking while it > > waits for a backend reply, the backend server's FD would be added to the > > Event Thread, and then when the reply is ready, any available worker > > thread would handle it, like they do new requests. > > > > This would work well for backend servers that might take a second or two > > for a reply, but it does add at least 3 context switches. (in some use > > cases this would work great, in others, it would hurt performance...) > > > > I don't think it would give any benefit. Well perhaps only on > forward proxies it could spare some keep-alive connections. > > Regards, > Mladen. -- Ronald Park <[EMAIL PROTECTED]>
Re: Multi-threaded proxy? was Re: re-do of proxy request body handling - ready for review
Paul Querna wrote: One thought I have been tossing around for a long time is tying the proxy code into the Event MPM. Instead of a thread blocking while it waits for a backend reply, the backend server's FD would be added to the Event Thread, and then when the reply is ready, any available worker thread would handle it, like they do new requests. This would work well for backend servers that might take a second or two for a reply, but it does add at least 3 context switches. (in some use cases this would work great, in others, it would hurt performance...) I don't think it would give any benefit. Well perhaps only on forward proxies it could spare some keep-alive connections. Regards, Mladen.
Re: Multi-threaded proxy? was Re: re-do of proxy request body handling - ready for review
Hmm, don't know enough about Event MPM to comment on that part of the message, but with regards to the performance for small requests, in my original 'design' for this, I figured it would do one synchronous read first, pass that through the filter chain and, if '(!seen_eos)', then it would pay the cost to set up the asynchronous details. Hopefully then most sites could alleviate performance issues by fiddling with the ProxyIOBufferSize directive to fit all requests within this value while large requests would be many multiples of this size. Ron On Wed, 2005-02-02 at 09:37 -0800, Paul Querna wrote: > Ronald Park wrote: > > I was recently considering a similar patch for mod_proxy along the lines > > of spool_reqbody_cl() method but it would go one step further: spawning > > off a thread to asynchronously read the request into a temp file (or > > files) while the initial thread would continue to stream the io_bufsize > > chunks down the filter chain. This would 'untie' the original client > > and the proxy server in cases where they ran at different speeds (more > > a problem for *large* proxy files where one side or the other could be > > tied up waiting for the slower side for long periods of time... and > > poor Apache caught in the middle). > > > > One thought I have been tossing around for a long time is tying the > proxy code into the Event MPM. Instead of a thread blocking while it > waits for a backend reply, the backend server's FD would be added to the > Event Thread, and then when the reply is ready, any available worker > thread would handle it, like they do new requests. > > This would work well for backend servers that might take a second or two > for a reply, but it does add at least 3 context switches. (in some > use cases this would work great, in others, it would hurt performance...) > > -Paul -- Ronald Park <[EMAIL PROTECTED]>
Multi-threaded proxy? was Re: re-do of proxy request body handling - ready for review
Ronald Park wrote: I was recently considering a similar patch for mod_proxy along the lines of spool_reqbody_cl() method but it would go one step further: spawning off a thread to asynchronously read the request into a temp file (or files) while the initial thread would continue to stream the io_bufsize chunks down the filter chain. This would 'untie' the original client and the proxy server in cases where they ran at different speeds (more a problem for *large* proxy files where one side or the other could be tied up waiting for the slower side for long periods of time... and poor Apache caught in the middle). One thought I have been tossing around for a long time is tying the proxy code into the Event MPM. Instead of a thread blocking while it waits for a backend reply, the backend server's FD would be added to the Event Thread, and then when the reply is ready, any available worker thread would handle it, like they do new requests. This would work well for backend servers that might take a second or two for a reply, but it does add at least 3 context switches. (in some use cases this would work great, in others, it would hurt performance...) -Paul
Re: re-do of proxy request body handling - ready for review
I was recently considering a similar patch for mod_proxy along the lines of spool_reqbody_cl() method but it would go one step further: spawning off a thread to asynchronously read the request into a temp file (or files) while the initial thread would continue to stream the io_bufsize chunks down the filter chain. This would 'untie' the original client and the proxy server in cases where they ran at different speeds (more a problem for *large* proxy files where one side or the other could be tied up waiting for the slower side for long periods of time... and poor Apache caught in the middle). I hadn't gotten too far along with my patch but with this, it's about 90% of the way done. :) Ron On Wed, 2005-02-02 at 09:56 -0500, Greg Ames wrote: > Jeff Trawick wrote: > > Please review the proxy-reqbody branch for proposed improvements to > > 2.1-dev. There is a 2.0.x equivalent of the patch at > > http://httpd.apache.org/~trawick/20proxyreqbody.txt. > > +1 (reviewed, not tested) > > certainly an improvement over what we have today. The brains (decisions > about > what to do with a body) are nicely isolated to a few lines of code in > send_request_body(). The rest of it is the muscle, and that shouldn't be > controversial...either it works or it doesn't, and I can't spot any problems. > > Greg > -- Ronald Park <[EMAIL PROTECTED]>
Re: [PATCH] set username from certificates at a more appropriate time
Geoffrey Young wrote: Joe Orton wrote: I presume this fixes #31418? Your patch makes sense to me. I could argue that it could even be done *before* the SSLRequire checking, such that the "username" is logged appropriately even if an SSLRequire triggers a 403, but I doubt that matters much. fwiw that's the route that the auth providers took when we last looked at this - make it possible to log the user, with the understanding that the user might not have passed the auth process so may be completely bogus. As Joe pointed out in a separate reply this is now done after the renegotiation, so it's a sensible place to do it and avoids one possible gotcha. david
Re: LDAPTrustedMode has the wrong scope...
Brad Nicholes wrote: +1, allowing mod_authnz_ldap to override the default makes a lot more sense. Unless you are already working on a patch, I will try to put something together today. But after today I will be offline for the next two days. I will only have a chance to work on this tomorrow :( Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: LDAPTrustedMode has the wrong scope...
+1, allowing mod_authnz_ldap to override the default makes a lot more sense. Unless you are already working on a patch, I will try to put something together today. But after today I will be offline for the next two days. Brad >>> [EMAIL PROTECTED] Wednesday, February 02, 2005 1:23:51 AM >>> Brad Nicholes said: > The attached patches convert LDAPTrustedMode into a per-directory > directive rather than a per-server. This allows the configuration to > specify which mode should be applied for the associated AuthLDAPURL. > > Thoughts on whether this should be the way to go or if LDAPTrustedMode > should be moved up into mod_authnz_ldap as AuthLDAPTrustedMode? Thinking about this some more - I'm not keen on the idea of adding another directive to mod_authnz_ldap, because when the configuration-via-LDAP happens, and other potential LDAP modules happen, then we end up with directive soup as each module has it's own variation of AuthLDAPTrustedMode. The idea that it be possible to set the SSL mode inside mod_authnz_ldap is still really useful - what we could do is this: AuthLDAPURL ldap:// AuthLDAPURL ldaps:// AuthLDAPURL SSL ldap:// AuthLDAPURL STARTTLS ldap:// In other words, combine the SSL mode in a TAKE12 option, with an optional SSL mode prefix. The LDAPTrustedMode directive could stay so as to define the default (and overriding the default if the user wanted). Thoughts? Regards, Graham --
Re: [PATCH] get a pointer to the raw cert from mod_ssl
At 04:17 AM 2/2/2005, David Reid wrote: >Basically this allows us to gain access to the actual cert structure. Agreed that raw cert isn't that useful, and somewhat frightens me in the environment table. The PEM or DER formats would be generally useful. Unpacking the extended X509 attributes might be even more useful. Bill
[PATCH] Re: mod_proxy and friends, FIX_15207
Does not include the required removal of the unused FIX_15207
code snippets...
Index: modules/proxy/proxy_ajp.c
===
--- modules/proxy/proxy_ajp.c (revision 149512)
+++ modules/proxy/proxy_ajp.c (working copy)
@@ -79,7 +79,7 @@
search = r->args;
/* process path */
-path = ap_proxy_canonenc(r->pool, url, strlen(url), enc_path,
r->proxyreq);
+path = ap_proxy_canonenc(r->pool, url, strlen(url), enc_path, 0,
r->proxyreq);
if (path == NULL)
return HTTP_BAD_REQUEST;
Index: modules/proxy/proxy_ftp.c
===
--- modules/proxy/proxy_ftp.c (revision 149512)
+++ modules/proxy/proxy_ftp.c (working copy)
@@ -166,7 +166,7 @@
strp = strchr(url, ';');
if (strp != NULL) {
*(strp++) = '\0';
-parms = ap_proxy_canonenc(p, strp, strlen(strp), enc_parm,
+parms = ap_proxy_canonenc(p, strp, strlen(strp), enc_parm, 0,
r->proxyreq);
if (parms == NULL)
return HTTP_BAD_REQUEST;
@@ -174,7 +174,7 @@
else
parms = "";
-path = ap_proxy_canonenc(p, url, strlen(url), enc_path,
r->proxyreq);
+path = ap_proxy_canonenc(p, url, strlen(url), enc_path, 0,
r->proxyreq);
if (path == NULL)
return HTTP_BAD_REQUEST;
if (!ftp_check_string(path))
@@ -182,13 +182,13 @@
if (r->proxyreq && r->args != NULL) {
if (strp != NULL) {
-strp = ap_proxy_canonenc(p, r->args, strlen(r->args),
enc_parm, 1);
+strp = ap_proxy_canonenc(p, r->args, strlen(r->args),
enc_parm, 1, r->proxyreq);
if (strp == NULL)
return HTTP_BAD_REQUEST;
parms = apr_pstrcat(p, parms, "?", strp, NULL);
}
else {
-strp = ap_proxy_canonenc(p, r->args, strlen(r->args),
enc_fpath, 1);
+strp = ap_proxy_canonenc(p, r->args, strlen(r->args),
enc_fpath, 1, r->proxyreq);
if (strp == NULL)
return HTTP_BAD_REQUEST;
path = apr_pstrcat(p, path, "?", strp, NULL);
Index: modules/proxy/proxy_util.c
===
--- modules/proxy/proxy_util.c (revision 149512)
+++ modules/proxy/proxy_util.c (working copy)
@@ -130,10 +130,10 @@
* Convert a URL-encoded string to canonical form.
* It decodes characters which need not be encoded,
* and encodes those which must be encoded, and does not touch
- * those which must not be touched.
+ * those which must not be touched.
*/
PROXY_DECLARE(char *)ap_proxy_canonenc(apr_pool_t *p, const char *x,
int len, enum enctype t,
-int isenc)
+int forcedec, int proxyreq)
{
int i, j, ch;
char *y;
@@ -174,8 +174,11 @@
y[j] = ch;
continue;
}
-/* decode it if not already done */
-if (isenc && ch == '%') {
+/*
+ * decode it if not already done. do not decode reverse proxied URLs
+ * unless specifically forced
+ */
+if ((forcedec || (proxyreq && proxyreq != PROXYREQ_REVERSE)) && ch
== '%') {
if (!apr_isxdigit(x[i + 1]) || !apr_isxdigit(x[i + 2]))
return NULL;
ch = ap_proxy_hex2c(&x[i + 1]);
@@ -238,12 +241,12 @@
strp = strchr(user, ':');
if (strp != NULL) {
*strp = '\0';
-password = ap_proxy_canonenc(p, strp + 1, strlen(strp + 1),
enc_user, 1);
+password = ap_proxy_canonenc(p, strp + 1, strlen(strp + 1),
enc_user, 1, 0);
if (password == NULL)
return "Bad %-escape in URL (password)";
}
-user = ap_proxy_canonenc(p, user, strlen(user), enc_user, 1);
+user = ap_proxy_canonenc(p, user, strlen(user), enc_user, 1, 0);
if (user == NULL)
return "Bad %-escape in URL (username)";
}
Index: modules/proxy/proxy_http.c
===
--- modules/proxy/proxy_http.c (revision 149512)
+++ modules/proxy/proxy_http.c (working copy)
@@ -79,7 +79,7 @@
search = r->args;
/* process path */
-path = ap_proxy_canonenc(r->pool, url, strlen(url), enc_path,
r->proxyreq);
+path = ap_proxy_canonenc(r->pool, url, strlen(url), enc_path, 0,
r->proxyreq);
if (path == NULL)
return HTTP_BAD_REQUEST;
Index: modules/proxy/mod_proxy.c
===
--- modules/proxy/mod_proxy.c (revision 149512)
+++ modules/proxy/mod_proxy.c (working copy)
@@ -1,4 +1,3 @@
-#define FIX_15207
/* Copyright 1999-2004 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
Index: modules/proxy/mod_proxy.h
===
--- modules/proxy/mod_proxy.h (revision 149512)
+++ modules/proxy/mod_proxy.h (working copy)
@@ -381,7 +381,7 @@
PROXY_DECLARE(int) ap_proxy_hex2c(const char *x);
PROXY_DECLARE(void) ap_pro
Re: 2.1.3 mod_disk_cache partial responses
On Thu, Feb 03, 2005 at 08:44:03AM +1100, dean wrote: > Im not sure where to find out the revision, but I checked out the src > from cvs just before my first post. I just did another update and > nothing has changed. Hope Im using the right command: > cvs checkout -d httpd-2.1 httpd-2.0 Oh. In November, we switched to Subversion. You need to follow the instructions at the top of: http://httpd.apache.org/dev/ to checkout the source code. So, the CVS checkouts are roughly 4 months out-of-date. > I guess I'll have to wait for mod_proxy to mature bit more. Compliments > to the developers, it works great with the new mod_filter which has some > really cool potential Actually, I bet that the race conditions you are seeing in mod_proxy are now fixed. We had a very similar report here on [EMAIL PROTECTED] that was fixed in the 2.1.2 timeframe (which was after we converted to Subversion). Hope this helps. -- justin
Re: re-do of proxy request body handling - ready for review
Jeff Trawick wrote: Please review the proxy-reqbody branch for proposed improvements to 2.1-dev. There is a 2.0.x equivalent of the patch at http://httpd.apache.org/~trawick/20proxyreqbody.txt. +1 (reviewed, not tested) certainly an improvement over what we have today. The brains (decisions about what to do with a body) are nicely isolated to a few lines of code in send_request_body(). The rest of it is the muscle, and that shouldn't be controversial...either it works or it doesn't, and I can't spot any problems. Greg
Re: mod_proxy and friends, FIX_15207
Please note that I'm not 100% happy, since the below "hides"
the meeting of isenc, or at least changes it... The real
fix is to also pass r->proxyreq (to determine type of
proxy request) as well as isenc, which should be renamed
something like forceenc to force the encoding
no matter what type of proxy request. I can provide that
if there's interest...
On Feb 2, 2005, at 9:22 AM, Jim Jagielski wrote:
There are now tests for both PR 32459 and PR 15207 in httpd-test's
t/modules/proxy.t; the former fails at the moment in 2.1 and the
latter
should fail instead if you flip the silly #define.
sorry - the PR 32459 test is in t/modules/rewrite.t not proxy.t
The below results in corrected behavior... Passes httpd-test
and some testing that I've done here, but would like more
feedback before I commit...
Index: modules/proxy/proxy_util.c
===
--- modules/proxy/proxy_util.c (revision 149512)
+++ modules/proxy/proxy_util.c (working copy)
@@ -175,7 +175,7 @@
continue;
}
/* decode it if not already done */
-if (isenc && ch == '%') {
+if (isenc != PROXYREQ_REVERSE && ch == '%') {
if (!apr_isxdigit(x[i + 1]) || !apr_isxdigit(x[i + 2]))
return NULL;
ch = ap_proxy_hex2c(&x[i + 1]);
Index: modules/proxy/mod_proxy.c
===
--- modules/proxy/mod_proxy.c (revision 149512)
+++ modules/proxy/mod_proxy.c (working copy)
@@ -1,4 +1,3 @@
-#define FIX_15207
/* Copyright 1999-2004 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
--
===
Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/
"There 10 types of people: those who read binary and everyone else."
Re: mod_proxy and friends, FIX_15207
There are now tests for both PR 32459 and PR 15207 in httpd-test's
t/modules/proxy.t; the former fails at the moment in 2.1 and the
latter
should fail instead if you flip the silly #define.
sorry - the PR 32459 test is in t/modules/rewrite.t not proxy.t
The below results in corrected behavior... Passes httpd-test
and some testing that I've done here, but would like more
feedback before I commit...
Index: modules/proxy/proxy_util.c
===
--- modules/proxy/proxy_util.c (revision 149512)
+++ modules/proxy/proxy_util.c (working copy)
@@ -175,7 +175,7 @@
continue;
}
/* decode it if not already done */
-if (isenc && ch == '%') {
+if (isenc != PROXYREQ_REVERSE && ch == '%') {
if (!apr_isxdigit(x[i + 1]) || !apr_isxdigit(x[i + 2]))
return NULL;
ch = ap_proxy_hex2c(&x[i + 1]);
Index: modules/proxy/mod_proxy.c
===
--- modules/proxy/mod_proxy.c (revision 149512)
+++ modules/proxy/mod_proxy.c (working copy)
@@ -1,4 +1,3 @@
-#define FIX_15207
/* Copyright 1999-2004 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
Re: [PATCH] set username from certificates at a more appropriate time
Joe Orton wrote: > I presume this fixes #31418? Your patch makes sense to me. I could > argue that it could even be done *before* the SSLRequire checking, such > that the "username" is logged appropriately even if an SSLRequire > triggers a 403, but I doubt that matters much. fwiw that's the route that the auth providers took when we last looked at this - make it possible to log the user, with the understanding that the user might not have passed the auth process so may be completely bogus. --Geoff
re-do of proxy request body handling - ready for review
Please review the proxy-reqbody branch for proposed improvements to 2.1-dev. There is a 2.0.x equivalent of the patch at http://httpd.apache.org/~trawick/20proxyreqbody.txt.
Re: [PATCH] set username from certificates at a more appropriate time
On Wed, 2 Feb 2005, Jim Jagielski wrote: > +1 Major +1 - though be aware of the fallout - internal redirects can play nice heavoc. DW.
Re: [PATCH] set username from certificates at a more appropriate time
+1
Re: [PATCH] get a pointer to the raw cert from mod_ssl
On Wed, Feb 02, 2005 at 11:36:41AM +, Ben Laurie wrote: > Joe Orton wrote: > >On Wed, Feb 02, 2005 at 11:09:47AM +, David Reid wrote: > >>The issue is a need to get access to the internals of the structure. > > > > > >By exposing the X509 * directly you expose a dependency on the > >underlying SSL toolkit. What if mod_ssl was built to use the RSA > >toolkit; will the X509 * have the same fields and layout? That's why > >it's preferable to just expose the DER: there's nothing you can't do > >with the DER that you can do with the X509 * anyway. > > This is a somewhat frivolous point though. If you expose the DER, then > the module will have to have some crypto toolkit to understand it. If > its a different one from the one mod_ssl is using, then it can always > convert to DER and into its own format, using the same kit mod_ssl uses > for the first step. So, it seems to me that this is simply making work > for modules to no real gain. That all presumes that random third-party modules can determine which type of SSL toolkit mod_ssl was built with and where the headers are etc, which isn't the case. > Certainly it should only be exported in this format if OpenSSL is in use. > > BTW, I'd advocate using a hook that was typesafe! Exactly my point: you can't do that without exposing OpenSSL headers through mod_ssl.h and making the hook declaration conditional on SSL support being enabled and all that mess. joe
Re: [PATCH] get a pointer to the raw cert from mod_ssl
Joe Orton wrote: On Wed, Feb 02, 2005 at 11:09:47AM +, David Reid wrote: Joe Orton wrote: On Wed, Feb 02, 2005 at 10:17:04AM +, David Reid wrote: Basically this allows us to gain access to the actual cert structure. I don't like the idea of exposing the X509 * directly especially not through a char * interface. Exposing the DER representation (e.g. base64-encoded) through ssl_var_lookup would be better. (of course that's essentially what _CERT_PEM is; but exporting it without the unnecessary PEM trimmings is useful too) The issue is a need to get access to the internals of the structure. By exposing the X509 * directly you expose a dependency on the underlying SSL toolkit. What if mod_ssl was built to use the RSA toolkit; will the X509 * have the same fields and layout? That's why it's preferable to just expose the DER: there's nothing you can't do with the DER that you can do with the X509 * anyway. This is a somewhat frivolous point though. If you expose the DER, then the module will have to have some crypto toolkit to understand it. If its a different one from the one mod_ssl is using, then it can always convert to DER and into its own format, using the same kit mod_ssl uses for the first step. So, it seems to me that this is simply making work for modules to no real gain. Certainly it should only be exported in this format if OpenSSL is in use. BTW, I'd advocate using a hook that was typesafe! Cheers, Ben. -- http://www.apache-ssl.org/ben.html http://www.thebunker.net/ "There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit." - Robert Woodruff
Re: [PATCH] get a pointer to the raw cert from mod_ssl
On Wed, Feb 02, 2005 at 11:09:47AM +, David Reid wrote: > Joe Orton wrote: > >On Wed, Feb 02, 2005 at 10:17:04AM +, David Reid wrote: > > > >>Basically this allows us to gain access to the actual cert structure. > > > > > >I don't like the idea of exposing the X509 * directly especially not > >through a char * interface. Exposing the DER representation (e.g. > >base64-encoded) through ssl_var_lookup would be better. (of course that's essentially what _CERT_PEM is; but exporting it without the unnecessary PEM trimmings is useful too) > The issue is a need to get access to the internals of the structure. By exposing the X509 * directly you expose a dependency on the underlying SSL toolkit. What if mod_ssl was built to use the RSA toolkit; will the X509 * have the same fields and layout? That's why it's preferable to just expose the DER: there's nothing you can't do with the DER that you can do with the X509 * anyway. joe
Re: [PATCH] get a pointer to the raw cert from mod_ssl
Joe Orton wrote:
On Wed, Feb 02, 2005 at 10:17:04AM +, David Reid wrote:
Basically this allows us to gain access to the actual cert structure.
I don't like the idea of exposing the X509 * directly especially not
through a char * interface. Exposing the DER representation (e.g.
base64-encoded) through ssl_var_lookup would be better.
The issue is a need to get access to the internals of the structure.
The existing function doesn't meet the requirements, but I'll work up an
alternative patch to add a new function that will meet them.
david
joe
Index: ssl_engine_vars.c
===
--- ssl_engine_vars.c (revision 123890)
+++ ssl_engine_vars.c (working copy)
@@ -364,6 +364,10 @@
else if (strcEQ(var, "CERT")) {
result = ssl_var_lookup_ssl_cert_PEM(p, xs);
}
+else if (strcEQ(var, "RAW_CERT")) {
+result = (char *)xs;
+resdup = FALSE;
+}
if (result != NULL && resdup)
result = apr_pstrdup(p, result);
Re: [PATCH] set username from certificates at a more appropriate time
Joe Orton wrote: On Wed, Feb 02, 2005 at 10:07:56AM +, David Reid wrote: William A. Rowe, Jr. wrote: My only concern ... does this new scope pair up properly if the user cert has been renegotiated? If so +1 I'm unable to test that here, but maybe if someone has a system where renegotiation is testable... You moved the code to be directly after the per-dir-reneg logic so it should be fine. I thought that would be the case, but as I said, being unable to test it... david
Re: 2.1.3 mod_disk_cache partial responses
Justin Erenkrantz wrote: --On Wednesday, February 2, 2005 12:25 AM +1100 dean <[EMAIL PROTECTED]> wrote: My first thought was that Firefox has an issue. Then I installed 2.0.53 as a cache-proxy BUT I couldn't reproduce the above, Google logo loads everytime. Another simple page I tried is news.com.au/wireless. Steps 1 -3 are same 4. refresh page -> no css applied (liveheaders shows 200 OK), logo appears broken (liveheaders show 206 Partial Response) 5. delete temp int. files in IE6 6. enter www.news.com.au/wireless in browser bar -> logo loads fine but no css applied 7. hit refresh button -> page loads fine with css & logo As before, I couldn't reproduce this with 2.0.53 Can you try reproduce this behaviour with 2.1.3 ?? My hunch is that you are being hit by mod_proxy brokenness not mod_cache issues. 2.0.53 uses the 'stable' mod_proxy, while 2.1.3 has a rewritten mod_proxy - which still has significant issues. For the most part, 2.0.53's mod_disk_cache (i.e. from SVN 2.0.x branch) is mostly equivalent to 2.1.3's mod_disk_cache. So, if it works with 2.0.53 from SVN and doesn't work with 2.1.3 from SVN, then I think mod_proxy is a more likely culprit. I know that there were some fixes recently made to mod_proxy that might help. So, what revision of 2.1.3 are you trying? I know that Mladen fixed some race conditions, and I know Sander and I have been making some recent commits in the last week to fix other brokenness. And, there's still stuff that we know just doesn't work: i.e. 'ProxyPass /foo/bar !'. So, I don't think I can recommend 2.1.3 for proxy functionality as it needs a lot more care and feeding before it is usable. Thanks! -- justin Thanks for your comments. The only reason I blamed mod_[disk]_cache is because the strange behaviour started happening once I enabled it. Proxy without cache seems to be working fine. Im not sure where to find out the revision, but I checked out the src from cvs just before my first post. I just did another update and nothing has changed. Hope Im using the right command: cvs checkout -d httpd-2.1 httpd-2.0 I guess I'll have to wait for mod_proxy to mature bit more. Compliments to the developers, it works great with the new mod_filter which has some really cool potential Thanks again Dean
Re: [PATCH] get a pointer to the raw cert from mod_ssl
On Wed, Feb 02, 2005 at 10:17:04AM +, David Reid wrote:
> Basically this allows us to gain access to the actual cert structure.
I don't like the idea of exposing the X509 * directly especially not
through a char * interface. Exposing the DER representation (e.g.
base64-encoded) through ssl_var_lookup would be better.
joe
> Index: ssl_engine_vars.c
> ===
> --- ssl_engine_vars.c (revision 123890)
> +++ ssl_engine_vars.c (working copy)
> @@ -364,6 +364,10 @@
> else if (strcEQ(var, "CERT")) {
> result = ssl_var_lookup_ssl_cert_PEM(p, xs);
> }
> +else if (strcEQ(var, "RAW_CERT")) {
> +result = (char *)xs;
> +resdup = FALSE;
> +}
>
> if (result != NULL && resdup)
> result = apr_pstrdup(p, result);
[PATCH] get a pointer to the raw cert from mod_ssl
Basically this allows us to gain access to the actual cert structure.
Index: ssl_engine_vars.c
===
--- ssl_engine_vars.c (revision 123890)
+++ ssl_engine_vars.c (working copy)
@@ -364,6 +364,10 @@
else if (strcEQ(var, "CERT")) {
result = ssl_var_lookup_ssl_cert_PEM(p, xs);
}
+else if (strcEQ(var, "RAW_CERT")) {
+result = (char *)xs;
+resdup = FALSE;
+}
if (result != NULL && resdup)
result = apr_pstrdup(p, result);
Re: [PATCH] set username from certificates at a more appropriate time
On Wed, Feb 02, 2005 at 10:07:56AM +, David Reid wrote: > William A. Rowe, Jr. wrote: > >My only concern ... does this new scope pair up properly if the > >user cert has been renegotiated? If so +1 > > I'm unable to test that here, but maybe if someone has a system where > renegotiation is testable... You moved the code to be directly after the per-dir-reneg logic so it should be fine. joe
Re: [PATCH] set username from certificates at a more appropriate time
William A. Rowe, Jr. wrote:
My only concern ... does this new scope pair up properly if the
user cert has been renegotiated? If so +1
I'm unable to test that here, but maybe if someone has a system where
renegotiation is testable...
david
Bill
At 07:17 PM 2/1/2005, you wrote:
Index: ssl_engine_kernel.c
===
--- ssl_engine_kernel.c (revision 123890)
+++ ssl_engine_kernel.c (working copy)
@@ -798,6 +798,20 @@
}
}
+/* If we're trying to have the user name set from a client
+ * certificate then we need to set it here. This should be safe as
+ * the user name probably isn't important from an auth checking point
+ * of view as the certificate supplied acts in that capacity.
+ * However, if FakeAuth is being used then this isn't the case so
+ * we need to postpone setting the username until later.
+ */
+if ((dc->nOptions & SSL_OPT_FAKEBASICAUTH) == 0 && dc->szUserName) {
+char *val = ssl_var_lookup(r->pool, r->server, r->connection,
+ r, (char *)dc->szUserName);
+if (val && val[0])
+r->user = val;
+}
+
/*
* Else access is granted from our point of view (except vendor
* handlers override). But we have to return DECLINED here instead
@@ -1042,17 +1056,6 @@
}
/*
- * Set r->user if requested
- */
-if (dc->szUserName) {
-val = ssl_var_lookup(r->pool, r->server, r->connection,
- r, (char *)dc->szUserName);
-if (val && val[0]) {
-r->user = val;
-}
-}
-
-/*
* Annotate the SSI/CGI environment with standard SSL information
*/
/* the always present HTTPS (=HTTP over SSL) flag! */
Re: [PATCH] set username from certificates at a more appropriate time
I presume this fixes #31418? Your patch makes sense to me. I could argue that it could even be done *before* the SSLRequire checking, such that the "username" is logged appropriately even if an SSLRequire triggers a 403, but I doubt that matters much. Thanks for looking at this!
Re: LDAPTrustedMode has the wrong scope...
Brad Nicholes said: > The attached patches convert LDAPTrustedMode into a per-directory > directive rather than a per-server. This allows the configuration to > specify which mode should be applied for the associated AuthLDAPURL. > > Thoughts on whether this should be the way to go or if LDAPTrustedMode > should be moved up into mod_authnz_ldap as AuthLDAPTrustedMode? Thinking about this some more - I'm not keen on the idea of adding another directive to mod_authnz_ldap, because when the configuration-via-LDAP happens, and other potential LDAP modules happen, then we end up with directive soup as each module has it's own variation of AuthLDAPTrustedMode. The idea that it be possible to set the SSL mode inside mod_authnz_ldap is still really useful - what we could do is this: AuthLDAPURL ldap:// AuthLDAPURL ldaps:// AuthLDAPURL SSL ldap:// AuthLDAPURL STARTTLS ldap:// In other words, combine the SSL mode in a TAKE12 option, with an optional SSL mode prefix. The LDAPTrustedMode directive could stay so as to define the default (and overriding the default if the user wanted). Thoughts? Regards, Graham --
