The new releases' win32 files
I noticed there's been a delta for a while, the unix packages drop all the docs .xml files. Have done the same for win32-src.zip and the .msi installers in this pass. I also note that in the future, we should drop the .xsl and changed the roll.sh script. Perhaps .dtd files as well? These all appear to be meaningless if you aren't using the xml source files. There are a ton of closed bugs and it would be really great if folks could take the new installers for a spin, which are located in http://www.apache.org/dist/httpd/binaries/win32/ (and even 1.3.41, parked over in http://archive.apache.org/dist/httpd/binaries/win32/ ) just to ensure all these changes are blessed and golden. I happen to be hibernating from the midwest cold most of this weekend, so there's a good chance I can address any problems very quickly. Hopefully we will see far fewer trouble reports. The only thing I really couldn't get to (since other issues all caved in on me since the holidays) was a special-flavor AWK.exe that would do utf-8 rewriting of our conf files from, unicode command line or environ variables and file names. That's a project I'm headed back to after my plate is a bit cleaner, so that folks installing in non-ASCII path names will someday soon have a much easier time of it. Doing this without help from apr will prove interesting ;-) Bill
Re: Is async the answer
On Jan 18, 2008 2:30 PM, Ruediger Pluem <[EMAIL PROTECTED]> wrote: > IMHO it doesn't for the first request of the entity (the request that causes > the entity to be cached) I'd expect the predominance of large numbers would reduce the impact of the one-time performance hit...but that conversion away from a fd and into flat buffers, I feel, is more of a bucket brigade problem than an intrinsic fault of mod_disk_cache. -- justin
Re: PR42829: graceful restart with multiple listeners using prefork MPM can result in hung processes
On Fri, Jan 04, 2008 at 02:42:05PM +0100, Stefan Fritsch wrote: > this bug can be quite annoying because of the resources used by the hung > processes. It happens e.g. under Linux when epoll is used. > > The patch from http://issues.apache.org/bugzilla/show_bug.cgi?id=42829#c14 > has been in Debian unstable/Ubuntu hardy for several weeks and there have > not been any complaints. I've been looking into this in more detail; excuse the length of this mail. The symptom in question is described as "children hang after graceful restart/stop in 2.2.x". I mentioned in the bug that the signal handler could cause undefined behaviour, but I'm not sure now whether that is true. On Linux I can reproduce some cases where this will happen, which are all due to well-defined behaviour: 1) with some (default on Linux) accept mutex types, apr_proc_mutex_lock() will loop on EINTR. Hence, children blocked waiting for the mutex do "hang" until the mutex is released. Fixing this would need some APR work, new interfaces, blah 2) prefork's apr_pollset_poll() loop-on-EINTR loop was not checking die_now; the child holding the mutex will not die immediately if poll fails with EINTR, and will hence appear to "hang" until a new connection is recevied. Fixed by http://svn.apache.org/viewvc?rev=613260&view=rev I can also reproduce a third case, but I'm not sure about the cause: 3) apr_pollset_poll() is blocking despite the fact that the listening fds are supposedly already closed before entering the syscall. I vaguely recall some issue with epoll being mentioned before in the context of graceful stop, but I can't find a reference. Colm? A very tempting explanation for (3) would be the fact that prefork only polls for POLLIN events, not POLLHUP or POLLERR, or indeed that it does not check that the returned event really is a POLLIN event; POSIX says on poll: " ... poll() shall set the POLLHUP, POLLERR, and POLLNVAL flag in revents if the condition is true, even if the application did not set the corresponding bit in events." and there's even a comment in the prefork poll code to the effect that maybe checking the returned event type would be a good idea. But from a brief play around here, fixing the poll code to DTRT doesn't help. I think more investigation is needed to understand exactly what is going on here. (Also, just to note; I can reproduce (3) even with my patch to dup2 against the listener fds.) joe
Re: Is async the answer
On 01/18/2008 10:29 PM, Colm MacCarthaigh wrote: > On Fri, Jan 18, 2008 at 04:17:16PM -0500, Akins, Brian wrote: >> For dynamic stuff, "X-sendfile" works well. (Just really starting to play >> with that, liking it so far). > > It's not a solve-all though, I mean even though CGI's or whatever > /could/ write their output to a file and then call X-sendfile, it'd be a > disaster latency-wise. Ironically enough the only way to solve that is > ... async ;-) > >> The proxy that the LiveJournal folks wrote, I think, copies all the data >> from the origin server into a file and then uses sendfile to send to the >> client... > > Erm, so does the one we wrote, mod_disk_cache ;p IMHO it doesn't for the first request of the entity (the request that causes the entity to be cached) Regards Rüdiger
Re: mod_proxy timeout
As a way to apologize for my 'spamming', I think I've uncovered the issue. This is either a bug, a known limitation or a misunderstanding on my part of the purpose of ProxyTimeout. Turns out that timeout is set on the socket in two places: first when it's attempting to get the connection to the proxy server, in this call: failed = ap_proxy_connect_to_backend(&p_conn->sock, "HTTP", p_conn->addr, p_conn->name, conf, r->server, c->pool); Here, the 'conf' argument passed allows p_conn->sock's timeout to be set with conf->timeout if conf->timeout_set is 1. Later, there's a call to: rc = ap_run_pre_connection(*origin, p_conn->sock); This eventually is uncovered to be the function core_pre_connection found in server/core.c. In this code, p_conn->sock's timeout is set with the timeout from *origin->base_server->timeout. The 'fix' is, after this call, use the timeout you really want to use. I *think* this should be conf->timeout... however, perhaps ProxyTimeout is supposed to be only used to timeout the connection, not the request. If that's true, the description of the directive is confusing. :/ For me, it's actually going to a timeout value set in the notes table by another module so we can control proxy timeouts per request "on the fly". :) Ron On Fri, 2008-01-18 at 12:30 -0500, Ronald Park wrote: > [[Resending as I suspect my junk mail filters flagged it > on sending it. :/ However, if this did make it, I apologize > for this duplicate sending.]] > > > Hi Folks, > > I'm trying to get ProxyTimeout to work and having no luck. > > I've set up 2 Apache instances in my network. These are > Apache 2.0.59 servers built on a 64 bit Redhat EL4 box. > > On the first, I've just added the following directives: > > ProxyTimeout 2 > > ProxyPass /cgi-bin http:///cgi-bin/ > > On the second machine, I have the following simple CGI: > > #!/bin/sh > sleep 30 > echo Content-type: text/html > echo > echo "" > echo "Hello" > echo "" > echo "Hello" > echo "" > > Why doesn't this timeout? > > For 'fun', I' fiddled with the code, namely proxy_http.c, > right before the call to ap_proxy_connect_to_backend, I > reduce the value of 'timeout' in the conf struct by a factor > of 10 (so from 200 to 20, etc) each request. As > I reload the page, nothing different happens: each request > takes 30 secs to finally, successfully return... until I > finally get down to '200'. Then, I finally get a 502, > BAD GATEWAY error. So at that tiny timeout (200 microsecs?), > I'm guessing the low level connect times out. So in some > regards, timeout is working. > > I did find what I think is a bug in the merge_proxy_config > function... although it merges the 'timeout' field member > in the config structure, it's not merging the 'timeout_set' > member. However, fixing that did not resolve my issue. :( > > Is there something wrong with my test? > > Thanks, > Ron >
Re: Is async the answer
On Fri, Jan 18, 2008 at 04:17:16PM -0500, Akins, Brian wrote: > For dynamic stuff, "X-sendfile" works well. (Just really starting to play > with that, liking it so far). It's not a solve-all though, I mean even though CGI's or whatever /could/ write their output to a file and then call X-sendfile, it'd be a disaster latency-wise. Ironically enough the only way to solve that is ... async ;-) > The proxy that the LiveJournal folks wrote, I think, copies all the data > from the origin server into a file and then uses sendfile to send to the > client... Erm, so does the one we wrote, mod_disk_cache ;p -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: Is async the answer
On 1/18/08 3:07 PM, "Colm MacCarthaigh" <[EMAIL PROTECTED]> wrote: > That's not even a consideration, > async is really for dynamic content, proxies, and other non-sendfile > content. For dynamic stuff, "X-sendfile" works well. (Just really starting to play with that, liking it so far). The proxy that the LiveJournal folks wrote, I think, copies all the data from the origin server into a file and then uses sendfile to send to the client... Also, we have driven apache as a proxy as far as we have squid... Paul Q and I have been kicking around the idea that even if we go to a completely async core, etc. that modules could mark some hooks as "blocking" and they would run basically how they do today. (One day, Paul, I'll actually think about this more...) Having a request tied to one thread for its lifetime does make some things easier. If the underlying IO is asynchronous and its faster/scalable/fun, then, all the better. I just am not a big fan of the "callback" method that squid uses (or used last time I looked at it). Yes, its doable, but just seems "not quite right" to me. That's just my opinion. I'd like to be able to say, "hey httpd, write this stuff to the client" and it just happen wonderfully fast :) Currently, worker is doing a great job for us. Maybe async would be fine as well, especially if the serf buckets are as easy to use as Justin says. I just don't want us to say "we must be async" with no real reason other than "we must." -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Is async the answer
On Fri, Jan 18, 2008 at 02:31:11PM -0500, Akins, Brian wrote: > On 1/18/08 2:20 PM, "Colm MacCarthaigh" <[EMAIL PROTECTED]> wrote: > > > I think so, in some environments anyway. If you have a server tuned for > > high throughput accross large bandwidth-delay product links then you > > have the general problem of equal-priority threads sitting around with > > quite a lot of large impending writes. > > Doesn't sendfile (and others) help in that case? Also RAM is cheap, > bandwidth isn't :) Oh if you can use sendfile, you use it sure, and whether its used async or not isn't going to make a big deal, all of the benefits are the zero copy, the DMA, the TOE, and so on. That's not even a consideration, async is really for dynamic content, proxies, and other non-sendfile content. -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: Is async the answer
On 1/18/08 2:20 PM, "Colm MacCarthaigh" <[EMAIL PROTECTED]> wrote: > I think so, in some environments anyway. If you have a server tuned for > high throughput accross large bandwidth-delay product links then you > have the general problem of equal-priority threads sitting around with > quite a lot of large impending writes. Doesn't sendfile (and others) help in that case? Also RAM is cheap, bandwidth isn't :) -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Is async the answer
On 1/18/08 2:16 PM, "Justin Erenkrantz" <[EMAIL PROTECTED]> wrote: > Speaking for myself, I think writing and using buckets with serf is > more straightforward than our complicated bucket brigade system with > mixed push/pull paradigms. It very well may be. Async may be easy. Except when my db connection blocks.. On stat calls.. Etc. I am by no means defending the buckets! Or anything for that matter... Just some observations. I just no longer buy into the idea that async is somehow inherently superior. It sounds good in theory, but in the "real world" I am just not seeing it. The whole reason I brought this up was to stimulate discussion. I really really would hate for us to spend many months porting everything over to async to discover that it made no positive impact on performance. Worse, it made extending httpd (or "D") much harder. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: Is async the answer
On Fri, Jan 18, 2008 at 01:52:02PM -0500, Akins, Brian wrote: > On 1/18/08 12:18 PM, "Colm MacCarthaigh" <[EMAIL PROTECTED]> wrote: > > Hmmm, it depends what you mean by scale really. Async doesn't help a > > daemon scale in terms of concurrency or throughput, if anything it might > > even impede it, but it certainly can help improve latency and > > responsivity greatly. On the whole, it's easy to see how it might make > > the end user experience of a very busy server much more pleasant. > > I also wonder is that has actually been tested or if it's just a "factoid"? I've tested, and it met my expectations on Linux 2.6 on Itanium, but I can't guarantee that the experiments were free from my own bias I guess. > >> Response time never increased in any measurable amount. > > > > I suspect it might though if the scheduler became bound, async would > > route the interupts more efficiently. > > But, I wonder if the scheduler would become bound in a "reasonable" amount > of traffic. I think so, in some environments anyway. If you have a server tuned for high throughput accross large bandwidth-delay product links then you have the general problem of equal-priority threads sitting around with quite a lot of large impending writes. Having them all in the polling loop is inefficient, and async is going to reduce the latency a little, though granted these days we may be talking about nanoseconds. And I guess responsivity and high BDP don't go together anyway, due to the speed of light. > > The scalability wars should really be over, > > everyone won - kernel's rule :-) > > Which is why I hate to see a ton of work go into async core if it actually > does very little to help performance (or if it hurts it) and makes writing > modules harder. It braindead simple nowadays to write well behaved high > performance modules (well, mostly) bcs you rarely worry about threads, > reads/writes, etc. Full async programming is just as challenging as > handling a ton of threads yourself. I think if it interests people and they want to work on it, cool stuff, but don't neccessarily expect any actual pay-off in terms of performance. One of the great things about an open source project is that sometimes what gets worked on isn't driven by considerations other than what people feel like working on. I'd be less worried about the effect on modules, many module authors already can't be bothered to make their modules thread-safe, but prefork still exists (and scales quite well, on many platforms). -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Re: Is async the answer
On Jan 18, 2008 10:52 AM, Akins, Brian <[EMAIL PROTECTED]> wrote: > Which is why I hate to see a ton of work go into async core if it actually > does very little to help performance (or if it hurts it) and makes writing > modules harder. It braindead simple nowadays to write well behaved high > performance modules (well, mostly) bcs you rarely worry about threads, > reads/writes, etc. Full async programming is just as challenging as > handling a ton of threads yourself. Speaking for myself, I think writing and using buckets with serf is more straightforward than our complicated bucket brigade system with mixed push/pull paradigms. YMMV. -- justin
Re: Is async the answer
On 1/18/08 12:18 PM, "Colm MacCarthaigh" <[EMAIL PROTECTED]> wrote: > Hmmm, it depends what you mean by scale really. Async doesn't help a > daemon scale in terms of concurrency or throughput, if anything it might > even impede it, but it certainly can help improve latency and > responsivity greatly. On the whole, it's easy to see how it might make > the end user experience of a very busy server much more pleasant. I also wonder is that has actually been tested or if it's just a "factoid"? >> Response time never increased in any measurable amount. > > I suspect it might though if the scheduler became bound, async would > route the interupts more efficiently. But, I wonder if the scheduler would become bound in a "reasonable" amount of traffic. > discussions on scalability baffling, the reality is that modern hardware > can outscale pretty much any amount of bandwidth you can buy regardless > of the software. Bandwidth generally isn't an issue for us anymore (thanks to gzip). We can still overrun the CPU with small objects requests/responses. On "large" objects (ie, over 16k or so), the CPU is bored when multiple gig interfaces are full. > The scalability wars should really be over, > everyone won - kernel's rule :-) Which is why I hate to see a ton of work go into async core if it actually does very little to help performance (or if it hurts it) and makes writing modules harder. It braindead simple nowadays to write well behaved high performance modules (well, mostly) bcs you rarely worry about threads, reads/writes, etc. Full async programming is just as challenging as handling a ton of threads yourself. My $.02 US worth (which ain't much). -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
mod_proxy timeout
[[Resending as I suspect my junk mail filters flagged it on sending it. :/ However, if this did make it, I apologize for this duplicate sending.]] Hi Folks, I'm trying to get ProxyTimeout to work and having no luck. I've set up 2 Apache instances in my network. These are Apache 2.0.59 servers built on a 64 bit Redhat EL4 box. On the first, I've just added the following directives: ProxyTimeout 2 ProxyPass /cgi-bin http:///cgi-bin/ On the second machine, I have the following simple CGI: #!/bin/sh sleep 30 echo Content-type: text/html echo echo "" echo "Hello" echo "" echo "Hello" echo "" Why doesn't this timeout? For 'fun', I' fiddled with the code, namely proxy_http.c, right before the call to ap_proxy_connect_to_backend, I reduce the value of 'timeout' in the conf struct by a factor of 10 (so from 200 to 20, etc) each request. As I reload the page, nothing different happens: each request takes 30 secs to finally, successfully return... until I finally get down to '200'. Then, I finally get a 502, BAD GATEWAY error. So at that tiny timeout (200 microsecs?), I'm guessing the low level connect times out. So in some regards, timeout is working. I did find what I think is a bug in the merge_proxy_config function... although it merges the 'timeout' field member in the config structure, it's not merging the 'timeout_set' member. However, fixing that did not resolve my issue. :( Is there something wrong with my test? Thanks, Ron
Re: Is async the answer?
On Fri, Jan 18, 2008 at 12:03:02PM -0500, Akins, Brian wrote: > Most of us seem to have convinced our self that high performance network > applications (including web servers) must be asynchronous in order to scale. > Is this still valid? For that matter, was it ever? Hmmm, it depends what you mean by scale really. Async doesn't help a daemon scale in terms of concurrency or throughput, if anything it might even impede it, but it certainly can help improve latency and responsivity greatly. On the whole, it's easy to see how it might make the end user experience of a very busy server much more pleasant. > It seems that modern OS's (this was Linux 2.6.something) deal with the > "thread overhead" and all the context switches very well. All the stuff > mentioned in the "the c10k problem" ( http://www.kegel.com/c10k.html) didn't > seem to apply. We could have easily doubled the amount of connections to > the server, I think. The c10k page has been hopelessly out of date for a long long time, I wrote to Dan Kegel some time about (maybe 3 or 4 years) pointing this out, but there's been no update :/ > Response time never increased in any measurable amount. I suspect it might though if the scheduler became bound, async would route the interupts more efficiently. > Yes, we are using sendfile, mmap, etc., so zero-copy helps us a lot. > > So, do we need apache 3 (or whatever it's called) to be fully asynchronous? > Is that just us reacting to "the market" trends, ie, lighttpd? Who knows, no harm in doing it anyway, if it's what interests people, cool. Personally I find comparisons between webservers and most discussions on scalability baffling, the reality is that modern hardware can outscale pretty much any amount of bandwidth you can buy regardless of the software. And to that end, the software is all near identical in the pipelines of syscall's used (hell even IIS) - which is what really matters. Most discussions seem to centre on some mindlessly ignorant comparison based on the suitable of defaults to a particular set of circumstances coupled with religion. The scalability wars should really be over, everyone won - kernel's rule :-) > All the apache httpd "is bloated and slow" is just plain horse crap. It's > not that hard to configure apache to be "fast." C programming is my > "hobby," and it's not that hard to write modules that don't do stupid things > and kill the performance. Yep! -- Colm MacCárthaighPublic Key: [EMAIL PROTECTED]
Is async the answer?
This is just some ramblings based on some observations, theories, and tests. Partially "devil's advocate" as well. Most of us seem to have convinced our self that high performance network applications (including web servers) must be asynchronous in order to scale. Is this still valid? For that matter, was it ever? We just ran a large scale test on a busy website (won't mention the name...) and ran about 95% of production traffic on a single server. This was about 30k connections. We set maxclients to 50k. Server did fine, had about 4GB Ram free and 55% cpu idle. This was the full production config, not stripped down or anything, using our cache and proxy and several "stock" modules. Granted these were fairly "beefy" servers, but nothing extraordinary: 2xdual core 2.4 Ghz CPU's with 8GB RAM, normal non-TOE Ethernet (but with checksum's on card). It seems that modern OS's (this was Linux 2.6.something) deal with the "thread overhead" and all the context switches very well. All the stuff mentioned in the "the c10k problem" ( http://www.kegel.com/c10k.html) didn't seem to apply. We could have easily doubled the amount of connections to the server, I think. We were using normal worker MPM with keepalives for this test. The current "stable" event would have helped with idle keepalive threads, but the system didn't seem to care. Response time never increased in any measurable amount. Yes, we are using sendfile, mmap, etc., so zero-copy helps us a lot. So, do we need apache 3 (or whatever it's called) to be fully asynchronous? Is that just us reacting to "the market" trends, ie, lighttpd? All the apache httpd "is bloated and slow" is just plain horse crap. It's not that hard to configure apache to be "fast." C programming is my "hobby," and it's not that hard to write modules that don't do stupid things and kill the performance. The biggest thing we do in our modules is to make trade-offs to avoid locking. Ie, we are happy to "waste" a few MB of RAM with some "per-module scoreboards" than to use per-proc or global locking. Most of our counters are per thread, and we just add them up when someone access the counter (ie, via mod_status). This made a huge difference in our performance. Also, don't get in httpd's way. Let the core handlers handle as much as possible. Just "encourage" them when needed. They are battle tested and improvements made there help everything out if you haven't tried to rewrite it in your own module. Like I said, just some ramblings. We may experiment with the current event MPM some more, but I honestly do not see the huge benefit to moving to a fully async IO architecture. It's very easy to write modules in the current "one thread per request" model. People will screw up the async thing and make it slower anyway, probably. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Basic Auth module for plain text password file
Hello, mod_auth_plain by Piotr Roszatycki was not working anymore since the 2.2 branch. Here is a new module based on mod_authn_file and the old mod_auth_plain It implements a new AuthBasicProvider named file_plain. It allow relative file path for the password file. Maybe you are interested by this. http://www.chocobon.fr/libapache2-mod-auth-plain-2.2.4 Regards, Patrick Amiel
Re: PR42829: graceful restart with multiple listeners using prefork MPM can result in hung processes
On Fri, Jan 04, 2008 at 02:42:05PM +0100, Stefan Fritsch wrote: > Hi, > > this bug can be quite annoying because of the resources used by the hung > processes. It happens e.g. under Linux when epoll is used. > > The patch from http://issues.apache.org/bugzilla/show_bug.cgi?id=42829#c14 > has been in Debian unstable/Ubuntu hardy for several weeks and there have > not been any complaints. > > It would be nice if you could look at it and commit it to svn. I can confirm that there are problems with the restart at least on FreeBSD-4.x/prefork. On FreeBSD-4.x/prefork I see this after a graceful restart: --snip-- $ apachectl status Apache Server Status for localhost Server Version: Apache/2.3.0-dev (Unix) mod_ssl/2.3.0-dev OpenSSL/0.9.7d-p1 DAV/2 Server Built: Jan 16 2008 04:19:11 [..] CPU Usage: u4.45313 s4.3125 cu0 cs0 - .00454% CPU load .0265 requests/sec - 9 B/second - 372 B/request 10 requests currently being processed, 7 idle workers GG_G__GGW... [...] --snip-- After another graceful restart, I see GGGWG... and the 'G' processes are stuck at state 'G'. With the patch applied, I no longer see any of the hanging "gracefully stuck" processes. So, from my side, I'd +1 the patch (although I understand the intention of the code, I have not "brain-traced" all code paths, so this is not a final "code +1" but just a "appears to fix the problem +1"). Anyone else? Martin -- <[EMAIL PROTECTED]>| Fujitsu Siemens http://www.fujitsu-siemens.com/imprint.html | 81730 Munich, Germany
