Re: getting rid of 'server reached MaxClients setting' error
Stas Bekman wrote:
I think this misleading error is really a bug in Apache:
I agree.
[Mon Apr 26 15:28:44 2004] [error] server reached MaxClients setting,
consider raising the MaxClients setting
It sounds like a one-off bug to me. It reports that error when the
number of workers is the same as MaxClients,
it's worse than that, I think.
which is perfectly fine. It
should only report a problem when a new request is coming in and there
are no free servers/threads to handle the request.
I've seen this recently while experimenting with event driven I/O built on top
of the worker MPM. It seems to happen when the parent is trying to increase the
number of child processes but they haven't initialized yet and the server is
temporarily out of worker threads. I didn't see the number of active workers
reach MaxClients. Re-examining the test that leads to the message:
else if (idle_thread_count < min_spare_threads) { <== cool
/* terminate the free list */
if (free_length == 0) { <== insufficient
I'm a little rusty on this code, but looks like free_length is increased when
the code finds a process with at least one worker thread with SERVER_DEAD state
in the scoreboard. I don't see where it checks for MaxClients.
Greg
Re: Regarding Apache 2.0.48 and specweb99
Joshua Schnee wrote: I am attempting to set up the latest Apache server and RHEL 64bit to use in a Specweb99 run and am running into cgi issues. I am very new to Apache, and am having difficulty getting apache to run/use/find my cgi-script. Static content works fine, but I am getting improper file returned errors on all but the command/reset. Would either of you will willing to help? I am copying the [EMAIL PROTECTED] mailing list where all the Apache specweb99 people hang out. You might want to subscribe if you are going to be working with specweb99 for a while. See http://httpd.apache.org/test/ for info on the mailing list and the project in general. There are two other sub-projects discussed on this list but the total traffic is low. OK, some questions. Are you trying to use mod_specweb99 for dynamic content, or are you trying to use just a CGI? If it's the latter, do any CGIs work for you? Apache ships two sample CGIs in the cgi-bin install directory, printenv and test-cgi. Make sure you chmod +x them or they won't work; it's a mystery to me why we install them without execute permissions. If you see the CGI code returned to the client rather than the result of executing the script, you probably need to tweak your httpd.conf to tell Apache that this is a CGI and not just a static file. Here's what I have: Options ExecCGI SetHandler cgi-script If you can get the Apache sample CGIs to work but still have problems, you are probably seeing failures at the beginning of the run where the SPEC client tests the various types of URLs. Fortunately, the client prints the URLs before it tests them. Cut-n-paste the failing URLs into a browser navigation bar and hit enter. What does the browser display? Hope this helps. Greg
Re: Regarding Apache 2.0.48 and specweb99
MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote: I enabled the POST transactions, and all of a sudden, the apache process is now hung (this is first time I'm seeing this behaviour).. The stack is : (gdb) t 21 [Switching to thread 21 (system thread 29207)] #0 0xc0306850:0 in _semop_sys+0x30 () from /usr/lib/hpux64/libc.so.1 (gdb) bt #0 0xc0306850:0 in _semop_sys+0x30 () from /usr/lib/hpux64/libc.so.1 #1 0xc0314b50:0 in semop+0xb0 () from /usr/lib/hpux64/libc.so.1 #2 0xc1b73220:0 in proc_mutex_sysv_acquire+0x70 () from /opt/hpws/apache/lib/libapr-0.sl.9 #3 0xc1b73fc0:0 in apr_proc_mutex_lock+0x60 () from /opt/hpws/apache/lib/libapr-0.sl.9 #4 0xc1b72950:0 in apr_global_mutex_lock+0x90 () from /opt/hpws/apache/lib/libapr-0.sl.9 #5 0xc1cf37d0:0 in do_post+0x780 () Two possibilities: * the command/Fetch URI is stuck also, or * something died holding the post lock, and it didn't get automagically cleaned up. I recall hearing that SysV sems are notorious for that on some platforms. Do either of those make sense? Any clues in the error_log? Greg
Re: Regarding Apache 2.0.48 and specweb99
MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote:
I think I found the problem (.. and it's not the cgid exiting problem).
The problem was because the default Listen Backlog in mod_cgid was a little
small (100 outstanding connections). I got the following tusc log for the
httpd processes :
{62717} connect(13, 0x9fffbebbfe60, 94) .. ERR#239
ECONNREFUSED
{62849} connect(29, 0x9fffbdcbfea0, 94) .. ERR#239
ECONNREFUSED
I increased listen backlog and the processes are a little more happy now :)
BTW, I had to increase the "ListenBacklog" for httpd also - the default 512
caused too many "Can't connect" errors. I increased it to 1024.
-Madhu
I'm glad you're making progress. But I'm wondering why raising the mod_cgid
Listen backlog was so important. If 100 mod_cgid connections wasn't enough at
some point, either the workload is spikey or the steady state arrival rate of
CGI requests is too fast for one daemon + your SPECweb99 CGI program to keep up.
If the latter is true, you should see more and more CGI requests building up
over time in server-status with ExtendedStatus on, and a big improvement in
throughput if you set DYNAMIC_CGI_GET=0 in the SPEC rc file. Then it would be
worthwhile using tusc/strace/truss with some timing options set to look for
unnecessary delays in mod_cgid. If that doesn't show a problem, perhaps we
should have multiple cgid daemons running in parallel for best throughput.
Someone brought up that idea a while back on [EMAIL PROTECTED]; cross-posting there.
Greg
Re: Regarding Apache 2.0.48 and specweb99
MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote: Hmmn. Interesting. 1. Did you include the cgid restart fix ? I don't think so. It's the httpd-2.0.48 tarball + mod_specweb99. 2. Are you driving the server with the SPECweb99 recommended CGI load ? DYNAMIC_CGI_GET=.005 I believe that's the standard. Looking at server-status, I see the CGI URIs dominating the active server threads even with this tiny percentage of the total workload. I'm using the perl script shipped with SPECweb99. An obvious improvement would be to use a compiled C implementation. But I'm not very interested in setting records or making CGIs run fast. I'm more interested in speeding up mod_specweb99 and base Apache. 3. do you mind posting the httpd.conf attached. > Oh.. BTW I hope you're also using worker MPM . [EMAIL PROTECTED] built]$ bin/httpd -l Compiled in modules: core.c worker.c http_core.c mod_suexec.c mod_so.c Greg ServerRoot /home/gregames/apache/httpd-2.0.48/built DocumentRoot /spec_docroot User webuser Group staff LoadModule specweb99_module modules/mod_specweb99.so LoadModule status_module modules/mod_status.so LoadModule info_module modules/mod_info.so LoadModule cgid_module modules/mod_cgid.so UseCanonicalName Off Listen 80 ServerLimit 700 MaxClients 450 StartServers 50 MinSpareServers 5 MaxSpareServers 100 ServerLimit 50 StartServers 2 MinSpareThreads 5 MaxSpareThreads 50 ThreadsPerChild 25 MaxClients 1000 ServerLimit 50 StartServers 3 MinSpareThreads 5 MaxSpareThreads 100 ThreadsPerChild 25 MaxClients 1000 # in case mod_mime isn't loaded DefaultType text/html # Assume no memory leaks at all MaxRequestsPerChild 0 # it's always nice to know the server has started #LogLevel debug ErrorLog logs/error_log # Some benchmarks require logging, which is a good requirement. Uncomment # this if you need logging. #TransferLog logs/access_log Options FollowSymLinks AllowOverride None Options ExecCGI SetHandler cgi-script # might want to disable ExtendedStatus to do serious runs ExtendedStatus On SetHandler server-status SetHandler server-info
Re: Regarding Apache 2.0.48 and specweb99
MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote: Hi Greg, The recent set of discussions prompted me to get some Apache numbers out there - and when I started with the SPECweb99 run, I experienced a major hang in the Apache, and the cgid daemon getting killed (I don't know how). Have you also seen something similar ? OK, I got past my build problems with httpd-2.0.48 (thanks Jeff!). So far I'm not seeing any similar problems with cgid or hangs in the core server. It's not going as fast as my last specweb99 run, but that's to be expected since I quit cheating and turned on CGIs. Greg
Re: Regarding Apache 2.0.48 and specweb99
MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote: cgid should _never_ exit without something in the error log. I tried attaching tusc to the cgi daemon - but since the daemon dies at a random time, my log file was getting too full, and I had to just stop it. Why not just let tusc write to a console? I would assume that once the problem hits, there won't be many more trace entries. The relevant traces will probably be right on the screen, or easy to scroll back to. I'm having build problems that don't make much sense to me with 2.0.48, so I can't tell you yet if I see any specweb99 weirdness with that release. Greg
Re: Regarding Apache 2.0.48 and specweb99
making public per Madhu MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote: I'm using 2.0.48 and also back-ported the cgid-restart patch to 2.0.48. I got a little further on the issue : The Apache hang is definitely caused by the cgid exiting. cgid should _never_ exit without something in the error log. That makes it sound like a core problem, i.e. ap_process_child_status() or a signal handler is fubar, in addition to whatever made the cgi daemon die. But if that is in fact happening, I would trace syscalls & signals for the cgid process. (can't remember what the HPUX trace program is called, but you want something similar to truss/strace) Here's what I think is happening : the SPECweb99 clients have stopped waiting for the post log, but the CGID exited in the middle of the processing - thus causing the SPECweb99 clients to hang (and hence Apache also to hang) > -Madhu hmmm, remember the mod_specweb99 locking change for the post log? I wonder if that is related?But AFAIK the DYN_CGI_SCRIPT cgi only does GETs, in which case locking on the post log won't matter. Or are you using a cgi for more than just DYN_CGI_SCRIPT? Greg -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, November 20, 2003 6:27 AM To: MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) Subject: Re: Regarding Apache 2.0.48 and specweb99 MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) wrote: Hi Greg, The recent set of discussions prompted me to get some Apache numbers out there - and when I started with the SPECweb99 run, I experienced a major hang in the Apache, and the cgid daemon getting killed (I don't know how). Have you also seen something similar ? Madhu, No, that sounds really bad. But to be honest, I had CGI turned off in the spec rc file because I had some config type problem with CGIs that I didn't bother to resolve. I will see if I can make some progress on 2.0.48 + spec + CGI today (barring day job interruptions) and let you know what happens. I would think that cgid would be more reliable than ever. I know Jeff Trawick fixed a double close bug and someone put in logic to let it restart automatically. But who knows? Greg
Re: [PATCH] Get mod_specweb99 to compile against 2.0.43
"MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1)" wrote: > > I noticed that there were some places where u_int32_t is being used instead > of apr_uint32_t. Is it purposefully done OR is it one of those "Oh, the apr > interface changed" stuff ?. > > Anyways, I've included a patch that atleast gets the module to compile > against 2.0.43. Pl. let me know if it's okay. It works fine on Linux. Committed, thanks! Greg
