Re: Error 503 ocurring when server under load
On 12 October 2010 19:47, Christopher Schultz wrote: > I would. > > I believe that is true. > > On the other hand, there is another case where you might have problems. > If you have, say, 512 worker threads in Apache httpd but you only have, > say, 200 request processor threads configured in Tomcat, then you will > get mod_jk connection failures on the httpd side. > > I would recommend that you have enough request processors configured in > Tomcat to handle the expected load. > > - -chris Thanks Chris (and everyone else for their comments). As an update: I updated Tomcat from 6.0.24 to 6.0.29 and commented out the cachesize directive from workers.properties. The server ran for a full day yesterday and not a single 503 error. :) So with a bit of luck it will continue to work as expected. Of course I'll monitor for any other issues for the next few days. Rob - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
Christopher Schultz wrote> This directive has been deprecated since 1.2.16. > Cachesize defines the number of connections made to the AJP backend that > are maintained as a connection pool. It will limit the number of those > connection that each web server child process can make. > > Cachesize property is used only for multi threaded web servers such as > Apache 2.0 (worker), IIS and Netscape. The cachesize property should > reflect the number of threads per child process. JK will discover the > number of threads per child process on Apache 2 web server with > worker-mpm and set its default value to match the ThreadsPerChild > Apache directive. [...] > " > > Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number > of worker threads (and therefore simultaneous requests that can be > accepted), you'll have to know what ThreadsPerChild is set to. That's > easy: just search for that directive in httpd.conf. If it's not there, > or commented-out, the default is 64 on mpm_winnt > (http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild). > > If you've left it as the default (64) and you've set worker.cachesize > (really worker.connection_pool_size) to 10, then you can accept 64 > connections from clients but only call-out to Tomcat over 10 of those > connections: the others will have to wait. I'm not sure what the > scheduling algorithm is for choosing which thread gets the next > available connection from the pool (FIFO?), but it's very possible to > starve your clients even after 10 simultaneous connections. > > - -chris So if I'm reading your email and the docs correctly. I should just comment out the cachesize=10 from the workers.properties. And since for connection_pool_size (that replaced it) JK will discover this number for the Apache web server automatically and set the pool size to this value, I don't need to add anything to the workers.properties file? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
Thanks to all for their replies to date. Much appreciated, I'll respond to some of the points raised. Apologies if I'm asking basic stuff but still getting my ahead around Tomcat and Apache integration. On 7 October 2010 18:31, André Warnier wrote: > Your configuration looks very clean to me (no unnecessary settings etc.), > which in this case is a plus (a good base to start tuning). Thank you > You may want to upgrade Tomcat to the latest version (6.0.29). Noted > For example, at the moment these errors happen, what are these Tomcats > really doing ? > Are they really busy each processing 200 requests, with 200 threads running > and actually doing something ? (200 is the default for the "maxThreads" > attribute of the AJP Connector). Unfortunately I'm not in a position to conduct realtime monitoring of the server and Tomcat. Is there a way to log this thread data? > If yes, then you may just need a leaner application, or a bigger system > (more RAM, faster CPU), or more systems. What does the Task Manager tell > you about the total system load ? Again, unfortunately not in a position to monitor in real time. > If not, and many of these threads are waiting, then you may have an issue > with a keepAlive that is too long. > See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the > connectionTimeout and keepAliveTimeout attributes. Will do > Whatever you do, first get an idea of the starting situation. Then modify > one setting at a time, and observe (and note) the effects. Sensible advice Pid wrote: > OK. Is there anything else different apart from the Servlet container? I'm not sure what you mean? > Why do you have two Tomcat instances? (It's not a trick question, I'm > interested in your reasoning.) Load balancing. I thought/guessed that two instances would be sufficent. Perhaps this was incorrect and I need more? > Was there an attachment I didn't see? No, all details for .conf files and server.xml was posted inline. > How many threads & server instances did you have before? >What were your Java -Xmx etc settings before, and now? > Are you using an Executor? I'll have to check, but I think it's safe to say that they are set to the default values as I don't recall changing anything relating to them. > Thread dumps from a maxed out Tomcat will tell you what each Thread is > waiting for. Collect a series of these during high load periods to find out what's happening. I'll have to figure out how to do this, but sounds a good idea. Christopher Schultz wrote > I think you might want to compare the request processor counts you have for > Tomcat versus mod_jk: the default maxThreads for a Tomcat is 200, > while your MaxClients settings might be different on your Apache httpd > configuration. Excuse my ignorance, but can you clarify where I check these settings? > I believe that only the Worker MPM is available on Microsoft Windows. Have > you explicitly configured an MPM? Since I don't know what an MPM is I'm going to say no. > Since you're using Apache 2.x, mod_jk should be able to auto-configure this > value for you. Do I need to set anything to allow this auto config to happen? > From my reading, you may be limiting yourself to 10 simultaneous connections > from Apache httpd to Tomcat, and others will get a 503. What lead you to that conclusion? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Error 503 ocurring when server under load
Hey all, Recently migrated a production site (mixture of Servlets and JSPs) from Oracle Application Server to Apache/Tomcat. Since then we have seen numerous HTTP Error 503 - Service unavailable errors at peak times when site is under load. mod_jk.log has the following error message(s): [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances failed, no more workers left I'm looking for help in trying to tweak settings to prevent this, or confirmation that I've configured the setup correctly. Many thanks Rob Platform: Windows Server 2003 SP2 Setup: Two tomcat instances with a single Apache front end, all on the same server Versions Tomcat: 6.0.24 Apache: 2.2.16 mod SSL: 2.2.16 Open SSL: 0.9.8 mod_JK:1.2.30 Conf: httpd.conf: JkWorkersFile conf/workers.properties JkLogFile logs/mod_jk.log JkLogLevel error JkLogStampFormat "[%a %b %d %H:%M:%S %Y] " JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat "%w %V %T" AllowOverride None deny from all JkMount /examples balancer JkMount /examples/* balancer JkMount /App1/* balancer JkMount /App2/* balancer Workers.properties: workers.tomcat_home="C:\tomcat1" workers.java_home="C:\Program Files\Java\jdk1.6.0_18" ps=\ # Define workers #worker.list=worker1,worker2 worker.list=balancer # Set properties for worker worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.lbfactor=1 worker.worker1.cachesize=10 worker.worker1.connection_pool_timeout=300 worker.worker1.socket_keepalive=FALSE #worker.worker1.recycle_timeout=300 # Set properties for worker worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8019 worker.worker2.lbfactor=1 worker.worker2.cachesize=10 worker.worker2.connection_pool_timeout=300 worker.worker2.socket_keepalive=FALSE #worker.worker2.recycle_timeout=300 worker.balancer.type=lb worker.balancer.balance_workers=worker1,worker2 worker.balancer.method=R worker.balancer.sticky_session=True Server.xml (same for both tomcat instances apart from jvmRoute): - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Restrict access to Manager from local ip address (Tomcat 6.0/Windows)
Well I had looked at http://tomcat.apache.org/tomcat-6.0-doc/manager-howto.html#Configuring Manager Application Access. And it has the docbase attribute too... > All too common, unfortunately. Use the real Tomcat doc first, then Google. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Restrict access to Manager from local ip address (Tomcat 6.0/Windows)
Charles and Mark, Thanks for your help. Your advice on removing the docBase and Resourcelink attibutes was correct and the restriction is now working correctly. I had based my changes to the conf files based on these articles: http://www.unidata.ucar.edu/Projects/THREDDS/tech/reference/TomcatSecurity.html http://www.theserverside.com/tt/articles/content/TomcatSecurity/TomcatSecurity.pdf Obviously they weren't entirely correct. Thanks again Regards Rob - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Restrict access to Manager from local ip address (Tomcat 6.0/Windows)
Hey all, My setup: Windows Server 2003 SP2 Single instance of Apache HTTP Server 2.2.15 (C:\Apache) Two instances of Apache Tomcat 6.0.24 in load balancing mode (C:\tomcat1 and C:\tomcat2) JK 1.2.30 used to connect the Apache front end to the two tomcat instances I've trying to configure the tomcat instances so that the manager webapp can only be access via browser on the Windows Server (i.e. 127.0.0.1). I've RTFM, STFW and nothing seems to work. I believe the process below is correct but it doesn't work. Can anyone see where I'm going wrong? 1) I've edited the following file: C:\tomcat1\webapps\manager\META-INF\context.xml. 2) It's contents is: 3) I've stopped and started the tomcat1 service. 4) I try to access the manager webapp from a remote machine ( http://hostname:8080/manager/html) and I can get access. I've also tried adding the text below to C:\tomcat1\conf\Catalina\localhost\manager.xml But no joy. Any help appreciated! Thanks Rob - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
