Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
Hello, if you would give detailed information about your problems, we could advise you, whether it is known, and whether it has been fixed in some version. The latest release is 1.2.15, we tried to produce a 1.2.16 release, that but it has now been withdrawn because of a regression bug. Another release (1.2.17) will be tried next week. Rainer Edmon Begoli schrieb: > We are running into some serious load and scalability issues related to the > mod_jk. > > I read some threads that indicate that these issues may have been fixed in > post 1.2.15 version. > > Available downoalds are only for 1.2.15. > > Some websites are indicating that the latest one should be 1.2.16. > > Can someone please tell me what is the latest version of mod_jk that we > should be using. > > > We are running 1.2.15 complied on RedHat ES 4.2.x. with Apache 2.0.52 > - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
Hi, First of all - thanks Rainer for willing to look into this issue. Environment description: Web Server: Apache 2.0.52 mod_jk: 1.2.15 OS: RedHat ES 4.2 App. Server: Tomcat 5.5.15 Cluster topology: 4 webservers with mod_jk doing sticky loadbalancing into 10 tomcats MOD_JK Configuration: Pretty much basic mod_jk and Tomcat settings for AJP13 Description of the issue: Under very heavy loads for our standards (about thousand of concurrent users) we start seeing errors (listed at end botton of this e-mail) in the mod_jk error log. Our concern is mod_jk behavior on the web server. We closely monitor apache and Apache itself is not under heavy stress. However, shortly after we start seeing these messages in the mod_jk's error log, web server becomes unresponsive and we have to re-start it. While the web server server is unresponsive we monitor 8009 port - nothing is going on it. We monitor Tomcats and they are fine. As soon as we re-start web servers traffic resumes. Then under heavy loads things break again We are planning on increasing the number of max threads on the Tomcats' AJP connector to imrpove the throughput of the Tomcats. However, our concern is with the web server side lock ups. It seems that mod_jk cannot recover after this oversaturation. TAIL -F FROM THE MOD_JK LOG RIGHT BEFORE IT "DIES" __ [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:45 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:45 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:46 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:46 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:46 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07
Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
What is your load average under these circumstance? Is each of the 10 tomcats on a seperate physical machine? What are your max thread settings in Tomcat? Are your tomcat machines CPU bound or I/O bound under full load? Alex On 7/9/06, Edmon Begoli <[EMAIL PROTECTED]> wrote: Hi, First of all - thanks Rainer for willing to look into this issue. Environment description: Web Server: Apache 2.0.52 mod_jk: 1.2.15 OS: RedHat ES 4.2 App. Server: Tomcat 5.5.15 Cluster topology: 4 webservers with mod_jk doing sticky loadbalancing into 10 tomcats MOD_JK Configuration: Pretty much basic mod_jk and Tomcat settings for AJP13 Description of the issue: Under very heavy loads for our standards (about thousand of concurrent users) we start seeing errors (listed at end botton of this e-mail) in the mod_jk error log. Our concern is mod_jk behavior on the web server. We closely monitor apache and Apache itself is not under heavy stress. However, shortly after we start seeing these messages in the mod_jk's error log, web server becomes unresponsive and we have to re-start it. While the web server server is unresponsive we monitor 8009 port - nothing is going on it. We monitor Tomcats and they are fine. As soon as we re-start web servers traffic resumes. Then under heavy loads things break again We are planning on increasing the number of max threads on the Tomcats' AJP connector to imrpove the throughput of the Tomcats. However, our concern is with the web server side lock ups. It seems that mod_jk cannot recover after this oversaturation. TAIL -F FROM THE MOD_JK LOG RIGHT BEFORE IT "DIES" __ [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:44 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:45 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:45 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1 [Fri Jul 07 09:59:46 2006] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (:8009), err=-110 [Fri Jul 07 09:59:46 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0 [Fri Jul 07 09:59:46 2006] [info] ajp_service::jk_ajp_common
Re: URGENT HELP NEEDED: mod_jk loadbalancing issues
Each Tomcat machine is running in 512 MB Max. (We recently increased that to ~850 MB. We are trying not to go up to far to avoid prolonged garbage collector pauses) CPU, I/O utilization under heavy loads is not excessive. 80% max usually 60% Max. thread settings are default. I was thinking to bump it by 35-50% as well as the backlog. BTW - What would be the impoact/side effects of increasing the Connectors max threads and backlog size. Thanks, Edmon _ What is your load average under these circumstance? Is each of the 10 tomcats on a seperate physical machine? What are your max thread settings in Tomcat? Are your tomcat machines CPU bound or I/O bound under full load? Alex On 7/9/06, Edmon Begoli <[EMAIL PROTECTED]> wrote: > > Hi, > > First of all - thanks Rainer for willing to look into this issue. > > Environment description: > > Web Server: Apache 2.0.52 > mod_jk: 1.2.15 > OS: RedHat ES 4.2 > App. Server: Tomcat 5.5.15 > > Cluster topology: 4 webservers with mod_jk doing sticky loadbalancing into > 10 tomcats > > MOD_JK Configuration: Pretty much basic mod_jk and Tomcat settings for > AJP13 > > Description of the issue: > > Under very heavy loads for our standards (about thousand of concurrent > users) > we start seeing errors (listed at end botton of this e-mail) in the mod_jk > error log. > > Our concern is mod_jk behavior on the web server. We closely monitor > apache > and Apache itself is not under heavy stress. However, shortly after we > start > seeing these messages in the mod_jk's error log, web server becomes > unresponsive > and we have to re-start it. > > While the web server server is unresponsive we monitor 8009 port - nothing > is going on it. > > We monitor Tomcats and they are fine. > > As soon as we re-start web servers traffic resumes. Then under heavy loads > things break again > > We are planning on increasing the number of max threads on the Tomcats' > AJP connector to imrpove the throughput of the Tomcats. > > However, our concern is with the web server side lock ups. It seems that > mod_jk cannot > recover after this oversaturation. > > > TAIL -F FROM THE MOD_JK LOG RIGHT BEFORE IT "DIES" > __ > > > [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1721): > Receiving from tomcat failed, recoverable operation attempt=0 > > [Fri Jul 07 09:59:43 2006] [info] ajp_service::jk_ajp_common.c (1749): > Sending request to tomcat failed, recoverable operation attempt=1 > > [Fri Jul 07 09:59:44 2006] [error] > ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the > response message from tomcat, network problems or tomcat is down ( REMOVED>:8009), err=-110 > > [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): > Tomcat is down or refused connection. No response has been sent to the > client (yet) > > [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): > Receiving from tomcat failed, recoverable operation attempt=0 > > [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): > Sending request to tomcat failed, recoverable operation attempt=1 > > [Fri Jul 07 09:59:44 2006] [error] > ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the > response message from tomcat, network problems or tomcat is down ( REMOVED>:8009), err=-110 > > [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): > Tomcat is down or refused connection. No response has been sent to the > client (yet) > > [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): > Receiving from tomcat failed, recoverable operation attempt=0 > > [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): > Sending request to tomcat failed, recoverable operation attempt=1 > > [Fri Jul 07 09:59:44 2006] [error] > ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the > response message from tomcat, network problems or tomcat is down ( REMOVED>:8009), err=-110 > > [Fri Jul 07 09:59:44 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): > Tomcat is down or refused connection. No response has been sent to the > client (yet) > > [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1721): > Receiving from tomcat failed, recoverable operation attempt=0 > > [Fri Jul 07 09:59:44 2006] [info] ajp_service::jk_ajp_common.c (1749): > Sending request to tomcat failed, recoverable operation attempt=1 > > [Fri Jul 07 09:59:45 2006] [error] > ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the > response message from tomcat, network problems or tomcat is down ( REMOVED>:8009), err=-110 > > [Fri Jul 07 09:59:45 2006] [error] ajp_get_reply::jk_ajp_common.c (1503): > Tomcat is down or refused connection. No response has been sent to the > client (yet) > > [Fri Jul 07 09:59:45 2006] [info] ajp_service::jk_ajp_common.c (1721): > Receiving from tomcat failed, recove
