Re: Mod_jk balancing, session problem
Choose the suggested access log methods to improve observability (URLs and Cookie and Set-Cookie Headers. We make some tests with error log set to debug, but it is so many informations in the log (even the decrypted password !!, should it be so ??) that it may take some time to analyse everything. But here I have another question about the errors reported by JK status manager. In our production environment we have log level set to error and there are about 200 requests per worker shown by JK status manager as error , but there isn't any line in the mod_jk.log with error. Why is that so ? Another question is about tomcat catalina.out. Please find below a cut from log: 2006-08-10 06:17:20 org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) at org.apache.coyote.Response.action(Response.java:182) at org.apache.coyote.Response.finish(Response.java:304) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) 2006-08-10 06:17:20 org.apache.jk.common.ChannelSocket processConnection WARNING: processCallbacks status 2 2006-08-10 08:38:53 org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) at org.apache.coyote.Response.action(Response.java:182) at org.apache.coyote.Response.finish(Response.java:304) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) 2006-08-10 08:38:53 org.apache.jk.common.ChannelSocket processConnection WARNING: processCallbacks status 2 2006-08-10 08:57:51 org.apache.jk.core.MsgContext action WARNING: Error sending end packet java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518) at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112) at org.apache.jk.core.MsgContext.action(MsgContext.java:293) at org.apache.coyote.Response.action(Response.java:182) at org.apache.coyote.Response.finish(Response.java:304) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) 2006-08-10 08:57:51 org.apache.jk.common.ChannelSocket processConnection WARNING: processCallbacks status 2 Do you have any idea what my be the reason ? Best regards Artur -- View this message in context: http://www.nabble.com/Mod_jk-balancing%2C-session-problem-tf2073073.html#a5765997 Sent from the
Re: Mod_jk balancing, session problem
Hi, Artur schrieb: - does it happen in times, when you get errors in the mod_jk-log? Does it happen at all times, or do you have peaks at special times? I can't figure it out, when this happens, but there isn't any special moment I can observe. It happens randomly. There isn't any error in mod_jk.log Choose the suggested access log methods to improve observability (URLs and Cookie and Set-Cookie Headers. And here I need some help. In mod_jk.log I always see the same worker - balancer worker, so I don't know which worker has been choosen. How can I change it ? I assume by adding request log to mod_jk you mean something like this: # JkRequestLogFormat set the request format JkRequestLogFormat %w %V %T OK, unfortunately mod_jk does not yet allow to log the sub worker of an lb. I'll check for improvement in 1.2.19... IfModule mpm_worker_module StartServers 5 MaxClients 448 MinSpareThreads 5 MaxSpareThreads 25 ThreadsPerChild 7 MaxRequestsPerChild 0 /IfModule Interesting, very few ThreadsPerChild. Not really a problem, but I wonder if this is efficient. We have decresed this value because we thought that maybe Apache is opening too many threads to Tomcat. We had in Tomcat's log entries saying All threads (500) are curently busy, increase maxThreds. Threads in Tomcat = Sum of numbers of apache threads used during peak times. So this relates more to MaxClients and not to ThreadsPerChild. If you want to connect many Apaches with a lot of threads to tomcat you need to use tricks like the APR connector on the tomcat side. A few hundred threads should be no problem, on a good OS even 1.000-2.000 should work. Think about using connect_timeout and prepost_timeout. Maybe also set recovery_options to 3. What values do you suggest for above parameters ? connect_timeout=2 This is in Milliseconds. Anything between 2000 and 2 should be OK. The right value depends on your trust in latency, reliability of your network and the responsiveness of the target machine. prepost_timeout=1 Values between 5000 and 3 are reasonable. Arguments the same as before. recovery_options=3 or 7 (here you'll need to experimnt on your own. The added 4 is very new, but maybe helpful). Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Mod_jk balancing, session problem
Artur wrote: Hello, We have environment with Apache (2.2.0) + Tomcat (5.5.17) connected via mod_jk (1.2.18). Further tomcat makes some queries to databese through Tuxedo services (BEA). Everything works fine until heavy load when something stops responding (Service not available). We think it is tomcat or mod_jk. It occures when Max busy connections on mod_jk status manager shows about 200. Mod_jk is in error status then. Understanding this situation would be helpful to decide, if adding another tomcat really helps (problem=database = more tomcats don't help etc.). So we decided to use mod_jk load balancing feature and we installed another instance of tomcat (on the same machine). Now service is available even under heavy load but we are losing sessions. What do you mean by losing sessions? I expect you mean users work in a session and then they send a request and get a response indicating, that the requested session has not been found? If so: - does it happen in times, when you get errors in the mod_jk-log? Does it happen at all times, or do you have peaks at special times? Stickyness sends users to the correct node, except when the node is not reachable (error, stopped). In this case users will be sent to one of the remaining nodes and - of course - without session replication their session will not be there. Session repllication means to establish a tomcat clusster, something you need to carefully test and that might force changes in your app. - if it happens even when there's no other problem at the same time: make session IDs trackable. Either your app uses URL rewriting, then your session ID is part of every request and logged in the apache access log. Or you use Cookies, then add logging of incoming header Cookie to your mod_log_config LogFormat. If you also add outgoing header Set-Cookie you are able to follow the sessions, and you can decide, if the users really send the correct session ID. Sometimes what you may find is: the previous request produced an error in the app, which then decided to invalidate the session. You can also add a request log to mod_jk, that logs, which worker has beend chosen for every request. That way you do not rely on your user feedback, but can measure the problem yourself (how many users/sessions, what time of day etc.). Of course sticky_session is set to true. Below I paste our config files. Please help if you only can. Best regards Artur ###Jk status for one day: NameTypejvmRouteHostAddrStatD F M V Acc Err Wr Rd BusyMax RR Cd worker2 ajp13 worker2 localhost:8009 127.0.0.1:8009 OK 0 1 1 2144824825 512 480M5.8G6 138 worker3 worker3 is in the original output and has iónly been removed by your copypaste? What was happening, when you got these 2144 Errors? The time stamps should be available from the mod_jk log. ###httpd.mpm section: # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves IfModule mpm_worker_module StartServers 5 MaxClients 448 MinSpareThreads 5 MaxSpareThreads 25 ThreadsPerChild 7 MaxRequestsPerChild 0 /IfModule Interesting, very few ThreadsPerChild. Not really a problem, but I wonder if this is efficient. ###workers.properties: # Define 3 workers, 2 real workers using ajp13 and the last one being a loadbalancing worker worker.list=worker1,jkstatus # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8009 worker.worker2.lbfactor=1 worker.worker2.connection_pool_size=7 This should be set top the same value as ThreadsPerChild by mod_jk automatically. worker.worker2.connection_pool_timeout=300 worker.worker2.socket_timeout=60 Think about using connect_timeout and prepost_timeout. Maybe also set recovery_options to 3. # Set properties for worker4 (lb) which use worker1 and worker2 worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8019 worker.worker3.lbfactor=1 worker.worker3.connection_pool_size=7 worker.worker3.connection_pool_timeout=300 worker.worker3.socket_timeout=60 # Define a 'jkstatus' worker using status worker.jkstatus.type=status worker.worker1.type=lb worker.worker1.sticky_session=true worker.worker1.method=R worker.worker1.balance_workers=worker2,worker3 ###server1.xml - connector section: !-- Define an AJP 1.3 Connector on port 8009 -- Connector port=8009
Re: Mod_jk balancing, session problem
Rainer Jung-3 wrote: What do you mean by losing sessions? I expect you mean users work in a session and then they send a request and get a response indicating, that the requested session has not been found? Thats exactly what I meant. If so: - does it happen in times, when you get errors in the mod_jk-log? Does it happen at all times, or do you have peaks at special times? I can't figure it out, when this happens, but there isn't any special moment I can observe. It happens randomly. There isn't any error in mod_jk.log - if it happens even when there's no other problem at the same time: You can also add a request log to mod_jk, that logs, which worker has beend chosen for every request. That way you do not rely on your user feedback, but can measure the problem yourself (how many users/sessions, what time of day etc.). And here I need some help. In mod_jk.log I always see the same worker - balancer worker, so I don't know which worker has been choosen. How can I change it ? I assume by adding request log to mod_jk you mean something like this: # JkRequestLogFormat set the request format JkRequestLogFormat %w %V %T Of course sticky_session is set to true. Below I paste our config files. Please help if you only can. Best regards Artur ###Jk status for one day: Name TypejvmRouteHostAddrStatD F M V Acc Err Wr Rd BusyMax RR Cd worker2 ajp13 worker2 localhost:8009 127.0.0.1:8009 OK 0 1 1 2144824825 512 480M 5.8G6 138 worker3 worker3 is in the original output and has iónly been removed by your copypaste? Right What was happening, when you got these 2144 Errors? The time stamps should be available from the mod_jk log. Loosing session occurs even when there arnen't any errors in mod_jk status manager, it just looks like with next request you have to log in once more. ###httpd.mpm section: # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves IfModule mpm_worker_module StartServers 5 MaxClients 448 MinSpareThreads 5 MaxSpareThreads 25 ThreadsPerChild 7 MaxRequestsPerChild 0 /IfModule Interesting, very few ThreadsPerChild. Not really a problem, but I wonder if this is efficient. We have decresed this value because we thought that maybe Apache is opening too many threads to Tomcat. We had in Tomcat's log entries saying All threads (500) are curently busy, increase maxThreds. Think about using connect_timeout and prepost_timeout. Maybe also set recovery_options to 3. What values do you suggest for above parameters ? -- View this message in context: http://www.nabble.com/Mod_jk-balancing%2C-session-problem-tf2073073.html#a5710742 Sent from the Tomcat - User forum at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
