Possible problem: ajp13 on Tomcat 3.2.x - need help
We are using Apache 1.3.26 and Tomcat 3.2.2. We've seen some clients to be hanging Apache on SSL read. The cause of this hang seems to be network/TCP/IP issue. The read will eventually times out. When it times out, here's where the problem starts. Tomcat seems to be grabbing some buffer that contains partial data of the previous request and call a servlet with it. The servlet will throw an exception because the data is bad. Here are the details (you will need to pull up the code to see what I mean). When Apache read times out, mod_jk's jk_ajp13_worker.c::read_into_msg_buff() will return error. mod_jk will drop its connection to Tomcat and retry (up to JK_RETRIES times - see the service() function). In the retry, the HTTP header gets resent to Tomcat. When Tomcat receives this header, it will eventually call Ajp13ConnectorRequest::decodeRequest(). This method will attempt to read more data from mod_jk (the call to con.receive()). This call will fail because mod_jk has yet again drop the connection (in its 2nd retry to call read_into_msg_buff() fails). Since this call (con.receive()) fails, the attribute blen is never set to the content-length of this request. Instead, blen has the value of the content-length of the previous request. Because the call to con.receive() fails, the method decodeRequest() will return -1. The caller to decodeRequest, Ajp13ConnectionHandler::processConnection(), doesn't check the error code and it proceeds to call the servlet. The servlet will issue a read and it will eventually call Ajp13ConnectorRequest::doRead(). In doRead(), I dumped some values of the variables. pos is always 0. blen is some number. The problem happens when blen is less than the length of the buffer (variable len). The method will copy some data from bodyBuff, which is a left-over from previous request. Note, I checked the source code for Tomcat 3.2.4. It doesn't seem to be fixed yet. We may not have the luxury to upgrade to Tomcat 3.3 yet. So I am attempting to fix this, but I have a question: Why does Ajp13ConnectionHandler::processConnection() not check the error code from decodeRequest()? It seems to me like if it gets an error in decodeRequest(), it should not call the service() method which calls the servlet. thanks, shinta -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
ajp13 stuck on read
Hi, We've been seeing some problems where both the Tomcat ajp13 connector and the Apache children (using mod_jk and ajp13) got stuck on read. The load on the system is not too heavy, i.e: there's only 12-20 Apache children around when we first saw the problem. We are using Tomcat 3.2.1 and Apache 1.3.14. Has anyone seen this problem and found an explaination for it? Tomcat's ajp13 connector stack trace looks like this: Full thread dump: Thread-34 (TID:0x178d2b8, sys_thread_t:0x178d1f0, state:R, thread_t: t@91, thr eadID:0xea421d78, stack_bottom:0xea422000, stack_size:0x2) prio=5 [1] java.net.SocketInputStream.socketRead(Native Method) [2] java.net.SocketInputStream.read(SocketInputStream.java:85) [3] org.apache.tomcat.service.connector.TcpConnector.receiveFully(TcpConnect or.j ava:148) [4] org.apache.tomcat.service.connector.TcpConnector.receive(TcpConnector.ja va:1 18) [5] org.apache.tomcat.service.connector.Ajp13ConnectorRequest.refeelReadBuff er(A jp13ConnectorRequest.java:288) [6] org.apache.tomcat.service.connector.Ajp13ConnectorRequest.doRead(Ajp13Co nnec torRequest.java:252) [7] org.apache.tomcat.service.connector.Ajp13ConnectorRequest.doRead(Ajp13Co nnec torRequest.java:261) [8] org.apache.tomcat.core.BufferedServletInputStream.doRead(BufferedServlet Inpu tStream.java:111) [9] org.apache.tomcat.core.BufferedServletInputStream.read(BufferedServletIn putS tream.java:137) [10] org.apache.tomcat.core.BufferedServletInputStream.read(BufferedServletIn put Stream.java:133) [11] myservlet.process(myservlet.java:97) [12] myservlet.doPost(myservlet.java:161) [13] javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [14] javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [15] org.apache.tomcat.core.ServletWrapper.doService(ServletWrapper.java:399) [16] org.apache.tomcat.core.Handler.service(Handler.java:250) [17] org.apache.tomcat.core.ServletWrapper.service(ServletWrapper.java:348) [18] org.apache.tomcat.core.ContextManager.internalService(ContextManager.jav a:7 62) [19] org.apache.tomcat.core.ContextManager.service(ContextManager.java:743) [20] org.apache.tomcat.service.connector.Ajp13ConnectionHandler.processConnec tio n(Ajp13ConnectionHandler.java:109) [21] org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:393 ) [22] org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:47 5) [23] java.lang.Thread.run(Thread.java:478) The Apache child's process stack trace looks like this: (gdb) where #0 0xff218190 in _so_recv () from /usr/lib/libc.so.1 #1 0xfefefa7c in jk_tcp_socket_recvfull () from /opt/apache/libexec/mod_jk.so #2 0xfefeff4c in connection_tcp_get_message () from /opt/apache/libexec/mod_jk.so #3 0xfeff12ec in getreply () from /opt/apache/libexec/mod_jk.so #4 0xfeff1634 in service () from /opt/apache/libexec/mod_jk.so #5 0xfeff6d58 in jk_handler () from /opt/apache/libexec/mod_jk.so #6 0x1f278 in ap_invoke_handler () #7 0x3c4e0 in process_request_internal () #8 0x3c564 in ap_process_request () #9 0x2fe8c in child_main () #10 0x30240 in make_child () #11 0x307b8 in perform_idle_server_maintenance () #12 0x31034 in standalone_main () #13 0x31998 in main () thanks, shinta -- To unsubscribe: mailto:[EMAIL PROTECTED] For additional commands: mailto:[EMAIL PROTECTED] Troubles with the list: mailto:[EMAIL PROTECTED]
RE: Re[2]: Using Apache/Tomcat in high-traffic site
Title: RE: Re[2]: Using Apache/Tomcat in high-traffic site We are also in the process of setting an environment like this. I'm curious about your comment in the first paragraph. Assuming the environment is: the loadbalancer is load balancing Apache, there are many Apache running mod_jk, the mod_jk is loadbalancing many Tomcat behind the Apache. I'm not familiar with BigIP, Steve, does it forward requests of the same sessions to the same Apache? If your loadbalancer can't be configured to send requests of the same sessions to the same Apache (Gavin's first question), I don't think mod_jk will solve this. Unless you want to just have one Apache, and let it be the load balancer essentially. Which of course, in our environment, this is not a scalable solution. shinta -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Monday, March 19, 2001 5:36 AM To: '[EMAIL PROTECTED]'; Tim Sullivan; [EMAIL PROTECTED] Subject: Re[2]: Using Apache/Tomcat in high-traffic site I'm just putting together a prototype on the Internet using Tomcat, we've got two Linux boxes and load is spread over the two hosts using a BigIP load-balancer. The mod_jk config is ok when you understand whats going on, local and remote Tomcat instances aren't specially distinguished apart from having a different IP address. mod_jk handles the sticky session part so going in on different web servers each time is fine. mod_jk could do with better documentation in this regard and its very easy to configure it so that it does absolutely nothing and logs no errors whatsoever ! Direct all contexts to the load balancer worker not the ajp13 workers and have the load balancer worker spread things out. I'm interested in your experience of heap sizes, in my prototype I've ignored it, how would I know if I'm near - do you have a feel for how much memory you need per session (assuming application doesn't store much data in the session). SteveQ. __ Reply Separator _ Subject: RE: Using Apache/Tomcat in high-traffic site Author: Tim Sullivan [EMAIL PROTECTED] at Internet-po Date: 16/03/01 11:24 Hello Gavin, I am considering doing the same thing on almost the same hardware (e4500 + e1). Did you end up finding any performance examples or otherwise for this? If so, could you forward it my way? Thanks in advance, -Tim -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Wednesday, March 14, 2001 12:55 PM To: [EMAIL PROTECTED] Subject: Using Apache/Tomcat in high-traffic site Hi, I am looking to use Apache and Tomcat in my company's production web site. Before I can convince management that this is a good idea I need some information so I am confident. If anyone can help I'll be very grateful. First of all I'll give you a picture of the overall architecture that I want to create: 1) For fault-tolerance and scaleability we want to have several instances of Apache running over a number of machines. These will be load balanced by a pair of Cisco Local Director boxes, because we already own a couple of them. From a brief look through the operating manual on Cisco's web site I get the impression that Local Director cannot support sticky load-balancing. Please correct me if I'm wrong. 2) We will run several instances of Tomcat (version 3.2.1 most likely) on our back-end servers. These are a couple of Sun E1s with 32 processors each. These are partitioned up into domains, so that we end up with 12 processors on each machine dedicated to running Tomcat. Each domain has 2GB of RAM. There are other domains dedicated to databases, etc... 3) Each instance of Apache will be using mod_jk in a sticky load-balanced configuration. Every instance of Apache (and so mod_jk) will have workers defined for every instance of Tomcat on both E1s. In other words, a Tomcat instance may receive a request from any instance of Apache. The things I need to know :) 1) Each request to our site will be assigned seemingly at random to one of the Apache instances by the Cisco boxes. Can the sticky part of mod_jk cope with this? i.e. Will mod_jk pass the request to the correct Tomcat instance even though it does not share context information with the other Apache instances? 2) What is the best way of running Tomcat on the large E1s? Should I have just a few very large processes (say 2 x 800MB max heap) for Tomcat or should I run many smaller processes (say 16 x 100MB max heap)? Has anyone got any experience with making Tomcat (or any Java based server) scale to machines of this size and power? I want to get good performance, but I don't want to compromise stability. 3) Has anyone used Tomcat in a mission critical
RE: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: problem w/ ajp13 - if Tomcat is shutdown Yes. I did have the ajp12 enabled. Shutdown itself wasn't the problem. I verified that the Tomcat process did go away and a new one was restarted properly. I'm putting more debugging into mod_jk and I will have more information about this problem shortly. shinta -Original Message- From: Sebastian Schulz [mailto:[EMAIL PROTECTED]] Sent: Tuesday, March 06, 2001 4:18 AM To: [EMAIL PROTECTED] Subject: Re: problem w/ ajp13 - if Tomcat is shutdown did you have a connector for mod_jserv at your configs at well? if i'm not mistaken, mod_jserv is needed for tomcat to shutdown. bAs T. Shinta Tjio schrieb: I'm having problem with mod_jk if ajp13 is used. The problem is often reproduced when Tomcat is shut down without Apache being shut down. When a request is fired through Apache as soon as Tomcat starts, I often get Internal Server Error. The mod_jk.log will have the following: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13 [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 [jk_worker.c (127)]: wc_get_worker_for_name, done found a worker [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - jk_tcp_socket_recvfull failed [jk_ajp13_worker.c (619)]: Error reading request [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done If I hit reload multiple times, eventually Tomcat will serve the servlet fine. Did anyone see this problem before? Is there anyway around this? shinta
problem w/ ajp13 - if Tomcat is shutdown
Title: problem w/ ajp13 - if Tomcat is shutdown I'm having problem with mod_jk if ajp13 is used. The problem is often reproduced when Tomcat is shut down without Apache being shut down. When a request is fired through Apache as soon as Tomcat starts, I often get Internal Server Error. The mod_jk.log will have the following: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13 [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 [jk_worker.c (127)]: wc_get_worker_for_name, done found a worker [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - jk_tcp_socket_recvfull failed [jk_ajp13_worker.c (619)]: Error reading request [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done If I hit reload multiple times, eventually Tomcat will serve the servlet fine. Did anyone see this problem before? Is there anyway around this? shinta
Tomcat's class loader
Title: Tomcat's class loader Hi, all, I have a servlet, ServletFoo, that calls a utility class that lives in a jar file under $TOMCAT_HOME/lib. This utility class calls the ResourceBundle.getBundle() on a resource bundle that lives in the servlet's home (i.e: $TOMCAT_HOME/webapps/SomeContext/WEB-INF/). What I found out was that ServletFoo was loaded by Tomcat's AdaptiveClassLoader. The utility class was loaded by the system's default class loader. And of course, that utility class called ResourceBundle.getBundle() it couldn't find the resouce file because it lived in the servlet's home and the system's default class loader doesn't have CLASSPATH pointing to that directory. I have been able to fix this by passing the servlet's class loader to the utility class, so that it passes it to the ResourceBundle.getBundle(). My question is, is this the expected behavior of Tomcat's class loader? Or is this a bug? thanks, shinta
JkMount for all context
Title: JkMount for all context How do you write a JkMount that will apply to all servlet context, assuming that the URI will always have 'servlet' in it? In other words, I don't want to write: JkMount /context1/servlet/myServlet ajp13 JkMount /context2/servlet/anotherOne ajp13 JkMount /context3/servlet/aThirdOne ajp13 I tried: JkMount /*/servlet/* ajp13 but this doesn't seem to give me what I want. The above line forwards ALL requests to Tomcat, including non servlet (static pages) requests. Any suggestions? thanks, shinta
