On Nov 6, 2013, at 11:23 PM, Muhammad Ali Orakzai <m.orak...@gmail.com> wrote:
First, please don't top post. This comment is an example of top posting. Instead reply inline, like the rest of my comments, or at the end. > I am using GUI version of tomcat and heap memory has been configured to > 500M min and 1000M max. So memory size is not an issue. GC is being > performed normally. Can you elaborate on this? How do you know it's performing normally? What's in your GC log at the time of the problem? > Took a heap dump on 60% memory. Analyzing it in Eclipse > MAT showed the following leak suspect. > > One instance of *"java.util.Vector"* loaded by *"<system class > loader>"*occupies *335,247,200 > (70.86%)* bytes. The instance is referenced by > *org.apache.tomcat.util.threads.TaskThread > @ 0x1ec45d98 http-apr-8082-exec-11* , loaded by > *"org.apache.catalina.loader.StandardClassLoader > @ 0x176e0ef0"*. The memory is accumulated in one instance of > *"java.lang.Object[]"* loaded by *"<system class loader>"*. > > > This was the stack trace. > > > http-apr-8082-exec-11 > at java.util.Vector$1.nextElement()Ljava/lang/Object; (Unknown Source) > at > com.sun.org.apache.xml.internal.resolver.Catalog.resolveLocalSystem(Ljava/lang/String;)Ljava/lang/String; > (Unknown Source) > at > com.sun.org.apache.xml.internal.resolver.Catalog.resolveSystem(Ljava/lang/String;)Ljava/lang/String; > (Unknown Source) > at > com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.getResolvedEntity(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; > (Unknown Source) > at > com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lorg/xml/sax/InputSource; > (Unknown Source) > at > com.sun.xml.ws.wsdl.parser.EntityResolverWrapper.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; > (EntityResolverWrapper.java:63) > at > com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; > (RuntimeWSDLParser.java:287) > at > com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;Lcom/sun/xml/ws/api/policy/PolicyResolver;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; > (RuntimeWSDLParser.java:151) > at > com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; > (RuntimeWSDLParser.java:131) > at > com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; > (WSServiceDelegate.java:267) > at > com.sun.xml.ws.client.WSServiceDelegate.<init>(Ljavax/xml/transform/Source;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V > (WSServiceDelegate.java:230) > at > com.sun.xml.ws.client.WSServiceDelegate.<init>(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V > (WSServiceDelegate.java:178) > at > com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)Ljavax/xml/ws/spi/ServiceDelegate; > (ProviderImpl.java:106) > at javax.xml.ws.Service.<init>(Ljava/net/URL;Ljavax/xml/namespace/QName;)V > (Unknown Source) > at com.example.xmlns._1366097065072.CashRechargeServiceServiceagent.<init>()V > (CashRechargeServiceServiceagent.java:42) > at > pk.com.telenor.topup.cbs.CbsCashRecharge.cashRecharge(Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeInput;)Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeOutput; > (CbsCashRecharge.java:119) > at > pk.com.telenor.topup.cbs.CbsCashRecharge.performCardRechargePerform(Lpk/com/telenor/topup/dto/CashRechargeRequest;)Lpk/com/telenor/topup/dto/CashRechargeResponse; > (CbsCashRecharge.java:58) > at > telenor.vas.topup.TopUpWS.TopUp(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ltelenor/vas/topup/PaymentResponse; > (TopUpWS.java:409) > at > sun.reflect.GeneratedMethodAccessor129.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; > (Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; > (Unknown Source) > at > java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; > (Unknown Source) > at > com.sun.xml.ws.api.server.InstanceResolver$1.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; > (InstanceResolver.java:246) > at > com.sun.xml.ws.server.InvokerTube$2.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; > (InvokerTube.java:146) > at > com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet; > (EndpointMethodHandler.java:257) > at > com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/pipe/NextAction; > (SEIInvokerTube.java:95) > at > com.sun.xml.ws.api.pipe.Fiber.__doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube; > (Fiber.java:629) > at > com.sun.xml.ws.api.pipe.Fiber._doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube; > (Fiber.java:588) > at > com.sun.xml.ws.api.pipe.Fiber.doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube; > (Fiber.java:573) > at > com.sun.xml.ws.api.pipe.Fiber.runSync(Lcom/sun/xml/ws/api/pipe/Tube;Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet; > (Fiber.java:470) > at > com.sun.xml.ws.server.WSEndpointImpl$2.process(Lcom/sun/xml/ws/api/message/Packet;Lcom/sun/xml/ws/api/server/WebServiceContextDelegate;Lcom/sun/xml/ws/api/server/TransportBackChannel;)Lcom/sun/xml/ws/api/message/Packet; > (WSEndpointImpl.java:295) > at > com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(Lcom/sun/xml/ws/transport/http/WSHTTPConnection;)V > (HttpAdapter.java:515) > at > com.sun.xml.ws.transport.http.HttpAdapter.handle(Lcom/sun/xml/ws/transport/http/WSHTTPConnection;)V > (HttpAdapter.java:285) > at > com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(Ljavax/servlet/ServletContext;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V > (ServletAdapter.java:143) > at > com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/ServletContext;)V > (WSServletDelegate.java:155) > at > com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/ServletContext;)V > (WSServletDelegate.java:189) > at > com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V > (WSServlet.java:76) > at > javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V > (HttpServlet.java:647) > at > javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V > (HttpServlet.java:728) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V > (ApplicationFilterChain.java:305) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V > (ApplicationFilterChain.java:210) > at > org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (StandardWrapperValve.java:222) > at > org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (StandardContextValve.java:123) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (AuthenticatorBase.java:472) > at > org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (StandardHostValve.java:171) > at > org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (ErrorReportValve.java:99) > at > org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (AccessLogValve.java:936) > at > org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V > (StandardEngineValve.java:118) > at > org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V > (CoyoteAdapter.java:407) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; > (AbstractHttp11Processor.java:1004) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; > (AbstractProtocol.java:589) > at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run()V > (AprEndpoint.java:1852) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V > (Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (Unknown Source) > at java.lang.Thread.run()V (Unknown Source) > Out of curiosity, where did you obtain this stack trace? Did it come from a thread dump? If so, did you take multiple thread dumps? If so, do you see the same thread stuck in each of the thread dumps? Also, the thread dump should indicate if it's running, blocked, etc… That would be helpful to know. Lastly, what's the CPU usage like when this is stuck? > It is stuck in the web service call. My web service client is embedded > in a jar file. Jar file is being used in 5 web apps deployed on this > server. Strange thing is that similar services calls have been used > and they are working fine without any delays. Delay is always in this > web service call. Hard to say exactly what's causing the problem, but it seems pretty unlikely that it's being caused by Tomcat. Dan > > > > On Wed, Nov 6, 2013 at 8:25 PM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> Muhammad, >> >> On 11/6/13, 1:43 AM, Muhammad Ali Orakzai wrote: >>> -XX:MaxPermSize=100m >> >> You don't have any other heap-sizing parameters. Perhaps you simply >> need more memory? It used to be that the default heap sizes were very >> small (sometimes like 64MiB) and you may simply need much more. How >> much memory is the JVM actually running with given your existing >> command-line? You can check with "jinfo" from the command-line. >> >>> -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled >>> -XX:+HeapDumpOnOutOfMemoryError >> >> Have you inspected one of these? You should get one for each OOM. You >> should be able to find out what kinds of objects are filling-up your >> heap. It should give you some clue as to what kind of action you >> should take next: either raise your heap size or fix your web application. >> >>> -Dcom.sun.management.jmxremote >>> -Dcom.sun.management.jmxremote.port=9004 >>> -Dcom.sun.management.jmxremote.ssl=false >>> -Dcom.sun.management.jmxremote.authenticate=false >> >> Um... the above doesn't look like a good idea, but only you know your >> own environment. >> >> - -chris >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.15 (Darwin) >> Comment: GPGTools - http://gpgtools.org >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iQIcBAEBCAAGBQJSel9OAAoJEBzwKT+lPKRYmDQP/3ZcHDmotCo1ZdX2IKMlfHAZ >> kHHZuuU62uAXhMDABYn768M0jKqs90P5yBPcb09PYxB0ZuPBSgq5hgANWAD/mcIc >> l558lg5LAc5yfbZedgLZzuMA+13/G/z8jBnND1j1S0psaMYA8LVk3ExidxwXbBB+ >> WEpRCZNhMovZkL2rHxO37vYGP0DNUtREjpVs+S4fWOJCYRrQmajfBwMttTVl36U/ >> 2md195/N0JccdCa119kV8WACIr5jeDXa6al3raP+dC893SUW2pZB/M6Wxk6eLND0 >> q8bIRjGuzfqZNn+tupNKKJqakT5kW8xASMrjUKKTq0/l6+nHD3DDSMa8Qi0YyWFD >> VL3YuW7IYCpGjniCWrXAeB5KxueS7OUoSq5EM290fs51ZYSY/LF5Stb7ybEAAL9a >> e4oisal9u/Jc3ZRsH3hgJDJusgAL7lF1GCOW/DVf5pvbrL75SdslAO67j92fJ3GI >> 9YtBQucpYLNvfUyqf+fOde5bcdECImmo2nM7fOt6fmd9rQBPNLC+holwvbixqfZO >> /Ny6e6NrRfxvHHw45/T0xE0IjUufSpmQQ0/n5hISg2YUtoEfIUqvc3Xv7FCEa+H4 >> hlDapQmJPGhfPWZhJSN8DqiGVHMBf/dUcyHfCeIb4Hs1FczX7/EXel6Pwn86N2MZ >> 3LDowe2VrWhevlfDBL07 >> =bzZV >> -----END PGP SIGNATURE----- >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
