I am using GUI version of tomcat and heap memory has been configured to 500M min and 1000M max. So memory size is not an issue. GC is being performed normally. Took a heap dump on 60% memory. Analyzing it in Eclipse MAT showed the following leak suspect.
One instance of *"java.util.Vector"* loaded by *"<system class loader>"*occupies *335,247,200 (70.86%)* bytes. The instance is referenced by *org.apache.tomcat.util.threads.TaskThread @ 0x1ec45d98 http-apr-8082-exec-11* , loaded by *"org.apache.catalina.loader.StandardClassLoader @ 0x176e0ef0"*. The memory is accumulated in one instance of *"java.lang.Object[]"* loaded by *"<system class loader>"*. This was the stack trace. http-apr-8082-exec-11 at java.util.Vector$1.nextElement()Ljava/lang/Object; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.Catalog.resolveLocalSystem(Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.Catalog.resolveSystem(Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.getResolvedEntity(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lorg/xml/sax/InputSource; (Unknown Source) at com.sun.xml.ws.wsdl.parser.EntityResolverWrapper.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; (EntityResolverWrapper.java:63) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; (RuntimeWSDLParser.java:287) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;Lcom/sun/xml/ws/api/policy/PolicyResolver;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (RuntimeWSDLParser.java:151) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (RuntimeWSDLParser.java:131) at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (WSServiceDelegate.java:267) at com.sun.xml.ws.client.WSServiceDelegate.<init>(Ljavax/xml/transform/Source;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V (WSServiceDelegate.java:230) at com.sun.xml.ws.client.WSServiceDelegate.<init>(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V (WSServiceDelegate.java:178) at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)Ljavax/xml/ws/spi/ServiceDelegate; (ProviderImpl.java:106) at javax.xml.ws.Service.<init>(Ljava/net/URL;Ljavax/xml/namespace/QName;)V (Unknown Source) at com.example.xmlns._1366097065072.CashRechargeServiceServiceagent.<init>()V (CashRechargeServiceServiceagent.java:42) at pk.com.telenor.topup.cbs.CbsCashRecharge.cashRecharge(Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeInput;)Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeOutput; (CbsCashRecharge.java:119) at pk.com.telenor.topup.cbs.CbsCashRecharge.performCardRechargePerform(Lpk/com/telenor/topup/dto/CashRechargeRequest;)Lpk/com/telenor/topup/dto/CashRechargeResponse; (CbsCashRecharge.java:58) at telenor.vas.topup.TopUpWS.TopUp(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ltelenor/vas/topup/PaymentResponse; (TopUpWS.java:409) at sun.reflect.GeneratedMethodAccessor129.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (InstanceResolver.java:246) at com.sun.xml.ws.server.InvokerTube$2.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (InvokerTube.java:146) at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet; (EndpointMethodHandler.java:257) at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/pipe/NextAction; (SEIInvokerTube.java:95) at com.sun.xml.ws.api.pipe.Fiber.__doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube; (Fiber.java:629) at com.sun.xml.ws.api.pipe.Fiber._doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube; (Fiber.java:588) at com.sun.xml.ws.api.pipe.Fiber.doRun(Lcom/sun/xml/ws/api/pipe/Tube;)Lcom/sun/xml/ws/api/pipe/Tube; (Fiber.java:573) at com.sun.xml.ws.api.pipe.Fiber.runSync(Lcom/sun/xml/ws/api/pipe/Tube;Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet; (Fiber.java:470) at com.sun.xml.ws.server.WSEndpointImpl$2.process(Lcom/sun/xml/ws/api/message/Packet;Lcom/sun/xml/ws/api/server/WebServiceContextDelegate;Lcom/sun/xml/ws/api/server/TransportBackChannel;)Lcom/sun/xml/ws/api/message/Packet; (WSEndpointImpl.java:295) at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(Lcom/sun/xml/ws/transport/http/WSHTTPConnection;)V (HttpAdapter.java:515) at com.sun.xml.ws.transport.http.HttpAdapter.handle(Lcom/sun/xml/ws/transport/http/WSHTTPConnection;)V (HttpAdapter.java:285) at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(Ljavax/servlet/ServletContext;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (ServletAdapter.java:143) at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/ServletContext;)V (WSServletDelegate.java:155) at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljavax/servlet/ServletContext;)V (WSServletDelegate.java:189) at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (WSServlet.java:76) at javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (HttpServlet.java:647) at javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (AccessLogValve.java:936) at org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (AbstractHttp11Processor.java:1004) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (AbstractProtocol.java:589) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run()V (AprEndpoint.java:1852) at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (Unknown Source) at java.lang.Thread.run()V (Unknown Source) It is stuck in the web service call. My web service client is embedded in a jar file. Jar file is being used in 5 web apps deployed on this server. Strange thing is that similar services calls have been used and they are working fine without any delays. Delay is always in this web service call. On Wed, Nov 6, 2013 at 8:25 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Muhammad, > > On 11/6/13, 1:43 AM, Muhammad Ali Orakzai wrote: > > -XX:MaxPermSize=100m > > You don't have any other heap-sizing parameters. Perhaps you simply > need more memory? It used to be that the default heap sizes were very > small (sometimes like 64MiB) and you may simply need much more. How > much memory is the JVM actually running with given your existing > command-line? You can check with "jinfo" from the command-line. > > > -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled > > -XX:+HeapDumpOnOutOfMemoryError > > Have you inspected one of these? You should get one for each OOM. You > should be able to find out what kinds of objects are filling-up your > heap. It should give you some clue as to what kind of action you > should take next: either raise your heap size or fix your web application. > > > -Dcom.sun.management.jmxremote > > -Dcom.sun.management.jmxremote.port=9004 > > -Dcom.sun.management.jmxremote.ssl=false > > -Dcom.sun.management.jmxremote.authenticate=false > > Um... the above doesn't look like a good idea, but only you know your > own environment. > > - -chris > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.15 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJSel9OAAoJEBzwKT+lPKRYmDQP/3ZcHDmotCo1ZdX2IKMlfHAZ > kHHZuuU62uAXhMDABYn768M0jKqs90P5yBPcb09PYxB0ZuPBSgq5hgANWAD/mcIc > l558lg5LAc5yfbZedgLZzuMA+13/G/z8jBnND1j1S0psaMYA8LVk3ExidxwXbBB+ > WEpRCZNhMovZkL2rHxO37vYGP0DNUtREjpVs+S4fWOJCYRrQmajfBwMttTVl36U/ > 2md195/N0JccdCa119kV8WACIr5jeDXa6al3raP+dC893SUW2pZB/M6Wxk6eLND0 > q8bIRjGuzfqZNn+tupNKKJqakT5kW8xASMrjUKKTq0/l6+nHD3DDSMa8Qi0YyWFD > VL3YuW7IYCpGjniCWrXAeB5KxueS7OUoSq5EM290fs51ZYSY/LF5Stb7ybEAAL9a > e4oisal9u/Jc3ZRsH3hgJDJusgAL7lF1GCOW/DVf5pvbrL75SdslAO67j92fJ3GI > 9YtBQucpYLNvfUyqf+fOde5bcdECImmo2nM7fOt6fmd9rQBPNLC+holwvbixqfZO > /Ny6e6NrRfxvHHw45/T0xE0IjUufSpmQQ0/n5hISg2YUtoEfIUqvc3Xv7FCEa+H4 > hlDapQmJPGhfPWZhJSN8DqiGVHMBf/dUcyHfCeIb4Hs1FczX7/EXel6Pwn86N2MZ > 3LDowe2VrWhevlfDBL07 > =bzZV > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
