Re: Web Service Client Response when Old Gen is 100%
On Nov 6, 2013, at 11:23 PM, Muhammad Ali Orakzai m.orak...@gmail.com wrote: First, please don't top post. This comment is an example of top posting. Instead reply inline, like the rest of my comments, or at the end. I am using GUI version of tomcat and heap memory has been configured to 500M min and 1000M max. So memory size is not an issue. GC is being performed normally. Can you elaborate on this? How do you know it's performing normally? What's in your GC log at the time of the problem? Took a heap dump on 60% memory. Analyzing it in Eclipse MAT showed the following leak suspect. One instance of *java.util.Vector* loaded by *system class loader*occupies *335,247,200 (70.86%)* bytes. The instance is referenced by *org.apache.tomcat.util.threads.TaskThread @ 0x1ec45d98 http-apr-8082-exec-11* , loaded by *org.apache.catalina.loader.StandardClassLoader @ 0x176e0ef0*. The memory is accumulated in one instance of *java.lang.Object[]* loaded by *system class loader*. This was the stack trace. http-apr-8082-exec-11 at java.util.Vector$1.nextElement()Ljava/lang/Object; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.Catalog.resolveLocalSystem(Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.Catalog.resolveSystem(Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.getResolvedEntity(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lorg/xml/sax/InputSource; (Unknown Source) at com.sun.xml.ws.wsdl.parser.EntityResolverWrapper.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; (EntityResolverWrapper.java:63) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; (RuntimeWSDLParser.java:287) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;Lcom/sun/xml/ws/api/policy/PolicyResolver;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (RuntimeWSDLParser.java:151) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (RuntimeWSDLParser.java:131) at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (WSServiceDelegate.java:267) at com.sun.xml.ws.client.WSServiceDelegate.init(Ljavax/xml/transform/Source;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V (WSServiceDelegate.java:230) at com.sun.xml.ws.client.WSServiceDelegate.init(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V (WSServiceDelegate.java:178) at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)Ljavax/xml/ws/spi/ServiceDelegate; (ProviderImpl.java:106) at javax.xml.ws.Service.init(Ljava/net/URL;Ljavax/xml/namespace/QName;)V (Unknown Source) at com.example.xmlns._1366097065072.CashRechargeServiceServiceagent.init()V (CashRechargeServiceServiceagent.java:42) at pk.com.telenor.topup.cbs.CbsCashRecharge.cashRecharge(Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeInput;)Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeOutput; (CbsCashRecharge.java:119) at pk.com.telenor.topup.cbs.CbsCashRecharge.performCardRechargePerform(Lpk/com/telenor/topup/dto/CashRechargeRequest;)Lpk/com/telenor/topup/dto/CashRechargeResponse; (CbsCashRecharge.java:58) at telenor.vas.topup.TopUpWS.TopUp(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ltelenor/vas/topup/PaymentResponse; (TopUpWS.java:409) at sun.reflect.GeneratedMethodAccessor129.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (InstanceResolver.java:246) at
Re: Web Service Client Response when Old Gen is 100%
On Wed, Nov 6, 2013 at 1:43 AM, Muhammad Ali Orakzai m.orak...@gmail.comwrote: I am using the following environment Windows Server 2003 32 bit How much RAM? Also, did you configure your virtual memory (or paging file) settings? Apache Tomcat 7.0.27 Others on the list will/may recommend you to upgrade to a newer version of Tomcat 7.0.x for security fixes/updates and bug fixes, but should not be necessary to fix this issue of yours. We have created a SOAP based web service which is calling 2 external SOAP services. We were getting OutOfMemoryError which was resloved by making the service object static. interesting. are you recycling or releasing memory to GC, or are you caching (or never releasing) data from your app... every time you call 2 external SOAP services? i think you may need to refactor your code/app to release memory at some point, if it's not doing that already. at your earliest convenience, read this post below: How to Fix Memory Leaks in Java[1] Now whenever the server memory reaches 100% one of the external service response is too slow (takes about 2-3 minutes). okay. Restarting tomcat resolves the issue but this issue is occurring 25-26 hours interval. i can definitely understand that this is not acceptable. I am using the following jvm settings -XX:MaxPermSize=100m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled In my app, I use the following: -Xms1024m -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled there are strategies on how to use -Xms and -Xmx to improve performance (and/or prevent out of memory error). you may want to research that a bit. [1] http://java.dzone.com/news/how-fix-memory-leaks-java
Re: Web Service Client Response when Old Gen is 100%
On Nov 6, 2013, at 1:43 AM, Muhammad Ali Orakzai m.orak...@gmail.com wrote: Hi, I am using the following environment Windows Server 2003 32 bit Apache Tomcat 7.0.27 Upgrade :) You're 20 versions out-of-date. Netbeans IDE 7.2 Metro 2.0 We have created a SOAP based web service which is calling 2 external SOAP services. We were getting OutOfMemoryError which was resloved by making the service object static. Now whenever the server memory reaches 100% one of the external service response is too slow (takes about 2-3 minutes). Sounds like a GC issue. I see you have GC logging enabled, what do you see in E:\topupfix\GC.log? Are you in the midst of a full gc? Also, while the response is hanging (i.e. during the 2-3 minute window) take some thread dumps (two or three is sufficient). Check them to see what is happening in the application during that time period. Dan Restarting tomcat resolves the issue but this issue is occurring 25-26 hours interval. I am using the following jvm settings -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer\endorsed -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer\conf\logging.properties -XX:MaxPermSize=100m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=E:\topupfix\ -Xloggc:E:\topupfix\GC.log -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Web Service Client Response when Old Gen is 100%
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Muhammad, On 11/6/13, 1:43 AM, Muhammad Ali Orakzai wrote: -XX:MaxPermSize=100m You don't have any other heap-sizing parameters. Perhaps you simply need more memory? It used to be that the default heap sizes were very small (sometimes like 64MiB) and you may simply need much more. How much memory is the JVM actually running with given your existing command-line? You can check with jinfo from the command-line. -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError Have you inspected one of these? You should get one for each OOM. You should be able to find out what kinds of objects are filling-up your heap. It should give you some clue as to what kind of action you should take next: either raise your heap size or fix your web application. -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false Um... the above doesn't look like a good idea, but only you know your own environment. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.15 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSel9OAAoJEBzwKT+lPKRYmDQP/3ZcHDmotCo1ZdX2IKMlfHAZ kHHZuuU62uAXhMDABYn768M0jKqs90P5yBPcb09PYxB0ZuPBSgq5hgANWAD/mcIc l558lg5LAc5yfbZedgLZzuMA+13/G/z8jBnND1j1S0psaMYA8LVk3ExidxwXbBB+ WEpRCZNhMovZkL2rHxO37vYGP0DNUtREjpVs+S4fWOJCYRrQmajfBwMttTVl36U/ 2md195/N0JccdCa119kV8WACIr5jeDXa6al3raP+dC893SUW2pZB/M6Wxk6eLND0 q8bIRjGuzfqZNn+tupNKKJqakT5kW8xASMrjUKKTq0/l6+nHD3DDSMa8Qi0YyWFD VL3YuW7IYCpGjniCWrXAeB5KxueS7OUoSq5EM290fs51ZYSY/LF5Stb7ybEAAL9a e4oisal9u/Jc3ZRsH3hgJDJusgAL7lF1GCOW/DVf5pvbrL75SdslAO67j92fJ3GI 9YtBQucpYLNvfUyqf+fOde5bcdECImmo2nM7fOt6fmd9rQBPNLC+holwvbixqfZO /Ny6e6NrRfxvHHw45/T0xE0IjUufSpmQQ0/n5hISg2YUtoEfIUqvc3Xv7FCEa+H4 hlDapQmJPGhfPWZhJSN8DqiGVHMBf/dUcyHfCeIb4Hs1FczX7/EXel6Pwn86N2MZ 3LDowe2VrWhevlfDBL07 =bzZV -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Web Service Client Response when Old Gen is 100%
I am using GUI version of tomcat and heap memory has been configured to 500M min and 1000M max. So memory size is not an issue. GC is being performed normally. Took a heap dump on 60% memory. Analyzing it in Eclipse MAT showed the following leak suspect. One instance of *java.util.Vector* loaded by *system class loader*occupies *335,247,200 (70.86%)* bytes. The instance is referenced by *org.apache.tomcat.util.threads.TaskThread @ 0x1ec45d98 http-apr-8082-exec-11* , loaded by *org.apache.catalina.loader.StandardClassLoader @ 0x176e0ef0*. The memory is accumulated in one instance of *java.lang.Object[]* loaded by *system class loader*. This was the stack trace. http-apr-8082-exec-11 at java.util.Vector$1.nextElement()Ljava/lang/Object; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.Catalog.resolveLocalSystem(Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.Catalog.resolveSystem(Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.getResolvedEntity(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; (Unknown Source) at com.sun.org.apache.xml.internal.resolver.tools.CatalogResolver.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lorg/xml/sax/InputSource; (Unknown Source) at com.sun.xml.ws.wsdl.parser.EntityResolverWrapper.resolveEntity(Ljava/lang/String;Ljava/lang/String;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; (EntityResolverWrapper.java:63) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/api/wsdl/parser/XMLEntityResolver$Parser; (RuntimeWSDLParser.java:287) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;Lcom/sun/xml/ws/api/policy/PolicyResolver;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (RuntimeWSDLParser.java:151) at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(Ljava/net/URL;Ljavax/xml/transform/Source;Lorg/xml/sax/EntityResolver;ZLcom/sun/xml/ws/api/server/Container;[Lcom/sun/xml/ws/api/wsdl/parser/WSDLParserExtension;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (RuntimeWSDLParser.java:131) at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(Ljava/net/URL;Ljavax/xml/transform/Source;)Lcom/sun/xml/ws/model/wsdl/WSDLModelImpl; (WSServiceDelegate.java:267) at com.sun.xml.ws.client.WSServiceDelegate.init(Ljavax/xml/transform/Source;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V (WSServiceDelegate.java:230) at com.sun.xml.ws.client.WSServiceDelegate.init(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)V (WSServiceDelegate.java:178) at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(Ljava/net/URL;Ljavax/xml/namespace/QName;Ljava/lang/Class;)Ljavax/xml/ws/spi/ServiceDelegate; (ProviderImpl.java:106) at javax.xml.ws.Service.init(Ljava/net/URL;Ljavax/xml/namespace/QName;)V (Unknown Source) at com.example.xmlns._1366097065072.CashRechargeServiceServiceagent.init()V (CashRechargeServiceServiceagent.java:42) at pk.com.telenor.topup.cbs.CbsCashRecharge.cashRecharge(Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeInput;)Lpk/telenor/eil/cashrecharge_service/messages/cash_recharge/v1/CashRechargeOutput; (CbsCashRecharge.java:119) at pk.com.telenor.topup.cbs.CbsCashRecharge.performCardRechargePerform(Lpk/com/telenor/topup/dto/CashRechargeRequest;)Lpk/com/telenor/topup/dto/CashRechargeResponse; (CbsCashRecharge.java:58) at telenor.vas.topup.TopUpWS.TopUp(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ltelenor/vas/topup/PaymentResponse; (TopUpWS.java:409) at sun.reflect.GeneratedMethodAccessor129.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (InstanceResolver.java:246) at com.sun.xml.ws.server.InvokerTube$2.invoke(Lcom/sun/xml/ws/api/message/Packet;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (InvokerTube.java:146) at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/message/Packet; (EndpointMethodHandler.java:257) at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(Lcom/sun/xml/ws/api/message/Packet;)Lcom/sun/xml/ws/api/pipe/NextAction; (SEIInvokerTube.java:95) at
Web Service Client Response when Old Gen is 100%
Hi, I am using the following environment Windows Server 2003 32 bit Apache Tomcat 7.0.27 Netbeans IDE 7.2 Metro 2.0 We have created a SOAP based web service which is calling 2 external SOAP services. We were getting OutOfMemoryError which was resloved by making the service object static. Now whenever the server memory reaches 100% one of the external service response is too slow (takes about 2-3 minutes). Restarting tomcat resolves the issue but this issue is occurring 25-26 hours interval. I am using the following jvm settings -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer\endorsed -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 7.0_Tomcat7Installer\conf\logging.properties -XX:MaxPermSize=100m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=E:\topupfix\ -Xloggc:E:\topupfix\GC.log -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
