So did you try changing your version of tomcat?
> Subject: RE: Interesting 4.2.1. Issue... > From: a...@opencloud.net.au > To: dev@cloudstack.apache.org > Date: Fri, 4 Apr 2014 10:35:42 +0800 > > cd /usr/share/tomcat6/bin/ > ./version.sh > > The output should be 6.0.33 instead of 6.0.35 > > Using CATALINA_BASE: /usr/share/tomcat6 > Using CATALINA_HOME: /usr/share/tomcat6 > Using CATALINA_TMPDIR: /usr/share/tomcat6/temp > Using JRE_HOME: /usr > Using CLASSPATH: /usr/share/tomcat6/bin/bootstrap.jar > Server version: Apache Tomcat/6.0.35 > Server built: > Server number: 6.0.35.0 > OS Name: Linux > OS Version: 3.11.0-18-generic > Architecture: amd64 > JVM Version: 1.6.0_30-b30 > JVM Vendor: Sun Microsystems Inc. > > > Kind Regards > Amin > > > > -----Original Message----- > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > Sent: Friday, 4 April 2014 10:31 AM > To: dev@cloudstack.apache.org > Subject: RE: Interesting 4.2.1. Issue... > > I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for the > next few days to see if we get the error again. > Do you know any way way to verify the version of tomcat that's running? > > > Subject: RE: Interesting 4.2.1. Issue... > > From: a...@opencloud.net.au > > To: dev@cloudstack.apache.org > > Date: Thu, 3 Apr 2014 10:35:29 +0800 > > > > No we didn't, it wouldn't matter because the memory would still fill > > up, the problem is it opens a thread and it fails to close it so > > whatever you will increase soon or later the memory will fill up (if I > > understand right) > > > > The error in catalina is as follows: > > > > SEVERE: The web application [/client] created a ThreadLocal with key of > > type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1bd66d2d]) and a > > value of type [com.cloud.api.SerializationContext] (value > > [com.cloud.api.SerializationContext@2f6baed9]) but failed to remove it when > > the web application was stopped. This is very likely to create a memory > > leak. > > > > If someone could help with this error generated in the catalina log, that > > would be much appreicated. > > > > Kind Regards > > Amin > > > > > > > > > > -----Original Message----- > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > Sent: Thursday, 3 April 2014 9:34 AM > > To: dev@cloudstack.apache.org > > Subject: RE: Interesting 4.2.1. Issue... > > > > A few other articles also mention setting the initial heap size "-Xms" to > > the same value as the heap size, to go ahead and fully commit the heap. > > Have you tried that? > > One other thing I am curious of is have you fiddled with the Perm Size > > "XX:Permsize" setting? > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > From: a...@opencloud.net.au > > > To: dev@cloudstack.apache.org > > > Date: Thu, 3 Apr 2014 09:26:31 +0800 > > > > > > Believe or not!! We have tried setting them in both formats and > > > still the catalina log produces java heap space error > > > > > > Kind Regards > > > Amin > > > > > > > > > > > > -----Original Message----- > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > Sent: Thursday, 3 April 2014 9:24 AM > > > To: dev@cloudstack.apache.org > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > This may be a silly question, but in all of the docs I am reading online > > > in regards to increasing the java heap size, they are specifying it as > > > -Xmx"size""MB" example -Xmx2048m vs -Xmx2g Is it possible that it's not > > > reading the 2g variable as 2GB? > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > From: a...@opencloud.net.au > > > > To: dev@cloudstack.apache.org > > > > Date: Thu, 3 Apr 2014 09:06:17 +0800 > > > > > > > > It is 6.0.35 and it still produces this error, even after increasing > > > > the Xmx to 4g, we have installed tomcat 6.0.33 and each time we install > > > > the cloudstack it does not sense the installed 6.0.33 and attempts to > > > > install 6.0.35 as it is dependent on it. Silly solution is that we > > > > scheduled a daily restart @ 2PM through a cron job!!!! But first you > > > > have to "killall jvsc" then restart the management server. > > > > > > > > What we are considering is to migrate the management server to CentOS > > > > 6.5 it comes with tomcat 6.0.24 and mysql 5.1, we attempted to restore > > > > the dump on a pilot environment and it worked. > > > > > > > > If someone else has a better solution than this would you please share > > > > it? > > > > > > > > Kind Regards > > > > Amin > > > > > > > > -----Original Message----- > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > Sent: Thursday, 3 April 2014 5:31 AM > > > > To: dev@cloudstack.apache.org > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > So I just checked my tomcat version and we are running 6.0.35 which > > > > must be the default that comes with Ubuntu 12.04 out of the box. Our > > > > memory settings are as follows: > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx4g > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M > > > > -XX:MaxPermSize=800m" > > > > So what version of tomcat are you running 6.0.35 or 6.0.33? > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > From: a...@opencloud.net.au > > > > > To: dev@cloudstack.apache.org > > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800 > > > > > > > > > > We have the same issue, after an upgrade from 2.2.14 to 4.2.1, > > > > > and during this upgrade we had upgrade from Ubuntu 10 LTS to > > > > > Ubuntu 12 LTS, it seems it related to tomcat 6.0.35, because it > > > > > is recommended to have tomcat 6.0.33 which doesn't come by > > > > > default with Ubuntu > > > > > 12.04.4 > > > > > > > > > > Kind Regards > > > > > Amin > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Marcus [mailto:shadow...@gmail.com] > > > > > Sent: Tuesday, 1 April 2014 6:22 AM > > > > > To: dev@cloudstack.apache.org > > > > > Subject: Re: Interesting 4.2.1. Issue... > > > > > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any issues > > > > > like that. You can send along your memory settings... here's > > > > > what I'm > > > > > running: > > > > > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m > > > > > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips > > > > > <mphilli7...@hotmail.com> wrote: > > > > > > So I have a redundant pair of management servers running on 4.2.1. > > > > > > At least once a day one of the management servers crashes and the > > > > > > log gets filled with the following messages: > > > > > > java.lang.OutOfMemoryError: Java heap > > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused by: > > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused by: > > > > > > javax.net.ssl.SSLHandshakeException: Remote host closed > > > > > > connection during handshake and there are a few others. When > > > > > > the one management server tanks, although the other management > > > > > > server is up and active, it won't actually process any UI > > > > > > commands until the crashed server is restarted. Example of > > > > > > won't process any UI command is create a new instance, create > > > > > > a new account, etc. After doing some searching I have found > > > > > > that others have noticed java heap errors in 4.2.1, and the > > > > > > suggested fix is to increase the heap size. I am planning on > > > > > > increasing it from 2g to 4g, however if the problem is > > > > > > something like a memory leak, then increasing the heap size > > > > > > will just delay the inevitable. Has anyone else fixed this > > > > > > issue by increasing the heap size? Or what is the recommended > > > > > > value? **updated...as expected I increased the heap size from > > > > > > 2g to 4g, and it just too > > > > > k longer for the problem to reoccur... > > > > > > In my honest opinion of bigger concern is the fact that when one > > > > > > management server crashes the other stops functioning as well. So > > > > > > this begs the question of why even bother with a redundant pair of > > > > > > servers..Anybody else experience this issue? I would love to hear > > > > > > any dev guys opinion on this.... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
