Gotcha... What hypervisor are you using with CS?
> Subject: RE: Interesting 4.2.1. Issue... > From: a...@opencloud.net.au > To: dev@cloudstack.apache.org > Date: Fri, 11 Apr 2014 09:05:29 +0800 > > Michael, > Just give me two days, will feedback you. I have to leave it till late night > where is the Ubuntu server crash times. > > Kind Regards > Amin > > -----Original Message----- > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > Sent: Thursday, 10 April 2014 11:10 PM > To: dev@cloudstack.apache.org > Subject: RE: Interesting 4.2.1. Issue... > > Amin, > Any luck on your project taking your 3 mgmt servers to 4.3 on Centos 6.5? > > > Subject: RE: Interesting 4.2.1. Issue... > > From: a...@opencloud.net.au > > To: dev@cloudstack.apache.org > > Date: Fri, 4 Apr 2014 11:10:28 +0800 > > > > Thanks and thanks for sharing the steps > > > > Kind Regards > > Amin > > > > -----Original Message----- > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > Sent: Friday, 4 April 2014 11:02 AM > > To: dev@cloudstack.apache.org > > Subject: RE: Interesting 4.2.1. Issue... > > > > So I manually downloaded tomcat 6.0.33 > > herehttps://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+up+ > > CloudStack+Development+Environment+on+Linux > > Then did the following1. extracted 6.0.33 to /usr/share/tomcat6.0.33. > > 2. Changed symlink of /usr/share/cloudstack-managemet/bin to > > /usr/share/tomcat6.0.33/bin3. Changed symlink of > > /usr/share/cloudstack-management/lib to /usr/share/tomcat6.0.33/lib4. > > Verified cloudstack was running tomcat 6.0.33 by creating a > > tomcat_version.jsp file in > > /usr/share/cloudstack-management/webapps/client > > code for tomcat_version.jsp can be found > > herehttp://stackoverflow.com/questions/14925073/how-to-find-out-runnin > > g-tomcat-version I'll definitely let you know how it goes... > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > From: a...@opencloud.net.au > > > To: dev@cloudstack.apache.org > > > Date: Fri, 4 Apr 2014 10:43:35 +0800 > > > > > > I tried but I failed to do so, each time cloudstack attempts to install > > > to go fetches the 6.0.35 from the repo, maybe you have installed it after > > > installing the cloudstack, if you managed to have a running cloudstack > > > version above the 6.0.33 feedback with the results. > > > > > > Kind Regards > > > Amin > > > > > > -----Original Message----- > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > Sent: Friday, 4 April 2014 10:41 AM > > > To: dev@cloudstack.apache.org > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > So did you try changing your version of tomcat? > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > From: a...@opencloud.net.au > > > > To: dev@cloudstack.apache.org > > > > Date: Fri, 4 Apr 2014 10:35:42 +0800 > > > > > > > > cd /usr/share/tomcat6/bin/ > > > > ./version.sh > > > > > > > > The output should be 6.0.33 instead of 6.0.35 > > > > > > > > Using CATALINA_BASE: /usr/share/tomcat6 > > > > Using CATALINA_HOME: /usr/share/tomcat6 > > > > Using CATALINA_TMPDIR: /usr/share/tomcat6/temp > > > > Using JRE_HOME: /usr > > > > Using CLASSPATH: /usr/share/tomcat6/bin/bootstrap.jar > > > > Server version: Apache Tomcat/6.0.35 > > > > Server built: > > > > Server number: 6.0.35.0 > > > > OS Name: Linux > > > > OS Version: 3.11.0-18-generic > > > > Architecture: amd64 > > > > JVM Version: 1.6.0_30-b30 > > > > JVM Vendor: Sun Microsystems Inc. > > > > > > > > > > > > Kind Regards > > > > Amin > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > Sent: Friday, 4 April 2014 10:31 AM > > > > To: dev@cloudstack.apache.org > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for > > > > the next few days to see if we get the error again. > > > > Do you know any way way to verify the version of tomcat that's running? > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > From: a...@opencloud.net.au > > > > > To: dev@cloudstack.apache.org > > > > > Date: Thu, 3 Apr 2014 10:35:29 +0800 > > > > > > > > > > No we didn't, it wouldn't matter because the memory would still > > > > > fill up, the problem is it opens a thread and it fails to close > > > > > it so whatever you will increase soon or later the memory will > > > > > fill up (if I understand right) > > > > > > > > > > The error in catalina is as follows: > > > > > > > > > > SEVERE: The web application [/client] created a ThreadLocal with key > > > > > of type [java.lang.ThreadLocal] (value > > > > > [java.lang.ThreadLocal@1bd66d2d]) and a value of type > > > > > [com.cloud.api.SerializationContext] (value > > > > > [com.cloud.api.SerializationContext@2f6baed9]) but failed to remove > > > > > it when the web application was stopped. This is very likely to > > > > > create a memory leak. > > > > > > > > > > If someone could help with this error generated in the catalina log, > > > > > that would be much appreicated. > > > > > > > > > > Kind Regards > > > > > Amin > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > > Sent: Thursday, 3 April 2014 9:34 AM > > > > > To: dev@cloudstack.apache.org > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > > > A few other articles also mention setting the initial heap size > > > > > "-Xms" to the same value as the heap size, to go ahead and fully > > > > > commit the heap. Have you tried that? > > > > > One other thing I am curious of is have you fiddled with the Perm > > > > > Size "XX:Permsize" setting? > > > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > From: a...@opencloud.net.au > > > > > > To: dev@cloudstack.apache.org > > > > > > Date: Thu, 3 Apr 2014 09:26:31 +0800 > > > > > > > > > > > > Believe or not!! We have tried setting them in both formats > > > > > > and still the catalina log produces java heap space error > > > > > > > > > > > > Kind Regards > > > > > > Amin > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > > > Sent: Thursday, 3 April 2014 9:24 AM > > > > > > To: dev@cloudstack.apache.org > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > > > > > This may be a silly question, but in all of the docs I am reading > > > > > > online in regards to increasing the java heap size, they are > > > > > > specifying it as -Xmx"size""MB" example -Xmx2048m vs -Xmx2g Is it > > > > > > possible that it's not reading the 2g variable as 2GB? > > > > > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > From: a...@opencloud.net.au > > > > > > > To: dev@cloudstack.apache.org > > > > > > > Date: Thu, 3 Apr 2014 09:06:17 +0800 > > > > > > > > > > > > > > It is 6.0.35 and it still produces this error, even after > > > > > > > increasing the Xmx to 4g, we have installed tomcat 6.0.33 and > > > > > > > each time we install the cloudstack it does not sense the > > > > > > > installed 6.0.33 and attempts to install 6.0.35 as it is > > > > > > > dependent on it. Silly solution is that we scheduled a daily > > > > > > > restart @ 2PM through a cron job!!!! But first you have to > > > > > > > "killall jvsc" then restart the management server. > > > > > > > > > > > > > > What we are considering is to migrate the management server to > > > > > > > CentOS 6.5 it comes with tomcat 6.0.24 and mysql 5.1, we > > > > > > > attempted to restore the dump on a pilot environment and it > > > > > > > worked. > > > > > > > > > > > > > > If someone else has a better solution than this would you please > > > > > > > share it? > > > > > > > > > > > > > > Kind Regards > > > > > > > Amin > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > > > > Sent: Thursday, 3 April 2014 5:31 AM > > > > > > > To: dev@cloudstack.apache.org > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > > > > > > > So I just checked my tomcat version and we are running 6.0.35 > > > > > > > which must be the default that comes with Ubuntu 12.04 out of the > > > > > > > box. Our memory settings are as follows: > > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx4g > > > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ > > > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m" > > > > > > > So what version of tomcat are you running 6.0.35 or 6.0.33? > > > > > > > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > From: a...@opencloud.net.au > > > > > > > > To: dev@cloudstack.apache.org > > > > > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800 > > > > > > > > > > > > > > > > We have the same issue, after an upgrade from 2.2.14 to > > > > > > > > 4.2.1, and during this upgrade we had upgrade from Ubuntu > > > > > > > > 10 LTS to Ubuntu 12 LTS, it seems it related to tomcat > > > > > > > > 6.0.35, because it is recommended to have tomcat 6.0.33 > > > > > > > > which doesn't come by default with Ubuntu > > > > > > > > 12.04.4 > > > > > > > > > > > > > > > > Kind Regards > > > > > > > > Amin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Marcus [mailto:shadow...@gmail.com] > > > > > > > > Sent: Tuesday, 1 April 2014 6:22 AM > > > > > > > > To: dev@cloudstack.apache.org > > > > > > > > Subject: Re: Interesting 4.2.1. Issue... > > > > > > > > > > > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any > > > > > > > > issues like that. You can send along your memory > > > > > > > > settings... here's what I'm > > > > > > > > running: > > > > > > > > > > > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g > > > > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ > > > > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m > > > > > > > > > > > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips > > > > > > > > <mphilli7...@hotmail.com> wrote: > > > > > > > > > So I have a redundant pair of management servers running on > > > > > > > > > 4.2.1. At least once a day one of the management servers > > > > > > > > > crashes and the log gets filled with the following messages: > > > > > > > > > java.lang.OutOfMemoryError: Java heap > > > > > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused by: > > > > > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused > > > > > > > > > by: > > > > > > > > > javax.net.ssl.SSLHandshakeException: Remote host closed > > > > > > > > > connection during handshake and there are a few others. > > > > > > > > > When the one management server tanks, although the other > > > > > > > > > management server is up and active, it won't actually > > > > > > > > > process any UI commands until the crashed server is > > > > > > > > > restarted. Example of won't process any UI command is > > > > > > > > > create a new instance, create a new account, etc. After > > > > > > > > > doing some searching I have found that others have > > > > > > > > > noticed java heap errors in 4.2.1, and the suggested fix > > > > > > > > > is to increase the heap size. I am planning on > > > > > > > > > increasing it from 2g to 4g, however if the problem is > > > > > > > > > something like a memory leak, then increasing the heap > > > > > > > > > size will just delay the inevitable. Has anyone else > > > > > > > > > fixed this issue by increasing the heap size? Or what is the > > > > > > > > > recommended value? > > > > > > > > > **updated...as expected I increased the heap size from > > > > > > > > > 2g to 4g, and it just too > > > > > > > > k longer for the problem to reoccur... > > > > > > > > > In my honest opinion of bigger concern is the fact that when > > > > > > > > > one management server crashes the other stops functioning as > > > > > > > > > well. So this begs the question of why even bother with a > > > > > > > > > redundant pair of servers..Anybody else experience this > > > > > > > > > issue? I would love to hear any dev guys opinion on this.... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
