Adam, I would have gone for the yum downgrade option. Please send us some logs.
-- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ----- Original Message ----- > From: "Adam" <adam.scarce...@gmail.com> > To: users@cloudstack.apache.org > Sent: Monday, 3 November, 2014 22:42:25 > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1 > ok, so I've rolled back and now none of my hosts are connecting. My system > VMs are not running, and the only thing I can see in the logs is that my > zone is not ready for either the Console Proxy or the Secondary Storage VM > to start because none of my hosts are reachable, even though in the UI it > says they're all up and running. > > I'm really losing all patience and confidence with CloudStack. I currently > have a 10 person team who cannot access any of their VMs because CloudStack > can't seem to find itself. > > I stopped all services and erased all 4.4.1 packages on the management > server as well as all the KVM hosts. > > # STOP ALL CLOUDSTACK SERVICES ON MANAGEMENT HOST > service cloudstack-agent stop > service cloudstack-usage stop > service cloudstack-management stop > > # STOP CLOUDSTACK AGENT ON KVM HOST #N > service cloudstack-agent stop > > # DROP ALL DATABASES > # THIS SCRIPT DROPS ALL DATABASES IN THE CLOUDSTACK MySQL DB (cloud, > cloud_usage, cloudbridge) > /root/cronScripts/mysql_DropAllDatabases.sh > > # MODIFIED/etc/yum.repos.d/cloudstack.repo to use 4.3 repository > [root@cs-east-prod1 /]# cat /etc/yum.repos.d/cloudstack.repo > [cloudstack] > name=cloudstack > baseurl=http://cloudstack.apt-get.eu/rhel/4.3/ > enabled=1 > gpgcheck=0 > [root@cs-east-prod1 /]# > > # YUM ERASE/FS CLEAN/YUM INSTALL ON MANAGEMENT HOST > yum -y erase cloudstack\* > rm -Rf /var/log/cloudstack/management > rm -Rf /usr/share/cloudstack* > rm -Rf /usr/share/doc/cloudstack* > rm -Rf /var/log/cloudstack > rm -Rf /etc/cloudstack > rm -Rf /var/cache/cloudstack > rm -Rf /var/cloudstack > yum clean all > yum -y install cloudstack\* > > # YUM ERASE/FS CLEAN/YUM INSTALL ON KVM HOST(S) > yum -y erase cloudstack\* > rm -Rf /usr/share/cloudstack* > rm -Rf /usr/share/doc/cloudstack* > rm -Rf /var/log/cloudstack > rm -Rf /etc/cloudstack > yum clean all > yum -y install cloudstack-agent > > # RAN cloudstack-setup-databases > cloudstack-setup-databases cloud:<PASSWORD> --deploy-as=root:<PASSWORD> -i > 10.97.38.10 > > # RESTORED MySQL BACKUPS > mysql -u root -p<PASSWORD> cloud < > /var/www/html/mysql-backups/cloud_11-01-2014.sql > mysql -u root -p<PASSWORD> cloud < > /var/www/html/mysql-backups/cloud_usage_11-01-2014.sql > mysql -u root -p<PASSWORD> cloud < > /var/www/html/mysql-backups/cloud_meta_11-01-2014.sql > mysql -u root -p<PASSWORD> cloud < > /var/www/html/mysql-backups/cloudbridge_11-01-2014.sql > > # START ALL CLOUDSTACK SERVICES ON MANAGEMENT HOST > service cloudstack-management start > service cloudstack-usage start > service cloudstack-agent start > > # START ALL CLOUDSTACK SERVICES ON KVM HOST(S) > service cloudstack-agent start > > Where do I go from here? > > Best Regards, > > > > Adam Scarcella > On Nov 3, 2014 2:06 PM, "Nux!" <n...@li.nux.ro> wrote: > >> Adam, >> >> I take it replacing the VR's disk did not work? Well, I'm hoping 4.4.2 >> will come with a fixed template. >> >> Re downgrading this is how I did it: >> 1 - stop all services (mgmt and agents) and "yum downgrade cloudstack-\*" >> 2 - drop the DBs and import backups >> 3 - restart the downgraded services >> 4 - you'll also need to rebuild your VR, possibly "virsh destroy" the >> 4.4.1 one if it's still around >> >> Good luck >> >> -- >> Sent from the Delta quadrant using Borg technology! >> >> Nux! >> www.nux.ro >> >> ----- Original Message ----- >> > From: "Adam" <adam.scarce...@gmail.com> >> > To: users@cloudstack.apache.org >> > Sent: Monday, 3 November, 2014 18:51:13 >> > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1 >> >> > I've decided to revert back to 4.3.1 since I cannot wait any loner to get >> > 4.4.1 working. >> > >> > I have a complete backup of all three databases just before I attempted >> to >> > upgrade so I'm assuming all I would have to do is remove 4.4.1, reinstall >> > 4.3.1, restore the db and then start everything back up correct? >> > >> > Are there any official steps to roll back? >> > >> > Best Regards, >> > >> > >> > >> > Adam Scarcella >> > >> > On Mon, Nov 3, 2014 at 12:13 PM, Nux! <n...@li.nux.ro> wrote: >> > >> >> What I am saying - and excuse my poor explanations - is that the HV may >> >> have already cached the old template, so the VR still uses the old >> broken >> >> one. >> >> >> >> What I was suggesting is find out which >> /var/lib/libvirt/images/blah.qcow2 >> >> the VR is using and replace it with my template. >> >> >> >> -- >> >> Sent from the Delta quadrant using Borg technology! >> >> >> >> Nux! >> >> www.nux.ro >> >> >> >> ----- Original Message ----- >> >> > From: "Adam" <adam.scarce...@gmail.com> >> >> > To: users@cloudstack.apache.org >> >> > Sent: Monday, 3 November, 2014 17:03:01 >> >> > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1 >> >> >> >> >> >> >> >> Not sure if this operation that you performed replaces the current >> >> >> systemvm template and *especially if it replaces it also on the >> >> >> hypervisor where it is running.* >> >> >> >> >> > >> >> > That procedure is the one documented in the official CloudStack >> >> > documentation so I hope it actually replaces the systemvm template. >> The >> >> > final output from that command tells it was successful: >> >> > >> >> > Successfully installed system VM template to >> >> > /export/secondary/template/tmpl/1/274/ >> >> > >> >> > Also, as stated before, the VR is not running on any hypervisor / host >> >> > because I cannot get it to run, so I'm unsure what you mean when you >> >> > say "*especially >> >> > if it replaces it also on the hypervisor where it is running.*" >> >> > >> >> > One more thing I would try is to make sure the VR on the HV is also >> using >> >> >> this image. You can try to compare md5sums of the image on the >> secondary >> >> >> storage and also the backend file for the VR's disk image (qemu-img >> info >> >> >> will show it to you) on the HV. >> >> >> If the sums differ then I would try to replace the VR disk image >> with my >> >> >> template directly on the HV and see if that does it. >> >> > >> >> > >> >> > I really don't understand what you're asking me to do. >> >> > >> >> > Best Regards, >> >> > >> >> > >> >> > >> >> > Adam Scarcella >> >> > >> >> > On Mon, Nov 3, 2014 at 11:50 AM, Nux! <n...@li.nux.ro> wrote: >> >> > >> >> >> Adam, >> >> >> >> >> >> Not sure if this operation that you performed replaces the current >> >> >> systemvm template and especially if it replaces it also on the >> >> hypervisor >> >> >> where it is running. >> >> >> >> >> >> One more thing I would try is to make sure the VR on the HV is also >> >> using >> >> >> this image. You can try to compare md5sums of the image on the >> secondary >> >> >> storage and also the backend file for the VR's disk image (qemu-img >> info >> >> >> will show it to you) on the HV. >> >> >> If the sums differ then I would try to replace the VR disk image >> with my >> >> >> template directly on the HV and see if that does it. >> >> >> >> >> >> -- >> >> >> Sent from the Delta quadrant using Borg technology! >> >> >> >> >> >> Nux! >> >> >> www.nux.ro >> >> >> >> >> >> ----- Original Message ----- >> >> >> > From: "Adam" <adam.scarce...@gmail.com> >> >> >> > To: users@cloudstack.apache.org >> >> >> > Sent: Monday, 3 November, 2014 16:08:05 >> >> >> > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1 >> >> >> >> >> >> > Thanks HTH, >> >> >> > >> >> >> > I decided to try to use your already modified qcow2 file: >> >> >> > >> >> >> > [root@cs-east-prod1 /]# >> >> >> > >> >> >> >> >> >> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt >> >> >> > -m /export/secondary -u >> >> >> http://tmp.nux.ro/systemvm-4.4.1_jira7781.qcow2.bz2 >> >> >> > -h kvm -F >> >> >> > --2014-11-03 10:11:32-- >> >> >> http://tmp.nux.ro/systemvm-4.4.1_jira7781.qcow2.bz2 >> >> >> > Resolving tmp.nux.ro... 217.19.15.108 >> >> >> > Connecting to tmp.nux.ro|217.19.15.108|:80... connected. >> >> >> > HTTP request sent, awaiting response... 200 OK >> >> >> > Length: 305498799 (291M) [application/x-bzip2] >> >> >> > Saving to: >> >> >> > >> >> >> >> >> >> “/usr/share/cloudstack-common/scripts/storage/secondary/00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2” >> >> >> > >> >> >> > >> >> >> >> >> >> 100%[===================================================================================================================>] >> >> >> > 305,498,799 9.02M/s in 37s >> >> >> > >> >> >> > 2014-11-03 10:12:10 (7.82 MB/s) - >> >> >> > >> >> >> >> >> >> “/usr/share/cloudstack-common/scripts/storage/secondary/00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2” >> >> >> > saved [305498799/305498799] >> >> >> > >> >> >> > Uncompressing to >> >> >> > >> >> >> >> >> >> /usr/share/cloudstack-common/scripts/storage/secondary/00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2.tmp >> >> >> > (type bz2)...could take a long time >> >> >> > Moving to >> >> >> > >> >> >> >> >> >> /export/secondary/template/tmpl/1/274///00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2...could >> >> >> > take a while >> >> >> > Successfully installed system VM template to >> >> >> > /export/secondary/template/tmpl/1/274/ >> >> >> > [root@cs-east-prod1 /]# >> >> >> > >> >> >> > Then I went into the Admin GUI and tried to start the VR, but it >> >> failed >> >> >> > with the exact same message: >> >> >> > >> >> >> > [image: Inline image 1] >> >> >> > >> >> >> > So I destroyed the Virtual Router and stopped and started the >> >> following >> >> >> > services: >> >> >> > >> >> >> > [root@cs-east-prod1 /]# service iptables stop >> >> >> > iptables: Setting chains to policy ACCEPT: nat filter [ OK ] >> >> >> > iptables: Flushing firewall rules: [ OK ] >> >> >> > iptables: Unloading modules: [ OK ] >> >> >> > [root@cs-east-prod1 /]# service cloudstack-management stop >> >> >> > Stopping cloudstack-management: [ OK ] >> >> >> > [root@cs-east-prod1 /]# service cloudstack-agent stop >> >> >> > Stopping Cloud Agent: >> >> >> > [root@cs-east-prod1 /]# service cloudstack-usage stop >> >> >> > Stopping CloudStack Usage Monitor cloudstack-usage [ OK ] >> >> >> > [root@cs-east-prod1 /]# service cloudstack-management start >> >> >> > Starting cloudstack-management: [ OK ] >> >> >> > [root@cs-east-prod1 /]# service cloudstack-usage start >> >> >> > Starting CloudStack Usage Monitor cloudstack-usage [ OK ] >> >> >> > [root@cs-east-prod1 /]# service cloudstack-agent start >> >> >> > Starting Cloud Agent: >> >> >> > [root@cs-east-prod1 /]# >> >> >> > >> >> >> > I then tried starting any VM instance to have CS automatically >> >> re-create >> >> >> > the VR, but that failed as well. >> >> >> > >> >> >> > Motty, I also looked at trying to increase the RAM and CPU for the >> >> router >> >> >> > but only saw one option >> >> >> > >> >> >> > [image: Inline image 2] >> >> >> > >> >> >> > So I created a new Service Offering with twice as much resources: >> >> >> > >> >> >> > [image: Inline image 3] >> >> >> > >> >> >> > Then upgraded the service offering for the VR >> >> >> > >> >> >> > [image: Inline image 4] >> >> >> > >> >> >> > [image: Inline image 6] >> >> >> > >> >> >> > But it failed still with the exact same error: >> >> >> > >> >> >> > [image: Inline image 7] >> >> >> > >> >> >> > Best Regards, >> >> >> > >> >> >> > >> >> >> > >> >> >> > Adam Scarcella >> >> >> > >> >> >> > On Mon, Nov 3, 2014 at 10:23 AM, motty cruz <motty.c...@gmail.com> >> >> >> wrote: >> >> >> > >> >> >> >> Adam, >> >> >> >> what are the specifications of the VR? can you upgrade the RAM and >> >> CPU >> >> >> and >> >> >> >> try to restart that VR? I had similar issue and giving the VR more >> >> RAM >> >> >> and >> >> >> >> CPU will start the VR. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> Motty >> >> >> >> >> >> >> >> On Mon, Nov 3, 2014 at 7:04 AM, Nux! <n...@li.nux.ro> wrote: >> >> >> >> >> >> >> >> > Adam, >> >> >> >> > >> >> >> >> > You can modify the qcow2 file that is in your secondary storage, >> >> i.e. >> >> >> you >> >> >> >> > should mount that storage somewhere and go to >> >> >> >> > /mount/point/template/tmpl/274 and modify the qcow2 file. >> >> >> >> > >> >> >> >> > Alternatively you could use my already modified copy to >> overwrite >> >> your >> >> >> >> > uuid.qcow2 with (at your own risk of course): >> >> >> >> > http://tmp.nux.ro/systemvm-4.4.1_jira7781.qcow2.bz2 >> >> >> >> > >> >> >> >> > HTH >> >> >> >> > >> >> >> >> > -- >> >> >> >> > Sent from the Delta quadrant using Borg technology! >> >> >> >> > >> >> >> >> > Nux! >> >> >> >> > www.nux.ro >> >> >> >> > >> >> >> >> > ----- Original Message ----- >> >> >> >> > > From: "Adam" <adam.scarce...@gmail.com> >> >> >> >> > > To: users@cloudstack.apache.org >> >> >> >> > > Sent: Monday, 3 November, 2014 14:58:58 >> >> >> >> > > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1 >> >> >> >> > >> >> >> >> > > [root@cs-east-prod1 management]# netstat -a|grep 9090 >> >> >> >> > > unix 3 [ ] STREAM CONNECTED 29090 >> >> >> >> > > [root@cs-east-prod1 management]# nmap localhost >> >> >> >> > > >> >> >> >> > > Starting Nmap 5.51 ( http://nmap.org ) at 2014-11-03 08:47 >> EST >> >> >> >> > > Nmap scan report for localhost (127.0.0.1) >> >> >> >> > > Host is up (0.000015s latency). >> >> >> >> > > Other addresses for localhost (not scanned): 127.0.0.1 >> >> >> >> > > Not shown: 990 closed ports >> >> >> >> > > PORT STATE SERVICE >> >> >> >> > > 22/tcp open ssh >> >> >> >> > > 25/tcp open smtp >> >> >> >> > > 80/tcp open http >> >> >> >> > > 111/tcp open rpcbind >> >> >> >> > > 199/tcp open smux >> >> >> >> > > 631/tcp open ipp >> >> >> >> > > 2049/tcp open nfs >> >> >> >> > > 3306/tcp open mysql >> >> >> >> > > 8080/tcp open http-proxy >> >> >> >> > > 9090/tcp open zeus-admin >> >> >> >> > > >> >> >> >> > > Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds >> >> >> >> > > [root@cs-east-prod1 management]# telnet localhost 9090 >> >> >> >> > > Trying ::1... >> >> >> >> > > Connected to localhost. >> >> >> >> > > Escape character is '^]'. >> >> >> >> > > Connection closed by foreign host. >> >> >> >> > > [root@cs-east-prod1 management]# nmap 10.97.38.10 >> >> >> >> > > >> >> >> >> > > Starting Nmap 5.51 ( http://nmap.org ) at 2014-11-03 08:49 >> EST >> >> >> >> > > Nmap scan report for cs-east-prod1.na.tibco.com (10.97.38.10) >> >> >> >> > > Host is up (0.000015s latency). >> >> >> >> > > Not shown: 993 closed ports >> >> >> >> > > PORT STATE SERVICE >> >> >> >> > > 22/tcp open ssh >> >> >> >> > > 80/tcp open http >> >> >> >> > > 111/tcp open rpcbind >> >> >> >> > > 2049/tcp open nfs >> >> >> >> > > 3306/tcp open mysql >> >> >> >> > > 8080/tcp open http-proxy >> >> >> >> > > 9090/tcp open zeus-admin >> >> >> >> > > >> >> >> >> > > Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds >> >> >> >> > > [root@cs-east-prod1 management]# telnet 10.97.38.10 9090 >> >> >> >> > > Trying 10.97.38.10... >> >> >> >> > > Connected to 10.97.38.10. >> >> >> >> > > Escape character is '^]'. >> >> >> >> > > Connection closed by foreign host. >> >> >> >> > > [root@cs-east-prod1 management]# >> >> >> >> > > >> >> >> >> > > Also, I checked for the /opt/cloud/bin/ directory on my >> >> Management >> >> >> >> Host, >> >> >> >> > > but it's not there: >> >> >> >> > > >> >> >> >> > > [root@cs-east-prod1 management]# cd /opt/cloud/bin/ >> >> >> >> > > -bash: cd: /opt/cloud/bin/: No such file or directory >> >> >> >> > > [root@cs-east-prod1 management]# cd / >> >> >> >> > > [root@cs-east-prod1 /]# find . -name "createipAlias.sh" >> >> >> >> > > [root@cs-east-prod1 /]# find . -name "createIpAlias.sh" >> >> >> >> > > [root@cs-east-prod1 /]# >> >> >> >> > > >> >> >> >> > > Are you saying that the /opt/cloud/bin/createipAlias.sh >> script is >> >> >> >> inside >> >> >> >> > > the Virtual Router VM? If so, how could I possibly change that >> >> >> since it >> >> >> >> > > won't start? >> >> >> >> > > >> >> >> >> > > Best Regards, >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > Adam Scarcella >> >> >> >> > > >> >> >> >> > > On Mon, Nov 3, 2014 at 9:07 AM, Pierre-Luc Dion < >> >> >> pdion...@apache.org> >> >> >> >> > wrote: >> >> >> >> > > >> >> >> >> > >> Could it be possible the firewall is blocking port 9090 from >> the >> >> >> >> > >> management-server? >> >> >> >> > >> >> >> >> >> > >> 2014-11-03 08:14:36,813 ERROR [c.c.c.ClusterManagerImpl] >> >> >> (main:null) >> >> >> >> > Unable >> >> >> >> > >> to ping management server at 10.97.38.10:9090 due to >> >> >> ConnectException >> >> >> >> > >> java.net.ConnectException: Connection refused >> >> >> >> > >> >> >> >> >> > >> On Mon, Nov 3, 2014 at 9:02 AM, Nux! <n...@li.nux.ro> wrote: >> >> >> >> > >> >> >> >> >> > >> > BTW the agent logs should help clarify this. >> >> >> >> > >> > The aforementioned changes must be done either in the >> current >> >> >> >> > (broken) VR >> >> >> >> > >> > or in the systemvm (after which you should redeploy the VR >> to >> >> >> pick >> >> >> >> up >> >> >> >> > the >> >> >> >> > >> > change). >> >> >> >> > >> > >> >> >> >> > >> > HTH >> >> >> >> > >> > Lucian >> >> >> >> > >> > >> >> >> >> > >> > -- >> >> >> >> > >> > Sent from the Delta quadrant using Borg technology! >> >> >> >> > >> > >> >> >> >> > >> > Nux! >> >> >> >> > >> > www.nux.ro >> >> >> >> > >> > >> >> >> >> > >> > ----- Original Message ----- >> >> >> >> > >> > > From: "Nux!" <n...@li.nux.ro> >> >> >> >> > >> > > To: users@cloudstack.apache.org >> >> >> >> > >> > > Sent: Monday, 3 November, 2014 13:53:10 >> >> >> >> > >> > > Subject: Re: How to Upgrade Virtual Router from 4.3 to >> 4.4.1 >> >> >> >> > >> > >> >> >> >> > >> > > Hello, >> >> >> >> > >> > > >> >> >> >> > >> > > I think you are hitting a problem I also had: >> >> >> >> > >> > > "2014-11-03 08:16:17,024 ERROR >> >> >> [c.c.v.VirtualMachineManagerImpl] >> >> >> >> > >> > > (Work-Job-Executor-1:ctx-5c33d547 job-24555/job-24556 >> >> >> >> ctx-fcb072b2) >> >> >> >> > >> > Failed to >> >> >> >> > >> > > start instance VM[DomainRouter|r-4758-VM] >> >> >> >> > >> > > com.cloud.utils.exception.ExecutionException: Unable to >> >> start >> >> >> >> > >> > > VM[DomainRouter|r-4758-VM] due to error in finalizeStart, >> >> not >> >> >> >> > retrying" >> >> >> >> > >> > > >> >> >> >> > >> > > It's >> https://issues.apache.org/jira/browse/CLOUDSTACK-7781 >> >> >> >> > >> > > >> >> >> >> > >> > > Basically the systemvm template has some problems, you >> need >> >> to >> >> >> >> > somehow >> >> >> >> > >> > copy >> >> >> >> > >> > > /opt/cloud/bin/createIpAlias.sh to >> >> >> /opt/cloud/bin/createipAlias.sh >> >> >> >> > to >> >> >> >> > >> > get it >> >> >> >> > >> > > working. >> >> >> >> > >> > > >> >> >> >> > >> > > -- >> >> >> >> > >> > > Sent from the Delta quadrant using Borg technology! >> >> >> >> > >> > > >> >> >> >> > >> > > Nux! >> >> >> >> > >> > > www.nux.ro >> >> >> >> > >> > > >> >> >> >> > >> > > ----- Original Message ----- >> >> >> >> > >> > >> From: "Adam" <adam.scarce...@gmail.com> >> >> >> >> > >> > >> To: users@cloudstack.apache.org >> >> >> >> > >> > >> Sent: Monday, 3 November, 2014 13:32:12 >> >> >> >> > >> > >> Subject: Re: How to Upgrade Virtual Router from 4.3 to >> >> 4.4.1 >> >> >> >> > >> > > >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# cat /etc/redhat-release >> >> >> >> > >> > >> CentOS release 6.6 (Final) >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# uname -a >> >> >> >> > >> > >> Linux cs-east-prod1 2.6.32-504.el6.x86_64 #1 SMP Wed >> Oct 15 >> >> >> >> > 04:27:16 >> >> >> >> > >> UTC >> >> >> >> > >> > >> 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# virsh --version >> >> >> >> > >> > >> 0.10.2 >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# sed -i 's/INFO/DEBUG/g' >> >> >> >> > >> > >> /etc/cloudstack/agent/log4j-cloud.xml >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# service cloudstack-management >> stop >> >> >> >> > >> > >> Stopping cloudstack-management: >> >> [ >> >> >> >> OK ] >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# mv >> >> >> >> > >> > >> /var/log/cloudstack/management/management-server.log >> >> >> >> > >> > >> /var/log/cloudstack/management/management-server.bak >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# service cloudstack-management >> >> start >> >> >> >> > >> > >> Starting cloudstack-management: >> >> [ >> >> >> >> OK ] >> >> >> >> > >> > >> [root@cs-east-prod1 ~]# >> >> >> >> > >> > >> >> >> >> >> > >> > >> Please see attached ZIP of clean management-server.log >> in >> >> >> DEBUG >> >> >> >> > mode. >> >> >> >> > >> At >> >> >> >> > >> > >> exactly (*2014-11-03 08:15:35,700 on line 3025 of the >> >> attached >> >> >> >> > >> > >> management-server.log*) I tried starting the Virtual >> Router >> >> >> from >> >> >> >> > the >> >> >> >> > >> > >> Management UI and received this "Status" message in the >> UI. >> >> >> >> > >> > >> >> >> >> >> > >> > >> [image: Inline image 1] >> >> >> >> > >> > >> >> >> >> >> > >> > >> I can see in the management-server.log that while the >> >> >> >> > >> > cloudstack-management >> >> >> >> > >> > >> service is initializing, it throws the following ERROR >> at >> >> >> exactly >> >> >> >> > >> > (*2014-11-03 >> >> >> >> > >> > >> 08:14:36,813 on line 342 of the attached >> >> >> management-server.log*): >> >> >> >> > >> > >> >> >> >> >> > >> > >> 2014-11-03 08:14:36,813 ERROR [c.c.c.ClusterManagerImpl] >> >> >> >> > (main:null) >> >> >> >> > >> > Unable >> >> >> >> > >> > >> to ping management server at 10.97.38.10:9090 due to >> >> >> >> > ConnectException >> >> >> >> > >> > >> java.net.ConnectException: Connection refused >> >> >> >> > >> > >> at sun.nio.ch.Net.connect0(Native Method) >> >> >> >> > >> > >> at sun.nio.ch.Net.connect(Net.java:465) >> >> >> >> > >> > >> at sun.nio.ch.Net.connect(Net.java:457) >> >> >> >> > >> > >> at >> >> >> >> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1123) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1092) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1170) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1057) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:114) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:110) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:56) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:167) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:339) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:108) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> >> org.apache.catalina.core.StandardContext.start(StandardContext.java:4709) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> >> >> >> >> >> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1041) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:964) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502) >> >> >> >> > >> > >> at >> >> >> >> > >> >> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) >> >> >> >> > >> > >> at >> >> >> >> > >> >> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:722) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> >> >> >> >> >> >> org.apache.catalina.core.StandardService.start(StandardService.java:516) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:710) >> >> >> >> > >> > >> at >> >> >> >> org.apache.catalina.startup.Catalina.start(Catalina.java:593) >> >> >> >> > >> > >> at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> >> >> Method) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> >> >> > >> > >> at >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> >> >> > >> > >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> >> >> > >> > >> at >> >> >> >> > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) >> >> >> >> > >> > >> at >> >> >> >> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) >> >> >> >> > >> > >> 2014-11-03 08:14:36,816 INFO [c.c.c.ClusterManagerImpl] >> >> >> >> > (main:null) >> >> >> >> > >> > >> Detected that another management node with the same IP >> >> >> >> 10.97.38.10 >> >> >> >> > is >> >> >> >> > >> > >> considered as running in DB, however it is not >> pingable, we >> >> >> will >> >> >> >> > >> > continue >> >> >> >> > >> > >> cluster initialization with this management server node >> >> >> >> > >> > >> >> >> >> >> > >> > >> I'm not sure what that error means, but more than likely >> >> >> that's >> >> >> >> the >> >> >> >> > >> > reason >> >> >> >> > >> > >> why the Virtual Router will not start. >> >> >> >> > >> > >> >> >> >> >> > >> > >> * PLEASE HELP ! * >> >> >> >> > >> > >> >> >> >> >> > >> > >> Best Regards, >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> >> > >> > >> >> >> >> >> > >> > >> Adam Scarcella >> >> >> >> > >> > >> >> >> >> >> > >> > >> On Sun, Nov 2, 2014 at 8:20 PM, Nux! <n...@li.nux.ro> >> >> wrote: >> >> >> >> > >> > >> >> >> >> >> > >> > >>> Hi, >> >> >> >> > >> > >>> >> >> >> >> > >> > >>> More agent and management logs please, make sure they >> are >> >> in >> >> >> >> > DEBUG[1] >> >> >> >> > >> > mode. >> >> >> >> > >> > >>> >> >> >> >> > >> > >>> [1] - >> >> >> >> > >> > >>> >> >> >> >> > >> >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug >> >> >> >> > >> > >>> >> >> >> >> > >> > >>> -- >> >> >> >> > >> > >>> Sent from the Delta quadrant using Borg technology! >> >> >> >> > >> > >>> >> >> >> >> > >> > >>> Nux! >> >> >> >> > >> > >>> www.nux.ro >> >> >> >> > >> > >>> >> >> >> >> > >> > >>> ----- Original Message ----- >> >> >> >> > >> > >>> > From: "Adam" <adam.scarce...@gmail.com> >> >> >> >> > >> > >>> > To: users@cloudstack.apache.org >> >> >> >> > >> > >>> > Sent: Sunday, 2 November, 2014 23:33:22 >> >> >> >> > >> > >>> > Subject: How to Upgrade Virtual Router from 4.3 to >> 4.4.1 >> >> >> >> > >> > >>> >> >> >> >> > >> > >>> > I've performed a complete upgrade from 4.3 to 4.4.1 >> and >> >> >> >> followed >> >> >> >> > >> the >> >> >> >> > >> > >>> > instructions perfectly, but my Virtual Router will >> not >> >> >> start, >> >> >> >> > nor >> >> >> >> > >> am >> >> >> >> > >> > I >> >> >> >> > >> > >>> able >> >> >> >> > >> > >>> > to "upgrade" it. >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > I tried destroying the Virtual Router, starting any >> VM >> >> >> >> Instance, >> >> >> >> > >> and >> >> >> >> > >> > >>> > allowing CS to recreate the VR. However that didn't >> >> help. >> >> >> The >> >> >> >> VR >> >> >> >> > >> > still >> >> >> >> > >> > >>> did >> >> >> >> > >> > >>> > not start and it was still saying that it required >> to be >> >> >> >> > upgraded. >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > [image: Inline image 1] >> >> >> >> > >> > >>> > [image: Inline image 2] >> >> >> >> > >> > >>> > [image: Inline image 3] >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > I know there are manual steps to finalize the >> systemvm >> >> >> >> upgrade, >> >> >> >> > but >> >> >> >> > >> > I did >> >> >> >> > >> > >>> > those steps: >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgrade-4.3.html#manual-hofix >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > [root@cs-east-prod1 cloudstack-logs]# mysql -u root >> -p >> >> >> >> > >> > >>> > Enter password: >> >> >> >> > >> > >>> > Welcome to the MySQL monitor. Commands end with ; or >> >> \g. >> >> >> >> > >> > >>> > Your MySQL connection id is 174 >> >> >> >> > >> > >>> > Server version: 5.1.73-log Source distribution >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > Copyright (c) 2000, 2013, Oracle and/or its >> affiliates. >> >> All >> >> >> >> > rights >> >> >> >> > >> > >>> reserved. >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > Oracle is a registered trademark of Oracle >> Corporation >> >> >> and/or >> >> >> >> > its >> >> >> >> > >> > >>> > affiliates. Other names may be trademarks of their >> >> >> respective >> >> >> >> > >> > >>> > owners. >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > Type 'help;' or '\h' for help. Type '\c' to clear the >> >> >> current >> >> >> >> > input >> >> >> >> > >> > >>> > statement. >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > mysql> select id,name from cloud.vm_template where >> name >> >> = >> >> >> >> > >> > >>> > 'systemvm-kvm-4.4'; >> >> >> >> > >> > >>> > +-----+------------------+ >> >> >> >> > >> > >>> >| id | name | >> >> >> >> > >> > >>> > +-----+------------------+ >> >> >> >> > >> > >>> >| 274 | systemvm-kvm-4.4 | >> >> >> >> > >> > >>> > +-----+------------------+ >> >> >> >> > >> > >>> > 1 row in set (0.00 sec) >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > mysql> select type from cloud.vm_template where >> >> id='274'; >> >> >> >> > >> > >>> > +--------+ >> >> >> >> > >> > >>> >| type | >> >> >> >> > >> > >>> > +--------+ >> >> >> >> > >> > >>> >| SYSTEM | >> >> >> >> > >> > >>> > +--------+ >> >> >> >> > >> > >>> > 1 row in set (0.00 sec) >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > mysql> select distinct(vm_template_id) from >> >> >> cloud.vm_instance >> >> >> >> > where >> >> >> >> > >> > >>> > type='ConsoleProxy' and hypervisor_type = 'KVM'; >> >> >> >> > >> > >>> > +----------------+ >> >> >> >> > >> > >>> >| vm_template_id | >> >> >> >> > >> > >>> > +----------------+ >> >> >> >> > >> > >>> >| 274 | >> >> >> >> > >> > >>> > +----------------+ >> >> >> >> > >> > >>> > 1 row in set (0.00 sec) >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > mysql> select distinct(vm_template_id) from >> >> >> cloud.vm_instance >> >> >> >> > where >> >> >> >> > >> > >>> > type='SecondaryStorageVm' and hypervisor_type = >> 'KVM'; >> >> >> >> > >> > >>> > +----------------+ >> >> >> >> > >> > >>> >| vm_template_id | >> >> >> >> > >> > >>> > +----------------+ >> >> >> >> > >> > >>> >| 274 | >> >> >> >> > >> > >>> > +----------------+ >> >> >> >> > >> > >>> > 1 row in set (0.03 sec) >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > mysql> select value from cloud.configuration where >> name >> >> = >> >> >> >> > >> > >>> > 'router.template.kvm'; >> >> >> >> > >> > >>> > +------------------+ >> >> >> >> > >> > >>> >| value | >> >> >> >> > >> > >>> > +------------------+ >> >> >> >> > >> > >>> >| systemvm-kvm-4.4 | >> >> >> >> > >> > >>> > +------------------+ >> >> >> >> > >> > >>> > 1 row in set (0.00 sec) >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > mysql> >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > * PLEASE HELP ! * >> >> >> >> > >> > >>> > >> >> >> >> > >> > >>> > Best Regards, >> >> >> >> > >> > >>> > >> >> >> >> > >> > > >> > Adam Scarcella >> >> >> >> > >> > >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Thanks for your support, >> >> >> >> Motty
