I have tried to do upgrade 2nd time and failed again. The sequence of actions is described here: http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgrade-4.4.html
Followed instructions until the end. VM upgrade has taken more than 5 hrs. The result of update is: ----------------- Stopping and starting 1 secondary storage vm(s)... ERROR: Failed to stop secondary storage vm with id 25 Done stopping and starting secondary storage vm(s) Stopping and starting 1 console proxy vm(s)... ERROR: Failed to stop console proxy vm with id 1 Done stopping and starting console proxy vm(s) . Stopping and starting 1 running routing vm(s)... 2 Done restarting router(s). ------------------- After restarting managment server workers are not querying status and resources anymore. The last lines at management server log are: ------------------- 2014-10-30 08:48:10,207 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin UcsManagerImpl 2014-10-30 08:48:10,208 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin VpcVirtualRouterElement 2014-10-30 08:48:10,208 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BaremetalPxeManagerImpl 2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BaremetalManagerImpl 2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ApiDiscoveryServiceImpl 2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ManagementServerImpl 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin DedicatedResourceManagerImpl 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin InternalLoadBalancerElement 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin LdapManagerImpl 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ApiRateLimitServiceImpl 2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BaremetalDhcpManagerImpl 2014-10-30 08:48:10,212 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ContrailVpcElementImpl 2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ContrailElementImpl 2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin MidoNetElement 2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin NetscalerElement 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin NiciraNvpElement 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin OpenDaylightControllerResourceManagerImpl 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin OvsElement 2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin PaloAltoExternalFirewallElement 2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin VirtualRouterElement 2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin SspElement 2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BigSwitchVnsElement 2014-10-30 08:48:10,250 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (NetworkStatsUpdater-1:ctx-ca0c3513) Successfully updated aggregate network stats 2014-10-30 08:48:10,379 INFO [c.c.a.ApiServer] (Thread-11:null) ApiServer listening on port 8096 ------------------------------------------ Full log is here: http://pastebin.com/Wpe7wsAV Log is not growing and changing. Cloudstack console is missing with 404 error. I thing VM update should not influence this. Any System VM in fact can be just re-created by "killing" from management GUI. Vadim. -----Original Message----- From: Andrija Panic [mailto:andrija.pa...@gmail.com] Sent: Wednesday, October 29, 2014 10:21 AM To: users@cloudstack.apache.org Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken ? If it makes it any easier - it failed for me, but with no error simply got "transaction revoked..." error, no description, nothing... On 28 October 2014 23:19, motty cruz <motty.c...@gmail.com> wrote: > Thanks Andrija, > but that did not fixed for me: > mysql> describe snapshot_policy; > +-----------+---------------------+------+-----+---------+----------------+ > | Field | Type | Null | Key | Default | Extra | > +-----------+---------------------+------+-----+---------+----------------+ > | id | bigint(20) unsigned | NO | PRI | NULL | auto_increment | > | uuid | varchar(40) | YES | UNI | NULL | | > | volume_id | bigint(20) unsigned | NO | MUL | NULL | | > | schedule | varchar(100) | NO | | NULL | | > | timezone | varchar(100) | NO | | NULL | | > | interval | int(4) | NO | | 4 | | > | max_snaps | int(8) | NO | | 0 | | > | active | tinyint(1) unsigned | NO | | NULL | | > | display | tinyint(1) | NO | | 1 | | > +-----------+---------------------+------+-----+---------+----------------+ > 9 rows in set (0.00 sec) > > I changed it as suggest: > ALTER TABLE `snapshot_policy` ADD `display` TINYINT( 1 ) NOT NULL > DEFAULT '1' > > am I doing something wrong? this is my 3rd attempt to upgrade.failed > with the same result. > > - Please help! > > > > > On Tue, Oct 28, 2014 at 2:56 PM, Andrija Panic > <andrija.pa...@gmail.com> > wrote: > > > Motty, here is copy/paste from someone's email here on mailing list: > > Maybe check if this is your issue... > > I did not manage to upgrade, and this solution came to late for > > me... did clean install, since this was test install in my case... > > > > <thomas.schnei...@euskill.com> wrote: > > > Hi, > > > > > > I solved the problem. > > > > > > The column display from the snapshot _policy was missing after update. > > > > > > Just do: ALTER TABLE `snapshot_policy` ADD `display` TINYINT( 1 > > > ) NOT NULL DEFAULT '1' > > > > > > Regards, > > > > On 28 October 2014 22:13, motty cruz <motty.c...@gmail.com> wrote: > > > > > forgot to mentioned about this logs in /etc/cloudstack/sysvm.log > > > nohup: ignoring input > > > > > > Stopping and starting 1 secondary storage vm(s)... > > > curl: (7) couldn't connect to host > > > ERROR: Failed to stop secondary storage vm with id 15 > > > > > > Done stopping and starting secondary storage vm(s) > > > > > > Stopping and starting 1 console proxy vm(s)... > > > curl: (7) couldn't connect to host > > > ERROR: Failed to stop console proxy vm with id 16 > > > > > > Done stopping and starting console proxy vm(s) . > > > > > > Stopping and starting 1 running routing vm(s)... > > > curl: (7) couldn't connect to host > > > 2 > > > Done restarting router(s). > > > > > > > > > Also /etc/cloudstack/cloud.log: > > > [2014.10.28-13.48.43] Stopping and starting 1 secondary storage > vm(s)... > > > [2014.10.28-13.48.43] INFO: Stopping secondary storage vm with id > > > 15 [2014.10.28-13.48.43] ERROR: Failed to stop secondary storage > > > vm with > id > > 15 > > > [2014.10.28-13.48.43] Done stopping and starting secondary storage > vm(s). > > > [2014.10.28-13.48.43] Stopping and starting 1 console proxy vm(s)... > > > [2014.10.28-13.48.43] INFO: Stopping console proxy with id 16 > > > [2014.10.28-13.48.43] ERROR: Failed to stop console proxy vm with > > > id 16 [2014.10.28-13.48.43] Done stopping and starting console proxy > > > vm(s) . > > > [2014.10.28-13.48.43] Stopping and starting 1 running routing vm(s)... > > > [2014.10.28-13.48.43] INFO: Restarting router with id 17 > > > [2014.10.28-13.48.43] ERROR: Failed to restart domainRouter with > > > id 17; unable to submit the job [2014.10.28-13.48.53] Done > > > restarting router(s). > > > > > > Thanks > > > -Motty > > > > > > On Tue, Oct 28, 2014 at 2:08 PM, motty cruz <motty.c...@gmail.com> > > wrote: > > > > > > > Thanks Vadim, > > > > I was able to restore back to ACS 4.4.0. upgraded again with the > > > > same > > bad > > > > result. has anybody successsfully upgrade from ACS 4.4.0 to > ACS-4.4.1? > > if > > > > yes do you mind sharing notes? > > > > I had used this URL: > > > > > > > > > > http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgr > ade-4.4.html > > > > > > > > here is my logs: > > > > upgradeACS4.4.1-http-status-404 > > > > *http://pastebin.com/raw.php?i=eeYQsVyD > > > > <http://pastebin.com/raw.php?i=eeYQsVyD>* > > > > > > > > *I really apreciate if someone can help me! * *in /etc/sudoers* > > > > > > > > *cloud ALL=NOPASSWD : ALL* > > > > > > > > *my suspecion, the issue have to do with database but am not > > > > certain > > and > > > > won't know how to fix it. * > > > > > > > > *Thanks, * > > > > *-Motty* > > > > > > > > On Tue, Oct 28, 2014 at 12:30 AM, Vadim Kimlaychuk < > > > > vadim.kimlayc...@elion.ee> wrote: > > > > > > > >> Motty, > > > >> > > > >> I got 404 after downgrading as well, but this was > > > >> database > > > issue. > > > >> You need to restore your DB first and then restart your > > > >> cloudstack management server. Better way (from my point of > > > >> view) - having DB > and > > > >> Management servers as VMs and restore them from snapshots. It > > > >> is > much > > > >> faster :) > > > >> > > > >> I will try to upgrade once again as Rohit Yadav suggested. > > > >> Didn't think about old VM-s running at hypervisors. They were > > > >> not > > > upgraded > > > >> that is true. Will keep in mind issue 7781 pointed by Nux! > > > >> Thank you all. Be back after update. > > > >> > > > >> Vadim. > > > >> > > > >> -----Original Message----- > > > >> From: motty cruz [mailto:motty.c...@gmail.com] > > > >> Sent: Tuesday, October 28, 2014 12:39 AM > > > >> To: users@cloudstack.apache.org > > > >> Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken ? > > > >> > > > >> Nux, > > > >> I just finished downgrading back to 4.4.0. but the error I get is: > > > >> HTTP STATUS 404 > > > >> type status report > > > >> > > > >> I will try again to upgrade as soon as is fully restore back to > > 4.4.0. I > > > >> am running XenServer 6.2 hypervison with Centos 6.5 as manager. > > > >> has > > > anybody > > > >> run into this issue? > > > >> > > > >> Vadim, I don't meant to hijack your post. > > > >> > > > >> Thanks, > > > >> Motty > > > >> > > > >> On Mon, Oct 27, 2014 at 3:09 PM, Nux! <n...@li.nux.ro> wrote: > > > >> > > > >> > Motty, > > > >> > > > > >> > Please provide DEBUG logs from management server and hypervisors. > > > >> > It's possible you may be hitting this (as I remember similar > > symptoms) > > > >> > https://issues.apache.org/jira/browse/CLOUDSTACK-7781 > > > >> > > > > >> > Lucian > > > >> > > > > >> > -- > > > >> > Sent from the Delta quadrant using Borg technology! > > > >> > > > > >> > Nux! > > > >> > www.nux.ro > > > >> > > > > >> > ----- Original Message ----- > > > >> > > From: "motty cruz" <motty.c...@gmail.com> > > > >> > > To: users@cloudstack.apache.org > > > >> > > Sent: Monday, 27 October, 2014 20:39:54 > > > >> > > Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken ? > > > >> > > > > >> > > Hello Vadim, > > > >> > > did you find the solution? > > > >> > > > > > >> > > I am having similar issue. I am running CentOS 6.5 > > > >> > > "nohup: ignoring input > > > >> > > > > > >> > > > > > >> > > > > > >> > > Stopping and starting 1 secondary storage vm(s)... > > > >> > > > > > >> > > curl: (56) Failure when receiving data from the peer > > > >> > > > > > >> > > ERROR: Failed to stop secondary storage vm with id 14 > > > >> > > > > > >> > > > > > >> > > > > > >> > > Done stopping and starting secondary storage vm(s) > > > >> > > > > > >> > > > > > >> > > > > > >> > > Stopping and starting 1 console proxy vm(s)... > > > >> > > > > > >> > > curl: (7) couldn't connect to host > > > >> > > > > > >> > > ERROR: Failed to stop console proxy vm with id 13 > > > >> > > > > > >> > > > > > >> > > > > > >> > > Done stopping and starting console proxy vm(s) . > > > >> > > > > > >> > > > > > >> > > > > > >> > > Stopping and starting 1 running routing vm(s)... > > > >> > > > > > >> > > curl: (7) couldn't connect to host > > > >> > > > > > >> > > 2 > > > >> > > > > > >> > > Done restarting router(s)." > > > >> > > > > > >> > > > > > >> > > Thanks, > > > >> > > > > > >> > > Motty > > > >> > > > > > >> > > > > > >> > > > > > >> > > On Mon, Oct 27, 2014 at 8:52 AM, Rohit Yadav > > > >> > > <rohit.ya...@shapeblue.com> > > > >> > > wrote: > > > >> > > > > > >> > >> Hi Vadim, > > > >> > >> > > > >> > >> See my reply in-line; > > > >> > >> > > > >> > >> > On 27-Oct-2014, at 8:46 pm, Vadim Kimlaychuk > > > >> > <vadim.kimlayc...@elion.ee> > > > >> > >> wrote: > > > >> > >> > > > > >> > >> > Yes, I did follow this guide. APT does not have > > > >> > >> > pre-compiled packages > > > >> > >> so I have compiled them by myself. This is the only difference. > > > >> > >> > > > > >> > >> > cloudstack-sysvmadm script that is mentioned in the > > > >> > >> > last > step > > > >> > >> > has the > > > >> > >> first line as follows: > > > >> > >> > > > > >> > >> > . /etc/rc.d/init.d/functions > > > >> > >> > > > > >> > >> > But there is no /etc/rc.d/ folder at my Ubuntu server. > Script > > > >> > >> > failed > > > >> > at > > > >> > >> the very beginning so I haven't waited until VMs are going > > > >> > >> to > > > >> migrate. > > > >> > >> > > > >> > >> You’re correct. This is supposed to be CentOS/RHEL/Fedora > > specific. > > > >> > >> I think your upgrade went fine it’s just this script that > > > >> > >> may > > need > > > >> > >> to be fixed. Can you create a ticket on CloudStack’s JIRA? > > > >> > >> As a workaround you may try to remove this line in the > > > >> > >> script, make > > sure > > > >> > >> “curl" is > > > >> > installed on > > > >> > >> Ubuntu and try again. > > > >> > >> > > > >> > >> > I have downgraded back to 4.4.0 until DEB packages will > > > >> > >> > be > > > >> released. > > > >> > >> Problem may be there though I have no errors building CS > > > >> > >> from > the > > > >> > source. > > > >> > >> > > > >> > >> I think you should be able to upgrade; until sysvmadm > > > >> > >> script is fixed so that it works on Debian based distro, > > > >> > >> after the > upgrade > > > >> > >> you may reboot > > > >> > or > > > >> > >> kill the existing/old 4.4.0 based systemvms using > > > >> > >> something > like > > a > > > >> > custom > > > >> > >> bash/python script or a tool like CloudMonkey or manually > > > >> > >> if > the > > > >> > >> deploy size is small. > > > >> > >> > > > >> > >> Regards. > > > >> > >> > > > >> > >> > > > > >> > >> > Vadim. > > > >> > >> > > > > >> > >> > -----Original Message----- > > > >> > >> > From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] > > > >> > >> > Sent: Monday, October 27, 2014 5:00 PM > > > >> > >> > To: users@cloudstack.apache.org > > > >> > >> > Subject: Re: Status: update from 4.4.0 -> 4.4.1 still > > > >> > >> > broken > ? > > > >> > >> > > > > >> > >> > Hi Vadim, > > > >> > >> > > > > >> > >> > Did you follow the upgrade instructions (especially the > > systemvm > > > >> > >> upgrading): > > > >> > >> > > > > >> > >> > > > >> > > > > http://cloudstack-release-notes.readthedocs.org/en/4.4.1/upgrade/u > > > pgra > > > >> > de-4.4.html > > > >> > >> > > > > >> > >> > If consoles are not coming up, please upgrade systemvms > > > >> > >> > to > > 4.4.1 > > > >> > >> > ones > > > >> > >> and delete existing console proxy VM(s) and retry. > > > >> > >> > > > > >> > >> > Any error log would help as well. > > > >> > >> > > > >> > >> Regards, > > > >> > >> Rohit Yadav > > > >> > >> Software Architect, ShapeBlue M. +91 88 262 30892 | > > > >> > >> rohit.ya...@shapeblue.com > > > >> > >> Blog: bhaisaab.org | Twitter: @_bhaisaab > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> Find out more about ShapeBlue and our range of CloudStack > related > > > >> > services > > > >> > >> > > > >> > >> IaaS Cloud Design & Build< > > > >> > >> http://shapeblue.com/iaas-cloud-design-and-build//> > > > >> > >> CSForge – rapid IaaS deployment > > > >> > >> framework<http://shapeblue.com/csforge/ > > > >> > > > > > >> > >> CloudStack Consulting< > > http://shapeblue.com/cloudstack-consultancy/ > > > > > > > >> > >> CloudStack Infrastructure Support< > > > >> > >> http://shapeblue.com/cloudstack-infrastructure-support/> > > > >> > >> CloudStack Bootcamp Training Courses< > > > >> > >> http://shapeblue.com/cloudstack-training/> > > > >> > >> > > > >> > >> This email and any attachments to it may be confidential > > > >> > >> and > are > > > >> > intended > > > >> > >> solely for the use of the individual to whom it is addressed. > Any > > > >> > >> views > > > >> > or > > > >> > >> opinions expressed are solely those of the author and do > > > >> > >> not necessarily represent those of Shape Blue Ltd or > > > >> > >> related > > companies. > > > >> > >> If you are not > > > >> > the > > > >> > >> intended recipient of this email, you must neither take > > > >> > >> any > > action > > > >> > >> based upon its contents, nor copy or show it to anyone. > > > >> > >> Please contact the > > > >> > sender > > > >> > >> if you believe you have received this email in error. > > > >> > >> Shape > Blue > > > >> > >> Ltd is > > > >> > a > > > >> > >> company incorporated in England & Wales. ShapeBlue > > > >> > >> Services > India > > > >> > >> LLP > > > >> > is a > > > >> > >> company incorporated in India and is operated under > > > >> > >> license > from > > > >> > >> Shape > > > >> > Blue > > > >> > >> Ltd. Shape Blue Brasil Consultoria Ltda is a company > incorporated > > > >> > >> in > > > >> > Brasil > > > >> > >> and is operated under license from Shape Blue Ltd. > > > >> > >> ShapeBlue SA > > Pty > > > >> > >> Ltd > > > >> > is > > > >> > >> a company registered by The Republic of South Africa and > > > >> > >> is > > traded > > > >> > >> under license from Shape Blue Ltd. ShapeBlue is a > > > >> > >> registered > > > >> trademark. > > > >> > >> > > > >> > > > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > Thanks for your support, > > > >> > > Motty > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> Thanks for your support, > > > >> Motty > > > >> > > > > > > > > > > > > > > > > -- > > > > Thanks for your support, > > > > Motty > > > > > > > > > > > > > > > > -- > > > Thanks for your support, > > > Motty > > > > > > > > > > > -- > > > > Andrija Panić > > -------------------------------------- > > http://admintweets.com > > -------------------------------------- > > > > > > -- > Thanks for your support, > Motty > -- Andrija Panić -------------------------------------- http://admintweets.com --------------------------------------
