Hi all, In my humble opinion, we should release 4.14 as it is (considering we have enough votes), but we'll further investigate the actual/behind-the-scene root-cause for the vSphere 6.7 harakiri (considering 6.0 and 6.5 are not affected) - this is possibly a VMware bug and we'll certainly try to address it.
If I don't hear any more concerns or -1 votes until tomorrow morning CET time, I will proceed with concluding the voting process and crafting the release. Thanks, Andrija On Tue, 19 May 2020 at 19:23, Pavan Kumar Aravapalli < pavankuma...@accelerite.com> wrote: > Thank you Bobby and Daan for the update. However I have not encountered > such issue while doing dev test with Vmware 5.5 & 6.5. > > > > > > Regards, > > Pavan Aravapalli. > > > ________________________________ > From: Daan Hoogland <daan.hoogl...@gmail.com> > Sent: 19 May 2020 20:56 > To: users <us...@cloudstack.apache.org> > Cc: dev@cloudstack.apache.org <dev@cloudstack.apache.org> > Subject: Re: [VOTE] Apache CloudStack 4.14.0.0 RC3 > > Thanks Bobby, > All, I've been closely working with Bobby and seen the same things. Does > anybody see any issues releasing 4.14 based on this code? I can confirm > that it is not Pavernalli's UEFI PR and we should not create a new PR to > revert it. > thanks for all of your patience, > > (this is me giving a binding +1) > > > On Tue, May 19, 2020 at 5:04 PM Boris Stoyanov < > boris.stoya...@shapeblue.com> > wrote: > > > Hi guys, > > > > I've done more testing around this and I can now confirm it has nothing > to > > do with cloudstack code. > > > > I've tested it with rc3, reverted UEFI PR and 4.13.1 (which does not > > happen to have the feature at all). Also I've used a matrix of VMware > > version of 6.0u2, 6.5u2 and 6.7u3. > > > > The bug is reproducible with all the cloudstack versions, and only vmware > > 6.7u3, I was not able to reproduce this with 6.5/6.0. All of my results > > during testing show it must be related to that specific version of > VMware. > > > > Therefore I'm reversing my '-1' and giving a +1 vote on the RC. I think > it > > needs to be included in release notes to refrain from that version for > now > > until further investigation is done. > > > > Thanks, > > Bobby. > > > > On 19.05.20, 10:08, "Boris Stoyanov" <boris.stoya...@shapeblue.com> > > wrote: > > > > Indeed it is severe, but please note it's a corner case which was > > unearthed almost by accident. It falls down to using a new feature of > > selecting a boot protocol and the template must be corrupted. So with > > already existing templates I would not expect to encounter it. > > > > As for recovery, we've managed to recover vCenter and Cloudstack > after > > reboots of the vCenter machine and the Cloudstack management service. > > There's no exact points to recover for now, but restart seems to work. > > By graceful failure I mean, cloudstack erroring out the deployment > and > > VM finished in ERROR state, meanwhile connection and operability with > > vCenter cluster remains the same. > > > > We're currently exploring options to fix this, one could be to > disable > > the feature for VMWare and work to introduce more sustainable fix in next > > release. Other is to look for more guarding code when installing a > > template, since VMware doesn’t actually allow you install that particular > > template but cloudstack does. We'll keep you posted. > > > > Thanks, > > Bobby. > > > > On 18.05.20, 23:01, "Marcus" <shadow...@gmail.com> wrote: > > > > The issue sounds severe enough that a release note probably won't > > suffice - > > unless there's a documented way to recover we'd never want to > > leave a > > system susceptible to being unrecoverable, even if it's rarely > > triggered. > > > > What's involved in "failing gracefully"? Is this a small fix, or > an > > overhaul? Perhaps the new feature could be disabled for VMware, > or > > disabled altogether until a fix is made in a patch release. > > > > Does it only affect new templates, or is there a risk that an > > existing > > template out in vSphere could suddenly cause problems? > > > > On Mon, May 18, 2020 at 12:49 AM Boris Stoyanov < > > boris.stoya...@shapeblue.com> wrote: > > > > > Hi guys, > > > > > > A little further info on this, it appears when we use a > > corrupted template > > > and UEFI/Legacy mode when deploy a VM, it breaks the connection > > between > > > cloudstack and vCenter. > > > > > > All hosts become unreachable and basically the cluster is not > > functional, > > > have not investigated a way to recover this but seems like a > > huge mess.. > > > Please note that user is not able to register such template in > > vCenter > > > directly, but cloudstack allows using it. > > > > > > Open to discuss if we'll fix this, since it's expected users to > > use > > > working templates, I think we should be failing gracefully and > > such action > > > should not be able to create downtime on such a large scale. > > > > > > I believe the boot type feature is new one and it's not > > available in older > > > releases, so this issue should be limited to 4.14/current > master. > > > > > > Thanks, > > > Bobby. > > > > > > On 15.05.20, 17:07, "Boris Stoyanov" < > > boris.stoya...@shapeblue.com> > > > wrote: > > > > > > I'll have to -1 RC3, we've discovered details about an > issue > > which is > > > causing severe consequences with a particular hypervisor in the > > afternoon. > > > We'll need more time to investigate before disclosing. > > > > > > Bobby. > > > > > > On 15.05.20, 9:12, "Boris Stoyanov" < > > boris.stoya...@shapeblue.com> > > > wrote: > > > > > > +1 (binding) > > > > > > I've executed upgrade tests with the following > > configurations: > > > > > > 4.13.1 with KVM on CentOS7 hosts > > > 4.13 with VMware6.5 hosts > > > 4.11.3 with KVM on CentOS7 hosts > > > 4.11.2 with XenServer7 hosts > > > 4.11.1 with VMware 6.7 > > > 4.9.3 with XenServer 7 hosts > > > 4.9.2 with KVM on CentOS 7 hosts > > > > > > Also I've run basic lifecycle operations on the > following > > > components: > > > VMs > > > Volumes > > > Infra (zones, pod, clusters, hosts) > > > Networks > > > and more > > > > > > I did not come across any problems during this testing. > > > > > > Thanks, > > > Bobby. > > > > > > > > > On 11.05.20, 18:21, "Andrija Panic" < > > andrija.pa...@gmail.com> > > > wrote: > > > > > > Hi All, > > > > > > I've created a 4.14.0.0 release (RC3), with the > > following > > > artefacts up for > > > testing and a vote: > > > > > > Git Branch and Commit SH: > > > > > > > > > https://gitbox.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/4.14.0.0-RC20200511T1503 > > > Commit: 6f96b3b2b391a9b7d085f76bcafa3989d9832b4e > > > > > > Source release (checksums and signatures are > > available at the > > > same > > > location): > > > > > https://dist.apache.org/repos/dist/dev/cloudstack/4.14.0.0/ > > > > > > PGP release keys (signed using 3DC01AE8): > > > > > https://dist.apache.org/repos/dist/release/cloudstack/KEYS > > > > > > The vote will be open until 14th May 2020, 17.00 > CET > > (72h). > > > > > > For sanity in tallying the vote, can PMC members > > please be > > > sure to indicate > > > "(binding)" with their vote? > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove (and reason why) > > > > > > Additional information: > > > > > > For users' convenience, I've built packages from > > > 6f96b3b2b391a9b7d085f76bcafa3989d9832b4e and > > published RC3 > > > repository here: > > > http://packages.shapeblue.com/testing/41400rc3/ > > (CentOS 7 and > > > Debian/generic, both with noredist support) > > > and here > > > > > > > > > https://download.cloudstack.org/testing/4.14.0.0-RC20200506T2028/ubuntu/bionic/ > > > (Ubuntu 18.04 specific, no noredist support - > > thanks to > > > Gabriel): > > > > > > The release notes are still work-in-progress, but > > for the > > > upgrade > > > instructions (including the new systemVM templates) > > you may > > > refer to the > > > following URL: > > > > > > > > > https://acs-www.shapeblue.com/docs/WIP-PROOFING/pr112/upgrading/index.html > > > > > > 4.14.0.0 systemVM templates are available from > here: > > > http://download.cloudstack.org/systemvm/4.14/ > > > > > > NOTES on issues fixed in this RC3 release: > > > > > > (this one does *NOT* require a full retest if you > > were testing > > > RC1/RC2 > > > already - just if you were affected this issue): > > > - https://github.com/apache/cloudstack/pull/4064 - > > affects > > > hostnames when > > > attaching a VM to additional networks > > > > > > Regards, > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > boris.stoya...@shapeblue.com > > > www.shapeblue.com<http://www.shapeblue.com> > > > 3 London Bridge Street, 3rd floor, News Building, London > > SE1 9SGUK > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > boris.stoya...@shapeblue.com > > > www.shapeblue.com<http://www.shapeblue.com> > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 > > 9SGUK > > > @shapeblue > > > > > > > > > > > > > > > > > > > > boris.stoya...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > @shapeblue > > > > > > > > > > > > > > boris.stoya...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > @shapeblue > > > > > > > > > > -- > Daan > -- Andrija Panić