Eric, I think you can get away with just downgrading to stock qemu-kvm packages (or patch your cloudstack installation). BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages don't get the same level of testing and QA.
-- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ----- Original Message ----- > From: "Eric Lee Green" <eric.lee.gr...@gmail.com> > To: "users" <users@cloudstack.apache.org> > Sent: Wednesday, 22 May, 2019 15:33:15 > Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN* > Okay. This makes sense. > > And people wonder why Amazon decided to make their own Linux rather than > use Centos and why Ubuntu has seized huge market share from Red Hat in > the past few years. SIGH. > > Downgrading my CentOS is not going to be easy. There were security > patches in the latest CentOS that I am required to have. It would > actually be easier to just switch to Ubuntu at this point since we're > talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software. > I hope IBM fixes them, but I have my doubts. > > On 5/22/2019 1:05 AM, Andrija Panic wrote: >> Hi Eric, all, >> >> I believe you might be hitting this one - issues on latest CentOS 7.6: >> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS >> itself... >> >> If you believe that is the case, please try with CentOS 7.4 (can confirm >> works fine) and/or CentOS 7.5. >> >> Best, >> Andrija >> >> >> >> On Wed, 22 May 2019 at 09:04, Eric Lee Green <eric.lee.gr...@gmail.com> >> wrote: >> >>> On 5/21/2019 11:10 PM, Thomas Joseph wrote: >>>> Hello Eric, >>>> >>>> If the router version is displayed as UNKNOWN in the portal, it >>> would be >>>> a connectivity issue. Check your connections related to firewall rules >>>> between the ACP management hosts, hypervisor and VR. Is your VR >>> management >>>> network setup correctly? >>> Communications between the hosts is working fine and I have not changed >>> the VR management network between the running 4.9 and the not-running >>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and >>> should properly forward VR traffic. >>> >>> More tomorrow when I have some sleep since it is now midnight here in >>> the SF Bay area. >>> >>> >>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <eric.lee.gr...@gmail.com> >>>> wrote: >>>> >>>>> Thanks for the response, sorry if I sound frustrated, but this is >>>>> supposed to be a simple easy process and it's been horrible all the way >>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2 >>>>> has failed to upgrade too. I've thus far spent around 16 hours of my >>>>> time on what should have taken an hour at most. I'm frustrated and >>> bummed. >>>>> [root@cloud2 ~]# rpm -q centos-release >>>>> centos-release-7-6.1810.2.el7.centos.x86_64 >>>>> [root@cloud2 ~]# rpm -q libvirt >>>>> libvirt-4.5.0-10.el7_6.9.x86_64 >>>>> [root@cloud1 ~]# rpm -qa | grep kvm >>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64 >>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64 >>>>> [root@cloud2 ~]# cat /proc/version >>>>> Linux version 3.10.0-862.6.3.el7.x86_64 >>>>> (buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat >>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018 >>>>> [root@cloud2 ~]# cat /proc/cpuinfo >>>>> ... >>>>> processor : 23 >>>>> vendor_id : GenuineIntel >>>>> cpu family : 6 >>>>> model : 44 >>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz >>>>> stepping : 2 >>>>> microcode : 0x1f >>>>> cpu MHz : 3068.000 >>>>> cache size : 12288 KB >>>>> physical id : 1 >>>>> siblings : 12 >>>>> core id : 10 >>>>> cpu cores : 6 >>>>> apicid : 53 >>>>> initial apicid : 53 >>>>> fpu : yes >>>>> fpu_exception : yes >>>>> cpuid level : 11 >>>>> wp : yes >>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge >>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe >>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good >>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl >>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt >>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm >>>>> ida arat >>>>> bogomips : 6133.21 >>>>> clflush size : 64 >>>>> cache_alignment : 64 >>>>> address sizes : 40 bits physical, 48 bits virtual >>>>> [root@cloud1 ~]# free >>>>> total used free shared buff/cache >>>>> available >>>>> Mem: 123596388 79795792 34047696 632116 9752900 >>> 39999332 >>>>> Swap: 4194300 1956824 2237476 >>>>> [root@cloud1 ~]# yum update >>>>> >>>>> >>> ========================================================================================================================================================================= >>>>> Package Arch Version Repository Size >>>>> >>>>> >>> ========================================================================================================================================================================= >>>>> Updating: >>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos >>>>> cloudstack411 47 M >>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos >>>>> cloudstack411 58 M >>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos >>>>> cloudstack411 82 M >>>>> >>>>> Three compute servers running the above processor averaging 128gb of >>>>> memory apiece, two data servers running NFS for primary and secondary >>>>> storage. Running the 4.11.2 community RPM's above. >>>>> >>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The >>>>> routers in fact started with the template, it said 4.11.2 when I opened >>>>> up the console window and looked at the login prompt. But they never got >>>>> configured, when I logged in with root/password at the console prompt >>>>> they had no networking set up and no configuration provided, the >>>>> cloud-init output in /var/log was essentially blank. >>>>> >>>>> Do I recall correctly that there is an issue with a particular version >>>>> of the hypervisor on the latest Centos 7? Any other information that you >>>>> need? I think I provided the complete log file for the cloud-init of the >>>>> router in another email... >>>>> >>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote: >>>>>> Hi Eric, >>>>>> >>>>>> Can you describe your environment in detail such as management server >>>>> host distro, hypervisor version, current CloudStack version, did you >>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc. >>>>>> Regards. >>>>>> >>>>>> Regards, >>>>>> Rohit Yadav >>>>>> >>>>>> ________________________________ >>>>>> From: Eric Lee Green <eric.lee.gr...@gmail.com> >>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM >>>>>> To: users@cloudstack.apache.org >>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN* >>>>>> >>>>>> You may remember me as the person who had to roll back to Cloudstack >>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines >>> once >>>>>> I upgraded to it, claiming that there were inadequate resources even >>>>>> though I had over 150 gigabytes of memory free in my cluster and oodles >>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a >>> 512mb >>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and >>>>>> *again* it's misbehaving. >>>>>> >>>>>> The symptom is that my virtual routers when I log into their console >>>>>> show 4.11.2 but when I look at them in the console they say 'Version: >>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link >>>>>> local IP address it fails. And when I try to start up a virtual machine >>>>>> that uses that virtual network, it says "Network unavailable', even >>>>>> though the router for that network is showing up and running. >>>>>> >>>>>> Clearly something's broken in the virtual routers but I don't know what >>>>>> because I can't get into the router virtual machines. How do I get the >>>>>> console password to get into the router virtual machines? It's >>> encrypted >>>>>> in the database (duh), how do I decrypt it? >>>>>> >>>>>> >>>>>> >>>>>> rohit.ya...@shapeblue.com >>>>>> www.shapeblue.com >>>>>> Amadeus House, Floral Street, London WC2E 9DPUK >>>>>> @shapeblue >>>>>> >>>>>> >>>>>>
