Hi, There is no need to get rid of qemu-ev packages. Just downgrade them to 2.10 and it works fine.
On Wed, May 22, 2019 at 10:52 PM Andrija Panic <andrija.pa...@gmail.com> wrote: > > Important to see you happy with 4.11.2 - the rest (HW) will follow ;) > > On Wed, 22 May 2019 at 21:49, Eric Lee Green <eric.lee.gr...@gmail.com> > wrote: > > > EXCELLENT! > > > > That did it. I downgraded to the libvirt and openssh rpms from Centos > > 7.5, which was the latest DVD I had hanging around (likely the one I > > used to install the systems with in the first place). Turns out it was > > the Red Hat updates that came in when I did 'rpm update' that broke > > things, not the Cloudstack upgrade. I am now happily running on > > Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had > > installed that at one point trying to resolve the problems, but have now > > downgraded it back to the release stuff. > > > > In case it is not obvious, I am the person who is serious about security > > policies. The company itself doesn't care as long as we have one to show > > to investors and customers. > > > > Regarding hardware, security policies don't cost a lot of money given > > all the open source tools we have available now. New hardware does cost > > money. Old commercial grade hardware is ridiculously cheap from Fleabay > > right now as big corporations dump "obsolete" gear. I bought three used > > commercial-grade 24-port 10 gigabit switches for the price of a single > > new 1 gigabit switch, for example. I won't have cheap white box junk in > > the back room, it's three racks of commercial-quality gear with dual > > processors, ECC, 10gbit Ethernet, IPMI for remote management, etc., > > reliable as a brick -- just old and a bit power hungry. Power is the > > biggest issue that will lead to an upgrade, not reliability or > > performance -- I only have power budget for three racks, and new > > hardware is much more dense for a specific power consumption (modern 16 > > core processors use about the same power as the 6 core processors I have > > in my nodes). When I have the budget to upgrade the hardware, I will, > > but what's there works, is reliable as a brick and not an issue, unlike > > Red Hat breaking my infrastructure with incompatible updates GRR! > > > > On 5/22/19 9:20 AM, Andrija Panic wrote: > > > +1 on what Nux said - simply downgrade the packages and check if that > > works > > > - no need to reinstall everything ! > > > As for the $3333 from github (CentOS 7.6 issue) - simple new line at the > > > end of id_rsa file and/or downgrading the ssh-keygen will work. > > > > > > Don't want to annoy you, but if the company is so serious about security > > > policies ask them to be serious about HW as well (regular white trash PC > > > will do...so you can test software itself) - otherwise, you personally > > > suffer the consequences and look bad (had the same pain many years > > ago...). > > > > > > On Wed, 22 May 2019 at 17:44, Nux! <n...@li.nux.ro> wrote: > > > > > >> Eric, > > >> > > >> I think you can get away with just downgrading to stock qemu-kvm > > packages > > >> (or patch your cloudstack installation). > > >> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages > > >> don't get the same level of testing and QA. > > >> > > >> -- > > >> Sent from the Delta quadrant using Borg technology! > > >> > > >> Nux! > > >> www.nux.ro > > >> > > >> ----- Original Message ----- > > >>> From: "Eric Lee Green" <eric.lee.gr...@gmail.com> > > >>> To: "users" <users@cloudstack.apache.org> > > >>> Sent: Wednesday, 22 May, 2019 15:33:15 > > >>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN* > > >>> Okay. This makes sense. > > >>> > > >>> And people wonder why Amazon decided to make their own Linux rather > > than > > >>> use Centos and why Ubuntu has seized huge market share from Red Hat in > > >>> the past few years. SIGH. > > >>> > > >>> Downgrading my CentOS is not going to be easy. There were security > > >>> patches in the latest CentOS that I am required to have. It would > > >>> actually be easier to just switch to Ubuntu at this point since we're > > >>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat > > Software. > > >>> I hope IBM fixes them, but I have my doubts. > > >>> > > >>> On 5/22/2019 1:05 AM, Andrija Panic wrote: > > >>>> Hi Eric, all, > > >>>> > > >>>> I believe you might be hitting this one - issues on latest CentOS 7.6: > > >>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the > > OS > > >>>> itself... > > >>>> > > >>>> If you believe that is the case, please try with CentOS 7.4 (can > > confirm > > >>>> works fine) and/or CentOS 7.5. > > >>>> > > >>>> Best, > > >>>> Andrija > > >>>> > > >>>> > > >>>> > > >>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green < > > eric.lee.gr...@gmail.com> > > >>>> wrote: > > >>>> > > >>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote: > > >>>>>> Hello Eric, > > >>>>>> > > >>>>>> If the router version is displayed as UNKNOWN in the portal, > > it > > >>>>> would be > > >>>>>> a connectivity issue. Check your connections related to firewall > > rules > > >>>>>> between the ACP management hosts, hypervisor and VR. Is your VR > > >>>>> management > > >>>>>> network setup correctly? > > >>>>> Communications between the hosts is working fine and I have not > > changed > > >>>>> the VR management network between the running 4.9 and the not-running > > >>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack > > and > > >>>>> should properly forward VR traffic. > > >>>>> > > >>>>> More tomorrow when I have some sleep since it is now midnight here in > > >>>>> the SF Bay area. > > >>>>> > > >>>>> > > >>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, < > > >> eric.lee.gr...@gmail.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Thanks for the response, sorry if I sound frustrated, but this is > > >>>>>>> supposed to be a simple easy process and it's been horrible all the > > >> way > > >>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now > > >> 4.11.2 > > >>>>>>> has failed to upgrade too. I've thus far spent around 16 hours of > > my > > >>>>>>> time on what should have taken an hour at most. I'm frustrated and > > >>>>> bummed. > > >>>>>>> [root@cloud2 ~]# rpm -q centos-release > > >>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64 > > >>>>>>> [root@cloud2 ~]# rpm -q libvirt > > >>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64 > > >>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm > > >>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64 > > >>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64 > > >>>>>>> [root@cloud2 ~]# cat /proc/version > > >>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64 > > >>>>>>> (buil...@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red > > >> Hat > > >>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018 > > >>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo > > >>>>>>> ... > > >>>>>>> processor : 23 > > >>>>>>> vendor_id : GenuineIntel > > >>>>>>> cpu family : 6 > > >>>>>>> model : 44 > > >>>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz > > >>>>>>> stepping : 2 > > >>>>>>> microcode : 0x1f > > >>>>>>> cpu MHz : 3068.000 > > >>>>>>> cache size : 12288 KB > > >>>>>>> physical id : 1 > > >>>>>>> siblings : 12 > > >>>>>>> core id : 10 > > >>>>>>> cpu cores : 6 > > >>>>>>> apicid : 53 > > >>>>>>> initial apicid : 53 > > >>>>>>> fpu : yes > > >>>>>>> fpu_exception : yes > > >>>>>>> cpuid level : 11 > > >>>>>>> wp : yes > > >>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr > > >> pge > > >>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe > > >>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts > > >> rep_good > > >>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor > > >> ds_cpl > > >>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt > > >>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid > > >> dtherm > > >>>>>>> ida arat > > >>>>>>> bogomips : 6133.21 > > >>>>>>> clflush size : 64 > > >>>>>>> cache_alignment : 64 > > >>>>>>> address sizes : 40 bits physical, 48 bits virtual > > >>>>>>> [root@cloud1 ~]# free > > >>>>>>> total used free shared > > buff/cache > > >>>>>>> available > > >>>>>>> Mem: 123596388 79795792 34047696 632116 9752900 > > >>>>> 39999332 > > >>>>>>> Swap: 4194300 1956824 2237476 > > >>>>>>> [root@cloud1 ~]# yum update > > >>>>>>> > > >>>>>>> > > >> > > ========================================================================================================================================================================= > > >>>>>>> Package Arch Version Repository > > Size > > >>>>>>> > > >>>>>>> > > >> > > ========================================================================================================================================================================= > > >>>>>>> Updating: > > >>>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos > > >>>>>>> cloudstack411 47 M > > >>>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos > > >>>>>>> cloudstack411 58 M > > >>>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos > > >>>>>>> cloudstack411 82 M > > >>>>>>> > > >>>>>>> Three compute servers running the above processor averaging 128gb > > of > > >>>>>>> memory apiece, two data servers running NFS for primary and > > secondary > > >>>>>>> storage. Running the 4.11.2 community RPM's above. > > >>>>>>> > > >>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. > > The > > >>>>>>> routers in fact started with the template, it said 4.11.2 when I > > >> opened > > >>>>>>> up the console window and looked at the login prompt. But they > > never > > >> got > > >>>>>>> configured, when I logged in with root/password at the console > > prompt > > >>>>>>> they had no networking set up and no configuration provided, the > > >>>>>>> cloud-init output in /var/log was essentially blank. > > >>>>>>> > > >>>>>>> Do I recall correctly that there is an issue with a particular > > >> version > > >>>>>>> of the hypervisor on the latest Centos 7? Any other information > > that > > >> you > > >>>>>>> need? I think I provided the complete log file for the cloud-init > > of > > >> the > > >>>>>>> router in another email... > > >>>>>>> > > >>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote: > > >>>>>>>> Hi Eric, > > >>>>>>>> > > >>>>>>>> Can you describe your environment in detail such as management > > >> server > > >>>>>>> host distro, hypervisor version, current CloudStack version, did > > you > > >>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc. > > >>>>>>>> Regards. > > >>>>>>>> > > >>>>>>>> Regards, > > >>>>>>>> Rohit Yadav > > >>>>>>>> > > >>>>>>>> ________________________________ > > >>>>>>>> From: Eric Lee Green <eric.lee.gr...@gmail.com> > > >>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM > > >>>>>>>> To: users@cloudstack.apache.org > > >>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN* > > >>>>>>>> > > >>>>>>>> You may remember me as the person who had to roll back to > > Cloudstack > > >>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual > > machines > > >>>>> once > > >>>>>>>> I upgraded to it, claiming that there were inadequate resources > > even > > >>>>>>>> though I had over 150 gigabytes of memory free in my cluster and > > >> oodles > > >>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a > > >>>>> 512mb > > >>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and > > >>>>>>>> *again* it's misbehaving. > > >>>>>>>> > > >>>>>>>> The symptom is that my virtual routers when I log into their > > console > > >>>>>>>> show 4.11.2 but when I look at them in the console they say > > >> 'Version: > > >>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or > > link > > >>>>>>>> local IP address it fails. And when I try to start up a virtual > > >> machine > > >>>>>>>> that uses that virtual network, it says "Network unavailable', > > even > > >>>>>>>> though the router for that network is showing up and running. > > >>>>>>>> > > >>>>>>>> Clearly something's broken in the virtual routers but I don't know > > >> what > > >>>>>>>> because I can't get into the router virtual machines. How do I get > > >> the > > >>>>>>>> console password to get into the router virtual machines? It's > > >>>>> encrypted > > >>>>>>>> in the database (duh), how do I decrypt it? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> rohit.ya...@shapeblue.com > > >>>>>>>> www.shapeblue.com > > >>>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK > > >>>>>>>> @shapeblue > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > > > > > > > > -- > > Andrija Panić
