This saga is to be continued. "Security groups" was the correct keyword to resolve my problem.
Now all is in order and all VMs run. One observation: This guide here suggests to configure /etc/libvirt/libvirtd.conf and /etc/sysconfig/libvirtd under Libvirt Configuration But these files get overwritten every time cloudstack-agent service is restarted. I think, there is inconsistency in this guide for sure. But Now I face the other problem , which is probably related to correct configuration of security groups, but maybe it a bug We have following config VM1 - running ngninx proxy VM2 - server hosting webapp on 8080 VM3 - server hosting another webapp on 8080. This webapp is exposing is connection over websocket - serving data stream (1) client -> VM1:80/app -> VM2:8080/app (2) client -> VM1:80/data -> VM3:8080/data This was working fine before the reinstallation. We found that it works, if we stop iptables. But with iptables ON, (1) works, but (2) does not work - it gives connection refused. How can this be resolved? On Fri, Mar 22, 2019 at 11:19 AM Dag Sonstebo <dag.sonst...@shapeblue.com> wrote: > Jevgeni - you've not provided any network troubleshooting findings - but > this is all down to security groups so check these are in place and working. > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > > On 21/03/2019, 19:47, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> > wrote: > > <<<Did you got your new installation running fine ? > Almost, but not completely. > > I am moving VMs one by one. They run and they get IP address from > Cloudstack and get connected to network. > > But I cannot connect to VMs from other PCs in the same LAN. Ping is not > responding too. > What can be the problem here? > > > On Thu, 21 Mar 2019, 21:23 Andrija Panic, <andrija.pa...@gmail.com> > wrote: > > > Stick to 4.11.2 - 4.12 should be released withing few days > officially. > > > > As for qemu-kvm-ev - yes, it's supposed to work - make sure to test > new > > versions obviously. > > > > Did you got your new installation running fine ? > > > > On Thu, 21 Mar 2019 at 19:26, Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > > wrote: > > > > > Andrija, > > > > > > I asked here in the group if its safe to try new version of KVM > and got > > > reply, that it works. It was back in September. So we installed it > with > > > yum install centos-release-qemu-ev > > > yum install qemu-kvm-ev > > > > > > It worked fine ever since. > > > But with new maintenance (yum update) apparently some breaking > changes > > were > > > introduced. > > > So, take care. > > > > > > Anyway, thanks. for help. > > > > > > As for your suggestion to use CS4.12. I haven't managed to find > systemvm > > > images for 4.12. Should I continue to use 4.11.12 systemvm? > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 21, 2019 at 7:19 PM Andrija Panic < > andrija.pa...@gmail.com> > > > wrote: > > > > > > > Jevgeni, qemu-kvm 1.5.3 is the lastest official one for CentoS > 7.6.XXX > > > > (latest) which I'm running atm in my lab (just checked for > update) - > > how > > > > did you manage to go to 2.0 (custom repo ?) > > > > > > > > On Thu, 21 Mar 2019 at 18:13, Ivan Kudryavtsev < > > kudryavtsev...@bw-sw.com > > > > > > > > wrote: > > > > > > > > > Jevgeniy, simplest and the most obvious way is to flatten their > > images > > > > with > > > > > "qemu-img convert", next import them as templates and recreate > VMs > > from > > > > > those templates. > > > > > > > > > > чт, 21 мар. 2019 г. в 13:05, Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com > > > > >: > > > > > > > > > > > What happened in the end was: qemu-kvm got updated to > version 2.0 > > > > during > > > > > > the maintenance. We could not manage to make this KVM to > work with > > > > > > Cloudstack. > > > > > > So we rolled back to version 1.5.3. > > > > > > > > > > > > And now we have clean cloudstack fully operational. We can > create > > new > > > > VMs > > > > > > and it works. I am almost happy. > > > > > > > > > > > > Now question - how do I get my old VMs to work, considering > I have > > > only > > > > > > their volumes? > > > > > > > > > > > > On Thu, Mar 21, 2019 at 6:24 PM Andrija Panic < > > > andrija.pa...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Just replace the URL for systemVM template from 4.11.1 with > > 4.11.2 > > > > > (there > > > > > > > is a PR for this now). > > > > > > > > > > > > > > On Thu, 21 Mar 2019 at 16:53, Andrija Panic < > > > andrija.pa...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Please use the one, updated specifically for CentOS 7 - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/cloudstack-documentation/blob/master/source/quickinstallationguide/qig.rst > > > > > > > > > > > > > > > > And please avoid collocating KVM and MGMT on same server > > > > (especially > > > > > in > > > > > > > > any production-like system) > > > > > > > > > > > > > > > > Please let me know if the guide above gives you problem > - we > > had > > > > > > multiple > > > > > > > > users explicitly following it - and successfully > installed > > (with > > > > some > > > > > > > minor > > > > > > > > modification, which we committed back to that guide). > > > > > > > > > > > > > > > > Thanks > > > > > > > > Andrija > > > > > > > > > > > > > > > > On Thu, 21 Mar 2019 at 16:34, Jevgeni Zolotarjov < > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> OS management - centos 7 (1810) > > > > > > > >> OS hypervisor - centos 7 (1810) > > > > > > > >> > > > > > > > >> Basic zone - yes > > > > > > > >> I am following this quide > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://docs.cloudstack.apache.org/en/4.11.2.0/quickinstallationguide/qig.html > > > > > > > >> > > > > > > > >> Right now from scratch - management ans hypervisor on > the same > > > > > machine > > > > > > > >> qemu - version 1.5.3 > > > > > > > >> libvirt - libvirt version: 4.5.0, package: 10.el7_6.6 > > > > > > > >> > > > > > > > >> Basically - everything out of the box of clean centos > install > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Thu, Mar 21, 2019 at 5:08 PM Andrija Panic < > > > > > > andrija.pa...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Hey Jevgeni, > > > > > > > >> > > > > > > > > >> > what OS mgmt, what OS hypervisor, what qemu/libvirt > > versions - > > > > > still > > > > > > > in > > > > > > > >> > Basic Zone, SG ? > > > > > > > >> > > > > > > > > >> > Andrija > > > > > > > >> > > > > > > > > >> > On Thu, 21 Mar 2019 at 13:06, Jevgeni Zolotarjov < > > > > > > > >> j.zolotar...@gmail.com> > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > >> > > I reinstalled cloudstack from scratch - everything > > > > > > > >> > > > > > > > > > >> > > But looks like I hit the same wall now > > > > > > > >> > > > > > > > > > >> > > In the last step of installation it cannot create > system > > > VMs. > > > > > > > >> > > > > > > > > > >> > > service libvirtd status -l > > > > > > > >> > > gives me > > > > > > > >> > > ------------------------------------ > > > > > > > >> > > ● libvirtd.service - Virtualization daemon > > > > > > > >> > > Loaded: loaded > > (/usr/lib/systemd/system/libvirtd.service; > > > > > > > enabled; > > > > > > > >> > > vendor preset: enabled) > > > > > > > >> > > Active: active (running) since Thu 2019-03-21 > 11:45:00 > > > GMT; > > > > > > 18min > > > > > > > >> ago > > > > > > > >> > > Docs: man:libvirtd(8) > > > > > > > >> > > https://libvirt.org > > > > > > > >> > > Main PID: 537 (libvirtd) > > > > > > > >> > > Tasks: 20 (limit: 32768) > > > > > > > >> > > CGroup: /system.slice/libvirtd.service > > > > > > > >> > > ├─ 537 /usr/sbin/libvirtd -l > > > > > > > >> > > ├─12206 /usr/sbin/dnsmasq > > > > > > > >> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > > --leasefile-ro > > > > > > > >> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > > > > >> > > └─12207 /usr/sbin/dnsmasq > > > > > > > >> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > > --leasefile-ro > > > > > > > >> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > > > > >> > > > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > > 2019-03-21 > > > > > > > >> > > 11:45:01.168+0000: 566: info : libvirt version: > 4.5.0, > > > > package: > > > > > > > >> > 10.el7_6.6 > > > > > > > >> > > (CentOS BuildSystem <http://bugs.centos.org>, > > > > > > 2019-03-14-10:21:47, > > > > > > > >> > > x86-01.bsys.centos.org) > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > > 2019-03-21 > > > > > > > >> > > 11:45:01.168+0000: 566: info : hostname: > > > > > > > mtl1-apphst03.mt.pbt.com.mt > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > > 2019-03-21 > > > > > > > >> > > 11:45:01.168+0000: 566: error : > > > > virFirewallApplyRuleDirect:709 : > > > > > > > >> internal > > > > > > > >> > > error: Failed to apply firewall rules > /usr/sbin/iptables > > -w > > > > > > --table > > > > > > > >> nat > > > > > > > >> > > --insert POSTROUTING --source 192.168.122.0/24 '!' > > > > > --destination > > > > > > > >> > > 192.168.122.0/24 --jump MASQUERADE: iptables > v1.4.21: > > can't > > > > > > > >> initialize > > > > > > > >> > > iptables table `nat': Table does not exist (do you > need to > > > > > > insmod?) > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > Perhaps > > > > > > > >> > > iptables > > > > > > > >> > > or your kernel needs to be upgraded. > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > dnsmasq[12206]: > > > > > read > > > > > > > >> > > /etc/hosts > > > > > > > >> > > - 4 addresses > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > dnsmasq[12206]: > > > > > read > > > > > > > >> > > /var/lib/libvirt/dnsmasq/default.addnhosts - 0 > addresses > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > > > > dnsmasq-dhcp[12206]: > > > > > > > read > > > > > > > >> > > /var/lib/libvirt/dnsmasq/default.hostsfile > > > > > > > >> > > Mar 21 11:45:01 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > > 2019-03-21 > > > > > > > >> > > 11:45:01.354+0000: 566: warning : > > virSecurityManagerNew:189 > > > : > > > > > > > >> Configured > > > > > > > >> > > security driver "none" disables default policy to > create > > > > > confined > > > > > > > >> guests > > > > > > > >> > > Mar 21 11:49:57 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > > 2019-03-21 > > > > > > > >> > > 11:49:57.354+0000: 542: warning : > qemuDomainObjTaint:7521 > > : > > > > > Domain > > > > > > > >> id=2 > > > > > > > >> > > name='s-1-VM' > uuid=1a06d3a7-4e3f-4cba-912f-74ae24569bac is > > > > > > tainted: > > > > > > > >> > > high-privileges > > > > > > > >> > > Mar 21 11:49:59 mtl1-apphst03.mt.pbt.com.mt > > libvirtd[537]: > > > > > > > 2019-03-21 > > > > > > > >> > > 11:49:59.402+0000: 540: warning : > qemuDomainObjTaint:7521 > > : > > > > > Domain > > > > > > > >> id=3 > > > > > > > >> > > name='v-2-VM' > uuid=af2a8342-cd9b-4b55-ba12-480634a31d65 is > > > > > > tainted: > > > > > > > >> > > high-privileges > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > What can be done about that ? > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > -- > > > > > > > >> > > > > > > > > >> > Andrija Panić > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > With best regards, Ivan Kudryavtsev > > > > > Bitworks LLC > > > > > Cell RU: +7-923-414-1515 > > > > > Cell USA: +1-201-257-1512 > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrija Panić > > > > > > > > > > > > > -- > > > > Andrija Panić > > > > > > dag.sonst...@shapeblue.com > www.shapeblue.com > Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > >
