Hi Sven,
ACS 4.12 does not have the fix to make it work with qemu-ev, we only recently got it in 4.11 and it is only available curently with 4.11.3.0 or latest master. Regards, Rohit Yadav Software Architect, ShapeBlue https://www.shapeblue.com ________________________________ From: Boris Stoyanov <boris.stoya...@shapeblue.com> Sent: Thursday, July 25, 2019 3:38:29 PM To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278 On my KVM host I have: CentOS Linux release 7.6.1810 (Core) Running the following qemu related packages qemu-img-ev-2.12.0-18.el7_6.5.1.x86_64 libvirt-daemon-driver-qemu-4.5.0-10.el7_6.12.x86_64 ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch centos-release-qemu-ev-1.0-4.el7.centos.noarch qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64 qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64 I was able to run system vms and they connect back to management, also was able to spin user vms and router I'm running ACS master (as of one of the latest meged PRs #3504) Am I missing something? Bobby. On 25.07.19, 12:38, "Sven Vogel" <s.vo...@ewerk.com> wrote: Hey Bobby, thanks Bobby. Which version of QEMU do you use? You mean 4.12 and later? Which CentOS and libvirt? We used the actually master, SystemVM and qemu 3.10 and 3.12. it seems it related to the cloudstack version 4.12 and above… Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actually master / SystemVM before 4.11.X --VR Dumpxml (two channel, first in use from libvirt, I know its disconnected but it seems here like the same not online) <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r-1741-VM.agent'/> <target type='virtio' name='r-1712-VM.vport' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel1'/> <address type='virtio-serial' controller='0' bus='0' port='2'/> </channel> — Socket seems there... [root@kvm08 /]# ll /var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0 srwxrwxr-x. 1 root root 0 Jul 23 11:49 /var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0 —related to the channel it seems to work [root@kvm08 /]# virsh qemu-agent-command r-1741-VM '{"execute":"guest-ping"}' {"return":{}} Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actually master / SystemVM 4.12.0 or later —VR Dumpxml <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> — Socket seems there… [root@kvm08 /]# ll /var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0 srwxrwxr-x. 1 root root 0 Jul 24 22:23 /var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0 —related to the channel it seems not to work [root@kvm08 /]# virsh qemu-agent-command r-1750-VM '{"execute":"guest-ping"}' error: Guest agent is not responding: QEMU guest agent is not connected After „second" restart of the systemVM the agent gets connected, patch.sh is running and cloud-early-config does configure the machine. The second channel is always connected from the first boot. The curious thing is that if we reboot the vpc provisioning is done correctly and the first channel is up too. It seems that libvirt use the first channel for its own. before it was a vport and agent.0 … but it seems only the second channel was used. —>https://wiki.libvirt.org/page/Qemu_guest_agent Configure guest agent without libvirt interference "In some cases, users might want to configure a guest agent in their domain XML but don't want libvirt to connect to guest agent socket at all. Because libvirt connects to a guest agent channel if and only if it is a virtio channel with org.qemu.guest_agent.0 name, all one need to do is void at least one of these conditions. However, the most feasible way is to change the target's name:" It seems we need to configure a second one as org.qemu.guest_agent.1 or use two of the channels! 1x org.qemu.guest_agent.1 or org.qemu.guest_agent.0 And org.qemu.guest_agent.1 Maybe I am wrong. Its only a theory that needs to be more investigation … I think we have a problem with the channels or qemu agent inside the SystemVM. Cheers, Sven __ Sven Vogel Teamlead Platform EWERK DIGITAL GmbH Brühl 24, D-04109 Leipzig P +49 341 42649 - 11 F +49 341 42649 - 18 s.vo...@ewerk.com www.ewerk.com<http://www.ewerk.com> Geschäftsführer: Dr. Erik Wende, Hendrik Schubert, Frank Richter Registergericht: Leipzig HRB 17023 Zertifiziert nach: ISO/IEC 27001:2013 DIN EN ISO 9001:2015 DIN ISO/IEC 20000-1:2011 ISAE 3402 Typ II Assessed EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/> Mit Handelsregistereintragung vom 09.07.2019 ist die EWERK RZ GmbH auf die EWERK IT GmbH verschmolzen und firmiert nun gemeinsam unter dem Namen: EWERK DIGITAL GmbH, für weitere Informationen klicken Sie hier<https://www.ewerk.com/ewerkdigital>. Auskünfte und Angebote per Mail sind freibleibend und unverbindlich. Disclaimer Privacy: Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank. The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you. Am 24.07.2019 um 09:47 schrieb Boris Stoyanov <boris.stoya...@shapeblue.com<mailto:boris.stoya...@shapeblue.com>>: Hi Sven, I've just installed master and upgraded my hosts to latest qemu-ev, then was able to deploy successfully the system vms, so looks like it's related to 4.12. Bobby. On 21.07.19, 18:23, "Sven Vogel" <s.vo...@ewerk.com<mailto:s.vo...@ewerk.com>> wrote: Hi Rohit, Thx for input. We don’t checked the 11.X.X version. We tied the upcoming master and 4.12. we check in the following week the XML from a actual Version (Master). Maybe you are right and we see only one. I think it’s a good idea to rid of the old adapter. I think what you mean is we should see only „path='/var/lib/libvirt/qemu/r-1710-VM.org.qemu.guest_agent.0'/><target type='virtio' name='org.qemu.guest_agent.0'/>“ that one. But why we don’t get an connection to the router/systemVM. So what i see is that the Agent socket it present. We will try the „virsh qemu-agent-command“ and see how if we get an connection. Do you think the problem is a Agent Problem into the SystemVM? Cheers Sven __ Sven Vogel Teamlead Platform EWERK RZ GmbH Brühl 24, D-04109 Leipzig P +49 341 42649 - 11 F +49 341 42649 - 18 s.vo...@ewerk.com<mailto:s.vo...@ewerk.com> www.ewerk.com<http://www.ewerk.com/> Geschäftsführer: Dr. Erik Wende, Hendrik Schubert, Frank Richter Registergericht: Leipzig HRB 17023 Zertifiziert nach: ISO/IEC 27001:2013 DIN EN ISO 9001:2015 DIN ISO/IEC 20000-1:2011 EWERK-Blog | LinkedIn | Xing | Twitter | Facebook Auskünfte und Angebote per Mail sind freibleibend und unverbindlich. Disclaimer Privacy: Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank. The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you. Am 21.07.2019 um 16:50 schrieb Rohit Yadav <rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>>: Hi Sven, Are you able to reproduce the failure against latest KVM qemu-ev + 4.11.3 systemvmtemplate with ACS 4.11.3.0? The PR https://github.com/apache/cloudstack/pull/3278 only fixes the issue towards 4.11.3.0 and upcoming master/4.13.0.0, qemu-ev will fail for 4.11.2.0 or 4.12.0.0. Also, with the PR/patch applied you should only see a single virtio serial port device on the domain xml, we got rid of the old/special case which would add a specific vport/serial-port device for systemvms. Regards, Rohit Yadav Software Architect, ShapeBlue https://www.shapeblue.com<https://www.shapeblue.com/> ________________________________ From: Sven Vogel <s.vo...@ewerk.com<mailto:s.vo...@ewerk.com>> Sent: Saturday, July 20, 2019 3:45:08 PM To: dev <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278 Hi Simon, Thanks for your information. We looking forward it you on a more recent mainline version. In the meantime we try find out more out more about this bug. Please feel free if you have new information add them here. In the meantime I wrote some comment below this PR #3278 Cheers Sven __ Sven Vogel Teamlead Platform EWERK DIGITAL GmbH Brühl 24, D-04109 Leipzig P +49 341 42649 - 11 F +49 341 42649 - 18 s.vo...@ewerk.com<mailto:s.vo...@ewerk.com> www.ewerk.com<http://www.ewerk.com> Geschäftsführer: Dr. Erik Wende, Hendrik Schubert, Frank Richter Registergericht: Leipzig HRB 17023 Zertifiziert nach: ISO/IEC 27001:2013 DIN EN ISO 9001:2015 DIN ISO/IEC 20000-1:2011 ISAE 3402 Typ II Assessed EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/> Mit Handelsregistereintragung vom 10.07.2019 ist die EWERK RZ GmbH auf die EWERK IT GmbH verschmolzen und firmiert zukünftig gemeinsam unter dem Namen: EWERK DIGITAL GmbH, für weitere Informationen klicken Sie hier<https://www.ewerk.com/ewerkdigital>. Auskünfte und Angebote per Mail sind freibleibend und unverbindlich. Disclaimer Privacy: Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank. The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you. Am 19.07.2019 um 15:04 schrieb Simon Weller <swel...@ena.com.INVALID<mailto:swel...@ena.com.INVALID>>: Hi Sven, We're still using 2.10 right now and we haven't tested the patches yet for 2.12. Having said that, we're currently trying to get our internal ACS release up to a more recent mainline version, so I'd suspect we'll have to dive into this fairly soon. -Si ________________________________ From: Sven Vogel <s.vo...@ewerk.com<mailto:s.vo...@ewerk.com>> Sent: Friday, July 19, 2019 6:48 AM To: dev <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278 Hi Guys, Sorry that I formal reopen this issue. We tested the actual system with this patch #3278 We use CentOS 1805 or 1810 and OVS and QEMU 2.12. It seems the first agent contact will not work so that the agent know the virtual machine is running. With 2.10 and the old agent, SystemVM it works. 2019-07-19 00:17:30,571 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:8ad14091) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n v-1728-VM -c template=domP type=consoleproxy host=10.24.48.46 port=8250 name=v-1728-VM premium=true zone=4 pod=4 guid=Proxy.1728 proxy_vm=1728 disable_rp_filter=true eth2ip=185.232.219.237 eth2mask=255.255.255.248 gateway=185.232.219.233 eth0ip=169.254.0.193 eth0mask=255.255.0.0 eth1ip=10.24.48.124 eth1mask=255.255.255.0 mgmtcidr=10.24.48.0/24 localgw=10.24.48.1 internaldns1=10.24.48.33 internaldns2=10.24.48.34 dns1=217.69.224.73 dns2=85.232.28.146 2019-07-19 00:17:30,573 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:8ad14091) Executing while with timeout : 300000 2019-07-19 00:17:30,616 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) (logid:7577151e) Unable to logon to 169.254.2.189 2019-07-19 00:17:30,616 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) (logid:7577151e) Trying to connect to 169.254.2.189 2019-07-19 00:17:31,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:04b4d6ce) Could not connect to 169.254.1.21 2019-07-19 00:17:33,622 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) (logid:7577151e) Could not connect to 169.254.2.189 2019-07-19 00:17:36,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:04b4d6ce) Unable to logon to 169.254.1.21 2019-07-19 00:17:36,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:04b4d6ce) Trying to connect to 169.254.1.21 2019-07-19 00:17:38,622 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) (logid:7577151e) Trying to connect to 169.254.2.189 @simonweller you encountered the same problem? @rohit maybe there is something what we forget or what’s wrong? @wido do you have ideas? thanks Sven __ Sven Vogel Teamlead Platform EWERK DIGITAL GmbH Brühl 24, D-04109 Leipzig P +49 341 42649 - 11 F +49 341 42649 - 18 s.vo...@ewerk.com<mailto:s.vo...@ewerk.com> www.ewerk.com<http://www.ewerk.com/><http://www.ewerk.com<http://www.ewerk.com/>> Geschäftsführer: Dr. Erik Wende, Hendrik Schubert, Frank Richter Registergericht: Leipzig HRB 17023 Zertifiziert nach: ISO/IEC 27001:2013 DIN EN ISO 9001:2015 DIN ISO/IEC 20000-1:2011 ISAE 3402 Typ II Assessed EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group> | Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group> | Facebook<https://de-de.facebook.com/EWERK.IT/> Mit Handelsregistereintragung vom 10.07.2019 ist die EWERK RZ GmbH auf die EWERK IT GmbH verschmolzen und firmiert zukünftig gemeinsam unter dem Namen: EWERK DIGITAL GmbH, für weitere Informationen klicken Sie hier<https://www.ewerk.com/ewerkdigital>. Auskünfte und Angebote per Mail sind freibleibend und unverbindlich. Disclaimer Privacy: Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank. The contents of this e-mail (including any attachments) are confidential and may be legally privileged. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. Thank you. Am 22.04.2019 um 21:29 schrieb Simon Weller <swel...@ena.com.INVALID<mailto:swel...@ena.com.INVALID><mailto:swel...@ena.com.INVALID>>: In our case the SystemVMs were booting fine, but ACS wasn't able to inject the payload via the socket. rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue boris.stoya...@shapeblue.com<mailto:boris.stoya...@shapeblue.com> www.shapeblue.com<http://www.shapeblue.com/> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue boris.stoya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue rohit.ya...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
