hi David, I cleared all the experimental environment and redeployed the test. pvlan can work normally, and a VM with a PVLAN network + an isolated network can also work normally.
However, I found that when the VR where the PVLAN is located and the VM are not in the same HOST, the VM cannot obtain the IP of the PVLAN. I checked ovs flows and the flow table was issued. (VM has Pvlan Network and Isolated Network, Isolated Network can DHCP to IP, Pvlan Network can not get. (Migrate VM to HOST where PVLAN VR is located, then VM can get all IPs) Does PVLAN need other switch support? My switch all port trunk all -----邮件原件----- 发件人: David Jumani <david.jum...@shapeblue.com> 发送时间: 2020年5月29日 12:36 收件人: users@cloudstack.apache.org 主题: Re: ACS 4.13.1 failed to create PVLAN network That's great! There will be no communication between the devices on a PVLAN if they're isolated except DHCP (since isolated should not be able to communicate with each other). About multiple NICs, in my setup, I've had a VM attached to a PVLAN as well as an Isolated and it worked. Haven't tried multiple PVLANs though ________________________________ From: li jerry <div...@hotmail.com> Sent: Thursday, May 28, 2020 2:05 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: ACS 4.13.1 failed to create PVLAN network Thanks to David, Boris and all friends. I am on the master branch of clone github.com, and then merge PR [https://github.com/apache/cloudstack/pull/4040]. I have successfully implemented a PVLAN network in my test environment. Realize no communication between VM and VM; Communication between VM and DHCP (dhcp request) However, during the test, the following two problems were encountered. 1. VM does not support multiple NICs When creating vm> 1 NIC, the cloudstack-agent not create ovs flows. 2. L3 PVLAN, Secondary Isolated VLAN Type = Isolated. VM cannot get userdata information I tested in the VM and found that only DHCP requests can be sent to the VR, and other requests HTTP 80 AND 8080 can not communicate. Is this a bug? Or am I doing something wrong? -----邮件原件----- 发件人: David Jumani <david.jum...@shapeblue.com> 发送时间: 2020年5月27日 19:38 收件人: users@cloudstack.apache.org 主题: Re: ACS 4.13.1 failed to create PVLAN network It's off master, I've built it using the PR that I shared, and yes, two bridges via ovs in my setup too! ________________________________ From: li jerry <div...@hotmail.com> Sent: Wednesday, May 27, 2020 3:34 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: 回复: ACS 4.13.1 failed to create PVLAN network Thanks David, is your environment 4.13.1 or 4.15? I can test it according to your version. In 4.13.1 I used openvswitch and created two bridges through ovs-vsctl -Jerry 发件人: David Jumani<mailto:david.jum...@shapeblue.com> 发送时间: 2020年5月27日 16:28 收件人: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 主题: Re: ACS 4.13.1 failed to create PVLAN network Hi Jerry, The way PVLAN works on KVM is that it uses OpenFlow rules to emulate PVLAN, so OVS must be used for Linux networking, not the default Linux bridge. Apart from that, there were certain issues with it which have been addressed in the PR provided by Boris. It's been tested and working on OVS 2.9.2 When creating an L3 PVLAN, the VR is automatically created only when an instance is brought up on that network. The steps I followed are : 1. Create the PVLAN network 2. Create an instance and attach it to the network 3. Wait until the router comes up The OpenFlow scripts run in the background on the agent Once the instance is up, it should work as expected I haven't tried adding a live host to a PVLAN network, but it worked when I tried it following the above steps. You can test the PVLAN connectivity by pinging another host using the interface attached to the PVLAN network ________________________________ From: Boris Stoyanov <boris.stoya...@shapeblue.com> Sent: Wednesday, May 27, 2020 1:45 PM To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Re: ACS 4.13.1 failed to create PVLAN network I can't advise if simply merging this code in 4.13 will work fine, it may cause some side issues since it's a big gap in code bases, therefore I'll advise you to install/upgrade you test env to this PR ( 4.15 ) Bobby. On 27.05.20, 11:07, "li jerry" <div...@hotmail.com> wrote: Thank you Boris for the information. I now go to merge this PR into 4.13.1 for testing. Provide test results later. -Jerry 发件人: Boris Stoyanov<mailto:boris.stoya...@shapeblue.com> 发送时间: 2020年5月27日 16:01 收件人: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> 主题: Re: ACS 4.13.1 failed to create PVLAN network Hi Li, Can you try your tests with this PR: https://github.com/apache/cloudstack/pull/4040 This one ^^ is enabling it on L2 networks, to be honest I'm not sure about L3 as I'm currently testing it. But to be able to test I'll need to facilitate an L3 network (arping requires IP), and then specify the interface to the other L2 network attached to the same VM. Please note that this is 4.15+ and it's currently under testing, so do it in a testing lab. Thanks, On 26.05.20, 18:14, "li jerry" <div...@hotmail.com> wrote: Dear All Who has successfully deployed a PVLAN network on ACS 4.13.1? I created the PVLAN network through the following process, but it failed Server: CentOS7 CloudStack 4.13.1 Zone 1： l Physical Network1 ：Public and Guest , Isolation method=VLAN, KVM traffic label=br1 l Physical Network2 : Management , Isolation method=VLAN, KVM traffic label=br2 SystemVM Template (KVM): Version 4.11.3 Hypervisor: CentOS7.7 3.10.0-1062 Openvswitch-2.12.0 libvirt 4.5.0 QEMU 1.5.3 1. Create L3 Guest Network, name= Pvlan-Net-01, vlan=700, isolatedpvlan=1700 ( this is api command) command=createNetwork&zoneId=409b04ea-d128-48ac-8e33-4df700da89cc&networkOfferingId=9778a4ab-0de8-4440-9879-a488416e0572&physicalnetworkid=ca0768c8-f068-4d88-b7bd-2766414a6415&name=Pvlan-Net-01&displayText=Pvlan-Net-01&vlan=700&bypassVlanOverlapCheck=false&isolatedpvlan=1700&acltype=domain&gateway=172.17.0.1&netmask=255.255.0.0&startip=172.17.1.1&endip=172.17.1.254&networkdomain=hyperx.com&response=json&_=1590497900407 2. attache network to VM vm1 (this VM is running and has an isolated network 10.0.0.x / 24) 3. attache failed, management throws the following error: this is management log 2020-05-26 21:59:40,268 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Seq 4-6374282322589515787: Received: { Ans: , MgmtId: 113349236140, via: 4(2222), Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer, NetworkUsageAnswer, Answer, Answer, Answer, Answer, Answer } } 2020-05-26 21:59:40,280 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) SETUP DHCP PVLAN RULES 2020-05-26 21:59:40,292 DEBUG [c.c.n.r.NetworkHelperImpl] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Router requires upgrade. Unable to send command to router:5, router template version : null, minimal required version : 4.10.0 2020-05-26 21:59:40,294 WARN [o.a.c.n.t.AdvancedNetworkVisitor] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Timed Out com.cloud.exception.ResourceUnavailableException: Resource [VirtualRouter:5] is unreachable: Unable to send command. Router requires upgrade at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:175) at org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:185) at com.cloud.network.rules.DhcpPvlanRules.accept(DhcpPvlanRules.java:61) at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.setupDhcpForPvlan(AdvancedNetworkTopology.java:131) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeStart(VirtualNetworkApplianceManagerImpl.java:2080) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1170) at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:904) at com.cloud.network.router.NetworkHelperImpl.start(NetworkHelperImpl.java:277) at com.cloud.network.router.NetworkHelperImpl.startVirtualRouter(NetworkHelperImpl.java:356) at com.cloud.network.router.NetworkHelperImpl.startRouters(NetworkHelperImpl.java:341) at org.cloud.network.router.deployment.RouterDeploymentDefinition.deployVirtualRouter(RouterDeploymentDefinition.java:205) at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:278) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1380) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1715) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createNicForVm(NetworkOrchestrator.java:3767) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:3504) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:5264) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5326) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-05-26 21:59:40,297 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) The guru did not like the answers so stopping VM[DomainRouter|r-5-VM] 2020-05-26 21:59:40,302 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Seq 4-6374282322589515789: Sending { Cmd , MgmtId: 113349236140, via: 4(2222), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.188.7","forceStop":false,"volumesToDisconnect":[],"vmName":"r-5-VM","executeInSequence":false,"wait":0}}] } Agent error 20-05-26 22:53:44,801 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-2:null) (logid:) Execution is successful. 2020-05-26 22:53:44,802 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-2:null) (logid:) Looking for libvirtd connection at: qemu:///system 2020-05-26 22:53:44,807 DEBUG [cloud.agent.Agent] (UgentTask-2:null) (logid:) Sending ping: Seq 4-4: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"r-11-VM":{"state":"PowerOn","host":"2222"},"r-4-VM":{"state":"PowerOn","host":"2222"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":4,"wait":0}}] } 2020-05-26 22:53:44,903 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Received response: Seq 4-4: { Ans: , MgmtId: 113349236140, via: 4, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":4,"wait":0},"result":true,"wait":0}}] } 2020-05-26 22:53:48,475 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Trying to connect to 169.254.208.61 2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.check.CheckSshCommand 2020-05-26 22:53:48,477 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-3:null) (logid:03678ec5) Ping command port, 169.254.208.61:3922 2020-05-26 22:53:48,477 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Trying to connect to 169.254.208.61 2020-05-26 22:53:48,477 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-3:null) (logid:03678ec5) Ping command port succeeded for vm r-11-VM 2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.GetDomRVersionCmd 2020-05-26 22:53:48,480 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh get_template_version.sh 169.254.208.61 2020-05-26 22:53:48,482 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout : 1800000 2020-05-26 22:53:48,986 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful. 2020-05-26 22:53:48,987 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing script in VR: get_template_version.sh 2020-05-26 22:53:48,988 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.NetworkUsageCommand 2020-05-26 22:53:48,988 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.208.61 -c 2020-05-26 22:53:48,989 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout : 3600000 2020-05-26 22:53:49,571 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful. 2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.routing.AggregationControlCommand 2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.routing.SetMonitorServiceCommand 2020-05-26 22:53:49,573 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.routing.AggregationControlCommand 2020-05-26 22:53:49,573 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Transforming com.cloud.agent.api.routing.SetMonitorServiceCommand to ConfigItems 2020-05-26 22:53:49,602 DEBUG [virtualnetwork.facade.AbstractConfigItemFacade] (agentRequest-Handler-3:null) (logid:03678ec5) Transformed filename: monitor_service.json to: monitor_service.json.e2aeaa96-5a74-4753-8edf-6a040717a8aa 2020-05-26 22:53:49,604 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Aggregate action timeout in seconds is 600 2020-05-26 22:53:49,605 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Creating file in VR, with ip: 169.254.208.61, file: VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg 2020-05-26 22:53:50,236 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 169.254.208.61 -c /var/cache/cloud/VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg 2020-05-26 22:53:50,239 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout : 600600 2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful. 2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing script in VR: vr_cfg.sh Note： 1. I used the same template to create the Isolated network and attache to the VM, everything works fine. 2. When I created the PVLAN VR, I used the virsh console to enter the VM and executed the get_template_version.sh script, which output cloudstack 4.11.3 I do n’t know what causes the PVLAN network to be created! Any help would be greatly appreciated! If you need more detailed information, please let me know Thank you -Jerry boris.stoya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue boris.stoya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue david.jum...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue david.jum...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue david.jum...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
