Dear All
Who has successfully deployed a PVLAN network on ACS 4.13.1?
I created the PVLAN network through the following process, but it failed
Server:
CentOS7
CloudStack 4.13.1
Zone 1：
l Physical Network1 ：Public and Guest , Isolation method=VLAN, KVM traffic
label=br1
l Physical Network2 : Management , Isolation method=VLAN, KVM traffic
label=br2
SystemVM Template (KVM): Version 4.11.3
Hypervisor:
CentOS7.7 3.10.0-1062
Openvswitch-2.12.0
libvirt 4.5.0
QEMU 1.5.3
1. Create L3 Guest Network, name= Pvlan-Net-01, vlan=700, isolatedpvlan=1700
( this is api command)
command=createNetwork&zoneId=409b04ea-d128-48ac-8e33-4df700da89cc&networkOfferingId=9778a4ab-0de8-4440-9879-a488416e0572&physicalnetworkid=ca0768c8-f068-4d88-b7bd-2766414a6415&name=Pvlan-Net-01&displayText=Pvlan-Net-01&vlan=700&bypassVlanOverlapCheck=false&isolatedpvlan=1700&acltype=domain&gateway=172.17.0.1&netmask=255.255.0.0&startip=172.17.1.1&endip=172.17.1.254&networkdomain=hyperx.com&response=json&_=1590497900407
2. attache network to VM vm1 (this VM is running and has an isolated
network 10.0.0.x / 24)
3. attache failed, management throws the following error:
this is management log
2020-05-26 21:59:40,268 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05)
Seq 4-6374282322589515787: Received: { Ans: , MgmtId: 113349236140, via:
4(2222), Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer,
GetDomRVersionAnswer, NetworkUsageAnswer, Answer, Answer, Answer, Answer,
Answer } }
2020-05-26 21:59:40,280 DEBUG [o.a.c.n.t.AdvancedNetworkTopology]
(Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05)
SETUP DHCP PVLAN RULES
2020-05-26 21:59:40,292 DEBUG [c.c.n.r.NetworkHelperImpl]
(Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05)
Router requires upgrade. Unable to send command to router:5, router template
version : null, minimal required version : 4.10.0
2020-05-26 21:59:40,294 WARN [o.a.c.n.t.AdvancedNetworkVisitor]
(Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05)
Timed Out
com.cloud.exception.ResourceUnavailableException: Resource [VirtualRouter:5] is
unreachable: Unable to send command. Router requires upgrade
at
com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:175)
at
org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:185)
at com.cloud.network.rules.DhcpPvlanRules.accept(DhcpPvlanRules.java:61)
at
org.apache.cloudstack.network.topology.AdvancedNetworkTopology.setupDhcpForPvlan(AdvancedNetworkTopology.java:131)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeStart(VirtualNetworkApplianceManagerImpl.java:2080)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1170)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:904)
at com.cloud.network.router.NetworkHelperImpl.start(NetworkHelperImpl.java:277)
at
com.cloud.network.router.NetworkHelperImpl.startVirtualRouter(NetworkHelperImpl.java:356)
at
com.cloud.network.router.NetworkHelperImpl.startRouters(NetworkHelperImpl.java:341)
at
org.cloud.network.router.deployment.RouterDeploymentDefinition.deployVirtualRouter(RouterDeploymentDefinition.java:205)
at
com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:278)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1380)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1715)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createNicForVm(NetworkOrchestrator.java:3767)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:3504)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:5264)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5326)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-05-26 21:59:40,297 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05)
The guru did not like the answers so stopping VM[DomainRouter|r-5-VM]
2020-05-26 21:59:40,302 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05)
Seq 4-6374282322589515789: Sending { Cmd , MgmtId: 113349236140, via: 4(2222),
Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.188.7","forceStop":false,"volumesToDisconnect":[],"vmName":"r-5-VM","executeInSequence":false,"wait":0}}]
}
Agent error
20-05-26 22:53:44,801 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Execution is successful.
2020-05-26 22:53:44,802 DEBUG [kvm.resource.LibvirtConnection]
(UgentTask-2:null) (logid:) Looking for libvirtd connection at: qemu:///system
2020-05-26 22:53:44,807 DEBUG [cloud.agent.Agent] (UgentTask-2:null) (logid:)
Sending ping: Seq 4-4: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"r-11-VM":{"state":"PowerOn","host":"2222"},"r-4-VM":{"state":"PowerOn","host":"2222"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":4,"wait":0}}]
}
2020-05-26 22:53:44,903 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
(logid:) Received response: Seq 4-4: { Ans: , MgmtId: 113349236140, via: 4,
Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":4,"wait":0},"result":true,"wait":0}}]
}
2020-05-26 22:53:48,475 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Trying to connect to
169.254.208.61
2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:03678ec5) Processing command: com.cloud.agent.api.check.CheckSshCommand
2020-05-26 22:53:48,477 DEBUG
[resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-3:null) (logid:03678ec5) Ping command port,
169.254.208.61:3922
2020-05-26 22:53:48,477 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Trying to connect to
169.254.208.61
2020-05-26 22:53:48,477 DEBUG
[resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-3:null) (logid:03678ec5) Ping command port succeeded for
vm r-11-VM
2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:03678ec5) Processing command: com.cloud.agent.api.GetDomRVersionCmd
2020-05-26 22:53:48,480 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
get_template_version.sh 169.254.208.61
2020-05-26 22:53:48,482 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout :
1800000
2020-05-26 22:53:48,986 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful.
2020-05-26 22:53:48,987 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing script in VR:
get_template_version.sh
2020-05-26 22:53:48,988 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:03678ec5) Processing command: com.cloud.agent.api.NetworkUsageCommand
2020-05-26 22:53:48,988 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh
169.254.208.61 -c
2020-05-26 22:53:48,989 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout :
3600000
2020-05-26 22:53:49,571 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful.
2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:03678ec5) Processing command:
com.cloud.agent.api.routing.AggregationControlCommand
2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:03678ec5) Processing command:
com.cloud.agent.api.routing.SetMonitorServiceCommand
2020-05-26 22:53:49,573 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:03678ec5) Processing command:
com.cloud.agent.api.routing.AggregationControlCommand
2020-05-26 22:53:49,573 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Transforming
com.cloud.agent.api.routing.SetMonitorServiceCommand to ConfigItems
2020-05-26 22:53:49,602 DEBUG [virtualnetwork.facade.AbstractConfigItemFacade]
(agentRequest-Handler-3:null) (logid:03678ec5) Transformed filename:
monitor_service.json to:
monitor_service.json.e2aeaa96-5a74-4753-8edf-6a040717a8aa
2020-05-26 22:53:49,604 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Aggregate action timeout in
seconds is 600
2020-05-26 22:53:49,605 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Creating file in VR, with ip:
169.254.208.61, file: VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg
2020-05-26 22:53:50,236 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh
169.254.208.61 -c /var/cache/cloud/VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg
2020-05-26 22:53:50,239 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout :
600600
2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful.
2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:03678ec5) Executing script in VR: vr_cfg.sh
Note：
1. I used the same template to create the Isolated network and attache to
the VM, everything works fine.
2. When I created the PVLAN VR, I used the virsh console to enter the VM
and executed the get_template_version.sh script, which output cloudstack 4.11.3
I do n’t know what causes the PVLAN network to be created!
Any help would be greatly appreciated!
If you need more detailed information, please let me know
Thank you
-Jerry
