I'm having an issue using bonded interfaces on gguest and public
networks. Details:
3 HVs, Xenserver 6.02 all patches applied, open vswitch backend, sixe
nics in each, configured as:
eth0 management
eth1+3 bonded, tagged as guest_pub
eth2, unused(management backup)
eth4+5 bonded, tagged as storage, which is iSCSI based
Each bond slave connects to pair of L3 switches that are properly stacked
If I unbond bond 1+3, vms are fine, guest and public traffic flows fine.
If I stop cloud-management service, open xencenter and bond eth1+3,
then name the resulting network "guest_pub", it now carries the label
for my network tag for guest and public traffic.
When I restart cloud-management, as soon as my system VMs try to start,
they start bouncing wildly in xencenter. The VIF is created, and looks
like "VLAN-SOMEUUID-15, where 15 is the vlan for my public network. The
VIF never gets assigned to the bond. If I look in the cloudstack log, I
see:
(DirectAgent-97:null) PV args are -- quiet
console=hvc0%template=domP%type=consoleproxy%host=172.16.5.2%port=8250%name=v-2-VM%premium=true%zone=1%pod=1%guid=Proxy.2%proxy_vm=2%disable_rp_filter=true%eth2ip=172.16.15.118%eth2mask=255.255.255.0%gateway=172.16.15.1%eth0ip=169.254.2.227%eth0mask=255.255.0.0%eth1ip=172.16.5.34%eth1mask=255.255.255.0%mgmtcidr=172.16.5.0/24%localgw=172.16.5.1%internaldns1=8.8.8.8%internaldns2=8.8.4.4%dns1=8.8.8.8%dns2=8.8.4.4
2012-09-05 12:54:31,762 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) VBD 14e0d8bc-8b5b-2316-cd1d-07c04389e242 created
for Vol[2|ROOT|a77d8900-aa57-4ce9-9c73-6db35d985f48|2147483648]
2012-09-05 12:54:31,824 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Creating VIF for v-2-VM on nic
[Nic:Public-172.16.15.118-vlan://15]
2012-09-05 12:54:31,824 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Looking for network named guest_pub
2012-09-05 12:54:31,829 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Found more than one network with the name guest_pub
2012-09-05 12:54:31,845 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Found a network called guest_pub on
host=172.16.5.5; Network=f008aa20-1ea1-f6e5-c881-08911e3549e5;
pif=64d27657-d3be-df29-d365-0baa2dab5303
2012-09-05 12:54:31,855 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Creating VLAN 15 on host 172.16.5.5 on device eth1
2012-09-05 12:54:31,885 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Catch Exception: class
com.xensource.xenapi.Types$XenAPIException due to
CANNOT_ADD_VLAN_TO_BOND_SLAVEOpaqueRef:70faa00e-7444-ee41-3cf3-7d151b6fd561
CANNOT_ADD_VLAN_TO_BOND_SLAVEOpaqueRef:70faa00e-7444-ee41-3cf3-7d151b6fd561
at com.xensource.xenapi.Types.checkResponse(Types.java:1731)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VLAN.create(VLAN.java:349)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.enableVlanNetwork(CitrixResourceBase.java:3646)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNetwork(CitrixResourceBase.java:635)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.createVif(CitrixResourceBase.java:671)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1104)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:466)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-09-05 12:54:31,886 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Unable to start v-2-VM due to
CANNOT_ADD_VLAN_TO_BOND_SLAVEOpaqueRef:70faa00e-7444-ee41-3cf3-7d151b6fd561
at com.xensource.xenapi.Types.checkResponse(Types.java:1731)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VLAN.create(VLAN.java:349)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.enableVlanNetwork(CitrixResourceBase.java:3646)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNetwork(CitrixResourceBase.java:635)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.createVif(CitrixResourceBase.java:671)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1104)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:466)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-09-05 12:54:31,977 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Unable to clean up VBD due to
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the
handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:211)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VBD.unplug(VBD.java:1058)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:929)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1167)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:466)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-09-05 12:54:31,996 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) Unable to clean up VBD due to
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the
handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:211)
at com.xensource.xenapi.Connection.dispatch(Connection.java:372)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:905)
at com.xensource.xenapi.VBD.unplug(VBD.java:1058)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:929)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1167)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:466)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-09-05 12:54:31,996 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-97:null) The VM is in stopped state, detected problem
during startup : v-2-VM
2012-09-05 12:54:31,996 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-97:null) Seq 4-204669124: Cancelling because one of the
answers is false and it is stop on error.
2012-09-05 12:54:31,996 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-97:null) Seq 4-204669124: Response Received:
2012-09-05 12:54:31,997 DEBUG [agent.transport.Request]
(DirectAgent-97:null) Seq 4-204669124: Processing: { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 110,
[{"StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"PyGrub","type":"ConsoleProxy","cpus":1,"speed":500,"minRam":1073741824,"maxRam":1073741824,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP type=consoleproxy
host=172.16.5.2 port=8250 name=v-2-VM premium=true zone=1 pod=1
guid=Proxy.2 proxy_vm=2 disable_rp_filter=true eth2ip=172.16.15.118
eth2mask=255.255.255.0 gateway=172.16.15.1 eth0ip=169.254.2.227
eth0mask=255.255.0.0 eth1ip=172.16.5.34 eth1mask=255.255.255.0
mgmtcidr=172.16.5.0/24 localgw=172.16.5.1 internaldns1=8.8.8.8
internaldns2=8.8.4.4 dns1=8.8.8.8
dns2=8.8.4.4","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPassword":"9d8805d384dc10ce","params":{},"disks":[{"id":2,"name":"ROOT-2","mountPoint":"/iqn.2012-01.com.nfinausa:san01/1","path":"a77d8900-aa57-4ce9-9c73-6db35d985f48","size":2147483648,"type":"ROOT","storagePoolType":"IscsiLUN","storagePoolUuid":"776299a8-a770-30f7-9f6b-8c422790f5b7","deviceId":0}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"ip":"172.16.15.118","netmask":"255.255.255.0","gateway":"172.16.15.1","mac":"06:18:16:00:00:27","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://15","isolationUri":"vlan://15","isSecurityGroupEnabled":false,"name":"guest_pub"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"ip":"169.254.2.227","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:e3","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":f
alse,"ip"
:"172.16.5.34","netmask":"255.255.255.0","gateway":"172.16.5.1","mac":"06:58:58:00:00:0f","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"management"}]},"result":false,"details":"Unable
to start v-2-VM due to ","wait":0}}] }
2012-09-05 12:54:31,997 WARN [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-97:null) Cleanup failed due to Unable to start v-2-VM due to
2012-09-05 12:54:31,997 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 4-204669124: Received: { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 110, { StartAnswer } }
2012-09-05 12:54:31,997 DEBUG [agent.manager.AgentAttache]
(DirectAgent-97:null) Seq 4-204669124: No more commands found
2012-09-05 12:54:31,997 WARN [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleanup failed due to Unable to start v-2-VM due to
2012-09-05 12:54:32,002 INFO [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Unable to start VM on Host[-4-Routing] due to
Unable to start v-2-VM due to
2012-09-05 12:54:32,005 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleaning up resources for the vm
VM[ConsoleProxy|v-2-VM] in Starting state
2012-09-05 12:54:32,006 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 4-204669125: Sending { Cmd , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"v-2-VM","wait":0}}] }
2012-09-05 12:54:32,006 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 4-204669125: Executing: { Cmd , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"v-2-VM","wait":0}}] }
2012-09-05 12:54:32,006 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-104:null) Seq 4-204669125: Executing request
2012-09-05 12:54:32,097 INFO [xen.resource.CitrixResourceBase]
(DirectAgent-104:null) VM does not exist on
XenServer0c091805-ebac-4279-9899-52018163a557
2012-09-05 12:54:32,097 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-104:null) Seq 4-204669125: Response Received:
2012-09-05 12:54:32,098 DEBUG [agent.transport.Request]
(DirectAgent-104:null) Seq 4-204669125: Processing: { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 110,
[{"StopAnswer":{"vncPort":0,"bytesSent":0,"bytesReceived":0,"result":true,"details":"VM
does not exist","wait":0}}] }
2012-09-05 12:54:32,098 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-104:null) Cleanup succeeded. Details VM does not exist
2012-09-05 12:54:32,098 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 4-204669125: Received: { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 110, { StopAnswer } }
2012-09-05 12:54:32,098 DEBUG [agent.manager.AgentAttache]
(DirectAgent-104:null) Seq 4-204669125: No more commands found
2012-09-05 12:54:32,098 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleanup succeeded. Details VM does not exist
2012-09-05 12:54:32,119 DEBUG [dc.dao.DataCenterIpAddressDaoImpl]
(consoleproxy-1:null) Releasing ip address for
reservationId=636f327b-5183-46bd-adf2-57bf0d5fae80, instance=7
2012-09-05 12:54:32,125 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Successfully cleanued up resources for the vm
VM[ConsoleProxy|v-2-VM] in Starting state
2012-09-05 12:54:32,126 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Root volume is ready, need to place VM in volume's
cluster
2012-09-05 12:54:32,126 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Vol[2|vm=2|ROOT] is READY, changing deployment
plan to use this pool's dcId: 1 , podId: 1 , and clusterId: 1
2012-09-05 12:54:32,127 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) DeploymentPlanner allocation algorithm: random
2012-09-05 12:54:32,127 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Trying to allocate a host and storage pools from
dc:1, pod:1,cluster:1, requested cpu: 500, requested ram: 1073741824
--
Regards,
Nik
Nik Martin
VP Business Development
Nfina Technologies, Inc.
+1.251.243.0043 x1003
Relentless Reliability
