Gerolamo Valcamonica created CLOUDSTACK-7696:
------------------------------------------------
Summary: Console proxy VM won't start. Unable to add port to OVS
bridge
Key: CLOUDSTACK-7696
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7696
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: SystemVM
Affects Versions: 4.3.0
Reporter: Gerolamo Valcamonica
I have a problem with console proxy.
After a long time I restarted it because of upgrading SSL certificates.
Unfortunately console proxy virtual machine won't start anymore.
I can see this in the libvirt logs:
=============================
2014-09-30 21:25:28.903+0000: 13547: error : virCommandWait:2399 : internal
error: Child process (ovs-vsctl --timeout=5 -- --may-exist add-port management
vnet1 -- set Interface vnet1 'external-ids:attached-mac="06:59:5a:00:00:1f"' --
set Interface vnet1
'external-ids:iface-id="3eac9df5-8288-4ca5-b76a-545b1b96e4cf"' -- set Interface
vnet1 'external-ids:vm-id="5c97bbfd-2161-441c-a6a6-68a1ec78402c"' -- set
Interface vnet1 external-ids:iface-status=active) unexpected exit status 1:
ovs-vsctl: no bridge named management
And this in agent log:
=============================
2014-09-30 23:11:16,779 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) LibvirtException
org.libvirt.LibvirtException: Unable to add port vnet1 to OVS bridge
management: Operation not permitted
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1213)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3659)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1307)
at com.cloud.agent.Agent.processRequest(Agent.java:498)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-09-30 23:11:16,780 DEBUG [kvm.storage.KVMStoragePoolManager]
(agentRequest-Handler-5:null) Disconnecting disk
ab234ec9-d8cf-410a-b849-0ec3859c2a06
2014-09-30 23:11:17,769 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
Seq 3-1904869421: { Ans: , MgmtId: 186916991823178, via: 3, Ver: v1, Flags:
10,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":405,"name":"v-405-VM","type":"ConsoleProxy","cpus":1,"minSpeed":454,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" template=domP type=consoleproxy
host=10.0.1.1 port=8250 name=v-405-VM premium=true zone=1 pod=1 guid=Proxy.405
proxy_vm=405 disable_rp_filter=true eth2ip=XXX.YYY.248.144
eth2mask=255.255.255.0 gateway=XXX.YYY.248.254 eth0ip=169.254.1.24
eth0mask=255.255.0.0 eth1ip=10.0.1.33 eth1mask=255.255.255.0
mgmtcidr=10.0.1.0/24 localgw=10.0.1.1 internaldns1=10.0.1.1 dns1=XXX.YYY.248.1
dns2=8.8.8.8","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"c21eae05c175155f","vncAddr":"10.0.1.104","params":{"memoryOvercommitRatio":"1","cpuOvercommitRatio":"1.1"},"uuid":"5c97bbfd-2161-441c-a6a6-68a1ec78402c","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"ab234ec9-d8cf-410a-b849-0ec3859c2a06","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0fe7281f-bfde-3260-964b-45a9d461006a","id":201,"poolType":"RBD","host":"ceph-int.xxxxxxxxxxxxx.com","path":"rbd","port":6789,"url":"RBD://ceph-int.xxxxxxxxxxxxx.com/rbd/?ROLE=Primary&STOREUUID=0fe7281f-bfde-3260-964b-45a9d461006a"}},"name":"ROOT-405","size":2621440000,"path":"ab234ec9-d8cf-410a-b849-0ec3859c2a06","volumeId":480,"vmName":"v-405-VM","accountId":1,"format":"RAW","id":480,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"ab234ec9-d8cf-410a-b849-0ec3859c2a06","type":"ROOT","_details":{"managed":"false","storagePort":"6789","storageHost":"ceph-int.xxxxxxxxxxxxx.com","volumeSize":"2621440000"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"ab6fca92-0aba-49bc-8024-08eba4686d8d","ip":"XXX.YYY.248.144","netmask":"255.255.255.0","gateway":"XXX.YYY.248.254","mac":"06:2e:e4:00:00:20","dns1":"XXX.YYY.248.1","dns2":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"1ec8fc52-a8ac-4c02-b389-7b6575ad4a0e","ip":"169.254.1.24","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:18","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"86779b11-a440-4b2d-ab4b-21128b0acaa0","ip":"10.0.1.33","netmask":"255.255.255.0","gateway":"10.0.1.1","mac":"06:91:ee:00:00:0e","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"result":false,"details":"Unable
to add port vnet1 to OVS bridge management: Operation not
permitted","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous failure","wait":0}}] }
Other data are:
=============================
ovs-vsctl show :
a2dcba6a-3aba-424a-9c8a-5c19f689ba69
Bridge "cloud0"
Port "vnet0"
Interface "vnet0"
Port "cloud0"
Interface "cloud0"
type: internal
Bridge "cloudbr0"
Port "cloudbr0"
Interface "cloudbr0"
type: internal
Port bondBE
Interface "eth1"
Interface "eth2"
Port management
Interface management
type: internal
Port storage
Interface storage
type: internal
Port "vnet1"
Interface "vnet1"
Bridge "cloudbr1"
Port bondFE
Interface "eth3"
Interface "eth0"
Port "vnet2"
Interface "vnet2"
Port "cloudbr1"
Interface "cloudbr1"
type: internal
ovs_version: "2.0.1"
=============================
agent.properties:
#Storage
#Tue Sep 30 23:43:36 CEST 2014
guest.network.device=cloudbr1
workers=5
private.network.device=management
network.bridge.type=openvswitch
port=8250
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
pod=1
storage.network.device=storage
zone=1
guid=5fba1db3-d7b2-379b-816f-0e6d0be99594
public.network.device=cloudbr1
cluster=1
local.storage.uuid=0c3487b4-0a53-4302-af9a-07d640826cc1
domr.scripts.dir=scripts/network/domr/kvm
LibvirtComputingResource.id=3
host=10.0.1.1
=============================
I tried a workaround changing agent.properties with
<< private.network.device=management
>> private.network.device=cloudbr0
With this, console proxy starts on that host, but the host use the storage
network as management network (assign the same ip to storage and management
networks)
In addiction all VMs started on that host use storage network as management
network.
Practically changing that setting in agent.properties cannot be a solution
because of unwantend effects
It looks like console proxy creation process ignores parameters from creation
command and takes network settings from private.network.device
But private.network.device value is not a bridge and so everithing goes wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
