VPC Issue with CS 4.1/KVM/Openvswitch
Hello, I've been testing Cloudstack 4.1 with KVM/Openvswitch networking. When trying to add a VPC, the VR's public interface default route was not assigned correctly, nor the source nat. Cloudstack reports the VPC is created successfully, however the VR is left in an "incomplete" state. Looking through the agent logs I found: 2013-07-19 16:39:20,961 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: com.cloud.agent.api.PlugNicCommand 2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] (agentRequest-Handler-2:null) plugging nic=[Nic:Public-192.168.1.68-vlan://32] 2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] (agentRequest-Handler-2:null) creating a vlan dev and bridge for public traffic per traffic label vswitch0 2013-07-19 16:39:21,116 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: com.cloud.agent.api.routing.IpAssocVpcCommand 2013-07-19 16:39:21,126 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vpc_ipassoc.sh 169.254.2.23 -A -l 192.168.1.68 -c ethnull -g 192.168.1.1 -m 24 -n 192.168.1.0 2013-07-19 16:39:29,107 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms 2013-07-19 16:39:29,233 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful. 2013-07-19 16:39:29,235 DEBUG [cloud.agent.Agent] (UgentTask-5:null) Sending ping: Seq 7-103: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, [{"PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":7,"wait":0}}] } 2013-07-19 16:39:29,243 DEBUG [cloud.agent.Agent] (Agent-Handler-5:null) Received response: Seq 7-103: { Ans: , MgmtId: 112938636298, via: 7, Ver: v1, Flags: 100010, [{"PingAnswer":{"_command":{"hostType":"Routing","hostId":7,"wait":0},"result":true,"wait":0}}] } 2013-07-19 16:39:38,707 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) Execution is successful. 2013-07-19 16:39:38,708 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) Device "ethnull" does not exist. Cannot find device "ethnull" Error: argument "Table_ethnull" is wrong: "table" value is invalid Error: argument "Table_ethnull" is wrong: "table" value is invalid RTNETLINK answers: No such process 2013-07-19 16:39:38,709 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: com.cloud.agent.api.routing.SetSourceNatCommand 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) failing to get physical interface from bridgecloud0, did not find an eth*, bond*, or vlan* in /sys/devices/virtual/net/cloud0/brif 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) failed to get vlan id from bridge cloud0attached to physical interface 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) failing to get physical interface from bridgevswitch0, did not find an eth*, bond*, or vlan* in /sys/devices/virtual/net/vswitch0/brif 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) failed to get vlan id from bridge vswitch0attached to physical interface 2013-07-19 16:39:38,719 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vpc_snat.sh 169.254.2.23 -A -l 192.168.1.68 -c eth2 2013-07-19 16:39:38,947 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) Execution is successful. 2013-07-19 16:39:38,948 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null) iptables: Bad rule (does a matching rule exist in that chain?). iptables: No chain/target/match by that name. Cloudstack is configured with advanced networking. Tags for physical networks - vswitch0 for public & guest traffic - vif9 for storage traffic - vif8 for management traffic Openvswitch configuration: # ovs-vsctl show Bridge "vswitch1" Port "vswitch1" Interface "vswitch1" type: internal Port "eth1" Interface "eth1" Port "vif9" tag: 9 Interface "vif9" type: internal Bridge "vswitch0" Port "vnet1" tag: 32 Interface "vnet1" Port "vswitch0" Interface "vswitch0" type: internal Port "vif8" tag: 8 Interface "vif8" type: internal Port "eth0" Interface "eth0" Bridge "cloud0" Port "vnet0"
Re: VPC Issue with CS 4.1/KVM/Openvswitch
I can only guess this to be a bug. I'd file that and probably raise it to critical since this seems to bet key functionality broken. Have you tried this with Cloudstack 4.2 that's nearing release, and you might be able to get it fixed there. Ahmad On Jul 23, 2013, at 10:07 AM, Dinu Arateanu wrote: > Hello, > > I've been testing Cloudstack 4.1 with KVM/Openvswitch networking. When trying > to add a VPC, the VR's public interface default route was not assigned > correctly, nor the source nat. Cloudstack reports the VPC is created > successfully, however the VR is left in an "incomplete" state. Looking > through the agent logs I found: > > 2013-07-19 16:39:20,961 DEBUG [cloud.agent.Agent] > (agentRequest-Handler-2:null) Processing command: > com.cloud.agent.api.PlugNicCommand > 2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] > (agentRequest-Handler-2:null) plugging nic=[Nic:Public-192.168.1.68-vlan://32] > 2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] > (agentRequest-Handler-2:null) creating a vlan dev and bridge for public > traffic per traffic label vswitch0 > 2013-07-19 16:39:21,116 DEBUG [cloud.agent.Agent] > (agentRequest-Handler-2:null) Processing command: > com.cloud.agent.api.routing.IpAssocVpcCommand > 2013-07-19 16:39:21,126 DEBUG > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-2:null) Executing: > /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh > vpc_ipassoc.sh 169.254.2.23 -A -l 192.168.1.68 -c ethnull -g 192.168.1.1 -m > 24 -n 192.168.1.0 > 2013-07-19 16:39:29,107 DEBUG [kvm.resource.LibvirtComputingResource] > (UgentTask-5:null) Executing: > /usr/share/cloudstack-common/scripts/vm/network/security_group.py > get_rule_logs_for_vms > 2013-07-19 16:39:29,233 DEBUG [kvm.resource.LibvirtComputingResource] > (UgentTask-5:null) Execution is successful. > 2013-07-19 16:39:29,235 DEBUG [cloud.agent.Agent] (UgentTask-5:null) Sending > ping: Seq 7-103: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, > [{"PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":7,"wait":0}}] > } > 2013-07-19 16:39:29,243 DEBUG [cloud.agent.Agent] (Agent-Handler-5:null) > Received response: Seq 7-103: { Ans: , MgmtId: 112938636298, via: 7, Ver: > v1, Flags: 100010, > [{"PingAnswer":{"_command":{"hostType":"Routing","hostId":7,"wait":0},"result":true,"wait":0}}] > } > 2013-07-19 16:39:38,707 DEBUG > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-2:null) Execution is successful. > 2013-07-19 16:39:38,708 DEBUG > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-2:null) Device "ethnull" does not exist. > Cannot find device "ethnull" > Error: argument "Table_ethnull" is wrong: "table" value is invalid > > Error: argument "Table_ethnull" is wrong: "table" value is invalid > > RTNETLINK answers: No such process > > 2013-07-19 16:39:38,709 DEBUG [cloud.agent.Agent] > (agentRequest-Handler-2:null) Processing command: > com.cloud.agent.api.routing.SetSourceNatCommand > 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] > (agentRequest-Handler-2:null) failing to get physical interface from > bridgecloud0, did not find an eth*, bond*, or vlan* in > /sys/devices/virtual/net/cloud0/brif > 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] > (agentRequest-Handler-2:null) failed to get vlan id from bridge > cloud0attached to physical interface > 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] > (agentRequest-Handler-2:null) failing to get physical interface from > bridgevswitch0, did not find an eth*, bond*, or vlan* in > /sys/devices/virtual/net/vswitch0/brif > 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] > (agentRequest-Handler-2:null) failed to get vlan id from bridge > vswitch0attached to physical interface > 2013-07-19 16:39:38,719 DEBUG > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-2:null) Executing: > /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vpc_snat.sh > 169.254.2.23 -A -l 192.168.1.68 -c eth2 > 2013-07-19 16:39:38,947 DEBUG > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-2:null) Execution is successful. > 2013-07-19 16:39:38,948 DEBUG > [resource.virtualnetwork.VirtualRoutingResource] > (agentRequest-Handler-2:null) iptables: Bad rule (does a matching rule exist > in that chain?). > iptables: No chain/target/match by that name. > > > Cloudstack is configured with advanced networking. Tags for physical networks > - vswitch0 for public & guest traffic > - vif9 for storage traffic > - vif8 for management traffic > > Openvswitch configuration: > # ovs-vsctl show > >Bridge "vswitch1" >Port "vswitch1" >Interface "vswitch1" >type: internal >Port "eth1
Re: VPC Issue with CS 4.1/KVM/Openvswitch
Check this thread http://markmail.org/thread/gaxuprs6vtifsybk On 7/23/13 10:37 PM, "Dinu Arateanu" wrote: >Hello, > >I've been testing Cloudstack 4.1 with KVM/Openvswitch networking. When >trying to add a VPC, the VR's public interface default route was not >assigned correctly, nor the source nat. Cloudstack reports the VPC is >created successfully, however the VR is left in an "incomplete" state. >Looking through the agent logs I found: > >2013-07-19 16:39:20,961 DEBUG [cloud.agent.Agent] >(agentRequest-Handler-2:null) Processing command: >com.cloud.agent.api.PlugNicCommand >2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] >(agentRequest-Handler-2:null) plugging >nic=[Nic:Public-192.168.1.68-vlan://32] >2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] >(agentRequest-Handler-2:null) creating a vlan dev and bridge for public >traffic per traffic label vswitch0 >2013-07-19 16:39:21,116 DEBUG [cloud.agent.Agent] >(agentRequest-Handler-2:null) Processing command: >com.cloud.agent.api.routing.IpAssocVpcCommand >2013-07-19 16:39:21,126 DEBUG >[resource.virtualnetwork.VirtualRoutingResource] >(agentRequest-Handler-2:null) Executing: >/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >vpc_ipassoc.sh 169.254.2.23 -A -l 192.168.1.68 -c ethnull -g >192.168.1.1 -m 24 -n 192.168.1.0 >2013-07-19 16:39:29,107 DEBUG [kvm.resource.LibvirtComputingResource] >(UgentTask-5:null) Executing: >/usr/share/cloudstack-common/scripts/vm/network/security_group.py >get_rule_logs_for_vms >2013-07-19 16:39:29,233 DEBUG [kvm.resource.LibvirtComputingResource] >(UgentTask-5:null) Execution is successful. >2013-07-19 16:39:29,235 DEBUG [cloud.agent.Agent] (UgentTask-5:null) >Sending ping: Seq 7-103: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, >[{"PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_g >atewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId >":7,"wait":0}}] } >2013-07-19 16:39:29,243 DEBUG [cloud.agent.Agent] (Agent-Handler-5:null) >Received response: Seq 7-103: { Ans: , MgmtId: 112938636298, via: 7, >Ver: v1, Flags: 100010, >[{"PingAnswer":{"_command":{"hostType":"Routing","hostId":7,"wait":0},"res >ult":true,"wait":0}}] } >2013-07-19 16:39:38,707 DEBUG >[resource.virtualnetwork.VirtualRoutingResource] >(agentRequest-Handler-2:null) Execution is successful. >2013-07-19 16:39:38,708 DEBUG >[resource.virtualnetwork.VirtualRoutingResource] >(agentRequest-Handler-2:null) Device "ethnull" does not exist. >Cannot find device "ethnull" >Error: argument "Table_ethnull" is wrong: "table" value is invalid > >Error: argument "Table_ethnull" is wrong: "table" value is invalid > >RTNETLINK answers: No such process > >2013-07-19 16:39:38,709 DEBUG [cloud.agent.Agent] >(agentRequest-Handler-2:null) Processing command: >com.cloud.agent.api.routing.SetSourceNatCommand >2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >(agentRequest-Handler-2:null) failing to get physical interface from >bridgecloud0, did not find an eth*, bond*, or vlan* in >/sys/devices/virtual/net/cloud0/brif >2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >(agentRequest-Handler-2:null) failed to get vlan id from bridge >cloud0attached to physical interface >2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >(agentRequest-Handler-2:null) failing to get physical interface from >bridgevswitch0, did not find an eth*, bond*, or vlan* in >/sys/devices/virtual/net/vswitch0/brif >2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >(agentRequest-Handler-2:null) failed to get vlan id from bridge >vswitch0attached to physical interface >2013-07-19 16:39:38,719 DEBUG >[resource.virtualnetwork.VirtualRoutingResource] >(agentRequest-Handler-2:null) Executing: >/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >vpc_snat.sh 169.254.2.23 -A -l 192.168.1.68 -c eth2 >2013-07-19 16:39:38,947 DEBUG >[resource.virtualnetwork.VirtualRoutingResource] >(agentRequest-Handler-2:null) Execution is successful. >2013-07-19 16:39:38,948 DEBUG >[resource.virtualnetwork.VirtualRoutingResource] >(agentRequest-Handler-2:null) iptables: Bad rule (does a matching rule >exist in that chain?). >iptables: No chain/target/match by that name. > > >Cloudstack is configured with advanced networking. Tags for physical >networks >- vswitch0 for public & guest traffic >- vif9 for storage traffic >- vif8 for management traffic > >Openvswitch configuration: ># ovs-vsctl show > >Bridge "vswitch1" >Port "vswitch1" >Interface "vswitch1" >type: internal >Port "eth1" >Interface "eth1" >Port "vif9" >tag: 9 >Interface "vif9" >type: internal >Bridge "vswitch0" >Port "vnet1" >tag: 32 >Interface "vnet1" >Port "vswitch0" >Interface "vswitch0" >type: internal >Port
Re: VPC Issue with CS 4.1/KVM/Openvswitch
I did file it - https://issues.apache.org/jira/browse/CLOUDSTACK-3783 I checked the code - the culprit seems to be a function called getVlanIdFromBridge() that doesn't take into account the type of networking that exists on the host. That particular code tries to list /sys/devices/virtual/net//brif/ - which is no longer populated by openvswitch. And I just tested it with 4.2 - it does exactly the same. Dinu On Aug 2, 2013, at 5:15 AM, Ahmad Emneina wrote: > I can only guess this to be a bug. I'd file that and probably raise it to > critical since this seems to bet key functionality broken. Have you tried > this with Cloudstack 4.2 that's nearing release, and you might be able to get > it fixed there. > > Ahmad > > On Jul 23, 2013, at 10:07 AM, Dinu Arateanu wrote: > >> Hello, >> >> I've been testing Cloudstack 4.1 with KVM/Openvswitch networking. When >> trying to add a VPC, the VR's public interface default route was not >> assigned correctly, nor the source nat. Cloudstack reports the VPC is >> created successfully, however the VR is left in an "incomplete" state. >> Looking through the agent logs I found: >> >> 2013-07-19 16:39:20,961 DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-2:null) Processing command: >> com.cloud.agent.api.PlugNicCommand >> 2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] >> (agentRequest-Handler-2:null) plugging >> nic=[Nic:Public-192.168.1.68-vlan://32] >> 2013-07-19 16:39:20,970 DEBUG [kvm.resource.OvsVifDriver] >> (agentRequest-Handler-2:null) creating a vlan dev and bridge for public >> traffic per traffic label vswitch0 >> 2013-07-19 16:39:21,116 DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-2:null) Processing command: >> com.cloud.agent.api.routing.IpAssocVpcCommand >> 2013-07-19 16:39:21,126 DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-2:null) Executing: >> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >> vpc_ipassoc.sh 169.254.2.23 -A -l 192.168.1.68 -c ethnull -g 192.168.1.1 >> -m 24 -n 192.168.1.0 >> 2013-07-19 16:39:29,107 DEBUG [kvm.resource.LibvirtComputingResource] >> (UgentTask-5:null) Executing: >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py >> get_rule_logs_for_vms >> 2013-07-19 16:39:29,233 DEBUG [kvm.resource.LibvirtComputingResource] >> (UgentTask-5:null) Execution is successful. >> 2013-07-19 16:39:29,235 DEBUG [cloud.agent.Agent] (UgentTask-5:null) Sending >> ping: Seq 7-103: { Cmd , MgmtId: -1, via: 7, Ver: v1, Flags: 11, >> [{"PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"newStates":{},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":7,"wait":0}}] >> } >> 2013-07-19 16:39:29,243 DEBUG [cloud.agent.Agent] (Agent-Handler-5:null) >> Received response: Seq 7-103: { Ans: , MgmtId: 112938636298, via: 7, Ver: >> v1, Flags: 100010, >> [{"PingAnswer":{"_command":{"hostType":"Routing","hostId":7,"wait":0},"result":true,"wait":0}}] >> } >> 2013-07-19 16:39:38,707 DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-2:null) Execution is successful. >> 2013-07-19 16:39:38,708 DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-2:null) Device "ethnull" does not exist. >> Cannot find device "ethnull" >> Error: argument "Table_ethnull" is wrong: "table" value is invalid >> >> Error: argument "Table_ethnull" is wrong: "table" value is invalid >> >> RTNETLINK answers: No such process >> >> 2013-07-19 16:39:38,709 DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-2:null) Processing command: >> com.cloud.agent.api.routing.SetSourceNatCommand >> 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >> (agentRequest-Handler-2:null) failing to get physical interface from >> bridgecloud0, did not find an eth*, bond*, or vlan* in >> /sys/devices/virtual/net/cloud0/brif >> 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >> (agentRequest-Handler-2:null) failed to get vlan id from bridge >> cloud0attached to physical interface >> 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >> (agentRequest-Handler-2:null) failing to get physical interface from >> bridgevswitch0, did not find an eth*, bond*, or vlan* in >> /sys/devices/virtual/net/vswitch0/brif >> 2013-07-19 16:39:38,719 DEBUG [kvm.resource.LibvirtComputingResource] >> (agentRequest-Handler-2:null) failed to get vlan id from bridge >> vswitch0attached to physical interface >> 2013-07-19 16:39:38,719 DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-2:null) Executing: >> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >> vpc_snat.sh 169.254.2.23 -A -l 192.168.1.68 -c eth2 >> 2013-07-19 16:39:38,947 DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-2:null) Execution is successful. >> 2013-07-19 16:39:38,948 DEBUG