Heya,
Comments inline.
I ran through the entire procedure I use to test the setup and documented it as
detailed as possible. Hope this helps:
#
# CentOS KVM installation
#
Base installation:
CentOS 6.2 Basic Server
Upgraded to CentOS 6.4:
# yum update
Install Virtualization tools
# yum groupinstall Virtualization "Virtualization Client"
"Virtualization Platform" "Virtualization Tools"
Verify Installed:
hypervkvpd.x86_64 0:0-0.9.el6 libguestfs.x86_64
1:1.16.34-2.el6 libvirt.x86_64 0:0.10.2-18.el6_4.3
libvirt-client.x86_64 0:0.10.2-18.el6_4.3 python-virtinst.noarch
0:0.600.0-15.el6 qemu-kvm.x86_64 2:0.12.1.2-2.355.0.1.el6.centos.2
virt-manager.x86_64 0:0.9.0-18.el6 virt-top.x86_64
0:1.0.4-3.15.el6 virt-viewer.x86_64 0:0.5.2-18.el6_4.2
Modify libvirt configuration:
Edit /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = 16059
auth_tcp = "none"
mdns_adv = 0
Edit /etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--listen"
Start libvirtd
# /etc/init.d/libvirtd start
# /etc/init.d/libvirtd status
Verify installation:
# virsh capabilities
- Should list two guest tags with os_type hvm
Build and install openvswitch:
Install build requirements:
# yum install rpmdevtools openssl-devel kernel-devel gcc redhat-rpm-config
Build packages:
# mkdir -p ~/rpmbuild/SOURCES
# curl -O http://openvswitch.org/releases/openvswitch-1.9.0.tar.gz
# cp openvswitch-1.9.0.tar.gz ~/rpmbuild/SOURCES
# cp centos64-openvswitch.patch ~/rpmbuild/SOURCES
# tar -xzf openvswitch-1.9.0.tar.gz
# cd openvswitch-1.9.0
# patch -p1 < ~/rpmbuild/SOURCES/centos64-openvswitch.patch
# rpmbuild -bb rhel/openvswitch.spec
# rpmbuild -bb -D "kversion `uname -r`"
rhel/openvswitch-kmod-rhel6.spec
Install openvswitch:
# yum install
~/rpmbuild/RPMS/x86_64/kmod-openvswitch-1.9.0-1.el6.x86_64.rpm
~/rpmbuild/RPMS/x86_64/openvswitch-1.9.0-1.x86_64.rpm
# echo 'blacklist bridge' >> /etc/modprobe.d/blacklist.conf
# reboot
Verify installation:
# lsmod |grep openvswitch
# ovs-vsctl -V
Network design:
cloudbr0 (Management, Storage)
ip: 172.16.10.10/24
gateway: 172.16.10.1
eth0 (physical port, no vlans)
cloudbr1 (Guest, Public)
eth1 (physical port, vlan trunk)
ip: none
Configure network interfaces:
/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
IPV6INIT=no
NM_CONTROLLED=no
ONBOOT=yes
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=cloudbr0
/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
IPV6INIT=no
NM_CONTROLLED=no
ONBOOT=yes
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=cloudbr1
/etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=172.16.10.10
GATEWAY=172.16.10.1
NETMASK=255.255.255.0
HOTPLUG=no
/etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
HOTPLUG=no
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=testkvm1
GATEWAY=172.10.10.1
Install cloudstack-agent:
# yum install cloudstack-agent
Edit /etc/cloudstack/agent/agent.properties
network.bridge.type=openvswitch
libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver
Now add the host to cloudstack.
================ centos64-openvswitch.patch =====================
diff -ru
openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/etherdevice.h
openvswitch-1.9.0/datapath/linux/compat/include/linux/etherdevice.h
--- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/etherdevice.h
2013-02-26 21:25:37.000000000 +0100
+++ openvswitch-1.9.0/datapath/linux/compat/include/linux/etherdevice.h
2013-04-25 10:45:09.942027933 +0200
@@ -4,16 +4,4 @@
#include <linux/version.h>
#include_next <linux/etherdevice.h>
-#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,36)
-static inline void eth_hw_addr_random(struct net_device *dev)
-{
- random_ether_addr(dev->dev_addr);
-}
-#elif LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0)
-static inline void eth_hw_addr_random(struct net_device *dev)
-{
- dev_hw_addr_random(dev, dev->dev_addr);
-}
-#endif
-
#endif
diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/if_vlan.h
openvswitch-1.9.0/datapath/linux/compat/include/linux/if_vlan.h
--- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/if_vlan.h
2013-02-26 21:25:37.000000000 +0100
+++ openvswitch-1.9.0/datapath/linux/compat/include/linux/if_vlan.h
2013-04-25 10:44:33.270023182 +0200
@@ -55,38 +55,4 @@
#define VLAN_TAG_PRESENT VLAN_CFI_MASK
#endif
-#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0)
-static inline void vlan_set_encap_proto(struct sk_buff *skb, struct vlan_hdr
*vhdr)
-{
- __be16 proto;
- unsigned char *rawp;
-
- /*
- * Was a VLAN packet, grab the encapsulated protocol, which the layer
- * three protocols care about.
- */
-
- proto = vhdr->h_vlan_encapsulated_proto;
- if (ntohs(proto) >= 1536) {
- skb->protocol = proto;
- return;
- }
-
- rawp = skb->data;
- if (*(unsigned short *) rawp == 0xFFFF)
- /*
- * This is a magic hack to spot IPX packets. Older Novell
- * breaks the protocol design and runs IPX over 802.3 without
- * an 802.2 LLC layer. We look for FFFF which isn't a used
- * 802.2 SSAP/DSAP. This won't work for fault tolerant netware
- * but does for the rest.
- */
- skb->protocol = htons(ETH_P_802_3);
- else
- /*
- * Real 802.2 LLC
- */
- skb->protocol = htons(ETH_P_802_2);
-}
-#endif
#endif /* linux/if_vlan.h wrapper */
diff -ru openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/skbuff.h
openvswitch-1.9.0/datapath/linux/compat/include/linux/skbuff.h
--- openvswitch-1.9.0-clean/datapath/linux/compat/include/linux/skbuff.h
2013-02-26 21:25:37.000000000 +0100
+++ openvswitch-1.9.0/datapath/linux/compat/include/linux/skbuff.h
2013-04-25 10:43:21.167021697 +0200
@@ -245,10 +245,4 @@
}
#endif
-#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,40)
-static inline void skb_reset_mac_len(struct sk_buff *skb)
-{
- skb->mac_len = skb->network_header - skb->mac_header;
-}
-#endif
#endif
diff -ru openvswitch-1.9.0-clean/rhel/openvswitch-kmod-rhel6.spec
openvswitch-1.9.0/rhel/openvswitch-kmod-rhel6.spec
--- openvswitch-1.9.0-clean/rhel/openvswitch-kmod-rhel6.spec 2013-02-26
21:25:52.000000000 +0100
+++ openvswitch-1.9.0/rhel/openvswitch-kmod-rhel6.spec 2013-04-25
10:40:41.039023915 +0200
@@ -22,6 +22,7 @@
Source0: %{oname}-%{version}.tar.gz
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
BuildRequires: %kernel_module_package_buildreqs
+Patch0: centos64-openvswitch.patch
# Without this we get an empty openvswitch-debuginfo package (whose name
# conflicts with the openvswitch-debuginfo package for OVS userspace).
@@ -43,6 +44,7 @@
%prep
%setup -n %{oname}-%{version}
+%patch0 -p1
%build
for flavor in %flavors_to_build; do
================ centos64-openvswitch.patch =====================
> -----Original Message-----
> From: Angeline Shen [mailto:angeline.s...@citrix.com]
> Sent: Thursday, April 25, 2013 5:31 AM
> To: '<dev@cloudstack.apache.org>'; Hugo Trippaers; Sheng Yang; Edison Su
> Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS
> support in KVM
>
> Hugo:
>
> For your configuration example :
>
>
> # The physical interface: eth1
> DEVICE=eth1
> BOOTPROTO=none
> HWADDR=BC:30:5B:D4:16:3C
> IPV6INIT=no
> #MTU=1500
> NM_CONTROLLED=no
> ONBOOT=yes
> TYPE=OVSPort
> DEVICETYPE=ovs
> OVS_BRIDGE=br0
>
>
> # The bridge interface: br0
> DEVICE=br0
> ONBOOT=yes
> DEVICETYPE=ovs
> TYPE=OVSBridge
> BOOTPROTO=static
> IPADDR= 10.223.58.195
> GATEWAY = 10.223.58.193
> NETMASK=255.255.255.192
> HOTPLUG=no
>
> QUESTION: will commands to configure openvswitch network interfaces be
> as follows:
>
> ovs-vsctl add-br br0
> ovs-vsctl add-port br0 eth1
>
The redhat network configuration should take care of creating all the bridges
and linking the ports. But yes the configuration above should have the same
effect as these two commands.
>
> Thanks
>
> -----Original Message-----
> From: Angeline Shen
> Sent: Wednesday, April 24, 2013 1:13 PM
> To: <dev@cloudstack.apache.org>; Hugo Trippaers; Sheng Yang; Edison Su
> Subject: RE: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS
> support in KVM
>
> Hugo:
>
> Please review my updates to
> https://issues.apache.org/jira/browse/CLOUDSTACK-101 .
>
> The latest experiment I conducted per your instructions is as follows:
>
> ovs Host 10.223.58.195 gateway 10.223.58.193 other
> host on
> same gateway but not in ovs setup 10.223.58.194
>
> On ovs host 10.223.58.195 , After I changed /etc/sysconfig/network-
> scripts/ifcfg-eth1 and /etc/sysconfig/network-scripts/ifcfg-cloudbr to your
> instructions, service network restart ,
>
> the ovs host can now ONLY ping its own gateway 10.223.58.193 and any host
> in same subnet, but CANNOT reach anything beyond the gateway
> 10.223.58.193.
>
> 1. /etc/sysconfig/network-scripts/ifcfg-eth1:
>
> DEVICE=eth1
> BOOTPROTO=none
> HWADDR=BC:30:5B:D4:16:3C
> IPV6INIT=no
> #MTU=1500
> NM_CONTROLLED=no
> ONBOOT=yes
> #TYPE=Ethernet
> TYPE=OVSPort
> DEVICETYPE=ovs
> OVS_BRIDGE=cloudbr
> #UUID="0142ba15-de11-499c-a1ba-82837b30f890"
> #IPADDR=10.223.58.195
> #NETMASK=255.255.255.192
> #GATEWAY=10.223.58.193
> #DNS1=10.223.110.254
> #USERCTL=yes
> #IPV4_FAILURE_FATAL=yes
> #DEFROUTE=yes
> #NAME="System eth1"
>
> 2. /etc/sysconfig/network-scripts/ifcfg-cloudbr:
>
> DEVICE=cloudbr
> ONBOOT=yes
> DEVICETYPE=ovs
> TYPE=OVSBridge
> BOOTPROTO=static
> IPADDR=10.223.58.195
> GATEWAY=10.23.58.193
> NETMASK=255.255.255.0
> HOTPLUG=no
>
> 3. On host 10.223.58.195, service network restart
>
>
> 4. on host 10.223.58.195:
>
> [root@Rack3Host16 ~]# ip route
> 10.223.58.0/24 dev cloudbr proto kernel scope link src 10.223.58.195
> 169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
> 169.254.0.0/16 dev eth1 scope link metric 1002
> 169.254.0.0/16 dev cloudbr scope link metric 1015
>
> 5. It can reach its gateway & other hosts on same gateway:
>
> [root@Rack3Host16 ~]# ping 10.223.58.193 PING 10.223.58.193
> (10.223.58.193) 56(84) bytes of data.
> 64 bytes from 10.223.58.193: icmp_seq=1 ttl=64 time=7.43 ms
> 64 bytes from 10.223.58.193: icmp_seq=2 ttl=64 time=2.82 ms
> 64 bytes from 10.223.58.193: icmp_seq=3 ttl=64 time=1.06 ms ^C
> --- 10.223.58.193 ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2364ms rtt
> min/avg/max/mdev = 1.061/3.771/7.430/2.685 ms
> [root@Rack3Host16 ~]# ping 10.223.58.194 PING 10.223.58.194
> (10.223.58.194) 56(84) bytes of data.
> 64 bytes from 10.223.58.194: icmp_seq=1 ttl=64 time=0.399 ms
> 64 bytes from 10.223.58.194: icmp_seq=2 ttl=64 time=0.181 ms ^C
> --- 10.223.58.194 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1446ms rtt
> min/avg/max/mdev = 0.181/0.290/0.399/0.109 ms
>
> [root@Rack3Host16 ~]# traceroute 10.223.58.194 traceroute to 10.223.58.194
> (10.223.58.194), 30 hops max, 60 byte packets
> 1 10.223.58.194 (10.223.58.194) 0.922 ms 0.851 ms 0.830 ms
>
> [root@Rack3Host16 ~]# traceroute 10.223.58.193 traceroute to 10.223.58.193
> (10.223.58.193), 30 hops max, 60 byte packets
> 1 10.223.58.193 (10.223.58.193) 7.017 ms 12.506 ms 7.675 ms
>
> 6. host CANNOT reach anything outside of its subnet:
>
> [root@Rack3Host16 ~]# ping 10.223.195.114
> connect: Network is unreachable
>
> [root@Rack3Host16 ~]# ping www.google.com
> ping: unknown host www.google.com
>
> [root@Rack3Host16 ~]# ping 8.8.8.8
> connect: Network is unreachable
>
> [root@Rack3Host16 ~]# traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 30
> hops max, 60 byte packets
> connect: Network is unreachable
>
> 7. [root@Rack3Host16 ~]# ifconfig
> cloudbr Link encap:Ethernet HWaddr BC:30:5B:D4:16:3C
> inet addr:10.223.58.195 Bcast:10.223.58.255 Mask:255.255.255.0
> inet6 addr: fc00:2::be30:5bff:fed4:163c/64 Scope:Global
> inet6 addr: fe80::68f7:52ff:fe03:a744/64 Scope:Link
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
> RX packets:46945 errors:0 dropped:0 overruns:0 frame:0
> TX packets:549 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:39235509 (37.4 MiB) TX bytes:102773 (100.3 KiB)
>
> cloud0 Link encap:Ethernet HWaddr AA:81:04:5F:5E:4A
> inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
> inet6 addr: fe80::a881:4ff:fe5f:5e4a/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:0 (0.0 b) TX bytes:4201 (4.1 KiB)
>
> eth1 Link encap:Ethernet HWaddr BC:30:5B:D4:16:3C
> inet6 addr: fe80::be30:5bff:fed4:163c/64 Scope:Link
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
> RX packets:48123 errors:0 dropped:0 overruns:0 frame:0
> TX packets:532 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:39518352 (37.6 MiB) TX bytes:101960 (99.5 KiB)
> Interrupt:16 Memory:da000000-da012800
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:340 errors:0 dropped:0 overruns:0 frame:0
> TX packets:340 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:46264 (45.1 KiB) TX bytes:46264 (45.1 KiB)
>
> 8. [root@Rack3Host16 ~]# brctl show
> bridge name bridge id STP enabled interfaces
> cloud0 /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory
> /sys/class/net/cloud0/bridge: No such file or directory 0000.aa81045f5e4a no
> cloudbr /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory
> /sys/class/net/cloudbr/bridge: No such file or directory 0000.bc305bd4163c
> no
>
>
> -----Original Message-----
> From: Sheng Yang [mailto:sh...@yasker.org]
> Sent: Wednesday, April 24, 2013 12:59 PM
> To: Hugo Trippaers
> Cc: <dev@cloudstack.apache.org>
> Subject: Re: https://issues.apache.org/jira/browse/CLOUDSTACK-101 OVS
> support in KVM
>
> Thank you Hugo!
>
> --Sheng
>
>
> On Wed, Apr 24, 2013 at 12:16 PM, Hugo Trippaers <
> htrippa...@schubergphilis.com> wrote:
>
> > Hey Sheng,
> >
> > Sure thing, I'm trying to help Angeline already, but it's quite a
> > complex setup.
> >
> > I'll try to write down some detailed instructions.
> >
> > Cheers,
> >
> > Hugo
> >
> > Sent from my iPhone
> >
> > On 24 apr. 2013, at 19:56, "Sheng Yang" <sh...@yasker.org> wrote:
> >
> > Hi Hugo,
> >
> > Could you help with KVM OVS setup? I cannot find much information on
> > the our wiki about OVS on KVM.
> >
> > --Sheng
> >
> > On Fri, Apr 19, 2013 at 8:14 PM, Angeline Shen
> <angeline.s...@citrix.com>wrote:
> >
> >> ASF 4.1 cloudstack with openvswitch support for RHEL 6.3 KVM test
> >> configuration setup:
> >>
> >> After completing
> >> http://nullworks.wordpress.com/2012/09/19/kvm-and-openvswitch-on-
> cent
> >> os-6-3-minimal/ to setup RHEL 6.3 openvswitch, encountered following
> >> problems:
> >>
> >> 1. In this scenario, after service openvswitch start, host is still
> >> up running but completely lost connection to network.
> >>
> >> [root@Rack3Host16 /]# service openvswitch start Inserting brcompat
> >> module [ OK ] Starting ovsdb-server [ OK ] Configuring Open vSwitch
> >> system IDs [ OK ] Starting ovs-vswitchd [ OK ] Starting ovs-brcompatd
> >> [ OK ] iptables already has a rule for gre, not explicitly enabling.
> >> [root@Rack3Host16 /]#
> >>
> >>
> >> 2. content of various files:
> >>
> >> /etc/sysconfig/network-scripts/ifcfg-ovsbr1 :
> >> DEVICE=ovsbr1
> >> ONBOOT=yes
> >> DEVICETYPE=ovs
> >> TYPE=OVSBridge
> >> BOOTPROTO=static
> >> IPADDR=10.223.58.195
> >> GATEWAY=10.223.58.193
> >> NETMASK=255.255.255.0
> >> HOTPLUG=no
> >>
> >>
> >> QUESTION: Is
> >> http://nullworks.wordpress.com/2012/09/19/kvm-and-openvswitch-on-
> centos-6-3-minimal/
> >> Valid configuration instructions for openvswitch RHEL 6.3
> >> or are there other updated correct configuration steps?
> >>
> >> Thanks
> >>
> >>
> >
