Re: sr-iov and bridges (mlx4)
On 09/20/2015 05:01 AM, Or Gerlitz wrote: > On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco > <matthew.mon...@colorado.edu> wrote: > [...] >> In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound >> DHCP from the virtio VMs work fine, but inbound frames are not making it >> back to >> the VM. > [...] >> Is there a know limitation of mixing SR-IOV and bridges in general? Does the >> SR-IOV switch specific to the mlx4 hardware not work well with linux >> bridges? ...? > > It would be a bit hard for bridge based promiscuous environment to work OOB > for one of the functions (PF or VF) in SRIOV, this is generic issue, > and not related > to specific vendor. > > You need to use the bridge (8) tool (part of iproute2) and add the P.V > VM MACs to > the PF interface as "self" see some slides (21/22/23) from netdev 0.1 > that deal with that > https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf > > Or. > > It also possible to mark one bridge port (the PF) as non-promiscuous, > but I haven't > played with that myself yet (slide 32), so can't just send you doing it... > Many thanks, that helps. Now to figure out the best approach for shoving this into OpenStack =) signature.asc Description: OpenPGP digital signature
Re: sr-iov and bridges (mlx4)
On 09/20/2015 12:18 PM, Nieścierowicz Adam wrote: > Hi Matthew, > in near future i want to do exactly the same, if you make progress with > SR-IOV+Bridge+OpenStack please leave here some advice. > I wrote a shell script which polls (~15s) each bridge /sys/class/net/brq*, and for each bridge figures out the uplink (not named /sys/class/net/brq*/brif/tap*). Then, for each port /sys/class/net//brif/tap*, determine the VM mac address and add an fdb entry if it doesn't already exist. In my environment, it seems the VM mac is the tap mac s/^fe:/fa:/. This is a little messier than the non-promsicuous bridge option, but I wasn't able to get that working on CentOS7/kernel-ml-4.2/iproute-3.10. But either way, this won't work well if you're trying to do nested virt, which thankfully I don't need at this time. If you're interested I can attach the script and systemd unit. Otherwise, I'll be looking to try to get this handled properly by neutron-linuxbridge-agent. signature.asc Description: OpenPGP digital signature
sr-iov and bridges (mlx4)
Hello. I have a Mellanox ConnectX-3 Pro EN (MCX314A-BCCT). I'm only using a single port so it must provide IP for my host as well as connectivity for VMs. SR-IOV VFs are working great, my KVM VMs have Ethernet and RDMA. However, I also want to support virtio VMs. Assuming eth0 is the first port on my mlx nic, I've tried placing VMs on a bridge with the primary physical interface, and giving an IP for management to a VF: br0 |--- eth0 |--- VM |--- VM vf0 (IP) vf1 -> VM vf2 -> VM vf3 -> VM I've tried placing VMs on a bridge with one of the VFs and using the primary iface for IP. eth0 (IP) br0 |--- vf0 |--- VM |--- VM vf1 -> VM vf2 -> VM vf3 -> VM And I've also tried using a veth pair to really spread things out: br0 (IP) |--- eth0 |--- veth-a br1 | |--- veth-b |--- VM |--- VM vf1 -> VM vf2 -> VM vf3 -> VM In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM. Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...? Thanks! Matt signature.asc Description: OpenPGP digital signature
