Re: sr-iov and bridges (mlx4)
Hi Matthew, in near future i want to do exactly the same, if you make progress with SR-IOV+Bridge+OpenStack please leave here some advice. --- Pozdrawiam Adam Nieścierowicz W dniu 20.09.2015 18:21, Matthew Monaco napisał(a): On 09/20/2015 05:01 AM, Or Gerlitz wrote: On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco wrote: [...] In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM. [...] Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...? It would be a bit hard for bridge based promiscuous environment to work OOB for one of the functions (PF or VF) in SRIOV, this is generic issue, and not related to specific vendor. You need to use the bridge (8) tool (part of iproute2) and add the P.V VM MACs to the PF interface as "self" see some slides (21/22/23) from netdev 0.1 that deal with that https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf [1] Or. It also possible to mark one bridge port (the PF) as non-promiscuous, but I haven't played with that myself yet (slide 32), so can't just send you doing it... Many thanks, that helps. Now to figure out the best approach for shoving this into OpenStack =) Links: -- [1] https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: sr-iov and bridges (mlx4)
On 09/20/2015 12:18 PM, Nieścierowicz Adam wrote: > Hi Matthew, > in near future i want to do exactly the same, if you make progress with > SR-IOV+Bridge+OpenStack please leave here some advice. > I wrote a shell script which polls (~15s) each bridge /sys/class/net/brq*, and for each bridge figures out the uplink (not named /sys/class/net/brq*/brif/tap*). Then, for each port /sys/class/net//brif/tap*, determine the VM mac address and add an fdb entry if it doesn't already exist. In my environment, it seems the VM mac is the tap mac s/^fe:/fa:/. This is a little messier than the non-promsicuous bridge option, but I wasn't able to get that working on CentOS7/kernel-ml-4.2/iproute-3.10. But either way, this won't work well if you're trying to do nested virt, which thankfully I don't need at this time. If you're interested I can attach the script and systemd unit. Otherwise, I'll be looking to try to get this handled properly by neutron-linuxbridge-agent. signature.asc Description: OpenPGP digital signature
Re: sr-iov and bridges (mlx4)
On 09/20/2015 05:01 AM, Or Gerlitz wrote: > On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco > wrote: > [...] >> In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound >> DHCP from the virtio VMs work fine, but inbound frames are not making it >> back to >> the VM. > [...] >> Is there a know limitation of mixing SR-IOV and bridges in general? Does the >> SR-IOV switch specific to the mlx4 hardware not work well with linux >> bridges? ...? > > It would be a bit hard for bridge based promiscuous environment to work OOB > for one of the functions (PF or VF) in SRIOV, this is generic issue, > and not related > to specific vendor. > > You need to use the bridge (8) tool (part of iproute2) and add the P.V > VM MACs to > the PF interface as "self" see some slides (21/22/23) from netdev 0.1 > that deal with that > https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf > > Or. > > It also possible to mark one bridge port (the PF) as non-promiscuous, > but I haven't > played with that myself yet (slide 32), so can't just send you doing it... > Many thanks, that helps. Now to figure out the best approach for shoving this into OpenStack =) signature.asc Description: OpenPGP digital signature
Re: sr-iov and bridges (mlx4)
On Sun, Sep 20, 2015 at 2:58 AM, Matthew Monaco wrote: [...] > In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound > DHCP from the virtio VMs work fine, but inbound frames are not making it back > to > the VM. [...] > Is there a know limitation of mixing SR-IOV and bridges in general? Does the > SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? > ...? It would be a bit hard for bridge based promiscuous environment to work OOB for one of the functions (PF or VF) in SRIOV, this is generic issue, and not related to specific vendor. You need to use the bridge (8) tool (part of iproute2) and add the P.V VM MACs to the PF interface as "self" see some slides (21/22/23) from netdev 0.1 that deal with that https://netdev01.org/docs/netdev_tutorial_bridge_makita_150213.pdf Or. It also possible to mark one bridge port (the PF) as non-promiscuous, but I haven't played with that myself yet (slide 32), so can't just send you doing it... -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
sr-iov and bridges (mlx4)
Hello. I have a Mellanox ConnectX-3 Pro EN (MCX314A-BCCT). I'm only using a single port so it must provide IP for my host as well as connectivity for VMs. SR-IOV VFs are working great, my KVM VMs have Ethernet and RDMA. However, I also want to support virtio VMs. Assuming eth0 is the first port on my mlx nic, I've tried placing VMs on a bridge with the primary physical interface, and giving an IP for management to a VF: br0 |--- eth0 |--- VM |--- VM vf0 (IP) vf1 -> VM vf2 -> VM vf3 -> VM I've tried placing VMs on a bridge with one of the VFs and using the primary iface for IP. eth0 (IP) br0 |--- vf0 |--- VM |--- VM vf1 -> VM vf2 -> VM vf3 -> VM And I've also tried using a veth pair to really spread things out: br0 (IP) |--- eth0 |--- veth-a br1 | |--- veth-b |--- VM |--- VM vf1 -> VM vf2 -> VM vf3 -> VM In all cases, VMs with SR-IOV work fine, IP on the host works fine, outbound DHCP from the virtio VMs work fine, but inbound frames are not making it back to the VM. Is there a know limitation of mixing SR-IOV and bridges in general? Does the SR-IOV switch specific to the mlx4 hardware not work well with linux bridges? ...? Thanks! Matt signature.asc Description: OpenPGP digital signature
