Hi Roland, This set of fixes is critical for Virtualization environments when the VM para-virtualized PKEY table isn't fully configured at the time the VF is probed, or when the management pkey is provisioned to non-zero index in the VF pkey table.
The first three patches are pretty much few liners (two of them with somehow long change log...). The forth one a bit larger. Would be happy to see them all going to -stable, either by you adding a Cc: sta...@vger.kernel.org when you push them or I can send them to Greg after they spend some time upstream. Changes from V0: - applied feedback from Sean on the IB core patch and added his review-by Or. Erez Shitrit (1): IB/ipoib: Fix pkey-change flow for Virtualization environments Jack Morgenstein (2): IB/core: Create QP1 using the pkey index which contains the default pkey IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz (1): IB/ipoib: Make sure child devices use valid/proper pkeys drivers/infiniband/core/mad.c | 8 +++- drivers/infiniband/hw/mlx4/mad.c | 10 +++- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 76 +++++++++++++++++++++----- drivers/infiniband/ulp/ipoib/ipoib_main.c | 2 +- drivers/infiniband/ulp/ipoib/ipoib_netlink.c | 9 +++ 5 files changed, 88 insertions(+), 17 deletions(-) This is the sequence of events when the IPoIB patch is applied at the guest: <-- the VM pkey table contains 0x0000 in index 0, and hence the mgid has 0x8000 as the pkey [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x0000 [root@xena017-3 ~]# ip addr show ib0 22: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256 link/infiniband 80:00:05:8b:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0 <-- the hypervisor changed pkey value in index 0 of the VM pkey table to contain 0x8001 [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x8001 [root@xena017-3 ~]# dmesg ib0: bringing up interface IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 [...] ib0: Event 12 on device mlx4_0 port 1 ib0: pkey changed from 0x8000 to 0x8001 ib0: downing ib_dev ib0: All sends and receives done. ib0: Created ah ffff88011a11f4e0 IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready ib0: Created ah ffff88011a11f660 ib0: Created ah ffff88011a11f780 <--- mgid changed to use 0x8001 [root@xena017-3 ~]# ip addr show ib0 22: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256 link/infiniband 80:00:05:8b:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:01:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0 ping works etc --> when the mlx4 patch isn't applied on the host we see these errors <mlx4_ib> create_pv_sqp: Couldn't change tunnel qp state to INIT (-22) <mlx4_ib> create_pv_resources: Couldn't create tunnel for QP1 (-22) --> this is the sequence of events when the IPoIB patch is not applied at the guest: <-- the VM pkey table contains 0x0000 in index 0, and hence the mgid has 0x8000 as the pkey [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x0000 [root@xena017-3 infiniband]# modprobe ib_ipoib debug_level=1 [root@xena017-3 infiniband]# ip a s ib0 30: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256 link/infiniband 80:00:05:93:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x8001 <-- ipoib got the event, but nothing changed [root@xena017-3 infiniband]# ip a s ib0 30: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256 link/infiniband 80:00:05:93:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0 [root@xena017-3 infiniband]# dmesg ib%d: max_srq_sge=31 ib%d: max_cm_mtu = 0xfff0, num_frags=16 ib%d: max_srq_sge=31 ib%d: max_cm_mtu = 0xfff0, num_frags=16 ib0: bringing up interface IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: Event 12 on device mlx4_0 port 1 ib0: downing ib_dev ib0: All sends and receives done. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html