On Tue, 2015-08-11 at 10:38 +0900, Christian Balzer wrote: > Hello Ben, > > thanks for the quick and detailed reply. > > On Mon, 10 Aug 2015 15:53:57 +0200 Ben Hutchings wrote: > > > Control: severity -1 important > > Control: tag -1 upstream > > > > On Mon, 2015-08-10 at 13:52 +0900, Christian Balzer wrote: > > [...] > > > I'm also not seeing this on several other machines we use for Ceph > > > with the current Jessie kernel, but to be fair they use slightly > > > different (QDR, not FDR) ConnectX-3 HBAs. > > > > If SR-IOV is enabled on the adapter then the ports will always operate > > in Ethernet mode as it's apparently not supported for IB. Perhaps SR > > -IOV enabled on some adapters but not others? > > > I was wondering about that, but wasn't aware of the Ethernet only bit of > SR-IOV. > Anyway, the previous cluster and one blade of this new one have Mellanox > firmware 2.30.8000, which doesn't offer the Flexboot Bios menu and thus > have no SR-IOV configuration option at boot time. > > However the other blade (replacement mobo for a DoA one) in the new server > does have firmware 2.33.5100 and the Flexboot menu and had SR-IOV enabled. > > Alas disabling it (and taking out the fake-install) did result in the same > behavior, mlx4_en was auto-loaded before mlx4_ib. [...] > I added that "options mlx4_core port_type_array=1" (since there is only > one port) to /etc/modprobe.d/local.conf, depmod -a, update-initramfs -u, > but no joy. > The mlx4_en module gets auto-loaded before the IB one as well with this > setting. [...]
There was a deliberate change in mlx4_core in Linux 3.15 to load
mlx4_en first if it finds any Ethernet port. But that is separate from
the decision of what types of port are configured.
What messages do the drivers log? (dmesg | grep mlx4)
What is the output of:
find /sys/bus/pci/drivers/mlx4_core/0*/ -name port_type | xargs grep -H .
Ben.
--
Ben Hutchings
Theory and practice are closer in theory than in practice.
- John Levine, moderator of comp.compilers
signature.asc
Description: This is a digitally signed message part
