Managed to figure out why this was happening - I had the NUM_VFS in the card firmware configured to 127 (the maximum value) - reducing this to a lower number allowed me to successfully switch the cards into switchdev mode at which point the ip tools all worked again.
The clue that pointed to this appeared when using the proposed 5.3 hwe edge kernel: [ 694.027106] infiniband (null): mlx5_ib_alloc_counters:5452:(pid 47479): couldn't allocate queue counter for port 128, err -12 (128 being the pertinent value) ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to iproute2 in Ubuntu. Matching subscriptions: iproute2 https://bugs.launchpad.net/bugs/1849856 Title: ip commands error with mellanox devices in switchdev mode Status in iproute2 package in Ubuntu: New Status in linux package in Ubuntu: New Bug description: Kernel: using 5.0.0-23-generic from hwe-edge Configuring a mellanox connectx device with configured VF's into switchdev (rather than legacy) mode result in the ip cli tool erroring when trying to query the interface state: $ sudo ip link Error: Buffer too small for object. Dump terminated ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: iproute2 4.15.0-2ubuntu1 ProcVersionSignature: Ubuntu 5.0.0-23.24~18.04.1-generic 5.0.15 Uname: Linux 5.0.0-23-generic x86_64 ApportVersion: 2.20.9-0ubuntu7.7 Architecture: amd64 Date: Fri Oct 25 14:55:36 2019 ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=C.UTF-8 SHELL=/bin/bash SourcePackage: iproute2 UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1849856/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp