On 2/22/2019 7:14 AM, Michael S. Tsirkin wrote:
On Thu, Feb 21, 2019 at 11:55:11PM -0800, si-wei liu wrote:
On 2/21/2019 11:00 PM, Samudrala, Sridhar wrote:
On 2/21/2019 7:33 PM, si-wei liu wrote:
On 2/21/2019 5:39 PM, Michael S. Tsirkin wrote:
On Thu, Feb 21, 2019 at 05:14:44PM -0800, Siwei Liu wrote:
Sorry for replying to this ancient thread. There was some remaining
issue that I don't think the initial net_failover patch got addressed
cleanly, see:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1815268
The renaming of 'eth0' to 'ens4' fails because the udev userspace was
not specifically writtten for such kernel automatic enslavement.
Specifically, if it is a bond or team, the slave would typically get
renamed *before* virtual device gets created, that's what udev can
control (without getting netdev opened early by the other part of
kernel) and other userspace components for e.g. initramfs,
init-scripts can coordinate well in between. The in-kernel
auto-enslavement of net_failover breaks this userspace convention,
which don't provides a solution if user care about consistent naming
on the slave netdevs specifically.
Previously this issue had been specifically called out when IFF_HIDDEN
and the 1-netdev was proposed, but no one gives out a solution to this
problem ever since. Please share your mind how to proceed and solve
this userspace issue if netdev does not welcome a 1-netdev model.
Above says:
there's no motivation in the systemd/udevd community at
this point to refactor the rename logic and make it work well with
3-netdev.
What would the fix be? Skip slave devices?
There's nothing user can get if just skipping slave devices - the
name is still unchanged and unpredictable e.g. eth0, or eth1 the
next reboot, while the rest may conform to the naming scheme (ens3
and such). There's no way one can fix this in userspace alone - when
the failover is created the enslaved netdev was opened by the kernel
earlier than the userspace is made aware of, and there's no
negotiation protocol for kernel to know when userspace has done
initial renaming of the interface. I would expect netdev list should
at least provide the direction in general for how this can be
solved...
I was just wondering what did you mean when you said
"refactor the rename logic and make it work well with 3-netdev" -
was there a proposal udev rejected?
No. I never believed this particular issue can be fixed in userspace
alone. Previously someone had said it could be, but I never see any work
or relevant discussion ever happened in various userspace communities
(for e.g. dracut, initramfs-tools, systemd, udev, and NetworkManager).
IMHO the root of the issue derives from the kernel, it makes more sense
to start from netdev, work out and decide on a solution: see what can be
done in the kernel in order to fix it, then after that engage userspace
community for the feasibility...
Anyway, can we write a time diagram for what happens in which order that
leads to failure? That would help look for triggers that we can tie
into, or add new ones.
See attached diagram.
Is there an issue if slave device names are not predictable? The user/admin
scripts are expected
to only work with the master failover device.
Where does this expectation come from?
Admin users may have ethtool or tc configurations that need to deal with
predictable interface name. Third-party app which was built upon specifying
certain interface name can't be modified to chase dynamic names.
Specifically, we have pre-canned image that uses ethtool to fine tune VF
offload settings post boot for specific workload. Those images won't work
well if the name is constantly changing just after couple rounds of live
migration.
It should be possible to specify the ethtool configuration on the
master and have it automatically propagated to the slave.
BTW this is something we should look at IMHO.
I was elaborating a few examples that the expectation and assumption
that user/admin scripts only deal with master failover device is
incorrect. It had never been taken good care of, although I did try to
emphasize it from the very beginning.
Basically what you said about propagating the ethtool configuration down
to the slave is the key pursuance of 1-netdev model. However, what I am
seeking now is any alternative that can also fix the specific udev
rename problem, before concluding that 1-netdev is the only solution.
Generally a 1-netdev scheme would take time to implement, while I'm
trying to find a way out to fix this particular naming problem under
3-netdev.
Moreover, you were suggesting hiding the lower slave devices anyway. There was
some discussion
about moving them to a hidden network namespace so that they are not visible
from the default namespace.
I looked into this sometime back, but did not find the right kernel api to
create a network namespace within
kernel. If so, we could use this mechanism to simulate a 1-netdev model.
Yes, that's one possible implementation (IMHO the key is to make 1-netdev
model as much transparent to a real NIC as possible, while a hidden netns is
just the vehicle). However, I recall there was resistance around this
discussion that even the concept of hiding itself is a taboo for Linux
netdev. I would like to summon potential alternatives before concluding
1-netdev is the only solution too soon.
Thanks,
-Siwei
Your scripts would not work at all then, right?
At this point we don't claim images with such usage as SR-IOV live
migrate-able. We would flag it as live migrate-able until this ethtool
config issue is fully addressed and a transparent live migration
solution emerges in upstream eventually.
Thanks,
-Siwei
-Siwei
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
net_failover(kernel) | network.service (user)
| systemd-udevd (user)
--------------------------------------------------+------------------------------+--------------------------------------------
(standby virtio-net and net_failover |
|
devices created and initialized, |
|
i.e. virtnet_probe()-> |
|
net_failover_create() |
|
was done.) |
|
|
|
| runs `ifup ens3' ->
|
| ip link set dev ens3 up
|
net_failover_open() |
|
dev_open(virtnet_dev) |
|
virtnet_open(virtnet_dev) |
|
netif_carrier_on(failover_dev) |
|
... |
|
|
|
(VF hot plugged in) |
|
ixgbevf_probe() |
|
register_netdev(ixgbevf_netdev) |
|
netdev_register_kobject(ixgbevf_netdev) |
|
kobject_add(ixgbevf_dev) |
|
device_add(ixgbevf_dev) |
|
kobject_uevent(&ixgbevf_dev->kobj, KOBJ_ADD) |
|
netlink_broadcast() |
|
... |
|
call_netdevice_notifiers(NETDEV_REGISTER) |
|
failover_event(..., NETDEV_REGISTER, ...) |
|
failover_slave_register(ixgbevf_netdev) |
|
net_failover_slave_register(ixgbevf_netdev) |
|
dev_open(ixgbevf_netdev) |
|
|
|
|
|
|
| received ADD uevent from netlink fd
|
| ...
|
| udev-builtin-net_id.c:dev_pci_slot()
|
| (decided to renamed 'eth0' )
|
| ip link set dev eth0 name ens4
(dev_change_name() returns -EBUSY as |
|
ixgbevf_netdev->flags has IFF_UP) |
|
|
|
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
