Hi Dimitrii,
This request seems like something we have never seen before. This
doesn't seem to be quite a complex thing to do. Right now, this is and
never has been in our plans to implement. This is also something that
has never come up before.
I'm going to mark this as a wishlist, and incomplete, until we
understand what are the needs and the impact on the project. I would
suggest you raise this during the sprint.
** Changed in: maas
Status: New => Incomplete
** Changed in: maas
Importance: Undecided => Wishlist
** Changed in: maas
Milestone: None => next
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1737428
Title:
VRF support to solve routing problems associated with multi-homing
Status in juju:
New
Status in MAAS:
Incomplete
Status in linux package in Ubuntu:
Incomplete
Bug description:
Problem description:
* a host is multi-homed if it has multiple network interfaces with L3
addresses configured (physical or virtual interfaces, natural to
OpenStack regardless of IPv4/IPv6 and IPv6 in general);
* if all hosts that need to participate in L3 communication are
located on the same L2 network there is no need for a routing device
to be present. ARP/NDP and auto-created directly connected routes are
enough;
* multi-homing with hosts located on different L2 networks requires more
intelligent routing:
- "directly connected" routes are no longer enough to talk to all relevant
hosts in the same network space;
- a default gateway in the main routing table may not be the correct
routing device that knows where to forward traffic (management network traffic
goes to a management switch and router, other traffic goes to L3 ToR switch but
may go via different bonds);
- even if a default gateway knows where to forward traffic, it may not be
the intended physical path (storage replication traffic must go through a
specific outgoing interface, not the same interface as storage access traffic
although both interfaces are connected to the same ToR);
- there is no longer a single "default gateway" as applications need either
per-logical-direction routers or to become routers themselves (if destination
== X, forward to next-hop Y). Leaf-spine architecture is a good example of how
multiple L2 networks force you to use spaces that have VLANs in different
switch fabrics => one or more hops between hosts with interfaces associated
with the same network space;
- while network spaces implicitly require L3 reachability between each host
that has a NIC associated with a network space, the current definition does not
mention routing infrastructure required for that. For a single L2 this problem
is hidden by directly connected routes, for multi-L2, no solution is provided
or discussed;
* existing solutions to multi-homing require routing table management
on a given host: complex static routing rules, dynamic routing (e.g.
running an OSPF or BGP daemon on a host);
* using static routes is rigid and requires network planning (i.e.
working with network engineers which may have varying degrees of
experience, doing VLSM planning etc.);
* using dynamic routing requires a broader integration into an
organization's L3 network infrastructure. Routing can be implemented
differently across different organizations and it is a security and
operational burden to integrate with a company's routing
infrastructure.
Summary: a mechanism is needed to associate an interface with a
forwarding table (FIB) which has its own default gateway and make an
application with a listen(2)ing socket(2) return connected sockets
associated with different FIBs. In other words, applications need to
implicitly get source/destination-based routing capabilities without
the need to use static routing schemes or dynamic routing and with
minimum or no modifications to the applications themselves.
Goals:
* avoid turning individual hosts into routers;
* avoid complex static rules;
* better support multi-fabric deployments with minimum effort (Juju, charms,
MAAS, applications, network infrastructure);
* reduce operational complexity (custom L3 infrastructure integration for
each deployment);
* reduce delivery risks (L3 infrastructure, L3 department responsiveness
varies);
* avoid any form of L2 stretching at the infrastructure level - this is
inefficient for various reasons.
NOTE: https://cumulusnetworks.com/blog/vrf-for-linux/ - I recommend to
read this post to understand suggestions below.
How to solve it?
What does it mean for Juju to support VRF devices?
* enslave certain devices on provisioning based on network space information
(physical NICs, VLAN devices, bonds AND bridges created for containers must be
considered) - VRF devices logically enslave devices similar to bridges but work
differently (on L3, not L2);
* the above is per network namespace so it will work equally well in a LXD
container;
Conceptually:
# echo 'net.ipv4.tcp_l3mdev_accept = 1' >> /etc/sysctl.conf
# echo 'net.ipv4.udp_l3mdev_accept = 1' >> /etc/sysctl.conf
# sysctl -p
# # create additional routing tables
# cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
1 mgmt
10 pub
20 storacc
30 storrepl
EOF
# # populate per-routing table default gateways
# ip route add mgmt default via 192.168.0.1
# ip route add pub default via 172.16.0.1
# ip route add storacc default via 10.10.4.1
# ip route add storrepl default via 10.10.5.1
# # add and bring up VRF devices
# ip link add mgmt type vrf table 1 && ip link set dev mgmt up
# ip link add pub type vrf table 10 && ip link set dev pub up
# ip link add storacc type vrf table 20 && ip link set dev storacc up
# ip link add storrepl type vrf table 30 && ip link set dev storrepl up
# # enslave actual devices to VRF devices
# ip link set mgmtbr0 master mgmt
# ip link set pubbr0 master pub
# ip link set storaccbr0 master storacc
# ip link set storreplbr0 master storrepl
# make your services use INADDR_ANY for listening sockets in charms if
not done already (use 0.0.0.0)
charm-related:
* (no-op) services with listening sockets on INADDR_ANY will not need
any modifications either on the charm side or at the application level
- this is the cheapest way to solve multi-homing problems;
* (later) a more advanced functionality for applications that do not
use INADDR_ANY but bind a listening socket to a specific address -
this requires `ip vrf exec` functionality in iproute2 or application
modifications.
Notes:
* Let's follow rule number 6 (https://tools.ietf.org/html/rfc1925) and move
routing problems to L3 departments. Juju deploy "router" is a different
scenario which should reside on a model separate from IAAS;
* We are not turning hosts into routers with this - this is a way to move
routing decisions to the next hop which is available on a directly connected
route. The problem we are solving here is N next hops instead of just one.
Those hops can worry about administrative distance/different routing protocols,
route costs/metrics, routing protocol peer authentication etc.
* Linux kernel functionality was mostly upstreamed in 4.4;
* Linux kernel only while a unit agent can run on Windows too (nothing we can
do here).
Implementation description:
1. Kernel
4.4 (GA xenial)
* CONFIG_NET_VRF=m - present in xenial GA kernels
http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/tree/debian.master/config/config.common.ubuntu?id=2c5158e82d497c5eb90d6e2b8aaf07d36cb175f6#n5172
* CONFIG_NET_L3_MASTER_DEV=y - present in xenial GA kernels
http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/tree/debian.master/config/config.common.ubuntu?id=2c5158e82d497c5eb90d6e2b8aaf07d36cb175f6#n5109
backports needed from 4.5 - required for VRF-unaware applications that
use INADDR_ANY:
6dd9a14e92e54895e143f10fef4d0b9abe109aa9 (tcp_l3mdev_accept)
63a6fff353d01da5a22b72670c434bf12fa0e3b8 (udp_l3mdev_accept)
only `ip vrf exec` related - NOT required for baseline functionality:
* http://man7.org/linux/man-pages/man8/ip-vrf.8.html CGROUPS and
CGROUP_BPF enabled - xenial HWE only (not HWE-edge)
2. User space (iproute2)
iproute2 supports the vrf keyword in a version packaged with Ubuntu
16.04.
More specific functionality like `ip vrf exec <vrf-name>` is available
in later versions:
https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/commit/?id=1949f82cdf62c074562f04acfbce40ada0aac7e0
git tag --contains=1949f82cdf62c074562f04acfbce40ada0aac7e0
v4.10.0
v4.11.0
...
3. MAAS - already hands over per-subnet default gateways
https://github.com/maas/maas/blob/2.3.0/src/maasserver/models/node.py#L3325-L3360
https://github.com/maas/maas/blob/2.3.0/src/maasserver/api/machines.py#L363-L378
4. Juju and/or MAAS:
* create per-network-space routing tables (default gateways must be taken
from subnets in MAAS - subnets related to the same space will have different
default gateways)
* create VRF devices relevant to network spaces;
* enslave interfaces to VRF devices (this includes Linux bridges created by
Juju for containers).
5. Nothing for baseline functionality other than configuring software
to use 0.0.0.0 (INADDR_ANY or "all interfaces") for listening sockets.
(future work) configure software to use `ip vrf exec` even if it
doesn't support VRFs directly when INADDR_ANY is not used.
See https://www.kernel.org/doc/Documentation/networking/vrf.txt, note
that setsockopt requirement is worked around via `ip vrf exec` in
iproute2 (no need to rewrite every application):
"Applications that are to work within a VRF need to bind their socket
to the VRF device:
setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
or to specify the output device using cmsg and IP_PKTINFO.
TCP & UDP services running in the default VRF context (ie., not bound
to any VRF device) can work across ***all VRF domains*** by enabling
the tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:
sysctl -w net.ipv4.tcp_l3mdev_accept=1
sysctl -w net.ipv4.udp_l3mdev_accept=1"
http://man7.org/linux/man-pages/man8/ip-vrf.8.html
"This ip-vrf command is a helper to run a command against a specific VRF with
the VRF association ***inherited parent to child***."
References:
https://en.wikipedia.org/wiki/Multihoming
http://blog.ipspace.net/2016/04/host-to-network-multihoming-kludges.html
http://blog.ipspace.net/2010/09/ribs-and-fibs.html
https://cumulusnetworks.com/blog/vrf-for-linux/ <--- this is a must-
read
https://docs.cumulusnetworks.com/display/DOCS/Virtual+Routing+and+Forwarding+-+VRF
http://netdevconf.org/1.2/session.html?david-ahern-talk
https://www.kernel.org/doc/Documentation/networking/vrf.txt
https://github.com/Mellanox/mlxsw/wiki/Virtual-Routing-and-
Forwarding-%28VRF%29
http://blog.ipspace.net/2016/02/running-bgp-on-servers.html
https://tools.ietf.org/html/rfc7938
http://www.routereflector.com/2016/11/working-with-vrf-on-linux/
(usage example on 16.04)
To manage notifications about this bug go to:
https://bugs.launchpad.net/juju/+bug/1737428/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
