Hi Wei, That's correct. The network operator would need to create static routes for the /64s, both for Isolated networks and VPCs. The next-hop for those being the outside interface of the Virtual Router, which will have an IP from the "public" /64.
Cheers Alex -----Original Message----- From: Wei ZHOU <ustcweiz...@gmail.com> Sent: 17 August 2021 10:20 To: dev@cloudstack.apache.org Subject: Re: IPV6 in Isolated/VPC networks Hi Wido, (cc to Rohit and Alex) It is a good suggestion to use FRR for ipv6. The configuration is quite simple and the VMs can get SLAAC, routes, etc. Privacy extension looks not the same as what you mentioned. see https://datatracker.ietf.org/doc/html/rfc4941 You are right. To use static routing, the admins need to configure the routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs and /64 for isolated networks) and their next-hop (which will be configured in VRs) in CloudStack. CloudStack will pick up an IPv6 range and assign it to an isolated network or vpc. @Rohit, correct me if I'm wrong. I have a question, it looks stateless dhcpv6 (SLAAC from router/VR, router/dns etc via RA messages) will be the only option for now (related to your pr https://github.com/apache/cloudstack/pull/3077) . Would it be good to provide stateful dhcpv6 (which can be implemented by dnsmasq) as an option in cloudstack ? The advantages are (1) support other ipv6 cidr sizes than /64. (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can be changed (4) an Ipv6 addresses can be re-used by multiple vms. The problem is, stateful dhcpv6 does not support routers,nameservers, etc. we need to figure it out (probably use radvd/frr and dnsmasq both). -Wei On Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote: > Hi, > > See my inline responses: > > Op 11-08-2021 om 14:26 schreef Rohit Yadav: > > Hi all, > > > > Thanks for your feedback and ideas, I've gone ahead with discussing > > them > with Alex and came up with a PoC/design which can be implemented in > the following phases: > > > > * Phase1: implement ipv6 support in isolated networks and VPC with > static routing > > * Phase2: discuss and implement support for dynamic routing (TBD) > > > > For Phase1 here's the high-level proposal: > > > > * IPv6 address management: > > * At the zone level root-admin specifies a /64 public range that > will be used for VRs, then they can add a /48, or /56 IPv6 range for > guest networks (to be used by isolated networks and VPC tiers) > > * On creation of any IPv6 enabled isolated network or VPC tier, > from the /48 or /56 block a /64 network is allocated/used > > * We assume SLAAC and autoconfiguration, no DHCPv6 in the zone > (discuss: is privacy a concern, can privacy extensions rfc4941 of > slaac be > explored?) > > Privacy Extensions are only a concern for client devices which roam > between different IPv6 networks. > > If you IPv6 address of a client keeps the same suffix (MAC based) and > switches network then only the prefix (/64) will change. > > This way a network like Google, Facebook, etc could track your device > moving from network to network if they only look at the last 64-bits > of the IPv6 address. > > For servers this is not a problem as you already know in which network > they are. > > > * Network offerings: root-admin can create new network offerings > (with VPC too) that specifies a network stack option: > > * ipv4 only (default, for backward compatibility all > networks/offerings post-upgrade migrate to this option) > > * ipv4-and-ipv6 > > * ipv6-only (this can be phase 1.b) > > * A new routing option: static (phase1), dynamic (phase2, with > multiple sub-options such as ospf/bgp etc...) > > This means that the network admin will need to statically route the > IPv6 subnet to the VR's outside IPv6 address, for example, on a JunOS router: > > set routing-options rib inet6.0 static route 2001:db8:500::/48 > next-hop > 2001:db8:100::50 > > I'm assuming that 2001:db8:100::50 is the address of the VR on the > outside (/64) network. In reality this will probably be a longer > address, but this is for just the example. > > > * VR changes: > > * VR gets its guest and public nics set to inet6 auto > > * For each /64 allocated to guest network and VPC tiers, radvd > is configured to do RA > > radvd is fine, but looking at phase 2 with dynamic routing you might > already want to look into FRRouting. FRR can also advertise RAs while > not doing any routing. > > interface ens4 > no ipv6 nd suppress-ra > ipv6 nd prefix 2001:db8:500::/64 > ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53 > > See: http://docs.frrouting.org/en/latest/ipv6.html > > > * Firewall: a new ipv6 zone/chain is created for ipv6 where ipv6 > firewall rules (ACLs, ingress, egress) are implemented; ACLs between > VPC tiers are managed/implemented by ipv6 firewall on VR > > Please take a look at the existing security_group.py script which > implements RFC4890 > > https://datatracker.ietf.org/doc/html/rfc4890 > > ICMPv6 is a vital part of IPv6 and certain packets should always be > allowed. > > > * It is assumed that static routes are created on the core/main > router by the admin or automated using some scripts/tools; for this > CloudStack will announce events with details of /64 networks and VR's > public IPv6 address that can be consumed by a rabbitmq/message bus > client (for example), or a custom cron job or script as part of orchestration. > (this wouldn't be necessary for dynamic routing bgp with phase2)\\ > > You would only need to announce the /48 or /56 allocated to the VR, > that's all. You don't need to inform the upstream router about the /64 > subnets created within that larger subnet. > > > * Guest Networking: With SLAAC, it's easy for CloudStack to > calculate allocate and use a /64 and determine the IPv6 address of VR > nics and guest VM nics > > * A user create an isolated network/VPC with an offering that is > ipv6 enabled > > * A user can manage firewall for the IPv6 address/guest nics; > there'll be no port forward and LB feature though for IPv6 > > * A users can run workloads in the guest VMs that listen on > publically routable ipv6 addresses > > * Usage/billing etc continue to work, no change needed > > > > Network layout: > > > > [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] > > -> > [guest VMs/nics] > > *core/ISP router needs static routes to be added (manually or > automated), assumes a /48 or /56 configured for the zone > > > > Thoughts, feedback? > > Looks doable! > > Side-note: It would be very cool if you could use parts of this > implementation to also route /48, /56, or /60 subnets to individual > VMs in Shared networks. > > Why? This allows for running Docker containers with native IPv6 inside > the VM or running a (Open)VPN server inside a VM which then assigns > public IPv6 addresses to clients connected. > > Instead of routing the subnet to a VR we route the subnet to a single > instance in a shared network. > > If we could then also move these subnets between Instances easily one > can quickly migrate to a different instance while keeping the same > IPv6 subnet. > > Wido > > > > > Proof-of-concept commentary: here's what I did to test the idea: > > > > * Created an isolated network and deployed a VM in my home lab > > The VR running on KVM has following nics > > eth0 - guest network > > eth1 - link local > > eth2 - public network > > > > * I setup a custom openwrt router on a RPi4 to serve as a toy-core > router where I create a wan6 IPv6 tunnel using tunnel broker and I got > a > /48 allocated. My configuration looks like: > > /48 - 2001:470:ed36::/48 (allocated by tunnel broker) > > /64 - 2001:470:36:3e2::/64 (default allocated by) > > > > I create a LAN ipv6 (public network for CloudStack VR): at > > subnet/prefix > 0: > > LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: > > SLAAC+stateless DHCP (no dhcpv6) > > * > > * > > In the isolated VR, I enabled ipv6 as: > > net.ipv6.conf.all.disable_ipv6 = 0 > > net.ipv6.conf.all.forwarding = 1 > > net.ipv6.conf.all.accept_ra = 1 > > net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf = > > 1 > > > > Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the > > nics: > > echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface > > eth2 inet6 auto >> /etc/network/interfaces /etc/init.d/networking > > restart Next, restart ACS isolated network without cleanup to have > > it > reconfigure IPv4 nics, firewall, NAT etc > > > > * > > Next, I created a /64 network for the isolated guest network on eth0 > > of > VR using radvd: > > > > # cat /etc/radvd.conf > > interface eth0 > > { > > AdvSendAdvert on; > > MinRtrAdvInterval 5; > > MaxRtrAdvInterval 15; > > prefix 2001:470:ed36:1::/64 > > { > > AdvOnLink on; > > AdvAutonomous on; > > }; > > }; > > systemctl restart radvd > > All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this > ...:1::/64 network > > * Finally I added a static route in toy core-router for the new /64 > IPv6 range in the isolated network > > 2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev > <local lan nic> > > * > > ... and I enabled firewall rules to allow any traffic to pass for > > the > new /64 network > > > > And voila all done! I create a domain AAAA record that points to my > guest VM IPv6 address a test webserver on > > http://ipv6-isolated-ntwk-demo.yadav.cloud/ > > > > (Note: I'll get rid of the tunnel and request a new /48 block after > > a > few days, sharing this solely for testing purposes) > > > > > > Regards. > > > > >
