Good point. These are the kinds of things we should be aware of right now. Even though we don't necessarily want to bite off too much right now in initial implementation, we want to plan so that certain things we may want to do aren't impossible.
I was thinking either the cloud provider would have a PI, or the end user may bring their own to the cloud provider. Either way, arrangements need to be made to get it routed up to the point where the VPC can service it, and entered as the 'global cidr' for that VPC (or at least a small portion of the PI). If we don't, then we have to NAT, which as you mentioned isn't really widely supported. I suppose nothing would preclude us from allowing a ULA space to be entered as the VPC global space later on if we wanted to support NAT for IPv6 on the router at some point. You could do either and we'd just route public space and NAT/port forward private space. So, to recap a bit: VPC is assigned a prefix (say /60). Admin assigns sub-prefix from that, per network tier, down to a /64. Route information is published both to a plugin framework and the event bus to enable automation of upstream route programming (point the /60 to the 'public traffic' ip of the VPC router), or admin can manually set up the upstream route per VPC. DNS can be provided by the (currently required) IPv4 recursor on the VPC router, but ultimately dnsmasq on IPv6 address can also provide it, looking forward to IPv6-only networks. 'public traffic' addressing for VPC routers (the internet-facing interface) will be handled by static assignment from a public range, much how the IPv4 public traffic is now. This allows for assigning extra LB addresses or static NAT addresses in the future. guest networks, my initial preference would be for SLAAC, but I think ultimately we'd want to be able to assign multiple ips to a guest. With the 64 bits of the SLAAC space dedicated to all of the unique MAC address possibilities, we can't really do that. We may want to consider DHCP with static assignments from the beginning. On Thu, Jan 16, 2014 at 11:32 PM, Chiradeep Vittal <chiradeep.vit...@citrix.com> wrote: > Are we assuming that the Cloud Provider has a Provider-Independent (PI) > space? > Using a Unique Local Address (ULA) space for the VPC *might* to desirable. > If VPC tiers (subnets) can span zones (not possible today), then NAT66 can > help keep addresses stable as workloads move to a different zone for > availability reasons. > But I see that only Ubuntu has support for NAT66. > > > On 1/16/14 11:33 AM, "Sheng Yang" <sh...@yasker.org> wrote: > >>On Thu, Jan 9, 2014 at 7:04 AM, Daan Hoogland >><dhoogl...@schubergphilis.com>wrote: >> >>> H Marcus, We had a small session on how we plan to go about ipv6 >>> configuration with your bullet list present. Comments are still >>> brainstorming phase quality but hopefully they will lead to something. >>> >>> > * VPC has no global IPv6 prefix (super CIDR as current private space), >>> it's simply IPv6 enabled or not. Admins can choose to route a /60 or a >>>/48 >>> to a vpc and carve it up among the networks there, but it's not >>>required or >>> enforced by cloudstack. >>> The idea at Schuberg Philis is that VPCs should have a standard size >>> network assigned to be taken from a pool. We should be able to configure >>> the width of the network. The idea is that per level (Physical >>> network/vpc/tier) only one width will be used but the size of it is >>> configurable. Default block for a VR / VPC would be a /56 and each >>> individual network would use a /64. Note that in a typical isolated >>>network >>> we only use the first. >>> >>> > * VPC networks get assigned one or multiple IPv6 prefixes, each prefix >>> can be marked 'SLAAC', 'DHCP', or 'Manual'. >>> For networks we are in favor of using SLAAC for configuring routing and >>> addressing, but would advise DHCPv6 to configure DNS and potentially >>>other >>> things (NTP?). No need to register the addresses in CloudStack. We >>>already >>> store the MAC of the NIC so we know which address will be configured on >>>the >>> interface with the configured prefix. Only the router will receive a >>> specific address (prefix::1/64) >>> >>> As mentioned earlier first implementation will do SLAAC. How will SLAAC >>>be >>> configured with ACL's? >>> >>> > * Mgmt server allows calling external plugins for pushing routes to >>> routers upstream of VPCs (SDN or router API), but could be manually >>>done by >>> admins as well >>> We discussed the routing issues in front of the VPC / VR. The goal is to >>> provide cloudstack with a way to deal with routable addresses inside an >>> isolated network, as actually this problem is relevant for both IPv4 and >>> IPv6. However for IPv4 we have a perfectly workable solution with NAT >>>and >>> in the IPv6 world we don't/. So we need to address this issue on how to >>> route a network from the provider edge into an isolated network. >>> >>> For IPv6 we have several options: >>> * Dynamic routing protocols, here we can think of iBGP, OSPF or >>>any >>> other protocol. Basically cloudstack assigns a block to a VR / VPC and >>>the >>> VPC / VR tells the outside world to route that block to the outside IP >>>of >>> the router. >>> * Block allocator, cloudstack knows about the supernet (/48 for >>> example) and assigns a block (/56) to a VR or VPC. Cloudstack tells the >>> block allocator to allocate (route) this block to the VR/VPC. Block >>> allocater needs to know about the device being managed (upstream router >>> like cisco/juniper) and configured the route using telnet or an API >>> * Broker, We introduce a new systemvm, the block broker. The admin >>> routes the /48 to the block broker and CloudStack tells the blok broker >>>to >>> reroute a specific subblock to this VR / VPC >>> * Static, Admin configures a list of block/ip pairs in the >>>upstream >>> router. CloudStack knows about these pairs and assigns the ip to the VR >>>/ >>> VPC and the block will be routed. >>> >>> >>> > * Work could be done in stages, e.g. SLAAC/manual network ranges would >>> be fairly straightforward, whereas DHCP ranges would require programming >>> scripts and ip allocation code. >>> >>> > * An issue was raised about privacy concerns with SLAAC using MAC, but >>> we think this revolves more around clients than servers; that is, a >>>client >>> moving around the country would be traceable because of the MAC, but a >>> server always has the same address anyway. >>> >>> > * Still need to figure out what to do about ACLs, that's a whole >>> separate issue, but a plan needs to be in place. >>> Are we going to put ACL's on networks or hosts or both? What is easiest >>>to >>> implement and enforce? >>> >> >>The routing is controlled by upstream router, so it's straightforward that >>ACL would be done by upstream router. >> >>But after rethink, modifying the upstream router automatically make me >>feel >>unsafe... Probably host side is a better choice, though it would be more >>work to do. >> >> >>> Do we care about portforwarding, static NAT, Load Balancing etc as well? >>> Or at least think >>> about what impact these decisions have. >>> This is not a version 1 consideration?!? >>> >>> > * We assume there will be an ipv6 public range assignable, allocated >>>for >>> VPC routers/static nat/loadbalancers to pull from. How will this range >>>be >>> made available? >>> Is a cidr configured? >>> Will we preconfigure all ranges or have a dynamic availability check? >>> >>> Radvd package is added to the systemvm template. (Done by Hugo this >>> morning) >>> >>> Some standard UI components are needed. >>> >>> (test-)attention needs to go to: >>> >>> HaProxy version/feature matrix has to be reviewed to check for ipv6 >>> compatability. >> >> >>> Does dnsmasq work with ipv6? >>> >> >>Yes. Already use it for IPv6 shared network implementation. >> >> >>> >>> Does keepalived work seemslessly? >>> >> >>According to http://www.keepalived.org/, it should work. But seems works >>are still going on. >> >>--Sheng >> >> >>> Ipv6 tables configuration needs to be pushed to the VRs >>> >>> -----Original Message----- >>> From: Marcus Sorensen [mailto:shadow...@gmail.com] >>> Sent: maandag 6 januari 2014 9:11 >>> To: cloudstack-...@incubator.apache.org >>> Subject: IPv6 in VPC (was Re: IPv6 plan - questions) >>> >>> I've discussed this a bit with various subject matter experts at our >>> datacenters/business, and so far we're leaning toward a rollout like >>> this: >>> >>> * VPC has no global IPv6 prefix (super CIDR as current private space), >>> it's simply IPv6 enabled or not. Admins can choose to route a /60 or a >>> /48 to a vpc and carve it up among the networks there, but it's not >>> required or enforced by cloudstack. >>> >>> * VPC networks get assigned one or multiple IPv6 prefixes, each prefix >>>can >>> be marked 'SLAAC', 'DHCP', or 'Manual'. >>> >>> * Mgmt server allows calling external plugins for pushing routes to >>> routers upstream of VPCs (SDN or router API), but could be manually >>>done by >>> admins as well >>> >>> * Work could be done in stages, e.g. SLAAC/manual network ranges would >>>be >>> fairly straightforward, whereas DHCP ranges would require programming >>> scripts and ip allocation code. >>> >>> * An issue was raised about privacy concerns with SLAAC using MAC, but >>>we >>> think this revolves more around clients than servers; that is, a client >>> moving around the country would be traceable because of the MAC, but a >>> server always has the same address anyway. >>> >>> * Still need to figure out what to do about ACLs, that's a whole >>>separate >>> issue, but a plan needs to be in place. Do we care about port >>>forwarding, >>> static NAT, Load Balancing etc as well? Or at least think about what >>>impact >>> these decisions have. >>> >>> * We assume there will be an ipv6 public range assignable, allocated for >>> VPC routers/static nat/loadbalancers to pull from. >>> >>> On Sat, Jan 4, 2014 at 6:11 AM, Marcus Sorensen <shadow...@gmail.com> >>> wrote: >>> > I've put together a rough draft spec: >>> > >>> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+in+VPC+Rou >>> > ter >>> > >>> > I basically just laid out some rough ideas. I know there has been a >>> > lot of discussion in the past about DHCPv6, etc. My hope is that we >>> > can at least decide on a spec, for future reference. >>> > >>> > >>> > On Fri, Jan 3, 2014 at 9:53 PM, Marcus Sorensen <shadow...@gmail.com> >>> wrote: >>> >> It's been a long time since I've heard anything in regards to IPv6, >>> >> let alone VPC support. Does anyone have plans for this at all? We'd >>> >> like to support IPv6, and we have enough CS knowledge and external >>> >> tools to hack something together, but I'd much prefer to build with >>> >> the community and/or be forward compatible with what it deploys. >>> >> >>> >> I'd like to start with something simple, like perhaps optionally >>> >> providing a /64 or larger as a parameter when creating a VPC (or a >>> >> separate call to add an IPV6 block), and network on the vpc. Then it >>> >> sounds like there's already a mechanism in place for tracking ipv6 >>> >> assignments to nics, that could be leveraged to pass dhcp assignments >>> >> to routers. >>> >> >>> >> Then there's the whole acl thing, that seems like at least as big of >>> >> a project as mentioned previously. >>> >> >>> >> On Mon, Aug 12, 2013 at 3:47 PM, Marcus Sorensen >>><shadow...@gmail.com> >>> wrote: >>> >>> has there been any further discussion that I might have missed >>> >>> around >>> >>> ipv6 in VPC? >>> >>> >>> >>> On Thu, Mar 7, 2013 at 12:09 PM, Sheng Yang <sh...@yasker.org> >>>wrote: >>> >>>> Hi Dave, >>> >>>> >>> >>>> I am glad it fits your need. That's our target. :) >>> >>>> >>> >>>> --Sheng >>> >>>> >>> >>>> On Thu, Mar 7, 2013 at 2:14 AM, Dave Cahill <dcah...@midokura.com> >>> wrote: >>> >>>>> Hi Sheng, >>> >>>>> >>> >>>>> Thanks for the quick reply, that helps a lot. >>> >>>>> >>> >>>>> My main purpose was to figure out how these changes affect virtual >>> >>>>> networking and pluggability. Having read through the IPv6 code >>> >>>>> today, it looks like it will work very nicely with virtual >>>networks. >>> >>>>> >>> >>>>> For example, when VMs are assigned an IPv6 address, the IPv6 >>> >>>>> address is stored in the NicProfile object. So, taking DHCP as an >>> >>>>> example, if the MidoNet plugin implements the DHCPServiceProvider >>> >>>>> interface, it will receive the NicProfile as one of the parameters >>> of addDhcpEntry. >>> >>>>> If we want to implement IPv6, we can then take the IPv6 address >>> >>>>> from the NicProfile, and just use it as needed. >>> >>>>> >>> >>>>> Thanks again for taking the time to respond, and for the detailed >>>FS. >>> >>>>> >>> >>>>> Dave. >>> >>>>> >>> >>>>> On Thu, Mar 7, 2013 at 4:57 AM, Sheng Yang <sh...@yasker.org> >>>wrote: >>> >>>>> >>> >>>>>> On Wed, Mar 6, 2013 at 1:36 AM, Dave Cahill >>><dcah...@midokura.com> >>> wrote: >>> >>>>>> > Hi, >>> >>>>>> >>> >>>>>> Hi Dave, >>> >>>>>> > >>> >>>>>> > I've been catching up on IPv6 plans by reading the functional >>> >>>>>> > specs and Jira tickets - it's great to have so much material to >>> refer to. >>> >>>>>> > >>> >>>>>> > I still have a few questions though, and I'm hoping someone >>> >>>>>> > involved with the feature can enlighten me. >>> >>>>>> > >>> >>>>>> > *[Support for Providers other than Virtual Router]* In [3], the >>> >>>>>> > spec says "No external device support in plan." >>> >>>>>> > What does this mean exactly? >>> >>>>>> >>> >>>>>> Because CloudStack also supports using external devices as >>> >>>>>> network controller e.g. Juniper SRX as firewall and NetScaler as >>> >>>>>> load balancer. The words here said is just we don't support these >>> >>>>>> devices when using IPv6. >>> >>>>>> > >>> >>>>>> > For example, if using Providers other than the Virtual Router, >>> >>>>>> > does the UI still allow setting IPv6 addresses? >>> >>>>>> > >>> >>>>>> > If so, do we attempt to pass IPv6 addresses to the Providers no >>> >>>>>> > matter what, or do we check whether the Provider has IPv6 >>>support? >>> >>>>>> >>> >>>>>> Yes, we checked it when you try to create a IPv6 >>> >>>>>> network(currently only support advance shared network). >>> >>>>>> >>> >>>>>> > >>> >>>>>> > *[Networking Modes]* >>> >>>>>> > Advanced Shared mode and Basic mode are mentioned in the Jira >>> >>>>>> > ticket [1] - "Isolated Network" is mentioned briefly in [2], >>> >>>>>> > but I wanted to check if the Advanced Isolated and VPC modes >>> >>>>>> > are on the roadmap? >>> >>>>>> >>> >>>>>> There is no "basic isolated" network, so "Isolated" network is >>> >>>>>> what we're talking about. We haven't got plan for VPC yet. >>> >>>>>> >>> >>>>>> And one correction: we didn't support "basic" mode for phase 1. >>> >>>>>> We support only "advance shared network" in phase 1. The >>> >>>>>> supported cases are described in FS. Jira ticket only provided a >>> >>>>>> rough idea at the time. >>> >>>>>> > >>> >>>>>> > *[IP Address Management / IPAM]* From [1], re: handing out IPv6 >>> >>>>>> > addresses: "One way could be that the network admin creates a >>> >>>>>> > static route for a /48 towards a Virtual Router and then the VR >>> >>>>>> > can hand out /64s to Instances." >>> >>>>>> > >>> >>>>>> > With IPv4, IPAM is handled by the CloudStack management server, >>> >>>>>> > and the VR is told which IP address to give to the VM over >>> >>>>>> > DHCP. Would this change with IPv6? "The VR can hand out /64s to >>> >>>>>> > instances" sounds like the VR is handling IPAM to some extent. >>> >>>>>> >>> >>>>>> Well, it's not how it works now. Please refer to the FS. The >>> >>>>>> current implementation works like before. VR get a /64 then >>> >>>>>> handle out IPv6 addresses to VM. >>> >>>>>> > >>> >>>>>> > From [3], "Router advertisement should be sent by public >>> >>>>>> > gateway in the network." - to double-check, does this mean the >>> >>>>>> > router outside the CloudStack network should send RAs, but the >>>VR >>> won't send RAs? >>> >>>>>> >>> >>>>>> Yes. Because in phase 1, we support only "advance shared >>> >>>>>> network", in which case, VR is NOT the gateway. So we assume the >>> >>>>>> gateway router outside CloudStack should send out RA to the VMs. >>> >>>>>> >>> >>>>>> But in the phase 2, VR would acting as gateway, then it would >>>send >>> out RAs. >>> >>>>>> >>> >>>>>> --Sheng >>> >>>>>> > >>> >>>>>> > Thanks, >>> >>>>>> > Dave. >>> >>>>>> > >>> >>>>>> > [1] IPv6 Support main Jira ticket >>> >>>>>> > https://issues.apache.org/jira/browse/CLOUDSTACK-452 >>> >>>>>> > >>> >>>>>> > [2] IPv6 Support in CloudStack FS >>> >>>>>> > https://cwiki.apache.org/CLOUDSTACK/ipv6-support-in-cloudstack. >>> >>>>>> > html >>> >>>>>> > >>> >>>>>> > [3] IPv6 Support FS >>> >>>>>> > https://cwiki.apache.org/CLOUDSTACK/ipv6-support.html >>> >>>>>> >>> >
