Re: [ovs-dev] OVN implementation of the NAT/FIP architecture
First of all, thank you very much for sharing your thoughts. I really appreciate your perspective on how things are going with OVN. On 02/11/2016 07:41 PM, Matt Kassawara wrote: > So, what do we do? Suggest a hybrid architecture with the following aspects: > > 1) Instances that require access from external networks contain a port on > the provider network. > 2) Instances that only require occasional access to external networks > contain a port on a private network and use SNAT on a router. If an > instance requires access from external networks at a later point, add a > port on the provider network. > 3) Instances optionally contain a port on one or more private networks to > facilitate communication among instances. > 4) Use a combination of multiple network nodes, clever network scheduling, > and L3 HA (neutron implementation of VRRP) to increase performance, > scalability, and reliability of routing among private networks and SNAT. > 5) Distribute DHCP and metadata services across multiple controller or > compute nodes. This deployment method seems quite reasonable. In fact, it seems we support a fairly simple form of this already. Without a gateway, private networks remain for inter-VM communication only and you *must* put an interface on the provider network for external connectivity. An OVN gateway solution will cover the rest. Obviously we'll have to keep working on the gateway story to ensure good scale and HA, but it won't have everything on the first pass. > What's missing? Floating IP addresses! > > I think OVN should support the NAT/FIP architecture, but only as a > temporary measure to increase adoption by operators using it. In fact, we > should use the words "classic" or "legacy" around it in any documentation. > For new deployments, suggest a hybrid architecture and provide ample > documentation for it similar to the scenarios in the networking guide. > Meanwhile, development should focus on supporting actual routing such as > advertisement of private networks via one or more dynamic routing > protocols. After release of this feature, deprecate and eventually remove > NAT/FIP support. Given how popular floating IPs are, I'm not sure I'm convinced we should get rid of it. It doesn't seem like a lot of extra work to keep it on top of what we'd need anyway. > P.S. - If OVN decides to support the NAT/FIP architecture, I strongly > recommend also offering some form of HA such as VRRP in the initial release > to increase appeal to operators, most of whom became wary of neutron's > inability to eliminate single points of failure for so many releases. > I don't want OVN to repeat mistakes of neutron, even for a little bit. It depends what you mean by "initial release". I don't think we should not merge code, but it could be a question of when we drop the "experimental" label from OVN, or at least the gateway functionality. -- Russell Bryant ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
Re: [ovs-dev] OVN implementation of the NAT/FIP architecture
Hi, I would also like to express my concerns. I agree that FIP and in general NAT traversal on a single node are bottlenecks and single points of failure. In general even supporting HA using VRRP save from the single point of failure but doesn’t really solve the bottleneck issue. We should also think about bigger providers and in general multi-datacenter deployments like my case for example where the validity of a publicly routable ip address is restricted to a subset of the compute nodes available in specific to a target availability zone. That said a deployment where vms that need public access are attached to a provider network seems more reasonable. And in general more productive because the network topology can be maintained valid even if the underlaying datacenter is different. It is also possible to provide remote access to vms by hooking a port in the network in atarget node that can run simply ovs+ovn-controller and this is replicable in more than onenode making effectively no single point of failure and in general no bottlenecks. As I can see right now the only real missing part is a way to provide external access to vms without attaching a provider network interface but this can be achieved in my personal opinion using NAT capability of the linux stack holding the hypervisor as described: VM (private_net, 10.0.0.10/24) -> ROUTER(10.0.0.1, 172.16.0.1) -> COMPUTE(172.16.0.2) Where 172.16.0.2 is an internal port directly attached to an ovs bridge and on the compute an iptables policy like “iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -o eth0 -j MASQUERADE" This solution will not permit implementation of a DNAT in general because ip addresses in the range of 172.16.0.0/24 are only known internally in every compute node even routing between this ip addresses on different computes is impossible (actually not easy but possible) The only missing point in OVN right now is the ability to route private network traffic to provider networks but the implementation of routing already make possible to have a virtual ip, 10.0.0.1 in the previous example, distributed on every compute node. In this way practically vms can access the external world using the network stack of the compute node and can be accessed by specific nodes (with hooked ports) This nodes in general can provide any king of service in my specific case I use this nodes to proxy and load balance traffic with a bunch of haproxy installations and a DNS round robin. For vms that need instead direct access the solution with a port in a provider network is just perfect. In this case a provider network can be extended over more physical locations and the FIP (in a publicly routable range) can be dynamically assigned by automatically configuring a DNAT on the network equipments. > On 12 Feb 2016, at 01:41, Matt Kassawara wrote: > > Hi, > > I see interest in OVN implementing the conventional NAT/FIP architecture > from the OpenStack Networking service (neutron) and think it warrants > further discussion to make sure the project takes an optimal approach to it. > > For those not familiar with neutron, routers between provider and private > networks primarily use NAT rather than actual routing. Each router contains > at least two interfaces, one on a provider network typically using publicly > routable IP space and one on each private network typically using RFC1918 > IP space. Instances on private networks access the provider network and > other external networks via SNAT on the router. Instances that require > access from external networks can use a FIP address, essentially DNAT on > the router. If you disable NAT on a router, you must create a static route > for each private network or an aggregate of them pointing to the router > interface on the provider network... definitely not practical or scalable. > Finally, the NAT/FIP architecture does not work with IPv6. > > Before you think I'm just crazy, I contribute significantly to the > OpenStack installation and networking guides and also provide support for a > variety of operators. In my opinion, many operators implemented the NAT/FIP > architecture because it was the only neutron architecture with some level > of documentation for a couple of years after the initial release of > neutron. The NAT/FIP architecture quickly leads to performance, scaling, > and reliability problems because all network traffic between private and > provider networks and among private networks must traverse a single node > (network node) that contains the neutron routers. Furthermore, the > architecture also runs DHCP and metadata services on the network node. For > many new and existing operators, the lack of documentation and plenty of > horror stories gave neutron the appearance of a complex and fragile "black > box"... leading to skepticism about it that continues today. Attempting to > rectify these issues with DVR only made the situation worse, but I'll spare >
[ovs-dev] OVN implementation of the NAT/FIP architecture
Hi, I see interest in OVN implementing the conventional NAT/FIP architecture from the OpenStack Networking service (neutron) and think it warrants further discussion to make sure the project takes an optimal approach to it. For those not familiar with neutron, routers between provider and private networks primarily use NAT rather than actual routing. Each router contains at least two interfaces, one on a provider network typically using publicly routable IP space and one on each private network typically using RFC1918 IP space. Instances on private networks access the provider network and other external networks via SNAT on the router. Instances that require access from external networks can use a FIP address, essentially DNAT on the router. If you disable NAT on a router, you must create a static route for each private network or an aggregate of them pointing to the router interface on the provider network... definitely not practical or scalable. Finally, the NAT/FIP architecture does not work with IPv6. Before you think I'm just crazy, I contribute significantly to the OpenStack installation and networking guides and also provide support for a variety of operators. In my opinion, many operators implemented the NAT/FIP architecture because it was the only neutron architecture with some level of documentation for a couple of years after the initial release of neutron. The NAT/FIP architecture quickly leads to performance, scaling, and reliability problems because all network traffic between private and provider networks and among private networks must traverse a single node (network node) that contains the neutron routers. Furthermore, the architecture also runs DHCP and metadata services on the network node. For many new and existing operators, the lack of documentation and plenty of horror stories gave neutron the appearance of a complex and fragile "black box"... leading to skepticism about it that continues today. Attempting to rectify these issues with DVR only made the situation worse, but I'll spare you the details for now. A couple of years ago, some OpenStack documentation contributors with networking experience including myself decided that neutron would benefit from a separate guide [1]. Most importantly, the guide would include a variety of practical deployment architectures or scenarios with operational details and functional example configurations that operators could use as a starting point. Looking at all of the scenarios [2], the classic (NAT/FIP) scenarios offer the least performance, scaling, and reliability. The provider network scenarios offer the highest performance, scaling, and reliability... mostly due to simplicity. However, these scenarios effectively disable "self-service" networking... a cornerstone of cloud computing. So, what do we do? Suggest a hybrid architecture with the following aspects: 1) Instances that require access from external networks contain a port on the provider network. 2) Instances that only require occasional access to external networks contain a port on a private network and use SNAT on a router. If an instance requires access from external networks at a later point, add a port on the provider network. 3) Instances optionally contain a port on one or more private networks to facilitate communication among instances. 4) Use a combination of multiple network nodes, clever network scheduling, and L3 HA (neutron implementation of VRRP) to increase performance, scalability, and reliability of routing among private networks and SNAT. 5) Distribute DHCP and metadata services across multiple controller or compute nodes. What's missing? Floating IP addresses! I think OVN should support the NAT/FIP architecture, but only as a temporary measure to increase adoption by operators using it. In fact, we should use the words "classic" or "legacy" around it in any documentation. For new deployments, suggest a hybrid architecture and provide ample documentation for it similar to the scenarios in the networking guide. Meanwhile, development should focus on supporting actual routing such as advertisement of private networks via one or more dynamic routing protocols. After release of this feature, deprecate and eventually remove NAT/FIP support. P.S. - If OVN decides to support the NAT/FIP architecture, I strongly recommend also offering some form of HA such as VRRP in the initial release to increase appeal to operators, most of whom became wary of neutron's inability to eliminate single points of failure for so many releases. I don't want OVN to repeat mistakes of neutron, even for a little bit. Matt [1] http://docs.openstack.org/liberty/networking-guide/ [2] http://docs.openstack.org/liberty/networking-guide/deploy.html ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
