Re: [ovs-discuss] OVN: availability zones concept
Hi, Thanks all! I will start working on it pretty soon then. > Lucas, I support you moving forward with 1), because: > a) it is more simple and straightforward, yet the configuration on > chassis won't (shouldn't) create compatibility issue > b) it doesn't conflict with further optimization with 2) or 3) on top > of it when needed in the future > That's very true. This work is about creating a logical separation for Chassis and 2) and 3) is mostly about how the tunnels are formed. > P.S. Just a comment on the troubleshooting concerns. It seems to me 2) > doesn't necessarily increase difficulty of trouble-shooting: if lport > A and lport B cannot communicate, the first step to check should > always be making sure the logical datapath allows them to talk, > regardless of the approaches, and if the answer is yes, you know the > tunnel is expected between the chassises. For monitoring, tunnels do > not have states unless BFD is enabled (for GW HA), so maybe checking > on BFD failure status itself is sufficient. > Good points! Cheers, Lucas ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Fri, Mar 22, 2019 at 4:36 AM Daniel Alvarez wrote: > > > > > > On 22 Mar 2019, at 13:22, Lucas Alvares Gomes wrote: > > > > Hi, > > > > Getting back to this topic because we do have people (read clients) > > interested in this work for their Telco/Edge computing uses cases as a > > way to preventing each edge site to try to create tunnels with every > > node on every site. > > > > Also, another request we saw which this feature would help is related > > to security where people want to create "trust zones" and prevent > > computes in a more secure zone to communicate with a less secure zone. > > > > As Dan Sneddon pointed out in this thread, it's possible to use > > firewalls to workaround these problems but the approach is not ideal > > (and may result in negative consequences) because each Chassis in OVN > > will still try again and again to form the tunnel mesh with every > > other Chassis but it will fail/timeout over and over. > > > > Regarding to the approach to implement this I see we have a couple > > ideas floating around: > > > > 1. Explicit setting the transport zone(s) in the OpenVSwitch table for > > the Chassis. > > > > 2. Dynamically creating tunnels between Chassis that are logically > > connected with each other. > > > > 3. Creating the tunnels on demand when the first packet is going > > to/from a Chassis (although Ben has stated that he is not pushing for > > this). > > > > So, while every approach has their pos/cons, personally (and also by > > talking to some work colleagues), I think that 1. is probably the > > best/most straight forward way to go with this because since it's > > explicit it is also the easiest to troubleshoot/monitor what's going > > on with the network. The approach 2. and 3., while very interesting > > from an engineering point of view, might not be so interesting for > > folks responsible for monitoring the network because it leaves the > > compute nodes in some "unknown" state (is there a tunnel between > > Chassis A and B ? no/yes/not yet ?). Lucas, I support you moving forward with 1), because: a) it is more simple and straightforward, yet the configuration on chassis won't (shouldn't) create compatibility issue b) it doesn't conflict with further optimization with 2) or 3) on top of it when needed in the future P.S. Just a comment on the troubleshooting concerns. It seems to me 2) doesn't necessarily increase difficulty of trouble-shooting: if lport A and lport B cannot communicate, the first step to check should always be making sure the logical datapath allows them to talk, regardless of the approaches, and if the answer is yes, you know the tunnel is expected between the chassises. For monitoring, tunnels do not have states unless BFD is enabled (for GW HA), so maybe checking on BFD failure status itself is sufficient. > > > > If there's no objection from people in this thread (specially Han and > > Ben) I would be more than happy in helping implementing this feature. > My vote doesn’t count much because that’s the approach I originally proposed > in the first email of this thread but having thought about everything thrown > here, I’m inclined to go ahead with it. It doesn’t do any harm if not used > and brings the value we need for the commented use case. > > Thanks a lot Lucas for volunteering! > > > > > > Cheers, > > Lucas > > > >> On Thu, Mar 7, 2019 at 6:54 PM Ben Pfaff wrote: > >> > >>> On Wed, Mar 06, 2019 at 10:32:29PM -0800, Han Zhou wrote: > On Wed, Mar 6, 2019 at 9:06 AM Ben Pfaff wrote: > > > On Tue, Mar 05, 2019 at 09:39:37PM -0800, Han Zhou wrote: > >> On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > >> What's the effective difference between an OVN deployment with 3 zones, > >> and a collection of 3 OVN deployments? Is it simply that the 3-zone > >> deployment shares databases? Is that a significant advantage? > > > > Hi Ben, based on the discussions there are two cases: > > > > For completely separated zones (no overlapping) v.s. separate OVN > > deployments, the difference is that separate OVN deployments requires > > some sort of federation at a higher layer, so that a single CMS can > > operate multiple OVN deployments. Of course separate zones in same OVN > > still requires changes in CMS to operate but the change may be smaller > > in some cases. > > > > For overlapping zones v.s. separate OVN deployments, the difference is > > more obvious. Separate OVN deployments doesn't allow overlapping. > > Overlapping zones allows sharing gateways between different groups of > > hypervisors. > > OK. The difference is obvious in the case where there is overlap. > > > If the purpose is only reducing tunnel mesh size, I think it may be > > better to avoid the zone concept but instead create tunnels (and bfd > > sessions) on-demand, as discussed here: > > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/0
Re: [ovs-discuss] OVN: availability zones concept
On Fri, Mar 22, 2019 at 11:22:50AM +, Lucas Alvares Gomes wrote: > Hi, > > Getting back to this topic because we do have people (read clients) > interested in this work for their Telco/Edge computing uses cases as a > way to preventing each edge site to try to create tunnels with every > node on every site. > > Also, another request we saw which this feature would help is related > to security where people want to create "trust zones" and prevent > computes in a more secure zone to communicate with a less secure zone. > > As Dan Sneddon pointed out in this thread, it's possible to use > firewalls to workaround these problems but the approach is not ideal > (and may result in negative consequences) because each Chassis in OVN > will still try again and again to form the tunnel mesh with every > other Chassis but it will fail/timeout over and over. > > Regarding to the approach to implement this I see we have a couple > ideas floating around: > > 1. Explicit setting the transport zone(s) in the OpenVSwitch table for > the Chassis. > > 2. Dynamically creating tunnels between Chassis that are logically > connected with each other. > > 3. Creating the tunnels on demand when the first packet is going > to/from a Chassis (although Ben has stated that he is not pushing for > this). > > So, while every approach has their pos/cons, personally (and also by > talking to some work colleagues), I think that 1. is probably the > best/most straight forward way to go with this because since it's > explicit it is also the easiest to troubleshoot/monitor what's going > on with the network. The approach 2. and 3., while very interesting > from an engineering point of view, might not be so interesting for > folks responsible for monitoring the network because it leaves the > compute nodes in some "unknown" state (is there a tunnel between > Chassis A and B ? no/yes/not yet ?). > > If there's no objection from people in this thread (specially Han and > Ben) I would be more than happy in helping implementing this feature. I don't object. I'm happy to see OVN become appropriate for more use cases. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
> On 22 Mar 2019, at 13:22, Lucas Alvares Gomes wrote: > > Hi, > > Getting back to this topic because we do have people (read clients) > interested in this work for their Telco/Edge computing uses cases as a > way to preventing each edge site to try to create tunnels with every > node on every site. > > Also, another request we saw which this feature would help is related > to security where people want to create "trust zones" and prevent > computes in a more secure zone to communicate with a less secure zone. > > As Dan Sneddon pointed out in this thread, it's possible to use > firewalls to workaround these problems but the approach is not ideal > (and may result in negative consequences) because each Chassis in OVN > will still try again and again to form the tunnel mesh with every > other Chassis but it will fail/timeout over and over. > > Regarding to the approach to implement this I see we have a couple > ideas floating around: > > 1. Explicit setting the transport zone(s) in the OpenVSwitch table for > the Chassis. > > 2. Dynamically creating tunnels between Chassis that are logically > connected with each other. > > 3. Creating the tunnels on demand when the first packet is going > to/from a Chassis (although Ben has stated that he is not pushing for > this). > > So, while every approach has their pos/cons, personally (and also by > talking to some work colleagues), I think that 1. is probably the > best/most straight forward way to go with this because since it's > explicit it is also the easiest to troubleshoot/monitor what's going > on with the network. The approach 2. and 3., while very interesting > from an engineering point of view, might not be so interesting for > folks responsible for monitoring the network because it leaves the > compute nodes in some "unknown" state (is there a tunnel between > Chassis A and B ? no/yes/not yet ?). > > If there's no objection from people in this thread (specially Han and > Ben) I would be more than happy in helping implementing this feature. My vote doesn’t count much because that’s the approach I originally proposed in the first email of this thread but having thought about everything thrown here, I’m inclined to go ahead with it. It doesn’t do any harm if not used and brings the value we need for the commented use case. Thanks a lot Lucas for volunteering! > > Cheers, > Lucas > >> On Thu, Mar 7, 2019 at 6:54 PM Ben Pfaff wrote: >> >>> On Wed, Mar 06, 2019 at 10:32:29PM -0800, Han Zhou wrote: On Wed, Mar 6, 2019 at 9:06 AM Ben Pfaff wrote: > On Tue, Mar 05, 2019 at 09:39:37PM -0800, Han Zhou wrote: >> On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: >> What's the effective difference between an OVN deployment with 3 zones, >> and a collection of 3 OVN deployments? Is it simply that the 3-zone >> deployment shares databases? Is that a significant advantage? > > Hi Ben, based on the discussions there are two cases: > > For completely separated zones (no overlapping) v.s. separate OVN > deployments, the difference is that separate OVN deployments requires > some sort of federation at a higher layer, so that a single CMS can > operate multiple OVN deployments. Of course separate zones in same OVN > still requires changes in CMS to operate but the change may be smaller > in some cases. > > For overlapping zones v.s. separate OVN deployments, the difference is > more obvious. Separate OVN deployments doesn't allow overlapping. > Overlapping zones allows sharing gateways between different groups of > hypervisors. OK. The difference is obvious in the case where there is overlap. > If the purpose is only reducing tunnel mesh size, I think it may be > better to avoid the zone concept but instead create tunnels (and bfd > sessions) on-demand, as discussed here: > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html Except in cases where we have BFD sessions, it is possible to entirely avoid having explicitly defined tunnels, since the tunnels can be defined in the flow table. The ovs-fields(7) manpage describes these under "flow-based tunnels" in the TUNNEL FIELDS section. Naively, doing it this way would require, on each hypervisor, a few OpenFlow flows per remote chassis, as opposed to one port per remote chassis. That probably scales better. If necessary, it could be made to scale better than that by using send-to-controller actions to add flows for tunnels as packets arrive for them or as packets need to go through them. >>> >>> Thanks Ben for the pointer. I have to admit I was not aware of these >>> different ways of using tunnels. The documentation is very clear, and >>> now I understand what OVN currently uses is "Intermediate models", >>> i.e. partially flow-based - remote-ips are port based while keys are >>> flow based. >> >> Thank
Re: [ovs-discuss] OVN: availability zones concept
Hi, Getting back to this topic because we do have people (read clients) interested in this work for their Telco/Edge computing uses cases as a way to preventing each edge site to try to create tunnels with every node on every site. Also, another request we saw which this feature would help is related to security where people want to create "trust zones" and prevent computes in a more secure zone to communicate with a less secure zone. As Dan Sneddon pointed out in this thread, it's possible to use firewalls to workaround these problems but the approach is not ideal (and may result in negative consequences) because each Chassis in OVN will still try again and again to form the tunnel mesh with every other Chassis but it will fail/timeout over and over. Regarding to the approach to implement this I see we have a couple ideas floating around: 1. Explicit setting the transport zone(s) in the OpenVSwitch table for the Chassis. 2. Dynamically creating tunnels between Chassis that are logically connected with each other. 3. Creating the tunnels on demand when the first packet is going to/from a Chassis (although Ben has stated that he is not pushing for this). So, while every approach has their pos/cons, personally (and also by talking to some work colleagues), I think that 1. is probably the best/most straight forward way to go with this because since it's explicit it is also the easiest to troubleshoot/monitor what's going on with the network. The approach 2. and 3., while very interesting from an engineering point of view, might not be so interesting for folks responsible for monitoring the network because it leaves the compute nodes in some "unknown" state (is there a tunnel between Chassis A and B ? no/yes/not yet ?). If there's no objection from people in this thread (specially Han and Ben) I would be more than happy in helping implementing this feature. Cheers, Lucas On Thu, Mar 7, 2019 at 6:54 PM Ben Pfaff wrote: > > On Wed, Mar 06, 2019 at 10:32:29PM -0800, Han Zhou wrote: > > On Wed, Mar 6, 2019 at 9:06 AM Ben Pfaff wrote: > > > > > > On Tue, Mar 05, 2019 at 09:39:37PM -0800, Han Zhou wrote: > > > > On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > > > > > What's the effective difference between an OVN deployment with 3 > > > > > zones, > > > > > and a collection of 3 OVN deployments? Is it simply that the 3-zone > > > > > deployment shares databases? Is that a significant advantage? > > > > > > > > Hi Ben, based on the discussions there are two cases: > > > > > > > > For completely separated zones (no overlapping) v.s. separate OVN > > > > deployments, the difference is that separate OVN deployments requires > > > > some sort of federation at a higher layer, so that a single CMS can > > > > operate multiple OVN deployments. Of course separate zones in same OVN > > > > still requires changes in CMS to operate but the change may be smaller > > > > in some cases. > > > > > > > > For overlapping zones v.s. separate OVN deployments, the difference is > > > > more obvious. Separate OVN deployments doesn't allow overlapping. > > > > Overlapping zones allows sharing gateways between different groups of > > > > hypervisors. > > > > > > OK. The difference is obvious in the case where there is overlap. > > > > > > > If the purpose is only reducing tunnel mesh size, I think it may be > > > > better to avoid the zone concept but instead create tunnels (and bfd > > > > sessions) on-demand, as discussed here: > > > > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html > > > > > > Except in cases where we have BFD sessions, it is possible to entirely > > > avoid having explicitly defined tunnels, since the tunnels can be > > > defined in the flow table. The ovs-fields(7) manpage describes these > > > under "flow-based tunnels" in the TUNNEL FIELDS section. Naively, doing > > > it this way would require, on each hypervisor, a few OpenFlow flows per > > > remote chassis, as opposed to one port per remote chassis. That > > > probably scales better. If necessary, it could be made to scale better > > > than that by using send-to-controller actions to add flows for tunnels > > > as packets arrive for them or as packets need to go through them. > > > > Thanks Ben for the pointer. I have to admit I was not aware of these > > different ways of using tunnels. The documentation is very clear, and > > now I understand what OVN currently uses is "Intermediate models", > > i.e. partially flow-based - remote-ips are port based while keys are > > flow based. > > Thanks for the documentation fixes! > > > While purely flow-based tunnel is attractive in terms of flexibility, > > it seems not fit very well for OVN use case because we do need BFD > > sessions. > > I think that OVN only uses BFD for a few of its ports--only for gateways > with HA, right? Those could continue to have ports. > > > For the "send-to-controller", i.e. reactively set up flows when > > packets arrives, I hope it is not
Re: [ovs-discuss] OVN: availability zones concept
On Wed, Mar 06, 2019 at 10:32:29PM -0800, Han Zhou wrote: > On Wed, Mar 6, 2019 at 9:06 AM Ben Pfaff wrote: > > > > On Tue, Mar 05, 2019 at 09:39:37PM -0800, Han Zhou wrote: > > > On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > > > > What's the effective difference between an OVN deployment with 3 zones, > > > > and a collection of 3 OVN deployments? Is it simply that the 3-zone > > > > deployment shares databases? Is that a significant advantage? > > > > > > Hi Ben, based on the discussions there are two cases: > > > > > > For completely separated zones (no overlapping) v.s. separate OVN > > > deployments, the difference is that separate OVN deployments requires > > > some sort of federation at a higher layer, so that a single CMS can > > > operate multiple OVN deployments. Of course separate zones in same OVN > > > still requires changes in CMS to operate but the change may be smaller > > > in some cases. > > > > > > For overlapping zones v.s. separate OVN deployments, the difference is > > > more obvious. Separate OVN deployments doesn't allow overlapping. > > > Overlapping zones allows sharing gateways between different groups of > > > hypervisors. > > > > OK. The difference is obvious in the case where there is overlap. > > > > > If the purpose is only reducing tunnel mesh size, I think it may be > > > better to avoid the zone concept but instead create tunnels (and bfd > > > sessions) on-demand, as discussed here: > > > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html > > > > Except in cases where we have BFD sessions, it is possible to entirely > > avoid having explicitly defined tunnels, since the tunnels can be > > defined in the flow table. The ovs-fields(7) manpage describes these > > under "flow-based tunnels" in the TUNNEL FIELDS section. Naively, doing > > it this way would require, on each hypervisor, a few OpenFlow flows per > > remote chassis, as opposed to one port per remote chassis. That > > probably scales better. If necessary, it could be made to scale better > > than that by using send-to-controller actions to add flows for tunnels > > as packets arrive for them or as packets need to go through them. > > Thanks Ben for the pointer. I have to admit I was not aware of these > different ways of using tunnels. The documentation is very clear, and > now I understand what OVN currently uses is "Intermediate models", > i.e. partially flow-based - remote-ips are port based while keys are > flow based. Thanks for the documentation fixes! > While purely flow-based tunnel is attractive in terms of flexibility, > it seems not fit very well for OVN use case because we do need BFD > sessions. I think that OVN only uses BFD for a few of its ports--only for gateways with HA, right? Those could continue to have ports. > For the "send-to-controller", i.e. reactively set up flows when > packets arrives, I hope it is not really needed for solving the tunnel > scaling problem, since it introduces data plane latency which could be > a bigger problem. (But I am not sure if reactive mode in general is a > good idea - it might be a reasonable trade-off for solving the scale > problem of each HV pre-installing flows for all related datapaths in a > full-mesh alike scenario. Anyway, not directly related to current > topic). It would introduce data plane latency for the first packet to go to or from a particular hypervisor. After that there would be no further additional latency. It would probably not be noticeable. Let me be clear that I am not pushing this solution. It will complicate things, and I do not like unnecessary complication. I am just pointing out that is possible. > So I would propose to keep the current partially flow-based tunnel > usage in OVN and optimize the tunnel setup only between peers that are > logically connected, if this satisfies the scaling goal of OVN users. > Even with this optimization, we may need to make it as a configurable > option, since in small scale use cases users may in practice prefer > the original behavior to avoid the latency of tunnel setup. OK. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Wed, Mar 6, 2019 at 9:06 AM Ben Pfaff wrote: > > On Tue, Mar 05, 2019 at 09:39:37PM -0800, Han Zhou wrote: > > On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > > > What's the effective difference between an OVN deployment with 3 zones, > > > and a collection of 3 OVN deployments? Is it simply that the 3-zone > > > deployment shares databases? Is that a significant advantage? > > > > Hi Ben, based on the discussions there are two cases: > > > > For completely separated zones (no overlapping) v.s. separate OVN > > deployments, the difference is that separate OVN deployments requires > > some sort of federation at a higher layer, so that a single CMS can > > operate multiple OVN deployments. Of course separate zones in same OVN > > still requires changes in CMS to operate but the change may be smaller > > in some cases. > > > > For overlapping zones v.s. separate OVN deployments, the difference is > > more obvious. Separate OVN deployments doesn't allow overlapping. > > Overlapping zones allows sharing gateways between different groups of > > hypervisors. > > OK. The difference is obvious in the case where there is overlap. > > > If the purpose is only reducing tunnel mesh size, I think it may be > > better to avoid the zone concept but instead create tunnels (and bfd > > sessions) on-demand, as discussed here: > > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html > > Except in cases where we have BFD sessions, it is possible to entirely > avoid having explicitly defined tunnels, since the tunnels can be > defined in the flow table. The ovs-fields(7) manpage describes these > under "flow-based tunnels" in the TUNNEL FIELDS section. Naively, doing > it this way would require, on each hypervisor, a few OpenFlow flows per > remote chassis, as opposed to one port per remote chassis. That > probably scales better. If necessary, it could be made to scale better > than that by using send-to-controller actions to add flows for tunnels > as packets arrive for them or as packets need to go through them. Thanks Ben for the pointer. I have to admit I was not aware of these different ways of using tunnels. The documentation is very clear, and now I understand what OVN currently uses is "Intermediate models", i.e. partially flow-based - remote-ips are port based while keys are flow based. While purely flow-based tunnel is attractive in terms of flexibility, it seems not fit very well for OVN use case because we do need BFD sessions. For the "send-to-controller", i.e. reactively set up flows when packets arrives, I hope it is not really needed for solving the tunnel scaling problem, since it introduces data plane latency which could be a bigger problem. (But I am not sure if reactive mode in general is a good idea - it might be a reasonable trade-off for solving the scale problem of each HV pre-installing flows for all related datapaths in a full-mesh alike scenario. Anyway, not directly related to current topic). So I would propose to keep the current partially flow-based tunnel usage in OVN and optimize the tunnel setup only between peers that are logically connected, if this satisfies the scaling goal of OVN users. Even with this optimization, we may need to make it as a configurable option, since in small scale use cases users may in practice prefer the original behavior to avoid the latency of tunnel setup. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Tue, Mar 05, 2019 at 09:39:37PM -0800, Han Zhou wrote: > On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > > What's the effective difference between an OVN deployment with 3 zones, > > and a collection of 3 OVN deployments? Is it simply that the 3-zone > > deployment shares databases? Is that a significant advantage? > > Hi Ben, based on the discussions there are two cases: > > For completely separated zones (no overlapping) v.s. separate OVN > deployments, the difference is that separate OVN deployments requires > some sort of federation at a higher layer, so that a single CMS can > operate multiple OVN deployments. Of course separate zones in same OVN > still requires changes in CMS to operate but the change may be smaller > in some cases. > > For overlapping zones v.s. separate OVN deployments, the difference is > more obvious. Separate OVN deployments doesn't allow overlapping. > Overlapping zones allows sharing gateways between different groups of > hypervisors. OK. The difference is obvious in the case where there is overlap. > If the purpose is only reducing tunnel mesh size, I think it may be > better to avoid the zone concept but instead create tunnels (and bfd > sessions) on-demand, as discussed here: > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html Except in cases where we have BFD sessions, it is possible to entirely avoid having explicitly defined tunnels, since the tunnels can be defined in the flow table. The ovs-fields(7) manpage describes these under "flow-based tunnels" in the TUNNEL FIELDS section. Naively, doing it this way would require, on each hypervisor, a few OpenFlow flows per remote chassis, as opposed to one port per remote chassis. That probably scales better. If necessary, it could be made to scale better than that by using send-to-controller actions to add flows for tunnels as packets arrive for them or as packets need to go through them. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
Thanks Dan for chiming in and others as well for your feedback! I also thought of having separate OVN deployments but that introduces the drawbacks that Han pointed out adding - maybe a lot of - burden to the CMS. Separate zones in the same OVN deployment will add minimal changes (at deployment size to define the zones and when selecting the nodes to schedule a gateway are the only ones I can think of). For the particular case of OpenStack, the overlapping TZs won't be of much help I guess as there's no notion of overlapping failure domains. However, if you see value for OVN itself or other CMS's, I think it's not too expensive to have (I'm possibly being naive here). That said, I think that Han's suggestion of establishing tunnels dynamically is *great*! It will improve scalability and will be more efficient generally (even if OpenStack is not using Availability Zones), reducing traffic and processing and avoiding deployer/ops to configure each node. The Transport/Availability zone concept will be then implemented at the CMS layer which makes sense to me (in the OpenStack case, just scheduling gateways). Only concern, as Han said, would be the latency between the time the port gets bound and it becomes available from other hypervisors (especially in the case of the encrypted tunnels). If we could have some figures before making the decision, it'd be great but my vote goes for this approach :) Han++ Thanks all, Daniel On Wed, Mar 6, 2019 at 11:27 AM Dan Sneddon wrote: > > > > On Tue, Mar 5, 2019 at 9:40 PM Han Zhou wrote: >> >> On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: >> > What's the effective difference between an OVN deployment with 3 zones, >> > and a collection of 3 OVN deployments? Is it simply that the 3-zone >> > deployment shares databases? Is that a significant advantage? >> >> Hi Ben, based on the discussions there are two cases: >> >> For completely separated zones (no overlapping) v.s. separate OVN >> deployments, the difference is that separate OVN deployments requires >> some sort of federation at a higher layer, so that a single CMS can >> operate multiple OVN deployments. Of course separate zones in same OVN >> still requires changes in CMS to operate but the change may be smaller >> in some cases. >> >> For overlapping zones v.s. separate OVN deployments, the difference is >> more obvious. Separate OVN deployments doesn't allow overlapping. >> Overlapping zones allows sharing gateways between different groups of >> hypervisors. >> >> If the purpose is only reducing tunnel mesh size, I think it may be >> better to avoid the zone concept but instead create tunnels (and bfd >> sessions) on-demand, as discussed here: >> https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html >> >> Daniel or other folks please comment if there are other benefit of >> creating zones. >> >> Thanks, >> Han > > > The original discussion came about when I was consulting with a very large > bank who were considering network designs for an application cloud. In that > case, all chassis were in a single site, and the desire was to be able to > separate groups of chassis into trust zones with no East-West communication > between zones. Of course this same result can be handled via network > segregation and firewalling, but zones would provide an additional layer of > security enforcement. In their case, the choice due to policy was to have > separate flow controllers and software routers in each zone rather than rely > on firewalls alone, but this increased the hardware footprint. > > When I discovered that there was no way to prevent tunnels from being formed > between all chassis, that became an obvious problem for edge scenarios. To me > that is the more pressing issue, which dynamic tunnels would solve. However, > the ability to have separate transit zones would also be a useful feature, in > my opinion. > > -- > Dan Sneddon > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Tue, Mar 5, 2019 at 9:40 PM Han Zhou wrote: > On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > > What's the effective difference between an OVN deployment with 3 zones, > > and a collection of 3 OVN deployments? Is it simply that the 3-zone > > deployment shares databases? Is that a significant advantage? > > Hi Ben, based on the discussions there are two cases: > > For completely separated zones (no overlapping) v.s. separate OVN > deployments, the difference is that separate OVN deployments requires > some sort of federation at a higher layer, so that a single CMS can > operate multiple OVN deployments. Of course separate zones in same OVN > still requires changes in CMS to operate but the change may be smaller > in some cases. > > For overlapping zones v.s. separate OVN deployments, the difference is > more obvious. Separate OVN deployments doesn't allow overlapping. > Overlapping zones allows sharing gateways between different groups of > hypervisors. > > If the purpose is only reducing tunnel mesh size, I think it may be > better to avoid the zone concept but instead create tunnels (and bfd > sessions) on-demand, as discussed here: > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html > > Daniel or other folks please comment if there are other benefit of > creating zones. > > Thanks, > Han > The original discussion came about when I was consulting with a very large bank who were considering network designs for an application cloud. In that case, all chassis were in a single site, and the desire was to be able to separate groups of chassis into trust zones with no East-West communication between zones. Of course this same result can be handled via network segregation and firewalling, but zones would provide an additional layer of security enforcement. In their case, the choice due to policy was to have separate flow controllers and software routers in each zone rather than rely on firewalls alone, but this increased the hardware footprint. When I discovered that there was no way to prevent tunnels from being formed between all chassis, that became an obvious problem for edge scenarios. To me that is the more pressing issue, which dynamic tunnels would solve. However, the ability to have separate transit zones would also be a useful feature, in my opinion. -- Dan Sneddon ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Tue, Mar 5, 2019 at 7:24 PM Ben Pfaff wrote: > What's the effective difference between an OVN deployment with 3 zones, > and a collection of 3 OVN deployments? Is it simply that the 3-zone > deployment shares databases? Is that a significant advantage? Hi Ben, based on the discussions there are two cases: For completely separated zones (no overlapping) v.s. separate OVN deployments, the difference is that separate OVN deployments requires some sort of federation at a higher layer, so that a single CMS can operate multiple OVN deployments. Of course separate zones in same OVN still requires changes in CMS to operate but the change may be smaller in some cases. For overlapping zones v.s. separate OVN deployments, the difference is more obvious. Separate OVN deployments doesn't allow overlapping. Overlapping zones allows sharing gateways between different groups of hypervisors. If the purpose is only reducing tunnel mesh size, I think it may be better to avoid the zone concept but instead create tunnels (and bfd sessions) on-demand, as discussed here: https://mail.openvswitch.org/pipermail/ovs-discuss/2019-March/048281.html Daniel or other folks please comment if there are other benefit of creating zones. Thanks, Han ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Thu, Feb 28, 2019 at 06:48:03PM +0100, Daniel Alvarez Sanchez wrote: > Just wanted to throw an idea here about introducing availability zones > (AZ) concept in OVN and get implementation ideas. From a CMS > perspective, it makes sense to be able to implement some sort of > logical division of resources into failure domains to maximize their > availability. > > In this sense, establishing a full mesh of Geneve tunnels is not > needed (and possibly undesired when strict firewalls are used between > AZs) as L2 connectivity will be constrained to the AZ boundaries. > > A possibility would be to let the deployer of the CMS set a key on the > OpenvSwitch table of the local OVS instance like > 'external_ids:ovn_az=' and if it's set, ovn-controller will > register itself as a Chassis with the same external ID and establish > tunnels to those Chassis within the same AZ, otherwise it'll keep the > current behavior. > > It'll be responsibility of the CMS to schedule gateway ports in the > right AZ as well to provide L3 AZ awareness. > > Does that make sense? Thoughts? What's the effective difference between an OVN deployment with 3 zones, and a collection of 3 OVN deployments? Is it simply that the 3-zone deployment shares databases? Is that a significant advantage? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Mon, Mar 4, 2019 at 4:48 PM Dan Sneddon wrote: > > > > On Mon, Mar 4, 2019 at 5:34 AM Lucas Alvares Gomes > wrote: >> >> Hi, >> >> On Sat, Mar 2, 2019 at 1:52 AM Han Zhou wrote: >> > >> > On Thu, Feb 28, 2019 at 9:58 AM Daniel Alvarez Sanchez >> > wrote: >> > > >> > > Hi folks, >> > > >> > > Just wanted to throw an idea here about introducing availability zones >> > > (AZ) concept in OVN and get implementation ideas. From a CMS >> > > perspective, it makes sense to be able to implement some sort of >> > > logical division of resources into failure domains to maximize their >> > > availability. >> > > >> > > In this sense, establishing a full mesh of Geneve tunnels is not >> > > needed (and possibly undesired when strict firewalls are used between >> > > AZs) as L2 connectivity will be constrained to the AZ boundaries. >> > > >> > > A possibility would be to let the deployer of the CMS set a key on the >> > > OpenvSwitch table of the local OVS instance like >> > > 'external_ids:ovn_az=' and if it's set, ovn-controller will >> > > register itself as a Chassis with the same external ID and establish >> > > tunnels to those Chassis within the same AZ, otherwise it'll keep the >> > > current behavior. >> > > >> > > It'll be responsibility of the CMS to schedule gateway ports in the >> > > right AZ as well to provide L3 AZ awareness. >> > > >> > > Does that make sense? Thoughts? >> > > >> > > Thanks a lot!! >> > > Daniel >> > > ___ >> > > discuss mailing list >> > > disc...@openvswitch.org >> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> > >> > This sounds like a good idea to me. Just a concern for the name "AZ". >> > The feature seems to be quite useful to optimize at scale when you >> > know there are different groups of chassises (and gateways) would >> > never need to communicate with each other. However, it doesn't sound >> > like availability zone concept, since it is managed by a single >> > control plane, which means they are not independently availability >> > zones. I'd call it TZ (transport zone), or maybe just cell. However, I >> > like the idea and it seems not hard to be implemented. >> > >> >> I agree with Han here, the idea is sound but the name seems a bit off. >> I specially liked the "transport zone" (TZs) suggested by Han here. So >> +1 to that name :D >> >> Quick question. Should we have a default TZ for the chassis/gateways >> that doesn't have that key set ? For example, if we have 9 chassis >> where three of them have the TZ key set to 1, three others setting TZ >> to 2 and remainder three left with no TZ key set. That should result >> in 3 different zones right ? I wanna clarify that because I don't >> think we should create a mash with all Chassis for those who doesn't >> have the TZ set, instead, if it's omitted ovn-controller could >> consider them to be part of a "default" TZ of some sort. >> >> What you think ? Is that aligned with your idea ? >> >> Cheers, >> Lucas > > > Hello, I'm chiming in here because I think Daniel was prompted to start this > thread based on a related feature request that I made against the OpenStack > OVN component. Transport Zones is a better name for this feature in my > opinion, and that is incidentally the name for this feature in VMWare NSX-T. > As far as I can tell, they are not claiming it to be a trademark, and it's a > fairly generic term. > > Having a default transport zone makes a lot of sense to me if we consider the > introduction of transport zones into an existing environment. If existing > chassis are not assigned to any transport zone, I think to most users it > would be expected behavior that chassis assigned to a new transport zone > would be separate. > > However, treating the default TZ as a separate TZ doesn't address a central > hub-and-spoke network where each chassis forms tunnels to other chassis in > the same TZ, and also forms tunnels with nodes at the central site, but > doesn't form tunnels with chassis in other specific TZs. In this alternative, > a chassis without a TZ forms tunnels with all other chassis. A chassis with a > specific TZ forms tunnels within that TZ and with chassis with no TZ > specified. That allows network functions such as gateways to be centralized > for all TZs. > > Perhaps an ideal solution would allow a chassis to be a member of more than > one TZ? That would allow flexibility, but may be more difficult to implement. > > -- > Dan Sneddon Good points on the default TZ and hub-and-spoke use case! To support that, I agree with the flexible way that a chassis is allowed to be a member of more than one TZ. So if TZ is not set, the chassis belongs to default TZ. If a chassis wants to join both default TZ and some other TZs, it should explicitely set to "default" and the other TZs. However, if our purpose here is just to avoid unnecessary tunnel mesh and improve tunnel/bfd scalability, another approach would be automatica
Re: [ovs-discuss] OVN: availability zones concept
On Mon, Mar 4, 2019 at 5:34 AM Lucas Alvares Gomes wrote: > Hi, > > On Sat, Mar 2, 2019 at 1:52 AM Han Zhou wrote: > > > > On Thu, Feb 28, 2019 at 9:58 AM Daniel Alvarez Sanchez > > wrote: > > > > > > Hi folks, > > > > > > Just wanted to throw an idea here about introducing availability zones > > > (AZ) concept in OVN and get implementation ideas. From a CMS > > > perspective, it makes sense to be able to implement some sort of > > > logical division of resources into failure domains to maximize their > > > availability. > > > > > > In this sense, establishing a full mesh of Geneve tunnels is not > > > needed (and possibly undesired when strict firewalls are used between > > > AZs) as L2 connectivity will be constrained to the AZ boundaries. > > > > > > A possibility would be to let the deployer of the CMS set a key on the > > > OpenvSwitch table of the local OVS instance like > > > 'external_ids:ovn_az=' and if it's set, ovn-controller will > > > register itself as a Chassis with the same external ID and establish > > > tunnels to those Chassis within the same AZ, otherwise it'll keep the > > > current behavior. > > > > > > It'll be responsibility of the CMS to schedule gateway ports in the > > > right AZ as well to provide L3 AZ awareness. > > > > > > Does that make sense? Thoughts? > > > > > > Thanks a lot!! > > > Daniel > > > ___ > > > discuss mailing list > > > disc...@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > This sounds like a good idea to me. Just a concern for the name "AZ". > > The feature seems to be quite useful to optimize at scale when you > > know there are different groups of chassises (and gateways) would > > never need to communicate with each other. However, it doesn't sound > > like availability zone concept, since it is managed by a single > > control plane, which means they are not independently availability > > zones. I'd call it TZ (transport zone), or maybe just cell. However, I > > like the idea and it seems not hard to be implemented. > > > > I agree with Han here, the idea is sound but the name seems a bit off. > I specially liked the "transport zone" (TZs) suggested by Han here. So > +1 to that name :D > > Quick question. Should we have a default TZ for the chassis/gateways > that doesn't have that key set ? For example, if we have 9 chassis > where three of them have the TZ key set to 1, three others setting TZ > to 2 and remainder three left with no TZ key set. That should result > in 3 different zones right ? I wanna clarify that because I don't > think we should create a mash with all Chassis for those who doesn't > have the TZ set, instead, if it's omitted ovn-controller could > consider them to be part of a "default" TZ of some sort. > > What you think ? Is that aligned with your idea ? > > Cheers, > Lucas > Hello, I'm chiming in here because I think Daniel was prompted to start this thread based on a related feature request that I made against the OpenStack OVN component. Transport Zones is a better name for this feature in my opinion, and that is incidentally the name for this feature in VMWare NSX-T. As far as I can tell, they are not claiming it to be a trademark, and it's a fairly generic term. Having a default transport zone makes a lot of sense to me if we consider the introduction of transport zones into an existing environment. If existing chassis are not assigned to any transport zone, I think to most users it would be expected behavior that chassis assigned to a new transport zone would be separate. However, treating the default TZ as a separate TZ doesn't address a central hub-and-spoke network where each chassis forms tunnels to other chassis in the same TZ, and also forms tunnels with nodes at the central site, but doesn't form tunnels with chassis in other specific TZs. In this alternative, a chassis without a TZ forms tunnels with all other chassis. A chassis with a specific TZ forms tunnels within that TZ and with chassis with no TZ specified. That allows network functions such as gateways to be centralized for all TZs. Perhaps an ideal solution would allow a chassis to be a member of more than one TZ? That would allow flexibility, but may be more difficult to implement. -- Dan Sneddon ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
Hi, On Sat, Mar 2, 2019 at 1:52 AM Han Zhou wrote: > > On Thu, Feb 28, 2019 at 9:58 AM Daniel Alvarez Sanchez > wrote: > > > > Hi folks, > > > > Just wanted to throw an idea here about introducing availability zones > > (AZ) concept in OVN and get implementation ideas. From a CMS > > perspective, it makes sense to be able to implement some sort of > > logical division of resources into failure domains to maximize their > > availability. > > > > In this sense, establishing a full mesh of Geneve tunnels is not > > needed (and possibly undesired when strict firewalls are used between > > AZs) as L2 connectivity will be constrained to the AZ boundaries. > > > > A possibility would be to let the deployer of the CMS set a key on the > > OpenvSwitch table of the local OVS instance like > > 'external_ids:ovn_az=' and if it's set, ovn-controller will > > register itself as a Chassis with the same external ID and establish > > tunnels to those Chassis within the same AZ, otherwise it'll keep the > > current behavior. > > > > It'll be responsibility of the CMS to schedule gateway ports in the > > right AZ as well to provide L3 AZ awareness. > > > > Does that make sense? Thoughts? > > > > Thanks a lot!! > > Daniel > > ___ > > discuss mailing list > > disc...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > This sounds like a good idea to me. Just a concern for the name "AZ". > The feature seems to be quite useful to optimize at scale when you > know there are different groups of chassises (and gateways) would > never need to communicate with each other. However, it doesn't sound > like availability zone concept, since it is managed by a single > control plane, which means they are not independently availability > zones. I'd call it TZ (transport zone), or maybe just cell. However, I > like the idea and it seems not hard to be implemented. > I agree with Han here, the idea is sound but the name seems a bit off. I specially liked the "transport zone" (TZs) suggested by Han here. So +1 to that name :D Quick question. Should we have a default TZ for the chassis/gateways that doesn't have that key set ? For example, if we have 9 chassis where three of them have the TZ key set to 1, three others setting TZ to 2 and remainder three left with no TZ key set. That should result in 3 different zones right ? I wanna clarify that because I don't think we should create a mash with all Chassis for those who doesn't have the TZ set, instead, if it's omitted ovn-controller could consider them to be part of a "default" TZ of some sort. What you think ? Is that aligned with your idea ? Cheers, Lucas ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVN: availability zones concept
On Thu, Feb 28, 2019 at 9:58 AM Daniel Alvarez Sanchez wrote: > > Hi folks, > > Just wanted to throw an idea here about introducing availability zones > (AZ) concept in OVN and get implementation ideas. From a CMS > perspective, it makes sense to be able to implement some sort of > logical division of resources into failure domains to maximize their > availability. > > In this sense, establishing a full mesh of Geneve tunnels is not > needed (and possibly undesired when strict firewalls are used between > AZs) as L2 connectivity will be constrained to the AZ boundaries. > > A possibility would be to let the deployer of the CMS set a key on the > OpenvSwitch table of the local OVS instance like > 'external_ids:ovn_az=' and if it's set, ovn-controller will > register itself as a Chassis with the same external ID and establish > tunnels to those Chassis within the same AZ, otherwise it'll keep the > current behavior. > > It'll be responsibility of the CMS to schedule gateway ports in the > right AZ as well to provide L3 AZ awareness. > > Does that make sense? Thoughts? > > Thanks a lot!! > Daniel > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss This sounds like a good idea to me. Just a concern for the name "AZ". The feature seems to be quite useful to optimize at scale when you know there are different groups of chassises (and gateways) would never need to communicate with each other. However, it doesn't sound like availability zone concept, since it is managed by a single control plane, which means they are not independently availability zones. I'd call it TZ (transport zone), or maybe just cell. However, I like the idea and it seems not hard to be implemented. Thanks, Han ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
