Re: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support
Ryan, I'll send out a V2 patch with the documentation changes you have suggested. In the IP_DNAT stage inport is reset to address cases which involve east-west communication via a floating ip where both the source and the destination VM's are in the same logical switch. This would require that the router send out the packet on the same interface it came in after applying the NAT rules. Thanks, Chandra Ryan Moats/Omaha/IBM wrote on 03/25/2016 10:15:22 AM: > From: Ryan Moats/Omaha/IBM > To: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS > Cc: dev@openvswitch.org > Date: 03/25/2016 10:15 AM > Subject: Re: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support > > "dev" <dev-boun...@openvswitch.org> wrote on 03/22/2016 04:19:07 PM: > > > From: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS > > To: dev@openvswitch.org > > Date: 03/22/2016 04:19 PM > > Subject: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support > > Sent by: "dev" <dev-boun...@openvswitch.org> > > > > This patch adds distributed floating ip support for ovn. The assumption made > > here is that the external network is a single L2 broadcast domain > and all the > > chassis have connectivity to the external network. > > > > 2 new tables are added in the LROUTER pipeline IN_IP_DNAT & IP_IN_SNAT. > > IN_IP_DNAT will modify the dst ip of the packet from floating ip to vm ip. > > IN_IP_SNAT will modify the src ip of the packet from vm ip to floating ip. > > > > Rules in IN_IP_DNAT: > > - Priority 100 rule to set the reg2 to 0x1 if dst & src networks are > > connectected via a router and both the networks are private. > > - Priority 90 rule to modify the dst ip from floating ip to vm ip. > > - Priority 0 rule to go to next table. > > > > Rules in IN_IP_SNAT: > > - Priority 100 rule to skip modifying the src ip when reg2 is set to 0x1 > > - Priority 90 rule to modify the src ip from vm ip to floating ip > and dst mac > > to floating ip port mac if the packet is egressing via the gateway port > > - Priority 50 rule to modify the src ip from vm ip to floating ip > > - Priority 0 rule to go to next table. > > > > Priority 100 rules in IN_IP_DNAT and IN_IP_SNAT serves 2 purposes. > > - Avoid NAT when vms in different LSWITCHES connected via a LROUTER talk to > > each other using private ips. > > - When 2 VMs connected to the same LSWITCH or different LSWITCHES connected > > via a router try to talk to each other, the dst ip of the packet should > > first be DNATed and then the src ip should be SNATed. > > > > The initial design was to stage DNAT in the ingress pipeline and the SNAT in > > the egress pipeline, but now both the stages are in the ingress > pipeline. This > > was done to solve the cases highlighted above [Priority 100 > rules]. There is a > > need to use information from DNAT stage when SNAT is being processed. This > > would require an explicit register to be burnt to store the information. > > > > Flows modified in the LSWITCH pipeline > > > > Rules in IN_PORT_SEC: > > - Priority 50 rule to allow packets ingressing the LSWITCH router port > > with a src mac of floating ip port > > > > Rules in ARP_RSP: > > - Priority 150 rule to respond to arp request for floating ip. To > prevent arp > > responses for floating ip's from all the chassis, "lport" optionis set in > > the external_id's column of the lflow table. lport will point to > > the vif-id of > > the vm that is associated with the floating ip. When ovn-controller is > > processing the flows, if it sees an lport option set in the external_ids > > column, it will install this lflow only if the lport is a local > port on the > > chassis. > > > > Rules in L2_LKUP: > > - Priority 50 rule to set the outport to the lrouter port when the dst mac > > matches the floating ip mac > > > > Rules in OUT_PORT_SEC: > > - Priority 50 rule to allow packet egressing the lrouter port witha mac of a > > floating ip port. > > > > Had to increase MAX_RESUBMIT_RECURSION from 64 to 96. When 2 VMs connected > > via vm1->LS->LR->LS->LR->LS->vm2 are trying to talk to each other, the > > resubmits are exceeding the existing 64 limit. > > > > When a floating ip is associated with a VM ip, NB will set the > options of the > > floating ip lport to "fixed-ip-port=, router- > port= > logical router port". > > > > If you want to try out this patch with openstack, add the > following patch [1] > > to networking-ovn. > > > > [1] https://review.openstack.org/#/c/295547/ > I know this is an RFC patch, but I'd really like to see two improvements > w.r.t the documentation: > > 1. include the needed changes to ovn-northd.8.xml. > 2. It looks like the DNAT code is also clearing the inport value and I > didn't find the reason why anywhere - I'm pretty sure I know why, but > confirmation would be good. > > Ryan Moats (regXboi) ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
Re: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support
10.1.2.5 responds to VM 10.1.1.3 [IN PVM3]: m_vm3, 10.1.2.5 -> m_net2, 10.1.1.3 The return path from 10.1.2.5 --> 10.1.1.3 will follow the exact same path as outlined in steps 1-8 above. The packet is not altered in the IP_DNAT and the IP_SNAT stages. [OUT PVM1]: m_net1, 10.1.2.5 -> m_vm1, 10.1.1.3 ___ Scenario 4: East <--> West via FIP, both endpoints on the same logical switch. (VM1 <--> FIP2 (VM2)) ___ [IN PVM1]: m_vm1, 10.1.1.3 -> m_net1, 198.44.1.4 1. Packet ingresses the Net1 logical switch on port PVM1 and will egress out of port P1 since the DMAC matches the router port's mac address. [OUT P1]: m_vm1, 10.1.1.3 -> m_net1, 198.44.1.4 [IN RP1]: m_vm1, 10.1.1.3 -> m_net1, 198.44.1.4 2. Packet ingresses the logical router on port RP1 and passes through the ingress security rules. 3. Packet gets processed in IP_DNAT stage where the priority 90 rule will be hit, since the DIP of the packet matches the FIP3 IP. The DIP of the packet will be changed from 198.44.1.4(FIP3 IP) to 10.1.1.4 (VM3 IP). The rule will also reset the inport, since this packet should egress out on the same interface it ingresses in. m_vm1, 10.1.1.3 -> m_net1, 10.1.1.4 (inport = " ") 4. Packet gets processed in the ROUTING stage and the SMAC is changed to m_net1 and the outport is set to RP1. m_net1, 10.1.1.3 -> m_net1, 10.1.1.4 (inport = " ") 5. Packet enters IP_SNAT stage where the priority 50 rule will be hit and the SIP is changed from 10.1.1.3(VM1 IP) to 198.44.1.3(FIP1 IP). Priority 90 rules do not match here, since the outport is not RP3. m_net1, 198.44.1.3 -> m_net1, 10.1.1.4 6. Packet gets processed in ARP_RESOLVE stage where the priority 100 rule will be hit and the DMAC of the packet is changed to VM2 mac (m_vm2) m_net1, 198.44.1.3 -> m_vm2, 10.1.1.4 7. Packet egresses out of the logical router via port RP1. The egress rules drops packets which match inport == outport, but in this case the inport was reset in IP_DNAT stage, so the packet egresses out. [OUT RP1]: m_net1, 198.44.1.3 -> m_vm2, 10.1.1.4 [IN P1 ]: m_net1, 198.44.1.3 -> m_vm2, 10.1.1.4 8. The packet ingresses Net1 logical switch on port P1 and after further processing, the packet will be sent out of port PVM2 to VM2. [OUT PVM2]: m_net1, 198.44.1.3 -> m_vm2, 10.1.1.4 VM 10.1.1.4 responds to 198.44.1.3 [IN PVM2]: m_vm2, 10.1.1.4 -> m_net1, 198.44.1.3 The return path from 10.1.1.4 --> 198.44.1.3 will follow the exact same path as outlined in steps 1-8 above. The DIP gets modified in IP_DNAT stage to 198.44.1.4 and the inport is reset. In the IP_SNAT stage the SIP gets modified to 10.1.1.3. [OUT PVM1]: m_net1, 198.44.1.4 -> m_vm1, 10.1.1.3 Guru Shetty <g...@ovn.org> wrote on 03/25/2016 10:56:20 AM: > From: Guru Shetty <g...@ovn.org> > To: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS > Cc: ovs dev <dev@openvswitch.org> > Date: 03/25/2016 10:56 AM > Subject: Re: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support > > It would really help if you give packet walkthrough when following 2 > simultaneous connections happen at the same time. > 1. east-west without NAT between the private IP addresses > 2 east-west with floating IP. > > You also mentioned in the meeting (if I remember correctly) on how > you have to keep a particular interface pinned to a hypervisor. A > little description on that would help. > > Also, does this work with overlay networking? > > On 22 March 2016 at 14:19, Chandra S Vejendla <csvej...@us.ibm.com> wrote: > This patch adds distributed floating ip support for ovn. The assumption made > here is that the external network is a single L2 broadcast domain and all the > chassis have connectivity to the external network. > > 2 new tables are added in the LROUTER pipeline IN_IP_DNAT & IP_IN_SNAT. > IN_IP_DNAT will modify the dst ip of the packet from floating ip to vm ip. > IN_IP_SNAT will modify the src ip of the packet from vm ip to floating ip. > > Rules in IN_IP_DNAT: > - Priority 100 rule to set the reg2 to 0x1 if dst & src networks are > connectected via a router and both the networks are private. > - Priority 90 rule to modify the dst ip from floating ip to vm ip. > - Priority 0 rule to go to next table. > > Rules in IN_IP_SNAT: > - Priority 100 rule to skip modifying the src ip when reg2 is set to 0x1 > - Priority 90 rule to modify the src ip from vm ip to floating ip and dst mac > to floating ip port mac if the packet is egressing via the gateway port > - Priority 50 rule to modify the src ip from vm ip to floating ip > - Priority 0 rule to go to next table. > > Priority 100 rules in IN_IP_DNAT and IN_IP_SNAT serves 2 purposes. >
Re: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support
It would really help if you give packet walkthrough when following 2 simultaneous connections happen at the same time. 1. east-west without NAT between the private IP addresses 2 east-west with floating IP. You also mentioned in the meeting (if I remember correctly) on how you have to keep a particular interface pinned to a hypervisor. A little description on that would help. Also, does this work with overlay networking? On 22 March 2016 at 14:19, Chandra S Vejendlawrote: > This patch adds distributed floating ip support for ovn. The assumption > made > here is that the external network is a single L2 broadcast domain and all > the > chassis have connectivity to the external network. > > 2 new tables are added in the LROUTER pipeline IN_IP_DNAT & IP_IN_SNAT. > IN_IP_DNAT will modify the dst ip of the packet from floating ip to vm ip. > IN_IP_SNAT will modify the src ip of the packet from vm ip to floating ip. > > Rules in IN_IP_DNAT: > - Priority 100 rule to set the reg2 to 0x1 if dst & src networks are > connectected via a router and both the networks are private. > - Priority 90 rule to modify the dst ip from floating ip to vm ip. > - Priority 0 rule to go to next table. > > Rules in IN_IP_SNAT: > - Priority 100 rule to skip modifying the src ip when reg2 is set to 0x1 > - Priority 90 rule to modify the src ip from vm ip to floating ip and dst > mac > to floating ip port mac if the packet is egressing via the gateway port > - Priority 50 rule to modify the src ip from vm ip to floating ip > - Priority 0 rule to go to next table. > > Priority 100 rules in IN_IP_DNAT and IN_IP_SNAT serves 2 purposes. > - Avoid NAT when vms in different LSWITCHES connected via a LROUTER talk to > each other using private ips. > - When 2 VMs connected to the same LSWITCH or different LSWITCHES connected > via a router try to talk to each other, the dst ip of the packet should > first be DNATed and then the src ip should be SNATed. > > The initial design was to stage DNAT in the ingress pipeline and the SNAT > in > the egress pipeline, but now both the stages are in the ingress pipeline. > This > was done to solve the cases highlighted above [Priority 100 rules]. There > is a > need to use information from DNAT stage when SNAT is being processed. This > would require an explicit register to be burnt to store the information. > > Flows modified in the LSWITCH pipeline > > Rules in IN_PORT_SEC: > - Priority 50 rule to allow packets ingressing the LSWITCH router port > with a src mac of floating ip port > > Rules in ARP_RSP: > - Priority 150 rule to respond to arp request for floating ip. To prevent > arp > responses for floating ip's from all the chassis, "lport" option is set > in > the external_id's column of the lflow table. lport will point to the > vif-id of > the vm that is associated with the floating ip. When ovn-controller is > processing the flows, if it sees an lport option set in the external_ids > column, it will install this lflow only if the lport is a local port on > the > chassis. > > Rules in L2_LKUP: > - Priority 50 rule to set the outport to the lrouter port when the dst mac > matches the floating ip mac > > Rules in OUT_PORT_SEC: > - Priority 50 rule to allow packet egressing the lrouter port with a mac > of a > floating ip port. > > Had to increase MAX_RESUBMIT_RECURSION from 64 to 96. When 2 VMs connected > via vm1->LS->LR->LS->LR->LS->vm2 are trying to talk to each other, the > resubmits are exceeding the existing 64 limit. > > When a floating ip is associated with a VM ip, NB will set the options of > the > floating ip lport to "fixed-ip-port=, router-port= the > logical router port". > > If you want to try out this patch with openstack, add the following patch > [1] > to networking-ovn. > > [1] https://review.openstack.org/#/c/295547/ > --- > ofproto/ofproto-dpif-xlate.c| 2 +- > ovn/controller/binding.c| 24 ++- > ovn/controller/binding.h| 4 +- > ovn/controller/lflow.c | 21 ++- > ovn/controller/lflow.h | 3 +- > ovn/controller/ovn-controller.c | 7 +- > ovn/northd/ovn-northd.c | 360 > +--- > 7 files changed, 378 insertions(+), 43 deletions(-) > > diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c > index 67504e8..4a5aae2 100644 > --- a/ofproto/ofproto-dpif-xlate.c > +++ b/ofproto/ofproto-dpif-xlate.c > @@ -68,7 +68,7 @@ VLOG_DEFINE_THIS_MODULE(ofproto_dpif_xlate); > > /* Maximum depth of flow table recursion (due to resubmit actions) in a > * flow translation. */ > -#define MAX_RESUBMIT_RECURSION 64 > +#define MAX_RESUBMIT_RECURSION 96 > #define MAX_INTERNAL_RESUBMITS 1 /* Max resbmits allowed using rules in >internal table. */ > > diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c > index d3ca9c9..f4e0f4a 100644 > --- a/ovn/controller/binding.c > +++
Re: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support
"dev" <dev-boun...@openvswitch.org> wrote on 03/22/2016 04:19:07 PM: > From: Chandra Sekhar Vejendla/San Jose/IBM@IBMUS > To: dev@openvswitch.org > Date: 03/22/2016 04:19 PM > Subject: [ovs-dev] [PATCH RFC] OVN: Openstack floating ip support > Sent by: "dev" <dev-boun...@openvswitch.org> > > This patch adds distributed floating ip support for ovn. The assumption made > here is that the external network is a single L2 broadcast domain and all the > chassis have connectivity to the external network. > > 2 new tables are added in the LROUTER pipeline IN_IP_DNAT & IP_IN_SNAT. > IN_IP_DNAT will modify the dst ip of the packet from floating ip to vm ip. > IN_IP_SNAT will modify the src ip of the packet from vm ip to floating ip. > > Rules in IN_IP_DNAT: > - Priority 100 rule to set the reg2 to 0x1 if dst & src networks are > connectected via a router and both the networks are private. > - Priority 90 rule to modify the dst ip from floating ip to vm ip. > - Priority 0 rule to go to next table. > > Rules in IN_IP_SNAT: > - Priority 100 rule to skip modifying the src ip when reg2 is set to 0x1 > - Priority 90 rule to modify the src ip from vm ip to floating ip and dst mac > to floating ip port mac if the packet is egressing via the gateway port > - Priority 50 rule to modify the src ip from vm ip to floating ip > - Priority 0 rule to go to next table. > > Priority 100 rules in IN_IP_DNAT and IN_IP_SNAT serves 2 purposes. > - Avoid NAT when vms in different LSWITCHES connected via a LROUTER talk to > each other using private ips. > - When 2 VMs connected to the same LSWITCH or different LSWITCHES connected > via a router try to talk to each other, the dst ip of the packet should > first be DNATed and then the src ip should be SNATed. > > The initial design was to stage DNAT in the ingress pipeline and the SNAT in > the egress pipeline, but now both the stages are in the ingress pipeline. This > was done to solve the cases highlighted above [Priority 100 rules]. There is a > need to use information from DNAT stage when SNAT is being processed. This > would require an explicit register to be burnt to store the information. > > Flows modified in the LSWITCH pipeline > > Rules in IN_PORT_SEC: > - Priority 50 rule to allow packets ingressing the LSWITCH router port > with a src mac of floating ip port > > Rules in ARP_RSP: > - Priority 150 rule to respond to arp request for floating ip. To prevent arp > responses for floating ip's from all the chassis, "lport" option is set in > the external_id's column of the lflow table. lport will point to > the vif-id of > the vm that is associated with the floating ip. When ovn-controller is > processing the flows, if it sees an lport option set in the external_ids > column, it will install this lflow only if the lport is a local port on the > chassis. > > Rules in L2_LKUP: > - Priority 50 rule to set the outport to the lrouter port when the dst mac > matches the floating ip mac > > Rules in OUT_PORT_SEC: > - Priority 50 rule to allow packet egressing the lrouter port with a mac of a > floating ip port. > > Had to increase MAX_RESUBMIT_RECURSION from 64 to 96. When 2 VMs connected > via vm1->LS->LR->LS->LR->LS->vm2 are trying to talk to each other, the > resubmits are exceeding the existing 64 limit. > > When a floating ip is associated with a VM ip, NB will set the options of the > floating ip lport to "fixed-ip-port=, router-port= logical router port". > > If you want to try out this patch with openstack, add the following patch [1] > to networking-ovn. > > [1] https://review.openstack.org/#/c/295547/ I know this is an RFC patch, but I'd really like to see two improvements w.r.t the documentation: 1. include the needed changes to ovn-northd.8.xml. 2. It looks like the DNAT code is also clearing the inport value and I didn't find the reason why anywhere - I'm pretty sure I know why, but confirmation would be good. Ryan Moats (regXboi) ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
[ovs-dev] [PATCH RFC] OVN: Openstack floating ip support
This patch adds distributed floating ip support for ovn. The assumption made here is that the external network is a single L2 broadcast domain and all the chassis have connectivity to the external network. 2 new tables are added in the LROUTER pipeline IN_IP_DNAT & IP_IN_SNAT. IN_IP_DNAT will modify the dst ip of the packet from floating ip to vm ip. IN_IP_SNAT will modify the src ip of the packet from vm ip to floating ip. Rules in IN_IP_DNAT: - Priority 100 rule to set the reg2 to 0x1 if dst & src networks are connectected via a router and both the networks are private. - Priority 90 rule to modify the dst ip from floating ip to vm ip. - Priority 0 rule to go to next table. Rules in IN_IP_SNAT: - Priority 100 rule to skip modifying the src ip when reg2 is set to 0x1 - Priority 90 rule to modify the src ip from vm ip to floating ip and dst mac to floating ip port mac if the packet is egressing via the gateway port - Priority 50 rule to modify the src ip from vm ip to floating ip - Priority 0 rule to go to next table. Priority 100 rules in IN_IP_DNAT and IN_IP_SNAT serves 2 purposes. - Avoid NAT when vms in different LSWITCHES connected via a LROUTER talk to each other using private ips. - When 2 VMs connected to the same LSWITCH or different LSWITCHES connected via a router try to talk to each other, the dst ip of the packet should first be DNATed and then the src ip should be SNATed. The initial design was to stage DNAT in the ingress pipeline and the SNAT in the egress pipeline, but now both the stages are in the ingress pipeline. This was done to solve the cases highlighted above [Priority 100 rules]. There is a need to use information from DNAT stage when SNAT is being processed. This would require an explicit register to be burnt to store the information. Flows modified in the LSWITCH pipeline Rules in IN_PORT_SEC: - Priority 50 rule to allow packets ingressing the LSWITCH router port with a src mac of floating ip port Rules in ARP_RSP: - Priority 150 rule to respond to arp request for floating ip. To prevent arp responses for floating ip's from all the chassis, "lport" option is set in the external_id's column of the lflow table. lport will point to the vif-id of the vm that is associated with the floating ip. When ovn-controller is processing the flows, if it sees an lport option set in the external_ids column, it will install this lflow only if the lport is a local port on the chassis. Rules in L2_LKUP: - Priority 50 rule to set the outport to the lrouter port when the dst mac matches the floating ip mac Rules in OUT_PORT_SEC: - Priority 50 rule to allow packet egressing the lrouter port with a mac of a floating ip port. Had to increase MAX_RESUBMIT_RECURSION from 64 to 96. When 2 VMs connected via vm1->LS->LR->LS->LR->LS->vm2 are trying to talk to each other, the resubmits are exceeding the existing 64 limit. When a floating ip is associated with a VM ip, NB will set the options of the floating ip lport to "fixed-ip-port=, router-port=https://review.openstack.org/#/c/295547/ --- ofproto/ofproto-dpif-xlate.c| 2 +- ovn/controller/binding.c| 24 ++- ovn/controller/binding.h| 4 +- ovn/controller/lflow.c | 21 ++- ovn/controller/lflow.h | 3 +- ovn/controller/ovn-controller.c | 7 +- ovn/northd/ovn-northd.c | 360 +--- 7 files changed, 378 insertions(+), 43 deletions(-) diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c index 67504e8..4a5aae2 100644 --- a/ofproto/ofproto-dpif-xlate.c +++ b/ofproto/ofproto-dpif-xlate.c @@ -68,7 +68,7 @@ VLOG_DEFINE_THIS_MODULE(ofproto_dpif_xlate); /* Maximum depth of flow table recursion (due to resubmit actions) in a * flow translation. */ -#define MAX_RESUBMIT_RECURSION 64 +#define MAX_RESUBMIT_RECURSION 96 #define MAX_INTERNAL_RESUBMITS 1 /* Max resbmits allowed using rules in internal table. */ diff --git a/ovn/controller/binding.c b/ovn/controller/binding.c index d3ca9c9..f4e0f4a 100644 --- a/ovn/controller/binding.c +++ b/ovn/controller/binding.c @@ -49,7 +49,7 @@ binding_register_ovs_idl(struct ovsdb_idl *ovs_idl) _interface_col_ingress_policing_burst); } -static void +void get_local_iface_ids(const struct ovsrec_bridge *br_int, struct shash *lports) { int i; @@ -149,7 +149,8 @@ update_qos(const struct ovsrec_interface *iface_rec, void binding_run(struct controller_ctx *ctx, const struct ovsrec_bridge *br_int, const char *chassis_id, struct simap *ct_zones, -unsigned long *ct_zone_bitmap, struct hmap *local_datapaths) +unsigned long *ct_zone_bitmap, struct hmap *local_datapaths, +struct sset *all_lports) { const struct sbrec_chassis *chassis_rec; const struct sbrec_port_binding *binding_rec; @@ -167,10 +168,9 @@ binding_run(struct controller_ctx *ctx, const