Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Mickey Spiegel]

Forgot to copy the list on the last reply, but also realized something and
asking for one change below.

On Fri, Nov 4, 2016 at 9:54 AM, Guru Shetty  wrote:

>
>
> On 3 November 2016 at 20:42, Mickey Spiegel  wrote:
>
>> On Thu, Nov 3, 2016 at 6:06 PM, Guru Shetty  wrote:
>>
>> 
>>
>>
>>> > 2. If a stateful action such as DNAT or LB is taken on a
>>> >   gateway router, such that it is necessary for the reverse
>>> >   packet flow to come back to the same gateway router,
>>> >   then there should be an SNAT action coupled with the
>>> >   DNAT or LB action. If we could implement such composite
>>> >   actions, then there would be no need for a general purpose
>>> >   SNAT 0.0.0.0/0 catch all. Perhaps instead of SNAT
>>> >   0.0.0.0/0, DNAT or LB could set a flag that indicates that
>>> >   SNAT should be applied?
>>>
>>> That is one option. But I could not think of a nice way to express it in
>>> nb database. The other option is to set it as an option in the gateway
>>> router itself.
>>
>>
>> Coming back to this option because IMO it is the most straightforward in
>> terms of understanding what the knobs do, even if figuring out what knobs
>> to provide is not so straightforward.
>>
>> For the trigger, I am somewhat torn between putting it in the DNAT and LB
>> rules, versus a gateway router option. I can think of cases where it is
>> driven by topology (your multiple gateway case), as well as cases where it
>> is driven by application (whether direct server return works with the
>> application or not). If the trigger goes in the DNAT and LB rules, then a
>> force_snat column of type boolean in the NAT and Load_Balancer tables would
>> take care of the trigger.
>>
>> The part that is not as obvious is where to specify the IP address that
>> replaces the packet's source IP address. I would lean towards specifying
>> this separately from the existing NAT table, partly because the terminology
>> of the existing NAT table (logical_ip/external_ip) does not match the use
>> case, and partly because my somewhat hazy memory of hardware load balancer
>> configuration involves specification of a reference to a NAT pool, i.e. it
>> is not a separate SNAT rule specification but just the extra bit of
>> specification necessary to make the SNAT part work. If my memory is right,
>> that involved a relatively complex reference from the DNAT or LB rule to a
>> NAT pool. The equivalent of your SNAT 0.0.0.0/0 rule would be just a
>> single SNAT IP address on the gateway router for all force_snat cases,
>> rather than a reference to one of many IP addresses or IP address pools.
>>
>> As long as the trigger is only needed for DNAT and LB in the gateway
>> router itself, a force_snat flag should work. If you want a trigger for
>> other cases such as stateful services in a switch upstream from the gateway
>> router, at first glance it does not seem like this option can handle such a
>> case. You would need interface related configuration in order to identify
>> the subset of traffic, and some pipeline stage with conntrack so that you
>> could weed out ct.rpl traffic.
>>
>
> Thanks. I think the trigger is only needed for LB and DNAT use cases. I am
> thinking of going with the following approach:
>
> * options:force-snat="$IP" in the gateway router.
>
> Whenever there is a DNAT or LB done on the gateway router, you get a force
> snat done with the set IP address.  If application specific use cases come
> up as we go forward, then it is a matter of providing the same knob in LB
> and NAT tables too - which can be designed better when the use cases are
> more obvious.
>

That works for me. A bit of explanation is required with regard to which
traffic is subject to force-snat, but all in all I think this is easy to
understand and avoids complex side effects.

I just realized that this is the missing piece that I need to allow
centralized load balancing on a distributed router. In that case, I need
force-snat for load balancing but not for dnat. Wondering if there should
be two options instead:
dnat-force-snat="$IP"
lb-force-snat="$IP"

That also takes care of the explanation.

Mickey
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Guru Shetty]

On 3 November 2016 at 20:42, Mickey Spiegel  wrote:

> On Thu, Nov 3, 2016 at 6:06 PM, Guru Shetty  wrote:
>
> 
>
>
>> > 2. If a stateful action such as DNAT or LB is taken on a
>> >   gateway router, such that it is necessary for the reverse
>> >   packet flow to come back to the same gateway router,
>> >   then there should be an SNAT action coupled with the
>> >   DNAT or LB action. If we could implement such composite
>> >   actions, then there would be no need for a general purpose
>> >   SNAT 0.0.0.0/0 catch all. Perhaps instead of SNAT
>> >   0.0.0.0/0, DNAT or LB could set a flag that indicates that
>> >   SNAT should be applied?
>>
>> That is one option. But I could not think of a nice way to express it in
>> nb database. The other option is to set it as an option in the gateway
>> router itself.
>
>
> Coming back to this option because IMO it is the most straightforward in
> terms of understanding what the knobs do, even if figuring out what knobs
> to provide is not so straightforward.
>
> For the trigger, I am somewhat torn between putting it in the DNAT and LB
> rules, versus a gateway router option. I can think of cases where it is
> driven by topology (your multiple gateway case), as well as cases where it
> is driven by application (whether direct server return works with the
> application or not). If the trigger goes in the DNAT and LB rules, then a
> force_snat column of type boolean in the NAT and Load_Balancer tables would
> take care of the trigger.
>
> The part that is not as obvious is where to specify the IP address that
> replaces the packet's source IP address. I would lean towards specifying
> this separately from the existing NAT table, partly because the terminology
> of the existing NAT table (logical_ip/external_ip) does not match the use
> case, and partly because my somewhat hazy memory of hardware load balancer
> configuration involves specification of a reference to a NAT pool, i.e. it
> is not a separate SNAT rule specification but just the extra bit of
> specification necessary to make the SNAT part work. If my memory is right,
> that involved a relatively complex reference from the DNAT or LB rule to a
> NAT pool. The equivalent of your SNAT 0.0.0.0/0 rule would be just a
> single SNAT IP address on the gateway router for all force_snat cases,
> rather than a reference to one of many IP addresses or IP address pools.
>
> As long as the trigger is only needed for DNAT and LB in the gateway
> router itself, a force_snat flag should work. If you want a trigger for
> other cases such as stateful services in a switch upstream from the gateway
> router, at first glance it does not seem like this option can handle such a
> case. You would need interface related configuration in order to identify
> the subset of traffic, and some pipeline stage with conntrack so that you
> could weed out ct.rpl traffic.
>

Thanks. I think the trigger is only needed for LB and DNAT use cases. I am
thinking of going with the following approach:

* options:force-snat="$IP" in the gateway router.

Whenever there is a DNAT or LB done on the gateway router, you get a force
snat done with the set IP address.  If application specific use cases come
up as we go forward, then it is a matter of providing the same knob in LB
and NAT tables too - which can be designed better when the use cases are
more obvious.




> Mickey
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Mickey Spiegel]

On Thu, Nov 3, 2016 at 6:06 PM, Guru Shetty  wrote:




> > 2. If a stateful action such as DNAT or LB is taken on a
> >   gateway router, such that it is necessary for the reverse
> >   packet flow to come back to the same gateway router,
> >   then there should be an SNAT action coupled with the
> >   DNAT or LB action. If we could implement such composite
> >   actions, then there would be no need for a general purpose
> >   SNAT 0.0.0.0/0 catch all. Perhaps instead of SNAT
> >   0.0.0.0/0, DNAT or LB could set a flag that indicates that
> >   SNAT should be applied?
>
> That is one option. But I could not think of a nice way to express it in
> nb database. The other option is to set it as an option in the gateway
> router itself.


Coming back to this option because IMO it is the most straightforward in
terms of understanding what the knobs do, even if figuring out what knobs
to provide is not so straightforward.

For the trigger, I am somewhat torn between putting it in the DNAT and LB
rules, versus a gateway router option. I can think of cases where it is
driven by topology (your multiple gateway case), as well as cases where it
is driven by application (whether direct server return works with the
application or not). If the trigger goes in the DNAT and LB rules, then a
force_snat column of type boolean in the NAT and Load_Balancer tables would
take care of the trigger.

The part that is not as obvious is where to specify the IP address that
replaces the packet's source IP address. I would lean towards specifying
this separately from the existing NAT table, partly because the terminology
of the existing NAT table (logical_ip/external_ip) does not match the use
case, and partly because my somewhat hazy memory of hardware load balancer
configuration involves specification of a reference to a NAT pool, i.e. it
is not a separate SNAT rule specification but just the extra bit of
specification necessary to make the SNAT part work. If my memory is right,
that involved a relatively complex reference from the DNAT or LB rule to a
NAT pool. The equivalent of your SNAT 0.0.0.0/0 rule would be just a single
SNAT IP address on the gateway router for all force_snat cases, rather than
a reference to one of many IP addresses or IP address pools.

As long as the trigger is only needed for DNAT and LB in the gateway router
itself, a force_snat flag should work. If you want a trigger for other
cases such as stateful services in a switch upstream from the gateway
router, at first glance it does not seem like this option can handle such a
case. You would need interface related configuration in order to identify
the subset of traffic, and some pipeline stage with conntrack so that you
could weed out ct.rpl traffic.

Mickey
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Guru Shetty]

> On Nov 3, 2016, at 7:16 PM, Mickey Spiegel  wrote:
> 
> See reply at the bottom.
> 
>> On Thu, Nov 3, 2016 at 6:06 PM, Guru Shetty  wrote:
>> 
>> 
>> 
>> > It seems to me that the root of the problem has to do with
>> > three issues:
>> > 1. SNAT (and DNAT) rules should not apply to ct.rpl traffic,
>> >   instead only UNSNAT (and UNDNAT) rules should apply.
>> >   Conntrack can tell which traffic is reply traffic, but this would
>> >   require going through conntrack with recirc prior to SNAT.
>> 
>> Correct
>> 
>> 
>  
>> > And to do that in the pipeline, we check whether a packet
>> >> has already been SNATted and if it has a transformation, we should not
>> >> do it again.
>> >
>> > I think this is a roundabout way of implementing 1 above. If all
>> > gateway router traffic is subject to SNAT, then reply traffic will
>> > always hit UNSNAT and so avoid the SNAT stage.
>> 
>> Right.
>> 
>> >
>> > The question is whether a solution that requires all traffic to
>> > be subject to SNAT is acceptable, for deployment scenarios
>> > where SNAT rules have coarse enough IP address prefixes
>> > to catch traffic in both directions?
>> 
>> I did not understand the above point. Can you elaborate a bit.
> 
> This approach imposes symmetry in terms of which traffic is
> subject to SNAT, when an SNAT rule is coarse enough to match
> source IP addresses in both directions, for example 0.0.0.0/0.
> If you wanted SNAT imposed for 0.0.0.0/0 in the inward direction,
> then you will have SNAT imposed for 0.0.0.0/0 in the outward
> direction, with no exceptions except for SNAT longest match
> overrides. I guess that is why you suggested the 'nosnat'
> second patch?

Thanks for the explanation. You are right about the situation. Do you have any 
design suggestions to solve this overall problem in a easy way?

> 
> Mickey
> 
>> 
>> >
>> > Mickey
>> > ___
>> > dev mailing list
>> > dev@openvswitch.org
>> > http://openvswitch.org/mailman/listinfo/dev
> 
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Mickey Spiegel]

See reply at the bottom.

On Thu, Nov 3, 2016 at 6:06 PM, Guru Shetty  wrote:



> It seems to me that the root of the problem has to do with
> > three issues:
> > 1. SNAT (and DNAT) rules should not apply to ct.rpl traffic,
> >   instead only UNSNAT (and UNDNAT) rules should apply.
> >   Conntrack can tell which traffic is reply traffic, but this would
> >   require going through conntrack with recirc prior to SNAT.
>
> Correct
>
> 
>


> > And to do that in the pipeline, we check whether a packet
> >> has already been SNATted and if it has a transformation, we should not
> >> do it again.
> >
> > I think this is a roundabout way of implementing 1 above. If all
> > gateway router traffic is subject to SNAT, then reply traffic will
> > always hit UNSNAT and so avoid the SNAT stage.
>
> Right.
>
> >
> > The question is whether a solution that requires all traffic to
> > be subject to SNAT is acceptable, for deployment scenarios
> > where SNAT rules have coarse enough IP address prefixes
> > to catch traffic in both directions?
>
> I did not understand the above point. Can you elaborate a bit.
>

This approach imposes symmetry in terms of which traffic is
subject to SNAT, when an SNAT rule is coarse enough to match
source IP addresses in both directions, for example 0.0.0.0/0.
If you wanted SNAT imposed for 0.0.0.0/0 in the inward direction,
then you will have SNAT imposed for 0.0.0.0/0 in the outward
direction, with no exceptions except for SNAT longest match
overrides. I guess that is why you suggested the 'nosnat'
second patch?

Mickey


> >
> > Mickey
> > ___
> > dev mailing list
> > dev@openvswitch.org
> > http://openvswitch.org/mailman/listinfo/dev
>
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Guru Shetty]

> On Nov 3, 2016, at 4:34 PM, Mickey Spiegel  wrote:
> 
> Interesting problem. See comments inline.
> 
>> On Thu, Nov 3, 2016 at 3:46 AM, Gurucharan Shetty  wrote:
>> 
>> When multiple gateway routers exist, a packet can
>> enter any gateway router. Once the packet reaches its
>> destination, its reverse direction should be via the
>> same gateway router.  This is achieved by doing a SNAT
>> of the packet in the inward direction (towards logical space)
>> with a IP address of the gateway router such that packet travels back
>> to the same gateway router.
>> 
>> The above means that you can have SNAT rules in the NB
>> database for both directions.  For e.g. if the logical
>> ip address are of the range 192.168.1.0/24, you will have
>> one SNAT rule to transform packet from 192.168.1.0/24 to
>> an external_ip and another SNAT rule to transform
>> "0.0.0.0/0" (all external initiated traffic) to a gateway_ip.
>> 
>> For a particular connection, we should do SNAT in only one
>> direction.
> 
> 
> In certain deployment scenarios, including the one you have
> mentioned, this might work. However, I wonder if this is really
> addressing the root of the problem, or latching on to a symptom
> specific to this deployment scenario.
> 
> It seems to me that the root of the problem has to do with
> three issues:
> 1. SNAT (and DNAT) rules should not apply to ct.rpl traffic,
>   instead only UNSNAT (and UNDNAT) rules should apply.
>   Conntrack can tell which traffic is reply traffic, but this would
>   require going through conntrack with recirc prior to SNAT.

Correct

> 2. If a stateful action such as DNAT or LB is taken on a
>   gateway router, such that it is necessary for the reverse
>   packet flow to come back to the same gateway router,
>   then there should be an SNAT action coupled with the
>   DNAT or LB action. If we could implement such composite
>   actions, then there would be no need for a general purpose
>   SNAT 0.0.0.0/0 catch all. Perhaps instead of SNAT
>   0.0.0.0/0, DNAT or LB could set a flag that indicates that
>   SNAT should be applied?

That is one option. But I could not think of a nice way to express it in nb 
database. The other option is to set it as an option in the gateway router 
itself.

> 3. Currently your gateway router has no sense of direction.
>   NAT rules are applied on all interfaces. In the scenario
>   described above, you really want the 192.168.1.0/24 SNAT
>   rule to apply to outward connections, while you want the
>   0.0.0.0/0 SNAT rule to apply to inward connections. Since
>   the gateway router has no sense of direction, both rules
>   are applied in both directions. Is there a way to introduce
>   a sense of direction to a gateway router?

It complicates deployment a bit. But can be given more thought.

> 
> And to do that in the pipeline, we check whether a packet
>> has already been SNATted and if it has a transformation, we should not
>> do it again.
> 
> I think this is a roundabout way of implementing 1 above. If all
> gateway router traffic is subject to SNAT, then reply traffic will
> always hit UNSNAT and so avoid the SNAT stage.

Right.

> 
> The question is whether a solution that requires all traffic to
> be subject to SNAT is acceptable, for deployment scenarios
> where SNAT rules have coarse enough IP address prefixes
> to catch traffic in both directions?

I did not understand the above point. Can you elaborate a bit.

> 
> Mickey
> ___
> dev mailing list
> dev@openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: [ovs-dev] [PATCH 3/3] ovn-northd: SNAT in either direction of gateway router.

[Mickey Spiegel]

Interesting problem. See comments inline.

On Thu, Nov 3, 2016 at 3:46 AM, Gurucharan Shetty  wrote:

> When multiple gateway routers exist, a packet can
> enter any gateway router. Once the packet reaches its
> destination, its reverse direction should be via the
> same gateway router.  This is achieved by doing a SNAT
> of the packet in the inward direction (towards logical space)
> with a IP address of the gateway router such that packet travels back
> to the same gateway router.
>
> The above means that you can have SNAT rules in the NB
> database for both directions.  For e.g. if the logical
> ip address are of the range 192.168.1.0/24, you will have
> one SNAT rule to transform packet from 192.168.1.0/24 to
> an external_ip and another SNAT rule to transform
> "0.0.0.0/0" (all external initiated traffic) to a gateway_ip.
>
> For a particular connection, we should do SNAT in only one
> direction.


In certain deployment scenarios, including the one you have
mentioned, this might work. However, I wonder if this is really
addressing the root of the problem, or latching on to a symptom
specific to this deployment scenario.

It seems to me that the root of the problem has to do with
three issues:
1. SNAT (and DNAT) rules should not apply to ct.rpl traffic,
   instead only UNSNAT (and UNDNAT) rules should apply.
   Conntrack can tell which traffic is reply traffic, but this would
   require going through conntrack with recirc prior to SNAT.
2. If a stateful action such as DNAT or LB is taken on a
   gateway router, such that it is necessary for the reverse
   packet flow to come back to the same gateway router,
   then there should be an SNAT action coupled with the
   DNAT or LB action. If we could implement such composite
   actions, then there would be no need for a general purpose
   SNAT 0.0.0.0/0 catch all. Perhaps instead of SNAT
   0.0.0.0/0, DNAT or LB could set a flag that indicates that
   SNAT should be applied?
3. Currently your gateway router has no sense of direction.
   NAT rules are applied on all interfaces. In the scenario
   described above, you really want the 192.168.1.0/24 SNAT
   rule to apply to outward connections, while you want the
   0.0.0.0/0 SNAT rule to apply to inward connections. Since
   the gateway router has no sense of direction, both rules
   are applied in both directions. Is there a way to introduce
   a sense of direction to a gateway router?

And to do that in the pipeline, we check whether a packet
> has already been SNATted and if it has a transformation, we should not
> do it again.
>

I think this is a roundabout way of implementing 1 above. If all
gateway router traffic is subject to SNAT, then reply traffic will
always hit UNSNAT and so avoid the SNAT stage.

The question is whether a solution that requires all traffic to
be subject to SNAT is acceptable, for deployment scenarios
where SNAT rules have coarse enough IP address prefixes
to catch traffic in both directions?

Mickey
___
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev