Re: [DISCUSS] Why we MARK packets?

[Ron Wheeler]

https://markandruth.co.uk/2016/08/08/testing-the-performance-of-the-linux-firewall
Does not directly address marking but does benchmark a number of 
iptables filtering tasks which may give some insight into the 
performance implications of using iptables for routing and filtering.


https://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture
A nice article on iptables. No performance info but a good read for 
anyone who needs to tackle routing and marking.


 https://www.markusschade.com/papers/firewall.pdf
Has some performance info and a discussion about the host limit with MARK.

I am a bit skeptical that MARK adds a lot of overhead but this is only 
based on the belief that CPUs are orders of magnitude faster than networks.
The decision process on entry and exit are both pretty simple and 
depending on the implementation of the internal table should not take a 
lot of memory of CPU to do the table lookups


I did not read all 60,400 links returned by Google but in the ones that 
I did read did not warn of any big problem with performance using MARK.


I use a simple static MARK configuration in my firewall to solve a 
routing problem but the performance requirements are not relevant to an 
datacentre with hundreds of CPUs and hundreds of entries in the MARK 
routing map.


I hope that this helps.

Ron


On 20/04/2018 9:04 AM, Rohit Yadav wrote:

Thanks Jayapal. I don't have any comparative study yet, but I'll explore this 
in future if we can get away without marking (mangling) packets which is 
generally an expensive task.


- Rohit

<https://cloudstack.apache.org>




From: Jayapal Uradi <jayapal.ur...@accelerite.com>
Sent: Thursday, April 19, 2018 10:33:25 AM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: [DISCUSS] Why we MARK packets?

Rohit,

My comments inline.


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
   
  


On Apr 19, 2018, at 1:52 AM, Rohit Yadav 
<rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>> wrote:

Nevermind, found the use of custom routing tables. In case someone want to 
refer, hints are here:

https://github.com/apache/cloudstack/pull/2514#issuecomment-382510915


Jayapal and others - I've another one, is there a way to do routing without 
marking packets at all, even in case of VRs with additional public interfaces?

AFAIK marking is the only way to do it.
Do you have any performance numbers with and without mark rules.

- Rohit

<https://cloudstack.apache.org<https://cloudstack.apache.org/>>




From: Rohit Yadav <rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>>
Sent: Wednesday, April 18, 2018 10:39:02 PM
To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>; 
us...@cloudstack.apache.org<mailto:us...@cloudstack.apache.org>
Subject: [DISCUSS] Why we MARK packets?

All,


I could not find any history around 'why' we MARK or CONNMARK packets in mangle 
table in VRs? I found an issue in case of VPCs where `MARK` iptable rules 
failed hair-pin nat (as described in this PR: 
https://github.com/apache/cloudstack/pull/2514)


The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
at all, it is commented:

https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141


Other than for debugging purposes in the VR, marking packets and connections I 
could not find any valid use. Please do share if you're using marked packets 
(such as VPN ones etc) outside of VR scope?


I propose we remove MARK on packets which is cpu intensive and slows the 
traffic (a bit), instead CONNMARK can still be used to mark connections and 
debug VRs without actually changing the packet marking permanently. Thoughts?


- Rohit

<https://cloudstack.apache.org>



rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com/><http://www.shapeblue.com<http://www.shapeblue.com/>>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com/>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infecte

Re: [DISCUSS] Why we MARK packets?

[Rohit Yadav]

Thanks Jayapal. I don't have any comparative study yet, but I'll explore this 
in future if we can get away without marking (mangling) packets which is 
generally an expensive task.


- Rohit

<https://cloudstack.apache.org>




From: Jayapal Uradi <jayapal.ur...@accelerite.com>
Sent: Thursday, April 19, 2018 10:33:25 AM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: [DISCUSS] Why we MARK packets?

Rohit,

My comments inline.


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

On Apr 19, 2018, at 1:52 AM, Rohit Yadav 
<rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>> wrote:

Nevermind, found the use of custom routing tables. In case someone want to 
refer, hints are here:

https://github.com/apache/cloudstack/pull/2514#issuecomment-382510915


Jayapal and others - I've another one, is there a way to do routing without 
marking packets at all, even in case of VRs with additional public interfaces?

AFAIK marking is the only way to do it.
Do you have any performance numbers with and without mark rules.

- Rohit

<https://cloudstack.apache.org<https://cloudstack.apache.org/>>




From: Rohit Yadav <rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>>
Sent: Wednesday, April 18, 2018 10:39:02 PM
To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>; 
us...@cloudstack.apache.org<mailto:us...@cloudstack.apache.org>
Subject: [DISCUSS] Why we MARK packets?

All,


I could not find any history around 'why' we MARK or CONNMARK packets in mangle 
table in VRs? I found an issue in case of VPCs where `MARK` iptable rules 
failed hair-pin nat (as described in this PR: 
https://github.com/apache/cloudstack/pull/2514)


The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
at all, it is commented:

https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141


Other than for debugging purposes in the VR, marking packets and connections I 
could not find any valid use. Please do share if you're using marked packets 
(such as VPN ones etc) outside of VR scope?


I propose we remove MARK on packets which is cpu intensive and slows the 
traffic (a bit), instead CONNMARK can still be used to mark connections and 
debug VRs without actually changing the packet marking permanently. Thoughts?


- Rohit

<https://cloudstack.apache.org>



rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com/><http://www.shapeblue.com<http://www.shapeblue.com/>>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com/>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: [DISCUSS] Why we MARK packets?

[Jayapal Uradi]

Rohit,

My comments inline.

On Apr 19, 2018, at 1:52 AM, Rohit Yadav 
<rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>> wrote:

Nevermind, found the use of custom routing tables. In case someone want to 
refer, hints are here:

https://github.com/apache/cloudstack/pull/2514#issuecomment-382510915


Jayapal and others - I've another one, is there a way to do routing without 
marking packets at all, even in case of VRs with additional public interfaces?

AFAIK marking is the only way to do it.
Do you have any performance numbers with and without mark rules.

- Rohit

<https://cloudstack.apache.org<https://cloudstack.apache.org/>>




From: Rohit Yadav <rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>>
Sent: Wednesday, April 18, 2018 10:39:02 PM
To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>; 
us...@cloudstack.apache.org<mailto:us...@cloudstack.apache.org>
Subject: [DISCUSS] Why we MARK packets?

All,


I could not find any history around 'why' we MARK or CONNMARK packets in mangle 
table in VRs? I found an issue in case of VPCs where `MARK` iptable rules 
failed hair-pin nat (as described in this PR: 
https://github.com/apache/cloudstack/pull/2514)


The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
at all, it is commented:

https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141


Other than for debugging purposes in the VR, marking packets and connections I 
could not find any valid use. Please do share if you're using marked packets 
(such as VPN ones etc) outside of VR scope?


I propose we remove MARK on packets which is cpu intensive and slows the 
traffic (a bit), instead CONNMARK can still be used to mark connections and 
debug VRs without actually changing the packet marking permanently. Thoughts?


- Rohit

<https://cloudstack.apache.org>



rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com/><http://www.shapeblue.com<http://www.shapeblue.com/>>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com/>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: [DISCUSS] Why we MARK packets?

[Rohit Yadav]

Nevermind, found the use of custom routing tables. In case someone want to 
refer, hints are here:

https://github.com/apache/cloudstack/pull/2514#issuecomment-382510915


Jayapal and others - I've another one, is there a way to do routing without 
marking packets at all, even in case of VRs with additional public interfaces?


- Rohit

<https://cloudstack.apache.org>




From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: Wednesday, April 18, 2018 10:39:02 PM
To: dev@cloudstack.apache.org; us...@cloudstack.apache.org
Subject: [DISCUSS] Why we MARK packets?

All,


I could not find any history around 'why' we MARK or CONNMARK packets in mangle 
table in VRs? I found an issue in case of VPCs where `MARK` iptable rules 
failed hair-pin nat (as described in this PR: 
https://github.com/apache/cloudstack/pull/2514)


The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
at all, it is commented:

https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141


Other than for debugging purposes in the VR, marking packets and connections I 
could not find any valid use. Please do share if you're using marked packets 
(such as VPN ones etc) outside of VR scope?


I propose we remove MARK on packets which is cpu intensive and slows the 
traffic (a bit), instead CONNMARK can still be used to mark connections and 
debug VRs without actually changing the packet marking permanently. Thoughts?


- Rohit

<https://cloudstack.apache.org>



rohit.ya...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [DISCUSS] Why we MARK packets?

[Rohit Yadav]

Hi Jayapal,


Thanks for replying. Can you point me to the code/rules that use the marking 
for packet routing (in case of additional public nics) in latest 4.11 or master 
branch? Don't we have routing tables and nat rules for routing across 
interfaces?


This started from reviewing Rafael's PR, whose fix I could not validate:

https://github.com/apache/cloudstack/pull/2514


I tested and found in my env that removing MARK rules in mangle table fixed 
access related issues for me, based on which I opened the PR:

https://github.com/apache/cloudstack/pull/2579


In case of VPC VRs and isolated network VRs, the nic number/order is different 
whose nic number/id is used to mark packets. I found that networking always 
failed when packets were marked with 0x1, but worked when 0x2 was used. I don't 
have an explaination for this. My test env was 4.11 based.


- Rohit

<https://cloudstack.apache.org>




From: Jayapal Uradi <jayapal.ur...@accelerite.com>
Sent: Wednesday, April 18, 2018 10:50:27 PM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: [DISCUSS] Why we MARK packets?

Hi,

Below are the uses of marking packets.

1. Marking is required to route the packets into correct interface in case 
additional public interfaces in VR.
2. Packets with VPN marking are accepted in first place of NAT POSTROUTING. 
Without marking these packets source ip will be replaced with source-nat IP.

Thanks,
Jayapal



rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

> On Apr 18, 2018, at 10:39 PM, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:
>
> All,
>
>
> I could not find any history around 'why' we MARK or CONNMARK packets in 
> mangle table in VRs? I found an issue in case of VPCs where `MARK` iptable 
> rules failed hair-pin nat (as described in this PR: 
> https://github.com/apache/cloudstack/pull/2514)
>
>
> The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
> at all, it is commented:
>
> https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141
>
>
> Other than for debugging purposes in the VR, marking packets and connections 
> I could not find any valid use. Please do share if you're using marked 
> packets (such as VPN ones etc) outside of VR scope?
>
>
> I propose we remove MARK on packets which is cpu intensive and slows the 
> traffic (a bit), instead CONNMARK can still be used to mark connections and 
> debug VRs without actually changing the packet marking permanently. Thoughts?
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: [DISCUSS] Why we MARK packets?

[Jayapal Uradi]

Hi,

Below are the uses of marking packets.

1. Marking is required to route the packets into correct interface in case 
additional public interfaces in VR.
2. Packets with VPN marking are accepted in first place of NAT POSTROUTING. 
Without marking these packets source ip will be replaced with source-nat IP.

Thanks,
Jayapal


> On Apr 18, 2018, at 10:39 PM, Rohit Yadav  wrote:
> 
> All,
> 
> 
> I could not find any history around 'why' we MARK or CONNMARK packets in 
> mangle table in VRs? I found an issue in case of VPCs where `MARK` iptable 
> rules failed hair-pin nat (as described in this PR: 
> https://github.com/apache/cloudstack/pull/2514)
> 
> 
> The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
> at all, it is commented:
> 
> https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141
> 
> 
> Other than for debugging purposes in the VR, marking packets and connections 
> I could not find any valid use. Please do share if you're using marked 
> packets (such as VPN ones etc) outside of VR scope?
> 
> 
> I propose we remove MARK on packets which is cpu intensive and slows the 
> traffic (a bit), instead CONNMARK can still be used to mark connections and 
> debug VRs without actually changing the packet marking permanently. Thoughts?
> 
> 
> - Rohit
> 
> 
> 
> 
> 
> rohit.ya...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
> 
> 
> 

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

[DISCUSS] Why we MARK packets?

[Rohit Yadav]

All,


I could not find any history around 'why' we MARK or CONNMARK packets in mangle 
table in VRs? I found an issue in case of VPCs where `MARK` iptable rules 
failed hair-pin nat (as described in this PR: 
https://github.com/apache/cloudstack/pull/2514)


The valid usage I found was wrt VPN_STATS, however, the usage is not exported 
at all, it is commented:

https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/vpc_netusage.sh#L141


Other than for debugging purposes in the VR, marking packets and connections I 
could not find any valid use. Please do share if you're using marked packets 
(such as VPN ones etc) outside of VR scope?


I propose we remove MARK on packets which is cpu intensive and slows the 
traffic (a bit), instead CONNMARK can still be used to mark connections and 
debug VRs without actually changing the packet marking permanently. Thoughts?


- Rohit





rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue