Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

[Jaime Caamaño Ruiz]

> Agree. I will try to address this issue. I think we can have a
> separate
> run time/log directory for OVN. ovn-controller needs to talk to the
> local ovsdb-server
> and br-int.mgmt and other related socket interfaces, so it needs
> access
> to
> the /var/run/openvswitch/ folder. I think we can solve this.

I have a (yet to be submitted patch) on my side that changes OVN to run
as the same user as OVS. Thought it as less disruptive than also
changing the log directory.

openvswitch-ipsec still logs as root though and I dont see a way around
that so it probably needs to log to a different directory.

Jaime.

-Original Message-
From: Numan Siddique 
To: Aaron Conole 
Cc: Jaime Caamaño Ruiz , ovs-discuss 
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Thu, 2 May 2019 19:20:28 +0530




On Mon, Apr 29, 2019, 9:36 PM Aaron Conole  wrote:
> Jaime Caamaño Ruiz  writes:
> 
> >> As a "security concern" you mean something among the lines where
> one
> >> of ovs-* processes running under openvswitch user would go ahead
> and
> >> create a file with its owner that later one of ovn processes would
> >> blindly reuse without checking that it actually belongs to
> root:root?
> >
> > One example is that the OVS user could create link as any OVN log
> file
> > to other file owned by root and then OVN would write to that file.
> 
> There's a long-standing issue of OVN running as root.  I don't see
> any
> reason it should.

Agree. I will try to address this issue. I think we can have a separate
run time/log directory for OVN. ovn-controller needs to talk to the
local ovsdb-server
and br-int.mgmt and other related socket interfaces, so it needs access
to
the /var/run/openvswitch/ folder. I think we can solve this.

Thanks
Numan

> >> Can you give a more concrete example? I believe logrotate is
> running
> >> under root and should be able to rotate everything?
> >
> > The logrotate configuration for openvswitch logs has a 'su'
> directive
> > to run under the openvswitch user, precisely to prevent something
> > similar to the above. So it wont be able to rotate root owned logs
> in
> > the openvswitch directory. How it fails precisely depends on global
> > logrotate configuration. For example it will fail to create the new
> log
> > file after rotation if the 'create' directive is used. Or it may
> fail
> > to compress the rotated log file as it wont be able to read it.
> >
> > BR
> > Jaime
> >
> > -Original Message-
> > From: Ansis Atteka 
> > To: jcaam...@suse.de
> > Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
> > Subject: Re: [ovs-discuss] Handling conf.db ownership on
> OVS_USER_ID
> > changes
> > Date: Wed, 24 Apr 2019 19:07:24 -0700
> >
> > On Tue, 23 Apr 2019 at 09:30, Jaime Caamaño Ruiz 
> > wrote:
> >> 
> >> Hello
> >> 
> >> So the non root owned log directory (and run directory) is shared
> >> between non root OVS processes and root OVN processes. Doesn't
> this
> >> raise some security concerns?
> >
> > As a "security concern" you mean something among the lines where
> one
> > of ovs-* processes running under openvswitch user would go ahead
> and
> > create a file with its owner that later one of ovn processes would
> > blindly reuse without checking that it actually belongs to
> root:root?
> >
> >
> >> 
> >> Also, since logrotate rotates the logs with the OVS user, it may
> fail
> >> to some extent to rotate the root owned OVN log files.
> Consequences
> >> vary but in it's current state some logging might be lost. This
> could
> >> be improved but I would guess that the approprioate thing to do is
> to
> >> etiher use a different log directory for OVN or make its processes
> >> run
> >> with the OVS user also. Has any of this been considered?
> >
> > Can you give a more concrete example? I believe logrotate is
> running
> > under root and should be able to rotate everything?
> >
> >
> >> 
> >> BR
> >> Jaime.
> >> 
> >> 
> >> -Original Message-
> >> From: Jaime Caamaño Ruiz 
> >> Reply-to: jcaam...@suse.com
> >> To: Ansis Atteka 
> >> Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
> >> Subject: Re: [ovs-discuss] Handling conf.db ownership on
> OVS_USER_ID
> >> changes
> >> Date: Wed, 17 Apr 2019 12:52:32 +0200
> >> 
> >> > You also need to chown /var/log/openvswitch.*.log files.
> >>

Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

[Jaime Caamaño Ruiz]

> As a "security concern" you mean something among the lines where one
> of ovs-* processes running under openvswitch user would go ahead and
> create a file with its owner that later one of ovn processes would
> blindly reuse without checking that it actually belongs to root:root?

One example is that the OVS user could create link as any OVN log file
to other file owned by root and then OVN would write to that file.

> Can you give a more concrete example? I believe logrotate is running
> under root and should be able to rotate everything?

The logrotate configuration for openvswitch logs has a 'su' directive
to run under the openvswitch user, precisely to prevent something
similar to the above. So it wont be able to rotate root owned logs in
the openvswitch directory. How it fails precisely depends on global
logrotate configuration. For example it will fail to create the new log
file after rotation if the 'create' directive is used. Or it may fail
to compress the rotated log file as it wont be able to read it.

BR
Jaime

-Original Message-
From: Ansis Atteka 
To: jcaam...@suse.de
Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Wed, 24 Apr 2019 19:07:24 -0700

On Tue, 23 Apr 2019 at 09:30, Jaime Caamaño Ruiz 
wrote:
> 
> Hello
> 
> So the non root owned log directory (and run directory) is shared
> between non root OVS processes and root OVN processes. Doesn't this
> raise some security concerns?

As a "security concern" you mean something among the lines where one
of ovs-* processes running under openvswitch user would go ahead and
create a file with its owner that later one of ovn processes would
blindly reuse without checking that it actually belongs to root:root?


> 
> Also, since logrotate rotates the logs with the OVS user, it may fail
> to some extent to rotate the root owned OVN log files. Consequences
> vary but in it's current state some logging might be lost. This could
> be improved but I would guess that the approprioate thing to do is to
> etiher use a different log directory for OVN or make its processes
> run
> with the OVS user also. Has any of this been considered?

Can you give a more concrete example? I believe logrotate is running
under root and should be able to rotate everything?


> 
> BR
> Jaime.
> 
> 
> -Original Message-
> From: Jaime Caamaño Ruiz 
> Reply-to: jcaam...@suse.com
> To: Ansis Atteka 
> Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Wed, 17 Apr 2019 12:52:32 +0200
> 
> > You also need to chown /var/log/openvswitch.*.log files.
> 
> OVS seems to be already handling this. I dont know the details but I
> guess that before dropping capabilities, OVS chowns these by itself.
> 
> > However,
> > what about other daemons, like ovn? Do they share run time dir?
> > Haven't looked into this in a while and don't have a setup to check
> > myself.
> 
> The permisions of the openvswitch run directory are already being
> handled by the service unit file. There is read access to files
> created
> there by OVS and AFAIK what it ultimately makes it a no problem for
> OVN
> is that it still runs as root with the provided unit files. If anyone
> changes this to a user different than the openvswitch user, then that
> is an already existing issue. Am I missing something here?
> 
> Agree with you other considerations.
> 
> BR
> Jaime.
> 
> -Original Message-
> From: Ansis Atteka 
> To: jcaam...@suse.de
> Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Tue, 16 Apr 2019 18:21:48 -0700
> 
> On Tue, 16 Apr 2019 at 17:20, Jaime Caamaño Ruiz 
> wrote:
> > 
> > My intention was doing it at systemd unit (prestart) for file
> > conf.db
> > (and .conf.db.~lock~ that stays around) only.
> 
> You also need to chown /var/log/openvswitch.*.log files.
> > 
> > I was not thinking on absolutely fool proof mechanism, among other
> > things because the admin might have customized the location for the
> > database. Also, updating and restarting the service are things that
> > would usually be monitored. So best effort.
> 
> make sure that best effort solution informs admin in case of error so
> that he does not have to go file by file to find the offending one.
> 
> > 
> > At systemd unit prestart the service is stopped and OVS nor nobody
> > should really be messing with the database files owned by OVS. If
> > the
> > spec file is changing things back to root unappropriately then that
> > sho

Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

[Jaime Caamaño Ruiz]

Hello

So the non root owned log directory (and run directory) is shared
between non root OVS processes and root OVN processes. Doesn't this
raise some security concerns? 

Also, since logrotate rotates the logs with the OVS user, it may fail
to some extent to rotate the root owned OVN log files. Consequences
vary but in it's current state some logging might be lost. This could
be improved but I would guess that the approprioate thing to do is to
etiher use a different log directory for OVN or make its processes run
with the OVS user also. Has any of this been considered?

BR
Jaime.


-Original Message-
From: Jaime Caamaño Ruiz 
Reply-to: jcaam...@suse.com
To: Ansis Atteka 
Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Wed, 17 Apr 2019 12:52:32 +0200

> You also need to chown /var/log/openvswitch.*.log files.

OVS seems to be already handling this. I dont know the details but I
guess that before dropping capabilities, OVS chowns these by itself.

> However,
> what about other daemons, like ovn? Do they share run time dir?
> Haven't looked into this in a while and don't have a setup to check
> myself.

The permisions of the openvswitch run directory are already being
handled by the service unit file. There is read access to files created
there by OVS and AFAIK what it ultimately makes it a no problem for OVN
is that it still runs as root with the provided unit files. If anyone
changes this to a user different than the openvswitch user, then that
is an already existing issue. Am I missing something here?

Agree with you other considerations.

BR
Jaime.

-Original Message-
From: Ansis Atteka 
To: jcaam...@suse.de
Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Tue, 16 Apr 2019 18:21:48 -0700

On Tue, 16 Apr 2019 at 17:20, Jaime Caamaño Ruiz 
wrote:
> 
> My intention was doing it at systemd unit (prestart) for file conf.db
> (and .conf.db.~lock~ that stays around) only.

You also need to chown /var/log/openvswitch.*.log files.
> 
> I was not thinking on absolutely fool proof mechanism, among other
> things because the admin might have customized the location for the
> database. Also, updating and restarting the service are things that
> would usually be monitored. So best effort.

make sure that best effort solution informs admin in case of error so
that he does not have to go file by file to find the offending one.

> 
> At systemd unit prestart the service is stopped and OVS nor nobody
> should really be messing with the database files owned by OVS. If the
> spec file is changing things back to root unappropriately then that
> should be changed too.

Yes.

> 
> The runtime /run/openvswitch directory is also controlled by systemd
> and at least on my system is removed if the service is not active,
> either by graceful or ungraceful exit.

I somewhat agree with this, because back in those days we were using
SystemV where the init.d managed lifecycle of run directory.. However,
what about other daemons, like ovn? Do they share run time dir?
Haven't looked into this in a while and don't have a setup to check
myself.

> 
> I guess we will need to look for specific issues. Any hint to find
> that
> patch?

1. test upgrade to your patched OVS and also from your patched OVS to
another future version
2. if you change existing spec files that are shared with on Fedora or
RHEL/CentOS make sure you don't break anything for them.
3. prefer consistency w.r.t. default behavior across deb and rpm
packages (if you change user by default consider to do that for all
platforms)
4. besides database you will also need to set right access bits to log
files. On my Ubuntu it is "-rw-r- 1 root adm " which means:
4.1. you have to chown it; OR
4.2. add openvswitch user to adm group and chmod that file
5. test logrotation. Maybe you can use a one forced logrotation
invocation to change the ownrer.
6. check that other stuff like ovs-monitor-ipsec is not affected. It
stil need to talk with StrongSwan or libreswan that best to my
knowledge still runs as root.
7. if you use USER= then:
7.1. For Linux datapath make sure you retain CAP_NET_ADMIN Linux
Capabilities
7.2. I don't know if there are special privileges you need to retain
for DPDK. And in future possibly AF_XDP.
8. I haven't looked in a while but check if ovn (that has its own
Systemd service file) shares run dir with OVS

> 
> BR
> Jaime.
> 
> -Original Message-
> From: Ansis Atteka 
> To: jcaam...@suse.de
> Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Tue, 16 Apr 2019 16:11:44 -0700
> 
> On Tue, 16 Apr 2019 at 15:36, Jaime Caamaño Ruiz 
> wrote:
> > 
> > > On any giv

Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

[Jaime Caamaño Ruiz]

> You also need to chown /var/log/openvswitch.*.log files.

OVS seems to be already handling this. I dont know the details but I
guess that before dropping capabilities, OVS chowns these by itself.

> However,
> what about other daemons, like ovn? Do they share run time dir?
> Haven't looked into this in a while and don't have a setup to check
> myself.

The permisions of the openvswitch run directory are already being
handled by the service unit file. There is read access to files created
there by OVS and AFAIK what it ultimately makes it a no problem for OVN
is that it still runs as root with the provided unit files. If anyone
changes this to a user different than the openvswitch user, then that
is an already existing issue. Am I missing something here?

Agree with you other considerations.

BR
Jaime.

-Original Message-
From: Ansis Atteka 
To: jcaam...@suse.de
Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Tue, 16 Apr 2019 18:21:48 -0700

On Tue, 16 Apr 2019 at 17:20, Jaime Caamaño Ruiz 
wrote:
> 
> My intention was doing it at systemd unit (prestart) for file conf.db
> (and .conf.db.~lock~ that stays around) only.

You also need to chown /var/log/openvswitch.*.log files.
> 
> I was not thinking on absolutely fool proof mechanism, among other
> things because the admin might have customized the location for the
> database. Also, updating and restarting the service are things that
> would usually be monitored. So best effort.

make sure that best effort solution informs admin in case of error so
that he does not have to go file by file to find the offending one.

> 
> At systemd unit prestart the service is stopped and OVS nor nobody
> should really be messing with the database files owned by OVS. If the
> spec file is changing things back to root unappropriately then that
> should be changed too.

Yes.

> 
> The runtime /run/openvswitch directory is also controlled by systemd
> and at least on my system is removed if the service is not active,
> either by graceful or ungraceful exit.

I somewhat agree with this, because back in those days we were using
SystemV where the init.d managed lifecycle of run directory.. However,
what about other daemons, like ovn? Do they share run time dir?
Haven't looked into this in a while and don't have a setup to check
myself.

> 
> I guess we will need to look for specific issues. Any hint to find
> that
> patch?

1. test upgrade to your patched OVS and also from your patched OVS to
another future version
2. if you change existing spec files that are shared with on Fedora or
RHEL/CentOS make sure you don't break anything for them.
3. prefer consistency w.r.t. default behavior across deb and rpm
packages (if you change user by default consider to do that for all
platforms)
4. besides database you will also need to set right access bits to log
files. On my Ubuntu it is "-rw-r- 1 root adm " which means:
4.1. you have to chown it; OR
4.2. add openvswitch user to adm group and chmod that file
5. test logrotation. Maybe you can use a one forced logrotation
invocation to change the ownrer.
6. check that other stuff like ovs-monitor-ipsec is not affected. It
stil need to talk with StrongSwan or libreswan that best to my
knowledge still runs as root.
7. if you use USER= then:
7.1. For Linux datapath make sure you retain CAP_NET_ADMIN Linux
Capabilities
7.2. I don't know if there are special privileges you need to retain
for DPDK. And in future possibly AF_XDP.
8. I haven't looked in a while but check if ovn (that has its own
Systemd service file) shares run dir with OVS

> 
> BR
> Jaime.
> 
> -Original Message-
> From: Ansis Atteka 
> To: jcaam...@suse.de
> Cc: ovs-discuss@openvswitch.org, Ben Pfaff 
> Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
> changes
> Date: Tue, 16 Apr 2019 16:11:44 -0700
> 
> On Tue, 16 Apr 2019 at 15:36, Jaime Caamaño Ruiz 
> wrote:
> > 
> > > On any given system, I would expect ovsdb-server to run as the
> > > same
> > > user
> > > every time.  Is there a reason to sometimes use a different user?
> > 
> > Well, changing from root which was the only supported option in the
> > past and current default to a different user (like the suggested
> > openvswitch), this one time, might be somewhat common. And the
> > scenario
> > that I am looking at is suporting a pianless upgrade through it.
> 
> Actually debian packages still by default use "root" user. Not
> "openvswitch".
> 
> Long time ago there was an attempt to change the default user across
> all different flavor packages to openvswitch (you can lookup the
> patch
> on mailing list). However, as you noticed upgrades are tricky. 

Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

[Jaime Caamaño Ruiz]

> On any given system, I would expect ovsdb-server to run as the same
> user
> every time.  Is there a reason to sometimes use a different user?

Well, changing from root which was the only supported option in the
past and current default to a different user (like the suggested
openvswitch), this one time, might be somewhat common. And the scenario
that I am looking at is suporting a pianless upgrade through it.

So if that makes sense to you, I will go ahead with the patch.

BR
Jaime.

-Original Message-
From: Ben Pfaff 
To: jcaam...@suse.de
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] Handling conf.db ownership on OVS_USER_ID
changes
Date: Tue, 16 Apr 2019 15:25:24 -0700

On Tue, Apr 16, 2019 at 11:32:21PM +0200, Jaime Caamaño Ruiz wrote:
> When sysconfig OVS_USER_ID is changed to a different user, it
> requires
> a manual ownership change of the OVS conf.db database if existing or
> otherwise ovsdb-server will fail to (re)start. I was wondering if I
> am
> missing any particular reason why this is change of ownership is not
> automatically being handled through the service unit file as it is
> being done with other items.

On any given system, I would expect ovsdb-server to run as the same
user
every time.  Is there a reason to sometimes use a different user?

Or, perhaps you are saying that, just before invoking ovsdb-server, the
service unit should chown (or whatever) the database file to the user
that it is going to use to invoke ovsdb-server.  That might be
reasonable, but no one has thought to do it yet.  Do you want to submit
a patch?

Thanks,

Ben.

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] Handling conf.db ownership on OVS_USER_ID changes

[Jaime Caamaño Ruiz]

Hello

When sysconfig OVS_USER_ID is changed to a different user, it requires
a manual ownership change of the OVS conf.db database if existing or
otherwise ovsdb-server will fail to (re)start. I was wondering if I am
missing any particular reason why this is change of ownership is not
automatically being handled through the service unit file as it is
being done with other items.

BR
Jaime.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Latest maintenace release not announced or available for download

[Jaime Caamaño Ruiz]

Hello

In the meantime, two related questions:

- Would github releases be safe to use? For exmaple

https://github.com/openvswitch/ovs/archive/v2.5.7.tar.gz

- Is 2.5.7 going to replace 2.5.6 as LTS release?

BR
Jaime.


-Original Message-
From: Ben Pfaff 
To: Justin Pettit , jcaam...@suse.de
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] Latest maintenace release not announced or
available for download
Date: Tue, 19 Mar 2019 10:15:29 -0700

Justin, you had some information on this, can you follow up?

On Tue, Mar 19, 2019 at 11:41:03AM +0100, Jaime Caamaño Ruiz wrote:
> Missing 2.9.4, 2.8.6, 2.7.8, 2.6.5, 2.5.7.
> 
> I don't know if they can actually be considered released and I guess
> that's what I am really asking. I do see them in the relases section
> in
> github.
> 
> BR
> Jaime.
> 
>  
> -Original Message-
> From: Ben Pfaff 
> To: jcaam...@suse.de
> Cc: ovs-discuss@openvswitch.org
> Subject: Re: [ovs-discuss] Latest maintenace release not announced or
> available for download
> Date: Mon, 18 Mar 2019 13:22:43 -0700
> 
> On Mon, Mar 18, 2019 at 12:09:07PM +0100, Jaime Caamaño Ruiz wrote:
> > Is there any reason why the latest maintenance releases have not
> > been
> > announced yet and are not available for download at the usual URL
> > 
> > https://www.openvswitch.org/releases/
> 
> What versions are missing?
> 


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Latest maintenace release not announced or available for download

[Jaime Caamaño Ruiz]

Missing 2.9.4, 2.8.6, 2.7.8, 2.6.5, 2.5.7.

I don't know if they can actually be considered released and I guess
that's what I am really asking. I do see them in the relases section in
github.

BR
Jaime.

 
-Original Message-
From: Ben Pfaff 
To: jcaam...@suse.de
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] Latest maintenace release not announced or
available for download
Date: Mon, 18 Mar 2019 13:22:43 -0700

On Mon, Mar 18, 2019 at 12:09:07PM +0100, Jaime Caamaño Ruiz wrote:
> Is there any reason why the latest maintenance releases have not been
> announced yet and are not available for download at the usual URL
> 
> https://www.openvswitch.org/releases/

What versions are missing?

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] Latest maintenace release not announced or available for download

[Jaime Caamaño Ruiz]

Hello

Is there any reason why the latest maintenance releases have not been
announced yet and are not available for download at the usual URL

https://www.openvswitch.org/releases/

Thanks,
Jaime.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Problems executing decap(eth)+encap(eth) actions

[Jaime Caamaño Ruiz]

Greg, I submitted this patch [1], let me know if anything looks bad.

[1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-October/353410.html

Thanks
Jaime.


-Original Message-
From: Jaime Caamaño Ruiz 
Reply-to: jcaam...@suse.com
To: Gregory Rose , ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] Problems executing decap(eth)+encap(eth)
actions
Date: Wed, 31 Oct 2018 12:07:59 +0100

Let me give it a try.

Aside for the fix on master, who takes care of mapping the fix to
bugfix releases?

BR
Jaime.

-Original Message-
From: Gregory Rose 
To: ovs-discuss@openvswitch.org, jcaam...@suse.de
Subject: Re: [ovs-discuss] Problems executing decap(eth)+encap(eth)
actions
Date: Tue, 30 Oct 2018 14:42:15 -0700

On 10/29/2018 3:38 AM, Jaime Caamaño Ruiz wrote:
> Hey Greg. Thanks for helping out. I did build OVS with the fix and it
> got my problem sorted without causing any additional ones on my
> environment. Let me know if I can help with anything else.
> 
> BR
> Jaime.

Jaime,

you seem to have identified a bug!

Using printks with a simple rule to just decap and then encap an 
Ethernet header we see this with the code
as it is right now:

[13568.973807] __ovs_nla_copy_actions:3007 <- decap
[13568.973812] __ovs_nla_copy_actions:3012 <- decap succeeds but sets 
mac_proto = MAC_PROTO_ETHERNET
[13568.973815] __ovs_nla_copy_actions:2999 <- encap
[13568.973818] openvswitch: netlink: Flow actions may not be safe on
all 
matching packets. <- returns -EINVAL

Note that the decap happens at lines 3007-3012 and is successful. 
However, the very next encap action
starting at line 2999 does not finish and returns -EINVAL so a printk
at 
line 3002 does not execute.
If I change the code as you suggested the flow of decap/encap works 
without complaint and without
returning -EINVAL:

[13838.435051] __ovs_nla_copy_actions:3007 <- decap
[13838.435054] __ovs_nla_copy_actions:3012 <-decap succeeds and sets 
mac_proto = MAC_PROTO_NONE
[13838.435055] __ovs_nla_copy_actions:2999 <- encap
[13838.435056] __ovs_nla_copy_actions:3002 <- encap succeeds and sets 
mac_proto = MAC_PROTO_ETHERNET

Thank you for finding this bug.  Do you wish to send the patch to fix
it 
or would you prefer me to do it?

Regards,

- Greg

> 
> 
> -Original Message-
> From: Gregory Rose 
> To: ovs-discuss@openvswitch.org, jcaam...@suse.de
> Subject: Re: [ovs-discuss] Problems executing decap(eth)+encap(eth)
> actions
> Date: Fri, 26 Oct 2018 15:42:51 -0700
> 
> On 10/19/2018 1:39 AM, Jaime Caamaño Ruiz wrote:
> > Hello
> > 
> > When using nsh encapsulation, it's useful to normalize your
> > pipeline
> > to
> > packet_type=nsh, poping an ethernet header on input if necessary
> > and
> > pushing an ethernet header again if required before output.
> > 
> > But it seems to be problematic:
> > 
> > ---
> > 2018-10-18T13:10:59.196Z|00010|dpif(handler3)|WARN|system@ovs-syste
> > m:
> > execute
> > pop_eth,push_eth(src=fe:16:3e:c1:9e:87,dst=fa:16:3e:c1:9e:87),5
> > failed (Invalid argument) on packet
> > vlan_tci=0x,dl_src=fa:16:3e:c2:e6:68,dl_dst=fe:16:3e:c2:e6:68,d
> > l_
> > ty
> > pe=0x894f,nsh_flags=0,nsh_ttl=63,nsh_mdtype=1,nsh_np=3,nsh_spi=0x1a
> > ,n
> > sh
> > _si=254,nsh_c1=0xc0a82a01,nsh_c2=0x3,nsh_c3=0x0,nsh_c4=0x9100,n
> > w_
> > pr
> > oto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
> >with metadata
> > skb_priority(0),tunnel(tun_id=0x0,src=192.168.42.1,dst=192.168.42.3
> > ,t
> > tl
> > =64,tp_src=47656,tp_dst=4789,flags(key)),skb_mark(0),in_port(4) mtu
> > 0
> > ---
> > 
> > Looking at the code datapath/flow_netlink.c @
> > __ovs_nla_copy_actions:
> > 
> >   case OVS_ACTION_ATTR_PUSH_ETH:
> >   /* Disallow pushing an Ethernet header if
> > one
> >* is already present */
> >   if (mac_proto != MAC_PROTO_NONE)
> >   return -EINVAL;
> >   mac_proto = MAC_PROTO_NONE;
> >   break;
> > 
> >   case OVS_ACTION_ATTR_POP_ETH:
> >   if (mac_proto != MAC_PROTO_ETHERNET)
> >   return -EINVAL;
> >   if (vlan_tci & htons(VLAN_TAG_PRESENT))
> >   return -EINVAL;
> >   mac_proto = MAC_PROTO_ETHERNET;
> >   break;
> > 
> > 
> > Isn't the mac_proto set inverted here, should'nt it look like this?
> > 
> > 
> >   case OVS_ACTION_ATTR_PUSH_ETH:
> >  

Re: [ovs-discuss] Problems executing decap(eth)+encap(eth) actions

[Jaime Caamaño Ruiz]

Let me give it a try.

Aside for the fix on master, who takes care of mapping the fix to
bugfix releases?

BR
Jaime.

-Original Message-
From: Gregory Rose 
To: ovs-discuss@openvswitch.org, jcaam...@suse.de
Subject: Re: [ovs-discuss] Problems executing decap(eth)+encap(eth)
actions
Date: Tue, 30 Oct 2018 14:42:15 -0700

On 10/29/2018 3:38 AM, Jaime Caamaño Ruiz wrote:
> Hey Greg. Thanks for helping out. I did build OVS with the fix and it
> got my problem sorted without causing any additional ones on my
> environment. Let me know if I can help with anything else.
> 
> BR
> Jaime.

Jaime,

you seem to have identified a bug!

Using printks with a simple rule to just decap and then encap an 
Ethernet header we see this with the code
as it is right now:

[13568.973807] __ovs_nla_copy_actions:3007 <- decap
[13568.973812] __ovs_nla_copy_actions:3012 <- decap succeeds but sets 
mac_proto = MAC_PROTO_ETHERNET
[13568.973815] __ovs_nla_copy_actions:2999 <- encap
[13568.973818] openvswitch: netlink: Flow actions may not be safe on
all 
matching packets. <- returns -EINVAL

Note that the decap happens at lines 3007-3012 and is successful. 
However, the very next encap action
starting at line 2999 does not finish and returns -EINVAL so a printk
at 
line 3002 does not execute.
If I change the code as you suggested the flow of decap/encap works 
without complaint and without
returning -EINVAL:

[13838.435051] __ovs_nla_copy_actions:3007 <- decap
[13838.435054] __ovs_nla_copy_actions:3012 <-decap succeeds and sets 
mac_proto = MAC_PROTO_NONE
[13838.435055] __ovs_nla_copy_actions:2999 <- encap
[13838.435056] __ovs_nla_copy_actions:3002 <- encap succeeds and sets 
mac_proto = MAC_PROTO_ETHERNET

Thank you for finding this bug.  Do you wish to send the patch to fix
it 
or would you prefer me to do it?

Regards,

- Greg

> 
> 
> -Original Message-
> From: Gregory Rose 
> To: ovs-discuss@openvswitch.org, jcaam...@suse.de
> Subject: Re: [ovs-discuss] Problems executing decap(eth)+encap(eth)
> actions
> Date: Fri, 26 Oct 2018 15:42:51 -0700
> 
> On 10/19/2018 1:39 AM, Jaime Caamaño Ruiz wrote:
> > Hello
> > 
> > When using nsh encapsulation, it's useful to normalize your
> > pipeline
> > to
> > packet_type=nsh, poping an ethernet header on input if necessary
> > and
> > pushing an ethernet header again if required before output.
> > 
> > But it seems to be problematic:
> > 
> > ---
> > 2018-10-18T13:10:59.196Z|00010|dpif(handler3)|WARN|system@ovs-syste
> > m:
> > execute
> > pop_eth,push_eth(src=fe:16:3e:c1:9e:87,dst=fa:16:3e:c1:9e:87),5
> > failed (Invalid argument) on packet
> > vlan_tci=0x,dl_src=fa:16:3e:c2:e6:68,dl_dst=fe:16:3e:c2:e6:68,d
> > l_
> > ty
> > pe=0x894f,nsh_flags=0,nsh_ttl=63,nsh_mdtype=1,nsh_np=3,nsh_spi=0x1a
> > ,n
> > sh
> > _si=254,nsh_c1=0xc0a82a01,nsh_c2=0x3,nsh_c3=0x0,nsh_c4=0x9100,n
> > w_
> > pr
> > oto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
> >with metadata
> > skb_priority(0),tunnel(tun_id=0x0,src=192.168.42.1,dst=192.168.42.3
> > ,t
> > tl
> > =64,tp_src=47656,tp_dst=4789,flags(key)),skb_mark(0),in_port(4) mtu
> > 0
> > ---
> > 
> > Looking at the code datapath/flow_netlink.c @
> > __ovs_nla_copy_actions:
> > 
> >   case OVS_ACTION_ATTR_PUSH_ETH:
> >   /* Disallow pushing an Ethernet header if
> > one
> >* is already present */
> >   if (mac_proto != MAC_PROTO_NONE)
> >   return -EINVAL;
> >   mac_proto = MAC_PROTO_NONE;
> >   break;
> > 
> >   case OVS_ACTION_ATTR_POP_ETH:
> >   if (mac_proto != MAC_PROTO_ETHERNET)
> >   return -EINVAL;
> >   if (vlan_tci & htons(VLAN_TAG_PRESENT))
> >   return -EINVAL;
> >   mac_proto = MAC_PROTO_ETHERNET;
> >   break;
> > 
> > 
> > Isn't the mac_proto set inverted here, should'nt it look like this?
> > 
> > 
> >   case OVS_ACTION_ATTR_PUSH_ETH:
> >   /* Disallow pushing an Ethernet header if
> > one
> >* is already present */
> >   if (mac_proto != MAC_PROTO_NONE)
> >   return -EINVAL;
> >   mac_proto = MAC_PROTO_ETHERNET;
> >   break;
> > 
> >  

Re: [ovs-discuss] Problems executing decap(eth)+encap(eth) actions

[Jaime Caamaño Ruiz]

Hey Greg. Thanks for helping out. I did build OVS with the fix and it
got my problem sorted without causing any additional ones on my
environment. Let me know if I can help with anything else.

BR
Jaime.


-Original Message-
From: Gregory Rose 
To: ovs-discuss@openvswitch.org, jcaam...@suse.de
Subject: Re: [ovs-discuss] Problems executing decap(eth)+encap(eth)
actions
Date: Fri, 26 Oct 2018 15:42:51 -0700

On 10/19/2018 1:39 AM, Jaime Caamaño Ruiz wrote:
> Hello
> 
> When using nsh encapsulation, it's useful to normalize your pipeline
> to
> packet_type=nsh, poping an ethernet header on input if necessary and
> pushing an ethernet header again if required before output.
> 
> But it seems to be problematic:
> 
> ---
> 2018-10-18T13:10:59.196Z|00010|dpif(handler3)|WARN|system@ovs-system:
> execute
> pop_eth,push_eth(src=fe:16:3e:c1:9e:87,dst=fa:16:3e:c1:9e:87),5
> failed (Invalid argument) on packet
> vlan_tci=0x,dl_src=fa:16:3e:c2:e6:68,dl_dst=fe:16:3e:c2:e6:68,dl_
> ty
> pe=0x894f,nsh_flags=0,nsh_ttl=63,nsh_mdtype=1,nsh_np=3,nsh_spi=0x1a,n
> sh
> _si=254,nsh_c1=0xc0a82a01,nsh_c2=0x3,nsh_c3=0x0,nsh_c4=0x9100,nw_
> pr
> oto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
>   with metadata
> skb_priority(0),tunnel(tun_id=0x0,src=192.168.42.1,dst=192.168.42.3,t
> tl
> =64,tp_src=47656,tp_dst=4789,flags(key)),skb_mark(0),in_port(4) mtu 0
> ---
> 
> Looking at the code datapath/flow_netlink.c @ __ovs_nla_copy_actions:
> 
>  case OVS_ACTION_ATTR_PUSH_ETH:
>  /* Disallow pushing an Ethernet header if
> one
>   * is already present */
>  if (mac_proto != MAC_PROTO_NONE)
>  return -EINVAL;
>  mac_proto = MAC_PROTO_NONE;
>  break;
> 
>  case OVS_ACTION_ATTR_POP_ETH:
>  if (mac_proto != MAC_PROTO_ETHERNET)
>  return -EINVAL;
>  if (vlan_tci & htons(VLAN_TAG_PRESENT))
>  return -EINVAL;
>  mac_proto = MAC_PROTO_ETHERNET;
>  break;
> 
> 
> Isn't the mac_proto set inverted here, should'nt it look like this?
> 
> 
>  case OVS_ACTION_ATTR_PUSH_ETH:
>  /* Disallow pushing an Ethernet header if
> one
>   * is already present */
>  if (mac_proto != MAC_PROTO_NONE)
>  return -EINVAL;
>  mac_proto = MAC_PROTO_ETHERNET;
>  break;
> 
>  case OVS_ACTION_ATTR_POP_ETH:
>  if (mac_proto != MAC_PROTO_ETHERNET)
>  return -EINVAL;
>  if (vlan_tci & htons(VLAN_TAG_PRESENT))
>  return -EINVAL;
>  mac_proto = MAC_PROTO_NONE;
>  break;

Jaime,

I am looking into this and at first sight this does look inverted but
we 
have no other reported bugs
in this area so I want to be careful that we don't break anything else 
while fixing this.  Have you tried
building OVS with this code change and been able to verify that this 
change works?

I'll be working on setting up a test bed to check this - hopefully by 
early next week I can report back so
stay tuned please.

Thanks,

- Greg

> 
> BR
> Jaime.
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Bad checksums observed with nsh encapsulation

[Jaime Caamaño Ruiz]

Hi Jiri

Did you have the chance to take a look at this?

Thanks,
Jaime.

-Original Message-
From: Jaime Caamaño Ruiz 
Reply-To: jcaam...@suse.com
To: Jiri Benc , Yi Y Yang 
Cc: ovs-discuss@openvswitch.org 
Subject: Re: [ovs-discuss] Bad checksums observed with nsh
encapsulation
Date: Fri, 29 Jun 2018 15:59:48 +0200

Hello Jiri

I will try to explain in more detail and provide a simple scenario to
reproduce.

Lets look first into the vm2vm scenario with no nsh, just a tcp packet
taken from on vm port and put into the other vmp port. The originating
vm requests checksum offload. OVS just clones the buffer into the
destination port. It will keep the same values for
ip_summed=CHECKSUM_PARTIAL, csum_offset, csum_start. The destination vm
assumes that it does not need to verify the checksum as it deems the
packet to be local (see [1]).

Now, lets assume the packet gets eth+nsh encapsulated and sent to
another intermediary service function vm on the same host, before being
unencapsulated and sent to the final destination vm. In my case the
intermediary vm is just a user space app that emulates a service
function. Opens a raw socket, inverts the outer eth header macs, and
decrements the path index before sending back the packet to OVS. At
this point, the nsh payload which is the original tcp packet has a bad
tcp checksum. Depending on what the service function does, this might
not be ok already. In any case, when the packet is back at OVS, it no
longer holds any of the previous csum metadata as it is a new buffer
generated via the service function user space app that is just
eth+nsh+payload. OVS unecapsulates and sends the packet to the final
destination vm, where is no longer deemed as a local packet as
ip_summed=CHECKSUM_NONE, so it is rejected when the tcp checksum does
not verify.

As a way to reproduce:

---

ovs-vsctl add-br br0

ip link add blueth0 type veth peer name veth0
ip netns add blue
ip link set veth0 netns blue
ovs-vsctl add-port ovsbr0 blueth0
ip netns exec blue ip addr add 10.0.0.1/24 dev veth0
ip link set blueth0 up
ip netns exec blue ip link set veth0 up

ip link add redeth0 type veth peer name veth0
ip netns add red
ip link set veth0 netns red
ovs-vsctl add-port ovsbr0 redeth0
ip netns exec red ip addr add 10.0.0.2/24 dev veth0
ip link set redeth0 up
ip netns exec red ip link set veth0 up

ovs-ofctl -Oopenflow13 add-flow br0
"priority=1,tcp,in_port=1,actions=encap(nsh),encap(ethernet),2"
ovs-ofctl -Oopenflow13 add-flow br0
"priority=1,tcp,in_port=2,actions=encap(nsh),encap(ethernet),1"

Capture on one namespace:
ip netns exec red tcpdump -i veth0 -U -w test.pcap

Connect form the other:
ip netns exec blue nc 10.0.0.2 80



If you inspect the capture (pic attached), you will see the SYN
connection attempts nsh encapsulated. The first attempt, the checksum
will be ok...that is because there is no flow in kernel datapath and
the checksum is calculated prior to upcall (see [2]). But successive
attempts will have wrong checksum since the flow has already been
installed in the kernel datapath and there is no upcall.

BR
Jaime

[1] https://github.com/torvalds/linux/blob/ea5f39f2f994e6fb8cb8d0304aa5
f422ae3bbf83/include/linux/skbuff.h#L3589
[2] https://github.com/openvswitch/ovs/blob/d22f8927c3c9034128df3859e98
e486ba1f06d60/datapath/datapath.c#L439

-Original Message-
From: Jiri Benc 
To: Yi Y Yang 
Cc: ovs-discuss@openvswitch.org , jcaamano
@suse.de
Subject: Re: [ovs-discuss] Bad checksums observed with nsh
encapsulation
Date: Tue, 26 Jun 2018 17:07:11 +0200

> But when we are pushing nsh headers, the first receiver may not be
> the final receiver and CHECKSUM_PARTIAL may not reach the final
> reciever which will then verify and reject a bad checksum.

I don't understand this. Could you please provide a minimal test case?
I will then reproduce locally and take a look. From the data provided,
it's unclear what's going on - the ofproto trace is too complex and
it's unclear what the individual interfaces are.

> So I think it may be necessary to handle the CHECKSUM_PARTIAL case on
> nsh_push, something like adding
> 
> if (skb->ip_summed == CHECKSUM_PARTIAL) {
> skb_checksum_help(skb);
> }
> 
> Tried that and got rid of my problem.

That's an interesting datapoint. However, this is not the right fix,
there's no reason for such code. The bug lies elsewhere.

 Jiri
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Bad checksums observed with nsh encapsulation

[Jaime Caamaño Ruiz]

Hello Jiri

I will try to explain in more detail and provide a simple scenario to
reproduce.

Lets look first into the vm2vm scenario with no nsh, just a tcp packet
taken from on vm port and put into the other vmp port. The originating
vm requests checksum offload. OVS just clones the buffer into the
destination port. It will keep the same values for
ip_summed=CHECKSUM_PARTIAL, csum_offset, csum_start. The destination vm
assumes that it does not need to verify the checksum as it deems the
packet to be local (see [1]).

Now, lets assume the packet gets eth+nsh encapsulated and sent to
another intermediary service function vm on the same host, before being
unencapsulated and sent to the final destination vm. In my case the
intermediary vm is just a user space app that emulates a service
function. Opens a raw socket, inverts the outer eth header macs, and
decrements the path index before sending back the packet to OVS. At
this point, the nsh payload which is the original tcp packet has a bad
tcp checksum. Depending on what the service function does, this might
not be ok already. In any case, when the packet is back at OVS, it no
longer holds any of the previous csum metadata as it is a new buffer
generated via the service function user space app that is just
eth+nsh+payload. OVS unecapsulates and sends the packet to the final
destination vm, where is no longer deemed as a local packet as
ip_summed=CHECKSUM_NONE, so it is rejected when the tcp checksum does
not verify.

As a way to reproduce:

---

ovs-vsctl add-br br0

ip link add blueth0 type veth peer name veth0
ip netns add blue
ip link set veth0 netns blue
ovs-vsctl add-port ovsbr0 blueth0
ip netns exec blue ip addr add 10.0.0.1/24 dev veth0
ip link set blueth0 up
ip netns exec blue ip link set veth0 up

ip link add redeth0 type veth peer name veth0
ip netns add red
ip link set veth0 netns red
ovs-vsctl add-port ovsbr0 redeth0
ip netns exec red ip addr add 10.0.0.2/24 dev veth0
ip link set redeth0 up
ip netns exec red ip link set veth0 up

ovs-ofctl -Oopenflow13 add-flow br0 
"priority=1,tcp,in_port=1,actions=encap(nsh),encap(ethernet),2"
ovs-ofctl -Oopenflow13 add-flow br0 
"priority=1,tcp,in_port=2,actions=encap(nsh),encap(ethernet),1"

Capture on one namespace:
ip netns exec red tcpdump -i veth0 -U -w test.pcap

Connect form the other:
ip netns exec blue nc 10.0.0.2 80



If you inspect the capture (pic attached), you will see the SYN
connection attempts nsh encapsulated. The first attempt, the checksum
will be ok...that is because there is no flow in kernel datapath and
the checksum is calculated prior to upcall (see [2]). But successive
attempts will have wrong checksum since the flow has already been
installed in the kernel datapath and there is no upcall.

BR
Jaime

[1] 
https://github.com/torvalds/linux/blob/ea5f39f2f994e6fb8cb8d0304aa5f422ae3bbf83/include/linux/skbuff.h#L3589
[2] 
https://github.com/openvswitch/ovs/blob/d22f8927c3c9034128df3859e98e486ba1f06d60/datapath/datapath.c#L439

-Original Message-
From: Jiri Benc 
To: Yi Y Yang 
Cc: ovs-discuss@openvswitch.org , jcaamano
@suse.de
Subject: Re: [ovs-discuss] Bad checksums observed with nsh
encapsulation
Date: Tue, 26 Jun 2018 17:07:11 +0200

> But when we are pushing nsh headers, the first receiver may not be
> the final receiver and CHECKSUM_PARTIAL may not reach the final
> reciever which will then verify and reject a bad checksum.

I don't understand this. Could you please provide a minimal test case?
I will then reproduce locally and take a look. From the data provided,
it's unclear what's going on - the ofproto trace is too complex and
it's unclear what the individual interfaces are.

> So I think it may be necessary to handle the CHECKSUM_PARTIAL case on
> nsh_push, something like adding
> 
> if (skb->ip_summed == CHECKSUM_PARTIAL) {
> skb_checksum_help(skb);
> }
> 
> Tried that and got rid of my problem.

That's an interesting datapoint. However, this is not the right fix,
there's no reason for such code. The bug lies elsewhere.

 Jiri
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Bad checksums observed with nsh encapsulation

[Jaime Caamaño Ruiz]

Hello

I looked a bit more into the issue.

This is happenning when OVS receives a CHECKSUM_PARTIAL. For a normal
vm2vm non nsh scenario, OVS provides the same CHECKSUM_PARTIAL to the
receiver which wont then verify the checksum.

But when we are pushing nsh headers, the first receiver may not be the
final receiver and CHECKSUM_PARTIAL may not reach the final reciever
which will then verify and reject a bad checksum.

So I think it may be necessary to handle the CHECKSUM_PARTIAL case on
nsh_push, something like adding

if (skb->ip_summed == CHECKSUM_PARTIAL) {
skb_checksum_help(skb);
}

Tried that and got rid of my problem.

Any thoughts?

BR
Jaime.


-Original Message-
From: Jaime Caamaño Ruiz 
Reply-To: jcaam...@suse.com
To: jcaam...@suse.com, ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] Bad checksums observed with nsh
encapsulation
Date: Thu, 14 Jun 2018 18:15:10 +0200

Hello

I have done a follow-up test very similar to the previous one, but this
time using two computes such that client and server reside in one of
them and the vnf on the other one. This means that packets coming from
either client/server that are being nsh encapsulated are then forwarded
to the vnf compute egressing through a vxlan tunnel port
(vxlan+eth+nsh+payload). 

In this scenario I dont observe the checksum problem. So it is a
combination of nsh encasulation + tap port egress when the checksum is
sometimes observed to be incorrect.

BR
Jaime.


-Original Message-
From: Jaime Caamaño Ruiz  
Reply-To: jcaam...@suse.com
To: ovs-discuss@openvswitch.org, jcaam...@suse.de
Subject: [ovs-discuss] Bad checksums observed with nsh encapsulation
Date: Wed, 13 Jun 2018 12:51:59 +0200

Hello

I am facing a problem where eth+nsh encapsulated packets egress OVS
with incorrect checksum. 

The scenario is

client  vnf  server

all guests on the same host so this is vm2vm traffic, tap ports are
directly added to the ovs bridge. TCP traffic from/to server port 80 is
encapsulated with eth+nsh and traverse the vnf. I exercise the traffic
by using nc both on client and server.

I include captures at the client [1] and at the vnf [2] where I attempt
three tcp connections on port 80. The general observation is that
packets generated on client/server are seen there with wrong checksums
due to offloading but then arrive at the vnf with correct checksum. But
not all of them. For the first conenction attempt you can see that SYN
(frame 74) and ACK (78) are ok, but then FIN (79) is not ok. A
retransmitted FIN (80) is still not ok and then a further FIN (93)
retranmission is ok. Much of the same happens for the second attempt.
The third attempt shows a bad SYN (104) coming from the server.

Two additional observations:

- This does not happen if I try the same on a port different than 80 so
that the traffic goes directly from the client to the server with no
eth+nsh encapsulation.

- This does not happen if I disable tx offloading both in the server
and the client.

I include also the flows [3] and the ofproto trace [4] for the FIN
(79), generated by the client, which is eth+nsh encapsulated and
forwarded to the vnf. The decision on whether packet should be eth+nsh
encapsulated or no happens on table 101 by setting reg2 which is then
checked on 221. Packet is nsh encapsulated on table 222 and then
ethernet encapsulated on table 83. If not encapsulated packet would go
from 221 back to 220 and output there without any further actions.

Using OVS 2.9.2 with OVS tree kernel module. Kernel is 4.4.

I am understanding the problem correctly in regards to OVS being
responsible for these checksums when offloading is enabled?
Any pointers on how I can debug this further?
Why would just some of the eth+nsh packets exhibit this problem and not
all?
Why would these bad packets be ok after retransmissions?

[1] https://filebin.net/8mnypc2qm4vninof/client.pcap?t=b097kh0m
[2] https://filebin.net/8mnypc2qm4vninof/vnf_eth0.pcap?t=b097kh0m
[3] https://hastebin.com/nuhexufaze.sql
[4] https://hastebin.com/yevufanula.http

Thanks for your help,
Jaime.







___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Bad checksums observed with nsh encapsulation

[Jaime Caamaño Ruiz]

Hello

I have done a follow-up test very similar to the previous one, but this
time using two computes such that client and server reside in one of
them and the vnf on the other one. This means that packets coming from
either client/server that are being nsh encapsulated are then forwarded
to the vnf compute egressing through a vxlan tunnel port
(vxlan+eth+nsh+payload). 

In this scenario I dont observe the checksum problem. So it is a
combination of nsh encasulation + tap port egress when the checksum is
sometimes observed to be incorrect.

BR
Jaime.


-Original Message-
From: Jaime Caamaño Ruiz  
Reply-To: jcaam...@suse.com
To: ovs-discuss@openvswitch.org, jcaam...@suse.de
Subject: [ovs-discuss] Bad checksums observed with nsh encapsulation
Date: Wed, 13 Jun 2018 12:51:59 +0200

Hello

I am facing a problem where eth+nsh encapsulated packets egress OVS
with incorrect checksum. 

The scenario is

client  vnf  server

all guests on the same host so this is vm2vm traffic, tap ports are
directly added to the ovs bridge. TCP traffic from/to server port 80 is
encapsulated with eth+nsh and traverse the vnf. I exercise the traffic
by using nc both on client and server.

I include captures at the client [1] and at the vnf [2] where I attempt
three tcp connections on port 80. The general observation is that
packets generated on client/server are seen there with wrong checksums
due to offloading but then arrive at the vnf with correct checksum. But
not all of them. For the first conenction attempt you can see that SYN
(frame 74) and ACK (78) are ok, but then FIN (79) is not ok. A
retransmitted FIN (80) is still not ok and then a further FIN (93)
retranmission is ok. Much of the same happens for the second attempt.
The third attempt shows a bad SYN (104) coming from the server.

Two additional observations:

- This does not happen if I try the same on a port different than 80 so
that the traffic goes directly from the client to the server with no
eth+nsh encapsulation.

- This does not happen if I disable tx offloading both in the server
and the client.

I include also the flows [3] and the ofproto trace [4] for the FIN
(79), generated by the client, which is eth+nsh encapsulated and
forwarded to the vnf. The decision on whether packet should be eth+nsh
encapsulated or no happens on table 101 by setting reg2 which is then
checked on 221. Packet is nsh encapsulated on table 222 and then
ethernet encapsulated on table 83. If not encapsulated packet would go
from 221 back to 220 and output there without any further actions.

Using OVS 2.9.2 with OVS tree kernel module. Kernel is 4.4.

I am understanding the problem correctly in regards to OVS being
responsible for these checksums when offloading is enabled?
Any pointers on how I can debug this further?
Why would just some of the eth+nsh packets exhibit this problem and not
all?
Why would these bad packets be ok after retransmissions?

[1] https://filebin.net/8mnypc2qm4vninof/client.pcap?t=b097kh0m
[2] https://filebin.net/8mnypc2qm4vninof/vnf_eth0.pcap?t=b097kh0m
[3] https://hastebin.com/nuhexufaze.sql
[4] https://hastebin.com/yevufanula.http

Thanks for your help,
Jaime.







___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] Bad checksums observed with nsh encapsulation

[Jaime Caamaño Ruiz]

Hello

I am facing a problem where eth+nsh encapsulated packets egress OVS
with incorrect checksum. 

The scenario is

client  vnf  server

all guests on the same host so this is vm2vm traffic, tap ports are
directly added to the ovs bridge. TCP traffic from/to server port 80 is
encapsulated with eth+nsh and traverse the vnf. I exercise the traffic
by using nc both on client and server.

I include captures at the client [1] and at the vnf [2] where I attempt
three tcp connections on port 80. The general observation is that
packets generated on client/server are seen there with wrong checksums
due to offloading but then arrive at the vnf with correct checksum. But
not all of them. For the first conenction attempt you can see that SYN
(frame 74) and ACK (78) are ok, but then FIN (79) is not ok. A
retransmitted FIN (80) is still not ok and then a further FIN (93)
retranmission is ok. Much of the same happens for the second attempt.
The third attempt shows a bad SYN (104) coming from the server.

Two additional observations:

- This does not happen if I try the same on a port different than 80 so
that the traffic goes directly from the client to the server with no
eth+nsh encapsulation.

- This does not happen if I disable tx offloading both in the server
and the client.

I include also the flows [3] and the ofproto trace [4] for the FIN
(79), generated by the client, which is eth+nsh encapsulated and
forwarded to the vnf. The decision on whether packet should be eth+nsh
encapsulated or no happens on table 101 by setting reg2 which is then
checked on 221. Packet is nsh encapsulated on table 222 and then
ethernet encapsulated on table 83. If not encapsulated packet would go
from 221 back to 220 and output there without any further actions.

Using OVS 2.9.2 with OVS tree kernel module. Kernel is 4.4.

I am understanding the problem correctly in regards to OVS being
responsible for these checksums when offloading is enabled?
Any pointers on how I can debug this further?
Why would just some of the eth+nsh packets exhibit this problem and not
all?
Why would these bad packets be ok after retransmissions?

[1] https://filebin.net/8mnypc2qm4vninof/client.pcap?t=b097kh0m
[2] https://filebin.net/8mnypc2qm4vninof/vnf_eth0.pcap?t=b097kh0m
[3] https://hastebin.com/nuhexufaze.sql
[4] https://hastebin.com/yevufanula.http

Thanks for your help,
Jaime.







___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [sfc-dev] NSH Patches for OVS 2.9.0

[Jaime Caamaño Ruiz]

Hello Srikanth

OVS 2.9 already has nsh support so those patches no longer apply.
Problem is that ODL SFC does not yet support OVS 2.9 nsh
implementation, support which is planned for next release Fluorine. If
you want to use ODL SFC I suggest that you stay with OVS 2.6.1 for now.

BR
Jaime.


-Original Message-
From: Srikanth Lingala 
To: sfc-...@lists.opendaylight.org ,
ovs-discuss@openvswitch.org , jcaamano@sus
e.de
Subject: [sfc-dev] NSH Patches for OVS 2.9.0
Date: Fri, 18 May 2018 11:31:20 +

Hi,
I am trying to apply NSH patches for OVS 2.9.0.
NSH patches are available for OVS 2.6.1 in the below link:
https://github.com/yyang13/ovs_nsh_patches.git
 
Does anyone have NSH patch set for OVS 2.9.0?
 
Regards,
Srikanth.
___
sfc-dev mailing list
sfc-...@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss