Re: [ovs-discuss] OvS DPDK takes too many hugepages

[Tobias Hofmann (tohofman) via discuss]

Thanks Ilya, I can confirm this solves my problem.

Regards
Tobias

From: Ilya Maximets 
Date: Wednesday, 15. February 2023 at 14:56
To: Tobias Hofmann (tohofman) , ovs-discuss@openvswitch.org 

Cc: i.maxim...@ovn.org 
Subject: Re: [ovs-discuss] OvS DPDK takes too many hugepages
On 2/15/23 22:46, Tobias Hofmann (tohofman) via discuss wrote:
> Hello everyone,
>
>
>
> I’m enabling DPDK on a system and I don’t see any errors while doing so. 
> However, after enabling DPDK, I can see that my system has way less free 
> hugepages than expected. It should only be using 512 at that point but it’s 
> using 1914. OvS is the only potential candidate that can be consuming 
> hugepages at that time. I  tried to verify that it is indeed OvS that is 
> allocating these hugepages by checking each process under 
> /proc//smaps but surprisingly I did not find any process having 
> hugepages under its process ID. I wonder if there is something failing 
> internally in OvS that is messing up the hugepage allocation.
>
> Here are a few details of the system I’m using:
>
> OvS version: 2.17.3
> DPDK version: 21.11.0
> Kernel version: 4.18.0-372.9.1.el8.x86_64
>
> HugePages_Total:5120
> HugePages_Free: 3206
> HugePages_Rsvd:0
> HugePages_Surp:0
> Hugepagesize:   2048 kB
> Hugetlb:10485760 kB
>
> I noticed that the OvS and DPDK versions are not precisely matching with the 
> official support listed here: 
> https://docs.openvswitch.org/en/latest/faq/releases/ 
> <https://docs.openvswitch.org/en/latest/faq/releases/>
>
> Could that be a reason for this behavior?
>
> I’ve attached the log files of the ovs-vswitchd.log of the time where DPDK 
> gets enabled to this email.

Hi, Tobias.

According to the log, you have DPDK initialized with the following
configuration:

  EAL ARGS: ovs-vswitchd --iova-mode=va --socket-mem 1024 --in-memory -l 0.

You have socket-mem set to 1024, that means that DPDK will *pre-allocate* 1024MB
on start up.  But you don't have socket-limit option, so the actual memory
consumption is unlimited.  Hence, OVS can allocate as much memory as it wants.

OVS stopped configuring socket-limit equal to socket-mem starting from
release 2.17.  You need to explicitly set the other_config:dpdk-socket-limit
option in order to limit the amount of memory OVS is allowed to allocate.

Here is what release notes in the NEWS file are saying abut 2.17:

 * EAL argument --socket-limit no longer takes on the value of --socket-mem
   by default.  'other_config:dpdk-socket-limit' can be set equal to
   the 'other_config:dpdk-socket-mem' to preserve the legacy memory
   limiting behavior.

Best regards, Ilya Maximets.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OvS DPDK takes too many hugepages

[Tobias Hofmann (tohofman) via discuss]

Hello everyone,

I’m enabling DPDK on a system and I don’t see any errors while doing so. 
However, after enabling DPDK, I can see that my system has way less free 
hugepages than expected. It should only be using 512 at that point but it’s 
using 1914. OvS is the only potential candidate that can be consuming hugepages 
at that time. I  tried to verify that it is indeed OvS that is allocating these 
hugepages by checking each process under /proc//smaps but surprisingly 
I did not find any process having hugepages under its process ID. I wonder if 
there is something failing internally in OvS that is messing up the hugepage 
allocation.


Here are a few details of the system I’m using:
OvS version: 2.17.3
DPDK version: 21.11.0
Kernel version: 4.18.0-372.9.1.el8.x86_64

HugePages_Total:5120
HugePages_Free: 3206
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:   2048 kB
Hugetlb:10485760 kB


I noticed that the OvS and DPDK versions are not precisely matching with the 
official support listed here: 
https://docs.openvswitch.org/en/latest/faq/releases/
Could that be a reason for this behavior?

I’ve attached the log files of the ovs-vswitchd.log of the time where DPDK gets 
enabled to this email.


Thanks for any help!
Tobias

Tobias Hofmann
Software Engineer
tohof...@cisco.com
Tel: +49 6196 7739014
Cisco Systems, Inc.
United States
This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.
Update Profile - 
Privacy
Please click 
here
 for Company Registration



ovs-vswitchd.log
Description: ovs-vswitchd.log
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] "ovs-vsctl get openvswitch . other_config" results in "WARN connection dropped"

[Tobias Hofmann (tohofman) via discuss]

I just see the warning, however on our system we periodically run “ovs-vsctl 
get openvswitch . other_config” which bloats the log file over time. I’d like 
to prevent that.

From: Ben Pfaff 
Date: Monday, 1. February 2021 at 22:09
To: Tobias Hofmann (tohofman) 
Cc: ovs-discuss@openvswitch.org 
Subject: Re: [ovs-discuss] "ovs-vsctl get openvswitch . other_config" results 
in "WARN connection dropped"
On Mon, Feb 01, 2021 at 01:41:05PM +0000, Tobias Hofmann (tohofman) via discuss 
wrote:
> Hello everybody,
>
> I have Open vSwitch 2.11.1 installed with DPDK 18.11.0 and DB Schema 7.16.1.
> Every time I execute: “ovs-vsctl get open_vswitch . other_config”. I get the 
> following WARN message in ovsdb-server.log:
> 2021-02-01T13:31:21.773Z|07709|jsonrpc|WARN|unix#7443: receive error: 
> Connection reset by peer
> 2021-02-01T13:31:21.774Z|07710|reconnect|WARN|unix#7443: connection dropped 
> (Connection reset by peer)
>
> When running: ”ovs-vsctl show” I don’t see any of these WARN logs.
> Interestingly, when running: “ovs-vsctl -vjsonrpc get open_vswitch . 
> other_config”. I also DO NOT see any WARN log messages.
>
> Does anyone have an idea what causes this issue and how to fix it?

Do you see some actual issue, or just a warning message in the log?  The
most probable reason for the message is that ovsdb-server has some RPC
message queued up to send ovs-vsctl, but ovs-vsctl is done and exits
without reading it.  Probably, adding -vjsonrpc slows ovs-vsctl down
enough that it reads that RPC, suppressing the log.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] "ovs-vsctl get openvswitch . other_config" results in "WARN connection dropped"

[Tobias Hofmann (tohofman) via discuss]

Hello everybody,

I have Open vSwitch 2.11.1 installed with DPDK 18.11.0 and DB Schema 7.16.1.
Every time I execute: “ovs-vsctl get open_vswitch . other_config”. I get the 
following WARN message in ovsdb-server.log:
2021-02-01T13:31:21.773Z|07709|jsonrpc|WARN|unix#7443: receive error: 
Connection reset by peer
2021-02-01T13:31:21.774Z|07710|reconnect|WARN|unix#7443: connection dropped 
(Connection reset by peer)

When running: ”ovs-vsctl show” I don’t see any of these WARN logs.
Interestingly, when running: “ovs-vsctl -vjsonrpc get open_vswitch . 
other_config”. I also DO NOT see any WARN log messages.

Does anyone have an idea what causes this issue and how to fix it?

In case anyone is interested, the output of “ovs-vsctl -vjsonrpc get 
open_vswitch . other_config” is:
2021-02-01T13:33:06Z|1|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: send 
request, method="get_schema", params=["_Server"], id=0
2021-02-01T13:33:06Z|2|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: 
received reply, result={"cksum":"3236486585 
698","name":"_Server","version":"1.1.0","tables":{"Database":{"columns":{"model":{"type":{"key":{"type":"string","enum":["set",["clustered","standalone"]]}}},"name":{"type":"string"},"connected":{"type":"boolean"},"leader":{"type":"boolean"},"schema":{"type":{"min":0,"key":"string"}},"sid":{"type":{"min":0,"key":"uuid"}},"cid":{"type":{"min":0,"key":"uuid"}},"index":{"type":{"min":0,"key":"integer"}},
 id=0
2021-02-01T13:33:06Z|3|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: send 
request, method="monitor_cond", 
params=["_Server",["monid","_Server"],{"Database":[{"where":[["name","==","Open_vSwitch"]],"columns":["cid","connected","index","leader","model","name","schema","sid"]}]}],
 id=1
2021-02-01T13:33:07Z|4|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock: 
received reply, 
result={"Database":{"fe27ef99-3410-426b-a71d-4d7314e7dfb8":{"initial":{"leader":true,"schema":"{\"cksum\":\"1452282319
 

Re: [ovs-discuss] OVS DPDK: Failed to create memory pool for netdev

[Tobias Hofmann (tohofman) via discuss]

Hi Flavio,

to follow up on this: I have just upgraded DPDK to 18.11 and OVS to 2.11 and I 
don't see this issue anymore. Also, I don't observe any "ring error" messages 
although the MTU is still at 9216 and OvS only has 1Gb of memory.
Do you have an idea which change in DPDK/OvS might have resolved it?

Thanks
Tobias

On 06.11.19, 14:44, "Tobias Hofmann (tohofman)"  wrote:

Hi Flavio,

the only error I saw in 'ovs-vsctl show' was related to the dpdk port. The 
other ports all came up fine.

Regarding the "ring error", I'm fine with having it, as long as DPDK is 
able to reserve the minimum amount of memory (which, after restarting OvS 
process is always the case).

Regards
Tobias

On 05.11.19, 21:07, "Flavio Leitner"  wrote:

On Tue, 5 Nov 2019 18:47:09 +0000
    "Tobias Hofmann \(tohofman\) via discuss" 
wrote:

> Hi Flavio,
> 
> thanks for the insights! Unfortunately, I don't know about the pdump
> and its relation to the ring.

pdump dumps packets from dpdk ports into rings/mempools, so that you
can inspect/use the traffic:
https://doc.dpdk.org/guides/howto/packet_capture_framework.html

But I looked at the dpdk sources now and I don't see it allocating any
memory when the library is initialized, so this is likely a red herring.

> Can you please specify where I can see that the port is not ready
> yet? Is that these three lines:
> 
> 2019-11-02T14:14:23.094Z|00070|dpdk|ERR|EAL: Cannot find unplugged
> device (:08:0b.2)

The above shows the device is not ready/bound yet.


> 2019-11-02T14:14:23.094Z|00071|netdev_dpdk|WARN|Error attaching
> device ':08:0b.2' to DPDK
> 2019-11-02T14:14:23.094Z|00072|netdev|WARN|dpdk-p0: could not set
> configuration (Invalid argument)
> 
> As far as I know, the ring allocation failure that you mentioned
> isn't necessarily a bad thing since it just indicates that DPDK
> reduces something internally (I can't remember what exactly it was)
> to support a high MTU with only 1GB of memory.

True for the memory allocated for DPDK ports. However, there is a
minimum which if it's not there, the mempool allocation will fail.

> I'm wondering now if it might help to change the timing of when
> openvswitch is started after a system reboot to prevent this problem
> as it only occurs after reboot. Do you think that this approach might
> fix the problem?

It will help to get the i40e port working, but that "ring error"
will continue as you see after restarting anyways.

I don't know the other interface types, maybe there is another
interface failing which is not in the log. Do you see any error
reported in 'ovs-vsctl show' after the restart?

fbl




___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS DPDK: Failed to create memory pool for netdev

[Tobias Hofmann (tohofman) via discuss]

Hi Flavio,

the only error I saw in 'ovs-vsctl show' was related to the dpdk port. The 
other ports all came up fine.

Regarding the "ring error", I'm fine with having it, as long as DPDK is able to 
reserve the minimum amount of memory (which, after restarting OvS process is 
always the case).

Regards
Tobias

On 05.11.19, 21:07, "Flavio Leitner"  wrote:

On Tue, 5 Nov 2019 18:47:09 +
    "Tobias Hofmann \(tohofman\) via discuss" 
wrote:

> Hi Flavio,
> 
> thanks for the insights! Unfortunately, I don't know about the pdump
> and its relation to the ring.

pdump dumps packets from dpdk ports into rings/mempools, so that you
can inspect/use the traffic:
https://doc.dpdk.org/guides/howto/packet_capture_framework.html

But I looked at the dpdk sources now and I don't see it allocating any
memory when the library is initialized, so this is likely a red herring.

> Can you please specify where I can see that the port is not ready
> yet? Is that these three lines:
> 
> 2019-11-02T14:14:23.094Z|00070|dpdk|ERR|EAL: Cannot find unplugged
> device (:08:0b.2)

The above shows the device is not ready/bound yet.


> 2019-11-02T14:14:23.094Z|00071|netdev_dpdk|WARN|Error attaching
> device ':08:0b.2' to DPDK
> 2019-11-02T14:14:23.094Z|00072|netdev|WARN|dpdk-p0: could not set
> configuration (Invalid argument)
> 
> As far as I know, the ring allocation failure that you mentioned
> isn't necessarily a bad thing since it just indicates that DPDK
> reduces something internally (I can't remember what exactly it was)
> to support a high MTU with only 1GB of memory.

True for the memory allocated for DPDK ports. However, there is a
minimum which if it's not there, the mempool allocation will fail.

> I'm wondering now if it might help to change the timing of when
> openvswitch is started after a system reboot to prevent this problem
> as it only occurs after reboot. Do you think that this approach might
> fix the problem?

It will help to get the i40e port working, but that "ring error"
will continue as you see after restarting anyways.

I don't know the other interface types, maybe there is another
interface failing which is not in the log. Do you see any error
reported in 'ovs-vsctl show' after the restart?

fbl


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS DPDK: Failed to create memory pool for netdev

[Tobias Hofmann (tohofman) via discuss]

Hi Flavio,

thanks for the insights! Unfortunately, I don't know about the pdump and its 
relation to the ring.

Can you please specify where I can see that the port is not ready yet? Is that 
these three lines:

2019-11-02T14:14:23.094Z|00070|dpdk|ERR|EAL: Cannot find unplugged device 
(:08:0b.2)
2019-11-02T14:14:23.094Z|00071|netdev_dpdk|WARN|Error attaching device 
':08:0b.2' to DPDK
2019-11-02T14:14:23.094Z|00072|netdev|WARN|dpdk-p0: could not set configuration 
(Invalid argument)

As far as I know, the ring allocation failure that you mentioned isn't 
necessarily a bad thing since it just indicates that DPDK reduces something 
internally (I can't remember what exactly it was) to support a high MTU with 
only 1GB of memory.

I'm wondering now if it might help to change the timing of when openvswitch is 
started after a system reboot to prevent this problem as it only occurs after 
reboot. Do you think that this approach might fix the problem?

Thanks for your help
Tobias

On 05.11.19, 14:08, "Flavio Leitner"  wrote:

On Mon, 4 Nov 2019 19:12:36 +
"Tobias Hofmann (tohofman)"  wrote:

> Hi Flavio,
> 
> thanks for reaching out.
> 
> The DPDK options used in OvS are:
> 
> other_config:pmd-cpu-mask=0x202
> other_config:dpdk-socket-mem=1024
> other_config:dpdk-init=true
> 
> 
> For the dpdk port, we set:
> 
> type=dpdk
> options:dpdk-devargs=:08:0b.2
> external_ids:unused-drv=i40evf 
> mtu_request=9216

Looks good to me, though the CPU has changed comparing to the log:
2019-11-02T14:51:26.940Z|00010|dpdk|INFO|EAL ARGS: ovs-vswitchd
--socket-mem 1024 -c 0x0001

What I see from the logs is that OvS is trying to add a port, but the
port is not ready yet, so it continues with other things which
also consumes memory. Unfortunately by the time that the i40 port is
ready then there is no memory.

When you restart, the i40 is ready and the memory can be allocated.
However, the ring allocation fails due to lack of memory:

2019-11-02T14:51:27.808Z|00136|dpdk|ERR|RING: Cannot reserve memory
2019-11-02T14:51:27.974Z|00137|dpdk|ERR|RING: Cannot reserve memory

If you reduce the MTU, then the minimum amount of memory required for
the DPDK port reduces drastically, which explains why it works.

Also increasing the total memory to 2G helps because then the minimum
amount for 9216 MTU and the ring seems to be sufficient.

The ring seems to be related to pdump, is that the case?
I don't known of the top of my head.

In summary, looks like 1G is not enough for large MTU and pdump.
HTH,
fbl

> 
> 
> Please let me know if this is what you asked for.
> 
> Thanks
> Tobias
>   
> On 04.11.19, 15:50, "Flavio Leitner"  wrote:
> 
> 
> It would be nice if you share the DPDK options used in OvS.
> 
> On Sat, 2 Nov 2019 15:43:18 +
> "Tobias Hofmann \(tohofman\) via discuss"
>  wrote:
> 
> > Hello community,
> > 
> > My team and I observe a strange behavior on our system with the
> > creation of dpdk ports in OVS. We have a CentOS 7 system with
> > OpenvSwitch and only one single port of type ‘dpdk’ attached to
> > a bridge. The MTU size of the DPDK port is 9216 and the reserved
> > HugePages for OVS are 512 x 2MB-HugePages, e.g. 1GB of total
> > HugePage memory.
> > 
> > Setting everything up works fine, however after I reboot my
> > box, the dpdk port is in error state and I can observe this
> > line in the logs (full logs attached to the mail):
> > 2019-11-02T14:46:16.914Z|00437|netdev_dpdk|ERR|Failed to create
> > memory pool for netdev dpdk-p0, with MTU 9216 on socket 0:
> > Invalid argument
> > 2019-11-02T14:46:16.914Z|00438|dpif_netdev|ERR|Failed to set
> > interface dpdk-p0 new configuration
> > 
> > I figured out that by restarting the openvswitch process, the
> > issue with the port is resolved and it is back in a working
> > state. However, as soon as I reboot the system a second time,
> > the port comes up in error state again. Now, we have also
> > observed a couple of other workarounds that I can’t really
> > explain why they help:
> > 
> >   *   When there is also a VM deployed on the system that is
> > using ports of type ‘dpdkvhostuserclient’, we never see any
> > issues like that. (MTU size of 

Re: [ovs-discuss] OVS DPDK: Failed to create memory pool for netdev

[Tobias Hofmann (tohofman) via discuss]

Hi Flavio,

thanks for reaching out.

The DPDK options used in OvS are:

other_config:pmd-cpu-mask=0x202
other_config:dpdk-socket-mem=1024
other_config:dpdk-init=true


For the dpdk port, we set:

type=dpdk
options:dpdk-devargs=:08:0b.2
external_ids:unused-drv=i40evf 
mtu_request=9216


Please let me know if this is what you asked for.

Thanks
Tobias

On 04.11.19, 15:50, "Flavio Leitner"  wrote:


It would be nice if you share the DPDK options used in OvS.

On Sat, 2 Nov 2019 15:43:18 +
    "Tobias Hofmann \(tohofman\) via discuss" 
wrote:

> Hello community,
> 
> My team and I observe a strange behavior on our system with the
> creation of dpdk ports in OVS. We have a CentOS 7 system with
> OpenvSwitch and only one single port of type ‘dpdk’ attached to a
> bridge. The MTU size of the DPDK port is 9216 and the reserved
> HugePages for OVS are 512 x 2MB-HugePages, e.g. 1GB of total HugePage
> memory.
> 
> Setting everything up works fine, however after I reboot my box, the
> dpdk port is in error state and I can observe this line in the logs
> (full logs attached to the mail):
> 2019-11-02T14:46:16.914Z|00437|netdev_dpdk|ERR|Failed to create
> memory pool for netdev dpdk-p0, with MTU 9216 on socket 0: Invalid
> argument 2019-11-02T14:46:16.914Z|00438|dpif_netdev|ERR|Failed to set
> interface dpdk-p0 new configuration
> 
> I figured out that by restarting the openvswitch process, the issue
> with the port is resolved and it is back in a working state. However,
> as soon as I reboot the system a second time, the port comes up in
> error state again. Now, we have also observed a couple of other
> workarounds that I can’t really explain why they help:
> 
>   *   When there is also a VM deployed on the system that is using
> ports of type ‘dpdkvhostuserclient’, we never see any issues like
> that. (MTU size of the VM ports is 9216 by the way)
>   *   When we increase the HugePage memory for OVS to 2GB, we also
> don’t see any issues.
>   *   Lowering the MTU size of the ‘dpdk’ type port to 1500 also
> helps to prevent this issue.
> 
> Can anyone explain this?
> 
> We’re using the following versions:
> Openvswitch: 2.9.3
> DPDK: 17.11.5
> 
> Appreciate any help!
> Tobias



___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] Link Control capabilities for DPDK ports

[Tobias Hofmann (tohofman) via discuss]

Hello community,

I need help in figuring out how if OpenvSwitch at the moment offers any 
capabilities to set the link speed and the duplex settings of a physical DPDK 
NIC.
For a non-DPDK port, there is ethtool however this only works for ports in the 
kernel space. The DPDK library offers a function to configure link speed but 
I’m wondering if it is already integrated into OVS’s source code and thus 
configurable through the OVS interface.

I saw on this website 
here some information 
on Flow Control but nothing related to link speed.

I highly appreciate your help!
Thanks in advance.

Tobias

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss