Before you start OVS with DPDK, if you execute cat /proc/meminfo how many
hugepages do you see available and how many free? (for 2mb I would assume 512
in both cases).
- In this specific case, I only saw 512 HugePages_Total and 0 free because
after the restart OVS was already using the 512 pages.
What memory commands are you passing to OVS with DPDK (e.g. dpdk-socket-mem parameter etc.)?
- Nothing, meaning the default memory of 1GB.
Is it just 1 bridge and a single DPDK interface you are adding or are there
more than 1 DPDK interface attached?
- There are 4 bridges in total but only one is using netdev datapath and
DPDK ports. Here is an extract of that one DPDK bridge:
Bridge lan-br
Port lan-br
Interface lan-br
type: internal
Port "dpdk-p0"
Interface "dpdk-p0"
type: dpdk
options: {dpdk-devargs="0000:08:0b.2"}
error: "could not add network device dpdk-p0 to ofproto (No such
device)"
Can you provide the entire log? I'd be interested in seeing the memory info at
initialization of OVS DPDK.
- I attached it to that mail.
What type of DPDK device are you adding? It seems to be a Virtual function from
the log above, can you provide more detail as regards the underlying NIC type
the VF is associated with?
- It's a VF. NIC type is Intel 710 and driver is i40
DPDK Version is 17.11.0
Thanks
Tobias
On 4/3/19, 9:07 AM, "Tobias Hofmann -T (tohofman - AAP3 INC at Cisco)"
<[email protected]> wrote:
Hi Ian,
To answer your questions, I just reproduced the issue on my system:
What commands have you used to configure the hugepage memory on your system?
- I have added the following kernel parameters: hugepagesz=2M
hugepages=512 and then rebooted the system. In other scenarios I allocated the
HugePages by writing into
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages.
The HugePages are also mounted on: hugetlbfs on /dev/hugepages type
hugetlbfs (rw,relatime,seclabel)
Before you start OVS with DPDK, if you execute cat /proc/meminfo how many hugepages do you see available and how many free? (for 2mb I would assume 512 in both cases).
- In this specific case, I only saw 512 HugePages_Total and 0 free because
after the restart OVS was already using the 512 pages.
What memory commands are you passing to OVS with DPDK (e.g. dpdk-socket-mem parameter etc.)?
On 4/3/19, 6:00 AM, "Ian Stokes" <[email protected]> wrote:
On 4/3/2019 1:04 AM, Tobias Hofmann -T (tohofman - AAP3 INC at Cisco)
via discuss wrote:
> Hello,
>
> I’m trying to attach a DPDK port with an mtu_size of 9216 to a
bridge.
> For this purpose, I have allocated 512 HugePages of size 2MB for OVS
> (1GB in total).
Hi,
I couldn't reproduce the behavior above on my own system with 512 x 2MB
hugepages. Ports were successfully configured with MTU 9216. Perhaps
some more detail as regards your setup will help reproduce/root cause.
Questions inline below.
What commands have you used to configure the hugepage memory on your system?
Before you start OVS with DPDK, if you execute cat /proc/meminfo how
many hugepages do you see available and how many free? (for 2mb I would
assume 512 in both cases).
What memory commands are you passing to OVS with DPDK (e.g.
dpdk-socket-mem parameter etc.)?
Is it just 1 bridge and a single DPDK interface you are adding or are
there more than 1 DPDK interface attached?
>
> Doing so will constantly fail, two workarounds to get it working were
> either to decrease the MTU size to 1500 or to increase the total
amount
> of HugePage memory to 3GB.
>
> Actually, I did expect the setup to also work with just 1GB because
if
> the amount of memory is not sufficient, OVS will try to halve the
number
> of buffers until 16K.
>
> However, inside the logs I couldn’t find any details regarding this.
The
> only error message I observed was:
>
> netdev_dpdk|ERR|Failed to create memory pool for netdev dpdk-p0, with
> MTU 9216 on socket 0: Invalid argument
Can you provide the entire log? I'd be interested in seeing the memory
info at initialization of OVS DPDK.
>
> That log message is weird as I would have expected an error message
> saying something like ‘could not reserve memory’ but not ‘Invalid
argument’.
>
> I then found this very similar bug on Openstack:
> https://bugs.launchpad.net/starlingx/+bug/1796380
>
> After having read this, I tried the exact same setup as described
above
> but this time with HugePages of size 1GB instead of 2MB. In this
> scenario, it also worked with just 1GB of memory reserved for OVS.
>
> Inside the logs I could observe this time:
>
> 2019-04-02T22:55:31.849Z|00098|dpdk|ERR|RING: Cannot reserve memory
>
> 2019-04-02T22:55:32.019Z|00099|dpdk|ERR|RING: Cannot reserve memory
>
> 2019-04-02T22:55:32.200Z|00100|netdev_dpdk|INFO|Virtual function
> detected, HW_CRRC_STRIP will be enabled
>
What type of DPDK device are you adding? It seems to be a Virtual
function from the log above, can you provide more detail as regards the
underlying NIC type the VF is associated with?
> 2019-04-02T22:55:32.281Z|00101|netdev_dpdk|INFO|Port 0: f6:e9:29:4d:f9:cf
>
> 2019-04-02T22:55:32.281Z|00102|dpif_netdev|INFO|Core 1 on numa node 0
> assigned port 'dpdk-p0' rx queue 0 (measured processing cycles 0).
>
> The two times where OVS cannot reserve memory are, I guess, the two
> times where it has to halve the number of buffers to get it working.
Yes this is correct. For example in my setup with 512 x 2MB pages I see
"Cannot reserve memory" message 4 times before it completes
configuration.
>
> My question now is, is the fact that it does not work for 2MB
HugePages
> a bug? Also, is the error message in the first log extract the
intended one?
>
Yes, it seems like a bug if it can be reproduced. The invalid argument
in this case would refer to the number of mbufs being requested by OVS
DPDK being less than the minimum allowed (4096 * 64). i.e. the amount
of
memory available cannot support the minimum 262,144 mbufs OVS DPDK
requires. The memory configuration at the system level is outside of
OVS
control however.
> *My version numbers:*
>
> * CentOS 7.6
> * Open vSwitch version: 2.9.3
> * DPDK version: 17.11
Is this DPDK 17.11.0? As an FYI The latest DPDK 17.11.x version
recommended for OVS is 17.11.4 with OVS 2.9.4, these typically include
bug fixes in DPDK so we recommend moving to them if possible..
Ian
> * System has a single NUMA node.
>
> Thank you
>
> Tobias
>
>
> _______________________________________________
> discuss mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
