Hi Ian, Just to close on this. Using openvswitch with version 2.9.3 and DPDK 17.11.5 fixed the issue.
Thanks for your help! On 4/16/19, 12:46 PM, "Ian Stokes" <[email protected]> wrote: On 4/3/2019 5:15 PM, Tobias Hofmann -T (tohofman - AAP3 INC at Cisco) wrote: > Sorry I just accidentally sent out the last mail too early... > > Hi Ian, > > To answer your questions, I just reproduced the issue on my system: > > What commands have you used to configure the hugepage memory on your system? > - I have added the following kernel parameters: hugepagesz=2M hugepages=512 and then rebooted the system. In other scenarios I allocated the HugePages by writing into /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages. The HugePages are also mounted on: hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) > Hi Tobias, I was able to reproduce the issue today. The error that occurs actually happens inside DPDK so as such it's not a bug in OVS. I was able to reproduce the issue with DPDK 17.11.0, 17.11.1 and 17.11.2. The issue itself seems to be resolved on my system by the following commit in DPDK 17.11.3: 222f91da4714 ("mem: do not use physical addresses in IOVA as VA mode") Since you are using OVS 2.9.3 I would suggest moving to use DPDK 17.11.4 at least. The mapping for OVS to DPDK versions is available in the OVS release notes: http://docs.openvswitch.org/en/latest/faq/releases/ Can you give this a try and see if it resolves the issue for you? HTH Ian > Before you start OVS with DPDK, if you execute cat /proc/meminfo how many hugepages do you see available and how many free? (for 2mb I would assume 512 in both cases). > - In this specific case, I only saw 512 HugePages_Total and 0 free because after the restart OVS was already using the 512 pages. > > What memory commands are you passing to OVS with DPDK (e.g. dpdk-socket-mem parameter etc.)? > - Nothing, meaning the default memory of 1GB. > > Is it just 1 bridge and a single DPDK interface you are adding or are there more than 1 DPDK interface attached? > - There are 4 bridges in total but only one is using netdev datapath and DPDK ports. Here is an extract of that one DPDK bridge: > Bridge lan-br > Port lan-br > Interface lan-br > type: internal > Port "dpdk-p0" > Interface "dpdk-p0" > type: dpdk > options: {dpdk-devargs="0000:08:0b.2"} > error: "could not add network device dpdk-p0 to ofproto (No such device)" > > Can you provide the entire log? I'd be interested in seeing the memory info at initialization of OVS DPDK. > - I attached it to that mail. > > What type of DPDK device are you adding? It seems to be a Virtual function from the log above, can you provide more detail as regards the underlying NIC type the VF is associated with? > - It's a VF. NIC type is Intel 710 and driver is i40 > > DPDK Version is 17.11.0 > > Thanks > Tobias > > On 4/3/19, 9:07 AM, "Tobias Hofmann -T (tohofman - AAP3 INC at Cisco)" <[email protected]> wrote: > > Hi Ian, > > To answer your questions, I just reproduced the issue on my system: > > What commands have you used to configure the hugepage memory on your system? > - I have added the following kernel parameters: hugepagesz=2M hugepages=512 and then rebooted the system. In other scenarios I allocated the HugePages by writing into /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages. > The HugePages are also mounted on: hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) > > Before you start OVS with DPDK, if you execute cat /proc/meminfo how many hugepages do you see available and how many free? (for 2mb I would assume 512 in both cases). > - In this specific case, I only saw 512 HugePages_Total and 0 free because after the restart OVS was already using the 512 pages. > > What memory commands are you passing to OVS with DPDK (e.g. dpdk-socket-mem parameter etc.)? > > On 4/3/19, 6:00 AM, "Ian Stokes" <[email protected]> wrote: > > On 4/3/2019 1:04 AM, Tobias Hofmann -T (tohofman - AAP3 INC at Cisco) > via discuss wrote: > > Hello, > > > > I’m trying to attach a DPDK port with an mtu_size of 9216 to a bridge. > > For this purpose, I have allocated 512 HugePages of size 2MB for OVS > > (1GB in total). > > Hi, > > I couldn't reproduce the behavior above on my own system with 512 x 2MB > hugepages. Ports were successfully configured with MTU 9216. Perhaps > some more detail as regards your setup will help reproduce/root cause. > > Questions inline below. > > What commands have you used to configure the hugepage memory on your system? > > Before you start OVS with DPDK, if you execute cat /proc/meminfo how > many hugepages do you see available and how many free? (for 2mb I would > assume 512 in both cases). > > What memory commands are you passing to OVS with DPDK (e.g. > dpdk-socket-mem parameter etc.)? > > Is it just 1 bridge and a single DPDK interface you are adding or are > there more than 1 DPDK interface attached? > > > > > Doing so will constantly fail, two workarounds to get it working were > > either to decrease the MTU size to 1500 or to increase the total amount > > of HugePage memory to 3GB. > > > > Actually, I did expect the setup to also work with just 1GB because if > > the amount of memory is not sufficient, OVS will try to halve the number > > of buffers until 16K. > > > > However, inside the logs I couldn’t find any details regarding this. The > > only error message I observed was: > > > > netdev_dpdk|ERR|Failed to create memory pool for netdev dpdk-p0, with > > MTU 9216 on socket 0: Invalid argument > > Can you provide the entire log? I'd be interested in seeing the memory > info at initialization of OVS DPDK. > > > > > That log message is weird as I would have expected an error message > > saying something like ‘could not reserve memory’ but not ‘Invalid argument’. > > > > I then found this very similar bug on Openstack: > > https://bugs.launchpad.net/starlingx/+bug/1796380 > > > > After having read this, I tried the exact same setup as described above > > but this time with HugePages of size 1GB instead of 2MB. In this > > scenario, it also worked with just 1GB of memory reserved for OVS. > > > > Inside the logs I could observe this time: > > > > 2019-04-02T22:55:31.849Z|00098|dpdk|ERR|RING: Cannot reserve memory > > > > 2019-04-02T22:55:32.019Z|00099|dpdk|ERR|RING: Cannot reserve memory > > > > 2019-04-02T22:55:32.200Z|00100|netdev_dpdk|INFO|Virtual function > > detected, HW_CRRC_STRIP will be enabled > > > > What type of DPDK device are you adding? It seems to be a Virtual > function from the log above, can you provide more detail as regards the > underlying NIC type the VF is associated with? > > > 2019-04-02T22:55:32.281Z|00101|netdev_dpdk|INFO|Port 0: f6:e9:29:4d:f9:cf > > > > 2019-04-02T22:55:32.281Z|00102|dpif_netdev|INFO|Core 1 on numa node 0 > > assigned port 'dpdk-p0' rx queue 0 (measured processing cycles 0). > > > > The two times where OVS cannot reserve memory are, I guess, the two > > times where it has to halve the number of buffers to get it working. > > Yes this is correct. For example in my setup with 512 x 2MB pages I see > "Cannot reserve memory" message 4 times before it completes configuration. > > > > > My question now is, is the fact that it does not work for 2MB HugePages > > a bug? Also, is the error message in the first log extract the intended one? > > > > Yes, it seems like a bug if it can be reproduced. The invalid argument > in this case would refer to the number of mbufs being requested by OVS > DPDK being less than the minimum allowed (4096 * 64). i.e. the amount of > memory available cannot support the minimum 262,144 mbufs OVS DPDK > requires. The memory configuration at the system level is outside of OVS > control however. > > > *My version numbers:* > > > > * CentOS 7.6 > > * Open vSwitch version: 2.9.3 > > * DPDK version: 17.11 > Is this DPDK 17.11.0? As an FYI The latest DPDK 17.11.x version > recommended for OVS is 17.11.4 with OVS 2.9.4, these typically include > bug fixes in DPDK so we recommend moving to them if possible.. > > Ian > > * System has a single NUMA node. > > > > Thank you > > > > Tobias > > > > > > _______________________________________________ > > discuss mailing list > > [email protected] > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > > > > _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
