On Wed, 24 Oct 2018 16:27:24 +0000 "Sirshak Das" <sirshak....@arm.com> wrote:
> FYI just to be on same page things are not broken in 1810 so I don’t think > its needed. Its only broken in current master. > > From: Marco Varlese <mvarl...@suse.de> > Sent: Wednesday, October 24, 2018 9:18 AM > To: dmar...@me.com; Sirshak Das <sirshak....@arm.com> > Cc: vpp-dev@lists.fd.io > Subject: Re: [vpp-dev] running VPP non-root broken > > Hi Damjan, > > On Wed, 2018-10-24 at 16:14 +0200, Damjan Marion via Lists.Fd.Io wrote: > > We merged patch which should fix things with 1G hugepages but I was not able > to test it on arm, so please try... > > Is this something which should go also on stable/1810 for a potential future > dot release? > If so could you please cherry pick it to that branch? > > > > -- > Damjan > > Cheers, > Marco > > > > On 24 Oct 2018, at 05:28, Sirshak Das > <sirshak....@arm.com<mailto:sirshak....@arm.com>> wrote: > 2M works but 1G still fails. > > I toned down the dpdk resource allocation to default: > dpdk > { > dev 0004:01:00.1 > dev 0004:01:00.2 > no-multi-seg > log-level debug > dev default > { > num-rx-queues 1 > # num-tx-queues 4 > num-rx-desc 2048 > num-tx-desc 2048 > } > # num-mbufs 128000 > # socket-mem 2048,2048 > no-tx-checksum-offload > > But here is the problem (for 16G of Hugepage memory): > With: > 2MB (nr_hugepages: 8192) > GRUB_CMDLINE_LINUX="default_hugepagesz=2M hugepagesz=1G hugepages=16 > hugepagesz=2M hugepages=8192 iommu.passthrough=1 isolcpus=16-45 > nohz_full=16-45 rcu_nocbs=16-45" > vs 1GB (nr_hugepages: 16) > GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=16 > hugepagesz=2M hugepages=8192 iommu.passthrough=1 isolcpus=16-45 > nohz_full=16-45 rcu_nocbs=16-45" > > I am getting a performance improvement of 49% when I use 1G hugepages > compared to 2MB. > I am not an expert on hugepages to pinpoint the exact reason but it will > surely help if you can fix the 1G hugepage issue. > > Thank you > Sirshak Das > From: Damjan Marion <dmar...@me.com<mailto:dmar...@me.com>> > Sent: Tuesday, October 23, 2018 3:43 PM > To: Sirshak Das <sirshak....@arm.com<mailto:sirshak....@arm.com>> > Cc: vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>>; Honnappa > Nagarahalli > <honnappa.nagaraha...@arm.com<mailto:honnappa.nagaraha...@arm.com>>; Lijian > Zhang (Arm Technology China) > <lijian.zh...@arm.com<mailto:lijian.zh...@arm.com>>; khemendra kumar > <khemendra.kuma...@gmail.com<mailto:khemendra.kuma...@gmail.com>>; Juraj > Linkeš <juraj.lin...@pantheon.tech<mailto:juraj.lin...@pantheon.tech>> > Subject: Re: [vpp-dev] running VPP non-root broken > > > OMG, you are good in wasting memory. 1G pages, 2G per socket given to dpdk to > hang empty :) > 128K buffers.... > > For a start can you switch default page size to 2M. newer x86 kernels ignore > it but maybe it behaves > differently on aarch64... > > In the meantime I will fix few coverity issues... > > -- > Damjan > > > > On 23 Oct 2018, at 20:45, Sirshak Das > <sirshak....@arm.com<mailto:sirshak....@arm.com>> wrote: > > Hi Damjan, > > I am getting the following error as well I don’t know if its related to this > issue: > vlib_plugin_early_init:361: plugin path > /home/sirdas/code/commita/vpp/build-root/install-vpp-native/vpp/lib/vpp_plugins > load_one_plugin:117: Plugin disabled (default): abf_plugin.so > load_one_plugin:117: Plugin disabled (default): acl_plugin.so > load_one_plugin:117: Plugin disabled (default): avf_plugin.so > load_one_plugin:117: Plugin disabled (default): cdp_plugin.so > load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development > Kit (DPDK)) > load_one_plugin:117: Plugin disabled (default): flowprobe_plugin.so > load_one_plugin:117: Plugin disabled (default): gbp_plugin.so > load_one_plugin:117: Plugin disabled (default): gtpu_plugin.so > load_one_plugin:117: Plugin disabled (default): igmp_plugin.so > load_one_plugin:117: Plugin disabled (default): ila_plugin.so > load_one_plugin:117: Plugin disabled (default): ioam_plugin.so > load_one_plugin:117: Plugin disabled (default): ixge_plugin.so > load_one_plugin:117: Plugin disabled (default): l2e_plugin.so > load_one_plugin:117: Plugin disabled (default): lacp_plugin.so > load_one_plugin:117: Plugin disabled (default): lb_plugin.so > load_one_plugin:117: Plugin disabled (default): mactime_plugin.so > load_one_plugin:117: Plugin disabled (default): map_plugin.so > load_one_plugin:117: Plugin disabled (default): memif_plugin.so > load_one_plugin:117: Plugin disabled (default): nat_plugin.so > load_one_plugin:117: Plugin disabled (default): nsh_plugin.so > load_one_plugin:117: Plugin disabled (default): nsim_plugin.so > load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so > load_one_plugin:117: Plugin disabled (default): pppoe_plugin.so > load_one_plugin:117: Plugin disabled (default): srv6ad_plugin.so > load_one_plugin:117: Plugin disabled (default): srv6am_plugin.so > load_one_plugin:117: Plugin disabled (default): srv6as_plugin.so > load_one_plugin:117: Plugin disabled (default): stn_plugin.so > load_one_plugin:117: Plugin disabled (default): svs_plugin.so > load_one_plugin:117: Plugin disabled (default): tlsmbedtls_plugin.so > load_one_plugin:117: Plugin disabled (default): tlsopenssl_plugin.so > load_one_plugin:117: Plugin disabled (default): unittest_plugin.so > load_one_plugin:117: Plugin disabled (default): vmxnet3_plugin.so > clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory > load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so > load_one_vat_plugin:67: Loaded plugin: map_test_plugin.so > vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip6-fa' not found > (before 'ip6-dvr-reinject', arc 'ip6-output') > vnet_feature_arc_init:206: feature node 'nat44-in2out-output' not found > (before 'ip4-dvr-reinject', arc 'ip4-output') > vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip4-fa' not found > (before 'ip4-dvr-reinject', arc 'ip4-output') > vlib_physmem_shared_map_create: pmalloc_map_pages: failed to mmap 153 pages > at 0xfffaa3c00000 fd 23 numa 0 flags 0x42031: Invalid argument > > dpdk_buffer_pool_create: failed to allocate mempool on socket 0 > > > This is the startup.conf I am using: > > ip > { > heap-size 4G > } > unix > { > nodaemon > interactive > cli-listen localhost:5002 > log /home/sirdas/var/log/vpp/vpp.log > } > ip6 > { > heap-size 4G > hash-buckets 2000000 > } > heapsize 4G > plugins > { > plugin default > { > disable > } > plugin dpdk_plugin.so > { > enable > } > } > cpu > { > corelist-workers 18,20 > main-core 17 > } > dpdk > { > dev 0004:01:00.1 > dev 0004:01:00.2 > no-multi-seg > log-level debug > dev default > { > num-rx-queues 2 > num-rx-desc 2048 > num-tx-desc 2048 > } > num-mbufs 128000 > socket-mem 2048,2048 > no-tx-checksum-offload > } > > More info for debugging: > Boot parameters: > default_hugepagesz=1G hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=2048 > iommu.passthrough=1 isolcpus=16-45 nohz_full=16-45 rcu_nocbs=16-45 > > OS & Kernel: > Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-38-generic aarch64) > > Meminfo: > $ cat /proc/meminfo > MemTotal: 98827048 kB > MemFree: 2802448 kB > MemAvailable: 2730464 kB > Buffers: 44960 kB > Cached: 477832 kB > SwapCached: 0 kB > Active: 368112 kB > Inactive: 232532 kB > Active(anon): 79504 kB > Inactive(anon): 1312 kB > Active(file): 288608 kB > Inactive(file): 231220 kB > Unevictable: 0 kB > Mlocked: 0 kB > SwapTotal: 96505852 kB > SwapFree: 96505852 kB > Dirty: 24 kB > Writeback: 0 kB > AnonPages: 78024 kB > Mapped: 144960 kB > Shmem: 2956 kB > Slab: 207728 kB > SReclaimable: 67780 kB > SUnreclaim: 139948 kB > KernelStack: 10464 kB > PageTables: 2300 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 98733456 kB > Committed_AS: 940272 kB > VmallocTotal: 135290290112<tel:135290290112> kB > VmallocUsed: 0 kB > VmallocChunk: 0 kB > HardwareCorrupted: 0 kB > AnonHugePages: 0 kB > ShmemHugePages: 0 kB > ShmemPmdMapped: 0 kB > CmaTotal: 0 kB > CmaFree: 0 kB > HugePages_Total: 86 > HugePages_Free: 86 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 1048576<tel:1048576> kB > > Let me know if I am doing anything wrong: > > This is VPP (master branch) with HEAD at: > > commit 68b4da67deb2e8ca224bb5abaeb9dbc7ae8e378c (HEAD -> master, > origin/master, origin/HEAD) > Author: Damjan Marion <damar...@cisco.com<mailto:damar...@cisco.com>> > Date: Sun Sep 30 18:26:20 2018 +0200 > > Numa-aware, growable physical memory allocator (pmalloc) > > Change-Id: Ic4c46bc733afae8bf0d8146623ed15633928de30 > Signed-off-by: Damjan Marion > <damar...@cisco.com<mailto:damar...@cisco.com>> > > > Thank you > Sirshak Das > From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> > <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion > via Lists.Fd.Io > Sent: Tuesday, October 23, 2018 11:40 AM > To: vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> > Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> > Subject: [vpp-dev] running VPP non-root broken > > > Folks, > > Looks like my big physmem patch breaks non-root operation of VPP, working on > it > and It will take a bit of time so as a workaround "make test" can be run with > sudo. > > Let me know if any issues, and I will revert, but would like to avoid that > due to the size of patch. > This change broke my system. Something is not right. Even just doing default startup fails: vlib_pci_bind_to_uio: Skipping PCI device 0000:04:00.0: missing kernel VFIO or UIO driver vlib_pci_bind_to_uio: Skipping PCI device 0000:07:00.0: missing kernel VFIO or UIO driver vlib_pci_bind_to_uio: Skipping PCI device 0000:08:00.0: missing kernel VFIO or UIO driver [New Thread 0x7ffab9754700 (LWP 8759)] [New Thread 0x7ffab8f53700 (LWP 8760)] EAL: FATAL: Cannot init memory dpdk_config: rte_eal_init returned -1 [Thread 0x7ffab9754700 (LWP 8759) exited] [Thread 0x7ffff7fda740 (LWP 8755) exited] [Inferior 1 (process 8755) exited normally]
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10967): https://lists.fd.io/g/vpp-dev/message/10967 Mute This Topic: https://lists.fd.io/mt/27570325/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
