Hi Damjan,
On Wed, 2018-10-24 at 16:14 +0200, Damjan Marion via Lists.Fd.Io wrote:
> We merged patch which should fix things with 1G hugepages but I was not able
> to test it on arm, so please try...
Is this something which should go also on stable/1810 for a potential future dot
release?
If so could you please cherry pick it to that branch?
> --
> Damjan
Cheers,
Marco
>
>
> > On 24 Oct 2018, at 05:28, Sirshak Das <sirshak....@arm.com> wrote:
> > 2M works but 1G still fails.
> >
> > I toned down the dpdk resource allocation to default:
> > dpdk
> > {
> > dev 0004:01:00.1
> > dev 0004:01:00.2
> > no-multi-seg
> > log-level debug
> > dev default
> > {
> > num-rx-queues 1
> > # num-tx-queues 4
> > num-rx-desc 2048
> > num-tx-desc 2048
> > }
> > # num-mbufs 128000
> > # socket-mem 2048,2048
> > no-tx-checksum-offload
> >
> > But here is the problem (for 16G of Hugepage memory):
> > With:
> > 2MB (nr_hugepages: 8192)
> > GRUB_CMDLINE_LINUX="default_hugepagesz=2M hugepagesz=1G hugepages=16
> > hugepagesz=2M hugepages=8192 iommu.passthrough=1 isolcpus=16-45
> > nohz_full=16-45 rcu_nocbs=16-45"
> > vs 1GB (nr_hugepages: 16)
> > GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=16
> > hugepagesz=2M hugepages=8192 iommu.passthrough=1 isolcpus=16-45
> > nohz_full=16-45 rcu_nocbs=16-45"
> >
> > I am getting a performance improvement of 49% when I use 1G hugepages
> > compared to 2MB.
> > I am not an expert on hugepages to pinpoint the exact reason but it will
> > surely help if you can fix the 1G hugepage issue.
> >
> > Thank you
> > Sirshak Das
> > From: Damjan Marion <dmar...@me.com>
> > Sent: Tuesday, October 23, 2018 3:43 PM
> > To: Sirshak Das <sirshak....@arm.com>
> > Cc: vpp-dev <vpp-dev@lists.fd.io>; Honnappa Nagarahalli <
> > honnappa.nagaraha...@arm.com>; Lijian Zhang (Arm Technology China) <
> > lijian.zh...@arm.com>; khemendra kumar <khemendra.kuma...@gmail.com>; Juraj
> > Linkeš <juraj.lin...@pantheon.tech>
> > Subject: Re: [vpp-dev] running VPP non-root broken
> >
> >
> > OMG, you are good in wasting memory. 1G pages, 2G per socket given to dpdk
> > to hang empty :)
> > 128K buffers....
> >
> > For a start can you switch default page size to 2M. newer x86 kernels ignore
> > it but maybe it behaves
> > differently on aarch64...
> >
> > In the meantime I will fix few coverity issues...
> >
> > --
> > Damjan
> >
> >
> > > On 23 Oct 2018, at 20:45, Sirshak Das <sirshak....@arm.com> wrote:
> > >
> > > Hi Damjan,
> > >
> > > I am getting the following error as well I don’t know if its related to
> > > this issue:
> > > vlib_plugin_early_init:361: plugin path
> > > /home/sirdas/code/commita/vpp/build-root/install-vpp-
> > > native/vpp/lib/vpp_plugins
> > > load_one_plugin:117: Plugin disabled (default): abf_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): acl_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): avf_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): cdp_plugin.so
> > > load_one_plugin:189: Loaded plugin: dpdk_plugin.so (Data Plane Development
> > > Kit (DPDK))
> > > load_one_plugin:117: Plugin disabled (default): flowprobe_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): gbp_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): gtpu_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): igmp_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): ila_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): ioam_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): ixge_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): l2e_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): lacp_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): lb_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): mactime_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): map_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): memif_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): nat_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): nsh_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): nsim_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): perfmon_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): pppoe_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): srv6ad_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): srv6am_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): srv6as_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): stn_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): svs_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): tlsmbedtls_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): tlsopenssl_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): unittest_plugin.so
> > > load_one_plugin:117: Plugin disabled (default): vmxnet3_plugin.so
> > > clib_elf_parse_file: open `linux-vdso.so.1': No such file or directory
> > > load_one_vat_plugin:67: Loaded plugin: lb_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: gtpu_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: memif_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: nsh_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: nsim_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: vmxnet3_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: pppoe_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: lacp_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: dpdk_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: cdp_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: flowprobe_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: nat_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: stn_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: acl_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: ioam_test_plugin.so
> > > load_one_vat_plugin:67: Loaded plugin: map_test_plugin.so
> > > vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip6-fa' not found
> > > (before 'ip6-dvr-reinject', arc 'ip6-output')
> > > vnet_feature_arc_init:206: feature node 'nat44-in2out-output' not found
> > > (before 'ip4-dvr-reinject', arc 'ip4-output')
> > > vnet_feature_arc_init:206: feature node 'acl-plugin-out-ip4-fa' not found
> > > (before 'ip4-dvr-reinject', arc 'ip4-output')
> > > vlib_physmem_shared_map_create: pmalloc_map_pages: failed to mmap 153
> > > pages at 0xfffaa3c00000 fd 23 numa 0 flags 0x42031: Invalid argument
> > >
> > > dpdk_buffer_pool_create: failed to allocate mempool on socket 0
> > >
> > >
> > > This is the startup.conf I am using:
> > >
> > > ip
> > > {
> > > heap-size 4G
> > > }
> > > unix
> > > {
> > > nodaemon
> > > interactive
> > > cli-listen localhost:5002
> > > log /home/sirdas/var/log/vpp/vpp.log
> > > }
> > > ip6
> > > {
> > > heap-size 4G
> > > hash-buckets 2000000
> > > }
> > > heapsize 4G
> > > plugins
> > > {
> > > plugin default
> > > {
> > > disable
> > > }
> > > plugin dpdk_plugin.so
> > > {
> > > enable
> > > }
> > > }
> > > cpu
> > > {
> > > corelist-workers 18,20
> > > main-core 17
> > > }
> > > dpdk
> > > {
> > > dev 0004:01:00.1
> > > dev 0004:01:00.2
> > > no-multi-seg
> > > log-level debug
> > > dev default
> > > {
> > > num-rx-queues 2
> > > num-rx-desc 2048
> > > num-tx-desc 2048
> > > }
> > > num-mbufs 128000
> > > socket-mem 2048,2048
> > > no-tx-checksum-offload
> > > }
> > >
> > > More info for debugging:
> > > Boot parameters:
> > > default_hugepagesz=1G hugepagesz=1G hugepages=32 hugepagesz=2M
> > > hugepages=2048 iommu.passthrough=1 isolcpus=16-45 nohz_full=16-45
> > > rcu_nocbs=16-45
> > >
> > > OS & Kernel:
> > > Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-38-generic aarch64)
> > >
> > > Meminfo:
> > > $ cat /proc/meminfo
> > > MemTotal: 98827048 kB
> > > MemFree: 2802448 kB
> > > MemAvailable: 2730464 kB
> > > Buffers: 44960 kB
> > > Cached: 477832 kB
> > > SwapCached: 0 kB
> > > Active: 368112 kB
> > > Inactive: 232532 kB
> > > Active(anon): 79504 kB
> > > Inactive(anon): 1312 kB
> > > Active(file): 288608 kB
> > > Inactive(file): 231220 kB
> > > Unevictable: 0 kB
> > > Mlocked: 0 kB
> > > SwapTotal: 96505852 kB
> > > SwapFree: 96505852 kB
> > > Dirty: 24 kB
> > > Writeback: 0 kB
> > > AnonPages: 78024 kB
> > > Mapped: 144960 kB
> > > Shmem: 2956 kB
> > > Slab: 207728 kB
> > > SReclaimable: 67780 kB
> > > SUnreclaim: 139948 kB
> > > KernelStack: 10464 kB
> > > PageTables: 2300 kB
> > > NFS_Unstable: 0 kB
> > > Bounce: 0 kB
> > > WritebackTmp: 0 kB
> > > CommitLimit: 98733456 kB
> > > Committed_AS: 940272 kB
> > > VmallocTotal: 135290290112 kB
> > > VmallocUsed: 0 kB
> > > VmallocChunk: 0 kB
> > > HardwareCorrupted: 0 kB
> > > AnonHugePages: 0 kB
> > > ShmemHugePages: 0 kB
> > > ShmemPmdMapped: 0 kB
> > > CmaTotal: 0 kB
> > > CmaFree: 0 kB
> > > HugePages_Total: 86
> > > HugePages_Free: 86
> > > HugePages_Rsvd: 0
> > > HugePages_Surp: 0
> > > Hugepagesize: 1048576 kB
> > >
> > > Let me know if I am doing anything wrong:
> > >
> > > This is VPP (master branch) with HEAD at:
> > >
> > > commit 68b4da67deb2e8ca224bb5abaeb9dbc7ae8e378c (HEAD -> master,
> > > origin/master, origin/HEAD)
> > > Author: Damjan Marion <damar...@cisco.com>
> > > Date: Sun Sep 30 18:26:20 2018 +0200
> > >
> > > Numa-aware, growable physical memory allocator (pmalloc)
> > >
> > > Change-Id: Ic4c46bc733afae8bf0d8146623ed15633928de30
> > > Signed-off-by: Damjan Marion <damar...@cisco.com>
> > >
> > >
> > > Thank you
> > > Sirshak Das
> > > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Damjan Marion
> > > via Lists.Fd.Io
> > > Sent: Tuesday, October 23, 2018 11:40 AM
> > > To: vpp-dev <vpp-dev@lists.fd.io>
> > > Cc: vpp-dev@lists.fd.io
> > > Subject: [vpp-dev] running VPP non-root broken
> > >
> > >
> > > Folks,
> > >
> > > Looks like my big physmem patch breaks non-root operation of VPP, working
> > > on it
> > > and It will take a bit of time so as a workaround "make test" can be run
> > > with sudo.
> > >
> > > Let me know if any issues, and I will revert, but would like to avoid that
> > > due to the size of patch.
> > >
> > > --
> > > Damjan
> > >
> > > IMPORTANT NOTICE: The contents of this email and any attachments are
> > > confidential and may also be privileged. If you are not the intended
> > > recipient, please notify the sender immediately and do not disclose the
> > > contents to any other person, use it for any purpose, or store or copy the
> > > information in any medium. Thank you.
> >
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are
> > confidential and may also be privileged. If you are not the intended
> > recipient, please notify the sender immediately and do not disclose the
> > contents to any other person, use it for any purpose, or store or copy the
> > information in any medium. Thank you.
>
> -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
> View/Reply Online (#10957): https://lists.fd.io/g/vpp-dev/message/10957
> Mute This Topic: https://lists.fd.io/mt/27570325/675056
> Group Owner: vpp-dev+ow...@lists.fd.io
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [mvarl...@suse.de]-=-=-=-=-
> =-=-=-=-=-=-=-
--
Marco V
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10958): https://lists.fd.io/g/vpp-dev/message/10958
Mute This Topic: https://lists.fd.io/mt/27570325/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
