Hi Dave and Damjan, Yes, as I confirmed before, the patch works, but I just discovered a side effect introduced by this patch.
That is, DPDK in-memory implys no shared config, then rte_eal_init/rte_eal_config_create will not check if another DPDK process is running. Therefore, we could run multiple vPP processes. I think it is not an expected behavior. Regards, Kingwel -------- 原始邮件 -------- 主题： Re: [vpp-dev] dpdk: switch to in-memory mode, deprecate use of socket-mem 来自： "Dave Barach via Lists.Fd.Io" <dbarach=cisco....@lists.fd.io> 发至： 2018年12月23日 上午12:37 抄送： Kingwel Xie <kingwel....@ericsson.com>,dmar...@me.com,Matthew Smith <mgsm...@netgate.com> Patch is in master/latest as of a couple of minutes ago... D. From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Kingwel Xie Sent: Friday, December 21, 2018 9:59 PM To: Kingwel Xie <kingwel....@ericsson.com>; dmar...@me.com; Matthew Smith <mgsm...@netgate.com> Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] dpdk: switch to in-memory mode, deprecate use of socket-mem Hi, After checking out the doc, code and making some tests, I have confirmed patch 16543 made by Damjan is working well, for both 2MB and 1GB cases. DPDK itself can grow or shrink based on request, the vPP master code base doesn’t allow this simple because vPP unmount & rmdir the hugepage mount point. Patch 16543 corrects this behavior, and makes life easier. Hi Matthew, You must be running master code without the patch, and you probably only have 2MB hugepages, so you see the allocation error when you are running out of socket-mem. Please switch to patch 16543 or specify how much you need with socket-mem option. Hi Damjan, Thanks for the heads-up. Regards, Kingwel From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Kingwel Xie Sent: Friday, December 21, 2018 11:04 AM To: dmar...@me.com<mailto:dmar...@me.com>; Matthew Smith <mgsm...@netgate.com<mailto:mgsm...@netgate.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Non-compliant mail �C action required, contact dm...@ericsson.se<mailto:dm...@ericsson.se> : Re: [vpp-dev] dpdk: switch to in-memory mode, deprecate use of socket-mem Hi Matthew, The patch (https://gerrit.fd.io/r/#/c/16287/) was intended to allocate crypto mem pool from DPDK, instead of from vPP. I guess you are using 2MB huge page, so you are experiencing out of memory with new patch created by Damjan. Please switch to 1GB, to see if it still happens. Hi Damjan, As I understand, DPDK could dynamically allocate mem segment when huge page is 1GB and there is still free space. In this case the socket-mem is valid always. Therefore, please do not merge the patch (https://gerrit.fd.io/r/#/c/16543/), it won’t work for 2MB case. Regards, Kingwel From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Damjan Marion via Lists.Fd.Io Sent: Friday, December 21, 2018 2:02 AM To: Matthew Smith <mgsm...@netgate.com<mailto:mgsm...@netgate.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] dpdk: switch to in-memory mode, deprecate use of socket-mem On 20 Dec 2018, at 18:46, Matthew Smith <mgsm...@netgate.com<mailto:mgsm...@netgate.com>> wrote: Hi Damjan, There is a comment that says "preallocate at least 16MB of hugepages per socket, if more is needed it is up to consumer to preallocate more". What does a consumer need to do in order to preallocate more? This is just equivalent of "sysctl -w vm.nr_hugepages=X", if you set more with sysctl or kernel boot parameters, vpp will not try to change that. We can make this configurable from startup.conf, but in reality it is just convenience feature. I've recently had problems using AES-GCM with IPsec on a test system. Mempool allocation for the DPDK aesni_gcm crypto vdev fails during initialization. An error message like this is logged: vnet[4765]: crypto_create_session_drv_pool: failed to create session drv mempool I found that the DPDK memory allocation set with socket-mem is 64 MB by default and this is being exhausted by the various mempool allocations. This behavior seems to have started with change https://gerrit.fd.io/r/#/c/16287/ ("dpdk-ipsec-mempool: allocate from dpdk mem specified by socket-mem in startup.conf"). When I increase socket-mem in startup.conf to 128 MB, the configured crypto devices successfully initialize. you can see what exactly dpdk allocated with "show dpdk physmem". When support for socket-mem goes away, what will need to be done to ensure that enough memory is available for crypto devices? DPDK is supposed to dynamically alloc more memory if needed, Unless I missed something... Thanks, -Matt On Thu, Dec 20, 2018 at 3:55 AM Damjan Marion via Lists.Fd.Io<http://lists.fd.io/> <dmarion=me....@lists.fd.io<mailto:me....@lists.fd.io>> wrote: Regarding: https://gerrit.fd.io/r/#/c/16543/ This patch switches dpdk to new in-memory mode, and reduces dpdk memory footprint as pages are allocated dynamically on-demand. I tested on both Ubuntu and Centos 7.5 and everything looks good but will appreciate feedback from people using non-standard configs before it is merged... Thanks, -- Damjan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11724): https://lists.fd.io/g/vpp-dev/message/11724 Mute This Topic: https://lists.fd.io/mt/28809973/675725 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev%2bow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [mgsm...@netgate.com<mailto:mgsm...@netgate.com>] -=-=-=-=-=-=-=-=-=-=-=- -- Damjan
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11805): https://lists.fd.io/g/vpp-dev/message/11805 Mute This Topic: https://lists.fd.io/mt/28809973/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
