Hi Nitin, Both 2M and 1G are supported by pmalloc, but they are not supported at same time, as system is configured with either 2M or 1G hugepages, not both at the same time. Thanks. From: [email protected] <[email protected]> On Behalf Of Nitin Saxena via lists.fd.io Sent: 2020年7月29日 15:23 To: Honnappa Nagarahalli <[email protected]>; Damjan Marion <[email protected]> Cc: Lijian Zhang <[email protected]>; vpp-dev <[email protected]>; nd <[email protected]>; Govindarajan Mohandoss <[email protected]>; Jieqiang Wang <[email protected]> Subject: Re: [vpp-dev] Create big tables on huge-page
Hi Lijian, +1 on the finding. It would be interested to know how much is the performance gain. Having said that, correct me if I am wrong, I think pmalloc module works only on single hugepage size (pm->def_log2_page_sz) which means either 1G or 2M and not both Thanks, Nitin From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Honnappa Nagarahalli Sent: Thursday, July 23, 2020 10:53 PM To: Damjan Marion <[email protected]<mailto:[email protected]>> Cc: Lijian Zhang <[email protected]<mailto:[email protected]>>; vpp-dev <[email protected]<mailto:[email protected]>>; nd <[email protected]<mailto:[email protected]>>; Govindarajan Mohandoss <[email protected]<mailto:[email protected]>>; Jieqiang Wang <[email protected]<mailto:[email protected]>>; Honnappa Nagarahalli <[email protected]<mailto:[email protected]>> Subject: [EXT] Re: [vpp-dev] Create big tables on huge-page External Email ________________________________ Sure. We will create couple of patches (in the areas we are analyzing currently) and we can decide from there. Thanks, Honnappa From: Damjan Marion <[email protected]<mailto:[email protected]>> Sent: Thursday, July 23, 2020 12:17 PM To: Honnappa Nagarahalli <[email protected]<mailto:[email protected]>> Cc: Lijian Zhang <[email protected]<mailto:[email protected]>>; vpp-dev <[email protected]<mailto:[email protected]>>; nd <[email protected]<mailto:[email protected]>>; Govindarajan Mohandoss <[email protected]<mailto:[email protected]>>; Jieqiang Wang <[email protected]<mailto:[email protected]>> Subject: Re: [vpp-dev] Create big tables on huge-page Hard to say without seeing the patch. Can you summarize what the changes will be in each particular .c file? On 23 Jul 2020, at 18:00, Honnappa Nagarahalli <[email protected]<mailto:[email protected]>> wrote: Hi Damjan, Thank you. Till your patch is ready, would you accept patches that would enable creating these tables in 1G huge pages as temporary solution? Thanks, Honnappa From: Damjan Marion <[email protected]<mailto:[email protected]>> Sent: Thursday, July 23, 2020 7:15 AM To: Lijian Zhang <[email protected]<mailto:[email protected]>> Cc: vpp-dev <[email protected]<mailto:[email protected]>>; nd <[email protected]<mailto:[email protected]>>; Honnappa Nagarahalli <[email protected]<mailto:[email protected]>>; Govindarajan Mohandoss <[email protected]<mailto:[email protected]>>; Jieqiang Wang <[email protected]<mailto:[email protected]>> Subject: Re: [vpp-dev] Create big tables on huge-page I started working on patch which addresses most of this points, few weeks ago, but likely I will not have it completed for 20.09. Even if it is completed, it is probably bad idea to merge it so late in the release process…. — Damjan On 23 Jul 2020, at 10:45, Lijian Zhang <[email protected]<mailto:[email protected]>> wrote: Hi Maintainers, From VPP source code, ip4-mtrie table is created on huge-page only when below parameters are set in configuration file. While adjacency table is created on normal-page always. 36 ip { 37 heap-size 256M 38 mtrie-hugetlb 39 } In the 10K flow testing, I configured 10K routing entries in ip4-mtrie and 10K entries in adjacency table. By creating ip4-mtrie table on 1G huge-page with above parameters set and similarly create adjacency table on 1G huge-page, although I don’t observe obvious throughput performance improvement, but TLB misses are dramatically reduced. Do you think configuration of 10K routing entries + 10K adjacency entries is a reasonable and possible config, or normally it would be 10K routing entries + only several adjacency entries? Does it make sense to create adjacency table on huge-pages? Another problem is although above assigned heap-size is 256M, but on 1G huge-page system, it seems to occupy a huge-page completely, other memory space within that huge-page seems will not be used by other tables. Same as the bihash based tables, only 2M huge-page system is supported. To support creating bihash based tables on 1G huge-page system, each table will occupy a 1G huge-page completely, but that will waste a lot of memories. Is it possible just like pmalloc module, reserving a big memory space on 1G/2M huge-pages in initialization stage, and then allocate memory pieces per requirement for Bihash, ip4-mtrie and adjacency tables, so that all tables could be created on huge-pages and will fully utilize the huge-pages. I tried to create MAC table on 1G huge-page, and it does improve throughput performance. vpp# show bihash Name Actual Configured GBP Endpoints - MAC/BD 1m 1m b4s 64m 64m b4s 64m 64m in2out 10.12m 10.12m in2out 10.12m 10.12m ip4-dr 2m 2m ip4-dr 2m 2m ip6 FIB fwding table 32m 32m ip6 FIB non-fwding table 32m 32m ip6 mFIB table 32m 32m l2fib mac table 512m 512m mapping_by_as4 64m 64m out2in 128m 128m out2in 128m 128m out2in 10.12m 10.12m out2in 10.12m 10.12m pppoe link table 8m 8m pppoe session table 8m 8m static_mapping_by_external 64m 64m static_mapping_by_local 64m 64m stn addresses 1m 1m users 648k 648k users 648k 648k vip_index_per_port 64m 64m vxlan4 1m 1m vxlan4-gbp 1m 1m Total 1.28g 1.28g Thanks.
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17126): https://lists.fd.io/g/vpp-dev/message/17126 Mute This Topic: https://lists.fd.io/mt/75742152/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
