Re: [PATCH v12 00/31] Speculative page faults
Hi Laurent, I downloaded your script and run it on Intel 2s skylake platform with spf-v12 patch serials. Here attached the output results of this script. The following comparison result is statistics from the script outputs. a). Enable THP SPF_0 change SPF_1 will-it-scale.page_fault2.per_thread_ops2664190.8 -11.7% 2353637.6 will-it-scale.page_fault3.per_thread_ops4480027.2 -14.7% 3819331.9 b). Disable THP SPF_0 change SPF_1 will-it-scale.page_fault2.per_thread_ops2653260.7 -10% 2385165.8 will-it-scale.page_fault3.per_thread_ops4436330.1 -12.4% 3886734.2 Thanks, Haiyan Song On Fri, Jun 14, 2019 at 10:44:47AM +0200, Laurent Dufour wrote: > Le 14/06/2019 à 10:37, Laurent Dufour a écrit : > > Please find attached the script I run to get these numbers. > > This would be nice if you could give it a try on your victim node and share > > the result. > > Sounds that the Intel mail fitering system doesn't like the attached shell > script. > Please find it there: > https://gist.github.com/ldu4/a5cc1a93f293108ea387d43d5d5e7f44 > > Thanks, > Laurent. > THP always SPF 0 average:2628818 average:2732209 average:2728392 average:2550695 average:2689873 average:2691963 average:2627612 average:2558295 average:2707877 average:2726174 SPF 1 average:2426260 average:2145674 average:2117769 average:2292502 average:2350403 average:2483327 average:2467324 average:2335393 average:2437859 average:2479865 THP never SPF 0 average:2712575 average:2711447 average:2672362 average:2701981 average:2668073 average:2579296 average:2662048 average:2637422 average:2579143 average:2608260 SPF 1 average:2348782 average:2203349 average:2312960 average:2402995 average:2318914 average:2543129 average:2390337 average:2490178 average:2416798 average:2424216 THP always SPF 0 average:4370143 average:4245754 average:4678884 average:4665759 average:4665809 average:4639132 average:4210755 average:4330552 average:4290469 average:4703015 SPF 1 average:3810608 average:3918890 average:3758003 average:3965024 average:3578151 average:3822748 average:3687293 average:3998701 average:3915771 average:3738130 THP never SPF 0 average:4505598 average:4672023 average:4701787 average:4355885 average:4338397 average:4446350 average:4360811 average:4653767 average:4016352 average:4312331 SPF 1 average:3685383 average:4029413 average:4051615 average:3747588 average:4058557 average:4042340 average:3971295 average:3752943 average:3750626 average:3777582
Re: [PATCH v12 00/31] Speculative page faults
Hi Laurent, Regression test for v12 patch serials have been run on Intel 2s skylake platform, some regressions were found by LKP-tools (linux kernel performance). Only tested the cases that have been run and found regressions on v11 patch serials. Get the patch serials from https://github.com/ldu4/linux/tree/spf-v12. Kernel commit: base: a297558ad4479e0c9c5c14f3f69fe43113f72d1c (v5.1-rc4-mmotm-2019-04-09-17-51) head: 02c5a1f984a8061d075cfd74986ac8aa01d81064 (spf-v12) Benchmark: will-it-scale Download link: https://github.com/antonblanchard/will-it-scale/tree/master Metrics: will-it-scale.per_thread_ops=threads/nr_cpu test box: lkp-skl-2sp8(nr_cpu=72,memory=192G) THP: enable / disable nr_task: 100% The following is benchmark results, tested 4 times for every case. a). Enable THP base %stddev changehead %stddev will-it-scale.page_fault3.per_thread_ops63216 ±3% -16.9%52537 ±4% will-it-scale.page_fault2.per_thread_ops36862 -9.8% 33256 b). Disable THP base %stddev changehead %stddev will-it-scale.page_fault3.per_thread_ops65111 -18.6%53023 ±2% will-it-scale.page_fault2.per_thread_ops38164 -12.0%33565 Best regards, Haiyan Song On Tue, Apr 16, 2019 at 03:44:51PM +0200, Laurent Dufour wrote: > This is a port on kernel 5.1 of the work done by Peter Zijlstra to handle > page fault without holding the mm semaphore [1]. > > The idea is to try to handle user space page faults without holding the > mmap_sem. This should allow better concurrency for massively threaded > process since the page fault handler will not wait for other threads memory > layout change to be done, assuming that this change is done in another part > of the process's memory space. This type of page fault is named speculative > page fault. If the speculative page fault fails because a concurrency has > been detected or because underlying PMD or PTE tables are not yet > allocating, it is failing its processing and a regular page fault is then > tried. > > The speculative page fault (SPF) has to look for the VMA matching the fault > address without holding the mmap_sem, this is done by protecting the MM RB > tree with RCU and by using a reference counter on each VMA. When fetching a > VMA under the RCU protection, the VMA's reference counter is incremented to > ensure that the VMA will not freed in our back during the SPF > processing. Once that processing is done the VMA's reference counter is > decremented. To ensure that a VMA is still present when walking the RB tree > locklessly, the VMA's reference counter is incremented when that VMA is > linked in the RB tree. When the VMA is unlinked from the RB tree, its > reference counter will be decremented at the end of the RCU grace period, > ensuring it will be available during this time. This means that the VMA > freeing could be delayed and could delay the file closing for file > mapping. Since the SPF handler is not able to manage file mapping, file is > closed synchronously and not during the RCU cleaning. This is safe since > the page fault handler is aborting if a file pointer is associated to the > VMA. > > Using RCU fixes the overhead seen by Haiyan Song using the will-it-scale > benchmark [2]. > > The VMA's attributes checked during the speculative page fault processing > have to be protected against parallel changes. This is done by using a per > VMA sequence lock. This sequence lock allows the speculative page fault > handler to fast check for parallel changes in progress and to abort the > speculative page fault in that case. > > Once the VMA has been found, the speculative page fault handler would check > for the VMA's attributes to verify that the page fault has to be handled > correctly or not. Thus, the VMA is protected through a sequence lock which > allows fast detection of concurrent VMA changes. If such a change is > detected, the speculative page fault is aborted and a *classic* page fault > is tried. VMA sequence lockings are added when VMA attributes which are > checked during the page fault are modified. > > When the PTE is fetched, the VMA is checked to see if it has been changed, > so once the page table is locked, the VMA is valid, so any other changes > leading to touching this PTE will need to lock the page table, so no > parallel change is possible at this time. > > The locking of the PTE is done with interrupts disabled, this allows > checking for the PMD to ensure that there is not an ongoing collapsing > operation. Since khugepaged is firstly set the PMD to pmd_none and then is > waiting for the other CPU to have caught the IPI interrupt, if the pmd is > valid at the ti
Re: [PATCH v11 00/26] Speculative page faults
Hi Laurent, Yes, these tests are done on V9 patch. Best regards, Haiyan Song On Mon, May 28, 2018 at 09:51:34AM +0200, Laurent Dufour wrote: > On 28/05/2018 07:23, Song, HaiyanX wrote: > > > > Some regression and improvements is found by LKP-tools(linux kernel > > performance) on V9 patch series > > tested on Intel 4s Skylake platform. > > Hi, > > Thanks for reporting this benchmark results, but you mentioned the "V9 patch > series" while responding to the v11 header series... > Were these tests done on v9 or v11 ? > > Cheers, > Laurent. > > > > > The regression result is sorted by the metric will-it-scale.per_thread_ops. > > Branch: Laurent-Dufour/Speculative-page-faults/20180316-151833 (V9 patch > > series) > > Commit id: > > base commit: d55f34411b1b126429a823d06c3124c16283231f > > head commit: 0355322b3577eeab7669066df42c550a56801110 > > Benchmark suite: will-it-scale > > Download link: > > https://github.com/antonblanchard/will-it-scale/tree/master/tests > > Metrics: > > will-it-scale.per_process_ops=processes/nr_cpu > > will-it-scale.per_thread_ops=threads/nr_cpu > > test box: lkp-skl-4sp1(nr_cpu=192,memory=768G) > > THP: enable / disable > > nr_task: 100% > > > > 1. Regressions: > > a) THP enabled: > > testcasebasechange head > > metric > > page_fault3/ enable THP 10092 -17.5% 8323 > > will-it-scale.per_thread_ops > > page_fault2/ enable THP 8300 -17.2% 6869 > > will-it-scale.per_thread_ops > > brk1/ enable THP 957.67 -7.6% 885 > > will-it-scale.per_thread_ops > > page_fault3/ enable THP172821-5.3%163692 > > will-it-scale.per_process_ops > > signal1/ enable THP 9125-3.2% 8834 > > will-it-scale.per_process_ops > > > > b) THP disabled: > > testcasebasechange head > > metric > > page_fault3/ disable THP10107 -19.1% 8180 > > will-it-scale.per_thread_ops > > page_fault2/ disable THP 8432 -17.8% 6931 > > will-it-scale.per_thread_ops > > context_switch1/ disable THP 215389-6.8%200776 > > will-it-scale.per_thread_ops > > brk1/ disable THP 939.67 -6.6% 877.33 > > will-it-scale.per_thread_ops > > page_fault3/ disable THP 173145-4.7%165064 > > will-it-scale.per_process_ops > > signal1/ disable THP 9162-3.9% 8802 > > will-it-scale.per_process_ops > > > > 2. Improvements: > > a) THP enabled: > > testcasebasechange head > > metric > > malloc1/ enable THP 66.33+469.8% 383.67 > > will-it-scale.per_thread_ops > > writeseek3/ enable THP 2531 +4.5% 2646 > > will-it-scale.per_thread_ops > > signal1/ enable THP 989.33 +2.8% 1016 > > will-it-scale.per_thread_ops > > > > b) THP disabled: > > testcasebasechange head > > metric > > malloc1/ disable THP 90.33+417.3% 467.33 > > will-it-scale.per_thread_ops > > read2/ disable THP 58934+39.2% 82060 > > will-it-scale.per_thread_ops > > page_fault1/ disable THP8607+36.4% 11736 > > will-it-scale.per_thread_ops > > read1/ disable THP314063+12.7%353934 > > will-it-scale.per_thread_ops > > writeseek3/ disable THP 2452+12.5% 2759 > > will-it-scale.per_thread_ops > > signal1/ disable THP 971.33 +5.5% 1024 > > will-it-scale.per_thread_ops > > > > Notes: for above values in column "change", the higher value means that the > > related testcase result > > on head commit is better than that on base commit for this benchmark. > > > > > > Best regards > > Haiyan Song > > > > > > From: owner-linux...@kvack.org [owner-linux...@kvack.org] on behalf of > > Laurent Dufour [lduf...@linux.vnet.ibm.com] > > Sent: Thu
