Hi Trond,

On 5/20/2019 1:54 PM, kernel test robot wrote:
Greeting,

FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to commit:


commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert socket page send 
code to use iov_iter()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: fsmark
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G 
memory
with following parameters:

        iterations: 1x
        nr_threads: 64t
        disk: 1BRD_48G
        fs: xfs
        fs2: nfsv4
        filesize: 4M
        test_size: 40G
        sync_method: fsyncBeforeClose
        cpufreq_governor: performance

test-description: The fsmark is a file system benchmark to test synchronous 
write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

         git clone https://github.com/intel/lkp-tests.git
         cd lkp-tests
         bin/lkp install job.yaml  # job file is attached in this email
         bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
   
gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-ep01/40G/fsmark

commit:
   e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use iov_iter_kvec()")
   0472e47660 ("SUNRPC: Convert socket page send code to use iov_iter()")

e791f8e9380d945e 0472e476604998c127f3c80d291
---------------- ---------------------------
        fail:runs  %reproduction    fail:runs
            |             |             |
            :4           50%           2:4     
dmesg.WARNING:at#for_ip_interrupt_entry/0x
          %stddev     %change         %stddev
              \          |                \
   15118573 ±  2%     +16.0%   17538083        fsmark.app_overhead
     510.93           -22.7%     395.12        fsmark.files_per_sec
      24.90           +22.8%      30.57        fsmark.time.elapsed_time
      24.90           +22.8%      30.57        fsmark.time.elapsed_time.max
     288.00 ±  2%     -27.8%     208.00        
fsmark.time.percent_of_cpu_this_job_got
      70.03 ±  2%     -11.3%      62.14        fsmark.time.system_time
    4391964           -16.7%    3658341        meminfo.max_used_kB
       6.10 ±  4%      +1.9        7.97 ±  3%  mpstat.cpu.all.iowait%
       0.27            -0.0        0.24 ±  3%  mpstat.cpu.all.soft%
   13668070 ± 40%    +118.0%   29801846 ± 19%  numa-numastat.node0.local_node
   13677774 ± 40%    +117.9%   29810258 ± 19%  numa-numastat.node0.numa_hit
       5.70 ±  3%     +32.1%       7.53 ±  3%  iostat.cpu.iowait
      16.42 ±  2%      -5.8%      15.47        iostat.cpu.system
       2.57            -4.1%       2.46        iostat.cpu.user
    1406781 ±  2%     -15.5%    1188498        vmstat.io.bo
     251792 ±  3%     -16.6%     209928        vmstat.system.cs
      84841            -1.9%      83239        vmstat.system.in
   97374502 ± 20%     +66.1%  1.617e+08 ± 17%  cpuidle.C1E.time
     573934 ± 19%     +44.6%     829662 ± 26%  cpuidle.C1E.usage
  5.892e+08 ±  8%     +15.3%  6.796e+08 ±  2%  cpuidle.C6.time
    1968016 ±  3%     -15.1%    1670867 ±  3%  cpuidle.POLL.time
     106420 ± 47%     +86.2%     198108 ± 35%  numa-meminfo.node0.Active
     106037 ± 48%     +86.2%     197395 ± 35%  numa-meminfo.node0.Active(anon)
     105052 ± 48%     +86.6%     196037 ± 35%  numa-meminfo.node0.AnonPages
     212876 ± 24%     -41.5%     124572 ± 56%  numa-meminfo.node1.Active
     211801 ± 24%     -41.5%     123822 ± 56%  numa-meminfo.node1.Active(anon)
     208559 ± 24%     -42.2%     120547 ± 57%  numa-meminfo.node1.AnonPages
       9955            +1.6%      10116        proc-vmstat.nr_kernel_stack
     452.25 ± 59%    +280.9%       1722 ±100%  
proc-vmstat.numa_hint_faults_local
   33817303           +55.0%   52421773 ±  5%  proc-vmstat.numa_hit
   33804286           +55.0%   52408807 ±  5%  proc-vmstat.numa_local
   33923002           +81.8%   61663426 ±  5%  proc-vmstat.pgalloc_normal
     184765            +9.3%     201985        proc-vmstat.pgfault
   12840986          +216.0%   40581327 ±  7%  proc-vmstat.pgfree
      31447 ± 11%     -26.1%      23253 ± 13%  
sched_debug.cfs_rq:/.min_vruntime.max
       4241 ±  3%     -12.2%       3724 ± 11%  
sched_debug.cfs_rq:/.min_vruntime.stddev
      20631 ± 11%     -36.7%      13069 ± 29%  sched_debug.cfs_rq:/.spread0.max
       4238 ±  4%     -12.1%       3724 ± 11%  
sched_debug.cfs_rq:/.spread0.stddev
     497105 ± 19%     -16.0%     417777 ±  4%  sched_debug.cpu.avg_idle.avg
      21199 ± 10%     -12.0%      18650 ±  3%  
sched_debug.cpu.nr_load_updates.max
       2229 ± 10%     -15.0%       1895 ±  4%  
sched_debug.cpu.nr_load_updates.stddev
       4.86 ±  5%     -23.6%       3.72 ±  5%  
sched_debug.cpu.nr_uninterruptible.stddev
     524.75 ±  2%     -10.7%     468.50        turbostat.Avg_MHz
       5.26 ± 41%      -1.6        3.66 ±  2%  turbostat.C1%
     573633 ± 19%     +44.6%     829267 ± 26%  turbostat.C1E
       8.53 ± 20%      +3.4       11.88 ± 17%  turbostat.C1E%
      76.75            -6.4%      71.86        turbostat.CorWatt
    2534071 ±  2%     +16.2%    2943968        turbostat.IRQ
       2.89 ± 24%     +52.9%       4.42 ± 22%  turbostat.Pkg%pc2
     104.28            -4.6%      99.51        turbostat.PkgWatt
       2289           +18.8%       2720        turbostat.SMI
      26496 ± 47%     +86.2%      49347 ± 35%  numa-vmstat.node0.nr_active_anon
      26251 ± 48%     +86.7%      49008 ± 35%  numa-vmstat.node0.nr_anon_pages
      26496 ± 47%     +86.2%      49347 ± 35%  
numa-vmstat.node0.nr_zone_active_anon
    8272770 ± 35%     +99.6%   16515977 ± 17%  numa-vmstat.node0.numa_hit
    8199681 ± 36%    +100.9%   16474282 ± 17%  numa-vmstat.node0.numa_local
      52953 ± 24%     -41.5%      30956 ± 56%  numa-vmstat.node1.nr_active_anon
      52144 ± 24%     -42.2%      30136 ± 57%  numa-vmstat.node1.nr_anon_pages
       2899 ±  2%     -19.4%       2336 ± 10%  numa-vmstat.node1.nr_writeback
      52953 ± 24%     -41.5%      30956 ± 56%  
numa-vmstat.node1.nr_zone_active_anon
       4549 ±  8%     -18.9%       3689 ±  7%  
numa-vmstat.node1.nr_zone_write_pending
      43.25            -1.6       41.69        perf-stat.i.cache-miss-rate%
     286377 ±  2%     -19.1%     231819 ±  2%  perf-stat.i.context-switches
  2.258e+10 ±  4%     -16.2%  1.893e+10 ± 10%  perf-stat.i.cpu-cycles
       4084           -34.0%       2696 ±  2%  perf-stat.i.cpu-migrations
   21563675 ± 24%     -30.8%   14929445 ±  5%  perf-stat.i.dTLB-load-misses
    1996341 ± 16%     -25.7%    1482826 ±  6%  perf-stat.i.dTLB-store-misses
  2.313e+09 ±  4%     -11.7%  2.042e+09 ±  6%  perf-stat.i.dTLB-stores
      75.93            -1.3       74.66        perf-stat.i.iTLB-load-miss-rate%
    1561381 ±  3%     -13.4%    1351731        perf-stat.i.iTLB-load-misses
     549643 ±  4%     -10.1%     494255 ±  3%  perf-stat.i.iTLB-loads
       6423           -10.0%       5779        perf-stat.i.minor-faults
      31.72 ±  2%      +4.4       36.14        perf-stat.i.node-load-miss-rate%
   23492890           +23.5%   29020305 ±  3%  perf-stat.i.node-load-misses
      15.13 ±  8%      +7.9       23.02 ±  3%  perf-stat.i.node-store-miss-rate%
   11517615           +71.3%   19734054 ±  2%  perf-stat.i.node-store-misses
       6423           -10.0%       5779        perf-stat.i.page-faults
      16.88 ± 14%     +33.4%      22.52 ± 23%  perf-stat.overall.MPKI
      46.67            -2.4       44.31        
perf-stat.overall.cache-miss-rate%
      31.96            +4.8       36.72        
perf-stat.overall.node-load-miss-rate%
      13.85            +8.0       21.87        
perf-stat.overall.node-store-miss-rate%
  2.266e+08            +2.8%  2.331e+08 ±  2%  perf-stat.ps.cache-references
     275634 ±  2%     -18.5%     224718 ±  2%  perf-stat.ps.context-switches
  2.174e+10 ±  4%     -15.6%  1.835e+10 ± 10%  perf-stat.ps.cpu-cycles
       3931           -33.5%       2614 ±  2%  perf-stat.ps.cpu-migrations
   20746040 ± 24%     -30.2%   14476157 ±  5%  perf-stat.ps.dTLB-load-misses
    1921077 ± 16%     -25.2%    1437750 ±  6%  perf-stat.ps.dTLB-store-misses
  2.227e+09 ±  4%     -11.1%  1.979e+09 ±  6%  perf-stat.ps.dTLB-stores
    1503433 ±  3%     -12.8%    1310741        perf-stat.ps.iTLB-load-misses
     529058 ±  4%      -9.4%     479204 ±  3%  perf-stat.ps.iTLB-loads
       6200            -9.4%       5620        perf-stat.ps.minor-faults
   22613159           +24.4%   28133123 ±  3%  perf-stat.ps.node-load-misses
   11085254           +72.6%   19131576 ±  2%  perf-stat.ps.node-store-misses
       6200            -9.4%       5620        perf-stat.ps.page-faults
       7008 ± 13%     +55.2%      10876 ± 31%  softirqs.CPU1.NET_RX
       6509 ±  4%     +51.8%       9883 ± 27%  softirqs.CPU1.RCU
       7294 ± 13%     +36.7%       9974 ±  8%  softirqs.CPU10.RCU
       7800 ± 14%     +85.5%      14469 ± 23%  softirqs.CPU12.NET_RX
       5697 ± 43%     +77.5%      10110 ± 24%  softirqs.CPU13.NET_RX
      15944 ±  9%     +14.6%      18278 ± 12%  softirqs.CPU14.TIMER
       6064 ± 19%     +68.6%      10223 ± 31%  softirqs.CPU15.NET_RX
       7796 ± 14%     +80.3%      14059 ± 25%  softirqs.CPU16.NET_RX
      15934 ± 10%     +22.1%      19452 ± 11%  softirqs.CPU18.TIMER
       6725 ±  7%     +40.0%       9413 ± 18%  softirqs.CPU19.NET_RX
       5710 ±  3%     +53.4%       8756 ± 12%  softirqs.CPU20.RCU
       7018 ± 14%     +65.5%      11616 ± 40%  softirqs.CPU21.NET_RX
       6389 ± 18%     +66.9%      10666 ± 31%  softirqs.CPU23.NET_RX
       7259 ±  7%     +36.1%       9881 ±  6%  softirqs.CPU24.RCU
       6491 ± 20%     +58.5%      10289 ± 33%  softirqs.CPU25.NET_RX
       7090 ± 10%     +58.7%      11256 ± 29%  softirqs.CPU27.NET_RX
       6333 ± 27%     +70.3%      10786 ± 27%  softirqs.CPU29.NET_RX
       5813 ± 20%     +84.2%      10706 ± 36%  softirqs.CPU3.NET_RX
       7041 ± 23%    +105.7%      14483 ± 21%  softirqs.CPU30.NET_RX
       6654 ±  7%     +64.1%      10918 ± 29%  softirqs.CPU31.NET_RX
      18019 ± 10%     +11.1%      20016 ±  7%  softirqs.CPU31.TIMER
       4666 ±  8%    +104.0%       9518 ± 23%  softirqs.CPU32.RCU
      15721 ± 16%     +35.9%      21371 ± 11%  softirqs.CPU32.TIMER
      15684 ± 13%     +40.0%      21959 ± 18%  softirqs.CPU34.TIMER
       6489 ± 15%     +69.7%      11013 ± 37%  softirqs.CPU35.NET_RX
       7930 ±  7%     +82.1%      14442 ± 29%  softirqs.CPU36.NET_RX
      15744 ± 14%     +24.3%      19563 ±  6%  softirqs.CPU36.TIMER
       7028 ± 13%     +29.3%       9085 ± 11%  softirqs.CPU39.NET_RX
       7491 ± 22%    +100.4%      15011 ± 24%  softirqs.CPU4.NET_RX
       6119 ± 13%     +58.7%       9710 ± 37%  softirqs.CPU5.NET_RX
       6980 ±  8%     +47.8%      10318 ± 42%  softirqs.CPU7.NET_RX
     285674           +65.7%     473395        softirqs.NET_RX
     267950 ±  5%     +21.1%     324597        softirqs.RCU
     238298 ±  2%     +15.1%     274371 ±  2%  softirqs.SCHED
     689305 ±  3%     +14.1%     786236 ±  3%  softirqs.TIMER
      56196 ±  2%     +19.9%      67389        
interrupts.CPU0.LOC:Local_timer_interrupts
      55971 ±  2%     +19.7%      67005        
interrupts.CPU1.LOC:Local_timer_interrupts
       5265 ± 17%     -32.2%       3568 ± 29%  
interrupts.CPU1.RES:Rescheduling_interrupts
      56163 ±  2%     +19.2%      66960        
interrupts.CPU10.LOC:Local_timer_interrupts
      56178 ±  2%     +19.5%      67139        
interrupts.CPU11.LOC:Local_timer_interrupts
       4669 ± 15%     -30.3%       3253 ± 29%  
interrupts.CPU11.RES:Rescheduling_interrupts
      56081           +19.7%      67142        
interrupts.CPU12.LOC:Local_timer_interrupts
      55999 ±  2%     +19.5%      66893        
interrupts.CPU13.LOC:Local_timer_interrupts
       4746 ± 23%     -30.0%       3324 ± 29%  
interrupts.CPU13.RES:Rescheduling_interrupts
      55991 ±  2%     +19.5%      66898        
interrupts.CPU14.LOC:Local_timer_interrupts
      55855 ±  2%     +20.0%      67031        
interrupts.CPU15.LOC:Local_timer_interrupts
      55943 ±  2%     +19.9%      67087        
interrupts.CPU16.LOC:Local_timer_interrupts
      56164           +18.9%      66802        
interrupts.CPU17.LOC:Local_timer_interrupts
       4579 ± 20%     -45.6%       2492 ± 16%  
interrupts.CPU17.RES:Rescheduling_interrupts
      55913 ±  2%     +19.6%      66860        
interrupts.CPU18.LOC:Local_timer_interrupts
      55820 ±  2%     +19.7%      66822        
interrupts.CPU19.LOC:Local_timer_interrupts
       4733 ± 17%     -35.5%       3052 ± 21%  
interrupts.CPU19.RES:Rescheduling_interrupts
      55989 ±  2%     +20.0%      67177        
interrupts.CPU2.LOC:Local_timer_interrupts
      55891 ±  2%     +20.3%      67258        
interrupts.CPU20.LOC:Local_timer_interrupts
      55966 ±  2%     +19.6%      66921        
interrupts.CPU21.LOC:Local_timer_interrupts
       4920 ± 21%     -33.4%       3278 ± 31%  
interrupts.CPU21.RES:Rescheduling_interrupts
      55945 ±  2%     +19.9%      67098        
interrupts.CPU22.LOC:Local_timer_interrupts
      55945 ±  2%     +19.9%      67073        
interrupts.CPU23.LOC:Local_timer_interrupts
       4972 ± 24%     -34.1%       3277 ± 36%  
interrupts.CPU23.RES:Rescheduling_interrupts
      56093 ±  2%     +19.8%      67185        
interrupts.CPU24.LOC:Local_timer_interrupts
      55910 ±  2%     +19.7%      66914        
interrupts.CPU25.LOC:Local_timer_interrupts
       4660 ± 25%     -28.9%       3313 ± 34%  
interrupts.CPU25.RES:Rescheduling_interrupts
      56105 ±  2%     +18.8%      66631        
interrupts.CPU26.LOC:Local_timer_interrupts
      55928 ±  2%     +19.5%      66827        
interrupts.CPU27.LOC:Local_timer_interrupts
      55934 ±  2%     +19.3%      66740        
interrupts.CPU28.LOC:Local_timer_interrupts
      55918 ±  2%     +19.5%      66812        
interrupts.CPU29.LOC:Local_timer_interrupts
       4825 ± 22%     -30.3%       3362 ± 30%  
interrupts.CPU29.RES:Rescheduling_interrupts
      55920 ±  2%     +19.8%      67004        
interrupts.CPU3.LOC:Local_timer_interrupts
      56007 ±  2%     +19.5%      66917        
interrupts.CPU30.LOC:Local_timer_interrupts
      56238           +18.8%      66838        
interrupts.CPU31.LOC:Local_timer_interrupts
       4859 ± 21%     -33.0%       3257 ± 36%  
interrupts.CPU31.RES:Rescheduling_interrupts
      56099           +19.3%      66921        
interrupts.CPU32.LOC:Local_timer_interrupts
      55963 ±  2%     +19.5%      66866        
interrupts.CPU33.LOC:Local_timer_interrupts
      55992 ±  2%     +19.2%      66767        
interrupts.CPU34.LOC:Local_timer_interrupts
      55964 ±  2%     +19.6%      66939        
interrupts.CPU35.LOC:Local_timer_interrupts
       4797 ± 26%     -33.6%       3185 ± 38%  
interrupts.CPU35.RES:Rescheduling_interrupts
      55770 ±  2%     +20.2%      67016        
interrupts.CPU36.LOC:Local_timer_interrupts
      55889           +20.0%      67095        
interrupts.CPU37.LOC:Local_timer_interrupts
      56140 ±  2%     +18.9%      66743        
interrupts.CPU38.LOC:Local_timer_interrupts
      56040 ±  2%     +19.2%      66828        
interrupts.CPU39.LOC:Local_timer_interrupts
       5126 ± 23%     -40.5%       3052 ± 11%  
interrupts.CPU39.RES:Rescheduling_interrupts
      56022 ±  2%     +19.9%      67149        
interrupts.CPU4.LOC:Local_timer_interrupts
      55850 ±  2%     +20.1%      67066        
interrupts.CPU5.LOC:Local_timer_interrupts
       4741 ± 23%     -31.9%       3227 ± 29%  
interrupts.CPU5.RES:Rescheduling_interrupts
      55912           +19.5%      66798        
interrupts.CPU6.LOC:Local_timer_interrupts
      56062 ±  2%     +19.6%      67052        
interrupts.CPU7.LOC:Local_timer_interrupts
      56118           +19.6%      67142        
interrupts.CPU8.LOC:Local_timer_interrupts
      56110 ±  2%     +19.6%      67121        
interrupts.CPU9.LOC:Local_timer_interrupts
       2216 ± 43%     -69.4%     677.50 ±112%  
interrupts.CPU9.NMI:Non-maskable_interrupts
       2216 ± 43%     -69.4%     677.50 ±112%  
interrupts.CPU9.PMI:Performance_monitoring_interrupts
    2240009 ±  2%     +19.6%    2678940        
interrupts.LOC:Local_timer_interrupts
     168296 ±  3%     -10.1%     151314        
interrupts.RES:Rescheduling_interrupts
      20.37            -3.3       17.04 ±  5%  
perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
      20.38            -3.3       17.04 ±  5%  
perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
      20.27            -3.3       16.95 ±  5%  
perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
      20.21            -3.3       16.91 ±  5%  
perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
      26.02            -3.1       22.91 ±  4%  
perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
       9.04 ±  2%      -1.9        7.18 ±  7%  
perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
       9.02 ±  2%      -1.9        7.17 ±  7%  
perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
       9.02 ±  2%      -1.9        7.16 ±  7%  
perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
       9.03 ±  2%      -1.9        7.18 ±  7%  
perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
       9.07 ±  2%      -1.9        7.22 ±  7%  
perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      10.50 ±  2%      -1.4        9.06 ±  4%  
perf-profile.calltrace.cycles-pp.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      10.51 ±  2%      -1.4        9.08 ±  4%  
perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      10.45 ±  2%      -1.4        9.02 ±  4%  
perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch
       9.82 ±  3%      -1.4        8.45 ±  3%  
perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound
       9.82 ±  3%      -1.4        8.45 ±  3%  
perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit
       9.82 ±  3%      -1.4        8.45 ±  3%  
perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      10.61 ±  5%      -1.1        9.46 ±  4%  
perf-profile.calltrace.cycles-pp.write
       7.54            -1.1        6.41 ±  6%  
perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
       7.54            -1.1        6.41 ±  6%  
perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
      10.27 ±  5%      -1.1        9.15 ±  4%  
perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
       7.49            -1.1        6.37 ±  6%  
perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev
      10.23 ±  5%      -1.1        9.11 ±  4%  
perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      10.35 ±  5%      -1.1        9.23 ±  4%  
perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      10.08 ±  5%      -1.1        8.96 ±  4%  
perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      10.33 ±  5%      -1.1        9.22 ±  4%  
perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      10.05 ±  5%      -1.1        8.95 ±  4%  
perf-profile.calltrace.cycles-pp.nfs_file_write.__vfs_write.vfs_write.ksys_write.do_syscall_64
       9.80 ±  5%      -1.1        8.73 ±  4%  
perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.__vfs_write.vfs_write.ksys_write
       5.31 ±  5%      -1.0        4.27 ±  5%  
perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
       5.31 ±  5%      -1.0        4.27 ±  5%  
perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
       4.38 ±  5%      -1.0        3.35 ±  2%  
perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv
       4.38 ±  6%      -1.0        3.36 ±  2%  
perf-profile.calltrace.cycles-pp.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd
       4.39 ±  6%      -1.0        3.40 ±  2%  
perf-profile.calltrace.cycles-pp.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd.kthread
       3.53 ±  5%      -0.9        2.62 ±  3%  
perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom.svc_tcp_recvfrom
       3.52 ±  5%      -0.9        2.62 ±  3%  
perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.svc_recvfrom
       5.01 ±  6%      -0.9        4.14 ±  3%  
perf-profile.calltrace.cycles-pp.fsync
       4.09 ±  7%      -0.8        3.24 ±  5%  
perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fsync
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.calltrace.cycles-pp.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.calltrace.cycles-pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
       5.84 ±  2%      -0.8        5.01 ±  4%  
perf-profile.calltrace.cycles-pp.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
       1.15 ±  7%      -0.8        0.36 ±100%  
perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.xfs_ilock
       4.38 ±  7%      -0.8        3.59 ±  3%  
perf-profile.calltrace.cycles-pp.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64
       5.17 ±  2%      -0.8        4.39 ±  5%  
perf-profile.calltrace.cycles-pp.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
       4.89 ±  4%      -0.7        4.18 ±  5%  
perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request.submit_bio
       1.38 ±  9%      -0.7        0.68 ± 22%  
perf-profile.calltrace.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
       1.38 ±  9%      -0.7        0.68 ± 22%  
perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev
       1.38 ±  9%      -0.7        0.68 ± 22%  
perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.xfs_ilock.xfs_file_buffered_aio_write
       1.38 ±  9%      -0.7        0.68 ± 22%  
perf-profile.calltrace.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
       4.01 ±  3%      -0.6        3.40 ±  5%  
perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepage
       4.01 ±  3%      -0.6        3.40 ±  5%  
perf-profile.calltrace.cycles-pp.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_writepages
       4.01 ±  3%      -0.6        3.40 ±  5%  
perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages
       3.99 ±  3%      -0.6        3.38 ±  5%  
perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs_add_to_ioend
       4.11 ±  3%      -0.6        3.51 ±  4%  
perf-profile.calltrace.cycles-pp.xfs_add_to_ioend.xfs_do_writepage.write_cache_pages.xfs_vm_writepages.do_writepages
       3.97 ±  5%      -0.5        3.43 ±  4%  
perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
       3.97 ±  5%      -0.5        3.43 ±  4%  
perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
       3.98 ±  5%      -0.5        3.44 ±  4%  
perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
       3.96 ±  5%      -0.5        3.43 ±  4%  
perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
       3.92 ±  5%      -0.5        3.40 ±  4%  
perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.xfs_submit_ioend
       2.89 ±  6%      -0.5        2.38 ±  6%  
perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
       2.77 ± 10%      -0.5        2.28 ±  2%  
perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
       2.77 ± 10%      -0.5        2.28 ±  2%  
perf-profile.calltrace.cycles-pp.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync
       2.77 ± 10%      -0.5        2.28 ±  2%  
perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync
       2.74 ± 10%      -0.5        2.25 ±  2%  
perf-profile.calltrace.cycles-pp.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range.filemap_write_and_wait_range
       3.12 ±  7%      -0.5        2.64 ±  4%  
perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write.vfs_write
       2.93 ±  7%      -0.5        2.47 ±  3%  
perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.__vfs_write
       3.37 ±  4%      -0.5        2.91 ±  4%  
perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write.vfs_write
       2.51 ±  4%      -0.4        2.06 ±  7%  
perf-profile.calltrace.cycles-pp.clear_page_erms.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec
       3.12 ±  5%      -0.4        2.68 ±  7%  
perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
       2.91 ±  5%      -0.4        2.48 ±  3%  
perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
       3.02 ±  5%      -0.4        2.59 ±  7%  
perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request
       2.59 ±  4%      -0.4        2.18 ±  5%  
perf-profile.calltrace.cycles-pp.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.__vfs_write
       2.84 ±  7%      -0.4        2.43 ±  3%  
perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
       3.27            -0.4        2.86 ±  6%  
perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
       3.20            -0.4        2.79 ±  6%  
perf-profile.calltrace.cycles-pp.memcpy_erms.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
       2.85 ±  5%      -0.4        2.44 ±  3%  
perf-profile.calltrace.cycles-pp.memcpy_erms._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
       2.50 ±  4%      -0.4        2.14 ±  7%  
perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
       2.41 ±  4%      -0.4        2.06 ±  7%  
perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
       2.02 ± 11%      -0.3        1.67 ±  3%  
perf-profile.calltrace.cycles-pp.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range
       0.60 ±  8%      -0.3        0.26 ±100%  
perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
       1.96 ± 11%      -0.3        1.62 ±  3%  
perf-profile.calltrace.cycles-pp.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages
       1.55 ±  2%      -0.3        1.22 ±  5%  
perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
       2.35 ±  4%      -0.3        2.03 ±  7%  
perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
       1.60 ±  2%      -0.3        1.32 ±  4%  
perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
       1.49 ±  6%      -0.2        1.24 ±  6%  
perf-profile.calltrace.cycles-pp._raw_spin_lock.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request
       1.03 ±  6%      -0.2        0.80 ±  7%  
perf-profile.calltrace.cycles-pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
       0.97 ±  7%      -0.2        0.74 ±  9%  
perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
       1.45 ±  6%      -0.2        1.23 ±  4%  
perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
       0.97 ±  4%      -0.2        0.77 ±  3%  
perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
       1.22 ±  6%      -0.2        1.03 ±  6%  
perf-profile.calltrace.cycles-pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work
       1.22 ±  6%      -0.2        1.03 ±  5%  
perf-profile.calltrace.cycles-pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work.worker_thread
       1.25 ±  5%      -0.2        1.06 ±  8%  
perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
       1.06 ±  2%      -0.2        0.87 ±  4%  
perf-profile.calltrace.cycles-pp.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync.do_fsync
       1.27 ±  5%      -0.2        1.11 ±  6%  
perf-profile.calltrace.cycles-pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
       0.78 ±  8%      -0.2        0.63 ±  6%  
perf-profile.calltrace.cycles-pp.__set_page_dirty_nobuffers.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
       1.11 ±  7%      -0.1        0.99 ±  6%  
perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
       0.68            -0.1        0.57 ±  5%  
perf-profile.calltrace.cycles-pp.nfs_create_request.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
       1.04 ±  7%      -0.1        0.93 ±  5%  
perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion
       1.02 ±  8%      -0.1        0.92 ±  4%  
perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback
       0.71 ±  2%      -0.1        0.60 ±  8%  
perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range
       0.72 ±  2%      -0.1        0.61 ±  8%  
perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range
       0.72            -0.1        0.62 ±  7%  
perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit_common.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_file_fsync
       0.93 ±  5%      -0.1        0.83 ±  5%  
perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
       0.73            -0.1        0.67 ±  5%  
perf-profile.calltrace.cycles-pp.end_page_writeback.xfs_destroy_ioend.process_one_work.worker_thread.kthread
       4.86 ±  6%      +0.5        5.39        
perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork
       0.00            +0.5        0.54 ±  3%  
perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.svc_recv.nfsd.kthread
       0.00            +0.6        0.59 ±  6%  
perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
       0.00            +0.6        0.63 ±  6%  
perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
       0.00            +0.7        0.65 ±  4%  
perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.svc_recv.nfsd.kthread.ret_from_fork
       0.00            +0.7        0.74 ±  8%  
perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
       0.00            +0.9        0.86 ±  8%  
perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
       9.98 ±  4%      +2.5       12.44 ±  4%  
perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
       9.61 ±  4%      +2.5       12.15 ±  4%  
perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      47.02 ±  3%      +2.7       49.68 ±  4%  
perf-profile.calltrace.cycles-pp.secondary_startup_64
       0.00            +3.5        3.54 ±  3%  
perf-profile.calltrace.cycles-pp.memcpy_erms.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
       0.00            +3.6        3.56 ±  3%  
perf-profile.calltrace.cycles-pp.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
       0.00            +3.7        3.67 ±  3%  
perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
       2.29 ±  6%      +3.8        6.05 ±  3%  
perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
       2.28 ±  6%      +3.8        6.05 ±  3%  
perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
       1.86 ±  3%      +3.8        5.64 ±  4%  
perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
       1.85 ±  3%      +3.8        5.64 ±  4%  
perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
       1.80 ±  4%      +3.8        5.60 ±  4%  
perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
       1.79 ±  3%      +3.8        5.60 ±  4%  
perf-profile.calltrace.cycles-pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute
       0.00            +5.5        5.47 ±  4%  
perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request
       0.00            +5.6        5.56 ±  4%  
perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit
       0.00            +5.6        5.57 ±  4%  
perf-profile.calltrace.cycles-pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit
      20.37            -3.3       17.04 ±  5%  
perf-profile.children.cycles-pp.svc_process_common
      20.38            -3.3       17.04 ±  5%  
perf-profile.children.cycles-pp.svc_process
      20.27            -3.3       16.95 ±  5%  
perf-profile.children.cycles-pp.nfsd_dispatch
      20.21            -3.3       16.91 ±  5%  
perf-profile.children.cycles-pp.nfsd4_proc_compound
      26.02            -3.1       22.91 ±  4%  
perf-profile.children.cycles-pp.nfsd
      15.88 ±  5%      -1.9       13.95 ±  3%  
perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      15.87 ±  5%      -1.9       13.95 ±  4%  
perf-profile.children.cycles-pp.do_syscall_64
      12.61 ±  3%      -1.9       10.73 ±  3%  
perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      12.60 ±  3%      -1.9       10.73 ±  3%  
perf-profile.children.cycles-pp.do_writepages
       9.04 ±  2%      -1.9        7.18 ±  7%  
perf-profile.children.cycles-pp.nfsd_vfs_write
       9.02 ±  2%      -1.9        7.17 ±  7%  
perf-profile.children.cycles-pp.do_iter_readv_writev
       9.02 ±  2%      -1.9        7.16 ±  7%  
perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
       9.03 ±  2%      -1.9        7.18 ±  7%  
perf-profile.children.cycles-pp.do_iter_write
       9.07 ±  2%      -1.9        7.22 ±  7%  
perf-profile.children.cycles-pp.nfsd4_write
       2.21 ±  4%      -1.8        0.41 ±  4%  
perf-profile.children.cycles-pp.inet_sendpage
       2.17 ±  4%      -1.8        0.40 ±  5%  
perf-profile.children.cycles-pp.tcp_sendpage
      10.50 ±  2%      -1.4        9.06 ±  4%  
perf-profile.children.cycles-pp.xfs_file_fsync
      10.51 ±  2%      -1.4        9.08 ±  4%  
perf-profile.children.cycles-pp.nfsd_commit
      10.45 ±  2%      -1.4        9.02 ±  4%  
perf-profile.children.cycles-pp.file_write_and_wait_range
       9.82 ±  3%      -1.4        8.45 ±  3%  
perf-profile.children.cycles-pp.xfs_vm_writepages
       8.59 ±  4%      -1.3        7.27 ±  4%  
perf-profile.children.cycles-pp.write_cache_pages
       8.02 ±  4%      -1.2        6.87 ±  4%  
perf-profile.children.cycles-pp.submit_bio
       8.02 ±  4%      -1.2        6.87 ±  4%  
perf-profile.children.cycles-pp.generic_make_request
       8.00 ±  4%      -1.1        6.85 ±  4%  
perf-profile.children.cycles-pp.brd_make_request
       7.94 ±  4%      -1.1        6.79 ±  4%  
perf-profile.children.cycles-pp.brd_do_bvec
      10.63 ±  5%      -1.1        9.49 ±  4%  
perf-profile.children.cycles-pp.write
       7.55            -1.1        6.41 ±  5%  
perf-profile.children.cycles-pp.iomap_apply
       7.54            -1.1        6.41 ±  6%  
perf-profile.children.cycles-pp.iomap_file_buffered_write
      10.28 ±  5%      -1.1        9.16 ±  4%  
perf-profile.children.cycles-pp.ksys_write
       7.50            -1.1        6.37 ±  6%  
perf-profile.children.cycles-pp.iomap_write_actor
       4.68 ±  5%      -1.1        3.56 ±  2%  
perf-profile.children.cycles-pp.inet_recvmsg
       4.68 ±  5%      -1.1        3.56 ±  2%  
perf-profile.children.cycles-pp.tcp_recvmsg
      10.24 ±  5%      -1.1        9.12 ±  4%  
perf-profile.children.cycles-pp.vfs_write
      10.08 ±  5%      -1.1        8.98 ±  4%  
perf-profile.children.cycles-pp.__vfs_write
      10.05 ±  5%      -1.1        8.95 ±  4%  
perf-profile.children.cycles-pp.nfs_file_write
       5.51 ±  5%      -1.1        4.43 ±  5%  
perf-profile.children.cycles-pp.rpc_free_task
       9.82 ±  5%      -1.1        8.74 ±  4%  
perf-profile.children.cycles-pp.generic_perform_write
       1.39 ±  5%      -1.1        0.32 ±  4%  
perf-profile.children.cycles-pp.tcp_sendpage_locked
       1.36 ±  5%      -1.1        0.31 ±  5%  
perf-profile.children.cycles-pp.do_tcp_sendpages
       5.31 ±  5%      -1.0        4.27 ±  5%  
perf-profile.children.cycles-pp.rpc_async_release
       4.39 ±  6%      -1.0        3.40 ±  2%  
perf-profile.children.cycles-pp.svc_recvfrom
       3.55 ±  5%      -0.9        2.64 ±  2%  
perf-profile.children.cycles-pp.__skb_datagram_iter
       3.55 ±  5%      -0.9        2.64 ±  2%  
perf-profile.children.cycles-pp.skb_copy_datagram_iter
       5.01 ±  6%      -0.9        4.15 ±  3%  
perf-profile.children.cycles-pp.fsync
       4.09 ±  7%      -0.9        3.24 ±  5%  
perf-profile.children.cycles-pp.nfs_write_completion
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.children.cycles-pp.__x64_sys_fsync
       4.91 ±  6%      -0.8        4.07 ±  3%  
perf-profile.children.cycles-pp.nfs_file_fsync
       4.90 ±  6%      -0.8        4.05 ±  3%  
perf-profile.children.cycles-pp.do_fsync
       5.36 ±  4%      -0.8        4.53 ±  5%  
perf-profile.children.cycles-pp.grab_cache_page_write_begin
       4.39 ±  7%      -0.8        3.60 ±  2%  
perf-profile.children.cycles-pp.filemap_write_and_wait_range
       5.18 ±  2%      -0.8        4.40 ±  5%  
perf-profile.children.cycles-pp.xfs_do_writepage
       5.21 ±  4%      -0.7        4.47 ±  5%  
perf-profile.children.cycles-pp.pagecache_get_page
       4.90 ±  4%      -0.7        4.19 ±  5%  
perf-profile.children.cycles-pp.brd_insert_page
       1.38 ±  9%      -0.7        0.68 ± 22%  
perf-profile.children.cycles-pp.xfs_ilock
       1.41 ±  8%      -0.7        0.71 ± 22%  
perf-profile.children.cycles-pp.call_rwsem_down_write_failed
       1.41 ±  8%      -0.7        0.71 ± 22%  
perf-profile.children.cycles-pp.rwsem_down_write_failed
       1.44 ±  7%      -0.7        0.74 ± 21%  
perf-profile.children.cycles-pp.down_write
       4.12 ±  3%      -0.6        3.52 ±  4%  
perf-profile.children.cycles-pp.xfs_add_to_ioend
       1.18 ±  6%      -0.6        0.61 ± 21%  
perf-profile.children.cycles-pp.rwsem_spin_on_owner
       3.98 ±  5%      -0.5        3.44 ±  4%  
perf-profile.children.cycles-pp.xfs_submit_ioend
       6.25            -0.5        5.72 ±  4%  
perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
       2.89 ±  6%      -0.5        2.38 ±  6%  
perf-profile.children.cycles-pp.nfs_end_page_writeback
       2.78 ± 10%      -0.5        2.29 ±  2%  
perf-profile.children.cycles-pp.nfs_writepages
       3.12 ±  7%      -0.5        2.64 ±  4%  
perf-profile.children.cycles-pp.nfs_write_begin
       3.37 ±  4%      -0.5        2.91 ±  4%  
perf-profile.children.cycles-pp.nfs_write_end
       2.54 ±  4%      -0.5        2.08 ±  7%  
perf-profile.children.cycles-pp.clear_page_erms
       2.92 ±  5%      -0.4        2.48 ±  3%  
perf-profile.children.cycles-pp._copy_to_iter
       2.60 ±  4%      -0.4        2.18 ±  5%  
perf-profile.children.cycles-pp.nfs_updatepage
       2.71 ±  4%      -0.4        2.29 ±  6%  
perf-profile.children.cycles-pp.add_to_page_cache_lru
       0.75 ±  3%      -0.4        0.37 ±  6%  
perf-profile.children.cycles-pp.release_sock
       2.51 ±  4%      -0.4        2.15 ±  7%  
perf-profile.children.cycles-pp.iomap_write_begin
       2.02 ± 10%      -0.3        1.67 ±  3%  
perf-profile.children.cycles-pp.nfs_writepages_callback
       2.23 ±  2%      -0.3        1.89 ±  4%  
perf-profile.children.cycles-pp.__filemap_fdatawait_range
       0.82 ±  7%      -0.3        0.47 ±  7%  
perf-profile.children.cycles-pp.__tcp_push_pending_frames
       1.96 ± 11%      -0.3        1.63 ±  3%  
perf-profile.children.cycles-pp.nfs_do_writepage
       1.55 ±  2%      -0.3        1.22 ±  5%  
perf-profile.children.cycles-pp.iomap_write_end
       0.78 ±  7%      -0.3        0.48 ±  6%  
perf-profile.children.cycles-pp.svc_send
       1.78 ±  3%      -0.3        1.48 ±  4%  
perf-profile.children.cycles-pp.end_page_writeback
       1.73 ±  3%      -0.3        1.45 ±  4%  
perf-profile.children.cycles-pp.test_clear_page_writeback
       0.90 ±  5%      -0.3        0.63 ±  8%  
perf-profile.children.cycles-pp.xas_load
       0.35 ± 14%      -0.2        0.10 ± 14%  
perf-profile.children.cycles-pp.simple_copy_to_iter
       1.78 ±  4%      -0.2        1.54 ±  7%  
perf-profile.children.cycles-pp.__wake_up_common
       0.43 ±  6%      -0.2        0.19 ±  7%  
perf-profile.children.cycles-pp.lock_sock_nested
       1.41 ±  5%      -0.2        1.18 ±  4%  
perf-profile.children.cycles-pp.nfs_commit_release
       1.41 ±  5%      -0.2        1.18 ±  4%  
perf-profile.children.cycles-pp.nfs_commit_release_pages
       1.95 ±  4%      -0.2        1.72 ±  5%  
perf-profile.children.cycles-pp.try_to_wake_up
       0.64 ±  8%      -0.2        0.41 ±  4%  
perf-profile.children.cycles-pp.svc_tcp_sendto
       0.63 ±  8%      -0.2        0.41 ±  5%  
perf-profile.children.cycles-pp.svc_sendto
       0.62 ±  8%      -0.2        0.41 ±  5%  
perf-profile.children.cycles-pp.svc_send_common
       0.62 ±  8%      -0.2        0.41 ±  5%  
perf-profile.children.cycles-pp.kernel_sendpage
       1.65 ±  5%      -0.2        1.44 ±  7%  
perf-profile.children.cycles-pp.autoremove_wake_function
       1.34 ±  3%      -0.2        1.14 ±  3%  
perf-profile.children.cycles-pp.wait_on_page_bit_common
       0.98 ±  4%      -0.2        0.79 ±  2%  
perf-profile.children.cycles-pp.poll_idle
       0.26 ±  8%      -0.2        0.08 ± 13%  
perf-profile.children.cycles-pp._raw_spin_lock_bh
       1.59 ±  5%      -0.2        1.42 ±  5%  
perf-profile.children.cycles-pp.schedule
       0.79 ±  5%      -0.2        0.62 ±  4%  
perf-profile.children.cycles-pp.tcp_v6_do_rcv
       0.64 ±  5%      -0.2        0.47 ±  8%  
perf-profile.children.cycles-pp.tcp_v6_rcv
       0.34 ± 12%      -0.2        0.18 ± 20%  
perf-profile.children.cycles-pp.xas_start
       0.24 ± 26%      -0.2        0.08 ± 40%  
perf-profile.children.cycles-pp.osq_lock
       0.76 ±  5%      -0.2        0.61 ±  5%  
perf-profile.children.cycles-pp.tcp_rcv_established
       0.49 ± 26%      -0.2        0.34 ± 14%  
perf-profile.children.cycles-pp.nfs_request_add_commit_list
       0.66 ±  5%      -0.1        0.52 ±  8%  
perf-profile.children.cycles-pp.ip6_input
       0.66 ±  4%      -0.1        0.51 ±  8%  
perf-profile.children.cycles-pp.ip6_input_finish
       0.65 ±  4%      -0.1        0.51 ±  9%  
perf-profile.children.cycles-pp.ip6_protocol_deliver_rcu
       0.79 ±  8%      -0.1        0.64 ±  7%  
perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
       0.83 ±  6%      -0.1        0.69 ±  8%  
perf-profile.children.cycles-pp.__nfs_commit_inode
       0.42 ±  4%      -0.1        0.29 ±  5%  
perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
       0.42 ±  4%      -0.1        0.29 ±  5%  
perf-profile.children.cycles-pp.xs_stream_data_receive
       0.70 ±  4%      -0.1        0.56 ±  6%  
perf-profile.children.cycles-pp.percpu_counter_add_batch
       0.71 ±  4%      -0.1        0.58 ±  8%  
perf-profile.children.cycles-pp.ipv6_rcv
       0.86 ±  6%      -0.1        0.73 ±  6%  
perf-profile.children.cycles-pp.__local_bh_enable_ip
       0.78 ±  5%      -0.1        0.65 ±  8%  
perf-profile.children.cycles-pp.process_backlog
       0.45 ± 14%      -0.1        0.32 ± 14%  
perf-profile.children.cycles-pp.__mutex_lock
       1.06 ±  4%      -0.1        0.93 ±  7%  
perf-profile.children.cycles-pp._raw_spin_lock_irqsave
       0.86 ±  6%      -0.1        0.73 ±  9%  
perf-profile.children.cycles-pp.pagevec_lru_move_fn
       0.75 ±  5%      -0.1        0.63 ±  8%  
perf-profile.children.cycles-pp.__netif_receive_skb_one_core
       0.61 ±  8%      -0.1        0.49 ±  9%  
perf-profile.children.cycles-pp.iomap_set_page_dirty
       0.94 ±  5%      -0.1        0.81 ±  8%  
perf-profile.children.cycles-pp.__lru_cache_add
       0.82 ±  6%      -0.1        0.70 ±  2%  
perf-profile.children.cycles-pp.clear_page_dirty_for_io
       0.80 ±  6%      -0.1        0.68 ±  7%  
perf-profile.children.cycles-pp.net_rx_action
       0.94 ±  2%      -0.1        0.82 ±  3%  
perf-profile.children.cycles-pp.io_schedule
       0.81 ±  7%      -0.1        0.70 ±  7%  
perf-profile.children.cycles-pp.do_softirq_own_stack
       0.69            -0.1        0.58 ±  5%  
perf-profile.children.cycles-pp.nfs_create_request
       0.62 ±  4%      -0.1        0.51 ±  5%  
perf-profile.children.cycles-pp.__pagevec_lru_add_fn
       0.32 ±  4%      -0.1        0.21 ± 11%  
perf-profile.children.cycles-pp.xs_sock_recvmsg
       0.49 ± 12%      -0.1        0.38 ±  7%  
perf-profile.children.cycles-pp.sched_ttwu_pending
       0.82 ±  6%      -0.1        0.72 ±  7%  
perf-profile.children.cycles-pp.do_softirq
       0.32 ± 10%      -0.1        0.22 ±  7%  
perf-profile.children.cycles-pp.xfs_map_blocks
       0.40 ± 13%      -0.1        0.30 ± 12%  
perf-profile.children.cycles-pp.mutex_spin_on_owner
       0.47 ±  2%      -0.1        0.38 ±  5%  
perf-profile.children.cycles-pp.kmem_cache_alloc
       0.26 ±  4%      -0.1        0.17 ±  8%  
perf-profile.children.cycles-pp.__wake_up_common_lock
       0.52 ±  6%      -0.1        0.43 ±  4%  
perf-profile.children.cycles-pp.account_page_dirtied
       0.22 ± 16%      -0.1        0.13 ±  9%  
perf-profile.children.cycles-pp.__check_object_size
       0.50 ±  8%      -0.1        0.42 ±  7%  
perf-profile.children.cycles-pp.iomap_set_range_uptodate
       0.41 ±  3%      -0.1        0.33 ± 14%  
perf-profile.children.cycles-pp.__set_page_dirty
       0.31 ± 11%      -0.1        0.24 ± 11%  
perf-profile.children.cycles-pp.nfs_io_completion_release
       0.41 ±  8%      -0.1        0.33 ±  7%  
perf-profile.children.cycles-pp.nfs_inode_remove_request
       0.25 ±  9%      -0.1        0.18 ±  8%  
perf-profile.children.cycles-pp.xfs_iomap_write_allocate
       0.33 ±  8%      -0.1        0.26 ±  4%  
perf-profile.children.cycles-pp.___might_sleep
       0.25 ±  5%      -0.1        0.18 ± 15%  
perf-profile.children.cycles-pp.__xa_set_mark
       0.31 ±  9%      -0.1        0.24 ±  5%  
perf-profile.children.cycles-pp.nfs_scan_commit_list
       0.25 ±  5%      -0.1        0.18 ±  8%  
perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
       0.45 ±  6%      -0.1        0.39 ±  4%  
perf-profile.children.cycles-pp.nfs_lock_and_join_requests
       0.50 ±  4%      -0.1        0.45 ±  5%  
perf-profile.children.cycles-pp.nfs_page_group_destroy
       0.19 ± 11%      -0.1        0.14 ±  8%  
perf-profile.children.cycles-pp.xfs_bmap_btalloc
       0.21 ±  2%      -0.1        0.15 ± 14%  
perf-profile.children.cycles-pp.__lock_sock
       0.17 ±  8%      -0.1        0.12 ±  7%  
perf-profile.children.cycles-pp.xfs_alloc_ag_vextent
       0.18 ±  9%      -0.1        0.13 ±  6%  
perf-profile.children.cycles-pp.xfs_alloc_vextent
       0.20 ± 12%      -0.1        0.15 ±  5%  
perf-profile.children.cycles-pp.xfs_bmapi_write
       0.30 ±  9%      -0.0        0.25 ±  5%  
perf-profile.children.cycles-pp.select_task_rq_fair
       0.16 ±  8%      -0.0        0.11 ±  7%  
perf-profile.children.cycles-pp.xfs_alloc_ag_vextent_near
       0.16 ± 15%      -0.0        0.12 ± 17%  
perf-profile.children.cycles-pp.__generic_write_end
       0.48 ±  2%      -0.0        0.44 ±  7%  
perf-profile.children.cycles-pp.dequeue_task_fair
       0.31 ±  2%      -0.0        0.27 ± 10%  
perf-profile.children.cycles-pp.release_pages
       0.20 ±  4%      -0.0        0.17 ± 10%  
perf-profile.children.cycles-pp.nfs_initiate_commit
       0.17 ± 10%      -0.0        0.13 ±  5%  
perf-profile.children.cycles-pp.__might_sleep
       0.29 ±  6%      -0.0        0.26 ±  4%  
perf-profile.children.cycles-pp.dec_zone_page_state
       0.23 ±  7%      -0.0        0.20 ±  7%  
perf-profile.children.cycles-pp.__pagevec_release
       0.19 ±  9%      -0.0        0.15 ±  9%  
perf-profile.children.cycles-pp.nfs_get_lock_context
       0.17 ± 13%      -0.0        0.13 ±  8%  
perf-profile.children.cycles-pp.check_preempt_curr
       0.09 ± 17%      -0.0        0.06 ± 20%  
perf-profile.children.cycles-pp.memset_erms
       0.14 ±  8%      -0.0        0.11 ±  6%  
perf-profile.children.cycles-pp.nfs_request_remove_commit_list
       0.07 ± 10%      -0.0        0.04 ± 58%  
perf-profile.children.cycles-pp.xfs_defer_finish_noroll
       0.15 ±  5%      -0.0        0.12 ±  6%  
perf-profile.children.cycles-pp.__nfs_find_lock_context
       0.19 ±  4%      -0.0        0.16 ± 11%  
perf-profile.children.cycles-pp.mem_cgroup_try_charge
       0.17 ±  8%      -0.0        0.15 ± 10%  
perf-profile.children.cycles-pp.vfs_create
       0.16 ±  2%      -0.0        0.14 ±  8%  
perf-profile.children.cycles-pp.__fprop_inc_percpu_max
       0.16 ±  8%      -0.0        0.13 ±  8%  
perf-profile.children.cycles-pp.xfs_create
       0.10 ±  4%      -0.0        0.07 ± 17%  
perf-profile.children.cycles-pp.xfs_file_aio_write_checks
       0.11 ± 10%      -0.0        0.08 ± 15%  
perf-profile.children.cycles-pp.nfs_pageio_doio
       0.11 ± 10%      -0.0        0.08 ± 15%  
perf-profile.children.cycles-pp.nfs_generic_pg_pgios
       0.19 ±  4%      -0.0        0.17 ±  7%  
perf-profile.children.cycles-pp.update_rq_clock
       0.15 ±  4%      -0.0        0.13 ±  5%  
perf-profile.children.cycles-pp.__xfs_trans_commit
       0.08 ± 15%      -0.0        0.06 ±  6%  
perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
       0.06 ± 15%      +0.0        0.07 ± 14%  
perf-profile.children.cycles-pp.selinux_ip_postroute
       0.08 ± 16%      +0.0        0.10 ± 10%  
perf-profile.children.cycles-pp.nf_hook_slow
       0.04 ± 60%      +0.0        0.08 ± 19%  
perf-profile.children.cycles-pp.__inc_numa_state
       0.03 ±100%      +0.0        0.07 ± 12%  
perf-profile.children.cycles-pp.get_task_policy
       0.41 ±  6%      +0.0        0.46        
perf-profile.children.cycles-pp.__release_sock
       0.14 ± 11%      +0.1        0.19 ±  8%  
perf-profile.children.cycles-pp.__list_add_valid
       0.09 ± 17%      +0.1        0.14 ± 10%  
perf-profile.children.cycles-pp.svc_xprt_do_enqueue
       0.14 ± 10%      +0.1        0.20 ±  2%  
perf-profile.children.cycles-pp.tcp_clean_rtx_queue
       0.00            +0.1        0.06 ± 11%  
perf-profile.children.cycles-pp.free_unref_page_commit
       0.10 ± 11%      +0.1        0.17 ±  6%  
perf-profile.children.cycles-pp.iov_iter_advance
       0.17 ±  9%      +0.1        0.24 ±  2%  
perf-profile.children.cycles-pp.tcp_ack
       0.11 ± 21%      +0.1        0.18 ±  2%  
perf-profile.children.cycles-pp.schedule_timeout
       0.00            +0.1        0.08 ± 17%  
perf-profile.children.cycles-pp.free_one_page
       0.23 ± 12%      +0.1        0.32 ±  4%  
perf-profile.children.cycles-pp.__kfree_skb
       0.10 ±  4%      +0.1        0.19 ±  6%  
perf-profile.children.cycles-pp.xas_create
       0.12 ± 17%      +0.1        0.24 ±  4%  
perf-profile.children.cycles-pp.skb_release_data
       0.00            +0.1        0.14 ±  5%  
perf-profile.children.cycles-pp.__free_pages_ok
       0.00            +0.2        0.21 ±  9%  
perf-profile.children.cycles-pp.skb_page_frag_refill
       0.00            +0.2        0.22 ±  9%  
perf-profile.children.cycles-pp.sk_page_frag_refill
       0.00            +0.2        0.25 ±  5%  
perf-profile.children.cycles-pp.__sk_flush_backlog
       0.03 ±100%      +0.4        0.39 ±  7%  
perf-profile.children.cycles-pp.free_pcppages_bulk
       0.07 ± 29%      +0.5        0.54 ±  6%  
perf-profile.children.cycles-pp.free_unref_page
       4.86 ±  6%      +0.5        5.40 ±  2%  
perf-profile.children.cycles-pp.svc_recv
       9.98 ±  4%      +2.5       12.44 ±  4%  
perf-profile.children.cycles-pp.worker_thread
       9.61 ±  4%      +2.5       12.15 ±  4%  
perf-profile.children.cycles-pp.process_one_work
      47.02 ±  3%      +2.7       49.68 ±  4%  
perf-profile.children.cycles-pp.secondary_startup_64
      47.02 ±  3%      +2.7       49.68 ±  4%  
perf-profile.children.cycles-pp.cpu_startup_entry
      47.02 ±  3%      +2.7       49.69 ±  4%  
perf-profile.children.cycles-pp.do_idle
       6.12 ±  2%      +2.8        8.88 ±  3%  
perf-profile.children.cycles-pp.memcpy_erms
       0.00            +3.6        3.58 ±  3%  
perf-profile.children.cycles-pp.memcpy_from_page
       0.00            +3.7        3.68 ±  3%  
perf-profile.children.cycles-pp._copy_from_iter_full
       2.43 ±  6%      +3.8        6.20 ±  3%  
perf-profile.children.cycles-pp.__rpc_execute
       2.29 ±  6%      +3.8        6.05 ±  3%  
perf-profile.children.cycles-pp.rpc_async_schedule
       1.93 ±  3%      +3.8        5.71 ±  4%  
perf-profile.children.cycles-pp.call_transmit
       1.92 ±  4%      +3.8        5.71 ±  4%  
perf-profile.children.cycles-pp.xprt_transmit
       1.87 ±  4%      +3.8        5.67 ±  4%  
perf-profile.children.cycles-pp.xs_tcp_send_request
       1.86 ±  4%      +3.8        5.67 ±  4%  
perf-profile.children.cycles-pp.xs_sendpages
       0.21 ±  7%      +5.3        5.54 ±  4%  
perf-profile.children.cycles-pp.tcp_sendmsg_locked
       0.24 ±  8%      +5.4        5.63 ±  4%  
perf-profile.children.cycles-pp.tcp_sendmsg
       0.25 ±  9%      +5.4        5.64 ±  4%  
perf-profile.children.cycles-pp.sock_sendmsg
       1.17 ±  7%      -0.6        0.61 ± 21%  
perf-profile.self.cycles-pp.rwsem_spin_on_owner
       2.50 ±  4%      -0.5        2.05 ±  7%  
perf-profile.self.cycles-pp.clear_page_erms
       2.46 ±  4%      -0.4        2.06 ±  5%  
perf-profile.self.cycles-pp.brd_do_bvec
       0.22 ± 10%      -0.2        0.03 ±100%  
perf-profile.self.cycles-pp.__skb_datagram_iter
       1.53 ±  6%      -0.2        1.34 ±  3%  
perf-profile.self.cycles-pp._raw_spin_lock
       0.91 ±  4%      -0.2        0.74 ±  2%  
perf-profile.self.cycles-pp.poll_idle
       0.24 ± 25%      -0.2        0.08 ± 40%  
perf-profile.self.cycles-pp.osq_lock
       0.74 ± 14%      -0.2        0.59 ± 10%  
perf-profile.self.cycles-pp.nfs_do_writepage
       0.32 ± 13%      -0.1        0.17 ± 18%  
perf-profile.self.cycles-pp.xas_start
       0.97 ±  3%      -0.1        0.84 ±  8%  
perf-profile.self.cycles-pp._raw_spin_lock_irqsave
       0.57 ±  6%      -0.1        0.45 ±  9%  
perf-profile.self.cycles-pp.xas_load
       0.55 ± 10%      -0.1        0.43 ±  7%  
perf-profile.self.cycles-pp.__schedule
       0.14 ± 12%      -0.1        0.03 ±100%  
perf-profile.self.cycles-pp._raw_spin_lock_bh
       0.53 ±  8%      -0.1        0.43 ±  9%  
perf-profile.self.cycles-pp.percpu_counter_add_batch
       0.40 ± 13%      -0.1        0.30 ± 12%  
perf-profile.self.cycles-pp.mutex_spin_on_owner
       0.33 ±  8%      -0.1        0.24 ± 12%  
perf-profile.self.cycles-pp.nfs_end_page_writeback
       0.74 ±  6%      -0.1        0.65 ±  8%  
perf-profile.self.cycles-pp.nfs_updatepage
       0.40 ±  8%      -0.1        0.31 ± 11%  
perf-profile.self.cycles-pp.__test_set_page_writeback
       0.50 ±  8%      -0.1        0.41 ±  7%  
perf-profile.self.cycles-pp.iomap_set_range_uptodate
       0.56 ±  5%      -0.1        0.48 ±  8%  
perf-profile.self.cycles-pp.try_to_wake_up
       0.38 ±  8%      -0.1        0.30 ±  5%  
perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
       0.28 ±  9%      -0.1        0.20 ±  6%  
perf-profile.self.cycles-pp.iomap_write_end
       0.45 ±  7%      -0.1        0.38 ±  6%  
perf-profile.self.cycles-pp.__pagevec_lru_add_fn
       0.15 ± 14%      -0.1        0.08 ± 27%  
perf-profile.self.cycles-pp.add_to_page_cache_lru
       0.43 ±  6%      -0.1        0.36        
perf-profile.self.cycles-pp.clear_page_dirty_for_io
       0.51 ±  8%      -0.1        0.45 ±  7%  
perf-profile.self.cycles-pp.test_clear_page_writeback
       0.20 ±  9%      -0.1        0.14 ± 11%  
perf-profile.self.cycles-pp.xas_store
       0.31 ±  8%      -0.1        0.25 ±  4%  
perf-profile.self.cycles-pp.___might_sleep
       0.15 ± 20%      -0.1        0.09 ± 12%  
perf-profile.self.cycles-pp.__check_object_size
       0.22 ±  6%      -0.1        0.17 ± 10%  
perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
       0.08 ± 11%      -0.0        0.03 ±100%  
perf-profile.self.cycles-pp.switch_mm_irqs_off
       0.28 ±  4%      -0.0        0.24 ± 10%  
perf-profile.self.cycles-pp.release_pages
       0.27 ±  8%      -0.0        0.23 ±  2%  
perf-profile.self.cycles-pp.dec_zone_page_state
       0.19 ±  6%      -0.0        0.15 ± 12%  
perf-profile.self.cycles-pp.kmem_cache_alloc
       0.15 ± 10%      -0.0        0.11 ±  7%  
perf-profile.self.cycles-pp.__might_sleep
       0.25 ± 10%      -0.0        0.21 ±  6%  
perf-profile.self.cycles-pp.wait_on_page_bit_common
       0.07 ± 12%      -0.0        0.03 ±100%  
perf-profile.self.cycles-pp.___slab_alloc
       0.23 ±  7%      -0.0        0.20 ± 10%  
perf-profile.self.cycles-pp._raw_spin_lock_irq
       0.18 ±  5%      -0.0        0.15 ±  2%  
perf-profile.self.cycles-pp.select_task_rq_fair
       0.14 ±  5%      -0.0        0.11 ± 17%  
perf-profile.self.cycles-pp.alloc_pages_current
       0.09 ± 16%      -0.0        0.06 ± 20%  
perf-profile.self.cycles-pp.memset_erms
       0.10 ±  8%      -0.0        0.07        
perf-profile.self.cycles-pp._cond_resched
       0.15 ±  4%      -0.0        0.12 ±  6%  
perf-profile.self.cycles-pp.__nfs_find_lock_context
       0.11 ± 13%      -0.0        0.09 ± 14%  
perf-profile.self.cycles-pp.nfs_request_add_commit_list_locked
       0.09 ±  4%      -0.0        0.07        
perf-profile.self.cycles-pp.nfs_request_remove_commit_list
       0.08 ± 10%      -0.0        0.06 ± 11%  
perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
       0.08 ± 10%      -0.0        0.06 ± 11%  
perf-profile.self.cycles-pp.nfs_request_add_commit_list
       0.10 ±  5%      -0.0        0.08 ±  5%  
perf-profile.self.cycles-pp.___perf_sw_event
       0.08 ± 13%      +0.0        0.11 ± 10%  
perf-profile.self.cycles-pp.mem_cgroup_commit_charge
       0.03 ±100%      +0.0        0.07 ± 12%  
perf-profile.self.cycles-pp.run_timer_softirq
       0.03 ±100%      +0.0        0.07 ± 12%  
perf-profile.self.cycles-pp.get_task_policy
       0.03 ±100%      +0.0        0.07 ± 17%  
perf-profile.self.cycles-pp.__inc_numa_state
       0.03 ±102%      +0.1        0.08 ±  8%  
perf-profile.self.cycles-pp.svc_recv
       0.11 ± 13%      +0.1        0.17 ± 12%  
perf-profile.self.cycles-pp.__list_add_valid
       0.09 ± 14%      +0.1        0.16 ±  9%  
perf-profile.self.cycles-pp.iov_iter_advance
       0.06 ± 13%      +0.1        0.15 ±  8%  
perf-profile.self.cycles-pp.xas_create
       0.00            +0.1        0.15 ± 14%  
perf-profile.self.cycles-pp.tcp_sendmsg_locked
       0.00            +0.2        0.20 ± 11%  
perf-profile.self.cycles-pp.free_pcppages_bulk
       0.04 ± 60%      +0.2        0.28 ±  7%  
perf-profile.self.cycles-pp.svc_tcp_recvfrom
       1.25            +0.3        1.52 ±  5%  
perf-profile.self.cycles-pp.get_page_from_freelist
       6.04 ±  2%      +2.7        8.74 ±  3%  
perf-profile.self.cycles-pp.memcpy_erms


fsmark.time.percent_of_cpu_this_job_got 320 +-+-------------------------------------------------------------------+
       |                                                                     |
   300 +-++.                                                          +      |
       |.   +..    .+.     +..  +..                              +. .. +  .+.|
   280 +-+     +.+.   +.. :    +    .+..+.  .+.  .+.  .+.+..   ..  +    +.   |
       |                  :   +    +      +.   +.   +.       .+              |
   260 +-+               +                                  +                |
       |                                                                     |
   240 +-+                                                                   |
       |                                                                     |
   220 +-+                                                                   |
       |       O                                         O  O O  O O  O      |
   200 O-+       O  O      O       O           O  O O  O                     |
       |  O O         O  O    O O    O  O O  O                               |
   180 +-+-------------------------------------------------------------------+
fsmark.time.elapsed_time 32 +-+O-------------------------------------------------------------------+
      O    O    O       O O  O O  O O  O  O O                                |
   31 +-+     O    O O                         O O  O O    O  O O  O O       |
   30 +-+                                                O                   |
      |                                                                      |
   29 +-+                                                                    |
   28 +-+                                                                    |
      |                                                                      |
   27 +-+                                                                    |
   26 +-+                                                    .+              |
      |                 +         +                 +.+.. .+.  +  .+.  .+.   |
   25 +-+             .. +  .+   + :   +..+.+..+. ..     +      +.   +.   +..|
   24 +-+    .+.+..+.+    +.  + +  : ..          +                           |
      |  +.+.                  +    +                                        |
   23 +-+--------------------------------------------------------------------+
fsmark.time.elapsed_time.max 32 +-+O-------------------------------------------------------------------+
      O    O    O       O O  O O  O O  O  O O                                |
   31 +-+     O    O O                         O O  O O    O  O O  O O       |
   30 +-+                                                O                   |
      |                                                                      |
   29 +-+                                                                    |
   28 +-+                                                                    |
      |                                                                      |
   27 +-+                                                                    |
   26 +-+                                                    .+              |
      |                 +         +                 +.+.. .+.  +  .+.  .+.   |
   25 +-+             .. +  .+   + :   +..+.+..+. ..     +      +.   +.   +..|
   24 +-+    .+.+..+.+    +.  + +  : ..          +                           |
      |  +.+.                  +    +                                        |
   23 +-+--------------------------------------------------------------------+
fsmark.files_per_sec 520 +-+-------------------------------------------------------------------+
       |..     +.+.  +     +.. .+.. :            .+.    .+..         .+.+..+ |
   500 +-+            +.. +   +     :   +.+..+.+.   +..+        .+.+.        |
   480 +-+               +         +                        +.+.             |
       |                                                                     |
   460 +-+                                                                   |
       |                                                                     |
   440 +-+                                                                   |
       |                                                                     |
   420 +-+                                                                   |
   400 +-+                                               O                   |
       |                                       O  O O  O    O O  O O  O      |
   380 O-+     O O  O O  O O            O O  O                               |
       |  O O                 O O  O O                                       |
   360 +-+-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


_______________________________________________
LKP mailing list
l...@lists.01.org
https://lists.01.org/mailman/listinfo/lkp


Do you have time to take a look at this regression?

--
Zhengjun Xing

Reply via email to