On 6/10/2020 11:07 AM, Xing Zhengjun wrote:
Hi Darrick,

    Do you have time to take a look at this? Thanks.



Ping...


On 6/6/2020 11:48 PM, kernel test robot wrote:
Greeting,

FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:


commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes to delalloc regions to unwritten")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

    disk: 1BRD_48G
    fs: xfs
    test: sync_disk_rw
    load: 600
    cpufreq_governor: performance
    ucode: 0x42e

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.c...@intel.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

         git clone https://github.com/intel/lkp-tests.git
         cd lkp-tests
         bin/lkp install job.yaml  # job file is attached in this email
         bin/lkp run     job.yaml

========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e

commit:
   590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
   a5949d3fae ("xfs: force writes to delalloc regions to unwritten")

590b16516ef38e2e a5949d3faedf492fa7863b914da
---------------- ---------------------------
        fail:runs  %reproduction    fail:runs
            |             |             |
            :4           50%           2:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          %stddev     %change         %stddev
              \          |                \
      35272           -33.6%      23430        aim7.jobs-per-min
     102.13           +50.5%     153.75        aim7.time.elapsed_time
     102.13           +50.5%     153.75        aim7.time.elapsed_time.max
    1388038           +40.2%    1945838 aim7.time.involuntary_context_switches       43420 ±  2%     +13.4%      49255 ±  2% aim7.time.minor_page_faults
       3123           +44.2%       4504 ±  2%  aim7.time.system_time
      59.31            +6.5%      63.18        aim7.time.user_time
   48595108           +58.6%   77064959 aim7.time.voluntary_context_switches
       1.44           -28.8%       1.02        iostat.cpu.user
       0.07 ±  6%      +0.4        0.44 ±  7%  mpstat.cpu.all.iowait%
       1.44            -0.4        1.02        mpstat.cpu.all.usr%
       8632 ± 50%     +75.6%      15156 ± 34% numa-meminfo.node0.KernelStack        6583 ±136%    +106.0%      13562 ± 82% numa-meminfo.node0.PageTables       63325 ± 11%     +14.3%      72352 ± 12% numa-meminfo.node0.SUnreclaim        8647 ± 50%     +75.3%      15156 ± 34% numa-vmstat.node0.nr_kernel_stack        1656 ±136%    +104.6%       3389 ± 82% numa-vmstat.node0.nr_page_table_pages       15831 ± 11%     +14.3%      18087 ± 12% numa-vmstat.node0.nr_slab_unreclaimable
      93640 ±  3%     +41.2%     132211 ±  2%  meminfo.AnonHugePages
      21641           +39.9%      30271 ±  4%  meminfo.KernelStack
     129269           +12.3%     145114        meminfo.SUnreclaim
      28000           -31.2%      19275        meminfo.max_used_kB
    1269307           -26.9%     927657        vmstat.io.bo
     149.75 ±  3%     -17.4%     123.75 ±  4%  vmstat.procs.r
     718992           +13.3%     814567        vmstat.system.cs
     231397            -9.3%     209881 ±  2%  vmstat.system.in
  6.774e+08           +70.0%  1.152e+09        cpuidle.C1.time
   18203372           +60.4%   29198744        cpuidle.C1.usage
  2.569e+08 ± 18%     +81.8%  4.672e+08 ±  5%  cpuidle.C1E.time
    2691402 ± 13%     +98.7%    5346901 ±  3%  cpuidle.C1E.usage
     990350           +95.0%    1931226 ±  2%  cpuidle.POLL.time
     520061           +97.7%    1028004 ±  2%  cpuidle.POLL.usage
      77231            +1.8%      78602        proc-vmstat.nr_active_anon
      19868            +3.8%      20615        proc-vmstat.nr_dirty
     381302            +1.0%     384969        proc-vmstat.nr_file_pages
       4388            -2.7%       4270 proc-vmstat.nr_inactive_anon       69865            +4.7%      73155 proc-vmstat.nr_inactive_file       21615           +40.0%      30251 ±  4% proc-vmstat.nr_kernel_stack
       7363            -3.2%       7127        proc-vmstat.nr_mapped
      12595 ±  3%      +5.2%      13255 ±  4%  proc-vmstat.nr_shmem
      19619            +3.2%      20247 proc-vmstat.nr_slab_reclaimable       32316           +12.3%      36280 proc-vmstat.nr_slab_unreclaimable       77231            +1.8%      78602 proc-vmstat.nr_zone_active_anon        4388            -2.7%       4270 proc-vmstat.nr_zone_inactive_anon       69865            +4.7%      73155 proc-vmstat.nr_zone_inactive_file       19165            +3.4%      19824 proc-vmstat.nr_zone_write_pending        9845 ± 27%     +68.0%      16539 ± 37% proc-vmstat.numa_hint_faults_local
    8195135            +2.5%    8402537        proc-vmstat.numa_hit
    8179414            +2.5%    8386209        proc-vmstat.numa_local
      15720            +3.9%      16328        proc-vmstat.numa_other
    8281098            +2.4%    8477157        proc-vmstat.pgalloc_normal
     342925 ±  2%     +39.8%     479279        proc-vmstat.pgfault
    8132789            +2.0%    8298189        proc-vmstat.pgfree
  1.327e+08            +9.7%  1.455e+08        proc-vmstat.pgpgout
     394.00          +151.7%     991.75 ±  5% slabinfo.biovec-max.active_objs      394.00          +152.1%     993.25 ±  5% slabinfo.biovec-max.num_objs
      21028 ±  6%      -7.8%      19389 ±  5%  slabinfo.filp.active_objs
      21520 ±  5%      -7.6%      19886 ±  5%  slabinfo.filp.num_objs
       4955          +234.9%      16594 ±  2% slabinfo.kmalloc-256.active_objs       77.25          +238.5%     261.50 ±  2% slabinfo.kmalloc-256.active_slabs        4955          +238.4%      16768 ±  2% slabinfo.kmalloc-256.num_objs       77.25          +238.5%     261.50 ±  2% slabinfo.kmalloc-256.num_slabs        3368 ±  2%     +22.6%       4128 ±  2% slabinfo.kmalloc-2k.active_objs        3446 ±  2%     +21.9%       4200 ±  2% slabinfo.kmalloc-2k.num_objs       11815 ±  8%     +29.0%      15239 ±  3% slabinfo.kmalloc-512.active_objs       11824 ±  9%     +29.0%      15249 ±  3% slabinfo.kmalloc-512.num_objs        6849 ±  5%      +9.0%       7463 ±  2% slabinfo.kmalloc-96.active_objs        6906 ±  5%      +8.9%       7521 ±  2% slabinfo.kmalloc-96.num_objs      765.00 ±  4%     +25.0%     956.25 ±  5% slabinfo.mnt_cache.active_objs      765.00 ±  4%     +25.0%     956.25 ±  5% slabinfo.mnt_cache.num_objs       13866           +12.5%      15594 slabinfo.proc_inode_cache.active_objs       13905           +12.2%      15598 slabinfo.proc_inode_cache.num_objs        1954           +24.4%       2431 ±  3% slabinfo.sighand_cache.active_objs        1954           +24.4%       2431 ±  3% slabinfo.sighand_cache.num_objs        2596           +13.7%       2951 ±  2% slabinfo.signal_cache.active_objs        2596           +13.7%       2951 ±  2% slabinfo.signal_cache.num_objs        4368 ±  6%     -11.7%       3856 ±  2% slabinfo.skbuff_head_cache.num_objs      822.25 ± 11%     +21.4%     998.50 ±  3% slabinfo.task_group.active_objs      822.25 ± 11%     +21.4%     998.50 ±  3% slabinfo.task_group.num_objs        1612           +32.1%       2129 ±  3% slabinfo.task_struct.active_objs      540.50           +31.6%     711.25 ±  3% slabinfo.task_struct.active_slabs        1622           +31.6%       2134 ±  3% slabinfo.task_struct.num_objs      540.50           +31.6%     711.25 ±  3% slabinfo.task_struct.num_slabs        6905           +20.0%       8289 slabinfo.vmap_area.active_objs        6933           +20.0%       8322 slabinfo.vmap_area.num_objs      646.25 ±  7%    +195.2%       1907 slabinfo.xfs_buf.active_objs
     646.25 ±  7%    +195.2%       1907        slabinfo.xfs_buf.num_objs
       1.78 ±  2%      -0.3        1.51 ±  2% perf-stat.i.branch-miss-rate%
   64079048            +3.9%   66551536        perf-stat.i.branch-misses
      17.99            -4.2       13.84 perf-stat.i.cache-miss-rate%
   66630506           -16.5%   55647472        perf-stat.i.cache-misses
  3.899e+08            +9.4%  4.265e+08 perf-stat.i.cache-references      731029           +13.1%     826756 perf-stat.i.context-switches
       3.79            -1.7%       3.73        perf-stat.i.cpi
     113320            +8.3%     122693        perf-stat.i.cpu-migrations
       1498           +20.4%       1803 perf-stat.i.cycles-between-cache-misses    50450547 ±  4%      -8.0%   46402743 ±  5% perf-stat.i.dTLB-load-misses        0.20 ±  3%      -0.0        0.16 ±  2% perf-stat.i.dTLB-store-miss-rate%     4587096 ±  3%     -22.9%    3538422 ±  2% perf-stat.i.dTLB-store-misses
  2.297e+09            -5.6%  2.168e+09        perf-stat.i.dTLB-stores
      86.68            -2.4       84.25 perf-stat.i.iTLB-load-miss-rate%    19484791            -6.1%   18297125 perf-stat.i.iTLB-load-misses
    3060023           +14.4%    3501587 ±  4%  perf-stat.i.iTLB-loads
       0.29            -1.5%       0.28        perf-stat.i.ipc
       0.21 ±  5%     -20.5%       0.17 ±  6%  perf-stat.i.metric.K/sec
       3120            -5.8%       2938        perf-stat.i.minor-faults
      42.61            +1.2       43.83 perf-stat.i.node-load-miss-rate%    24887573           -10.1%   22384693 perf-stat.i.node-load-misses
   31814752           -12.5%   27830232        perf-stat.i.node-loads
      36.62            +0.8       37.45 perf-stat.i.node-store-miss-rate%    22617213           -20.3%   18015607 perf-stat.i.node-store-misses
   35707079           -19.9%   28604167        perf-stat.i.node-stores
       3120            -5.8%       2938        perf-stat.i.page-faults
      13.96            +9.3%      15.25        perf-stat.overall.MPKI
       0.99            +0.0        1.03 perf-stat.overall.branch-miss-rate%       17.09            -4.0       13.05 perf-stat.overall.cache-miss-rate%        1555           +19.4%       1856 perf-stat.overall.cycles-between-cache-misses        0.69 ±  4%      -0.1        0.64 ±  5% perf-stat.overall.dTLB-load-miss-rate%        0.20 ±  3%      -0.0        0.16 ±  2% perf-stat.overall.dTLB-store-miss-rate%       86.43            -2.5       83.94 perf-stat.overall.iTLB-load-miss-rate%        1433            +6.7%       1528 perf-stat.overall.instructions-per-iTLB-miss       43.89            +0.7       44.58 perf-stat.overall.node-load-miss-rate%   6.389e+09            +0.8%  6.438e+09 perf-stat.ps.branch-instructions
   63452800            +4.2%   66110454        perf-stat.ps.branch-misses
   65977234           -16.2%   55277568        perf-stat.ps.cache-misses
  3.861e+08            +9.8%  4.237e+08 perf-stat.ps.cache-references      723870           +13.5%     821265 perf-stat.ps.context-switches      112209            +8.6%     121881 perf-stat.ps.cpu-migrations    49955941 ±  4%      -7.7%   46094243 ±  5% perf-stat.ps.dTLB-load-misses     4542094 ±  3%     -22.6%    3514899 ±  2% perf-stat.ps.dTLB-store-misses
  2.275e+09            -5.3%  2.154e+09        perf-stat.ps.dTLB-stores
   19294019            -5.8%   18175580 perf-stat.ps.iTLB-load-misses
    3030049           +14.8%    3478314 ±  4%  perf-stat.ps.iTLB-loads
       3089            -5.5%       2918        perf-stat.ps.minor-faults
   24643655            -9.8%   22235787 perf-stat.ps.node-load-misses
   31502901           -12.2%   27645143        perf-stat.ps.node-loads
   22395376           -20.1%   17895417 perf-stat.ps.node-store-misses
   35356897           -19.6%   28413366        perf-stat.ps.node-stores
       3089            -5.5%       2918        perf-stat.ps.page-faults
   2.85e+12           +50.6%  4.293e+12 perf-stat.total.instructions       84193 ± 36%    +135.9%     198575 ± 25% sched_debug.cfs_rq:/.MIN_vruntime.avg      919053           +98.2%    1821460 ±  2% sched_debug.cfs_rq:/.MIN_vruntime.max      256686 ± 16%    +114.7%     551162 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev       19497          +106.7%      40307 sched_debug.cfs_rq:/.exec_clock.avg       21240          +101.1%      42717 sched_debug.cfs_rq:/.exec_clock.max       19079          +107.3%      39542 sched_debug.cfs_rq:/.exec_clock.min      394.52 ±  2%     +54.3%     608.79 ± 13% sched_debug.cfs_rq:/.exec_clock.stddev      965.25 ± 15%     +59.7%       1541 ±  4% sched_debug.cfs_rq:/.load_avg.avg        2866 ± 21%     +57.1%       4503 ± 10% sched_debug.cfs_rq:/.load_avg.max      659.77 ± 15%     +68.9%       1114 ±  9% sched_debug.cfs_rq:/.load_avg.stddev       84193 ± 36%    +135.9%     198575 ± 25% sched_debug.cfs_rq:/.max_vruntime.avg      919053           +98.2%    1821462 ±  2% sched_debug.cfs_rq:/.max_vruntime.max      256686 ± 16%    +114.7%     551163 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev      901765           +97.1%    1777741 ±  2% sched_debug.cfs_rq:/.min_vruntime.avg      941794           +95.8%    1844445 ±  2% sched_debug.cfs_rq:/.min_vruntime.max      872925           +96.5%    1715658 ±  2% sched_debug.cfs_rq:/.min_vruntime.min       18358 ± 44%    +146.6%      45281 ± 34% sched_debug.cfs_rq:/.min_vruntime.stddev        1.22 ±  8%    +159.9%       3.18 ± 20% sched_debug.cfs_rq:/.nr_spread_over.avg        6.75 ± 23%     +65.4%      11.17 ± 17% sched_debug.cfs_rq:/.nr_spread_over.max        1.38 ± 11%     +61.4%       2.22 ± 12% sched_debug.cfs_rq:/.nr_spread_over.stddev       18357 ± 44%    +146.5%      45250 ± 34% sched_debug.cfs_rq:/.spread0.stddev      773.25 ±  3%     +13.2%     875.02 sched_debug.cfs_rq:/.util_avg.avg      285624           -30.3%     199111 ±  4% sched_debug.cpu.avg_idle.avg      576507 ±  2%     -26.2%     425271 ±  7% sched_debug.cpu.avg_idle.max       51029 ± 68%     -83.4%       8469 ± 29% sched_debug.cpu.avg_idle.min      135149 ±  7%     -23.4%     103460 ± 10% sched_debug.cpu.avg_idle.stddev
      57492           +51.0%      86835        sched_debug.cpu.clock.avg
      57495           +51.0%      86841        sched_debug.cpu.clock.max
      57487           +51.0%      86828        sched_debug.cpu.clock.min
       1.98 ±  4%     +74.8%       3.46 ± 10% sched_debug.cpu.clock.stddev       57492           +51.0%      86835 sched_debug.cpu.clock_task.avg       57495           +51.0%      86841 sched_debug.cpu.clock_task.max       57487           +51.0%      86828 sched_debug.cpu.clock_task.min        1.98 ±  4%     +74.8%       3.46 ± 10% sched_debug.cpu.clock_task.stddev        2831           +43.0%       4049 sched_debug.cpu.curr->pid.max      417519          +133.3%     974078 sched_debug.cpu.nr_switches.avg      427854          +134.0%    1001016 sched_debug.cpu.nr_switches.max      409942          +131.8%     950218 sched_debug.cpu.nr_switches.min        4770 ± 42%    +238.7%      16156 ± 22% sched_debug.cpu.nr_switches.stddev        4.70 ± 23%     +37.1%       6.44 ±  4% sched_debug.cpu.nr_uninterruptible.avg      776.62 ± 19%     +86.1%       1445 ± 12% sched_debug.cpu.nr_uninterruptible.max     -472.62           +98.3%    -937.33 sched_debug.cpu.nr_uninterruptible.min      289.80 ±  5%    +109.7%     607.68 ±  3% sched_debug.cpu.nr_uninterruptible.stddev      415893          +133.8%     972526 sched_debug.cpu.sched_count.avg      423960          +135.5%     998457 sched_debug.cpu.sched_count.max      409113          +132.0%     949068 sched_debug.cpu.sched_count.min        4194 ± 50%    +274.3%      15698 ± 22% sched_debug.cpu.sched_count.stddev      110094          +125.8%     248571 sched_debug.cpu.sched_goidle.avg      112236          +125.6%     253163 sched_debug.cpu.sched_goidle.max      108281          +125.8%     244506 sched_debug.cpu.sched_goidle.min      772.04 ± 16%    +175.8%       2129 ± 18% sched_debug.cpu.sched_goidle.stddev      297267          +137.2%     705173 sched_debug.cpu.ttwu_count.avg      320112          +135.9%     755096 sched_debug.cpu.ttwu_count.max      272487 ±  2%    +142.2%     660099 sched_debug.cpu.ttwu_count.min       10662 ± 10%    +110.9%      22491 ± 18% sched_debug.cpu.ttwu_count.stddev       12295 ±  2%    +834.9%     114954 ±  2% sched_debug.cpu.ttwu_local.avg       13726 ±  2%    +769.7%     119376 ±  2% sched_debug.cpu.ttwu_local.max       10620 ±  4%    +942.4%     110710 sched_debug.cpu.ttwu_local.min      693.29 ± 14%    +228.7%       2278 ± 18% sched_debug.cpu.ttwu_local.stddev
      57488           +51.0%      86829        sched_debug.cpu_clk
      56998           +51.5%      86337        sched_debug.ktime
      57977           +50.6%      87317        sched_debug.sched_clk
      72.50 ± 37%     +55.5%     112.75 ± 26% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1       53.25 ±  2%     +84.5%      98.25 ± 22% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2       73.50 ± 26%    +116.3%     159.00 ± 60% interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3      204541           +49.0%     304702 ±  2% interrupts.CPU0.LOC:Local_timer_interrupts      290268 ±  2%     +29.1%     374641 ±  4% interrupts.CPU0.RES:Rescheduling_interrupts      204580           +48.9%     304667 ±  2% interrupts.CPU1.LOC:Local_timer_interrupts      290342 ±  2%     +28.5%     373110 ±  4% interrupts.CPU1.RES:Rescheduling_interrupts      204537           +48.9%     304619 ±  2% interrupts.CPU10.LOC:Local_timer_interrupts      291605 ±  2%     +28.3%     374190 ±  4% interrupts.CPU10.RES:Rescheduling_interrupts      204140           +49.2%     304587 ±  2% interrupts.CPU11.LOC:Local_timer_interrupts      291988 ±  2%     +28.1%     374024 ±  4% interrupts.CPU11.RES:Rescheduling_interrupts      204367           +49.0%     304559 ±  2% interrupts.CPU12.LOC:Local_timer_interrupts      288632           +28.4%     370669 ±  3% interrupts.CPU12.RES:Rescheduling_interrupts      204555           +48.9%     304669 ±  2% interrupts.CPU13.LOC:Local_timer_interrupts      289354 ±  2%     +28.5%     371782 ±  3% interrupts.CPU13.RES:Rescheduling_interrupts      204536           +49.0%     304726 ±  2% interrupts.CPU14.LOC:Local_timer_interrupts        4937 ± 25%     +41.6%       6991 ±  2% interrupts.CPU14.NMI:Non-maskable_interrupts        4937 ± 25%     +41.6%       6991 ±  2% interrupts.CPU14.PMI:Performance_monitoring_interrupts      287759           +28.2%     368986 ±  3% interrupts.CPU14.RES:Rescheduling_interrupts      204527           +49.0%     304707 ±  2% interrupts.CPU15.LOC:Local_timer_interrupts      286950           +28.6%     369104 ±  3% interrupts.CPU15.RES:Rescheduling_interrupts      204556           +49.0%     304718 ±  2% interrupts.CPU16.LOC:Local_timer_interrupts        5444 ± 32%     -19.9%       4360 ± 32% interrupts.CPU16.NMI:Non-maskable_interrupts        5444 ± 32%     -19.9%       4360 ± 32% interrupts.CPU16.PMI:Performance_monitoring_interrupts      287623           +28.1%     368583 ±  3% interrupts.CPU16.RES:Rescheduling_interrupts      204528           +49.0%     304728 ±  2% interrupts.CPU17.LOC:Local_timer_interrupts      287734           +28.2%     368920 ±  3% interrupts.CPU17.RES:Rescheduling_interrupts      204465           +49.0%     304707 ±  2% interrupts.CPU18.LOC:Local_timer_interrupts        5382 ± 31%     -21.8%       4211 ± 27% interrupts.CPU18.NMI:Non-maskable_interrupts        5382 ± 31%     -21.8%       4211 ± 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts      288778           +28.1%     369871 ±  3% interrupts.CPU18.RES:Rescheduling_interrupts      204302           +49.2%     304727 ±  2% interrupts.CPU19.LOC:Local_timer_interrupts      289794           +27.8%     370247 ±  3% interrupts.CPU19.RES:Rescheduling_interrupts      204552           +49.0%     304821 ±  2% interrupts.CPU2.LOC:Local_timer_interrupts      291056 ±  2%     +28.3%     373290 ±  4% interrupts.CPU2.RES:Rescheduling_interrupts      204468           +48.9%     304484 ±  2% interrupts.CPU20.LOC:Local_timer_interrupts      290778           +27.6%     371106 ±  3% interrupts.CPU20.RES:Rescheduling_interrupts      204546           +48.9%     304619 ±  2% interrupts.CPU21.LOC:Local_timer_interrupts      289974           +28.0%     371074 ±  3% interrupts.CPU21.RES:Rescheduling_interrupts      204694           +48.8%     304581 ±  2% interrupts.CPU22.LOC:Local_timer_interrupts      290312           +28.0%     371734 ±  3% interrupts.CPU22.RES:Rescheduling_interrupts      204374           +49.0%     304493 ±  2% interrupts.CPU23.LOC:Local_timer_interrupts      290266           +28.5%     372865 ±  3% interrupts.CPU23.RES:Rescheduling_interrupts      204518           +49.0%     304653 ±  2% interrupts.CPU24.LOC:Local_timer_interrupts      288815 ±  2%     +28.7%     371621 ±  4% interrupts.CPU24.RES:Rescheduling_interrupts      204414           +49.0%     304640 ±  2% interrupts.CPU25.LOC:Local_timer_interrupts      288779 ±  2%     +28.2%     370202 ±  4% interrupts.CPU25.RES:Rescheduling_interrupts      204536           +49.0%     304730 ±  2% interrupts.CPU26.LOC:Local_timer_interrupts      287615 ±  2%     +28.6%     369843 ±  4% interrupts.CPU26.RES:Rescheduling_interrupts       72.50 ± 37%     +55.5%     112.75 ± 26% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1      204514           +49.0%     304668 ±  2% interrupts.CPU27.LOC:Local_timer_interrupts      287014 ±  2%     +28.5%     368825 ±  4% interrupts.CPU27.RES:Rescheduling_interrupts       53.25 ±  2%     +84.5%      98.25 ± 22% interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2      204331           +49.1%     304759 ±  2% interrupts.CPU28.LOC:Local_timer_interrupts      287278 ±  2%     +28.4%     368876 ±  4% interrupts.CPU28.RES:Rescheduling_interrupts       73.50 ± 26%    +116.3%     159.00 ± 60% interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3      204427           +49.0%     304651 ±  2% interrupts.CPU29.LOC:Local_timer_interrupts      287382 ±  2%     +28.9%     370434 ±  4% interrupts.CPU29.RES:Rescheduling_interrupts      204538           +49.0%     304765 ±  2% interrupts.CPU3.LOC:Local_timer_interrupts        5034 ± 27%     -29.4%       3552 interrupts.CPU3.NMI:Non-maskable_interrupts        5034 ± 27%     -29.4%       3552 interrupts.CPU3.PMI:Performance_monitoring_interrupts      288881 ±  2%     +28.3%     370740 ±  4% interrupts.CPU3.RES:Rescheduling_interrupts      204750           +48.8%     304674 ±  2% interrupts.CPU30.LOC:Local_timer_interrupts      287139 ±  2%     +28.8%     369825 ±  4% interrupts.CPU30.RES:Rescheduling_interrupts      204429           +48.9%     304416 ±  2% interrupts.CPU31.LOC:Local_timer_interrupts      288411 ±  2%     +28.4%     370199 ±  4% interrupts.CPU31.RES:Rescheduling_interrupts      204395           +49.3%     305106 ±  2% interrupts.CPU32.LOC:Local_timer_interrupts      289178 ±  2%     +28.6%     371830 ±  4% interrupts.CPU32.RES:Rescheduling_interrupts      204347           +49.2%     304799 ±  2% interrupts.CPU33.LOC:Local_timer_interrupts      289689 ±  2%     +28.2%     371280 ±  4% interrupts.CPU33.RES:Rescheduling_interrupts      204280           +49.1%     304566 ±  2% interrupts.CPU34.LOC:Local_timer_interrupts        6418 ± 24%     -44.5%       3564 interrupts.CPU34.NMI:Non-maskable_interrupts        6418 ± 24%     -44.5%       3564 interrupts.CPU34.PMI:Performance_monitoring_interrupts      290382 ±  2%     +28.3%     372606 ±  4% interrupts.CPU34.RES:Rescheduling_interrupts      204331           +49.0%     304517 ±  2% interrupts.CPU35.LOC:Local_timer_interrupts      289736 ±  2%     +28.3%     371869 ±  4% interrupts.CPU35.RES:Rescheduling_interrupts      204578           +48.8%     304423 ±  2% interrupts.CPU36.LOC:Local_timer_interrupts        7298           -39.0%       4450 ± 34% interrupts.CPU36.NMI:Non-maskable_interrupts        7298           -39.0%       4450 ± 34% interrupts.CPU36.PMI:Performance_monitoring_interrupts      287187           +28.3%     368516 ±  3% interrupts.CPU36.RES:Rescheduling_interrupts      204538           +49.0%     304729 ±  2% interrupts.CPU37.LOC:Local_timer_interrupts      288676           +28.1%     369792 ±  3% interrupts.CPU37.RES:Rescheduling_interrupts      204504           +49.0%     304687 ±  2% interrupts.CPU38.LOC:Local_timer_interrupts        6301 ± 24%     -29.8%       4424 ± 34% interrupts.CPU38.NMI:Non-maskable_interrupts        6301 ± 24%     -29.8%       4424 ± 34% interrupts.CPU38.PMI:Performance_monitoring_interrupts      287510           +28.1%     368349 ±  3% interrupts.CPU38.RES:Rescheduling_interrupts      204566           +48.9%     304668 ±  2% interrupts.CPU39.LOC:Local_timer_interrupts      286909           +28.1%     367622 ±  3% interrupts.CPU39.RES:Rescheduling_interrupts      204445           +49.0%     304686 ±  2% interrupts.CPU4.LOC:Local_timer_interrupts      288608 ±  2%     +28.7%     371493 ±  3% interrupts.CPU4.RES:Rescheduling_interrupts      204538           +48.9%     304582 ±  2% interrupts.CPU40.LOC:Local_timer_interrupts      286219           +27.9%     365983 ±  3% interrupts.CPU40.RES:Rescheduling_interrupts      204551           +49.0%     304739 ±  2% interrupts.CPU41.LOC:Local_timer_interrupts        5170 ± 29%     -18.5%       4215 ± 27% interrupts.CPU41.NMI:Non-maskable_interrupts        5170 ± 29%     -18.5%       4215 ± 27% interrupts.CPU41.PMI:Performance_monitoring_interrupts      286946           +27.8%     366717 ±  3% interrupts.CPU41.RES:Rescheduling_interrupts      204492           +48.9%     304442 ±  2% interrupts.CPU42.LOC:Local_timer_interrupts      287384           +28.0%     367805 ±  3% interrupts.CPU42.RES:Rescheduling_interrupts      204545           +49.0%     304679 ±  2% interrupts.CPU43.LOC:Local_timer_interrupts      289100           +27.3%     368059 ±  3% interrupts.CPU43.RES:Rescheduling_interrupts      204489           +49.0%     304635 ±  2% interrupts.CPU44.LOC:Local_timer_interrupts        6393 ± 24%     -32.4%       4319 ± 31% interrupts.CPU44.NMI:Non-maskable_interrupts        6393 ± 24%     -32.4%       4319 ± 31% interrupts.CPU44.PMI:Performance_monitoring_interrupts      288835           +27.8%     369182 ±  3% interrupts.CPU44.RES:Rescheduling_interrupts      204473           +49.2%     304983 ±  2% interrupts.CPU45.LOC:Local_timer_interrupts      289778           +27.4%     369059 ±  3% interrupts.CPU45.RES:Rescheduling_interrupts      204493           +49.0%     304598 ±  2% interrupts.CPU46.LOC:Local_timer_interrupts      289485           +27.9%     370172 ±  3% interrupts.CPU46.RES:Rescheduling_interrupts      204511           +49.2%     305090 ±  2% interrupts.CPU47.LOC:Local_timer_interrupts      286611           +28.8%     369021 ±  3% interrupts.CPU47.RES:Rescheduling_interrupts      204441           +49.1%     304794 ±  2% interrupts.CPU5.LOC:Local_timer_interrupts      289245 ±  2%     +28.4%     371335 ±  4% interrupts.CPU5.RES:Rescheduling_interrupts      204246           +49.0%     304414 ±  2% interrupts.CPU6.LOC:Local_timer_interrupts      290006 ±  2%     +28.3%     372180 ±  4% interrupts.CPU6.RES:Rescheduling_interrupts      204445           +49.0%     304719 ±  2% interrupts.CPU7.LOC:Local_timer_interrupts      290524 ±  2%     +28.2%     372414 ±  4% interrupts.CPU7.RES:Rescheduling_interrupts      204178           +49.2%     304663 ±  2% interrupts.CPU8.LOC:Local_timer_interrupts      291625 ±  2%     +27.8%     372747 ±  4% interrupts.CPU8.RES:Rescheduling_interrupts      204502           +48.9%     304407 ±  3% interrupts.CPU9.LOC:Local_timer_interrupts      291615 ±  2%     +27.9%     373024 ±  4% interrupts.CPU9.RES:Rescheduling_interrupts     9814627           +49.0%   14624016 ±  2% interrupts.LOC:Local_timer_interrupts    13869791           +28.2%   17785831 ±  3% interrupts.RES:Rescheduling_interrupts
      25728           +47.6%      37969        softirqs.CPU0.RCU
      34944           +41.7%      49503 ±  3%  softirqs.CPU0.SCHED
      42375           +38.5%      58673 ±  2%  softirqs.CPU0.TIMER
      25348           +53.8%      38976 ±  5%  softirqs.CPU1.RCU
      30986           +46.2%      45315 ±  3%  softirqs.CPU1.SCHED
      41265           +44.3%      59538 ±  3%  softirqs.CPU1.TIMER
      25149           +48.3%      37288        softirqs.CPU10.RCU
      29752           +47.1%      43764 ±  2%  softirqs.CPU10.SCHED
      40609           +41.5%      57449        softirqs.CPU10.TIMER
      25218           +48.3%      37399        softirqs.CPU11.RCU
      30294           +44.6%      43800        softirqs.CPU11.SCHED
      41371           +39.2%      57578        softirqs.CPU11.TIMER
      24427           +48.9%      36361        softirqs.CPU12.RCU
      30438           +44.0%      43835        softirqs.CPU12.SCHED
      41045 ±  3%     +43.0%      58687        softirqs.CPU12.TIMER
      24247           +47.0%      35645        softirqs.CPU13.RCU
      30155           +44.5%      43577        softirqs.CPU13.SCHED
      41148 ±  2%     +40.3%      57720        softirqs.CPU13.TIMER
      24506           +46.3%      35856        softirqs.CPU14.RCU
      29991           +44.1%      43229        softirqs.CPU14.SCHED
      41190 ±  2%     +39.7%      57552        softirqs.CPU14.TIMER
      24412           +46.7%      35800        softirqs.CPU15.RCU
      30135           +44.1%      43430        softirqs.CPU15.SCHED
      45081 ± 15%     +28.0%      57684        softirqs.CPU15.TIMER
      25327           +47.0%      37225        softirqs.CPU16.RCU
      30230           +43.8%      43461        softirqs.CPU16.SCHED
      41157           +39.6%      57448        softirqs.CPU16.TIMER
      25136           +47.8%      37142        softirqs.CPU17.RCU
      29894           +44.5%      43212        softirqs.CPU17.SCHED
      41961 ±  5%     +36.5%      57281        softirqs.CPU17.TIMER
      25431           +46.1%      37160        softirqs.CPU18.RCU
      30062           +44.0%      43281        softirqs.CPU18.SCHED
      40956 ±  2%     +42.1%      58217        softirqs.CPU18.TIMER
      25496           +44.8%      36923 ±  2%  softirqs.CPU19.RCU
      29989           +46.5%      43946        softirqs.CPU19.SCHED
      41740 ±  2%     +67.8%      70057 ± 17%  softirqs.CPU19.TIMER
      25414           +48.3%      37678        softirqs.CPU2.RCU
      30905           +43.4%      44309 ±  2%  softirqs.CPU2.SCHED
      41083           +40.6%      57768 ±  2%  softirqs.CPU2.TIMER
      25124           +46.8%      36892        softirqs.CPU20.RCU
      30001           +44.7%      43416        softirqs.CPU20.SCHED
      41060 ±  7%     +39.2%      57151        softirqs.CPU20.TIMER
      24992           +47.6%      36896        softirqs.CPU21.RCU
      30109           +44.4%      43467        softirqs.CPU21.SCHED
      40055 ±  3%     +42.5%      57081 ±  2%  softirqs.CPU21.TIMER
      25072           +46.2%      36656        softirqs.CPU22.RCU
      30296           +43.3%      43425        softirqs.CPU22.SCHED
      24882           +48.8%      37031        softirqs.CPU23.RCU
      29900           +43.7%      42971        softirqs.CPU23.SCHED
      40388 ±  3%     +44.8%      58494 ±  3%  softirqs.CPU23.TIMER
      25007           +48.1%      37036        softirqs.CPU24.RCU
      29755           +45.9%      43399        softirqs.CPU24.SCHED
      40786           +40.6%      57353        softirqs.CPU24.TIMER
      25031           +47.3%      36875        softirqs.CPU25.RCU
      29841 ±  2%     +45.3%      43355        softirqs.CPU25.SCHED
      40971           +39.8%      57266        softirqs.CPU25.TIMER
      25145           +47.6%      37114        softirqs.CPU26.RCU
      29462 ±  2%     +48.5%      43739 ±  2%  softirqs.CPU26.SCHED
      40553           +41.8%      57521        softirqs.CPU26.TIMER
      25184           +47.6%      37169        softirqs.CPU27.RCU
      29867           +45.5%      43450 ±  3%  softirqs.CPU27.SCHED
      41118           +39.3%      57261        softirqs.CPU27.TIMER
      25452           +47.1%      37440        softirqs.CPU28.RCU
      29714           +46.2%      43440 ±  2%  softirqs.CPU28.SCHED
      41209 ±  2%     +37.8%      56783 ±  2%  softirqs.CPU28.TIMER
      25334           +48.8%      37696 ±  2%  softirqs.CPU29.RCU
      29894           +46.3%      43749        softirqs.CPU29.SCHED
      41413 ±  2%     +43.3%      59346 ±  5%  softirqs.CPU29.TIMER
      25514           +47.0%      37495        softirqs.CPU3.RCU
      30016           +46.0%      43831 ±  2%  softirqs.CPU3.SCHED
      41371 ±  2%     +39.3%      57611        softirqs.CPU3.TIMER
      25206           +47.4%      37149        softirqs.CPU30.RCU
      30167           +44.0%      43451 ±  2%  softirqs.CPU30.SCHED
      41166 ±  2%     +58.1%      65092 ± 19%  softirqs.CPU30.TIMER
      25211           +46.8%      37016        softirqs.CPU31.RCU
      30004 ±  2%     +45.2%      43569        softirqs.CPU31.SCHED
      41541 ±  2%     +36.9%      56874        softirqs.CPU31.TIMER
      24432           +46.9%      35885        softirqs.CPU32.RCU
      30206           +43.4%      43322 ±  2%  softirqs.CPU32.SCHED
      41069           +40.7%      57781 ±  2%  softirqs.CPU32.TIMER
      24303           +48.0%      35973        softirqs.CPU33.RCU
      30238           +43.8%      43489        softirqs.CPU33.SCHED
      41161           +38.2%      56872        softirqs.CPU33.TIMER
      24308           +47.6%      35876        softirqs.CPU34.RCU
      29803           +46.5%      43647 ±  2%  softirqs.CPU34.SCHED
      40809           +39.2%      56817        softirqs.CPU34.TIMER
      24291           +47.7%      35877        softirqs.CPU35.RCU
      30092           +44.2%      43386        softirqs.CPU35.SCHED
      40607           +41.5%      57480 ±  2%  softirqs.CPU35.TIMER
      25605           +49.5%      38286        softirqs.CPU36.RCU
      29900           +44.6%      43238        softirqs.CPU36.SCHED
      45207 ± 15%     +27.4%      57602        softirqs.CPU36.TIMER
      25555 ±  2%     +49.0%      38070        softirqs.CPU37.RCU
      29926           +45.2%      43460        softirqs.CPU37.SCHED
      42619 ±  8%     +36.7%      58242        softirqs.CPU37.TIMER
      27683 ±  9%     +50.2%      41577 ±  4%  softirqs.CPU38.RCU
      29700           +45.6%      43230        softirqs.CPU38.SCHED
      41779 ±  4%     +43.5%      59948        softirqs.CPU38.TIMER
      26699 ±  7%     +42.1%      37951        softirqs.CPU39.RCU
      30061           +43.4%      43102        softirqs.CPU39.SCHED
      42312 ±  5%     +36.1%      57581 ±  2%  softirqs.CPU39.TIMER
      25219           +47.7%      37247        softirqs.CPU4.RCU
      30363 ±  3%     +45.9%      44295        softirqs.CPU4.SCHED
      41108 ±  2%     +39.6%      57371 ±  2%  softirqs.CPU4.TIMER
      25413           +50.6%      38268        softirqs.CPU40.RCU
      29910           +46.0%      43665        softirqs.CPU40.SCHED
      39987 ±  2%     +45.9%      58322        softirqs.CPU40.TIMER
      25569           +49.5%      38238        softirqs.CPU41.RCU
      29867           +44.8%      43247        softirqs.CPU41.SCHED
      25492           +49.8%      38185        softirqs.CPU42.RCU
      29871           +45.4%      43440        softirqs.CPU42.SCHED
      40076           +44.8%      58020        softirqs.CPU42.TIMER
      25281           +48.9%      37640        softirqs.CPU43.RCU
      30025           +43.9%      43220        softirqs.CPU43.SCHED
      40222 ±  3%     +44.7%      58195        softirqs.CPU43.TIMER
      25183           +49.3%      37611        softirqs.CPU44.RCU
      30123           +43.6%      43249        softirqs.CPU44.SCHED
      41090 ±  6%     +42.5%      58537        softirqs.CPU44.TIMER
      25314           +49.6%      37863        softirqs.CPU45.RCU
      29998           +43.2%      42957        softirqs.CPU45.SCHED
      39705 ±  3%     +43.6%      57033        softirqs.CPU45.TIMER
      25500           +46.2%      37274        softirqs.CPU46.RCU
      29921           +44.7%      43286        softirqs.CPU46.SCHED
      41088 ±  5%     +39.9%      57463 ±  3%  softirqs.CPU46.TIMER
      25564           +48.4%      37931        softirqs.CPU47.RCU
      30178           +42.4%      42980        softirqs.CPU47.SCHED
      40738 ±  4%     +42.2%      57939 ±  3%  softirqs.CPU47.TIMER
      26467 ±  8%     +40.8%      37278        softirqs.CPU5.RCU
      30334 ±  2%     +44.1%      43714        softirqs.CPU5.SCHED
      42268 ±  4%     +35.8%      57394        softirqs.CPU5.TIMER
      25343           +47.0%      37245        softirqs.CPU6.RCU
      30100           +45.0%      43643 ±  2%  softirqs.CPU6.SCHED
      41270           +43.6%      59249 ±  5%  softirqs.CPU6.TIMER
      25331           +47.7%      37418        softirqs.CPU7.RCU
      30210           +44.7%      43727 ±  2%  softirqs.CPU7.SCHED
      41028           +41.9%      58206 ±  2%  softirqs.CPU7.TIMER
      25483           +47.2%      37514        softirqs.CPU8.RCU
      30324 ±  2%     +43.4%      43483 ±  2%  softirqs.CPU8.SCHED
      41027           +40.4%      57591 ±  2%  softirqs.CPU8.TIMER
      25546           +47.5%      37687        softirqs.CPU9.RCU
      30218           +44.8%      43771 ±  2%  softirqs.CPU9.SCHED
      41193           +40.1%      57716 ±  2%  softirqs.CPU9.TIMER
    1212606           +47.7%    1790808        softirqs.RCU
    1448198           +44.7%    2095500        softirqs.SCHED
    1990064           +40.7%    2800202        softirqs.TIMER
       8.56 ±  3%      -4.3        4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write        8.50 ±  3%      -4.3        4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write        8.47 ±  3%      -4.3        4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync       78.55            -4.1       74.44 perf-profile.calltrace.cycles-pp.write        8.74 ±  3%      -4.0        4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write       77.69            -3.9       73.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write       77.64            -3.9       73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write       77.13            -3.8       73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write       77.08            -3.8       73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write       76.61            -3.7       72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe       76.43            -3.6       72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64       69.22            -2.6       66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write        4.70 ±  2%      -2.2        2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range        4.63 ±  2%      -2.2        2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range        4.60 ±  2%      -2.2        2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages        4.32 ±  2%      -2.1        2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages        3.72 ±  4%      -2.1        1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range        3.71 ±  4%      -2.1        1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range        2.63 ±  4%      -2.0        0.63 ±  3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend        7.72            -2.0        5.74 ±  2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write        3.08 ±  5%      -1.9        1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages        2.49 ±  5%      -1.9        0.55 ±  3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio        2.48 ±  5%      -1.9        0.55 ±  3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request        2.41 ±  6%      -1.8        0.61 ±  2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages        4.90            -1.5        3.40 ±  3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync        4.78            -1.5        3.30 ±  3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn        2.35            -0.6        1.77 ±  2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write        2.36            -0.6        1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write        3.40            -0.5        2.87 ±  6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write        1.83            -0.5        1.33 ±  2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write        3.26            -0.5        2.75 ±  6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks        2.85            -0.4        2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork        2.79            -0.4        2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread        2.83            -0.4        2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread        1.48            -0.4        1.05 ±  2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend        0.61 ±  2%      -0.4        0.26 ±100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock        1.89            -0.3        1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync        1.71            -0.3        1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn        1.65            -0.3        1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn        0.69            -0.3        0.38 ± 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write        1.59            -0.3        1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work        0.86            -0.3        0.58 ±  2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time        1.52            -0.3        1.26 ±  3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time        1.75            -0.2        1.50 ±  2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work        1.74            -0.2        1.49 ±  2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked        1.74            -0.2        1.50 ±  2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete        2.15            -0.2        1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate        0.79 ±  3%      -0.2        0.60 ±  4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time        1.81            -0.2        1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair        1.87            -0.2        1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task        0.70 ±  4%      -0.2        0.54 ±  4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit        0.81            -0.1        0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity        0.65 ±  2%      -0.1        0.55 ±  2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn        0.62 ±  2%      -0.1        0.52 ±  3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn        1.18            -0.0        1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64        1.01            -0.0        0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry        1.00            -0.0        0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle        0.96            -0.0        0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending        1.01            -0.0        0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary        0.59 ±  3%      +0.1        0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64        0.58 ±  3%      +0.1        0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary        0.53            +0.1        0.63 ±  3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work        0.91 ±  3%      +0.1        1.02 ±  2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread        0.54            +0.1        0.65 ±  3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread        0.57            +0.1        0.69 ±  3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread        1.83            +0.2        1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork        0.00            +0.6        0.57 ±  2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common        0.00            +0.6        0.57 ±  2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function        0.00            +0.6        0.57 ±  2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit        0.00            +0.7        0.70 ±  2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend        0.00            +0.8        0.75 ±  4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io        0.00            +0.8        0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend        0.00            +0.8        0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends        0.00            +0.8        0.80 ± 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks        0.00            +0.8        0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend        0.00            +0.8        0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io        0.00            +0.9        0.89 ±  2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work        0.00            +0.9        0.92 ± 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io        0.00            +1.0        0.97 ± 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work       35.68            +1.0       36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write        0.00            +1.0        1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work        0.00            +1.0        1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread        9.35            +1.2       10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry        9.61            +1.2       10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary        9.63            +1.2       10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64       12.24            +1.3       13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64       12.25            +1.3       13.60 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64       12.25            +1.3       13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64       12.49            +1.4       13.89 perf-profile.calltrace.cycles-pp.secondary_startup_64       14.38            +1.4       15.81 ±  2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write       13.41            +1.4       14.86 ±  2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync       13.37            +1.5       14.82 ±  2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn       13.34            +1.5       14.80 ±  2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn       58.91            +1.6       60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write        0.00            +2.2        2.24 ± 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread        5.63            +2.5        8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn        5.65            +2.5        8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync        5.69            +2.5        8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write        7.04            +2.8        9.80 ±  2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork        7.54            +2.9       10.45 ±  2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork        7.55            +2.9       10.46 ±  2% perf-profile.calltrace.cycles-pp.ret_from_fork        7.55            +2.9       10.46 ±  2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork        0.00            +3.3        3.30 ±  9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread        0.00            +3.3        3.32 ±  8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork        8.56 ±  3%      -4.3        4.25 perf-profile.children.cycles-pp.__filemap_fdatawrite_range        8.47 ±  3%      -4.3        4.18 perf-profile.children.cycles-pp.xfs_vm_writepages        8.50 ±  3%      -4.3        4.20 perf-profile.children.cycles-pp.do_writepages       78.64            -4.1       74.51 perf-profile.children.cycles-pp.write       78.55            -4.1       74.44 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe       78.50            -4.1       74.41 perf-profile.children.cycles-pp.do_syscall_64        8.75 ±  3%      -4.1        4.70 perf-profile.children.cycles-pp.file_write_and_wait_range       77.14            -3.8       73.37 perf-profile.children.cycles-pp.ksys_write       77.09            -3.8       73.33 perf-profile.children.cycles-pp.vfs_write       76.62            -3.7       72.97 perf-profile.children.cycles-pp.new_sync_write       76.44            -3.6       72.81 perf-profile.children.cycles-pp.xfs_file_buffered_aio_write       69.23            -2.6       66.62 perf-profile.children.cycles-pp.xfs_file_fsync        4.70 ±  2%      -2.2        2.50 perf-profile.children.cycles-pp.iomap_submit_ioend        4.89 ±  2%      -2.1        2.75 perf-profile.children.cycles-pp.submit_bio        4.85 ±  2%      -2.1        2.71 perf-profile.children.cycles-pp.generic_make_request        3.73 ±  4%      -2.1        1.63 perf-profile.children.cycles-pp.iomap_writepages        3.72 ±  4%      -2.1        1.63 perf-profile.children.cycles-pp.write_cache_pages        4.43 ±  2%      -2.1        2.38 perf-profile.children.cycles-pp.brd_make_request        7.73            -2.0        5.75 ±  2% perf-profile.children.cycles-pp.__flush_work        3.08 ±  4%      -1.9        1.14 perf-profile.children.cycles-pp.iomap_writepage_map        6.00            -1.8        4.20 ±  2% perf-profile.children.cycles-pp._raw_spin_lock_irq        2.50 ±  4%      -1.8        0.70 ±  3% perf-profile.children.cycles-pp.end_page_writeback        2.50 ±  5%      -1.8        0.70 ±  4% perf-profile.children.cycles-pp.test_clear_page_writeback        2.42 ±  6%      -1.8        0.61 ±  2% perf-profile.children.cycles-pp.__test_set_page_writeback        2.08 ±  7%      -1.6        0.44 ±  4% perf-profile.children.cycles-pp.sb_mark_inode_writeback        2.08 ±  6%      -1.6        0.44 ±  5% perf-profile.children.cycles-pp.sb_clear_inode_writeback        2.70 ±  4%      -1.0        1.67 ±  2% perf-profile.children.cycles-pp.iomap_finish_ioend        2.36            -0.6        1.79 perf-profile.children.cycles-pp.iomap_file_buffered_write        2.35            -0.6        1.78 perf-profile.children.cycles-pp.iomap_apply        1.84            -0.5        1.33 ±  2% perf-profile.children.cycles-pp.iomap_write_actor        2.41 ±  4%      -0.5        1.94 ±  2% perf-profile.children.cycles-pp.reschedule_interrupt        3.44 ±  2%      -0.5        2.98 perf-profile.children.cycles-pp.sched_ttwu_pending        2.87            -0.5        2.42 perf-profile.children.cycles-pp.complete        2.83            -0.4        2.38 perf-profile.children.cycles-pp.swake_up_locked        2.28 ±  4%      -0.4        1.84 ±  2% perf-profile.children.cycles-pp.scheduler_ipi        1.56            -0.4        1.17 ±  2% perf-profile.children.cycles-pp.brd_do_bvec        1.89            -0.4        1.54 perf-profile.children.cycles-pp.wait_for_completion        1.71            -0.3        1.38 perf-profile.children.cycles-pp.schedule_timeout        0.88            -0.2        0.68 ±  2% perf-profile.children.cycles-pp.rwsem_down_read_slowpath        0.69            -0.2        0.51 ±  2% perf-profile.children.cycles-pp.iomap_write_begin        0.67 ±  2%      -0.2        0.49 perf-profile.children.cycles-pp.lseek64        1.33 ±  5%      -0.2        1.17 ±  3% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore        0.51            -0.2        0.36 perf-profile.children.cycles-pp.grab_cache_page_write_begin        0.47            -0.1        0.33 perf-profile.children.cycles-pp.pagecache_get_page        0.51 ±  3%      -0.1        0.38 ±  2% perf-profile.children.cycles-pp.iomap_write_end        0.57 ±  2%      -0.1        0.46 perf-profile.children.cycles-pp.blkdev_issue_flush        0.38            -0.1        0.26 ±  5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic        0.34            -0.1        0.23 ±  6% perf-profile.children.cycles-pp.copyin        0.33            -0.1        0.22 ±  6% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string        0.65 ±  2%      -0.1        0.56 ±  2% perf-profile.children.cycles-pp.rwsem_optimistic_spin        0.37            -0.1        0.28 ±  2% perf-profile.children.cycles-pp.bio_alloc_bioset        0.31 ±  4%      -0.1        0.22 perf-profile.children.cycles-pp.syscall_return_via_sysret        0.56 ±  3%      -0.1        0.48 ±  3% perf-profile.children.cycles-pp.osq_lock        0.26 ±  3%      -0.1        0.18 ±  4% perf-profile.children.cycles-pp.entry_SYSCALL_64        0.28 ±  2%      -0.1        0.21 ±  5% perf-profile.children.cycles-pp.mempool_alloc        0.26 ±  6%      -0.1        0.18 ±  8% perf-profile.children.cycles-pp.xlog_state_get_iclog_space        0.29            -0.1        0.22 ±  3% perf-profile.children.cycles-pp.find_get_entry        0.34 ±  2%      -0.1        0.27 ±  4% perf-profile.children.cycles-pp.generic_make_request_checks        0.30 ±  5%      -0.1        0.23 ±  3% perf-profile.children.cycles-pp.iomap_set_page_dirty        0.32 ±  2%      -0.1        0.26 ±  5% perf-profile.children.cycles-pp.brd_lookup_page        0.18 ±  2%      -0.1        0.12 ±  6% perf-profile.children.cycles-pp.iomap_set_range_uptodate        0.15 ±  2%      -0.1        0.09 ±  4% perf-profile.children.cycles-pp.__mod_lruvec_state        0.15 ±  5%      -0.1        0.10 ±  9% perf-profile.children.cycles-pp.wake_up_q        0.21 ±  5%      -0.1        0.15 ±  5% perf-profile.children.cycles-pp.xfs_ilock        0.13 ±  3%      -0.0        0.08 ± 10% perf-profile.children.cycles-pp.rwsem_wake        0.20 ±  2%      -0.0        0.15 ±  5% perf-profile.children.cycles-pp.ksys_lseek        0.20 ±  3%      -0.0        0.15 ±  3% perf-profile.children.cycles-pp.xas_load        0.20 ±  5%      -0.0        0.16 ±  2% perf-profile.children.cycles-pp.__set_page_dirty        0.19 ±  2%      -0.0        0.15 ±  2% perf-profile.children.cycles-pp.xfs_iunlock        0.40            -0.0        0.36 ±  4% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin        0.17 ±  2%      -0.0        0.12 ±  4% perf-profile.children.cycles-pp.clear_page_dirty_for_io        0.20 ±  3%      -0.0        0.16 ±  4% perf-profile.children.cycles-pp.submit_bio_wait        0.18 ±  4%      -0.0        0.15 ±  7% perf-profile.children.cycles-pp.security_file_permission        0.10            -0.0        0.06 ±  6% perf-profile.children.cycles-pp.evict        0.10            -0.0        0.06 ±  6% perf-profile.children.cycles-pp.truncate_inode_pages_range        0.06 ±  6%      -0.0        0.03 ±100% perf-profile.children.cycles-pp.vfs_unlink        0.08 ±  6%      -0.0        0.04 ± 58% perf-profile.children.cycles-pp.add_to_page_cache_lru        0.26 ±  2%      -0.0        0.23 ±  3% perf-profile.children.cycles-pp.reweight_entity        0.12 ±  6%      -0.0        0.09 ±  4% perf-profile.children.cycles-pp.bio_endio        0.10 ±  7%      -0.0        0.07 ±  7% perf-profile.children.cycles-pp.do_sys_open        0.10 ±  7%      -0.0        0.07 ±  7% perf-profile.children.cycles-pp.do_sys_openat2        0.17 ±  2%      -0.0        0.14 perf-profile.children.cycles-pp.__pagevec_release        0.13 ±  5%      -0.0        0.10 ±  4% perf-profile.children.cycles-pp.down_write        0.10 ±  5%      -0.0        0.06 ±  6% perf-profile.children.cycles-pp.do_filp_open        0.15 ±  4%      -0.0        0.12 ±  5% perf-profile.children.cycles-pp.tag_pages_for_writeback        0.11 ±  9%      -0.0        0.08 ±  5% perf-profile.children.cycles-pp.bio_associate_blkg        0.10 ±  5%      -0.0        0.07 ±  7% perf-profile.children.cycles-pp.open64        0.09 ±  4%      -0.0        0.06 ±  6% perf-profile.children.cycles-pp.path_openat        0.16 ±  8%      -0.0        0.13 ±  3% perf-profile.children.cycles-pp.pagevec_lookup_range_tag        0.12 ±  8%      -0.0        0.10 ±  5% perf-profile.children.cycles-pp.account_page_dirtied        0.10 ±  4%      -0.0        0.08 ±  6% perf-profile.children.cycles-pp.__fdget_pos        0.14 ±  3%      -0.0        0.11 ±  7% perf-profile.children.cycles-pp.memset_erms        0.11 ±  3%      -0.0        0.09 ±  4% perf-profile.children.cycles-pp.switch_fpu_return        0.23            -0.0        0.21 ±  3% perf-profile.children.cycles-pp.__radix_tree_lookup        0.18 ±  2%      -0.0        0.16 ±  9% perf-profile.children.cycles-pp.insert_work        0.12 ±  4%      -0.0        0.10 ±  7% perf-profile.children.cycles-pp.common_file_perm        0.09 ±  4%      -0.0        0.07 ±  6% perf-profile.children.cycles-pp.__fget_light        0.09 ±  4%      -0.0        0.07 ±  6% perf-profile.children.cycles-pp.bvec_alloc        0.15 ±  7%      -0.0        0.12 ±  3% perf-profile.children.cycles-pp.find_get_pages_range_tag        0.08 ±  8%      -0.0        0.06 ±  7% perf-profile.children.cycles-pp.iov_iter_fault_in_readable        0.09 ±  5%      -0.0        0.06 ± 13% perf-profile.children.cycles-pp.blk_throtl_bio        0.24 ±  4%      -0.0        0.22 ±  3% perf-profile.children.cycles-pp._cond_resched        0.13 ±  3%      -0.0        0.11 ±  4% perf-profile.children.cycles-pp.release_pages        0.10 ±  4%      -0.0        0.08 ±  6% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap        0.08 ± 10%      -0.0        0.06 perf-profile.children.cycles-pp.unlink        0.08 ± 10%      -0.0        0.06 ± 11% perf-profile.children.cycles-pp.__sb_start_write        0.08 ±  5%      -0.0        0.06 ±  7% perf-profile.children.cycles-pp.generic_write_checks        0.08 ±  8%      -0.0        0.06 perf-profile.children.cycles-pp.do_unlinkat        0.07 ± 10%      -0.0        0.05 perf-profile.children.cycles-pp.xfs_break_layouts        0.18 ±  2%      -0.0        0.16 ±  2% perf-profile.children.cycles-pp.__list_add_valid        0.12 ±  3%      -0.0        0.10 ±  8% perf-profile.children.cycles-pp.__might_sleep        0.14 ±  3%      -0.0        0.12 ±  7% perf-profile.children.cycles-pp.xas_find_marked        0.07 ± 10%      -0.0        0.05 ±  8% perf-profile.children.cycles-pp.bio_associate_blkg_from_css        0.08 ±  5%      -0.0        0.06 ± 11% perf-profile.children.cycles-pp.bio_free        0.08 ±  6%      -0.0        0.06 perf-profile.children.cycles-pp.xfs_file_llseek        0.08 ±  5%      -0.0        0.06 ±  6% perf-profile.children.cycles-pp.xfs_file_write_iter        0.16 ±  2%      +0.0        0.17 ±  2% perf-profile.children.cycles-pp.__switch_to_asm        0.05            +0.0        0.07 ±  7% perf-profile.children.cycles-pp.cpumask_next_wrap        0.10 ±  4%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp._find_next_bit        0.07 ±  7%      +0.0        0.09 ±  5% perf-profile.children.cycles-pp.xlog_state_done_syncing        0.16 ±  2%      +0.0        0.19 ±  2% perf-profile.children.cycles-pp.check_preempt_curr        0.19 ±  4%      +0.0        0.22 perf-profile.children.cycles-pp.ttwu_do_wakeup        0.56 ±  3%      +0.0        0.60 perf-profile.children.cycles-pp.dequeue_entity        0.20 ±  6%      +0.0        0.23 ±  4% perf-profile.children.cycles-pp.xfs_trans_committed_bulk        0.07 ±  7%      +0.0        0.10 ±  7% perf-profile.children.cycles-pp.check_preempt_wakeup        0.82 ±  2%      +0.0        0.86 perf-profile.children.cycles-pp.update_load_avg        0.31 ±  3%      +0.0        0.35 perf-profile.children.cycles-pp.__unwind_start        0.08 ±  8%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp.destroy_inode        0.08 ± 10%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp.xfs_fs_destroy_inode        0.08 ± 10%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp.xfs_inactive        0.08 ± 15%      +0.0        0.12 ±  5% perf-profile.children.cycles-pp.xfs_bmap_validate_extent        3.96            +0.0        4.01 perf-profile.children.cycles-pp.stack_trace_save_tsk        0.28            +0.0        0.33 perf-profile.children.cycles-pp.set_next_entity        0.00            +0.1        0.05 perf-profile.children.cycles-pp.xfs_trans_roll        0.00            +0.1        0.05 ±  8% perf-profile.children.cycles-pp.xfs_btree_delrec        0.00            +0.1        0.05 ±  8% perf-profile.children.cycles-pp.xfs_btree_insert        0.16 ±  4%      +0.1        0.21 ±  5% perf-profile.children.cycles-pp.memcpy_erms        0.04 ± 57%      +0.1        0.10 ±  4% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags        0.00            +0.1        0.06 ± 11% perf-profile.children.cycles-pp.xfs_buf_item_format_segment        0.00            +0.1        0.06 ± 11% perf-profile.children.cycles-pp.xfs_btree_delete        0.00            +0.1        0.06 perf-profile.children.cycles-pp.xfs_btree_update        0.00            +0.1        0.07 ±  7% perf-profile.children.cycles-pp.xfs_cil_prepare_item        0.00            +0.1        0.07 ±  6% perf-profile.children.cycles-pp.xfs_bmbt_update        0.00            +0.1        0.07 ± 10% perf-profile.children.cycles-pp.xfs_buf_item_format        0.56            +0.1        0.64 ±  2% perf-profile.children.cycles-pp.pick_next_task_fair        0.03 ±100%      +0.1        0.10 ±  4% perf-profile.children.cycles-pp.xfs_inactive_truncate        0.08 ±  5%      +0.1        0.16 ± 11% perf-profile.children.cycles-pp.xfs_iext_lookup_extent        0.00            +0.1        0.08 ±  5% perf-profile.children.cycles-pp.xfs_defer_finish        0.00            +0.1        0.08 ±  5% perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents        0.00            +0.1        0.08 ±  5% perf-profile.children.cycles-pp.xfs_verify_agbno        0.00            +0.1        0.09 ±  9% perf-profile.children.cycles-pp.xfs_defer_finish_noroll        0.60 ±  3%      +0.1        0.70 perf-profile.children.cycles-pp.schedule_idle        0.39 ±  2%      +0.1        0.49 perf-profile.children.cycles-pp.xlog_state_release_iclog        0.00            +0.1        0.10 ±  9% perf-profile.children.cycles-pp.xfs_buf_find        0.93 ±  3%      +0.1        1.03 ±  2% perf-profile.children.cycles-pp.xlog_write        2.58            +0.1        2.69 perf-profile.children.cycles-pp.schedule        1.52            +0.1        1.63 perf-profile.children.cycles-pp.__wake_up_common_lock        0.00            +0.1        0.11 ±  7% perf-profile.children.cycles-pp.xfs_buf_get_map        4.62            +0.1        4.73 perf-profile.children.cycles-pp.__account_scheduler_latency        0.00            +0.1        0.12 ± 10% perf-profile.children.cycles-pp.xfs_buf_read_map        1.83            +0.2        1.99 perf-profile.children.cycles-pp.xlog_cil_push_work        5.94            +0.2        6.12 perf-profile.children.cycles-pp.ttwu_do_activate        0.00            +0.2        0.17 ±  6% perf-profile.children.cycles-pp.xfs_btree_read_buf_block        5.93            +0.2        6.11 perf-profile.children.cycles-pp.activate_task        5.88            +0.2        6.06 perf-profile.children.cycles-pp.enqueue_task_fair        0.00            +0.2        0.18 ±  5% perf-profile.children.cycles-pp.xfs_trans_read_buf_map        5.41            +0.2        5.59 perf-profile.children.cycles-pp.enqueue_entity        3.14            +0.2        3.33 perf-profile.children.cycles-pp.__sched_text_start        0.00            +0.2        0.21 ±  6% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block        0.00            +0.2        0.23 ± 57% perf-profile.children.cycles-pp.xlog_grant_head_wait        0.00            +0.2        0.24 ±  6% perf-profile.children.cycles-pp.io_schedule        0.11 ±  3%      +0.3        0.38 ±  4% perf-profile.children.cycles-pp.__filemap_fdatawait_range        0.00            +0.3        0.27 ±  8% perf-profile.children.cycles-pp.wait_on_page_bit        0.00            +0.3        0.34 ±  5% perf-profile.children.cycles-pp.xfs_btree_lookup        0.08 ± 14%      +0.4        0.43 perf-profile.children.cycles-pp.__queue_work        0.08 ±  8%      +0.4        0.44 perf-profile.children.cycles-pp.queue_work_on        0.00            +0.4        0.41 perf-profile.children.cycles-pp.xfs_end_bio        3.51            +0.4        3.94 ±  9% perf-profile.children.cycles-pp.__xfs_trans_commit        0.20 ±  9%      +0.5        0.72 ± 54% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant        0.06 ± 11%      +0.5        0.58 ± 67% perf-profile.children.cycles-pp.xfs_log_space_wake        0.45 ±  5%      +0.6        1.04 ± 36% perf-profile.children.cycles-pp.xfs_log_reserve        0.65 ±  3%      +0.6        1.25 ± 29% perf-profile.children.cycles-pp.xfs_trans_alloc        0.48 ±  5%      +0.6        1.09 ± 34% perf-profile.children.cycles-pp.xfs_trans_reserve        0.04 ± 57%      +0.6        0.65 ± 60% perf-profile.children.cycles-pp.xlog_grant_head_check        4.83            +0.7        5.52 perf-profile.children.cycles-pp.try_to_wake_up        0.00            +0.7        0.70 ±  2% perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real        0.00            +0.8        0.75 ±  4% perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten        0.04 ± 57%      +0.8        0.80 perf-profile.children.cycles-pp.autoremove_wake_function        0.05 ±  8%      +0.8        0.85 perf-profile.children.cycles-pp.wake_up_page_bit        1.50            +0.9        2.36 perf-profile.children.cycles-pp.__wake_up_common        0.00            +0.9        0.89 ±  2% perf-profile.children.cycles-pp.xfs_bmapi_write       24.29            +0.9       25.18 perf-profile.children.cycles-pp._raw_spin_lock_irqsave        0.07 ±  5%      +1.0        1.05 perf-profile.children.cycles-pp.iomap_finish_ioends       35.68            +1.0       36.69 perf-profile.children.cycles-pp.xlog_cil_force_lsn        9.53            +1.2       10.76 perf-profile.children.cycles-pp.intel_idle        9.81            +1.3       11.10 perf-profile.children.cycles-pp.cpuidle_enter_state        9.82            +1.3       11.11 perf-profile.children.cycles-pp.cpuidle_enter       12.25            +1.3       13.60 perf-profile.children.cycles-pp.start_secondary       12.49            +1.4       13.89 perf-profile.children.cycles-pp.secondary_startup_64       12.49            +1.4       13.89 perf-profile.children.cycles-pp.cpu_startup_entry       12.48            +1.4       13.88 perf-profile.children.cycles-pp.do_idle       14.78            +1.4       16.20 ±  2% perf-profile.children.cycles-pp.xlog_wait_on_iclog       58.91            +1.7       60.57 perf-profile.children.cycles-pp.xfs_log_force_lsn        0.00            +2.2        2.24 ± 13% perf-profile.children.cycles-pp.xfs_iomap_write_unwritten        7.05            +2.8        9.81 ±  2% perf-profile.children.cycles-pp.process_one_work        7.54            +2.9       10.45 ±  2% perf-profile.children.cycles-pp.worker_thread        7.55            +2.9       10.46 ±  2% perf-profile.children.cycles-pp.ret_from_fork        7.55            +2.9       10.46 ±  2% perf-profile.children.cycles-pp.kthread        0.10            +3.2        3.30 ±  9% perf-profile.children.cycles-pp.xfs_end_ioend        0.10 ±  4%      +3.2        3.32 ±  8% perf-profile.children.cycles-pp.xfs_end_io       19.57            +4.0       23.62 perf-profile.children.cycles-pp.remove_wait_queue        1.20            -0.3        0.90 ±  2% perf-profile.self.cycles-pp.brd_do_bvec        0.55 ±  3%      -0.2        0.39 ±  4% perf-profile.self.cycles-pp.do_syscall_64        0.31            -0.1        0.22 ±  5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string        0.31 ±  3%      -0.1        0.22 perf-profile.self.cycles-pp.syscall_return_via_sysret        0.54 ±  3%      -0.1        0.47 ±  3% perf-profile.self.cycles-pp.osq_lock        0.23 ±  3%      -0.1        0.16 ±  5% perf-profile.self.cycles-pp.entry_SYSCALL_64        0.10 ±  4%      -0.1        0.04 ± 57% perf-profile.self.cycles-pp.__mod_lruvec_state        0.18 ±  4%      -0.1        0.12 ±  5% perf-profile.self.cycles-pp.iomap_set_range_uptodate        0.60 ±  4%      -0.1        0.54 ±  2% perf-profile.self.cycles-pp._raw_spin_lock        0.77            -0.0        0.72 perf-profile.self.cycles-pp._raw_spin_lock_irqsave        0.14            -0.0        0.10 ±  8% perf-profile.self.cycles-pp.find_get_entry        0.14 ±  3%      -0.0        0.11 ±  4% perf-profile.self.cycles-pp.xas_load        0.17 ±  4%      -0.0        0.14 ±  3% perf-profile.self.cycles-pp.write        0.30 ±  2%      -0.0        0.27 ±  6% perf-profile.self.cycles-pp._raw_spin_lock_irq        0.18 ±  5%      -0.0        0.15 ±  7% perf-profile.self.cycles-pp.__flush_work        0.08 ±  5%      -0.0        0.05 perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork        0.12 ±  3%      -0.0        0.09 ±  4% perf-profile.self.cycles-pp.brd_lookup_page        0.12 ±  7%      -0.0        0.09 ±  9% perf-profile.self.cycles-pp.vfs_write        0.18 ±  4%      -0.0        0.15 ±  7% perf-profile.self.cycles-pp.insert_work        0.13            -0.0        0.10 ±  8% perf-profile.self.cycles-pp.memset_erms        0.11 ±  4%      -0.0        0.08 perf-profile.self.cycles-pp.xfs_file_buffered_aio_write        0.11 ±  6%      -0.0        0.08 ±  5% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin        0.12 ±  3%      -0.0        0.09 ±  4% perf-profile.self.cycles-pp.iomap_writepage_map        0.08 ±  8%      -0.0        0.06 ± 15% perf-profile.self.cycles-pp.bio_alloc_bioset        0.06 ±  6%      -0.0        0.04 ± 57% perf-profile.self.cycles-pp.xfs_bmbt_to_iomap        0.09 ±  5%      -0.0        0.06 ±  6% perf-profile.self.cycles-pp.__fget_light        0.06            -0.0        0.04 ± 57% perf-profile.self.cycles-pp.xfs_map_blocks        0.18 ±  2%      -0.0        0.15 ±  3% perf-profile.self.cycles-pp.__list_add_valid        0.08 ±  6%      -0.0        0.06 ±  9% perf-profile.self.cycles-pp.xfs_ilock        0.08 ±  6%      -0.0        0.06 ±  9% perf-profile.self.cycles-pp.iov_iter_fault_in_readable        0.11 ±  4%      -0.0        0.09 ±  4% perf-profile.self.cycles-pp.switch_fpu_return        0.07            -0.0        0.05 perf-profile.self.cycles-pp.iomap_write_actor        0.23 ±  2%      -0.0        0.21 ±  5% perf-profile.self.cycles-pp.__radix_tree_lookup        0.18 ±  4%      -0.0        0.16 ±  6% perf-profile.self.cycles-pp.xlog_cil_force_lsn        0.20 ±  2%      -0.0        0.18 ±  4% perf-profile.self.cycles-pp.kmem_cache_alloc        0.10 ±  5%      -0.0        0.08 ± 11% perf-profile.self.cycles-pp.xfs_file_fsync        0.07 ±  5%      -0.0        0.05 ±  9% perf-profile.self.cycles-pp.clear_page_dirty_for_io        0.07 ±  6%      -0.0        0.05 perf-profile.self.cycles-pp.__test_set_page_writeback        0.11            -0.0        0.09 ±  8% perf-profile.self.cycles-pp.common_file_perm        0.23            -0.0        0.21 ±  4% perf-profile.self.cycles-pp.update_rq_clock        0.08 ±  6%      -0.0        0.06 perf-profile.self.cycles-pp.xfs_file_write_iter        0.10 ±  5%      -0.0        0.08 ±  8% perf-profile.self.cycles-pp.up_read        0.08 ±  5%      -0.0        0.07 perf-profile.self.cycles-pp.write_cache_pages        0.10            -0.0        0.09 perf-profile.self.cycles-pp.reweight_entity        0.05 ±  8%      +0.0        0.07 ±  7% perf-profile.self.cycles-pp.dequeue_entity        0.16 ±  4%      +0.0        0.17 ±  2% perf-profile.self.cycles-pp.__switch_to_asm        0.11 ±  4%      +0.0        0.12 ±  3% perf-profile.self.cycles-pp.set_next_entity        0.15 ±  3%      +0.0        0.16 ±  2% perf-profile.self.cycles-pp.orc_find        0.09 ±  4%      +0.0        0.11 ±  6% perf-profile.self.cycles-pp.enqueue_entity        0.10 ±  4%      +0.0        0.12 ±  3% perf-profile.self.cycles-pp._find_next_bit        0.01 ±173%      +0.0        0.06 ±  7% perf-profile.self.cycles-pp.check_preempt_wakeup        0.01 ±173%      +0.0        0.06 ± 11% perf-profile.self.cycles-pp.xfs_trans_committed_bulk        0.45 ±  2%      +0.1        0.51 ±  2% perf-profile.self.cycles-pp.__sched_text_start        0.15 ±  6%      +0.1        0.21 ±  3% perf-profile.self.cycles-pp.memcpy_erms        0.00            +0.1        0.06 ±  6% perf-profile.self.cycles-pp.xfs_verify_agbno        0.08 ±  8%      +0.1        0.16 ±  9% perf-profile.self.cycles-pp.xfs_iext_lookup_extent        9.53            +1.2       10.76 perf-profile.self.cycles-pp.intel_idle


                                   aim7.jobs-per-min
   45000 +-------------------------------------------------------------------+          |        + +                                                       |    40000 |.+.+    :   + .+.   +. .+.     |    35000 |-+ :    :    +   +.+ +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+   ++.+.|          |   : :                                                          |    30000 |-+ : :                                                           |    25000 |-O :O O:O OO                                                     |          |   :   :  O    O O OO O O O O                                      |    20000 |-+  : :                                                           |    15000 |-+  : :                                                           |          |    : :                                                           |    10000 |-+  : :                                                            |     5000 |-+  : :                                                            |          |    : :                                                            |        0 +-------------------------------------------------------------------+
                                aim7.time.system_time
   5000 +--------------------------------------------------------------------+    4500 |-+        O   OO   O    O O O                                       |         | O OO O O   O    O   O O                                            |    4000 |-+                                                                  |    3500 |-+                                                                  |         |             .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+ .+.+.+. .++.+.|    3000 |.+.+       .+  +     +                               + +      |    2500 |-+ : +.+                                                         |    2000 |-+ : :                                                           |         |   : :                                                            |    1500 |-+  : :                                                            |    1000 |-+  : :                                                            |         |    : :                                                            |     500 |-+  : :                                                             |       0 +--------------------------------------------------------------------+
                               aim7.time.elapsed_time
   160 +---------------------------------------------------------------------+        |   O OO O O O O OO O O O O                                         |    140 |-O                                                                   |    120 |-+                                                                   | |                                                                     |    100 |-+           .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|        |.+.+    +. .+   +    + +      |     80 |-+ :    : +                                                          |        |    : :                                                            |     60 |-+  : :                                                             |     40 |-+  : :                                                             |        |    : :                                                             |     20 |-+   : :                                                             |        | ::                                                              |      0 +---------------------------------------------------------------------+
                             aim7.time.elapsed_time.max
   160 +---------------------------------------------------------------------+        |   O OO O O O O OO O O O O                                         |    140 |-O                                                                   |    120 |-+                                                                   | |                                                                     |    100 |-+           .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|        |.+.+    +. .+   +    + +      |     80 |-+ :    : +                                                          |        |    : :                                                            |     60 |-+  : :                                                             |     40 |-+  : :                                                             |        |    : :                                                             |     20 |-+   : :                                                             |        | ::                                                              |      0 +---------------------------------------------------------------------+
                         aim7.time.voluntary_context_switches
   8e+07 +-------------------------------------------------------------------+          | O          O  O O OO O O                                        |    7e+07 |-+                                                                 |    6e+07 |-+                                                                 | |                                                                   |    5e+07 |-+           +.+.+.++.+.+.+.++.+.+. .+   .+.+.    .+. .+.   +.+.|          |.+.+       .+                      +  +.+     ++.+   ++.+ +.+    |    4e+07 |-+ : +.+                                                        |          |   : :                                                          |    3e+07 |-+ : :                                                           |    2e+07 |-+  : :                                                           |          |    : :                                                           |    1e+07 |-+  : :                                                           |          |    : :                                                            |        0 +-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


_______________________________________________
LKP mailing list -- l...@lists.01.org
To unsubscribe send an email to lkp-le...@lists.01.org



--
Zhengjun Xing

Reply via email to