On Fri, Dec 11, 2020 at 11:06:03AM -0500, Vivek Goyal wrote: [..] > > > > Could we measure at what point does a large window size actually make > > performance worse? > > Will do. Will run tests with varying window sizes (small to large) > and see how does it impact performance for same workload with > same guest memory. >
Ok, I ran some tests with virtiofs cache=auto and varied dax window size from 1G to 64G. (1, 2, 4, 8, 16, 32, 64). Still using file size 4G each. And -multi test use 4 files of 4G each. Here are the test results. To run tests faster, I ran one iteration of each job (instead of running 3 iterations and taking average). This should be good enough to give some idea. I have also pushed raw data file and test results here. https://github.com/rhvgoyal/virtiofs-tests/tree/master/performance-results/dec-10-2020 First line is vtfs cache auto without dax to see with what dax window size dax helps as opposed to without dax. Conclusion ---------- - Decreasing window size hurts tremendously for random read and random write workloads. - seqread/seqwrite still perform reasonably well even with smaller window size. - Increasing DAX window does not hurt. I will try to spend some time with smaller dax window and random read and ramdom write workloads and see if there is anything I can do to reduce the performance drop. Thanks Vivek NAME WORKLOAD Bandwidth IOPS vtfs-auto seqread-psync 650.0mb 162.5k vtfs-auto-dax-1G seqread-psync 653.1mb 163.2k vtfs-auto-dax-2G seqread-psync 668.2mb 167.0k vtfs-auto-dax-4G seqread-psync 657.8mb 164.4k vtfs-auto-dax-8G seqread-psync 699.5mb 174.8k vtfs-auto-dax-16G seqread-psync 674.3mb 168.5k vtfs-auto-dax-32G seqread-psync 667.6mb 166.9k vtfs-auto-dax-64G seqread-psync 696.1mb 174.0k vtfs-auto seqread-psync-multi 2530.7mb 632.6k vtfs-auto-dax-1G seqread-psync-multi 1931.1mb 482.7k vtfs-auto-dax-2G seqread-psync-multi 2105.6mb 526.4k vtfs-auto-dax-4G seqread-psync-multi 2142.2mb 535.5k vtfs-auto-dax-8G seqread-psync-multi 2229.7mb 557.4k vtfs-auto-dax-16G seqread-psync-multi 2324.3mb 581.0k vtfs-auto-dax-32G seqread-psync-multi 2421.1mb 605.2k vtfs-auto-dax-64G seqread-psync-multi 2469.7mb 617.4k vtfs-auto seqread-mmap 570.6mb 142.6k vtfs-auto-dax-1G seqread-mmap 823.9mb 205.9k vtfs-auto-dax-2G seqread-mmap 798.1mb 199.5k vtfs-auto-dax-4G seqread-mmap 874.4mb 218.6k vtfs-auto-dax-8G seqread-mmap 960.5mb 240.1k vtfs-auto-dax-16G seqread-mmap 917.1mb 229.2k vtfs-auto-dax-32G seqread-mmap 884.4mb 221.1k vtfs-auto-dax-64G seqread-mmap 855.6mb 213.9k vtfs-auto seqread-mmap-multi 2210.1mb 552.5k vtfs-auto-dax-1G seqread-mmap-multi 2722.9mb 680.7k vtfs-auto-dax-2G seqread-mmap-multi 2828.2mb 707.0k vtfs-auto-dax-4G seqread-mmap-multi 2777.4mb 694.3k vtfs-auto-dax-8G seqread-mmap-multi 2880.4mb 720.1k vtfs-auto-dax-16G seqread-mmap-multi 3159.2mb 789.8k vtfs-auto-dax-32G seqread-mmap-multi 2989.2mb 747.3k vtfs-auto-dax-64G seqread-mmap-multi 3007.3mb 751.8k vtfs-auto seqread-libaio 303.3mb 75.8k vtfs-auto-dax-1G seqread-libaio 396.4mb 99.1k vtfs-auto-dax-2G seqread-libaio 391.9mb 97.9k vtfs-auto-dax-4G seqread-libaio 422.9mb 105.7k vtfs-auto-dax-8G seqread-libaio 419.0mb 104.7k vtfs-auto-dax-16G seqread-libaio 411.2mb 102.8k vtfs-auto-dax-32G seqread-libaio 436.1mb 109.0k vtfs-auto-dax-64G seqread-libaio 418.0mb 104.5k vtfs-auto seqread-libaio-multi 1500.0mb 375.0k vtfs-auto-dax-1G seqread-libaio-multi 1271.7mb 317.9k vtfs-auto-dax-2G seqread-libaio-multi 1338.1mb 334.5k vtfs-auto-dax-4G seqread-libaio-multi 1306.6mb 326.6k vtfs-auto-dax-8G seqread-libaio-multi 1416.8mb 354.2k vtfs-auto-dax-16G seqread-libaio-multi 1461.9mb 365.4k vtfs-auto-dax-32G seqread-libaio-multi 1531.9mb 382.9k vtfs-auto-dax-64G seqread-libaio-multi 1495.4mb 373.8k vtfs-auto randread-psync 77.9mb 19.4k vtfs-auto-dax-1G randread-psync 18.7mb 4792 vtfs-auto-dax-2G randread-psync 25.2mb 6463 vtfs-auto-dax-4G randread-psync 59.9mb 14.9k vtfs-auto-dax-8G randread-psync 693.7mb 173.4k vtfs-auto-dax-16G randread-psync 646.0mb 161.5k vtfs-auto-dax-32G randread-psync 635.4mb 158.8k vtfs-auto-dax-64G randread-psync 663.4mb 165.8k vtfs-auto randread-psync-multi 285.2mb 71.3k vtfs-auto-dax-1G randread-psync-multi 28.7mb 7357 vtfs-auto-dax-2G randread-psync-multi 29.6mb 7596 vtfs-auto-dax-4G randread-psync-multi 35.0mb 8969 vtfs-auto-dax-8G randread-psync-multi 51.3mb 12.8k vtfs-auto-dax-16G randread-psync-multi 497.1mb 124.2k vtfs-auto-dax-32G randread-psync-multi 2092.9mb 523.2k vtfs-auto-dax-64G randread-psync-multi 2051.0mb 512.7k vtfs-auto randread-mmap 57.7mb 14.4k vtfs-auto-dax-1G randread-mmap 15.1mb 3889 vtfs-auto-dax-2G randread-mmap 21.5mb 5506 vtfs-auto-dax-4G randread-mmap 59.4mb 14.8k vtfs-auto-dax-8G randread-mmap 970.1mb 242.5k vtfs-auto-dax-16G randread-mmap 883.1mb 220.7k vtfs-auto-dax-32G randread-mmap 874.2mb 218.5k vtfs-auto-dax-64G randread-mmap 954.3mb 238.5k vtfs-auto randread-mmap-multi 243.5mb 60.8k vtfs-auto-dax-1G randread-mmap-multi 11.7mb 3005 vtfs-auto-dax-2G randread-mmap-multi 20.4mb 5232 vtfs-auto-dax-4G randread-mmap-multi 23.1mb 5937 vtfs-auto-dax-8G randread-mmap-multi 35.3mb 9058 vtfs-auto-dax-16G randread-mmap-multi 1009.8mb 252.4k vtfs-auto-dax-32G randread-mmap-multi 2612.6mb 653.1k vtfs-auto-dax-64G randread-mmap-multi 2908.5mb 727.1k vtfs-auto randread-libaio 276.2mb 69.0k vtfs-auto-dax-1G randread-libaio 19.4mb 4968 vtfs-auto-dax-2G randread-libaio 23.3mb 5981 vtfs-auto-dax-4G randread-libaio 60.6mb 15.1k vtfs-auto-dax-8G randread-libaio 413.5mb 103.3k vtfs-auto-dax-16G randread-libaio 410.8mb 102.7k vtfs-auto-dax-32G randread-libaio 392.0mb 98.0k vtfs-auto-dax-64G randread-libaio 412.9mb 103.2k vtfs-auto randread-libaio-multi 271.0mb 67.7k vtfs-auto-dax-1G randread-libaio-multi 29.2mb 7479 vtfs-auto-dax-2G randread-libaio-multi 30.4mb 7803 vtfs-auto-dax-4G randread-libaio-multi 36.6mb 9387 vtfs-auto-dax-8G randread-libaio-multi 49.0mb 12.2k vtfs-auto-dax-16G randread-libaio-multi 364.3mb 91.0k vtfs-auto-dax-32G randread-libaio-multi 1435.0mb 358.7k vtfs-auto-dax-64G randread-libaio-multi 1358.4mb 339.6k vtfs-auto seqwrite-psync 78.6mb 19.6k vtfs-auto-dax-1G seqwrite-psync 440.7mb 110.1k vtfs-auto-dax-2G seqwrite-psync 428.6mb 107.1k vtfs-auto-dax-4G seqwrite-psync 476.5mb 119.1k vtfs-auto-dax-8G seqwrite-psync 547.8mb 136.9k vtfs-auto-dax-16G seqwrite-psync 530.1mb 132.5k vtfs-auto-dax-32G seqwrite-psync 486.6mb 121.6k vtfs-auto-dax-64G seqwrite-psync 515.8mb 128.9k vtfs-auto seqwrite-psync-multi 305.7mb 76.4k vtfs-auto-dax-1G seqwrite-psync-multi 1405.0mb 351.2k vtfs-auto-dax-2G seqwrite-psync-multi 1447.0mb 361.7k vtfs-auto-dax-4G seqwrite-psync-multi 1329.4mb 332.3k vtfs-auto-dax-8G seqwrite-psync-multi 1250.4mb 312.5k vtfs-auto-dax-16G seqwrite-psync-multi 1264.7mb 316.1k vtfs-auto-dax-32G seqwrite-psync-multi 1224.5mb 306.1k vtfs-auto-dax-64G seqwrite-psync-multi 1165.6mb 291.4k vtfs-auto seqwrite-mmap 199.7mb 49.9k vtfs-auto-dax-1G seqwrite-mmap 1262.6mb 315.6k vtfs-auto-dax-2G seqwrite-mmap 1369.4mb 342.3k vtfs-auto-dax-4G seqwrite-mmap 1363.5mb 340.8k vtfs-auto-dax-8G seqwrite-mmap 1368.9mb 342.2k vtfs-auto-dax-16G seqwrite-mmap 1305.2mb 326.3k vtfs-auto-dax-32G seqwrite-mmap 1299.4mb 324.8k vtfs-auto-dax-64G seqwrite-mmap 1301.1mb 325.2k vtfs-auto seqwrite-mmap-multi 384.1mb 96.0k vtfs-auto-dax-1G seqwrite-mmap-multi 2162.9mb 540.7k vtfs-auto-dax-2G seqwrite-mmap-multi 2191.8mb 547.9k vtfs-auto-dax-4G seqwrite-mmap-multi 2045.1mb 511.2k vtfs-auto-dax-8G seqwrite-mmap-multi 1973.0mb 493.2k vtfs-auto-dax-16G seqwrite-mmap-multi 1911.3mb 477.8k vtfs-auto-dax-32G seqwrite-mmap-multi 2263.9mb 565.9k vtfs-auto-dax-64G seqwrite-mmap-multi 1718.4mb 429.6k vtfs-auto seqwrite-libaio 279.1mb 69.7k vtfs-auto-dax-1G seqwrite-libaio 294.2mb 73.5k vtfs-auto-dax-2G seqwrite-libaio 308.7mb 77.1k vtfs-auto-dax-4G seqwrite-libaio 308.3mb 77.0k vtfs-auto-dax-8G seqwrite-libaio 339.0mb 84.7k vtfs-auto-dax-16G seqwrite-libaio 323.3mb 80.8k vtfs-auto-dax-32G seqwrite-libaio 311.4mb 77.8k vtfs-auto-dax-64G seqwrite-libaio 334.2mb 83.5k vtfs-auto seqwrite-libaio-multi 306.4mb 76.6k vtfs-auto-dax-1G seqwrite-libaio-multi 968.7mb 242.1k vtfs-auto-dax-2G seqwrite-libaio-multi 1050.7mb 262.6k vtfs-auto-dax-4G seqwrite-libaio-multi 989.4mb 247.3k vtfs-auto-dax-8G seqwrite-libaio-multi 1017.8mb 254.4k vtfs-auto-dax-16G seqwrite-libaio-multi 1101.5mb 275.3k vtfs-auto-dax-32G seqwrite-libaio-multi 915.0mb 228.7k vtfs-auto-dax-64G seqwrite-libaio-multi 1067.5mb 266.8k vtfs-auto randwrite-psync 82.2mb 20.5k vtfs-auto-dax-1G randwrite-psync 20.4mb 5240 vtfs-auto-dax-2G randwrite-psync 29.6mb 7585 vtfs-auto-dax-4G randwrite-psync 65.4mb 16.3k vtfs-auto-dax-8G randwrite-psync 502.6mb 125.6k vtfs-auto-dax-16G randwrite-psync 501.6mb 125.4k vtfs-auto-dax-32G randwrite-psync 461.6mb 115.4k vtfs-auto-dax-64G randwrite-psync 481.2mb 120.2k vtfs-auto randwrite-psync-multi 302.1mb 75.5k vtfs-auto-dax-1G randwrite-psync-multi 30.9mb 7915 vtfs-auto-dax-2G randwrite-psync-multi 35.0mb 8968 vtfs-auto-dax-4G randwrite-psync-multi 36.7mb 9399 vtfs-auto-dax-8G randwrite-psync-multi 57.1mb 14.2k vtfs-auto-dax-16G randwrite-psync-multi 499.0mb 124.7k vtfs-auto-dax-32G randwrite-psync-multi 1123.0mb 280.7k vtfs-auto-dax-64G randwrite-psync-multi 1058.3mb 264.5k vtfs-auto randwrite-mmap 55.9mb 13.9k vtfs-auto-dax-1G randwrite-mmap 16.4mb 4221 vtfs-auto-dax-2G randwrite-mmap 22.0mb 5648 vtfs-auto-dax-4G randwrite-mmap 60.6mb 15.1k vtfs-auto-dax-8G randwrite-mmap 763.3mb 190.8k vtfs-auto-dax-16G randwrite-mmap 859.2mb 214.8k vtfs-auto-dax-32G randwrite-mmap 792.5mb 198.1k vtfs-auto-dax-64G randwrite-mmap 807.7mb 201.9k vtfs-auto randwrite-mmap-multi 208.9mb 52.2k vtfs-auto-dax-1G randwrite-mmap-multi 12.4mb 3191 vtfs-auto-dax-2G randwrite-mmap-multi 19.8mb 5072 vtfs-auto-dax-4G randwrite-mmap-multi 22.3mb 5710 vtfs-auto-dax-8G randwrite-mmap-multi 33.2mb 8511 vtfs-auto-dax-16G randwrite-mmap-multi 1061.2mb 265.3k vtfs-auto-dax-32G randwrite-mmap-multi 2113.2mb 528.3k vtfs-auto-dax-64G randwrite-mmap-multi 2099.4mb 524.8k vtfs-auto randwrite-libaio 258.7mb 64.6k vtfs-auto-dax-1G randwrite-libaio 20.4mb 5231 vtfs-auto-dax-2G randwrite-libaio 26.4mb 6774 vtfs-auto-dax-4G randwrite-libaio 64.9mb 16.2k vtfs-auto-dax-8G randwrite-libaio 326.7mb 81.6k vtfs-auto-dax-16G randwrite-libaio 303.6mb 75.9k vtfs-auto-dax-32G randwrite-libaio 298.2mb 74.5k vtfs-auto-dax-64G randwrite-libaio 328.5mb 82.1k vtfs-auto randwrite-libaio-multi 320.9mb 80.2k vtfs-auto-dax-1G randwrite-libaio-multi 31.1mb 7977 vtfs-auto-dax-2G randwrite-libaio-multi 34.2mb 8772 vtfs-auto-dax-4G randwrite-libaio-multi 38.8mb 9945 vtfs-auto-dax-8G randwrite-libaio-multi 55.9mb 13.9k vtfs-auto-dax-16G randwrite-libaio-multi 363.2mb 90.8k vtfs-auto-dax-32G randwrite-libaio-multi 800.4mb 200.1k vtfs-auto-dax-64G randwrite-libaio-multi 805.5mb 201.3k vtfs-auto randrw-psync 38.1mb/12.7mb 9775/3265 vtfs-auto-dax-1G randrw-psync 15.0mb/5134kb 3855/1283 vtfs-auto-dax-2G randrw-psync 19.0mb/6490kb 4866/1622 vtfs-auto-dax-4G randrw-psync 41.9mb/14.0mb 10.4k/3595 vtfs-auto-dax-8G randrw-psync 422.2mb/141.1mb 105.5k/35.2k vtfs-auto-dax-16G randrw-psync 395.9mb/132.3mb 98.9k/33.0k vtfs-auto-dax-32G randrw-psync 378.2mb/126.4mb 94.5k/31.5k vtfs-auto-dax-64G randrw-psync 419.2mb/140.1mb 104.8k/35.0k vtfs-auto randrw-psync-multi 151.4mb/50.7mb 37.8k/12.6k vtfs-auto-dax-1G randrw-psync-multi 23.5mb/8091kb 6020/2022 vtfs-auto-dax-2G randrw-psync-multi 24.7mb/8515kb 6343/2128 vtfs-auto-dax-4G randrw-psync-multi 27.9mb/9606kb 7148/2401 vtfs-auto-dax-8G randrw-psync-multi 38.6mb/12.9mb 9882/3309 vtfs-auto-dax-16G randrw-psync-multi 277.7mb/93.0mb 69.4k/23.2k vtfs-auto-dax-32G randrw-psync-multi 1233.6mb/412.5mb308.4k/103.1k vtfs-auto-dax-64G randrw-psync-multi 1447.4mb/483.9mb361.8k/120.9k vtfs-auto randrw-mmap 44.4mb/14.8mb 11.1k/3806 vtfs-auto-dax-1G randrw-mmap 11.6mb/3978kb 2980/994 vtfs-auto-dax-2G randrw-mmap 15.0mb/5133kb 3854/1283 vtfs-auto-dax-4G randrw-mmap 36.1mb/12.0mb 9247/3088 vtfs-auto-dax-8G randrw-mmap 618.2mb/206.6mb 154.5k/51.6k vtfs-auto-dax-16G randrw-mmap 638.2mb/213.3mb 159.5k/53.3k vtfs-auto-dax-32G randrw-mmap 554.6mb/185.3mb 138.6k/46.3k vtfs-auto-dax-64G randrw-mmap 608.5mb/203.3mb 152.1k/50.8k vtfs-auto randrw-mmap-multi 179.3mb/60.0mb 44.8k/15.0k vtfs-auto-dax-1G randrw-mmap-multi 11.7mb/4069kb 3007/1017 vtfs-auto-dax-2G randrw-mmap-multi 14.7mb/5088kb 3768/1272 vtfs-auto-dax-4G randrw-mmap-multi 16.8mb/5806kb 4307/1451 vtfs-auto-dax-8G randrw-mmap-multi 24.7mb/8515kb 6341/2128 vtfs-auto-dax-16G randrw-mmap-multi 479.4mb/160.4mb 119.8k/40.1k vtfs-auto-dax-32G randrw-mmap-multi 1290.1mb/431.3mb322.5k/107.8k vtfs-auto-dax-64G randrw-mmap-multi 1828.4mb/611.4mb457.1k/152.8k vtfs-auto randrw-libaio 83.4mb/27.9mb 20.8k/7161 vtfs-auto-dax-1G randrw-libaio 13.9mb/4725kb 3565/1181 vtfs-auto-dax-2G randrw-libaio 17.1mb/5866kb 4396/1466 vtfs-auto-dax-4G randrw-libaio 38.1mb/12.7mb 9756/3259 vtfs-auto-dax-8G randrw-libaio 253.0mb/84.5mb 63.2k/21.1k vtfs-auto-dax-16G randrw-libaio 252.5mb/84.4mb 63.1k/21.1k vtfs-auto-dax-32G randrw-libaio 239.9mb/80.1mb 59.9k/20.0k vtfs-auto-dax-64G randrw-libaio 255.7mb/85.4mb 63.9k/21.3k vtfs-auto randrw-libaio-multi 209.6mb/70.2mb 52.4k/17.5k vtfs-auto-dax-1G randrw-libaio-multi 23.8mb/8197kb 6105/2049 vtfs-auto-dax-2G randrw-libaio-multi 21.5mb/7418kb 5507/1854 vtfs-auto-dax-4G randrw-libaio-multi 28.0mb/9651kb 7186/2412 vtfs-auto-dax-8G randrw-libaio-multi 34.6mb/11.6mb 8873/2976 vtfs-auto-dax-16G randrw-libaio-multi 197.4mb/66.1mb 49.3k/16.5k vtfs-auto-dax-32G randrw-libaio-multi 633.8mb/211.9mb 158.4k/52.9k vtfs-auto-dax-64G randrw-libaio-multi 943.8mb/315.5mb 235.9k/78.8k
