[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Excellent news. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Hi Colin, thanks so much for giving this another look. I don't see any
references to trying different zfs_arc_meta_limit* settings here!
I raised zfs_arc_meta_limit_percent from the default 75 to 95 and my
system has been running this ripgrep for four hours without falling to
pieces.
I think we can declare success on this one!
Thanks
# grep . /sys/module/zfs/parameters/zfs_arc_{dnode,meta}_limit*
/sys/module/zfs/parameters/zfs_arc_dnode_limit:0
/sys/module/zfs/parameters/zfs_arc_dnode_limit_percent:50
/sys/module/zfs/parameters/zfs_arc_meta_limit:0
/sys/module/zfs/parameters/zfs_arc_meta_limit_percent:95
7.97% [kernel] [k] SHA256TransformBlocks
3.33% [kernel] [k] _raw_spin_lock
2.59% [kernel] [k] arc_evict_state
2.39% [kernel] [k] mutex_lock
2.08% [kernel] [k] osq_lock
PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
28555 sarnold 20 0 28.648g 0.023t 2160 S 262.8 18.8 546:39.58 rg
1567 root 20 0 0 0 0 R 71.4 0.0 1200:10 arc_reclaim
2 root 20 0 0 0 0 R 23.7 0.0 238:17.77 kthreadd
1966 root 0 -20 0 0 0 D 11.2 0.0 163:24.49 z_rd_int_2
1967 root 0 -20 0 0 0 S 11.2 0.0 163:25.54 z_rd_int_3
1968 root 0 -20 0 0 0 S 11.2 0.0 163:25.52 z_rd_int_4
# zpool iostat 1
capacity operations bandwidth
poolalloc free read write read write
[...]
-- - - - - - -
fst 2.11T 1.52T 13.8K 0 112M 0
srv 1.84T 6.32T 0 0 0 0
-- - - - - - -
fst 2.11T 1.52T 17.4K 0 146M 0
srv 1.84T 6.32T 0 0 0 0
-- - - - - - -
fst 2.11T 1.52T 17.3K 0 147M 0
srv 1.84T 6.32T 0 0 0 0
-- - - - - - -
# arc_summary
ZFS Subsystem ReportTue Oct 27 19:32:40 2020
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted:688.20m
Mutex Misses: 115.54m
Evict Skips:99.92b
ARC Size: 89.14% 56.07 GiB
Target Size: (Adaptive) 9.30% 5.85GiB
Min Size (Hard Limit): 6.25% 3.93GiB
Max Size (High Water): 16:162.90 GiB
ARC Size Breakdown:
Recently Used Cache Size: 81.96% 24.51 GiB
Frequently Used Cache Size: 18.04% 5.39GiB
ARC Hash Breakdown:
Elements Max: 9.78m
Elements Current: 56.79% 5.55m
Collisions: 201.32m
Chain Max: 8
Chains: 736.70k
ARC Total accesses: 12.23b
Cache Hit Ratio:90.87% 11.12b
Cache Miss Ratio: 9.13% 1.12b
Actual Hit Ratio: 90.27% 11.04b
Data Demand Efficiency: 82.63% 666.79m
Data Prefetch Efficiency: 27.20% 14.18m
CACHE HITS BY CACHE LIST:
Most Recently Used: 30.85% 3.43b
Most Frequently Used: 68.49% 7.61b
Most Recently Used Ghost: 0.13% 13.99m
Most Frequently Used Ghost: 0.61% 67.51m
CACHE HITS BY DATA TYPE:
Demand Data: 4.96% 550.94m
Prefetch Data:0.03% 3.86m
Demand Metadata: 94.20% 10.47b
Prefetch Metadata:0.80% 89.48m
CACHE MISSES BY DATA TYPE:
Demand Data: 10.37% 115.85m
Prefetch Data:0.92% 10.32m
Demand Metadata: 82.55% 921.91m
Prefetch Metadata:6.15% 68.73m
L2 ARC Summary: (HEALTHY)
Low Memory Aborts: 116
Free on Write: 21.13k
R/W Clashes:3
Bad Checksums: 0
IO Errors: 0
L2 ARC Size: (Adaptive) 418.26 GiB
Compressed: 90.78% 379.71 GiB
Header Size:0.08% 360.27 MiB
L2 AR
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
OK, Seems like we can resolve this by using some tuning. I was able to reproduce this on a single drive SSD pool configuration with 30 clones of the linux source and grep'ing for various strings. So, the ARC determines that it can't free up enough memory by releasing unpinned buffers so the prune kicks in. With the use case you have here a large fraction of the ARC is pinned which causes the issue you see here. Tuning can resolve this for your specific use case. I recommend bumping the settings in: /sys/module/zfs/parameters/zfs_arc_meta_limit_percent /sys/module/zfs/parameters/zfs_arc_dnode_limit_percent For example, when I set these to 100% the performance improved dramatically and the pruning reduced. I recommend increasing these settings until you see performance running at the level you require. Try to avoid setting them too high - make sure they are set high enough for your use case but no higher. Let me know if this resolves your issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
The fix seems to be causing some issues, as reported here: https://github.com/openzfs/zfs/pull/10331#issuecomment-636502835 ..so I'm watching the upstream fixes to see how it all shakes out. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Thanks for the heads-up on this fix. It does not seem to have been merged yet, so I'll wait for it to get through the regression tests and once it's upstream I'll backport it for you to test. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Hello Colin, this looks promising for my arc_prune spinlock contention problems: https://github.com/openzfs/zfs/pull/10331 with some background here: https://github.com/openzfs/zfs/issues/7559 This might have a simple ~dozen line fix! It's not yet reviewed by the openzfs gurus but it sure *looks* plausible to me. I tried to bodge this patch into the dkms module you gave me for my other zfs problem, but I have no confidence in my work -- I tried to reduce the number of arc_prune threads but failed, so I'm skeptical if I was able to change anything anyway. (I'm finding dkms to be *very* confusing.) Any chance I can ask you to put together a dkms with this patch? I'd trust your work way further. :) Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
I forgot to mention, my l2arc is used on the second pool on this system: $ zpool iostat -v capacity operations bandwidth pool alloc free read write read write --- - - - - - - fst 1.79T 1.84T611 52 5.51M 968K raidz1 1.79T 1.84T611 52 5.51M 968K wwn-0x5002538e4095da39 - -154 12 1.40M 242K wwn-0x5002538e4095bdd6 - -150 12 1.36M 242K wwn-0x5002538e4093c6fd - -154 13 1.40M 242K wwn-0x5002538e4095da30 - -150 13 1.36M 242K --- - - - - - - srv 1.69T 6.47T 45 11 4.08M 624K mirror 576G 2.16T 15 3 1.36M 206K ata-HGST_HUS724030ALA640_PN2234P8KTWJYY - - 5 1 464K 68.8K ata-HGST_HUS724030ALA640_PN2234P9G620TW - - 5 1 465K 68.8K ata-HGST_HUS724030ALA640_PN2234P9G66E2U - - 5 1 465K 68.8K mirror 576G 2.16T 15 3 1.36M 209K ata-HGST_HUS724030ALA640_PN2234P9G69TKU - - 5 1 465K 69.7K ata-HGST_HUS724030ALA640_PN2234P9G69TXU - - 5 1 464K 69.7K ata-HGST_HUS724030ALA640_PN2234P9G69U2U - - 5 1 464K 69.7K mirror 576G 2.16T 15 3 1.36M 208K ata-HGST_HUS724030ALA640_PN2234P9G6EBUU - - 5 1 464K 69.3K ata-HGST_HUS724030ALA640_PN2234P9G6ESAU - - 5 1 464K 69.3K ata-HGST_HUS724030ALA640_PN2234P9G6G70U - - 5 1 464K 69.3K logs - - - - - - nvme0n1p1 776K 19.9G 0 0 1 1 cache- - - - - - nvme0n1p2 91.4G 1006G 11 1 52.5K 155K --- - - - - - - -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Colin, thanks for the link to https://github.com/openzfs/zfs/issues/9966 ; unfortunately I think that's a different problem, my meta use seems less drastic than the github issue arc_prune 41859269059 arc_meta_used 43590932168 arc_meta_limit 494868305920 arc_dnode_limit 44294967296 arc_meta_max479969748624 arc_meta_min416777216 It looks like there's still 14898557296 bytes available for arc_meta. Do you know off-hand what the comment about the l2arc issue is? My l2arc is 1TB, well above 320 gigs. Here's where my stats are now (the system is not currently under any real load): $ cat /proc/spl/kstat/zfs/arcstats 13 1 0x01 96 4608 10492531405 1374854032619736 nametype data hits46820744874 misses 4953303948 demand_data_hits41163033450 demand_data_misses 4112593263 demand_metadata_hits45398565264 demand_metadata_misses 4771209639 prefetch_data_hits 43978361 prefetch_data_misses44454061 prefetch_metadata_hits 4255167799 prefetch_metadata_misses465046985 mru_hits42596176358 mru_ghost_hits 47960783 mfu_hits43972213379 mfu_ghost_hits 437066000 deleted 4624581823 mutex_miss 466283716 access_skip 4112 evict_skip 459660318004 evict_not_enough4531586443 evict_l2_cached 4758952236544 evict_l2_eligible 46629993222656 evict_l2_ineligible 41055824726016 evict_l2_skip 44093 hash_elements 41382698 hash_elements_max 47705756 hash_collisions 4101258453 hash_chains 454374 hash_chain_max 46 p 43403412095 c 49636886736 c_min 44221281536 c_max 4126491074560 size49501317832 compressed_size 46533731328 uncompressed_size 412946349568 overhead_size 41324591616 hdr_size4125140504 data_size 45910385664 metadata_size 41947937280 dbuf_size 4313455312 dnode_size 4801582080 bonus_size 4304865408 anon_size 469632 anon_evictable_data 40 anon_evictable_metadata 40 mru_size42874187264 mru_evictable_data 42820684800 mru_evictable_metadata 411149312 mru_ghost_size 41934149120 mru_ghost_evictable_data4254414848 mru_ghost_evictable_metadata41679734272 mfu_size44984066048 mfu_evictable_data 43089700864 mfu_evictable_metadata 4293028864 mfu_ghost_size 432707072 mfu_ghost_evictable_data432707072 mfu_ghost_evictable_metadata40 l2_hits 416313870 l2_misses 4936990042 l2_feeds41342134 l2_rw_clash 41 l2_read_bytes 473933849600 l2_write_bytes 4218946362880 l2_writes_sent 445529 l2_writes_done 445529 l2_writes_error 40 l2_writes_lock_retry421 l2_evict_lock_retry 40 l2_evict_reading40 l2_evict_l1cached 40 l2_free_on_write4878 l2_abort_lowmem 4496 l2_cksum_bad40 l2_io_error 40 l2_size 4130902416896 l2_asize498157307392 l2_hdr_size 497951584 memory_throttle_count 40 memory_direct_count 4856012 memory_indirect_count 4233680 memory_all_bytes4135081009152 memory_free_bytes 4120346943488 memory_available_bytes 3118236303360 arc_no_grow 40 arc_tempreserve 40 arc_loaned_bytes40 arc_prune 41859269059 arc_meta_used 43590932168 arc_meta_limit 494868305920 arc_dnode_limit 44294967296 arc_meta_max479969748624 arc
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Andreas, this system is running 18.04 LTS, 0.7.5-1ubuntu16.8, 4.15.0-91-generic. It has 128 gigs of ram; the workload is running ripgrep on an entire unpacked Ubuntu source archive, roughly 193 million files, 3.8 TB of data, on a single raidz1 ssd vdev. So I have no illusions that this workload fits into the available memory, but it's unfortunate that the best performance I've found so far includes dropping caches every few minutes. Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Hi there, https://github.com/openzfs/zfs/issues/9966 contains some advice that seems pertinent to this issue. ** Bug watch added: github.com/openzfs/zfs/issues #9966 https://github.com/openzfs/zfs/issues/9966 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
I've filed an issue with the upstream bug tracker: https://github.com/openzfs/zfs/issues/10222 ** Bug watch added: github.com/openzfs/zfs/issues #10222 https://github.com/openzfs/zfs/issues/10222 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
For what is worth, I also have the arc_prune eating cpu problem. I thought it was because my "machine" has just 3Gb of RAM, but this report makes me think there is something wrong in that area with zfs 0.8.x, since you have what, 64Gb of RAM? The same 3Gb machine didn't have any such problems with the previous version of zfs, 0.7.5. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Writing 8589934592 (eight gigs) into the zfs_arc_dnode_limit seemed to make an improvement: the arc_prune threads would periodically spike above 20% CPU use, but would return to 2-5% CPU use quickly; it worked well for a long time, perhaps even hours, before all the arc_prune threads returned to ~50% CPU use. I tried one gigabyte for a bit but the arc_prune threads returned quickly. Tuning this may take a bit. Thanks for the tip. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Thanks Colin, I believe that bountysource.com url is a scrape of https://github.com/openzfs/zfs/issues/6223 I hadn't seen much of this information before. Sadly my machine doesn't have enough memory to just keep turning up the knobs -- the working set far exceeds the memory of my computer. A recent archive-wide search I performed, I tried the following: echo 126491074560 >> /sys/module/zfs/parameters/zfs_arc_max # eight gigs less than RAM echo 117901139968 >> /sys/module/zfs/parameters/zfs_arc_dnode_limit # eight gigs less than ^^ I picked these numbers entirely at a guess. The zfs arc had previously been 0 -- for half of ram, 64 gigabytes. This workload seemed to run fine for longer than usual (arc_prune cpu numbers were low, good read iops reported in zpool status output -- not ideal indicator of actual progress, but a reasonable proxy) but eventually the arc_prune tasks started consuming most of the CPUs, and read iops fell to the dozens. I haven't tried a run with silly low numbers for the arc dnode limit, but I'm curious... Thanks ** Bug watch added: github.com/openzfs/zfs/issues #6223 https://github.com/openzfs/zfs/issues/6223 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
It may be worth reading the following article as it has some tuning tweaks that may help resolve your issue: https://www.bountysource.com/issues/46175251-arc_prune-high-load-and- soft-lockups ** Bug watch added: github.com/openzfs/zfs/issues #7559 https://github.com/openzfs/zfs/issues/7559 ** Also affects: zfs via https://github.com/openzfs/zfs/issues/7559 Importance: Unknown Status: Unknown ** Changed in: zfs-linux (Ubuntu) Status: Confirmed => In Progress ** Changed in: zfs-linux (Ubuntu) Status: In Progress => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/zfs/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: zfs-linux (Ubuntu) Importance: Undecided => High ** Changed in: zfs-linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
echo 1 > /proc/sys/vm/drop_caches # still slow echo 2 > /proc/sys/vm/drop_caches # goes faster Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
This is also repeatable on my pool of slower spinning metal disks, srv: top - 22:59:43 up 71 days, 22:35, 3 users, load average: 18.88, 17.98, 10.68 Tasks: 804 total, 20 running, 605 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 58.0 sy, 0.0 ni, 41.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 13191572+total, 14715308 free, 1172+used, 7199696 buff/cache KiB Swap: 5970940 total, 5950460 free,20480 used. 20842640 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 1523 root 20 0 0 0 0 R 98.3 0.0 512:59.56 arc_reclaim 11330 root 20 0 0 0 0 R 57.1 0.0 0:04.91 arc_prune 11388 root 20 0 0 0 0 S 57.1 0.0 0:02.48 arc_prune 1522 root 20 0 0 0 0 R 56.8 0.0 282:45.37 arc_prune 11259 root 20 0 0 0 0 S 56.8 0.0 0:08.04 arc_prune 11323 root 20 0 0 0 0 R 56.8 0.0 0:05.20 arc_prune 11395 root 20 0 0 0 0 R 56.8 0.0 0:02.23 arc_prune 11400 root 20 0 0 0 0 S 56.8 0.0 0:01.96 arc_prune 11350 root 20 0 0 0 0 R 56.1 0.0 0:03.92 arc_prune 11375 root 20 0 0 0 0 S 56.1 0.0 0:02.86 arc_prune 11373 root 20 0 0 0 0 S 55.8 0.0 0:02.91 arc_prune 11379 root 20 0 0 0 0 S 55.1 0.0 0:02.65 arc_prune 11396 root 20 0 0 0 0 R 54.8 0.0 0:02.06 arc_prune 11378 root 20 0 0 0 0 R 54.5 0.0 0:02.67 arc_prune 11399 root 20 0 0 0 0 R 54.1 0.0 0:01.92 arc_prune 11406 root 20 0 0 0 0 R 54.1 0.0 0:01.68 arc_prune 11407 root 20 0 0 0 0 R 52.5 0.0 0:01.59 arc_prune 11409 root 20 0 0 0 0 R 46.5 0.0 0:01.41 arc_prune 11410 root 20 0 0 0 0 S 45.9 0.0 0:01.39 arc_prune 11417 root 20 0 0 0 0 S 40.3 0.0 0:01.22 arc_prune 11421 root 20 0 0 0 0 R 36.3 0.0 0:01.10 arc_prune 11424 root 20 0 0 0 0 S 32.7 0.0 0:00.99 arc_prune 11428 root 20 0 0 0 0 R 28.4 0.0 0:00.86 arc_prune 11429 root 20 0 0 0 0 S 27.4 0.0 0:00.83 arc_prune 11430 root 20 0 0 0 0 S 22.8 0.0 0:00.69 arc_prune 11433 root 20 0 0 0 0 S 20.1 0.0 0:00.61 arc_prune 11434 root 20 0 0 0 0 R 17.5 0.0 0:00.53 arc_prune 11439 root 20 0 0 0 0 R 11.2 0.0 0:00.34 arc_prune 11440 root 20 0 0 0 0 R 10.2 0.0 0:00.31 arc_prune 11442 root 20 0 0 0 0 S 6.6 0.0 0:00.20 arc_prune 11445 root 20 0 0 0 0 R 3.0 0.0 0:00.09 arc_prune 8021 sarnold 20 0 58756 19168 4792 D 1.7 0.0 0:03.33 dpkg-source 11446 root 20 0 0 0 0 R 1.7 0.0 0:00.05 arc_prune sarnold@wopr:/tmp$ sudo -s [sudo] password for sarnold: root@wopr:/tmp# echo 3 > /proc/sys/vm/drop_caches top - 23:01:37 up 71 days, 22:37, 3 users, load average: 6.43, 14.26, 10.22 Tasks: 823 total, 2 running, 637 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.3 us, 4.7 sy, 0.0 ni, 93.3 id, 0.5 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 13191572+total, 11750942+free, 14105432 used, 300868 buff/cache KiB Swap: 5970940 total, 5950460 free,20480 used. 11703465+avail Mem PID USER PR NIVIRTRESSHR S
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Oh yes, and perf top output after dropping caches: Samples: 11M of event 'cycles:ppp', Event count (approx.): 648517108833 Overhead Shared Object Symbol 19.77% [kernel][k] zfs_prune 18.96% [kernel][k] _raw_spin_lock 12.89% [kernel][k] d_prune_aliases 7.12% [kernel][k] igrab 7.10% [kernel][k] _atomic_dec_and_lock 6.98% [kernel][k] iput 2.24% [kernel][k] Skein_512_Process_Block 1.65% [kernel][k] arc_evict_state 0.89% [kernel][k] lz4_decompress_zfs 0.78% [kernel][k] mutex_unlock 0.78% [kernel][k] mutex_lock 0.69% [kernel][k] native_queued_spin_lock_slowpath 0.52% [kernel][k] clear_page_erms 0.46% [kernel][k] _raw_spin_lock_irqsave 0.44% [kernel][k] Skein_Get64_LSB_First 0.35% [kernel][k] __schedule 0.27% [kernel][k] mutex_trylock 0.26% [kernel][k] kmem_cache_free 0.26% [kernel][k] _cond_resched 0.23% [kernel][k] arc_space_consume 0.21% [kernel][k] kmem_cache_alloc 0.20% [kernel][k] menu_select 0.20% [kernel][k] multilist_sublist_move_forward 0.19% [kernel][k] arc_space_return 0.18% [kernel][k] taskq_thread 0.18% [kernel][k] __cv_signal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
Debugging by voodoo dolls: I made a wild guess that stumbling over cached objects was getting in the way. I dropped caches with: echo 3 > /proc/sys/vm/drop_caches and immediately the read iops to the pool skyrocketed and the ripgrep output sure looks like it's going WAY faster: capacity operations bandwidth pool alloc free read write read write --- - - - - - - fst 1.15T 679G 75 0 403K 0 mirror 588G 340G 35 0 211K 0 sdj - - 23 0 128K 0 sdk - - 11 0 83.7K 0 mirror 588G 340G 39 0 191K 0 sdl - - 16 0 79.7K 0 sdm - - 22 0 112K 0 --- - - - - - - srv 1.30T 6.86T 5 0 47.8K 0 mirror 443G 2.29T 1 0 15.9K 0 ata-HGST_HUS724030ALA640_PN2234P8KTWJYY - - 0 0 7.97K 0 ata-HGST_HUS724030ALA640_PN2234P9G620TW - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G66E2U - - 0 0 7.97K 0 mirror 443G 2.29T 0 0 7.97K 0 ata-HGST_HUS724030ALA640_PN2234P9G69TKU - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69TXU - - 0 0 7.97K 0 ata-HGST_HUS724030ALA640_PN2234P9G69U2U - - 0 0 0 0 mirror 443G 2.29T 2 0 23.9K 0 ata-HGST_HUS724030ALA640_PN2234P9G6EBUU - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6ESAU - - 0 0 7.97K 0 ata-HGST_HUS724030ALA640_PN2234P9G6G70U - - 1 0 15.9K 0 logs - - - - - - nvme0n1p1 900K 19.9G 0 0 0 0 cache- - - - - - nvme0n1p2 334G 764G 3 0 16.4K 0 --- - - - - - - capacity operations bandwidth pool alloc free read write read write --- - - - - - - fst 1.15T 679G 67 0 375K 0 mirror 588G 340G 43 0 255K 0 sdj - - 25 0 147K 0 sdk - - 17 0 108K 0 mirror 588G 340G 23 0 120K 0 sdl - - 6 0 31.9K 0 sdm - - 16 0 87.7K 0 --- - - - - - - srv 1.30T 6.86T 3 0 15.9K 0 mirror 443G 2.29T 0 0 3.99K 0 ata-HGST_HUS724030ALA640_PN2234P8KTWJYY - - 0 0 3.99K 0 ata-HGST_HUS724030ALA640_PN2234P9G620TW - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G66E2U - - 0 0 0 0 mirror 443G 2.29T 1 0 7.97K 0 ata-HGST_HUS724030ALA640_PN2234P9G69TKU - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69TXU - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69U2U - - 1 0 7.97K 0 mirror 443G 2.29T 0 0 3.99K 0 ata-HGST_HUS724030ALA640_PN2234P9G6EBUU - - 0 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6ESAU - - 0 0 3.99K 0 ata-HGST_HUS724030ALA640_PN2234P9G6G70U - - 0 0 0 0 logs - - - - - - nvme0n1p1
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
And top(1) output after dropping caches: top - 18:42:09 up 70 days, 18:18, 3 users, load average: 17.33, 22.19, 27.87 Tasks: 826 total, 2 running, 644 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.2 us, 22.9 sy, 0.0 ni, 63.3 id, 11.5 wa, 0.0 hi, 1.1 si, 0.0 st GiB Mem : 125.805 total, 32.176 free, 91.669 used,1.960 buff/cache GiB Swap:5.694 total,5.688 free,0.007 used. 33.172 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 13548 sarnold 20 0 9231484 8.695g 3836 S 340.8 6.9 96:59.86 rg 2 root 20 0 0 0 0 S 23.7 0.0 38:06.54 kthreadd 1523 root 20 0 0 0 0 S 16.4 0.0 419:11.17 arc_reclaim 8491 root 0 -20 0 0 0 R 14.8 0.0 5:56.88 z_rd_int_0 8494 root 0 -20 0 0 0 S 14.8 0.0 5:56.75 z_rd_int_3 8489 root 0 -20 0 0 0 S 14.5 0.0 6:52.34 z_null_int 8492 root 0 -20 0 0 0 S 14.5 0.0 5:56.99 z_rd_int_1 8498 root 0 -20 0 0 0 S 14.5 0.0 5:56.66 z_rd_int_6 8499 root 0 -20 0 0 0 S 14.5 0.0 5:56.71 z_rd_int_7 8493 root 0 -20 0 0 0 S 14.1 0.0 5:56.58 z_rd_int_2 8495 root 0 -20 0 0 0 S 14.1 0.0 5:56.75 z_rd_int_4 8497 root 0 -20 0 0 0 S 14.1 0.0 5:56.61 z_rd_int_5 1271 root 0 -20 0 0 0 D 11.2 0.0 18:28.48 spl_dynamic_tas 1531 root 39 19 0 0 0 S 10.5 0.0 21:26.74 dbuf_evict 1522 root 20 0 0 0 0 S 8.2 0.0 228:11.90 arc_prune 3790 root 0 -204520 1648 1556 S 6.6 0.0 13:23.43 atopacctd 6457 mirror21 1 70928 30628 2976 D 5.6 0.0 0:44.18 rsync 30826 root 20 0 0 0 0 S 3.0 0.0 0:00.09 arc_prune 1272 root 0 -20 0 0 0 S 2.0 0.0 3:39.48 spl_kmem_cache 15868 root 20 0 0 0 0 S 1.0 0.0 0:00.03 arc_prune 16231 root 20 0 0 0 0 S 1.0 0.0 0:00.03 arc_prune 533 root 0 -20 0 0 0 I 0.7 0.0 1:38.82 kworker/3:1H 601 root 0 -20 0 0 0 I 0.7 0.0 1:37.84 kworker/21:1H 1530 root 20 0 0 0 0 S 0.7 0.0 7:11.37 dbu_evict 2116 root 20 0 0 0 0 I 0.7 0.0 0:18.64 kworker/1:1 4489 root 20 09916848664 S 0.7 0.0 6:18.54 rngd 7302 root 20 0 0 0 0 I 0.7 0.0 0:16.69 kworker/5:1 8952 root 20 0 0 0 0 I 0.7 0.0 0:22.74 kworker/2:2 12217 root 20 0 0 0 0 I 0.7 0.0 0:21.09 kworker/27:2 12258 sarnold 20 0 42620 4568 3256 R 0.7 0.0 0:00.42 top 16346 root 20 0 0 0 0 S 0.7 0.0 0:00.02 arc_prune 16394 root 20 0 0 0 0 S 0.7 0.0 0:00.02 arc_prune -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1814983] Re: zfs poor sustained read performance from ssd pool
sudo perf top output: Samples: 1M of event 'cycles:ppp', Event count (approx.): 476944550835 Overhead Shared Object Symbol 24.93% [kernel] [k] _raw_spin_lock 21.52% [kernel] [k] zfs_prune 17.29% [kernel] [k] d_prune_aliases 9.48% [kernel] [k] _atomic_dec_and_lock 9.20% [kernel] [k] iput 7.95% [kernel] [k] igrab 0.75% [kernel] [k] arc_evict_state 0.50% [kernel] [k] clear_page_erms 0.48% [kernel] [k] mutex_unlock 0.39% [kernel] [k] arc_space_consume 0.36% [kernel] [k] hdr_full_cons 0.36% [kernel] [k] arc_space_return 0.34% [kernel] [k] mutex_lock 0.28% [kernel] [k] kmem_cache_free 0.27% [kernel] [k] spl_kmem_cache_alloc 0.24% [kernel] [k] __cv_destroy 0.22% [kernel] [k] multilist_sublist_unlock 0.22% [kernel] [k] _raw_spin_lock_irqsave 0.20% [kernel] [k] _cond_resched 0.20% [kernel] [k] kmem_cache_alloc 0.17% [kernel] [k] multilist_sublist_lock 0.14% [kernel] [k] rcu_all_qs 0.14% [kernel] [k] osq_lock 0.12% [kernel] [k] spl_kmem_cache_free 0.12% [kernel] [k] __indirect_thunk_start 0.12% [kernel] [k] native_queued_spin_lock_slowpath 0.11% [kernel] [k] __init_waitqueue_head 0.11% [kernel] [k] arc_prune_async 0.11% [kernel] [k] __cv_init 0.09% [kernel] [k] __schedule -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1814983 Title: zfs poor sustained read performance from ssd pool To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1814983/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
