The perf report is very similar to the one that started this effort..
I'm afraid we'll need to resolve the per-target m->lock in order
to scale with NUMA...
Could be. Just for testing, you can try the 2 topmost commits I've put
here (once applied both __multipath_map and multipath_busy won't have
_any_ locking.. again, very much test-only):
http://git.kernel.org/cgit/linux/kernel/git/snitzer/linux.git/log/?h=devel2
Hi Mike,
So I still don't see the IOPs scale like I expected. With these two
patches applied I see ~670K IOPs while the perf output is different
and does not indicate a clear lock contention.
--
- 4.67% fio [kernel.kallsyms] [k]
blk_account_io_start
- blk_account_io_start
- 56.05% blk_insert_cloned_request
map_request
dm_mq_queue_rq
__blk_mq_run_hw_queue
blk_mq_run_hw_queue
blk_mq_insert_requests
blk_mq_flush_plug_list
blk_flush_plug_list
blk_finish_plug
do_io_submit
SyS_io_submit
entry_SYSCALL_64_fastpath
+ io_submit
- 43.94% blk_mq_bio_to_request
blk_mq_make_request
generic_make_request
submit_bio
do_blockdev_direct_IO
__blockdev_direct_IO
blkdev_direct_IO
generic_file_read_iter
blkdev_read_iter
aio_run_iocb
io_submit_one
do_io_submit
SyS_io_submit
entry_SYSCALL_64_fastpath
+ io_submit
- 2.52% fio [dm_mod] [k] dm_mq_queue_rq
- dm_mq_queue_rq
- 99.16% __blk_mq_run_hw_queue
blk_mq_run_hw_queue
blk_mq_insert_requests
blk_mq_flush_plug_list
blk_flush_plug_list
blk_finish_plug
do_io_submit
SyS_io_submit
entry_SYSCALL_64_fastpath
+ io_submit
- 2.52% fio [dm_mod] [k] dm_mq_queue_rq
- dm_mq_queue_rq
- 99.16% __blk_mq_run_hw_queue
blk_mq_run_hw_queue
blk_mq_insert_requests
blk_mq_flush_plug_list
blk_flush_plug_list
blk_finish_plug
do_io_submit
SyS_io_submit
entry_SYSCALL_64_fastpath
+ io_submit
+ 0.84% blk_mq_run_hw_queue
- 2.46% fio [kernel.kallsyms] [k]
blk_mq_hctx_mark_pending
- blk_mq_hctx_mark_pending
- 99.79% blk_mq_insert_requests
blk_mq_flush_plug_list
blk_flush_plug_list
blk_finish_plug
do_io_submit
SyS_io_submit
entry_SYSCALL_64_fastpath
+ io_submit
- 2.07% ksoftirqd/6 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
- blk_mq_run_hw_queues
- 99.70% rq_completed
dm_done
dm_softirq_done
blk_done_softirq
+ __do_softirq
+ 2.06% ksoftirqd/0 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 2.02% ksoftirqd/9 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 2.00% ksoftirqd/20 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 2.00% ksoftirqd/12 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.99% ksoftirqd/11 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.97% ksoftirqd/18 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.96% ksoftirqd/1 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.95% ksoftirqd/14 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.95% ksoftirqd/13 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.94% ksoftirqd/5 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.94% ksoftirqd/8 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.93% ksoftirqd/2 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.92% ksoftirqd/21 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.92% ksoftirqd/17 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.92% ksoftirqd/7 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.91% ksoftirqd/23 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.84% ksoftirqd/4 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.81% ksoftirqd/19 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.76% ksoftirqd/3 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.76% ksoftirqd/16 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.75% ksoftirqd/15 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.74% ksoftirqd/22 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.72% ksoftirqd/10 [kernel.kallsyms] [k]
blk_mq_run_hw_queues
+ 1.38% perf [kernel.kallsyms] [k]
copy_user_generic_string
+ 1.20% fio [kernel.kallsyms] [k] enqueue_task_fair
+ 1.18% fio [kernel.kallsyms] [k] part_round_stats
+ 1.08% fio [kernel.kallsyms] [k] enqueue_entity
+ 1.07% fio [kernel.kallsyms] [k] _raw_spin_lock
+ 1.02% fio [kernel.kallsyms] [k]
__blk_mq_run_hw_queue
+ 0.79% fio [dm_multipath] [k] multipath_busy
+ 0.57% fio [kernel.kallsyms] [k] insert_work
+ 0.54% fio [kernel.kallsyms] [k] blk_flush_plug_list
--
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
