Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Sagi Grimberg
FYI, here is the latest revised patch: https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.6&id=a5b835282422ec41991c1dbdb88daa4af7d166d2 (revised patch header and fixed a thinko in the dm.c:rq_completed() change from the RFC patch I posted earlier) Hi Mike,

[dm-devel] [dm-4.6 PATCH v2 07/15] dm: add 'blk_mq_nr_hw_queues' and 'blk_mq_queue_depth' module params

2016-02-07 Thread Mike Snitzer
Allow user to change these values via module params or sysfs. 'blk_mq_nr_hw_queues' defaults to 1 (max 32). 'blk_mq_queue_depth' defaults to 2048 (up from 64, which proved far too small under moderate sized workloads -- the dm-multipath device would continuously block waiting for tags (requests)

[dm-devel] [dm-4.6 PATCH v2 02/15] dm: remove unused dm_get_rq_mapinfo()

2016-02-07 Thread Mike Snitzer
Signed-off-by: Mike Snitzer --- drivers/md/dm.c | 8 1 file changed, 8 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 846e1bb..873512d 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -106,14 +106,6 @@ struct dm_rq_clone_bio_info { struct bio clone; };

[dm-devel] [dm-4.6 PATCH v2 12/15] dm mpath: cleanup 'struct dm_mpath_io' management code

2016-02-07 Thread Mike Snitzer
Refactor and rename existing interfaces to be more specific and self-documenting. Signed-off-by: Mike Snitzer --- drivers/md/dm-mpath.c | 27 --- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c index 021ea39..6

[dm-devel] [dm-4.6 PATCH v2 08/15] dm: allocate blk_mq_tag_set rather than embed in mapped_device

2016-02-07 Thread Mike Snitzer
The blk_mq_tag_set is only needed for dm-mq support. There is point wasting space in 'struct mapped_device' for non-dm-mq devices. Signed-off-by: Mike Snitzer --- drivers/md/dm.c | 42 -- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/dri

[dm-devel] [dm-4.6 PATCH v2 05/15] dm: optimize dm_mq_queue_rq()

2016-02-07 Thread Mike Snitzer
DM multipath is the only dm-mq target. But that aside, request-based DM only supports tables with a single target that is immutable. Leverage this fact in dm_mq_queue_rq() by using the 'immutable_target' stored in the mapped_device when the table was made active. This saves the need to even take

[dm-devel] [dm-4.6 PATCH v2 03/15] dm: cleanup dm_any_congested()

2016-02-07 Thread Mike Snitzer
The request-based DM support for checking queue congestion doesn't require access to the live DM table. Signed-off-by: Mike Snitzer --- drivers/md/dm.c | 17 - 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 873512d..c92e356 1

[dm-devel] [dm-4.6 PATCH v2 09/15] dm: rename target's per_bio_data_size to per_io_data_size

2016-02-07 Thread Mike Snitzer
Request-based DM will also make use of per_bio_data_size. Signed-off-by: Mike Snitzer --- drivers/md/dm-cache-target.c | 2 +- drivers/md/dm-crypt.c | 2 +- drivers/md/dm-delay.c | 2 +- drivers/md/dm-flakey.c| 2 +- drivers/md/dm-log-writes.c| 2 +- drivers/md/

[dm-devel] [dm-4.6 PATCH v2 06/15] dm: optimize dm_request_fn()

2016-02-07 Thread Mike Snitzer
DM multipath is the only request-based DM target -- which only supports tables with a single target that is immutable. Leverage this fact in dm_request_fn(). Signed-off-by: Mike Snitzer --- drivers/md/dm.c | 47 +-- 1 file changed, 17 insertions(+), 3

[dm-devel] [dm-4.6 PATCH v2 11/15] dm mpath: use blk-mq pdu for per-request 'struct dm_mpath_io'

2016-02-07 Thread Mike Snitzer
Allow the multipath target to avoid making small allocations for each 'struct dm_mpath_io' that is needed for each request. Signed-off-by: Mike Snitzer --- drivers/md/dm-mpath.c | 40 +--- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/drivers

[dm-devel] [dm-4.6 PATCH v2 13/15] dm mpath: use blk_mq_alloc_request() and blk_mq_free_request() directly

2016-02-07 Thread Mike Snitzer
There isn't any need to support both old .request_fn and blk-mq paths in the blk-mq specific portion of __multipath_map(). Call blk_mq_alloc_request() directly rather than use blk_get_request(). Similarly, call blk_mq_free_request(), rather than blk_put_request(), in multipath_release_clone(). S

[dm-devel] [dm-4.6 PATCH v2 10/15] dm: allow immutable request-based targets to use blk-mq pdu

2016-02-07 Thread Mike Snitzer
This will allow DM multipath to use a portion of the blk-mq pdu space for target data (e.g. struct dm_mpath_io). Signed-off-by: Mike Snitzer --- drivers/md/dm-ioctl.c | 2 +- drivers/md/dm.c | 45 + drivers/md/dm.h | 2 +- 3 files changed

[dm-devel] [dm-4.5 PATCH v2 01/15] dm: fix excessive dm-mq context switching

2016-02-07 Thread Mike Snitzer
Request-based DM's blk-mq support (dm-mq) was reported to be 50% slower than if an underlying null_blk device were used directly. One of the reasons for this drop in performance is that blk_insert_clone_request() was calling blk_mq_insert_request() with @async=true. This forced the use of kblockd

[dm-devel] [dm-4.6 PATCH v2 04/15] dm: set DM_TARGET_WILDCARD feature on "error" target

2016-02-07 Thread Mike Snitzer
The DM_TARGET_WILDCARD feature indicates that the "error" target may replace any target; even immutable targets. This feature will be useful to preserve the ability to replace the "multipath" target even once it is formally converted over to having the DM_TARGET_IMMUTABLE feature. Also, implicit

[dm-devel] [dm-4.6 PATCH v2 14/15] dm mpath: reduce granularity of locking in __multipath_map

2016-02-07 Thread Mike Snitzer
No need to hold m->lock after path has been selected (and 'struct multipath' state updated). Signed-off-by: Mike Snitzer --- drivers/md/dm-mpath.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c index 7986446..199d3d3 100644

[dm-devel] [PATCH v2 00/15] dm: improve request-based DM and multipath

2016-02-07 Thread Mike Snitzer
These changes have been staged in linux-dm.git's 'for-next', see: https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/log/?h=for-next The first patch is a stable@ candidate to go into 4.5-rc4+, the remainder are targetting the 4.6 merge window. Any review would be appreciated.

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Mike Snitzer
On Sun, Feb 07 2016 at 10:41am -0500, Sagi Grimberg wrote: > > >FYI, here is the latest revised patch: > >https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.6&id=a5b835282422ec41991c1dbdb88daa4af7d166d2 > > > >(revised patch header and fixed a thinko in the dm

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Sagi Grimberg
Hi Mike, So I gave your patches a go (dm-4.6) but I still don't see the improvement you reported (while I do see a minor improvement). null_blk queue_mode=2 submit_queues=24 dm_mod blk_mq_nr_hw_queues=24 blk_mq_queue_depth=4096 use_blk_mq=Y I see 620K IOPs on dm_mq vs. 1750K IOPs on raw nullb

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Sagi Grimberg
Hello Sagi, Hey Bart, Did you run your test on a NUMA system ? I did. If so, can you check with e.g. perf record -ags -e LLC-load-misses sleep 10 && perf report whether this workload triggers perhaps lock contention ? What you need to look for in the perf output is whether any functions

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Bart Van Assche
On 02/07/16 07:41, Sagi Grimberg wrote: FYI, here is the latest revised patch: https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.6&id=a5b835282422ec41991c1dbdb88daa4af7d166d2 (revised patch header and fixed a thinko in the dm.c:rq_completed() change from th

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Sagi Grimberg
If so, can you check with e.g. perf record -ags -e LLC-load-misses sleep 10 && perf report whether this workload triggers perhaps lock contention ? What you need to look for in the perf output is whether any functions occupy more than 10% CPU time. I will, thanks for the tip! The perf report

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Mike Snitzer
On Sun, Feb 07 2016 at 11:43am -0500, Sagi Grimberg wrote: > > >Hello Sagi, > > Hey Bart, > > >Did you run your test on a NUMA system ? > > I did. > > >If so, can you check with e.g. > >perf record -ags -e LLC-load-misses sleep 10 && perf report whether this > >workload triggers perhaps lock

Re: [dm-devel] [RFC PATCH] dm: fix excessive dm-mq context switching

2016-02-07 Thread Mike Snitzer
On Sun, Feb 07 2016 at 11:54am -0500, Sagi Grimberg wrote: > > >>If so, can you check with e.g. > >>perf record -ags -e LLC-load-misses sleep 10 && perf report whether this > >>workload triggers perhaps lock contention ? What you need to look for in > >>the perf output is whether any functions o