[PATCHSET v3] blk-mq scheduling framework

2016-12-14 Thread Jens Axboe
This is version 3 of the blk-mq scheduling framework. Version 2 was posted here: https://marc.info/?l=linux-block&m=148122805026762&w=2 It's fully stable. In fact I'm running it on my laptop [1]. That may or may not have been part of a dare. In any case, it's been stable on that too, and has surv

[PATCH 1/7] block: move existing elevator ops to union

2016-12-14 Thread Jens Axboe
Prep patch for adding MQ ops as well, since doing anon unions with named initializers doesn't work on older compilers. Signed-off-by: Jens Axboe --- block/blk-ioc.c | 8 +++ block/blk-merge.c| 4 ++-- block/blk.h | 10 block/cfq-iosched.c | 2 +-

[PATCH 3/7] block: move rq_ioc() to blk.h

2016-12-14 Thread Jens Axboe
We want to use it outside of blk-core.c. Signed-off-by: Jens Axboe --- block/blk-core.c | 16 block/blk.h | 16 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 61ba08c58b64..92baea07acbc 100644 ---

[PATCH 5/7] blk-mq-sched: add framework for MQ capable IO schedulers

2016-12-14 Thread Jens Axboe
Signed-off-by: Jens Axboe --- block/Makefile | 2 +- block/blk-core.c | 7 +- block/blk-exec.c | 3 +- block/blk-flush.c| 7 +- block/blk-mq-sched.c | 375 +++ block/blk-mq-sched.h | 190

[PATCH 4/7] blk-mq: export some helpers we need to the scheduling framework

2016-12-14 Thread Jens Axboe
Signed-off-by: Jens Axboe --- block/blk-mq.c | 31 +-- block/blk-mq.h | 25 + 2 files changed, 42 insertions(+), 14 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 87b7eaa1cb74..8d1cec8e25d1 100644 --- a/block/blk-mq.c +++ b/blo

[PATCH 2/7] blk-mq: make mq_ops a const pointer

2016-12-14 Thread Jens Axboe
We never change it, make that clear. Signed-off-by: Jens Axboe --- block/blk-mq.c | 2 +- include/linux/blk-mq.h | 2 +- include/linux/blkdev.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index d79fdc11b1ee..87b7eaa1cb74 100644 -

[PATCH 6/7] mq-deadline: add blk-mq adaptation of the deadline IO scheduler

2016-12-14 Thread Jens Axboe
Signed-off-by: Jens Axboe --- block/Kconfig.iosched | 6 + block/Makefile| 1 + block/mq-deadline.c | 638 ++ 3 files changed, 645 insertions(+) create mode 100644 block/mq-deadline.c diff --git a/block/Kconfig.iosched b/block/Kconfi

[PATCH 7/7] blk-mq-sched: allow setting of default IO scheduler

2016-12-14 Thread Jens Axboe
Signed-off-by: Jens Axboe --- block/Kconfig.iosched | 43 +-- block/blk-mq-sched.c| 19 +++ block/blk-mq-sched.h| 2 ++ block/blk-mq.c | 3 +++ block/elevator.c| 5 - drivers/nvme/host/pci.c | 1 + include/

Re: [RFC] block: check partition alignment

2016-12-14 Thread Damien Le Moal
Christoph, On 12/15/16 02:07, Christoph Hellwig wrote: >> To prevent partitions that are not aligned to the physical blocksize >> of a device check for the alignment in the blkpg_ioctl. > > We'd also need to reject this when reading partitions from disk, right? Only for DASD devices, no ? Logi

Re: [RFC] block: check partition alignment

2016-12-14 Thread Damien Le Moal
> sd.c ensures that the logical block size (sector size in sd.c) is a > power of 2 between 512 and 4096. So you can use: > > if (p.start & (bdev_physical_block_size(bdev) - 1)) Sorry, that was a little too short as a complete proof: sd.c ensures that the logical block size (sector size in sd.c)

Re: [RFC] block: check partition alignment

2016-12-14 Thread Damien Le Moal
Stefan, On 12/15/16 01:47, Stefan Haberland wrote: > Partitions that are not aligned to the blocksize of a device may cause > invalid I/O requests because the blocklayer cares only about alignment > within the partition when building requests on partitions. > > device > |4096|

[GIT PULL] Block IO fixes for 4.10-rc1

2016-12-14 Thread Jens Axboe
Hi Linus, A few fixes that I collected as post-merge. I was going to wait a bit with sending this out, but the O_DIRECT fix should really go in sooner rather than later. Please pull! git://git.kernel.dk/linux-block.git for-linus --

[GIT PULL] fs meta data unmap

2016-12-14 Thread Jens Axboe
Hi Linus, A series from Jan Kara, providing a more efficient way for unmapping meta data from in the buffer cache than doing it block-by-block. Provide a general helper that existing callers can use. Please pull! git://git.kernel.dk/linux-block.git for-4.10/fs-unmap

Re: [PATCH v3] blk-mq: Fix failed allocation path when mapping queues

2016-12-14 Thread Jens Axboe
On 12/14/2016 01:48 PM, Gabriel Krisman Bertazi wrote: > From: Gabriel Krisman Bertazi > > In blk_mq_map_swqueue, there is a memory optimization that frees the > tags of a queue that has gone unmapped. Later, if that hctx is remapped > after another topology change, the tags need to be reallocat

[PATCH v3] blk-mq: Fix failed allocation path when mapping queues

2016-12-14 Thread Gabriel Krisman Bertazi
From: Gabriel Krisman Bertazi In blk_mq_map_swqueue, there is a memory optimization that frees the tags of a queue that has gone unmapped. Later, if that hctx is remapped after another topology change, the tags need to be reallocated. If this allocation fails, a simple WARN_ON triggers, but the

[PATCH] dm-crypt: fix lost ioprio when queuing crypto bios from task with ioprio

2016-12-14 Thread Eric Wheeler
Since dm-crypt queues writes (and sometimes reads) to a different kernel thread (workqueue), the bios will dispatch from tasks with different io_context->ioprio settings than the submitting task, thus giving incorrect ioprio hints to the io scheduler. By assigning the ioprio to the bio before queu

Re: [Lsf-pc] [LSF/MM TOPIC] Un-addressable device memory and block/fs implications

2016-12-14 Thread Jerome Glisse
On Wed, Dec 14, 2016 at 12:13:51PM +0100, Jan Kara wrote: > On Tue 13-12-16 16:24:33, Jerome Glisse wrote: > > On Wed, Dec 14, 2016 at 08:10:41AM +1100, Dave Chinner wrote: > > > On Tue, Dec 13, 2016 at 03:31:13PM -0500, Jerome Glisse wrote: > > > > On Wed, Dec 14, 2016 at 07:15:15AM +1100, Dave Ch

Re: [RFC] block: check partition alignment

2016-12-14 Thread Christoph Hellwig
> To prevent partitions that are not aligned to the physical blocksize > of a device check for the alignment in the blkpg_ioctl. We'd also need to reject this when reading partitions from disk, right? > + /* check if partition is aligned to blocksize */ > +

[RFC] block: check partition alignment

2016-12-14 Thread Stefan Haberland
Partitions that are not aligned to the blocksize of a device may cause invalid I/O requests because the blocklayer cares only about alignment within the partition when building requests on partitions. device |4096|4096|4096| partition offset 512byte

Re: [LSF/MM TOPIC] Un-addressable device memory and block/fs implications

2016-12-14 Thread Jerome Glisse
On Wed, Dec 14, 2016 at 03:23:13PM +1100, Dave Chinner wrote: > On Tue, Dec 13, 2016 at 08:07:58PM -0500, Jerome Glisse wrote: > > On Wed, Dec 14, 2016 at 11:14:22AM +1100, Dave Chinner wrote: > > > On Tue, Dec 13, 2016 at 05:55:24PM -0500, Jerome Glisse wrote: > > > > On Wed, Dec 14, 2016 at 09:13

Re: [PATCH RESEND v2 2/2] blk-mq: Avoid memory reclaim when remapping queues

2016-12-14 Thread Jens Axboe
On 12/06/2016 08:31 AM, Gabriel Krisman Bertazi wrote: > While stressing memory and IO at the same time we changed SMT settings, > we were able to consistently trigger deadlocks in the mm system, which > froze the entire machine. > > I think that under memory stress conditions, the large allocatio

Re: [PATCH RESEND v2 1/2] blk-mq: Fix failed allocation path when mapping queues

2016-12-14 Thread Jens Axboe
On 12/06/2016 08:31 AM, Gabriel Krisman Bertazi wrote: > This should apply cleanly on top of Jen's for-next branch. Jens, not Jen. > @@ -1893,6 +1893,15 @@ static void blk_mq_map_swqueue(struct request_queue *q, > if (!cpumask_test_cpu(i, online_mask)) > contin

Re: [PATCH 5/7] blk-mq-sched: add framework for MQ capable IO schedulers

2016-12-14 Thread Jens Axboe
On 12/14/2016 03:31 AM, Bart Van Assche wrote: > On 12/13/2016 04:14 PM, Jens Axboe wrote: >> On 12/13/2016 06:56 AM, Bart Van Assche wrote: >>> On 12/08/2016 09:13 PM, Jens Axboe wrote: +struct request *blk_mq_sched_alloc_shadow_request(struct request_queue *q, +

Re: [PATCH 6/7] mq-deadline: add blk-mq adaptation of the deadline IO scheduler

2016-12-14 Thread Jens Axboe
On 12/14/2016 01:09 AM, Bart Van Assche wrote: > On 12/08/2016 09:13 PM, Jens Axboe wrote: >> +static inline bool dd_rq_is_shadow(struct request *rq) >> +{ >> +return rq->rq_flags & RQF_ALLOCED; >> +} > > Hello Jens, > > Something minor: because req_flags_t has been defined using __bitwise >

Re: [PATCH RESEND v2 1/2] blk-mq: Fix failed allocation path when mapping queues

2016-12-14 Thread Douglas Miller
On 12/13/2016 06:39 PM, Gabriel Krisman Bertazi wrote: On 12/06/2016 09:31 AM, Gabriel Krisman Bertazi wrote: In blk_mq_map_swqueue, there is a memory optimization that frees the tags of a queue that has gone unmapped. Later, if that hctx is remapped after another topology change, the tags need

Re: [Lsf-pc] [LSF/MM TOPIC] Un-addressable device memory and block/fs implications

2016-12-14 Thread Jan Kara
On Tue 13-12-16 16:24:33, Jerome Glisse wrote: > On Wed, Dec 14, 2016 at 08:10:41AM +1100, Dave Chinner wrote: > > On Tue, Dec 13, 2016 at 03:31:13PM -0500, Jerome Glisse wrote: > > > On Wed, Dec 14, 2016 at 07:15:15AM +1100, Dave Chinner wrote: > > > > On Tue, Dec 13, 2016 at 01:15:11PM -0500, Jer

Re: [PATCH 6/6] media/cobalt: use pci_irq_allocate_vectors

2016-12-14 Thread Hans Verkuil
On 14/12/16 11:47, Christoph Hellwig wrote: On Wed, Dec 14, 2016 at 11:37:17AM +0100, Hans Verkuil wrote: Completely forgot this. Is it OK to queue it for 4.11? Or is it blocking other follow-up work you want to do for 4.10? My plan was to see if Bjorn would take the patch to do the trivial re

Re: [PATCH 6/6] media/cobalt: use pci_irq_allocate_vectors

2016-12-14 Thread Christoph Hellwig
On Wed, Dec 14, 2016 at 11:37:17AM +0100, Hans Verkuil wrote: > Completely forgot this. Is it OK to queue it for 4.11? Or is it blocking > other follow-up work you want to do for 4.10? My plan was to see if Bjorn would take the patch to do the trivial removal of pci_enable_msix_exact and pci_enabl

Re: [PATCH] block_dev: don't update file access position for sync direct IO

2016-12-14 Thread Christoph Hellwig
On Tue, Dec 13, 2016 at 09:08:18PM -0700, Jens Axboe wrote: > That's not great... Thanks, added. Ooops, yeah - we're still going through ->direct_IO for block devices. I'll take a stab at removing that, as it's just a pointless indirect call. -- To unsubscribe from this list: send the line "unsubs

Re: [PATCH 6/6] media/cobalt: use pci_irq_allocate_vectors

2016-12-14 Thread Hans Verkuil
On 14/12/16 11:29, Christoph Hellwig wrote: Hi Hans, just checked the current Linux tree and cobalt still uses the old pci_enable_msi_range call. Did you queue this patch up for 4.10? -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...

Re: [PATCH 5/7] blk-mq-sched: add framework for MQ capable IO schedulers

2016-12-14 Thread Bart Van Assche
On 12/13/2016 04:14 PM, Jens Axboe wrote: > On 12/13/2016 06:56 AM, Bart Van Assche wrote: >> On 12/08/2016 09:13 PM, Jens Axboe wrote: >>> +struct request *blk_mq_sched_alloc_shadow_request(struct request_queue *q, >>> + struct blk_mq_alloc_data >>> *da

Re: [PATCH 6/6] media/cobalt: use pci_irq_allocate_vectors

2016-12-14 Thread Christoph Hellwig
Hi Hans, just checked the current Linux tree and cobalt still uses the old pci_enable_msi_range call. Did you queue this patch up for 4.10? -- To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to majord...@vger.kernel.org More majordomo info at http:

Re: [PATCH 6/7] mq-deadline: add blk-mq adaptation of the deadline IO scheduler

2016-12-14 Thread Bart Van Assche
On 12/08/2016 09:13 PM, Jens Axboe wrote: > +static inline bool dd_rq_is_shadow(struct request *rq) > +{ > + return rq->rq_flags & RQF_ALLOCED; > +} Hello Jens, Something minor: because req_flags_t has been defined using __bitwise (typedef __u32 __bitwise req_flags_t) sparse complains for the