Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 07 feb 2018, alle ore 16:50, Oleksandr Natalenko > ha scritto: > > Hi. > > 07.02.2018 12:27, Paolo Valente wrote: >> Hi Oleksandr, Holger, >> before I prepare a V2 candidate patch, could you please test my >> instrumentation patch too, with the above

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 07 feb 2018, alle ore 16:50, Oleksandr Natalenko > ha scritto: > > Hi. > > 07.02.2018 12:27, Paolo Valente wrote: >> Hi Oleksandr, Holger, >> before I prepare a V2 candidate patch, could you please test my >> instrumentation patch too, with the above change made. For your >>

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 07.02.2018 12:27, Paolo Valente wrote: Hi Oleksandr, Holger, before I prepare a V2 candidate patch, could you please test my instrumentation patch too, with the above change made. For your convenience, I have attached a compressed archive with both the instrumentation patch and a patch

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 07.02.2018 12:27, Paolo Valente wrote: Hi Oleksandr, Holger, before I prepare a V2 candidate patch, could you please test my instrumentation patch too, with the above change made. For your convenience, I have attached a compressed archive with both the instrumentation patch and a patch

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 12:12 +0100, Paolo Valente wrote: > Just to be certain, before submitting a new patch: you changed *only* > the BUG_ON at line 4742, on top of my instrumentation patch. Nah, I completely rewrite it with only a little help from an ouija board to compensate for missing (all)

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 12:12 +0100, Paolo Valente wrote: > Just to be certain, before submitting a new patch: you changed *only* > the BUG_ON at line 4742, on top of my instrumentation patch. Nah, I completely rewrite it with only a little help from an ouija board to compensate for missing (all)

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 07 feb 2018, alle ore 12:03, Mike Galbraith ha > scritto: > > On Wed, 2018-02-07 at 11:27 +0100, Paolo Valente wrote: >> >> 2. Could you please turn that BUG_ON into: >> if (!(rq->rq_flags & RQF_ELVPRIV)) >> return; >> and see what happens? > > That seems to

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 07 feb 2018, alle ore 12:03, Mike Galbraith ha > scritto: > > On Wed, 2018-02-07 at 11:27 +0100, Paolo Valente wrote: >> >> 2. Could you please turn that BUG_ON into: >> if (!(rq->rq_flags & RQF_ELVPRIV)) >> return; >> and see what happens? > > That seems to make it forgets

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 11:27 +0100, Paolo Valente wrote: > > 2. Could you please turn that BUG_ON into: > if (!(rq->rq_flags & RQF_ELVPRIV)) > return; > and see what happens? That seems to make it forgets how to make boom. -Mike

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 11:27 +0100, Paolo Valente wrote: > > 2. Could you please turn that BUG_ON into: > if (!(rq->rq_flags & RQF_ELVPRIV)) > return; > and see what happens? That seems to make it forgets how to make boom. -Mike

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 11:27 +0100, Paolo Valente wrote: > > 1. Could you paste a stack trace for this OOPS, just to understand how we > get there? [ 442.421058] kernel BUG at block/bfq-iosched.c:4742! [ 442.421762] invalid opcode: [#1] SMP PTI [ 442.422436] Dumping ftrace buffer: [

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 11:27 +0100, Paolo Valente wrote: > > 1. Could you paste a stack trace for this OOPS, just to understand how we > get there? [ 442.421058] kernel BUG at block/bfq-iosched.c:4742! [ 442.421762] invalid opcode: [#1] SMP PTI [ 442.422436] Dumping ftrace buffer: [

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 07 feb 2018, alle ore 11:15, Mike Galbraith ha > scritto: > > On Wed, 2018-02-07 at 10:45 +0100, Paolo Valente wrote: >> >>> Il giorno 07 feb 2018, alle ore 10:23, Mike Galbraith ha >>> scritto: >>> >>> On Wed, 2018-02-07 at 10:08 +0100, Paolo

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 07 feb 2018, alle ore 11:15, Mike Galbraith ha > scritto: > > On Wed, 2018-02-07 at 10:45 +0100, Paolo Valente wrote: >> >>> Il giorno 07 feb 2018, alle ore 10:23, Mike Galbraith ha >>> scritto: >>> >>> On Wed, 2018-02-07 at 10:08 +0100, Paolo Valente wrote: The

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 10:45 +0100, Paolo Valente wrote: > > > Il giorno 07 feb 2018, alle ore 10:23, Mike Galbraith ha > > scritto: > > > > On Wed, 2018-02-07 at 10:08 +0100, Paolo Valente wrote: > >> > >> The first piece of information I need is whether this failure happens >

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 10:45 +0100, Paolo Valente wrote: > > > Il giorno 07 feb 2018, alle ore 10:23, Mike Galbraith ha > > scritto: > > > > On Wed, 2018-02-07 at 10:08 +0100, Paolo Valente wrote: > >> > >> The first piece of information I need is whether this failure happens > >> even without

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 10:08 +0100, Paolo Valente wrote: > > The first piece of information I need is whether this failure happens > even without "BFQ hierarchical scheduling support". I presume you mean BFQ_GROUP_IOSCHED, which I do not have enabled. -Mike 

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Wed, 2018-02-07 at 10:08 +0100, Paolo Valente wrote: > > The first piece of information I need is whether this failure happens > even without "BFQ hierarchical scheduling support". I presume you mean BFQ_GROUP_IOSCHED, which I do not have enabled. -Mike 

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 19:35, Oleksandr Natalenko > ha scritto: > > Hi. > > 06.02.2018 15:50, Paolo Valente wrote: >> Could you please do a >> gdb /block/bfq-iosched.o # or vmlinux.o if bfq is builtin >> list *(bfq_finish_requeue_request+0x54) >> list

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 19:35, Oleksandr Natalenko > ha scritto: > > Hi. > > 06.02.2018 15:50, Paolo Valente wrote: >> Could you please do a >> gdb /block/bfq-iosched.o # or vmlinux.o if bfq is builtin >> list *(bfq_finish_requeue_request+0x54) >> list *(bfq_put_queue+0x10b) >> for

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 15:50, Paolo Valente wrote: Could you please do a gdb /block/bfq-iosched.o # or vmlinux.o if bfq is builtin list *(bfq_finish_requeue_request+0x54) list *(bfq_put_queue+0x10b) for me? Fresh crashes and gdb output are given below. A side note: it is harder to trigger things on

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 15:50, Paolo Valente wrote: Could you please do a gdb /block/bfq-iosched.o # or vmlinux.o if bfq is builtin list *(bfq_finish_requeue_request+0x54) list *(bfq_put_queue+0x10b) for me? Fresh crashes and gdb output are given below. A side note: it is harder to trigger things on

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On 02/06/18 15:55, Paolo Valente wrote: > > >> Il giorno 06 feb 2018, alle ore 14:40, Holger Hoffstätte >> ha scritto: >> >> >> The plot thickens! >> > > Yep, the culprit seems clearer, though ... > >> Just as I was about to post that I didn't have any problems

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On 02/06/18 15:55, Paolo Valente wrote: > > >> Il giorno 06 feb 2018, alle ore 14:40, Holger Hoffstätte >> ha scritto: >> >> >> The plot thickens! >> > > Yep, the culprit seems clearer, though ... > >> Just as I was about to post that I didn't have any problems - because >> I didn't have any

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

06.02.2018 15:50, Paolo Valente wrote: Could you please do a gdb /block/bfq-iosched.o # or vmlinux.o if bfq is builtin list *(bfq_finish_requeue_request+0x54) list *(bfq_put_queue+0x10b) for me? Yes. Just give me some time to recompile the kernel with minimal debug info enabled. I'll post

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

06.02.2018 15:50, Paolo Valente wrote: Could you please do a gdb /block/bfq-iosched.o # or vmlinux.o if bfq is builtin list *(bfq_finish_requeue_request+0x54) list *(bfq_put_queue+0x10b) for me? Yes. Just give me some time to recompile the kernel with minimal debug info enabled. I'll post

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 14:40, Holger Hoffstätte > ha scritto: > > > The plot thickens! > Yep, the culprit seems clearer, though ... > Just as I was about to post that I didn't have any problems - because > I didn't have any - I decided to do a

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 14:40, Holger Hoffstätte > ha scritto: > > > The plot thickens! > Yep, the culprit seems clearer, though ... > Just as I was about to post that I didn't have any problems - because > I didn't have any - I decided to do a second test, activated bfq on my >

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 15:07, Oleksandr Natalenko > ha scritto: > > Hi. > > 06.02.2018 14:46, Mike Galbraith wrote: >>> Sorry for the noise, but just to make it clear, are we talking about >>> "deadline" or "mq-deadline" now? >> mq-deadline. > > Okay,

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 15:07, Oleksandr Natalenko > ha scritto: > > Hi. > > 06.02.2018 14:46, Mike Galbraith wrote: >>> Sorry for the noise, but just to make it clear, are we talking about >>> "deadline" or "mq-deadline" now? >> mq-deadline. > > Okay, I've spent a little bit more

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 13:43 +0100, Holger Hoffstätte wrote: > > A much more interesting question to me is why there is kyber in the middle. :) Yeah, given per sysfs I have zero devices using kyber. -Mike

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 13:43 +0100, Holger Hoffstätte wrote: > > A much more interesting question to me is why there is kyber in the middle. :) Yeah, given per sysfs I have zero devices using kyber. -Mike

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 14:46, Mike Galbraith wrote: Sorry for the noise, but just to make it clear, are we talking about "deadline" or "mq-deadline" now? mq-deadline. Okay, I've spent a little bit more time on stressing the VM with BFQ + this patch enabled, and managed to get it crashed relatively

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 14:46, Mike Galbraith wrote: Sorry for the noise, but just to make it clear, are we talking about "deadline" or "mq-deadline" now? mq-deadline. Okay, I've spent a little bit more time on stressing the VM with BFQ + this patch enabled, and managed to get it crashed relatively

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 13:26 +0100, Paolo Valente wrote: > > ok, right in the middle of bfq this time ... Was this the first OOPS in your > kernel log? Yeah.

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 13:26 +0100, Paolo Valente wrote: > > ok, right in the middle of bfq this time ... Was this the first OOPS in your > kernel log? Yeah.

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 13:16 +0100, Oleksandr Natalenko wrote: > Hi. > > 06.02.2018 12:57, Mike Galbraith wrote: > > Not me.  Box seems to be fairly sure that it is bfq. Twice again box > > went belly up on me in fairly short order with bfq, but seemed fine > > with deadline. I'm currently

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 13:16 +0100, Oleksandr Natalenko wrote: > Hi. > > 06.02.2018 12:57, Mike Galbraith wrote: > > Not me.  Box seems to be fairly sure that it is bfq. Twice again box > > went belly up on me in fairly short order with bfq, but seemed fine > > with deadline. I'm currently

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

The plot thickens! Just as I was about to post that I didn't have any problems - because I didn't have any - I decided to do a second test, activated bfq on my workstation, on a hunch typed "sync" and .. the machine locked up, hard. Rebooted, activated bfq, typed sync..sync hangs. Luckily this

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

The plot thickens! Just as I was about to post that I didn't have any problems - because I didn't have any - I decided to do a second test, activated bfq on my workstation, on a hunch typed "sync" and .. the machine locked up, hard. Rebooted, activated bfq, typed sync..sync hangs. Luckily this

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On 02/06/18 13:26, Paolo Valente wrote: (..) > As Oleksadr asked too, is it deadline or mq-deadline? You can use deadline as alias as long as blk-mq is active. This doesn't work when mq-deadline is built as a module, but that doesn't seem to be the problem here. >> [ 484.179292] BUG: unable to

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On 02/06/18 13:26, Paolo Valente wrote: (..) > As Oleksadr asked too, is it deadline or mq-deadline? You can use deadline as alias as long as blk-mq is active. This doesn't work when mq-deadline is built as a module, but that doesn't seem to be the problem here. >> [ 484.179292] BUG: unable to

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 13:26, Paolo Valente > ha scritto: > > > >> Il giorno 06 feb 2018, alle ore 12:57, Mike Galbraith ha >> scritto: >> >> On Tue, 2018-02-06 at 10:38 +0100, Paolo Valente wrote: >>> >>> Hi Mike, >>> as you can

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 13:26, Paolo Valente > ha scritto: > > > >> Il giorno 06 feb 2018, alle ore 12:57, Mike Galbraith ha >> scritto: >> >> On Tue, 2018-02-06 at 10:38 +0100, Paolo Valente wrote: >>> >>> Hi Mike, >>> as you can imagine, I didn't get any failure in my

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 12:57, Mike Galbraith ha > scritto: > > On Tue, 2018-02-06 at 10:38 +0100, Paolo Valente wrote: >> >> Hi Mike, >> as you can imagine, I didn't get any failure in my pre-submission >> tests on this patch. In addition, it is not that easy to

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 12:57, Mike Galbraith ha > scritto: > > On Tue, 2018-02-06 at 10:38 +0100, Paolo Valente wrote: >> >> Hi Mike, >> as you can imagine, I didn't get any failure in my pre-submission >> tests on this patch. In addition, it is not that easy to link this >>

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 12:57, Mike Galbraith wrote: Not me.  Box seems to be fairly sure that it is bfq. Twice again box went belly up on me in fairly short order with bfq, but seemed fine with deadline. I'm currently running deadline again, and box again seems solid, thought I won't say _is_ solid

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 12:57, Mike Galbraith wrote: Not me.  Box seems to be fairly sure that it is bfq. Twice again box went belly up on me in fairly short order with bfq, but seemed fine with deadline. I'm currently running deadline again, and box again seems solid, thought I won't say _is_ solid

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 10:38 +0100, Paolo Valente wrote: > > Hi Mike, > as you can imagine, I didn't get any failure in my pre-submission > tests on this patch. In addition, it is not that easy to link this > patch, which just adds some internal bfq housekeeping in case of a > requeue, with a

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 10:38 +0100, Paolo Valente wrote: > > Hi Mike, > as you can imagine, I didn't get any failure in my pre-submission > tests on this patch. In addition, it is not that easy to link this > patch, which just adds some internal bfq housekeeping in case of a > requeue, with a

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 08:56, Mike Galbraith ha > scritto: > > On Tue, 2018-02-06 at 08:44 +0100, Oleksandr Natalenko wrote: >> Hi, Paolo. >> >> I can confirm that this patch fixes cfdisk hang for me. I've also tried >> to trigger the issue Mike has encountered,

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

> Il giorno 06 feb 2018, alle ore 08:56, Mike Galbraith ha > scritto: > > On Tue, 2018-02-06 at 08:44 +0100, Oleksandr Natalenko wrote: >> Hi, Paolo. >> >> I can confirm that this patch fixes cfdisk hang for me. I've also tried >> to trigger the issue Mike has encountered, but with no luck

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 09:37 +0100, Oleksandr Natalenko wrote: > Hi. > > 06.02.2018 08:56, Mike Galbraith wrote: > > I was doing kbuilds, and it blew up on me twice. Switching back to cfq > > seemed to confirm it was indeed the patch causing trouble, but that's > > by no means a certainty. > >

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 09:37 +0100, Oleksandr Natalenko wrote: > Hi. > > 06.02.2018 08:56, Mike Galbraith wrote: > > I was doing kbuilds, and it blew up on me twice. Switching back to cfq > > seemed to confirm it was indeed the patch causing trouble, but that's > > by no means a certainty. > >

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 08:56, Mike Galbraith wrote: I was doing kbuilds, and it blew up on me twice. Switching back to cfq seemed to confirm it was indeed the patch causing trouble, but that's by no means a certainty. Just to note, I was using v4.15.1, not the latest git HEAD. Are you able to

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi. 06.02.2018 08:56, Mike Galbraith wrote: I was doing kbuilds, and it blew up on me twice. Switching back to cfq seemed to confirm it was indeed the patch causing trouble, but that's by no means a certainty. Just to note, I was using v4.15.1, not the latest git HEAD. Are you able to

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 08:44 +0100, Oleksandr Natalenko wrote: > Hi, Paolo. > > I can confirm that this patch fixes cfdisk hang for me. I've also tried > to trigger the issue Mike has encountered, but with no luck (maybe, I > wasn't insistent enough, just was doing dd on usb-storage device in

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

On Tue, 2018-02-06 at 08:44 +0100, Oleksandr Natalenko wrote: > Hi, Paolo. > > I can confirm that this patch fixes cfdisk hang for me. I've also tried > to trigger the issue Mike has encountered, but with no luck (maybe, I > wasn't insistent enough, just was doing dd on usb-storage device in

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi, Paolo. I can confirm that this patch fixes cfdisk hang for me. I've also tried to trigger the issue Mike has encountered, but with no luck (maybe, I wasn't insistent enough, just was doing dd on usb-storage device in the VM). So, with regard to cfdisk hang on usb-storage: Tested-by:

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi, Paolo. I can confirm that this patch fixes cfdisk hang for me. I've also tried to trigger the issue Mike has encountered, but with no luck (maybe, I wasn't insistent enough, just was doing dd on usb-storage device in the VM). So, with regard to cfdisk hang on usb-storage: Tested-by:

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi Paolo, I applied this to master.today, flipped udev back to bfq and took it for a spin.  Unfortunately, box fairly quickly went boom under load. [ 454.739975] [ cut here ] [ 454.739979] list_add corruption. prev->next should be next (5f99a42a), but was

Re: [PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Hi Paolo, I applied this to master.today, flipped udev back to bfq and took it for a spin.  Unfortunately, box fairly quickly went boom under load. [ 454.739975] [ cut here ] [ 454.739979] list_add corruption. prev->next should be next (5f99a42a), but was

[PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Commit 'a6a252e64914 ("blk-mq-sched: decide how to handle flush rq via RQF_FLUSH_SEQ")' makes all non-flush re-prepared requests for a device be re-inserted into the active I/O scheduler for that device. As a consequence, I/O schedulers may get the same request inserted again, even several times,

[PATCH BUGFIX 1/1] block, bfq: add requeue-request hook

Commit 'a6a252e64914 ("blk-mq-sched: decide how to handle flush rq via RQF_FLUSH_SEQ")' makes all non-flush re-prepared requests for a device be re-inserted into the active I/O scheduler for that device. As a consequence, I/O schedulers may get the same request inserted again, even several times,