Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-19 Thread Sinan Kaya
On 7/19/2017 5:20 AM, Sagi Grimberg wrote: >> Code is moving the completion queue doorbell after processing all completed >> events and sending callbacks to the block layer on each iteration. >> >> This is causing a performance drop when a lot of jobs are queued towards >> the HW. Move the

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-19 Thread Sinan Kaya
On 7/19/2017 5:20 AM, Sagi Grimberg wrote: >> Code is moving the completion queue doorbell after processing all completed >> events and sending callbacks to the block layer on each iteration. >> >> This is causing a performance drop when a lot of jobs are queued towards >> the HW. Move the

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-19 Thread Sagi Grimberg
Code is moving the completion queue doorbell after processing all completed events and sending callbacks to the block layer on each iteration. This is causing a performance drop when a lot of jobs are queued towards the HW. Move the completion queue doorbell on each loop instead and allow new

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-19 Thread Sagi Grimberg
Code is moving the completion queue doorbell after processing all completed events and sending callbacks to the block layer on each iteration. This is causing a performance drop when a lot of jobs are queued towards the HW. Move the completion queue doorbell on each loop instead and allow new

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-18 Thread Keith Busch
On Tue, Jul 18, 2017 at 02:52:26PM -0400, Sinan Kaya wrote: > On 7/18/2017 10:36 AM, Keith Busch wrote: > > I do see that the NVMe driver is creating a completion interrupt on > each CPU core for the completions. No problems with that. > > However, I don't think you can guarantee that there

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-18 Thread Keith Busch
On Tue, Jul 18, 2017 at 02:52:26PM -0400, Sinan Kaya wrote: > On 7/18/2017 10:36 AM, Keith Busch wrote: > > I do see that the NVMe driver is creating a completion interrupt on > each CPU core for the completions. No problems with that. > > However, I don't think you can guarantee that there

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-18 Thread Sinan Kaya
On 7/18/2017 10:36 AM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 07:07:00PM -0400, ok...@codeaurora.org wrote: >> Maybe, I need to understand the design better. I was curious why completion >> and submission queues were protected by a single lock causing lock >> contention. > Ideally the queues

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-18 Thread Sinan Kaya
On 7/18/2017 10:36 AM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 07:07:00PM -0400, ok...@codeaurora.org wrote: >> Maybe, I need to understand the design better. I was curious why completion >> and submission queues were protected by a single lock causing lock >> contention. > Ideally the queues

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-18 Thread Keith Busch
On Mon, Jul 17, 2017 at 07:07:00PM -0400, ok...@codeaurora.org wrote: > Maybe, I need to understand the design better. I was curious why completion > and submission queues were protected by a single lock causing lock > contention. Ideally the queues are tied to CPUs, so you couldn't have one

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-18 Thread Keith Busch
On Mon, Jul 17, 2017 at 07:07:00PM -0400, ok...@codeaurora.org wrote: > Maybe, I need to understand the design better. I was curious why completion > and submission queues were protected by a single lock causing lock > contention. Ideally the queues are tied to CPUs, so you couldn't have one

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread okaya
On 2017-07-17 18:56, Keith Busch wrote: On Mon, Jul 17, 2017 at 06:46:11PM -0400, Sinan Kaya wrote: Hi Keith, On 7/17/2017 6:45 PM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: >> Code is moving the completion queue doorbell after processing all completed >>

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread okaya
On 2017-07-17 18:56, Keith Busch wrote: On Mon, Jul 17, 2017 at 06:46:11PM -0400, Sinan Kaya wrote: Hi Keith, On 7/17/2017 6:45 PM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: >> Code is moving the completion queue doorbell after processing all completed >>

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Keith Busch
On Mon, Jul 17, 2017 at 06:46:11PM -0400, Sinan Kaya wrote: > Hi Keith, > > On 7/17/2017 6:45 PM, Keith Busch wrote: > > On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: > >> Code is moving the completion queue doorbell after processing all completed > >> events and sending callbacks

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Keith Busch
On Mon, Jul 17, 2017 at 06:46:11PM -0400, Sinan Kaya wrote: > Hi Keith, > > On 7/17/2017 6:45 PM, Keith Busch wrote: > > On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: > >> Code is moving the completion queue doorbell after processing all completed > >> events and sending callbacks

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Sinan Kaya
Hi Keith, On 7/17/2017 6:45 PM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: >> Code is moving the completion queue doorbell after processing all completed >> events and sending callbacks to the block layer on each iteration. >> >> This is causing a performance

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Sinan Kaya
Hi Keith, On 7/17/2017 6:45 PM, Keith Busch wrote: > On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: >> Code is moving the completion queue doorbell after processing all completed >> events and sending callbacks to the block layer on each iteration. >> >> This is causing a performance

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Keith Busch
On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: > Code is moving the completion queue doorbell after processing all completed > events and sending callbacks to the block layer on each iteration. > > This is causing a performance drop when a lot of jobs are queued towards > the HW.

Re: [PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Keith Busch
On Mon, Jul 17, 2017 at 06:36:23PM -0400, Sinan Kaya wrote: > Code is moving the completion queue doorbell after processing all completed > events and sending callbacks to the block layer on each iteration. > > This is causing a performance drop when a lot of jobs are queued towards > the HW.

[PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Sinan Kaya
Code is moving the completion queue doorbell after processing all completed events and sending callbacks to the block layer on each iteration. This is causing a performance drop when a lot of jobs are queued towards the HW. Move the completion queue doorbell on each loop instead and allow new

[PATCH] nvme: Acknowledge completion queue on each iteration

2017-07-17 Thread Sinan Kaya
Code is moving the completion queue doorbell after processing all completed events and sending callbacks to the block layer on each iteration. This is causing a performance drop when a lot of jobs are queued towards the HW. Move the completion queue doorbell on each loop instead and allow new