Re: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a channel for I/O requests

[Martin K. Petersen]

Michael,

> Update the algorithm in storvsc_do_io to look for a channel starting
> with the current CPU + 1 and wrap around (within the current NUMA
> node). This spreads VMbus interrupts more evenly across CPUs. Previous
> code always started with first CPU in the current NUMA node, skewing
> the interrupt load to that CPU.

Applied to 4.16/scsi-fixes. Thanks!

-- 
Martin K. Petersen  Oracle Linux Engineering

RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a channel for I/O requests

[Long Li]

> Subject: RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a
> channel for I/O requests
> 
> > From: Long Li
> > Sent: Wednesday, January 31, 2018 12:23 PM
> > To: Michael Kelley (EOSG) <michael.h.kel...@microsoft.com>; KY
> > Srinivasan <k...@microsoft.com>; Stephen Hemminger
> > <sthem...@microsoft.com>; martin.peter...@oracle.com;
> > de...@linuxdriverproject.org; linux-ker...@vger.kernel.org;
> > linux-scsi@vger.kernel.org; James E . J . Bottomley
> > <j...@linux.vnet.ibm.com>
> > Subject: RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking
> > a channel for I/O requests
> >
> > > Subject: RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when
> > > picking a channel for I/O requests
> > >
> > > Updated/corrected two email addresses ...
> > >
> > > > -Original Message-
> > > > From: Michael Kelley (EOSG)
> > > > Sent: Wednesday, January 24, 2018 2:14 PM
> > > > To: KY Srinivasan <k...@microsoft.com>; Stephen Hemminger
> > > > <sthem...@microsoft.com>; martin.peter...@oracle.com;
> > > > lo...@microsoft.com; jbottom...@odin.com;
> > > > de...@linuxdriverproject.org; linux-ker...@vger.kernel.org;
> > > > linux-scsi@vger.kernel.org
> > > > Cc: Michael Kelley (EOSG) <michael.h.kel...@microsoft.com>
> > > > Subject: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking
> > > > a channel for I/O requests
> > > >
> > > > Update the algorithm in storvsc_do_io to look for a channel
> > > > starting with the current CPU + 1 and wrap around (within the
> > > > current NUMA node). This spreads VMbus interrupts more evenly
> > > > across CPUs. Previous code always started with first CPU in the
> > > > current NUMA node, skewing the interrupt load to that CPU.
> > > >
> > > > Signed-off-by: Michael Kelley <mikel...@microsoft.com>

Reviewed-by: Long Li <lon...@microsoft.com>

> > > > ---
> > > >  drivers/scsi/storvsc_drv.c | 3 ++-
> > > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/scsi/storvsc_drv.c
> > > > b/drivers/scsi/storvsc_drv.c index e07907d..f3264c4 100644
> > > > --- a/drivers/scsi/storvsc_drv.c
> > > > +++ b/drivers/scsi/storvsc_drv.c
> > > > @@ -1310,7 +1310,8 @@ static int storvsc_do_io(struct hv_device
> *device,
> > > >  */
> > > > cpumask_and(_mask, _device-
> alloced_cpus,
> > > >
> > > cpumask_of_node(cpu_to_node(q_num)));
> > > > -   for_each_cpu(tgt_cpu, _mask) {
> > > > +   for_each_cpu_wrap(tgt_cpu, _mask,
> > > > +   outgoing_channel->target_cpu + 
> > > > 1) {
> >
> > Does it work when target_cpu is the last CPU on the system?
> >
> > Otherwise, looking good.
> 
> Yes, it works.  for_each_cpu_wrap() correctly wraps in the case where the
> 3rd parameter ('start') is one past the end of the mask.  Arguably, we
> shouldn't rely on that, and should do the wrap to 0 before calling
> for_each_cpu_wrap().
> 
> >
> > > > if (tgt_cpu != 
> > > > outgoing_channel->target_cpu)
> > > {
> > > > outgoing_channel =
> > > > stor_device->stor_chns[tgt_cpu];
> > > > --
> > > > 1.8.3.1

RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a channel for I/O requests

[Michael Kelley (EOSG)]

> From: Long Li
> Sent: Wednesday, January 31, 2018 12:23 PM
> To: Michael Kelley (EOSG) <michael.h.kel...@microsoft.com>; KY Srinivasan
> <k...@microsoft.com>; Stephen Hemminger <sthem...@microsoft.com>;
> martin.peter...@oracle.com; de...@linuxdriverproject.org; 
> linux-ker...@vger.kernel.org;
> linux-scsi@vger.kernel.org; James E . J . Bottomley <j...@linux.vnet.ibm.com>
> Subject: RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a 
> channel for I/O
> requests
> 
> > Subject: RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a
> > channel for I/O requests
> >
> > Updated/corrected two email addresses ...
> >
> > > -Original Message-
> > > From: Michael Kelley (EOSG)
> > > Sent: Wednesday, January 24, 2018 2:14 PM
> > > To: KY Srinivasan <k...@microsoft.com>; Stephen Hemminger
> > > <sthem...@microsoft.com>; martin.peter...@oracle.com;
> > > lo...@microsoft.com; jbottom...@odin.com;
> > > de...@linuxdriverproject.org; linux-ker...@vger.kernel.org;
> > > linux-scsi@vger.kernel.org
> > > Cc: Michael Kelley (EOSG) <michael.h.kel...@microsoft.com>
> > > Subject: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a
> > > channel for I/O requests
> > >
> > > Update the algorithm in storvsc_do_io to look for a channel starting
> > > with the current CPU + 1 and wrap around (within the current NUMA
> > > node). This spreads VMbus interrupts more evenly across CPUs. Previous
> > > code always started with first CPU in the current NUMA node, skewing
> > > the interrupt load to that CPU.
> > >
> > > Signed-off-by: Michael Kelley <mikel...@microsoft.com>
> > > ---
> > >  drivers/scsi/storvsc_drv.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> > > index e07907d..f3264c4 100644
> > > --- a/drivers/scsi/storvsc_drv.c
> > > +++ b/drivers/scsi/storvsc_drv.c
> > > @@ -1310,7 +1310,8 @@ static int storvsc_do_io(struct hv_device *device,
> > >*/
> > >   cpumask_and(_mask, _device-
> > >alloced_cpus,
> > >
> > cpumask_of_node(cpu_to_node(q_num)));
> > > - for_each_cpu(tgt_cpu, _mask) {
> > > + for_each_cpu_wrap(tgt_cpu, _mask,
> > > + outgoing_channel->target_cpu + 1) {
> 
> Does it work when target_cpu is the last CPU on the system?
> 
> Otherwise, looking good.

Yes, it works.  for_each_cpu_wrap() correctly wraps in the case where
the 3rd parameter ('start') is one past the end of the mask.  Arguably,
we shouldn't rely on that, and should do the wrap to 0 before calling
for_each_cpu_wrap().

> 
> > >   if (tgt_cpu != outgoing_channel->target_cpu)
> > {
> > >   outgoing_channel =
> > >   stor_device->stor_chns[tgt_cpu];
> > > --
> > > 1.8.3.1

RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a channel for I/O requests

[Long Li]

> Subject: RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a
> channel for I/O requests
> 
> Updated/corrected two email addresses ...
> 
> > -Original Message-
> > From: Michael Kelley (EOSG)
> > Sent: Wednesday, January 24, 2018 2:14 PM
> > To: KY Srinivasan <k...@microsoft.com>; Stephen Hemminger
> > <sthem...@microsoft.com>; martin.peter...@oracle.com;
> > lo...@microsoft.com; jbottom...@odin.com;
> > de...@linuxdriverproject.org; linux-ker...@vger.kernel.org;
> > linux-scsi@vger.kernel.org
> > Cc: Michael Kelley (EOSG) <michael.h.kel...@microsoft.com>
> > Subject: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a
> > channel for I/O requests
> >
> > Update the algorithm in storvsc_do_io to look for a channel starting
> > with the current CPU + 1 and wrap around (within the current NUMA
> > node). This spreads VMbus interrupts more evenly across CPUs. Previous
> > code always started with first CPU in the current NUMA node, skewing
> > the interrupt load to that CPU.
> >
> > Signed-off-by: Michael Kelley <mikel...@microsoft.com>
> > ---
> >  drivers/scsi/storvsc_drv.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> > index e07907d..f3264c4 100644
> > --- a/drivers/scsi/storvsc_drv.c
> > +++ b/drivers/scsi/storvsc_drv.c
> > @@ -1310,7 +1310,8 @@ static int storvsc_do_io(struct hv_device *device,
> >  */
> > cpumask_and(_mask, _device-
> >alloced_cpus,
> >
> cpumask_of_node(cpu_to_node(q_num)));
> > -   for_each_cpu(tgt_cpu, _mask) {
> > +   for_each_cpu_wrap(tgt_cpu, _mask,
> > +   outgoing_channel->target_cpu + 1) {

Does it work when target_cpu is the last CPU on the system?

Otherwise, looking good.

> > if (tgt_cpu != outgoing_channel->target_cpu)
> {
> > outgoing_channel =
> > stor_device->stor_chns[tgt_cpu];
> > --
> > 1.8.3.1

RE: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a channel for I/O requests

[Michael Kelley (EOSG)]

Updated/corrected two email addresses ...

> -Original Message-
> From: Michael Kelley (EOSG)
> Sent: Wednesday, January 24, 2018 2:14 PM
> To: KY Srinivasan ; Stephen Hemminger 
> ;
> martin.peter...@oracle.com; lo...@microsoft.com; jbottom...@odin.com;
> de...@linuxdriverproject.org; linux-ker...@vger.kernel.org; 
> linux-scsi@vger.kernel.org
> Cc: Michael Kelley (EOSG) 
> Subject: [PATCH 1/1] scsi: storvsc: Spread interrupts when picking a channel 
> for I/O requests
> 
> Update the algorithm in storvsc_do_io to look for a channel
> starting with the current CPU + 1 and wrap around (within the
> current NUMA node). This spreads VMbus interrupts more evenly
> across CPUs. Previous code always started with first CPU in
> the current NUMA node, skewing the interrupt load to that CPU.
> 
> Signed-off-by: Michael Kelley 
> ---
>  drivers/scsi/storvsc_drv.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> index e07907d..f3264c4 100644
> --- a/drivers/scsi/storvsc_drv.c
> +++ b/drivers/scsi/storvsc_drv.c
> @@ -1310,7 +1310,8 @@ static int storvsc_do_io(struct hv_device *device,
>*/
>   cpumask_and(_mask, _device->alloced_cpus,
>   cpumask_of_node(cpu_to_node(q_num)));
> - for_each_cpu(tgt_cpu, _mask) {
> + for_each_cpu_wrap(tgt_cpu, _mask,
> + outgoing_channel->target_cpu + 1) {
>   if (tgt_cpu != outgoing_channel->target_cpu) {
>   outgoing_channel =
>   stor_device->stor_chns[tgt_cpu];
> --
> 1.8.3.1