Re: Observing Softlockup's while running heavy IOs

2016-09-12 Thread Neil Horman
On Mon, Sep 12, 2016 at 01:48:39PM +0530, Sreekanth Reddy wrote: > On Thu, Sep 8, 2016 at 7:09 PM, Neil Horman wrote: > > On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote: > >> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: > >> >

Re: Observing Softlockup's while running heavy IOs

2016-09-12 Thread Neil Horman
On Mon, Sep 12, 2016 at 01:48:39PM +0530, Sreekanth Reddy wrote: > On Thu, Sep 8, 2016 at 7:09 PM, Neil Horman wrote: > > On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote: > >> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: > >> > On Wed, Sep 07, 2016 at 11:30:04AM +0530,

Re: Observing Softlockup's while running heavy IOs

2016-09-12 Thread Sreekanth Reddy
On Thu, Sep 8, 2016 at 7:09 PM, Neil Horman wrote: > On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote: >> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: >> > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: >> >> On

Re: Observing Softlockup's while running heavy IOs

2016-09-12 Thread Sreekanth Reddy
On Thu, Sep 8, 2016 at 7:09 PM, Neil Horman wrote: > On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote: >> On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: >> > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: >> >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman

Re: Observing Softlockup's while running heavy IOs

2016-09-08 Thread Neil Horman
On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote: > On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: > > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: > >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman wrote: > >> >

Re: Observing Softlockup's while running heavy IOs

2016-09-08 Thread Neil Horman
On Thu, Sep 08, 2016 at 11:12:40AM +0530, Sreekanth Reddy wrote: > On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: > > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: > >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman wrote: > >> > On Tue, Sep 06, 2016 at 04:52:37PM +0530,

Re: Observing Softlockup's while running heavy IOs

2016-09-08 Thread Sreekanth Reddy
On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman wrote: >> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote: >> >> On

Re: Observing Softlockup's while running heavy IOs

2016-09-08 Thread Sreekanth Reddy
On Wed, Sep 7, 2016 at 6:54 PM, Neil Horman wrote: > On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: >> On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman wrote: >> > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote: >> >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche

Re: Observing Softlockup's while running heavy IOs

2016-09-07 Thread Neil Horman
On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: > On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman wrote: > > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote: > >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche > >>

Re: Observing Softlockup's while running heavy IOs

2016-09-07 Thread Neil Horman
On Wed, Sep 07, 2016 at 11:30:04AM +0530, Sreekanth Reddy wrote: > On Tue, Sep 6, 2016 at 8:36 PM, Neil Horman wrote: > > On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote: > >> On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche > >> wrote: > >> > On 09/01/2016 03:31 AM, Sreekanth

Re: Observing Softlockup's while running heavy IOs

2016-09-06 Thread Neil Horman
On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote: > On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche > wrote: > > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote: > >> > >> I reduced the ISR workload by one third in-order to reduce the time > >> that is

Re: Observing Softlockup's while running heavy IOs

2016-09-06 Thread Neil Horman
On Tue, Sep 06, 2016 at 04:52:37PM +0530, Sreekanth Reddy wrote: > On Fri, Sep 2, 2016 at 4:34 AM, Bart Van Assche > wrote: > > On 09/01/2016 03:31 AM, Sreekanth Reddy wrote: > >> > >> I reduced the ISR workload by one third in-order to reduce the time > >> that is spent per CPU in interrupt

Re: Observing Softlockup's while running heavy IOs

2016-09-01 Thread Bart Van Assche
On 09/01/2016 03:31 AM, Sreekanth Reddy wrote: I reduced the ISR workload by one third in-order to reduce the time that is spent per CPU in interrupt context, even then I am observing softlockups. As I mentioned before only same single CPU in the set of CPUs(enabled in affinity_hint) is busy

Re: Observing Softlockup's while running heavy IOs

2016-09-01 Thread Bart Van Assche
On 09/01/2016 03:31 AM, Sreekanth Reddy wrote: I reduced the ISR workload by one third in-order to reduce the time that is spent per CPU in interrupt context, even then I am observing softlockups. As I mentioned before only same single CPU in the set of CPUs(enabled in affinity_hint) is busy

Re: Observing Softlockup's while running heavy IOs

2016-09-01 Thread Sreekanth Reddy
On Fri, Aug 19, 2016 at 9:26 PM, Bart Van Assche wrote: > On 08/19/2016 04:44 AM, Sreekanth Reddy wrote: >> >> [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms >> [ ... ] >> [ +0.05] [] ? finish_task_switch+0x6b/0x200 >> [ +0.06] []

Re: Observing Softlockup's while running heavy IOs

2016-09-01 Thread Sreekanth Reddy
On Fri, Aug 19, 2016 at 9:26 PM, Bart Van Assche wrote: > On 08/19/2016 04:44 AM, Sreekanth Reddy wrote: >> >> [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms >> [ ... ] >> [ +0.05] [] ? finish_task_switch+0x6b/0x200 >> [ +0.06] [] __schedule+0x36c/0x950 >> [ +0.02]

RE: Observing Softlockup's while running heavy IOs

2016-08-23 Thread Kashyap Desai
ai; Sathya Prakash Veerichetty; > Chaitra Basappa; Suganath Prabu Subramani > Subject: RE: Observing Softlockup's while running heavy IOs > > > > > -Original Message- > > From: Sreekanth Reddy [mailto:sreekanth.re...@broadcom.com] > > Sent: Friday, August 19, 2016

RE: Observing Softlockup's while running heavy IOs

2016-08-23 Thread Kashyap Desai
ai; Sathya Prakash Veerichetty; > Chaitra Basappa; Suganath Prabu Subramani > Subject: RE: Observing Softlockup's while running heavy IOs > > > > > -Original Message- > > From: Sreekanth Reddy [mailto:sreekanth.re...@broadcom.com] > > Sent: Friday, August 19, 2016

RE: Observing Softlockup's while running heavy IOs

2016-08-19 Thread Elliott, Robert (Persistent Memory)
> -Original Message- > From: Sreekanth Reddy [mailto:sreekanth.re...@broadcom.com] > Sent: Friday, August 19, 2016 6:45 AM > To: Elliott, Robert (Persistent Memory) <elli...@hpe.com> > Subject: Re: Observing Softlockup's while running heavy IOs > ... > Yes I

RE: Observing Softlockup's while running heavy IOs

2016-08-19 Thread Elliott, Robert (Persistent Memory)
> -Original Message- > From: Sreekanth Reddy [mailto:sreekanth.re...@broadcom.com] > Sent: Friday, August 19, 2016 6:45 AM > To: Elliott, Robert (Persistent Memory) > Subject: Re: Observing Softlockup's while running heavy IOs > ... > Yes I am also observing th

Re: Observing Softlockup's while running heavy IOs

2016-08-19 Thread Bart Van Assche
On 08/19/2016 04:44 AM, Sreekanth Reddy wrote: [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms [ ... ] [ +0.05] [] ? finish_task_switch+0x6b/0x200 [ +0.06] [] __schedule+0x36c/0x950 [ +0.02] [] schedule+0x37/0x80 [ +0.06] [] futex_wait_queue_me+0xbc/0x120 [

Re: Observing Softlockup's while running heavy IOs

2016-08-19 Thread Bart Van Assche
On 08/19/2016 04:44 AM, Sreekanth Reddy wrote: [ +0.000439] __blk_mq_run_hw_queue() finished after 10058 ms [ ... ] [ +0.05] [] ? finish_task_switch+0x6b/0x200 [ +0.06] [] __schedule+0x36c/0x950 [ +0.02] [] schedule+0x37/0x80 [ +0.06] [] futex_wait_queue_me+0xbc/0x120 [

Re: Observing Softlockup's while running heavy IOs

2016-08-19 Thread Sreekanth Reddy
18, 2016 12:56 AM >> Subject: Observing Softlockup's while running heavy IOs >> >> Problem statement: >> Observing softlockups while running heavy IOs on 8 SSD drives >> connected behind our LSI SAS 3004 HBA. >> > ... >> Observing a loop in th

Re: Observing Softlockup's while running heavy IOs

2016-08-19 Thread Sreekanth Reddy
19, 2016 at 2:38 AM, Elliott, Robert (Persistent Memory) wrote: > > >> -Original Message- >> From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel- >> ow...@vger.kernel.org] On Behalf Of Sreekanth Reddy >> Sent: Thursday, August 18, 2016 12:56 AM >>

Re: Observing Softlockup's while running heavy IOs

2016-08-18 Thread Bart Van Assche
On 08/17/16 22:55, Sreekanth Reddy wrote: > Observing softlockups while running heavy IOs on 8 SSD drives > connected behind our LSI SAS 3004 HBA. Hello Sreekanth, This means that more than 23s was spent before the scheduler was invoked, probably due to a loop. Can you give the attached

Re: Observing Softlockup's while running heavy IOs

2016-08-18 Thread Bart Van Assche
On 08/17/16 22:55, Sreekanth Reddy wrote: > Observing softlockups while running heavy IOs on 8 SSD drives > connected behind our LSI SAS 3004 HBA. Hello Sreekanth, This means that more than 23s was spent before the scheduler was invoked, probably due to a loop. Can you give the attached

RE: Observing Softlockup's while running heavy IOs

2016-08-18 Thread Elliott, Robert (Persistent Memory)
> -Original Message- > From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel- > ow...@vger.kernel.org] On Behalf Of Sreekanth Reddy > Sent: Thursday, August 18, 2016 12:56 AM > Subject: Observing Softlockup's while running heavy IOs > > Problem statement: &g

RE: Observing Softlockup's while running heavy IOs

2016-08-18 Thread Elliott, Robert (Persistent Memory)
> -Original Message- > From: linux-kernel-ow...@vger.kernel.org [mailto:linux-kernel- > ow...@vger.kernel.org] On Behalf Of Sreekanth Reddy > Sent: Thursday, August 18, 2016 12:56 AM > Subject: Observing Softlockup's while running heavy IOs > > Problem statement: &g

Observing Softlockup's while running heavy IOs

2016-08-17 Thread Sreekanth Reddy
Hi, Problem statement: Observing softlockups while running heavy IOs on 8 SSD drives connected behind our LSI SAS 3004 HBA. System configuration: OS & kernel version: Fedora 23, v4.2.3-300.fc23.x86_64 NUMA : disabled, CPUs : 24 logical cpus, SCSI_MQ: enabled Driver : mpt3sas, MSIx vectors: we

Observing Softlockup's while running heavy IOs

2016-08-17 Thread Sreekanth Reddy
Hi, Problem statement: Observing softlockups while running heavy IOs on 8 SSD drives connected behind our LSI SAS 3004 HBA. System configuration: OS & kernel version: Fedora 23, v4.2.3-300.fc23.x86_64 NUMA : disabled, CPUs : 24 logical cpus, SCSI_MQ: enabled Driver : mpt3sas, MSIx vectors: we