Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Fri, 6 Nov 2015, Tetsuo Handa wrote:

> So, if you refer to the blocking of the execution of vmstat updates,
> description for patch 3/3 sould be updated to something like below?

Ok that is much better.

> --
> Since __GFP_WAIT memory allocations do not call schedule()
> when there is nothing to reclaim, and workqueue does not kick
> remaining workqueue items unless in-flight workqueue item calls
> schedule(), __GFP_WAIT memory allocation requests by workqueue
> items can block vmstat_update work item forever.
>
> Since zone_reclaimable() decision depends on vmstat counters
> to be up to dated, a silent lockup occurs because a workqueue
> item doing a __GFP_WAIT memory allocation request continues
> using outdated vmstat counters.
>
> In order to fix this problem, we need to allocate a dedicated
> workqueue for vmstat. Note that this patch itself does not fix
> lockup problem. Tejun will develop a patch which detects lockup
> situation and kick remaining workqueue items.
> --
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Sat, 31 Oct 2015, Tetsuo Handa wrote:
> 
> > Then, you need to update below description (or drop it) because
> > patch 3/3 alone will not guarantee that the counters are up to date.
> 
> The vmstat system does not guarantee that the counters are up to date
> always. The whole point is the deferral of updates for performance
> reasons. They are updated *at some point* within stat_interval. That needs
> to happen and that is what this patchset is fixing.

So, if you refer to the blocking of the execution of vmstat updates,
description for patch 3/3 sould be updated to something like below?

--
Since __GFP_WAIT memory allocations do not call schedule()
when there is nothing to reclaim, and workqueue does not kick
remaining workqueue items unless in-flight workqueue item calls
schedule(), __GFP_WAIT memory allocation requests by workqueue
items can block vmstat_update work item forever.

Since zone_reclaimable() decision depends on vmstat counters
to be up to dated, a silent lockup occurs because a workqueue
item doing a __GFP_WAIT memory allocation request continues
using outdated vmstat counters.

In order to fix this problem, we need to allocate a dedicated
workqueue for vmstat. Note that this patch itself does not fix
lockup problem. Tejun will develop a patch which detects lockup
situation and kick remaining workqueue items.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Sat, 31 Oct 2015, Tetsuo Handa wrote:
> 
> > Then, you need to update below description (or drop it) because
> > patch 3/3 alone will not guarantee that the counters are up to date.
> 
> The vmstat system does not guarantee that the counters are up to date
> always. The whole point is the deferral of updates for performance
> reasons. They are updated *at some point* within stat_interval. That needs
> to happen and that is what this patchset is fixing.

So, if you refer to the blocking of the execution of vmstat updates,
description for patch 3/3 sould be updated to something like below?

--
Since __GFP_WAIT memory allocations do not call schedule()
when there is nothing to reclaim, and workqueue does not kick
remaining workqueue items unless in-flight workqueue item calls
schedule(), __GFP_WAIT memory allocation requests by workqueue
items can block vmstat_update work item forever.

Since zone_reclaimable() decision depends on vmstat counters
to be up to dated, a silent lockup occurs because a workqueue
item doing a __GFP_WAIT memory allocation request continues
using outdated vmstat counters.

In order to fix this problem, we need to allocate a dedicated
workqueue for vmstat. Note that this patch itself does not fix
lockup problem. Tejun will develop a patch which detects lockup
situation and kick remaining workqueue items.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Fri, 6 Nov 2015, Tetsuo Handa wrote:

> So, if you refer to the blocking of the execution of vmstat updates,
> description for patch 3/3 sould be updated to something like below?

Ok that is much better.

> --
> Since __GFP_WAIT memory allocations do not call schedule()
> when there is nothing to reclaim, and workqueue does not kick
> remaining workqueue items unless in-flight workqueue item calls
> schedule(), __GFP_WAIT memory allocation requests by workqueue
> items can block vmstat_update work item forever.
>
> Since zone_reclaimable() decision depends on vmstat counters
> to be up to dated, a silent lockup occurs because a workqueue
> item doing a __GFP_WAIT memory allocation request continues
> using outdated vmstat counters.
>
> In order to fix this problem, we need to allocate a dedicated
> workqueue for vmstat. Note that this patch itself does not fix
> lockup problem. Tejun will develop a patch which detects lockup
> situation and kick remaining workqueue items.
> --
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

On Mon, Nov 02, 2015 at 12:10:04PM -0600, Christoph Lameter wrote:
> Well true that is dependend on the correct workqueue operation. I though
> that was fixed by Tejun?

At least for now, we're going with Tetsuo's short sleep patch.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Tue, 3 Nov 2015, Tetsuo Handa wrote:

> I'm still unclear. I think that the result of this patchset is
>
>   The counters are never updated even after stat_interval
>   if some workqueue item is doing a __GFP_WAIT memory allocation.
>
> but the patch description sounds as if
>
>   The counters will be updated even if some workqueue item is
>   doing a __GFP_WAIT memory allocation.
>
> which denies the actual result I tested with this patchset applied.

Well true that is dependend on the correct workqueue operation. I though
that was fixed by Tejun?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Sat, 31 Oct 2015, Tetsuo Handa wrote:
> 
> > Then, you need to update below description (or drop it) because
> > patch 3/3 alone will not guarantee that the counters are up to date.
> 
> The vmstat system does not guarantee that the counters are up to date
> always. The whole point is the deferral of updates for performance
> reasons. They are updated *at some point* within stat_interval. That needs
> to happen and that is what this patchset is fixing.
> 
I'm still unclear. I think that the result of this patchset is

  The counters are never updated even after stat_interval
  if some workqueue item is doing a __GFP_WAIT memory allocation.

but the patch description sounds as if

  The counters will be updated even if some workqueue item is
  doing a __GFP_WAIT memory allocation.

which denies the actual result I tested with this patchset applied.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Sat, 31 Oct 2015, Tetsuo Handa wrote:

> Then, you need to update below description (or drop it) because
> patch 3/3 alone will not guarantee that the counters are up to date.

The vmstat system does not guarantee that the counters are up to date
always. The whole point is the deferral of updates for performance
reasons. They are updated *at some point* within stat_interval. That needs
to happen and that is what this patchset is fixing.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Tue, 3 Nov 2015, Tetsuo Handa wrote:

> I'm still unclear. I think that the result of this patchset is
>
>   The counters are never updated even after stat_interval
>   if some workqueue item is doing a __GFP_WAIT memory allocation.
>
> but the patch description sounds as if
>
>   The counters will be updated even if some workqueue item is
>   doing a __GFP_WAIT memory allocation.
>
> which denies the actual result I tested with this patchset applied.

Well true that is dependend on the correct workqueue operation. I though
that was fixed by Tejun?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

On Mon, Nov 02, 2015 at 12:10:04PM -0600, Christoph Lameter wrote:
> Well true that is dependend on the correct workqueue operation. I though
> that was fixed by Tejun?

At least for now, we're going with Tetsuo's short sleep patch.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Sat, 31 Oct 2015, Tetsuo Handa wrote:

> Then, you need to update below description (or drop it) because
> patch 3/3 alone will not guarantee that the counters are up to date.

The vmstat system does not guarantee that the counters are up to date
always. The whole point is the deferral of updates for performance
reasons. They are updated *at some point* within stat_interval. That needs
to happen and that is what this patchset is fixing.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Sat, 31 Oct 2015, Tetsuo Handa wrote:
> 
> > Then, you need to update below description (or drop it) because
> > patch 3/3 alone will not guarantee that the counters are up to date.
> 
> The vmstat system does not guarantee that the counters are up to date
> always. The whole point is the deferral of updates for performance
> reasons. They are updated *at some point* within stat_interval. That needs
> to happen and that is what this patchset is fixing.
> 
I'm still unclear. I think that the result of this patchset is

  The counters are never updated even after stat_interval
  if some workqueue item is doing a __GFP_WAIT memory allocation.

but the patch description sounds as if

  The counters will be updated even if some workqueue item is
  doing a __GFP_WAIT memory allocation.

which denies the actual result I tested with this patchset applied.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Thu, 29 Oct 2015, Tejun Heo wrote:
> 
> > Wait, this series doesn't include Tetsuo's change.  Of course it won't
> > fix the deadlock problem.  What's necessary is Tetsuo's patch +
> > WQ_MEM_RECLAIM.
> 
> This series is only dealing with vmstat changes. Do I get an ack here?
> 

Then, you need to update below description (or drop it) because
patch 3/3 alone will not guarantee that the counters are up to date.

  Christoph Lameter wrote:
  > Seems that vmstat needs its own workqueue now since the general
  > workqueue mechanism has been *enhanced* which means that the
  > vmstat_updates cannot run reliably but are being blocked by
  > work requests doing memory allocation. Which causes vmstat
  > to be unable to keep the counters up to date.

I am waiting for decision from candidates listed at
http://lkml.kernel.org/r/201510251952.cef04109.osotlfhfvfj...@i-love.sakura.ne.jp
 .
If your series is not for backporting, please choose one from
the candidates. Can you accept the original patch at
http://lkml.kernel.org/r/201510212126.jif90648.hoofjvfqlms...@i-love.sakura.ne.jp
which implements (1) from the candidates?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

On Thu, Oct 29, 2015 at 08:01:12PM -0500, Christoph Lameter wrote:
> On Thu, 29 Oct 2015, Tejun Heo wrote:
> 
> > Wait, this series doesn't include Tetsuo's change.  Of course it won't
> > fix the deadlock problem.  What's necessary is Tetsuo's patch +
> > WQ_MEM_RECLAIM.
> 
> This series is only dealing with vmstat changes. Do I get an ack here?

Yeap, please feel free to add my acked-by.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Thu, 29 Oct 2015, Tejun Heo wrote:
> 
> > Wait, this series doesn't include Tetsuo's change.  Of course it won't
> > fix the deadlock problem.  What's necessary is Tetsuo's patch +
> > WQ_MEM_RECLAIM.
> 
> This series is only dealing with vmstat changes. Do I get an ack here?
> 

Then, you need to update below description (or drop it) because
patch 3/3 alone will not guarantee that the counters are up to date.

  Christoph Lameter wrote:
  > Seems that vmstat needs its own workqueue now since the general
  > workqueue mechanism has been *enhanced* which means that the
  > vmstat_updates cannot run reliably but are being blocked by
  > work requests doing memory allocation. Which causes vmstat
  > to be unable to keep the counters up to date.

I am waiting for decision from candidates listed at
http://lkml.kernel.org/r/201510251952.cef04109.osotlfhfvfj...@i-love.sakura.ne.jp
 .
If your series is not for backporting, please choose one from
the candidates. Can you accept the original patch at
http://lkml.kernel.org/r/201510212126.jif90648.hoofjvfqlms...@i-love.sakura.ne.jp
which implements (1) from the candidates?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

On Thu, Oct 29, 2015 at 08:01:12PM -0500, Christoph Lameter wrote:
> On Thu, 29 Oct 2015, Tejun Heo wrote:
> 
> > Wait, this series doesn't include Tetsuo's change.  Of course it won't
> > fix the deadlock problem.  What's necessary is Tetsuo's patch +
> > WQ_MEM_RECLAIM.
> 
> This series is only dealing with vmstat changes. Do I get an ack here?

Yeap, please feel free to add my acked-by.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Thu, 29 Oct 2015, Tejun Heo wrote:

> Wait, this series doesn't include Tetsuo's change.  Of course it won't
> fix the deadlock problem.  What's necessary is Tetsuo's patch +
> WQ_MEM_RECLAIM.

This series is only dealing with vmstat changes. Do I get an ack here?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Thu, 29 Oct 2015, Tejun Heo wrote:

> Wait, this series doesn't include Tetsuo's change.  Of course it won't
> fix the deadlock problem.  What's necessary is Tetsuo's patch +
> WQ_MEM_RECLAIM.

This series is only dealing with vmstat changes. Do I get an ack here?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

On Thu, Oct 29, 2015 at 11:24:47AM +0900, Tejun Heo wrote:
> Hello,
> 
> That's weird.
> 
> On Wed, Oct 28, 2015 at 08:57:28PM +0900, Tetsuo Handa wrote:
> > [  272.851035] Showing busy workqueues and worker pools:
> > [  272.852583] workqueue events: flags=0x0
> > [  272.853942]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
> > [  272.855781] pending: vmw_fb_dirty_flush [vmwgfx]
> > [  272.857500]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
> > [  272.859359] pending: vmpressure_work_fn
> > [  272.860840] workqueue events_freezable_power_: flags=0x84
> > [  272.862461]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
> > [  272.864479] in-flight: 11286:disk_events_workfn
> 
> What's this guy doing?  Can you get stack dump on 11286 (or whatever
> is in flight in the next lockup)?

Wait, this series doesn't include Tetsuo's change.  Of course it won't
fix the deadlock problem.  What's necessary is Tetsuo's patch +
WQ_MEM_RECLAIM.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

Hello,

That's weird.

On Wed, Oct 28, 2015 at 08:57:28PM +0900, Tetsuo Handa wrote:
> [  272.851035] Showing busy workqueues and worker pools:
> [  272.852583] workqueue events: flags=0x0
> [  272.853942]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
> [  272.855781] pending: vmw_fb_dirty_flush [vmwgfx]
> [  272.857500]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
> [  272.859359] pending: vmpressure_work_fn
> [  272.860840] workqueue events_freezable_power_: flags=0x84
> [  272.862461]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
> [  272.864479] in-flight: 11286:disk_events_workfn

What's this guy doing?  Can you get stack dump on 11286 (or whatever
is in flight in the next lockup)?

> [  272.866065] pending: disk_events_workfn
> [  272.867587] workqueue vmstat: flags=0x8
> [  272.868942]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
> [  272.870785] pending: vmstat_update
> [  272.872248] pool 2: cpus=1 node=0 flags=0x0 nice=0 workers=4 idle: 14 218 
> 43

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Wed, 28 Oct 2015, Tetsuo Handa wrote:

> Christoph Lameter wrote:
> > On Wed, 28 Oct 2015, Tejun Heo wrote:
> >
> > > The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
> > > WQ_SYSFS and WQ_FREEZABLE make sense here.
> >
> I can still trigger silent livelock with this patchset applied.

Ok so why the vmstat updater still deferred, Tejun?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Wed, 28 Oct 2015, Tejun Heo wrote:
> 
> > The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
> > WQ_SYSFS and WQ_FREEZABLE make sense here.
> 
I can still trigger silent livelock with this patchset applied.

--
[  272.283217] MemAlloc-Info: 9 stalling task, 0 dying task, 0 victim task.
[  272.285089] MemAlloc: a.out(11325) gfp=0x24280ca order=0 delay=19164
[  272.286817] MemAlloc: a.out(11326) gfp=0x242014a order=0 delay=19104
[  272.288512] MemAlloc: vmtoolsd(1897) gfp=0x242014a order=0 delay=19072
[  272.290280] MemAlloc: kworker/1:3(11286) gfp=0x240 order=0 delay=19056
[  272.292114] MemAlloc: sshd(11202) gfp=0x242014a order=0 delay=18927
[  272.293908] MemAlloc: tuned(2073) gfp=0x242014a order=0 delay=18799
[  272.297360] MemAlloc: nmbd(4752) gfp=0x242014a order=0 delay=16532
[  272.299115] MemAlloc: auditd(529) gfp=0x242014a order=0 delay=13073
[  272.302248] MemAlloc: irqbalance(1696) gfp=0x242014a order=0 delay=10529
(...snipped...)
[  272.851035] Showing busy workqueues and worker pools:
[  272.852583] workqueue events: flags=0x0
[  272.853942]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[  272.855781] pending: vmw_fb_dirty_flush [vmwgfx]
[  272.857500]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
[  272.859359] pending: vmpressure_work_fn
[  272.860840] workqueue events_freezable_power_: flags=0x84
[  272.862461]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[  272.864479] in-flight: 11286:disk_events_workfn
[  272.866065] pending: disk_events_workfn
[  272.867587] workqueue vmstat: flags=0x8
[  272.868942]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
[  272.870785] pending: vmstat_update
[  272.872248] pool 2: cpus=1 node=0 flags=0x0 nice=0 workers=4 idle: 14 218 43
--

> 2. Create a separate workqueue so that the vmstat updater
>is not blocked by other work requeusts. This creates a
>new kernel thread  and avoids the issue of
>differentials not folded in a timely fashion.

Did you really mean "the vmstat updater is not blocked by other
work requeusts"?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tetsuo Handa]

Christoph Lameter wrote:
> On Wed, 28 Oct 2015, Tejun Heo wrote:
> 
> > The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
> > WQ_SYSFS and WQ_FREEZABLE make sense here.
> 
I can still trigger silent livelock with this patchset applied.

--
[  272.283217] MemAlloc-Info: 9 stalling task, 0 dying task, 0 victim task.
[  272.285089] MemAlloc: a.out(11325) gfp=0x24280ca order=0 delay=19164
[  272.286817] MemAlloc: a.out(11326) gfp=0x242014a order=0 delay=19104
[  272.288512] MemAlloc: vmtoolsd(1897) gfp=0x242014a order=0 delay=19072
[  272.290280] MemAlloc: kworker/1:3(11286) gfp=0x240 order=0 delay=19056
[  272.292114] MemAlloc: sshd(11202) gfp=0x242014a order=0 delay=18927
[  272.293908] MemAlloc: tuned(2073) gfp=0x242014a order=0 delay=18799
[  272.297360] MemAlloc: nmbd(4752) gfp=0x242014a order=0 delay=16532
[  272.299115] MemAlloc: auditd(529) gfp=0x242014a order=0 delay=13073
[  272.302248] MemAlloc: irqbalance(1696) gfp=0x242014a order=0 delay=10529
(...snipped...)
[  272.851035] Showing busy workqueues and worker pools:
[  272.852583] workqueue events: flags=0x0
[  272.853942]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
[  272.855781] pending: vmw_fb_dirty_flush [vmwgfx]
[  272.857500]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
[  272.859359] pending: vmpressure_work_fn
[  272.860840] workqueue events_freezable_power_: flags=0x84
[  272.862461]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[  272.864479] in-flight: 11286:disk_events_workfn
[  272.866065] pending: disk_events_workfn
[  272.867587] workqueue vmstat: flags=0x8
[  272.868942]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
[  272.870785] pending: vmstat_update
[  272.872248] pool 2: cpus=1 node=0 flags=0x0 nice=0 workers=4 idle: 14 218 43
--

> 2. Create a separate workqueue so that the vmstat updater
>is not blocked by other work requeusts. This creates a
>new kernel thread  and avoids the issue of
>differentials not folded in a timely fashion.

Did you really mean "the vmstat updater is not blocked by other
work requeusts"?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

On Thu, Oct 29, 2015 at 11:24:47AM +0900, Tejun Heo wrote:
> Hello,
> 
> That's weird.
> 
> On Wed, Oct 28, 2015 at 08:57:28PM +0900, Tetsuo Handa wrote:
> > [  272.851035] Showing busy workqueues and worker pools:
> > [  272.852583] workqueue events: flags=0x0
> > [  272.853942]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
> > [  272.855781] pending: vmw_fb_dirty_flush [vmwgfx]
> > [  272.857500]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
> > [  272.859359] pending: vmpressure_work_fn
> > [  272.860840] workqueue events_freezable_power_: flags=0x84
> > [  272.862461]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
> > [  272.864479] in-flight: 11286:disk_events_workfn
> 
> What's this guy doing?  Can you get stack dump on 11286 (or whatever
> is in flight in the next lockup)?

Wait, this series doesn't include Tetsuo's change.  Of course it won't
fix the deadlock problem.  What's necessary is Tetsuo's patch +
WQ_MEM_RECLAIM.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

Hello,

That's weird.

On Wed, Oct 28, 2015 at 08:57:28PM +0900, Tetsuo Handa wrote:
> [  272.851035] Showing busy workqueues and worker pools:
> [  272.852583] workqueue events: flags=0x0
> [  272.853942]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256
> [  272.855781] pending: vmw_fb_dirty_flush [vmwgfx]
> [  272.857500]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
> [  272.859359] pending: vmpressure_work_fn
> [  272.860840] workqueue events_freezable_power_: flags=0x84
> [  272.862461]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
> [  272.864479] in-flight: 11286:disk_events_workfn

What's this guy doing?  Can you get stack dump on 11286 (or whatever
is in flight in the next lockup)?

> [  272.866065] pending: disk_events_workfn
> [  272.867587] workqueue vmstat: flags=0x8
> [  272.868942]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256
> [  272.870785] pending: vmstat_update
> [  272.872248] pool 2: cpus=1 node=0 flags=0x0 nice=0 workers=4 idle: 14 218 
> 43

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Wed, 28 Oct 2015, Tetsuo Handa wrote:

> Christoph Lameter wrote:
> > On Wed, 28 Oct 2015, Tejun Heo wrote:
> >
> > > The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
> > > WQ_SYSFS and WQ_FREEZABLE make sense here.
> >
> I can still trigger silent livelock with this patchset applied.

Ok so why the vmstat updater still deferred, Tejun?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Wed, 28 Oct 2015, Tejun Heo wrote:

> The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
> WQ_SYSFS and WQ_FREEZABLE make sense here.


Subject: vmstat: Remove WQ_FREEZABLE and WQ_SYSFS

Signed-off-by: Christoph Lameter 

Index: linux/mm/vmstat.c
===
--- linux.orig/mm/vmstat.c
+++ linux/mm/vmstat.c
@@ -1546,8 +1546,6 @@ static int __init setup_vmstat(void)
start_shepherd_timer();
cpu_notifier_register_done();
vmstat_wq = alloc_workqueue("vmstat",
-   WQ_FREEZABLE|
-   WQ_SYSFS|
WQ_MEM_RECLAIM, 0);
 #endif
 #ifdef CONFIG_PROC_FS
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

Hello,

On Tue, Oct 27, 2015 at 09:41:17PM -0500, Christoph Lameter wrote:
> + vmstat_wq = alloc_workqueue("vmstat",
> + WQ_FREEZABLE|
> + WQ_SYSFS|
> + WQ_MEM_RECLAIM, 0);

The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
WQ_SYSFS and WQ_FREEZABLE make sense here.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Tejun Heo]

Hello,

On Tue, Oct 27, 2015 at 09:41:17PM -0500, Christoph Lameter wrote:
> + vmstat_wq = alloc_workqueue("vmstat",
> + WQ_FREEZABLE|
> + WQ_SYSFS|
> + WQ_MEM_RECLAIM, 0);

The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
WQ_SYSFS and WQ_FREEZABLE make sense here.

Thanks.

-- 
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 3/3] vmstat: Create our own workqueue

[Christoph Lameter]

On Wed, 28 Oct 2015, Tejun Heo wrote:

> The only thing necessary here is WQ_MEM_RECLAIM.  I don't see how
> WQ_SYSFS and WQ_FREEZABLE make sense here.


Subject: vmstat: Remove WQ_FREEZABLE and WQ_SYSFS

Signed-off-by: Christoph Lameter 

Index: linux/mm/vmstat.c
===
--- linux.orig/mm/vmstat.c
+++ linux/mm/vmstat.c
@@ -1546,8 +1546,6 @@ static int __init setup_vmstat(void)
start_shepherd_timer();
cpu_notifier_register_done();
vmstat_wq = alloc_workqueue("vmstat",
-   WQ_FREEZABLE|
-   WQ_SYSFS|
WQ_MEM_RECLAIM, 0);
 #endif
 #ifdef CONFIG_PROC_FS
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/