Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-12-05 Thread Jerome Marchand
On 11/26/2014 06:35 PM, Michal Hocko wrote: > On Tue 25-11-14 16:00:06, Michal Hocko wrote: >> On Tue 25-11-14 16:13:16, Konstantin Khlebnikov wrote: >>> On Tue, Nov 25, 2014 at 1:59 PM, Michal Hocko wrote: On Mon 24-11-14 11:09:40, Konstantin Khlebnikov wrote: > On Thu, Nov 20, 2014 at 6

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-26 Thread Michal Hocko
On Tue 25-11-14 16:00:06, Michal Hocko wrote: > On Tue 25-11-14 16:13:16, Konstantin Khlebnikov wrote: > > On Tue, Nov 25, 2014 at 1:59 PM, Michal Hocko wrote: > > > On Mon 24-11-14 11:09:40, Konstantin Khlebnikov wrote: > > >> On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov > > >> wrote:

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-25 Thread Michal Hocko
On Tue 25-11-14 16:13:16, Konstantin Khlebnikov wrote: > On Tue, Nov 25, 2014 at 1:59 PM, Michal Hocko wrote: > > On Mon 24-11-14 11:09:40, Konstantin Khlebnikov wrote: > >> On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov > >> wrote: > >> > On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel w

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-25 Thread Konstantin Khlebnikov
On Tue, Nov 25, 2014 at 1:59 PM, Michal Hocko wrote: > On Mon 24-11-14 11:09:40, Konstantin Khlebnikov wrote: >> On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov >> wrote: >> > On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel wrote: >> >> -BEGIN PGP SIGNED MESSAGE- >> >> Hash: SHA1 >

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-25 Thread Michal Hocko
On Mon 24-11-14 11:09:40, Konstantin Khlebnikov wrote: > On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov > wrote: > > On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel wrote: > >> -BEGIN PGP SIGNED MESSAGE- > >> Hash: SHA1 > >> > >> On 11/20/2014 09:42 AM, Konstantin Khlebnikov wrote:

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-23 Thread Konstantin Khlebnikov
On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov wrote: > On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 11/20/2014 09:42 AM, Konstantin Khlebnikov wrote: >> >>> I'm thinking about limitation for reusing anon_vmas which migh

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-20 Thread Michel Lespinasse
On Thu, Nov 20, 2014 at 3:42 PM, Konstantin Khlebnikov wrote: > On Thu, Nov 20, 2014 at 2:14 AM, Michel Lespinasse wrote: >> On Wed, Nov 19, 2014 at 8:58 AM, Konstantin Khlebnikov >> wrote: >>> On Wed, Nov 19, 2014 at 7:09 PM, Vlastimil Babka wrote: Also from reading http://lwn.net/Articl

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-20 Thread Konstantin Khlebnikov
On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 11/20/2014 09:42 AM, Konstantin Khlebnikov wrote: > >> I'm thinking about limitation for reusing anon_vmas which might >> increase performance without breaking asymptotic estimation of >>

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-20 Thread Rik van Riel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/20/2014 09:42 AM, Konstantin Khlebnikov wrote: > I'm thinking about limitation for reusing anon_vmas which might > increase performance without breaking asymptotic estimation of > count anon_vma in the worst case. For example this heuristic: all

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-20 Thread Konstantin Khlebnikov
On Thu, Nov 20, 2014 at 2:14 AM, Michel Lespinasse wrote: > On Wed, Nov 19, 2014 at 8:58 AM, Konstantin Khlebnikov > wrote: >> On Wed, Nov 19, 2014 at 7:09 PM, Vlastimil Babka wrote: >>> Also from reading http://lwn.net/Articles/383162/ I understand that >>> correctness >>> also depends on the

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-19 Thread Michel Lespinasse
On Wed, Nov 19, 2014 at 8:58 AM, Konstantin Khlebnikov wrote: > On Wed, Nov 19, 2014 at 7:09 PM, Vlastimil Babka wrote: >> Also from reading http://lwn.net/Articles/383162/ I understand that >> correctness >> also depends on the hierarchy and I wonder if there's a danger of >> reintroducing >>

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-19 Thread Konstantin Khlebnikov
On Wed, Nov 19, 2014 at 7:09 PM, Vlastimil Babka wrote: > On 11/19/2014 03:36 PM, Konstantin Khlebnikov wrote: >> On Wed, Nov 19, 2014 at 2:50 AM, Vlastimil Babka wrote: >>> On 11/19/2014 12:02 AM, Konstantin Khlebnikov wrote: On Wed, Nov 19, 2014 at 1:15 AM, Konstantin Khlebnikov wro

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-19 Thread Vlastimil Babka
On 11/19/2014 03:36 PM, Konstantin Khlebnikov wrote: > On Wed, Nov 19, 2014 at 2:50 AM, Vlastimil Babka wrote: >> On 11/19/2014 12:02 AM, Konstantin Khlebnikov wrote: >>> On Wed, Nov 19, 2014 at 1:15 AM, Konstantin Khlebnikov >>> wrote: On Tue, Nov 18, 2014 at 11:19 PM, Andrew Morton

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-19 Thread Konstantin Khlebnikov
On Wed, Nov 19, 2014 at 2:50 AM, Vlastimil Babka wrote: > On 11/19/2014 12:02 AM, Konstantin Khlebnikov wrote: >> On Wed, Nov 19, 2014 at 1:15 AM, Konstantin Khlebnikov >> wrote: >>> On Tue, Nov 18, 2014 at 11:19 PM, Andrew Morton >>> wrote: On Mon, 17 Nov 2014 21:41:57 -0500 Rik van Riel

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-18 Thread Rik van Riel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/18/2014 03:19 PM, Andrew Morton wrote: > On Mon, 17 Nov 2014 21:41:57 -0500 Rik van Riel > wrote: > >> That way people can understand what the code does simply by >> looking at the changelog - no need to go find old linux-kernel >> mailing lis

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-18 Thread Vlastimil Babka
On 11/19/2014 12:02 AM, Konstantin Khlebnikov wrote: > On Wed, Nov 19, 2014 at 1:15 AM, Konstantin Khlebnikov > wrote: >> On Tue, Nov 18, 2014 at 11:19 PM, Andrew Morton >> wrote: >>> On Mon, 17 Nov 2014 21:41:57 -0500 Rik van Riel wrote: >>> > Because of the serial forking there does inde

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-18 Thread Konstantin Khlebnikov
On Wed, Nov 19, 2014 at 1:15 AM, Konstantin Khlebnikov wrote: > On Tue, Nov 18, 2014 at 11:19 PM, Andrew Morton > wrote: >> On Mon, 17 Nov 2014 21:41:57 -0500 Rik van Riel wrote: >> >>> > Because of the serial forking there does indeed end up being an >>> > infinite number of vmas. The initial

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-18 Thread Konstantin Khlebnikov
On Tue, Nov 18, 2014 at 11:19 PM, Andrew Morton wrote: > On Mon, 17 Nov 2014 21:41:57 -0500 Rik van Riel wrote: > >> > Because of the serial forking there does indeed end up being an >> > infinite number of vmas. The initial vma can never be deleted >> > (even though the initial parent process h

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-18 Thread Andrew Morton
On Mon, 17 Nov 2014 21:41:57 -0500 Rik van Riel wrote: > > Because of the serial forking there does indeed end up being an > > infinite number of vmas. The initial vma can never be deleted > > (even though the initial parent process has long since terminated) > > because the initial vma is refer

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-17 Thread Rik van Riel
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/17/2014 08:41 PM, Daniel Forrest wrote: > On Mon, Nov 17, 2014 at 04:02:12PM -0800, Andrew Morton wrote: >> On Fri, 14 Nov 2014 10:30:53 -0600 Daniel Forrest >> wrote: >> >>> There have been a couple of inquiries about the status of this >>> pa

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-17 Thread Daniel Forrest
On Mon, Nov 17, 2014 at 04:02:12PM -0800, Andrew Morton wrote: > On Fri, 14 Nov 2014 10:30:53 -0600 Daniel Forrest > wrote: > > > There have been a couple of inquiries about the status of this patch > > over the last few months, so I am going to try pushing it out. > > > > Andrea Arcangeli has

Re: [PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-17 Thread Andrew Morton
On Fri, 14 Nov 2014 10:30:53 -0600 Daniel Forrest wrote: > There have been a couple of inquiries about the status of this patch > over the last few months, so I am going to try pushing it out. > > Andrea Arcangeli has commented: > > > Agreed. The only thing I don't like about this patch is the

[PATCH] Repeated fork() causes SLAB to grow without bound

2014-11-14 Thread Daniel Forrest
There have been a couple of inquiries about the status of this patch over the last few months, so I am going to try pushing it out. Andrea Arcangeli has commented: > Agreed. The only thing I don't like about this patch is the hardcoding > of number 5: could we make it a variable to tweak with sys

Re: [RFC PATCH] Re: Repeated fork() causes SLAB to grow without bound

2013-06-05 Thread Andrea Arcangeli
On Tue, Jun 04, 2013 at 06:37:25AM -0400, Rik van Riel wrote: > On 06/03/2013 03:50 PM, Daniel Forrest wrote: > > On Tue, Aug 21, 2012 at 11:29:54PM -0400, Rik van Riel wrote: > >> On 08/21/2012 11:20 PM, Michel Lespinasse wrote: > >>> On Mon, Aug 20, 2012 at 02:39:26AM -0700, Michel Lespinasse wro

Re: [RFC PATCH] Re: Repeated fork() causes SLAB to grow without bound

2013-06-04 Thread Rik van Riel
On 06/03/2013 03:50 PM, Daniel Forrest wrote: On Tue, Aug 21, 2012 at 11:29:54PM -0400, Rik van Riel wrote: On 08/21/2012 11:20 PM, Michel Lespinasse wrote: On Mon, Aug 20, 2012 at 02:39:26AM -0700, Michel Lespinasse wrote: Instead of adding an atomic count for page references, we could limit

Re: [RFC PATCH] Re: Repeated fork() causes SLAB to grow without bound

2013-06-03 Thread Daniel Forrest
On Tue, Aug 21, 2012 at 11:29:54PM -0400, Rik van Riel wrote: > On 08/21/2012 11:20 PM, Michel Lespinasse wrote: > >On Mon, Aug 20, 2012 at 02:39:26AM -0700, Michel Lespinasse wrote: > >>Instead of adding an atomic count for page references, we could limit > >>the anon_vma stacking depth. In fork,

Re: [RFC PATCH] Re: Repeated fork() causes SLAB to grow without bound

2012-08-21 Thread Rik van Riel
On 08/21/2012 11:20 PM, Michel Lespinasse wrote: On Mon, Aug 20, 2012 at 02:39:26AM -0700, Michel Lespinasse wrote: Instead of adding an atomic count for page references, we could limit the anon_vma stacking depth. In fork, we would only clone anon_vmas that have a low enough generation count. I

[RFC PATCH] Re: Repeated fork() causes SLAB to grow without bound

2012-08-21 Thread Michel Lespinasse
On Mon, Aug 20, 2012 at 02:39:26AM -0700, Michel Lespinasse wrote: > Instead of adding an atomic count for page references, we could limit > the anon_vma stacking depth. In fork, we would only clone anon_vmas > that have a low enough generation count. I think that's not great > (adds a special case

Re: Repeated fork() causes SLAB to grow without bound

2012-08-20 Thread Michel Lespinasse
On Mon, Aug 20, 2012 at 4:53 AM, Michel Lespinasse wrote: > I wonder if it might help to add the child VMA onto the parent's > anon_vma only at the first child COW event. That way it would at least > be possible (with userspace changes) for any forking servers to > separate the areas they want to

Re: Repeated fork() causes SLAB to grow without bound

2012-08-20 Thread Michel Lespinasse
On Mon, Aug 20, 2012 at 4:17 AM, Rik van Riel wrote: > Without the anon_vma_chains, we end up scanning every single > one of the child processes (and the parent) for every COWed > page, which can be a real issue when the VM runs into 1000 > such pages, for 1000 child processes. > > Unfortunately,

Re: Repeated fork() causes SLAB to grow without bound

2012-08-20 Thread Rik van Riel
On 08/20/2012 05:39 AM, Michel Lespinasse wrote: I would still prefer if we could just remove the anon_vma_chain stuff, though. If only we could. That simply replaces a medium issue at fork time, with the potential for a catastrophic issue at page reclaim time, in any workload with heavily fo

Re: Repeated fork() causes SLAB to grow without bound

2012-08-20 Thread Andi Kleen
Michel Lespinasse writes: > > I would still prefer if we could just remove the anon_vma_chain stuff, though. Would probably help with the fork locking problems too. We never really recovered from that regression. -Andi -- a...@linux.intel.com -- Speaking for myself only -- To unsubscribe from

Re: Repeated fork() causes SLAB to grow without bound

2012-08-20 Thread Michel Lespinasse
On Mon, Aug 20, 2012 at 1:00 AM, Hugh Dickins wrote: > On Fri, 17 Aug 2012, Rik van Riel wrote: >> Of course, that leaves the big question: do we want the >> overhead of having the atomic addition and decrement for >> every anonymous memory page, or is it easier to fix this >> issue in userspace?

Re: Repeated fork() causes SLAB to grow without bound

2012-08-20 Thread Hugh Dickins
On Fri, 17 Aug 2012, Rik van Riel wrote: > On 08/17/2012 08:03 PM, Daniel Forrest wrote: > > > Based on your comments, I came up with the following patch. It boots > > and the anon_vma/anon_vma_chain SLAB usage is stable, but I don't know > > if I've overlooked something. I'm not a kernel hacker

Re: Repeated fork() causes SLAB to grow without bound

2012-08-17 Thread Rik van Riel
On 08/18/2012 12:07 AM, Daniel Forrest wrote: I was being careful since I wasn't certain about the locking. Does the test need to be protected by "lock_anon_vma_root"? That's why I chose the overhead of the possible wasted "anon_vma_chain_alloc". The function anon_vma_clone is being called f

Re: Repeated fork() causes SLAB to grow without bound

2012-08-17 Thread Daniel Forrest
On Fri, Aug 17, 2012 at 11:46:18PM -0400, Rik van Riel wrote: > On 08/17/2012 08:03 PM, Daniel Forrest wrote: > > >Based on your comments, I came up with the following patch. It boots > >and the anon_vma/anon_vma_chain SLAB usage is stable, but I don't know > >if I've overlooked something. I'm

Re: Repeated fork() causes SLAB to grow without bound

2012-08-17 Thread Rik van Riel
On 08/17/2012 08:03 PM, Daniel Forrest wrote: Based on your comments, I came up with the following patch. It boots and the anon_vma/anon_vma_chain SLAB usage is stable, but I don't know if I've overlooked something. I'm not a kernel hacker. The patch looks reasonable to me. There is one spo

Re: Repeated fork() causes SLAB to grow without bound

2012-08-17 Thread Daniel Forrest
On Thu, Aug 16, 2012 at 02:58:45PM -0400, Rik van Riel wrote: > Oh dear. > > Basically, what happens is that at fork time, a new > "level" is created for the anon_vma hierarchy. This > works great for normal forking daemons, since the > parent process just keeps running, and forking off > childre

Re: Repeated fork() causes SLAB to grow without bound

2012-08-16 Thread Rik van Riel
On 08/15/2012 10:46 PM, Daniel Forrest wrote: I'm hoping someone has seen this before... I've been trying to track down a performance problem with Linux 3.0.4. The symptom is system-mode load increasing over time while user-mode load remains constant while running a data ingest/processing progra

Repeated fork() causes SLAB to grow without bound

2012-08-15 Thread Daniel Forrest
I'm hoping someone has seen this before... I've been trying to track down a performance problem with Linux 3.0.4. The symptom is system-mode load increasing over time while user-mode load remains constant while running a data ingest/processing program. Looking at /proc/meminfo I noticed SUnreclai