Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-31 Thread David Rientjes
On Thu, 30 Jul 2015, Mel Gorman wrote: > There will be different opinions on periodic compaction but to be honest, > periodic compaction also could be implemented from userspace using the > compact_node sysfs files. The risk with periodic compaction is that it > can cause stalls in applications

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-31 Thread David Rientjes
On Thu, 30 Jul 2015, Mel Gorman wrote: There will be different opinions on periodic compaction but to be honest, periodic compaction also could be implemented from userspace using the compact_node sysfs files. The risk with periodic compaction is that it can cause stalls in applications that

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-30 Thread Mel Gorman
On Thu, Jul 02, 2015 at 10:46:32AM +0200, Vlastimil Babka wrote: > Memory compaction can be currently performed in several contexts: > > - kswapd balancing a zone after a high-order allocation failure Which potentially was a problem that's hard to detect. > - direct compaction to satisfy a

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-30 Thread Mel Gorman
On Thu, Jul 02, 2015 at 10:46:32AM +0200, Vlastimil Babka wrote: Memory compaction can be currently performed in several contexts: - kswapd balancing a zone after a high-order allocation failure Which potentially was a problem that's hard to detect. - direct compaction to satisfy a

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-29 Thread Dave Chinner
On Wed, Jul 29, 2015 at 08:34:06AM +0200, Vlastimil Babka wrote: > On 07/29/2015 02:33 AM, David Rientjes wrote: > > On Fri, 24 Jul 2015, Vlastimil Babka wrote: > > > >> > Two issues I want to bring up: > >> > > >> > (1) do non-thp configs benefit from periodic compaction? > >> > > >> >

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-29 Thread David Rientjes
On Wed, 29 Jul 2015, Vlastimil Babka wrote: > > I see the two mechanisms different enough that they need to be defined > > separately: periodic compaction that would be done at certain intervals > > regardless of fragmentation or allocation failures to keep fragmentation > > low, and

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-29 Thread Vlastimil Babka
On 07/29/2015 02:33 AM, David Rientjes wrote: > On Fri, 24 Jul 2015, Vlastimil Babka wrote: > >> > Two issues I want to bring up: >> > >> > (1) do non-thp configs benefit from periodic compaction? >> > >> > In my experience, no, but perhaps there are other use cases where >> >

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-29 Thread David Rientjes
On Wed, 29 Jul 2015, Vlastimil Babka wrote: I see the two mechanisms different enough that they need to be defined separately: periodic compaction that would be done at certain intervals regardless of fragmentation or allocation failures to keep fragmentation low, and background

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-29 Thread Dave Chinner
On Wed, Jul 29, 2015 at 08:34:06AM +0200, Vlastimil Babka wrote: On 07/29/2015 02:33 AM, David Rientjes wrote: On Fri, 24 Jul 2015, Vlastimil Babka wrote: Two issues I want to bring up: (1) do non-thp configs benefit from periodic compaction? In my experience, no,

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-29 Thread Vlastimil Babka
On 07/29/2015 02:33 AM, David Rientjes wrote: On Fri, 24 Jul 2015, Vlastimil Babka wrote: Two issues I want to bring up: (1) do non-thp configs benefit from periodic compaction? In my experience, no, but perhaps there are other use cases where this has been a pain.

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-28 Thread David Rientjes
On Fri, 24 Jul 2015, Vlastimil Babka wrote: > > Two issues I want to bring up: > > > > (1) do non-thp configs benefit from periodic compaction? > > > > In my experience, no, but perhaps there are other use cases where > > this has been a pain. The primary candidates, in my

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-28 Thread David Rientjes
On Fri, 24 Jul 2015, Vlastimil Babka wrote: Two issues I want to bring up: (1) do non-thp configs benefit from periodic compaction? In my experience, no, but perhaps there are other use cases where this has been a pain. The primary candidates, in my opinion,

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-24 Thread Vlastimil Babka
On 07/23/2015 11:21 PM, David Rientjes wrote: On Thu, 23 Jul 2015, Vlastimil Babka wrote: When a khugepaged allocation fails for a node, it could easily kick off background compaction on that node and revisit the range later, very similar to how we can kick off background compaction in the

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-24 Thread Joonsoo Kim
On Thu, Jul 23, 2015 at 02:21:29PM -0700, David Rientjes wrote: > On Thu, 23 Jul 2015, Vlastimil Babka wrote: > > > > When a khugepaged allocation fails for a node, it could easily kick off > > > background compaction on that node and revisit the range later, very > > > similar to how we can

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-24 Thread Vlastimil Babka
On 07/23/2015 11:21 PM, David Rientjes wrote: On Thu, 23 Jul 2015, Vlastimil Babka wrote: When a khugepaged allocation fails for a node, it could easily kick off background compaction on that node and revisit the range later, very similar to how we can kick off background compaction in the

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-24 Thread Joonsoo Kim
On Thu, Jul 23, 2015 at 02:21:29PM -0700, David Rientjes wrote: On Thu, 23 Jul 2015, Vlastimil Babka wrote: When a khugepaged allocation fails for a node, it could easily kick off background compaction on that node and revisit the range later, very similar to how we can kick off

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread Joonsoo Kim
On Thu, Jul 23, 2015 at 01:58:20PM -0700, David Rientjes wrote: > On Thu, 23 Jul 2015, Joonsoo Kim wrote: > > > > The slub allocator does try to allocate its high-order memory with > > > __GFP_WAIT before falling back to lower orders if possible. I would > > > think > > > that this would be

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread David Rientjes
On Thu, 23 Jul 2015, Vlastimil Babka wrote: > > When a khugepaged allocation fails for a node, it could easily kick off > > background compaction on that node and revisit the range later, very > > similar to how we can kick off background compaction in the page allocator > > when async or

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread David Rientjes
On Thu, 23 Jul 2015, Joonsoo Kim wrote: > > The slub allocator does try to allocate its high-order memory with > > __GFP_WAIT before falling back to lower orders if possible. I would think > > that this would be the greatest sign of on-demand memory compaction being > > a problem, especially

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread Vlastimil Babka
On 07/23/2015 12:36 AM, David Rientjes wrote: > On Wed, 22 Jul 2015, Vlastimil Babka wrote: > >> > I don't think the criteria for waking up khugepaged should become any more >> > complex beyond its current state, which is impacted by two different >> > tunables, and whether it actually has memory

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread David Rientjes
On Thu, 23 Jul 2015, Joonsoo Kim wrote: The slub allocator does try to allocate its high-order memory with __GFP_WAIT before falling back to lower orders if possible. I would think that this would be the greatest sign of on-demand memory compaction being a problem, especially since

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread Joonsoo Kim
On Thu, Jul 23, 2015 at 01:58:20PM -0700, David Rientjes wrote: On Thu, 23 Jul 2015, Joonsoo Kim wrote: The slub allocator does try to allocate its high-order memory with __GFP_WAIT before falling back to lower orders if possible. I would think that this would be the greatest

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread David Rientjes
On Thu, 23 Jul 2015, Vlastimil Babka wrote: When a khugepaged allocation fails for a node, it could easily kick off background compaction on that node and revisit the range later, very similar to how we can kick off background compaction in the page allocator when async or sync_light

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-23 Thread Vlastimil Babka
On 07/23/2015 12:36 AM, David Rientjes wrote: On Wed, 22 Jul 2015, Vlastimil Babka wrote: I don't think the criteria for waking up khugepaged should become any more complex beyond its current state, which is impacted by two different tunables, and whether it actually has memory to scan.

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-22 Thread Joonsoo Kim
Hello, On Thu, Jul 09, 2015 at 02:53:27PM -0700, David Rientjes wrote: > The slub allocator does try to allocate its high-order memory with > __GFP_WAIT before falling back to lower orders if possible. I would think > that this would be the greatest sign of on-demand memory compaction being

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-22 Thread David Rientjes
On Wed, 22 Jul 2015, Vlastimil Babka wrote: > > I don't think the criteria for waking up khugepaged should become any more > > complex beyond its current state, which is impacted by two different > > tunables, and whether it actually has memory to scan. During this > > additional wakeup, you'd

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-22 Thread Vlastimil Babka
On 07/22/2015 01:07 AM, David Rientjes wrote: On Tue, 21 Jul 2015, Vlastimil Babka wrote: Khugepaged benefits from the periodic memory compaction being done immediately before it attempts to compact memory, and that may be lost with a de-coupled approach like this. Meant to say "before it

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-22 Thread Vlastimil Babka
On 07/22/2015 01:07 AM, David Rientjes wrote: On Tue, 21 Jul 2015, Vlastimil Babka wrote: Khugepaged benefits from the periodic memory compaction being done immediately before it attempts to compact memory, and that may be lost with a de-coupled approach like this. Meant to say before it

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-22 Thread Joonsoo Kim
Hello, On Thu, Jul 09, 2015 at 02:53:27PM -0700, David Rientjes wrote: The slub allocator does try to allocate its high-order memory with __GFP_WAIT before falling back to lower orders if possible. I would think that this would be the greatest sign of on-demand memory compaction being a

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-22 Thread David Rientjes
On Wed, 22 Jul 2015, Vlastimil Babka wrote: I don't think the criteria for waking up khugepaged should become any more complex beyond its current state, which is impacted by two different tunables, and whether it actually has memory to scan. During this additional wakeup, you'd also need

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-21 Thread David Rientjes
On Tue, 21 Jul 2015, Vlastimil Babka wrote: > > Khugepaged benefits from the periodic memory compaction being done > > immediately before it attempts to compact memory, and that may be lost > > with a de-coupled approach like this. > Meant to say "before it attempts to allocate a hugepage", but

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-21 Thread Vlastimil Babka
On 07/09/2015 11:53 PM, David Rientjes wrote: On Thu, 2 Jul 2015, Vlastimil Babka wrote: Memory compaction can be currently performed in several contexts: - kswapd balancing a zone after a high-order allocation failure - direct compaction to satisfy a high-order allocation, including THP page

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-21 Thread David Rientjes
On Tue, 21 Jul 2015, Vlastimil Babka wrote: Khugepaged benefits from the periodic memory compaction being done immediately before it attempts to compact memory, and that may be lost with a de-coupled approach like this. Meant to say before it attempts to allocate a hugepage, but it seems

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-21 Thread Vlastimil Babka
On 07/09/2015 11:53 PM, David Rientjes wrote: On Thu, 2 Jul 2015, Vlastimil Babka wrote: Memory compaction can be currently performed in several contexts: - kswapd balancing a zone after a high-order allocation failure - direct compaction to satisfy a high-order allocation, including THP page

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-09 Thread David Rientjes
On Thu, 2 Jul 2015, Vlastimil Babka wrote: > Memory compaction can be currently performed in several contexts: > > - kswapd balancing a zone after a high-order allocation failure > - direct compaction to satisfy a high-order allocation, including THP page > fault attemps > - khugepaged trying

Re: [RFC 1/4] mm, compaction: introduce kcompactd

2015-07-09 Thread David Rientjes
On Thu, 2 Jul 2015, Vlastimil Babka wrote: Memory compaction can be currently performed in several contexts: - kswapd balancing a zone after a high-order allocation failure - direct compaction to satisfy a high-order allocation, including THP page fault attemps - khugepaged trying to

[RFC 1/4] mm, compaction: introduce kcompactd

2015-07-02 Thread Vlastimil Babka
Memory compaction can be currently performed in several contexts: - kswapd balancing a zone after a high-order allocation failure - direct compaction to satisfy a high-order allocation, including THP page fault attemps - khugepaged trying to collapse a hugepage - manually from /proc The

[RFC 1/4] mm, compaction: introduce kcompactd

2015-07-02 Thread Vlastimil Babka
Memory compaction can be currently performed in several contexts: - kswapd balancing a zone after a high-order allocation failure - direct compaction to satisfy a high-order allocation, including THP page fault attemps - khugepaged trying to collapse a hugepage - manually from /proc The