Re: [RFC] mm: Proactive compaction

2019-09-24 Thread Khalid Aziz
On 9/24/19 7:39 AM, Vlastimil Babka wrote: > On 9/20/19 1:37 AM, Nitin Gupta wrote: >> On Tue, 2019-08-20 at 10:46 +0200, Vlastimil Babka wrote: >>> >>> That's a lot of control knobs - how is an admin supposed to tune them to >>> their >>> needs? >> >> >> Yes, it's difficult for an admin to get so

Re: [RFC] mm: Proactive compaction

2019-09-24 Thread Vlastimil Babka
On 9/20/19 1:37 AM, Nitin Gupta wrote: > On Tue, 2019-08-20 at 10:46 +0200, Vlastimil Babka wrote: >> >> That's a lot of control knobs - how is an admin supposed to tune them to >> their >> needs? > > > Yes, it's difficult for an admin to get so many tunable right unless > targeting a very

Re: [RFC] mm: Proactive compaction

2019-09-19 Thread Nitin Gupta
On Tue, 2019-08-20 at 10:46 +0200, Vlastimil Babka wrote: > > This patch is largely based on ideas from Michal Hocko posted here: > > https://lore.kernel.org/linux-mm/20161230131412.gi13...@dhcp22.suse.cz/ > > > > Testing done (on x86): > > - Set

Re: [RFC] mm: Proactive compaction

2019-09-19 Thread Nitin Gupta
On Thu, 2019-08-22 at 09:51 +0100, Mel Gorman wrote: > As unappealing as it sounds, I think it is better to try improve the > allocation latency itself instead of trying to hide the cost in a kernel > thread. It's far harder to implement as compaction is not easy but it > would be more obvious

Re: [RFC] mm: Proactive compaction

2019-09-17 Thread David Rientjes
On Tue, 17 Sep 2019, John Hubbard wrote: > > We've had good success with periodically compacting memory on a regular > > cadence on systems with hugepages enabled. The cadence itself is defined > > by the admin but it causes khugepaged[*] to periodically wakeup and invoke > > compaction in an

Re: [RFC] mm: Proactive compaction

2019-09-17 Thread John Hubbard
On 9/16/19 1:16 PM, David Rientjes wrote: > On Fri, 16 Aug 2019, Nitin Gupta wrote: ... > > We've had good success with periodically compacting memory on a regular > cadence on systems with hugepages enabled. The cadence itself is defined > by the admin but it causes khugepaged[*] to

Re: [RFC] mm: Proactive compaction

2019-09-16 Thread Nitin Gupta
On Mon, 2019-09-16 at 13:16 -0700, David Rientjes wrote: > On Fri, 16 Aug 2019, Nitin Gupta wrote: > > > For some applications we need to allocate almost all memory as > > hugepages. However, on a running system, higher order allocations can > > fail if the memory is fragmented. Linux kernel

Re: [RFC] mm: Proactive compaction

2019-09-16 Thread David Rientjes
On Fri, 16 Aug 2019, Nitin Gupta wrote: > For some applications we need to allocate almost all memory as > hugepages. However, on a running system, higher order allocations can > fail if the memory is fragmented. Linux kernel currently does > on-demand compaction as we request more hugepages but

Re: [RFC] mm: Proactive compaction

2019-08-27 Thread Nitin Gupta
On Mon, 2019-08-26 at 12:47 +0100, Mel Gorman wrote: > On Thu, Aug 22, 2019 at 09:57:22PM +, Nitin Gupta wrote: > > > Note that proactive compaction may reduce allocation latency but > > > it is not > > > free either. Even though the scanning and migration may happen in > > > a kernel > > >

Re: [RFC] mm: Proactive compaction

2019-08-26 Thread Mel Gorman
On Thu, Aug 22, 2019 at 09:57:22PM +, Nitin Gupta wrote: > > Note that proactive compaction may reduce allocation latency but it is not > > free either. Even though the scanning and migration may happen in a kernel > > thread, tasks can incur faults while waiting for compaction to complete if

Re: [RFC] mm: Proactive compaction

2019-08-24 Thread Khalid Aziz
On 8/20/19 2:46 AM, Vlastimil Babka wrote: > +CC Khalid Aziz who proposed a different approach: > https://lore.kernel.org/linux-mm/20190813014012.30232-1-khalid.a...@oracle.com/T/#u > > On 8/16/19 11:43 PM, Nitin Gupta wrote: >> The patch has plenty of rough edges but posting it early to see if

Re: [RFC] mm: Proactive compaction

2019-08-22 Thread Nitin Gupta
> ; Qian Cai ; Andrey Ryabinin > ; Roman Gushchin ; Greg Kroah- > Hartman ; Kees Cook > ; Jann Horn ; Johannes > Weiner ; Arun KS ; Janne > Huttunen ; Konstantin Khlebnikov > ; linux-kernel@vger.kernel.org; linux- > m...@kvack.org > Subject: Re: [RFC] mm: Proactive compaction >

Re: [RFC] mm: Proactive compaction

2019-08-22 Thread Mel Gorman
On Fri, Aug 16, 2019 at 02:43:30PM -0700, Nitin Gupta wrote: > For some applications we need to allocate almost all memory as > hugepages. However, on a running system, higher order allocations can > fail if the memory is fragmented. Linux kernel currently does > on-demand compaction as we request

RE: [RFC] mm: Proactive compaction

2019-08-21 Thread Nitin Gupta
ntel.com; Yu Zhao ; Qian Cai > ; Andrey Ryabinin ; Roman > Gushchin ; Greg Kroah-Hartman > ; Kees Cook ; Jann > Horn ; Johannes Weiner ; Arun > KS ; Janne Huttunen > ; Konstantin Khlebnikov > ; linux-kernel@vger.kernel.org; linux- > m...@kvack.org > Subject: Re: [RFC] mm: P

Re: [RFC] mm: Proactive compaction

2019-08-20 Thread Matthew Wilcox
On Fri, Aug 16, 2019 at 02:43:30PM -0700, Nitin Gupta wrote: > Testing done (on x86): > - Set /sys/kernel/mm/compaction/order-9/extfrag_{low,high} = {25, 30} > respectively. > - Use a test program to fragment memory: the program allocates all memory > and then for each 2M aligned section,

RE: [RFC] mm: Proactive compaction

2019-08-20 Thread Nitin Gupta
binin ; Roman > Gushchin ; Greg Kroah-Hartman > ; Kees Cook ; Jann > Horn ; Johannes Weiner ; Arun > KS ; Janne Huttunen > ; Konstantin Khlebnikov > ; linux-kernel@vger.kernel.org; linux- > m...@kvack.org; Khalid Aziz > Subject: Re: [RFC] mm: Proactive compaction >

Re: [RFC] mm: Proactive compaction

2019-08-20 Thread Vlastimil Babka
+CC Khalid Aziz who proposed a different approach: https://lore.kernel.org/linux-mm/20190813014012.30232-1-khalid.a...@oracle.com/T/#u On 8/16/19 11:43 PM, Nitin Gupta wrote: > For some applications we need to allocate almost all memory as > hugepages. However, on a running system, higher order