Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-05-07 Thread Greg Kroah-Hartman
On Tue, May 07, 2019 at 12:27:21AM -0700, Sultan Alsawaf wrote: > On Tue, May 07, 2019 at 09:04:30AM +0200, Greg Kroah-Hartman wrote: > > Um, why can't "all" Android devices take the same patches that the Pixel > > phones are using today? They should all be in the public android-common > > kernel

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-05-07 Thread Sultan Alsawaf
On Tue, May 07, 2019 at 09:04:30AM +0200, Greg Kroah-Hartman wrote: > Um, why can't "all" Android devices take the same patches that the Pixel > phones are using today? They should all be in the public android-common > kernel repositories that all Android devices should be syncing with on a >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-05-07 Thread Greg Kroah-Hartman
On Mon, May 06, 2019 at 07:16:22PM -0700, Sultan Alsawaf wrote: > This is a complete low memory killer solution for Android that is small > and simple. Processes are killed according to the priorities that > Android gives them, so that the least important processes are always >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-05-06 Thread Sultan Alsawaf
This is a complete low memory killer solution for Android that is small and simple. Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Processes are killed until memory deficits are satisfied, as observed from

Re: [PATCH v2 2/5] mips: Perform early low memory test

2019-05-06 Thread Paul Burton
Hello, Serge Semin wrote: > memblock subsystem provides a method to optionally test the passed > memory region in case if it was requested via special kernel boot > argument. Lets add the function at the bottom of the arch_mem_init() > method. Testing at this point in the boot sequence should be

[PATCH v2 2/5] mips: Perform early low memory test

2019-05-03 Thread Serge Semin
memblock subsystem provides a method to optionally test the passed memory region in case if it was requested via special kernel boot argument. Lets add the function at the bottom of the arch_mem_init() method. Testing at this point in the boot sequence should be safe since all critical areas are

[PATCH 09/12] mips: Perform early low memory test

2019-04-23 Thread Serge Semin
memblock subsystem provides a method to optionally test the passed memory region in case if it was requested via special kernel boot argument. Lets add the function at the bottom of the arch_mem_init() method. Testing at this point in the boot sequence should be safe since all critical areas are

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-04-05 Thread Catalin Marinas
On Mon, Apr 01, 2019 at 10:12:01PM +0200, Michal Hocko wrote: > On Fri 29-03-19 16:16:38, Catalin Marinas wrote: > > On Fri, Mar 29, 2019 at 01:02:37PM +0100, Michal Hocko wrote: > > > On Thu 28-03-19 14:59:17, Catalin Marinas wrote: > > > [...] > > > > >From

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-04-01 Thread Michal Hocko
On Fri 29-03-19 16:16:38, Catalin Marinas wrote: > On Fri, Mar 29, 2019 at 01:02:37PM +0100, Michal Hocko wrote: > > On Thu 28-03-19 14:59:17, Catalin Marinas wrote: > > [...] > > > >From 09eba8f0235eb16409931e6aad77a45a12bedc82 Mon Sep 17 00:00:00 2001 > > > From: Catalin Marinas > > > Date:

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-29 Thread Catalin Marinas
On Fri, Mar 29, 2019 at 01:02:37PM +0100, Michal Hocko wrote: > On Thu 28-03-19 14:59:17, Catalin Marinas wrote: > [...] > > >From 09eba8f0235eb16409931e6aad77a45a12bedc82 Mon Sep 17 00:00:00 2001 > > From: Catalin Marinas > > Date: Thu, 28 Mar 2019 13:26:07 + > > Subject: [PATCH] mm:

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-29 Thread Michal Hocko
On Thu 28-03-19 14:59:17, Catalin Marinas wrote: [...] > >From 09eba8f0235eb16409931e6aad77a45a12bedc82 Mon Sep 17 00:00:00 2001 > From: Catalin Marinas > Date: Thu, 28 Mar 2019 13:26:07 + > Subject: [PATCH] mm: kmemleak: Use mempool allocations for kmemleak objects > > This patch adds

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Qian Cai
On Thu, 2019-03-28 at 15:05 +, Catalin Marinas wrote: > > It takes 2 runs of LTP oom01 tests to disable kmemleak. > > What configuration are you using (number of CPUs, RAM)? I tried this on > an arm64 guest under kvm with 4 CPUs and 512MB of RAM, together with > fault injection on

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Catalin Marinas
t; > provides a good balance between enabling kmemleak in a low-memory > > > > situation and not introducing too much hackiness into the existing > > > > code for now. > > On Thu, Mar 28, 2019 at 08:05:31AM +0200, Pekka Enberg wrote: > > > Unfortunatel

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Qian Cai
On Thu, 2019-03-28 at 14:59 +, Catalin Marinas wrote: 2 > +/* minimum memory pool sizes */ > +#define MIN_OBJECT_POOL (NR_CPUS * 4) > +#define MIN_SCAN_AREA_POOL (NR_CPUS * 1) I am thinking about making those are tunable, so people could have a big pool depends on

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Catalin Marinas
On Wed, Mar 27, 2019 at 02:02:27PM -0400, Qian Cai wrote: > On 3/27/19 1:29 PM, Catalin Marinas wrote: > > From dc4194539f8191bb754901cea74c86e7960886f8 Mon Sep 17 00:00:00 2001 > > From: Catalin Marinas > > Date: Wed, 27 Mar 2019 17:20:57 + > > Subject: [PATCH] mm: kmemleak: Add an emergency

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Catalin Marinas
On Wed, Mar 27, 2019 at 11:21:58AM -0700, Matthew Wilcox wrote: > On Wed, Mar 27, 2019 at 05:29:57PM +, Catalin Marinas wrote: > > On Wed, Mar 27, 2019 at 09:44:32AM +0100, Michal Hocko wrote: > > > As long as there is an implicit __GFP_NOFAIL then kmemleak is simply > > > broken no matter

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Pekka Enberg
Hi Catalin, On 27/03/2019 2.59, Qian Cai wrote: Unless there is a brave soul to reimplement the kmemleak to embed it's metadata into the tracked memory itself in a foreseeable future, this provides a good balance between enabling kmemleak in a low-memory situation and not introducing too much

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Catalin Marinas
ance between enabling kmemleak in a low-memory > > situation and not introducing too much hackiness into the existing > > code for now. > > Unfortunately I am not that brave soul, but I'm wondering what the > complication here is? It shouldn't be too hard to teach calculate_sizes(

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-28 Thread Pekka Enberg
Hi, On 27/03/2019 2.59, Qian Cai wrote: Unless there is a brave soul to reimplement the kmemleak to embed it's metadata into the tracked memory itself in a foreseeable future, this provides a good balance between enabling kmemleak in a low-memory situation and not introducing too much hackiness

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Qian Cai
On 3/27/19 1:29 PM, Catalin Marinas wrote: > From dc4194539f8191bb754901cea74c86e7960886f8 Mon Sep 17 00:00:00 2001 > From: Catalin Marinas > Date: Wed, 27 Mar 2019 17:20:57 + > Subject: [PATCH] mm: kmemleak: Add an emergency allocation pool for kmemleak > objects > > This patch adds an

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Michal Hocko
On Wed 27-03-19 17:29:57, Catalin Marinas wrote: [...] > Quick attempt below and it needs some more testing (pretty random pick > of the EMERGENCY_POOL_SIZE value). Also, with __GFP_NOFAIL removed, are > the other flags safe or we should trim them further? I would be still careful about

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Matthew Wilcox
On Wed, Mar 27, 2019 at 05:29:57PM +, Catalin Marinas wrote: > On Wed, Mar 27, 2019 at 09:44:32AM +0100, Michal Hocko wrote: > > As long as there is an implicit __GFP_NOFAIL then kmemleak is simply > > broken no matter what other gfp flags you play with. Has anybody looked > > at some sort of

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Catalin Marinas
ak memory scanning code, that's not actually possible without some long periods with kmemleak_lock held. The scanner relies on the kmemleak_object (refcounted) being around even when the actual memory block has been freed. > > this > > provides a good balance between enabling kmemleak

[PATCH v5] kmemleak: survive in a low-memory situation

2019-03-27 Thread Qian Cai
Kmemleak could quickly fail to allocate an object structure and then disable itself below in a low-memory situation. For example, running a mmap() workload triggering swapping and OOM. This is especially problematic for running things like LTP testsuite where one OOM test case would disable

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Michal Hocko
On Wed 27-03-19 09:05:31, Qian Cai wrote: > On 3/27/19 7:44 AM, Michal Hocko wrote> What? Normal spin lock implementation > doesn't disable interrupts. So > > either I misunderstand what you are saying or you seem to be confused. > > the thing is that in_atomic relies on preempt_count to work

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Qian Cai
On 3/27/19 7:44 AM, Michal Hocko wrote> What? Normal spin lock implementation doesn't disable interrupts. So > either I misunderstand what you are saying or you seem to be confused. > the thing is that in_atomic relies on preempt_count to work properly and > if you have CONFIG_PREEMPT_COUNT=n then

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Michal Hocko
On Wed 27-03-19 07:34:32, Qian Cai wrote: > On 3/27/19 4:44 AM, Michal Hocko wrote: > >> diff --git a/mm/kmemleak.c b/mm/kmemleak.c > >> index a2d894d3de07..7f4545ab1f84 100644 > >> --- a/mm/kmemleak.c > >> +++ b/mm/kmemleak.c > >> @@ -580,7 +580,16 @@ static struct kmemleak_object

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Qian Cai
On 3/27/19 4:44 AM, Michal Hocko wrote: >> diff --git a/mm/kmemleak.c b/mm/kmemleak.c >> index a2d894d3de07..7f4545ab1f84 100644 >> --- a/mm/kmemleak.c >> +++ b/mm/kmemleak.c >> @@ -580,7 +580,16 @@ static struct kmemleak_object *create_object(unsigned >> long ptr, size_t size, >> struct

Re: [PATCH v4] kmemleak: survive in a low-memory situation

2019-03-27 Thread Michal Hocko
On Tue 26-03-19 20:59:48, Qian Cai wrote: [...] > Unless there is a brave soul to reimplement the kmemleak to embed it's > metadata into the tracked memory itself in a foreseeable future, this > provides a good balance between enabling kmemleak in a low-memory > situation and not int

[PATCH v4] kmemleak: survive in a low-memory situation

2019-03-26 Thread Qian Cai
Kmemleak could quickly fail to allocate an object structure and then disable itself below in a low-memory situation. For example, running a mmap() workload triggering swapping and OOM. This is especially problematic for running things like LTP testsuite where one OOM test case would disable

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Qian Cai
. I suppose this is not going to work well, as DMA_API_DEBUG use a similar approach [1] but I still saw it is struggling in a low-memory situation and disable itself occasionally. [1] https://lkml.org/lkml/2018/12/10/383

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Michal Hocko
ata into the tracked memory itself in a foreseeable future, this > > > provides a good balance between enabling kmemleak in a low-memory > > > situation and not introducing too much hackiness into the existing > > > code for now. > > > > I don't understand k

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Qian Cai
On 3/26/19 12:00 PM, Christopher Lameter wrote: >> + */ >> +gfp = (in_atomic() || irqs_disabled()) ? GFP_ATOMIC : >> + gfp_kmemleak_mask(gfp) | __GFP_DIRECT_RECLAIM; >> +object = kmem_cache_alloc(object_cache, gfp); >> +} >> + >> if

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Catalin Marinas
> provides a good balance between enabling kmemleak in a low-memory > > situation and not introducing too much hackiness into the existing > > code for now. > > I don't understand kmemleak. Kirill pointed me at this a few days ago: > > https://gist.github.com/kiryl/3225

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Catalin Marinas
On Tue, Mar 26, 2019 at 11:43:38AM -0400, Qian Cai wrote: > Kmemleak could quickly fail to allocate an object structure and then > disable itself in a low-memory situation. For example, running a mmap() > workload triggering swapping and OOM. This is especially problematic for > ru

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Matthew Wilcox
On Tue, Mar 26, 2019 at 11:43:38AM -0400, Qian Cai wrote: > Unless there is a brave soul to reimplement the kmemleak to embed it's > metadata into the tracked memory itself in a foreseeable future, this > provides a good balance between enabling kmemleak in a low-memory &g

Re: [PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Christopher Lameter
On Tue, 26 Mar 2019, Qian Cai wrote: > + if (!object) { > + /* > + * The tracked memory was allocated successful, if the kmemleak > + * object failed to allocate for some reasons, it ends up with > + * the whole kmemleak disabled, so let it

[PATCH v3] kmemleaak: survive in a low-memory situation

2019-03-26 Thread Qian Cai
Kmemleak could quickly fail to allocate an object structure and then disable itself in a low-memory situation. For example, running a mmap() workload triggering swapping and OOM. This is especially problematic for running things like LTP testsuite where one OOM test case would disable the whole

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-19 Thread Joel Fernandes
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner wrote: > On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione wrote: > > On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner > > wrote: > > > So I dislike the idea of allocating new inodes from the procfs super > > > block. I would

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-19 Thread Christian Brauner
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione wrote: > On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner > wrote: > > So I dislike the idea of allocating new inodes from the procfs super > > block. I would like to avoid pinning the whole pidfd concept exclusively > > to proc. The

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-19 Thread Daniel Colascione
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner wrote: > So I dislike the idea of allocating new inodes from the procfs super > block. I would like to avoid pinning the whole pidfd concept exclusively > to proc. The idea is that the pidfd API will be useable through procfs > via open("/proc/")

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-19 Thread Joel Fernandes
On Tue, Mar 19, 2019 at 11:14:17PM +0100, Christian Brauner wrote: [snip] > > > > ---8<--- > > > > From: Joel Fernandes > > Subject: [PATCH] Partial skeleton prototype of pidfd_wait frontend > > > > Signed-off-by: Joel Fernandes > > --- > >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-19 Thread Christian Brauner
On Mon, Mar 18, 2019 at 07:50:52PM -0400, Joel Fernandes wrote: > On Mon, Mar 18, 2019 at 01:29:51AM +0100, Christian Brauner wrote: > > On Sun, Mar 17, 2019 at 08:40:19AM -0700, Daniel Colascione wrote: > > > On Sun, Mar 17, 2019 at 4:42 AM Christian Brauner > > > wrote: > > > > > > > > On Sat,

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-18 Thread Joel Fernandes
On Mon, Mar 18, 2019 at 01:29:51AM +0100, Christian Brauner wrote: > On Sun, Mar 17, 2019 at 08:40:19AM -0700, Daniel Colascione wrote: > > On Sun, Mar 17, 2019 at 4:42 AM Christian Brauner > > wrote: > > > > > > On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > > > > On Sat, Mar

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Christian Brauner
On Sun, Mar 17, 2019 at 08:40:19AM -0700, Daniel Colascione wrote: > On Sun, Mar 17, 2019 at 4:42 AM Christian Brauner > wrote: > > > > On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > > > On Sat, Mar 16, 2019 at 12:37:18PM -0700, Suren Baghdasaryan wrote: > > > > On Sat, Mar

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Suren Baghdasaryan
On Sun, Mar 17, 2019 at 10:16 AM Serge E. Hallyn wrote: > > On Sun, Mar 17, 2019 at 10:11:10AM -0700, Daniel Colascione wrote: > > On Sun, Mar 17, 2019 at 9:35 AM Serge E. Hallyn wrote: > > > > > > On Sun, Mar 17, 2019 at 12:42:40PM +0100, Christian Brauner wrote: > > > > On Sat, Mar 16, 2019 at

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Serge E. Hallyn
On Sun, Mar 17, 2019 at 10:11:10AM -0700, Daniel Colascione wrote: > On Sun, Mar 17, 2019 at 9:35 AM Serge E. Hallyn wrote: > > > > On Sun, Mar 17, 2019 at 12:42:40PM +0100, Christian Brauner wrote: > > > On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > > > > On Sat, Mar 16, 2019

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Daniel Colascione
On Sun, Mar 17, 2019 at 9:35 AM Serge E. Hallyn wrote: > > On Sun, Mar 17, 2019 at 12:42:40PM +0100, Christian Brauner wrote: > > On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > > > On Sat, Mar 16, 2019 at 12:37:18PM -0700, Suren Baghdasaryan wrote: > > > > On Sat, Mar 16, 2019

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Serge E. Hallyn
On Sun, Mar 17, 2019 at 12:42:40PM +0100, Christian Brauner wrote: > On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > > On Sat, Mar 16, 2019 at 12:37:18PM -0700, Suren Baghdasaryan wrote: > > > On Sat, Mar 16, 2019 at 11:57 AM Christian Brauner > > > wrote: > > > > > > > > On

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Daniel Colascione
On Sun, Mar 17, 2019 at 4:42 AM Christian Brauner wrote: > > On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > > On Sat, Mar 16, 2019 at 12:37:18PM -0700, Suren Baghdasaryan wrote: > > > On Sat, Mar 16, 2019 at 11:57 AM Christian Brauner > > > wrote: > > > > > > > > On Sat, Mar

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-17 Thread Christian Brauner
On Sat, Mar 16, 2019 at 09:53:06PM -0400, Joel Fernandes wrote: > On Sat, Mar 16, 2019 at 12:37:18PM -0700, Suren Baghdasaryan wrote: > > On Sat, Mar 16, 2019 at 11:57 AM Christian Brauner > > wrote: > > > > > > On Sat, Mar 16, 2019 at 11:00:10AM -0700, Daniel Colascione wrote: > > > > On Sat,

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-16 Thread Joel Fernandes
On Sat, Mar 16, 2019 at 12:37:18PM -0700, Suren Baghdasaryan wrote: > On Sat, Mar 16, 2019 at 11:57 AM Christian Brauner > wrote: > > > > On Sat, Mar 16, 2019 at 11:00:10AM -0700, Daniel Colascione wrote: > > > On Sat, Mar 16, 2019 at 10:31 AM Suren Baghdasaryan > > > wrote: > > > > > > > > On

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-16 Thread Suren Baghdasaryan
On Sat, Mar 16, 2019 at 11:57 AM Christian Brauner wrote: > > On Sat, Mar 16, 2019 at 11:00:10AM -0700, Daniel Colascione wrote: > > On Sat, Mar 16, 2019 at 10:31 AM Suren Baghdasaryan > > wrote: > > > > > > On Fri, Mar 15, 2019 at 11:49 AM Joel Fernandes > > > wrote: > > > > > > > > On Fri,

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-16 Thread Christian Brauner
On Sat, Mar 16, 2019 at 11:00:10AM -0700, Daniel Colascione wrote: > On Sat, Mar 16, 2019 at 10:31 AM Suren Baghdasaryan wrote: > > > > On Fri, Mar 15, 2019 at 11:49 AM Joel Fernandes > > wrote: > > > > > > On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote: > > > [..] > > > > >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-16 Thread Daniel Colascione
On Sat, Mar 16, 2019 at 10:31 AM Suren Baghdasaryan wrote: > > On Fri, Mar 15, 2019 at 11:49 AM Joel Fernandes > wrote: > > > > On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote: > > [..] > > > > why do we want to add a new syscall (pidfd_wait) though? Why not just > > > > use

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-16 Thread Suren Baghdasaryan
On Fri, Mar 15, 2019 at 11:49 AM Joel Fernandes wrote: > > On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote: > [..] > > > why do we want to add a new syscall (pidfd_wait) though? Why not just use > > > standard poll/epoll interface on the proc fd like Daniel was suggesting. > > >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote: [..] > > why do we want to add a new syscall (pidfd_wait) though? Why not just use > > standard poll/epoll interface on the proc fd like Daniel was suggesting. > > AFAIK, once the proc file is opened, the struct pid is essentially

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Christian Brauner
that we're finally getting it. > > > Adding new system calls is not expensive; going to great lengths to > > > avoid adding one is like calling a helicopter to avoid crossing the > > > street. I don't think we should present an abuse of the debugging and > > >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Joel Fernandes
> > performance monitoring infrastructure as an alternative to a robust > > and desperately-needed bit of core functionality that's neither hard > > to add nor complex to implement nor expensive to use. > > > > Regarding the proposal for a new kernel-side lmkd: when poss

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Christian Brauner
d present an abuse of the debugging and > performance monitoring infrastructure as an alternative to a robust > and desperately-needed bit of core functionality that's neither hard > to add nor complex to implement nor expensive to use. > > Regarding the proposal for a new kernel-

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Daniel Colascione
On Fri, Mar 15, 2019 at 9:43 AM Steven Rostedt wrote: > > On Thu, 14 Mar 2019 21:36:43 -0700 > Daniel Colascione wrote: > > > On Thu, Mar 14, 2019 at 8:16 PM Steven Rostedt wrote: > > > > > > On Thu, 14 Mar 2019 13:49:11 -0700 > > > Sultan Alsawaf wrote: > > > > > > > Perhaps I'm missing

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Steven Rostedt
On Thu, 14 Mar 2019 21:36:43 -0700 Daniel Colascione wrote: > On Thu, Mar 14, 2019 at 8:16 PM Steven Rostedt wrote: > > > > On Thu, 14 Mar 2019 13:49:11 -0700 > > Sultan Alsawaf wrote: > > > > > Perhaps I'm missing something, but if you want to know when a process has > > > died > > > after

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Daniel Colascione
gt; > performance monitoring infrastructure as an alternative to a robust > > and desperately-needed bit of core functionality that's neither hard > > to add nor complex to implement nor expensive to use. > > > > Regarding the proposal for a new kernel-side lmkd: when

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Suren Baghdasaryan
an abuse of the debugging and > performance monitoring infrastructure as an alternative to a robust > and desperately-needed bit of core functionality that's neither hard > to add nor complex to implement nor expensive to use. > > Regarding the proposal for a new kernel-side lm

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-15 Thread Joel Fernandes
roposal for a new kernel-side lmkd: when possible, the > kernel should provide mechanism, not policy. Putting the low memory > killer back into the kernel after we've spent significant effort > making it possible for userspace to do that job. Compared to kernel > code, more easily under

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Daniel Colascione
kd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updat

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Sultan Alsawaf
On Thu, Mar 14, 2019 at 11:16:41PM -0400, Steven Rostedt wrote: > How would you implement such a method in userspace? kill() doesn't take > any parameters but the pid of the process you want to send a signal to, > and the signal to send. This would require a new system call, and be > quite a bit

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Sultan Alsawaf
On Thu, Mar 14, 2019 at 10:54:48PM -0400, Joel Fernandes wrote: > I'm not sure if that makes much semantic sense for how the signal handling is > supposed to work. Imagine a parent sends SIGKILL to its child, and then does > a wait(2). Because the SIGKILL blocks in your idea, then the wait cannot

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Steven Rostedt
On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf wrote: > Perhaps I'm missing something, but if you want to know when a process has died > after sending a SIGKILL to it, then why not just make the SIGKILL optionally > block until the process has died completely? It'd be rather trivial to just >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Joel Fernandes
On Thu, Mar 14, 2019 at 01:49:11PM -0700, Sultan Alsawaf wrote: > On Thu, Mar 14, 2019 at 10:47:17AM -0700, Joel Fernandes wrote: > > About the 100ms latency, I wonder whether it is that high because of > > the way Android's lmkd is observing that a process has died. There is > > a gap between

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Sultan Alsawaf
On Thu, Mar 14, 2019 at 10:47:17AM -0700, Joel Fernandes wrote: > About the 100ms latency, I wonder whether it is that high because of > the way Android's lmkd is observing that a process has died. There is > a gap between when a process memory is freed and when it disappears > from the

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-14 Thread Joel Fernandes
Hi Tim, Thanks for the detailed and excellent write-up. It will serve as a good future reference for low memory killer requirements. I made some comments below on the "how to kill" part. On Tue, Mar 12, 2019 at 10:17 AM Tim Murray wrote: > > On Tue, Mar 12, 2019 at 9:37 AM Sulta

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Christian Brauner
On Tue, Mar 12, 2019 at 7:43 PM Tim Murray wrote: > > On Tue, Mar 12, 2019 at 10:45 AM Sultan Alsawaf > wrote: > > > > On Tue, Mar 12, 2019 at 10:17:43AM -0700, Tim Murray wrote: > > > Knowing whether a SIGKILL'd process has finished reclaiming is as far > > > as I know not possible without

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Tim Murray
On Tue, Mar 12, 2019 at 10:45 AM Sultan Alsawaf wrote: > > On Tue, Mar 12, 2019 at 10:17:43AM -0700, Tim Murray wrote: > > Knowing whether a SIGKILL'd process has finished reclaiming is as far > > as I know not possible without something like procfds. That's where > > the 100ms timeout in lmkd

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Sultan Alsawaf
On Tue, Mar 12, 2019 at 10:17:43AM -0700, Tim Murray wrote: > Knowing whether a SIGKILL'd process has finished reclaiming is as far > as I know not possible without something like procfds. That's where > the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both > attempt to wait up to

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Suren Baghdasaryan
On Tue, Mar 12, 2019 at 9:58 AM Michal Hocko wrote: > > On Tue 12-03-19 09:37:41, Sultan Alsawaf wrote: > > I have not had a chance to look at PSI yet, but > > unless a PSI-enabled solution allows allocations to reach the same point as > > when > > the OOM killer is invoked (which is

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Tim Murray
On Tue, Mar 12, 2019 at 9:37 AM Sultan Alsawaf wrote: > > On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > > The only way to control the OOM behavior pro-actively is to throttle > > allocation speed. We have memcg high limit for that purpose. Along with > > PSI, I can imagine a

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Michal Hocko
On Tue 12-03-19 09:37:41, Sultan Alsawaf wrote: > I have not had a chance to look at PSI yet, but > unless a PSI-enabled solution allows allocations to reach the same point as > when > the OOM killer is invoked (which is contradictory to what it sets out to do), > then it cannot take advantage of

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Michal Hocko
On Tue 12-03-19 09:37:41, Sultan Alsawaf wrote: > On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > > The only way to control the OOM behavior pro-actively is to throttle > > allocation speed. We have memcg high limit for that purpose. Along with > > PSI, I can imagine a reasonably

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Sultan Alsawaf
On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > The only way to control the OOM behavior pro-actively is to throttle > allocation speed. We have memcg high limit for that purpose. Along with > PSI, I can imagine a reasonably working user space early oom > notifications and

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Michal Hocko
On Tue 12-03-19 16:33:15, Michal Hocko wrote: > On Tue 12-03-19 08:25:41, Matthew Wilcox wrote: > > On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > > > On Mon 11-03-19 15:15:35, Suren Baghdasaryan wrote: > > > > Yeah, killing speed is a well-known problem which we are considering >

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Michal Hocko
On Tue 12-03-19 08:25:41, Matthew Wilcox wrote: > On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > > On Mon 11-03-19 15:15:35, Suren Baghdasaryan wrote: > > > Yeah, killing speed is a well-known problem which we are considering > > > in LMKD. For example the recent LMKD change to

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Matthew Wilcox
On Tue, Mar 12, 2019 at 09:05:32AM +0100, Michal Hocko wrote: > On Mon 11-03-19 15:15:35, Suren Baghdasaryan wrote: > > Yeah, killing speed is a well-known problem which we are considering > > in LMKD. For example the recent LMKD change to assign process being > > killed to a cpuset cgroup

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Suren Baghdasaryan
On Tue, Mar 12, 2019 at 1:05 AM Michal Hocko wrote: > > On Mon 11-03-19 15:15:35, Suren Baghdasaryan wrote: > > On Mon, Mar 11, 2019 at 1:46 PM Sultan Alsawaf > > wrote: > > > > > > On Mon, Mar 11, 2019 at 01:10:36PM -0700, Suren Baghdasaryan wrote: > > > > The idea seems interesting although I

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-12 Thread Michal Hocko
On Mon 11-03-19 15:15:35, Suren Baghdasaryan wrote: > On Mon, Mar 11, 2019 at 1:46 PM Sultan Alsawaf wrote: > > > > On Mon, Mar 11, 2019 at 01:10:36PM -0700, Suren Baghdasaryan wrote: > > > The idea seems interesting although I need to think about this a bit > > > more. Killing processes based on

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Sultan Alsawaf
On Mon, Mar 11, 2019 at 03:15:35PM -0700, Suren Baghdasaryan wrote: > This what LMKD currently is - a userspace RT process. > My point was that this page allocation queue that you implemented > can't be implemented in userspace, at least not without extensive > communication with kernel. Oh,

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Suren Baghdasaryan
On Mon, Mar 11, 2019 at 1:46 PM Sultan Alsawaf wrote: > > On Mon, Mar 11, 2019 at 01:10:36PM -0700, Suren Baghdasaryan wrote: > > The idea seems interesting although I need to think about this a bit > > more. Killing processes based on failed page allocation might backfire > > during transient

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Sultan Alsawaf
On Mon, Mar 11, 2019 at 05:11:25PM -0400, Joel Fernandes wrote: > But the point is that a transient temporary memory spike should not be a > signal to kill _any_ process. The reaction to kill shouldn't be so > spontaneous that unwanted tasks are killed because the system went into > panic mode.

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Joel Fernandes
On Mon, Mar 11, 2019 at 01:46:26PM -0700, Sultan Alsawaf wrote: > On Mon, Mar 11, 2019 at 01:10:36PM -0700, Suren Baghdasaryan wrote: > > The idea seems interesting although I need to think about this a bit > > more. Killing processes based on failed page allocation might backfire > > during

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Sultan Alsawaf
On Mon, Mar 11, 2019 at 01:10:36PM -0700, Suren Baghdasaryan wrote: > The idea seems interesting although I need to think about this a bit > more. Killing processes based on failed page allocation might backfire > during transient spikes in memory usage. This issue could be alleviated if tasks

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Suren Baghdasaryan
Hi Sultan, On Mon, Mar 11, 2019 at 10:58 AM Sultan Alsawaf wrote: > > On Mon, Mar 11, 2019 at 06:43:20PM +0100, Michal Hocko wrote: > > I am sorry but we are not going to maintain two different OOM > > implementations in the kernel. From a quick look the implementation is > > quite a hack which

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Sultan Alsawaf
On Mon, Mar 11, 2019 at 06:43:20PM +0100, Michal Hocko wrote: > I am sorry but we are not going to maintain two different OOM > implementations in the kernel. From a quick look the implementation is > quite a hack which is not really suitable for anything but a very > specific usecase. E.g.

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Michal Hocko
On Sun 10-03-19 13:34:03, Sultan Alsawaf wrote: > From: Sultan Alsawaf > > This is a complete low memory killer solution for Android that is small > and simple. It kills the largest, least-important processes it can find > whenever a page allocation has completely failed (rig

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Joel Fernandes
On Mon, Mar 11, 2019 at 12:32:33PM -0400, Joel Fernandes wrote: > On Sun, Mar 10, 2019 at 01:34:03PM -0700, Sultan Alsawaf wrote: > [...] > > > > /* Perform scheduler related setup. Assign this task to a CPU. */ > > retval = sched_fork(clone_flags, p); > > diff --git a/mm/page_alloc.c

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-11 Thread Joel Fernandes
On Sun, Mar 10, 2019 at 01:34:03PM -0700, Sultan Alsawaf wrote: [...] > > /* Perform scheduler related setup. Assign this task to a CPU. */ > retval = sched_fork(clone_flags, p); > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 3eb01dedf..fd0d697c6 100644 > ---

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-10 Thread Sultan Alsawaf
On Sun, Mar 10, 2019 at 10:03:35PM +0100, Greg Kroah-Hartman wrote: > On Sun, Mar 10, 2019 at 01:34:03PM -0700, Sultan Alsawaf wrote: > > From: Sultan Alsawaf > > > > This is a complete low memory killer solution for Android that is small > > and simple. It kills t

Re: [RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-10 Thread Greg Kroah-Hartman
On Sun, Mar 10, 2019 at 01:34:03PM -0700, Sultan Alsawaf wrote: > From: Sultan Alsawaf > > This is a complete low memory killer solution for Android that is small > and simple. It kills the largest, least-important processes it can find > whenever a page allocation has completel

[RFC] simple_lmk: Introduce Simple Low Memory Killer for Android

2019-03-10 Thread Sultan Alsawaf
From: Sultan Alsawaf This is a complete low memory killer solution for Android that is small and simple. It kills the largest, least-important processes it can find whenever a page allocation has completely failed (right after direct reclaim). Processes are killed according to the priorities

Re: [PATCH v2] kmemleak: survive in a low-memory situation

2019-01-07 Thread Qian Cai
object(unsigned long ptr, size_t size, >>>>>> struct rb_node **link, *rb_parent; >>>>>> >>>>>> object = kmem_cache_alloc(object_cache, gfp_kmemleak_mask(gfp)); >>>>>> +#ifdef CONFIG_PREEMPT_COUNT >>

Re: [PATCH v2] kmemleak: survive in a low-memory situation

2019-01-07 Thread Qian Cai
ink, *rb_parent; >>>>> >>>>> object = kmem_cache_alloc(object_cache, gfp_kmemleak_mask(gfp)); >>>>> +#ifdef CONFIG_PREEMPT_COUNT >>>>> + if (!object) { >>>>> + /* last-ditch effort in a low-memory situation */ >>

<    1   2   3   4   5   6   7   >