Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-18 Thread Dave Chinner
On Fri, Apr 16, 2021 at 10:14:39AM +0530, Bharata B Rao wrote: > On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > > On Mon, Apr 05, 2021 at 11:18:48AM +0530, Bharata B Rao wrote: > > > > > As an alternative approach, I have this below hack that does lazy > > > list_lru creation.

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-15 Thread Bharata B Rao
On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > On Mon, Apr 05, 2021 at 11:18:48AM +0530, Bharata B Rao wrote: > > > As an alternative approach, I have this below hack that does lazy > > list_lru creation. The memcg-specific list is created and initialized > > only when there is a

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-15 Thread Bharata B Rao
On Thu, Apr 15, 2021 at 08:54:43AM +0200, Michal Hocko wrote: > On Thu 15-04-21 10:53:00, Bharata B Rao wrote: > > On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > > > > > > Another approach may be to identify filesystem types that do not > > > need memcg awareness and feed that

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-15 Thread Michal Hocko
On Thu 15-04-21 10:53:00, Bharata B Rao wrote: > On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > > > > Another approach may be to identify filesystem types that do not > > need memcg awareness and feed that into alloc_super() to set/clear > > the SHRINKER_MEMCG_AWARE flag. This

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-14 Thread Bharata B Rao
On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > > Another approach may be to identify filesystem types that do not > need memcg awareness and feed that into alloc_super() to set/clear > the SHRINKER_MEMCG_AWARE flag. This could be based on fstype - most > virtual filesystems that

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Shakeel Butt
On Wed, Apr 7, 2021 at 4:55 AM Michal Hocko wrote: > > On Mon 05-04-21 11:18:48, Bharata B Rao wrote: > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > > consumption increases quite a

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Christian Brauner
On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > On Mon, Apr 05, 2021 at 11:18:48AM +0530, Bharata B Rao wrote: > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > >

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Michal Hocko
On Wed 07-04-21 19:13:42, Bharata B Rao wrote: > On Wed, Apr 07, 2021 at 01:54:48PM +0200, Michal Hocko wrote: > > On Mon 05-04-21 11:18:48, Bharata B Rao wrote: > > > Hi, > > > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > > server(160 CPUs, 2 NUMA nodes,

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Bharata B Rao
On Wed, Apr 07, 2021 at 01:54:48PM +0200, Michal Hocko wrote: > On Mon 05-04-21 11:18:48, Bharata B Rao wrote: > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > > consumption

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Christian Brauner
On Wed, Apr 07, 2021 at 01:54:48PM +0200, Michal Hocko wrote: > On Mon 05-04-21 11:18:48, Bharata B Rao wrote: > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > > consumption

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Kirill Tkhai
On 07.04.2021 14:47, Bharata B Rao wrote: > On Wed, Apr 07, 2021 at 01:07:27PM +0300, Kirill Tkhai wrote: >>> Here is how the calculation turns out to be in my setup: >>> >>> Number of possible NUMA nodes = 2 >>> Number of mounts per container = 7 (Check below to see which are these) >>> Number of

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Michal Hocko
On Mon 05-04-21 11:18:48, Bharata B Rao wrote: > Hi, > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > consumption increases quite a lot (around 172G) when the containers are > running. Most of it

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Bharata B Rao
On Wed, Apr 07, 2021 at 01:07:27PM +0300, Kirill Tkhai wrote: > > Here is how the calculation turns out to be in my setup: > > > > Number of possible NUMA nodes = 2 > > Number of mounts per container = 7 (Check below to see which are these) > > Number of list creation requests per mount = 2 > >

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-07 Thread Kirill Tkhai
On 07.04.2021 08:05, Bharata B Rao wrote: > On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: >> On Mon, Apr 05, 2021 at 11:18:48AM +0530, Bharata B Rao wrote: >>> Hi, >>> >>> When running 1 (more-or-less-empty-)containers on a bare-metal Power9 >>> server(160 CPUs, 2 NUMA nodes,

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-06 Thread Bharata B Rao
On Wed, Apr 07, 2021 at 08:28:07AM +1000, Dave Chinner wrote: > On Mon, Apr 05, 2021 at 11:18:48AM +0530, Bharata B Rao wrote: > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > >

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-06 Thread Yang Shi
On Tue, Apr 6, 2021 at 3:05 AM Bharata B Rao wrote: > > On Mon, Apr 05, 2021 at 11:08:26AM -0700, Yang Shi wrote: > > On Sun, Apr 4, 2021 at 10:49 PM Bharata B Rao wrote: > > > > > > Hi, > > > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > > server(160 CPUs,

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-06 Thread Dave Chinner
On Mon, Apr 05, 2021 at 11:18:48AM +0530, Bharata B Rao wrote: > Hi, > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > consumption increases quite a lot (around 172G) when the containers are >

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-06 Thread Bharata B Rao
On Mon, Apr 05, 2021 at 11:38:44AM -0700, Roman Gushchin wrote: > > > @@ -534,7 +521,17 @@ static void memcg_drain_list_lru_node(struct > > > list_lru *lru, int nid, > > > spin_lock_irq(>lock); > > > > > > src = list_lru_from_memcg_idx(nlru, src_idx); > > > + if (!src) > > >

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-06 Thread Bharata B Rao
On Mon, Apr 05, 2021 at 11:08:26AM -0700, Yang Shi wrote: > On Sun, Apr 4, 2021 at 10:49 PM Bharata B Rao wrote: > > > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > > consumption

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-05 Thread Roman Gushchin
On Mon, Apr 05, 2021 at 11:08:26AM -0700, Yang Shi wrote: > On Sun, Apr 4, 2021 at 10:49 PM Bharata B Rao wrote: > > > > Hi, > > > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > > consumption

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-05 Thread Yang Shi
On Sun, Apr 4, 2021 at 10:49 PM Bharata B Rao wrote: > > Hi, > > When running 1 (more-or-less-empty-)containers on a bare-metal Power9 > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > consumption increases quite a lot (around 172G) when the containers are > running.

Re: High kmalloc-32 slab cache consumption with 10k containers

2021-04-05 Thread kernel test robot
submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Bharata-B-Rao/High-kmalloc-32-slab-cache-consumption-with-10k-containers/20210405-135124 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds

High kmalloc-32 slab cache consumption with 10k containers

2021-04-04 Thread Bharata B Rao
Hi, When running 1 (more-or-less-empty-)containers on a bare-metal Power9 server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory consumption increases quite a lot (around 172G) when the containers are running. Most of it comes from slab (149G) and within slab, the majority of it