On Fri, Jan 29, 2021 at 09:05:05PM +0100, Uladzislau Rezki (Sony) wrote:
> Running an rcuscale stress-suite can lead to "Out of memory"
> of a system. This can happen under high memory pressure with
> a small amount of physical memory.
>
> For example a KVM test configuration with 64 CPUs and 512 megabytes
> can lead to of memory after running rcuscale with below parameters:
>
> ../kvm.sh --torture rcuscale --allcpus --duration 10 --kconfig
> CONFIG_NR_CPUS=64 \
> --bootargs "rcuscale.kfree_rcu_test=1 rcuscale.kfree_nthreads=16
> rcuscale.holdoff=20 \
> rcuscale.kfree_loops=1 torture.disable_onoff_at_boot" --trust-make
>
>
> [ 12.054448] kworker/1:1H invoked oom-killer:
> gfp_mask=0x2cc0(GFP_KERNEL|__GFP_NOWARN), order=0, oom_score_adj=0
> [ 12.055303] CPU: 1 PID: 377 Comm: kworker/1:1H Not tainted 5.11.0-rc3+ #510
> [ 12.055416] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> 1.12.0-1 04/01/2014
> [ 12.056485] Workqueue: events_highpri fill_page_cache_func
> [ 12.056485] Call Trace:
> [ 12.056485] dump_stack+0x57/0x6a
> [ 12.056485] dump_header+0x4c/0x30a
> [ 12.056485] ? del_timer_sync+0x20/0x30
> [ 12.056485] out_of_memory.cold.47+0xa/0x7e
> [ 12.056485] __alloc_pages_slowpath.constprop.123+0x82f/0xc00
> [ 12.056485] __alloc_pages_nodemask+0x289/0x2c0
> [ 12.056485] __get_free_pages+0x8/0x30
> [ 12.056485] fill_page_cache_func+0x39/0xb0
> [ 12.056485] process_one_work+0x1ed/0x3b0
> [ 12.056485] ? process_one_work+0x3b0/0x3b0
> [ 12.060485] worker_thread+0x28/0x3c0
> [ 12.060485] ? process_one_work+0x3b0/0x3b0
> [ 12.060485] kthread+0x138/0x160
> [ 12.060485] ? kthread_park+0x80/0x80
> [ 12.060485] ret_from_fork+0x22/0x30
> [ 12.062156] Mem-Info:
> [ 12.062350] active_anon:0 inactive_anon:0 isolated_anon:0
> [ 12.062350] active_file:0 inactive_file:0 isolated_file:0
> [ 12.062350] unevictable:0 dirty:0 writeback:0
> [ 12.062350] slab_reclaimable:2797 slab_unreclaimable:80920
> [ 12.062350] mapped:1 shmem:2 pagetables:8 bounce:0
> [ 12.062350] free:10488 free_pcp:1227 free_cma:0
> ...
> [ 12.101610] Out of memory and no killable processes...
> [ 12.102042] Kernel panic - not syncing: System is deadlocked on memory
> [ 12.102583] CPU: 1 PID: 377 Comm: kworker/1:1H Not tainted 5.11.0-rc3+ #510
> [ 12.102600] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> 1.12.0-1 04/01/2014
>
>
> Having a fallback mechanism we should not go with "GFP_KERNEL | __GFP_NOWARN"
> that implies a "hard" page request involving OOM killer. Replace such set with
> the same as the one used for a single argument.
>
> Thus it will follow same rules:
> a) minimize a fallback hitting;
> b) avoid of OOM invoking;
> c) do a light-wait page request;
> d) avoid of dipping into the emergency reserves.
>
> With this change an rcuscale and the parameters which are in question
> never runs into "Kernel panic".
>
> Signed-off-by: Uladzislau Rezki (Sony)
I did have some misgivings about this one, but after a closer look at
the GFP flags you suggest along with offlist discussions it looks like
what needs to happen. So thank you for persisting! ;-)
I did the usual wordsmithing as shown below, so please check to make
sure that I did not mess anything up.
Thanx, Paul
commit f60c420f9de536e7fc397f0ad8d6677d782d0141
Author: Uladzislau Rezki (Sony)
Date: Fri Jan 29 21:05:05 2021 +0100
kvfree_rcu: Use same set of GFP flags as does single-argument
Running an rcuscale stress-suite can lead to "Out of memory" of a
system. This can happen under high memory pressure with a small amount
of physical memory.
For example, a KVM test configuration with 64 CPUs and 512 megabytes
can result in OOM when running rcuscale with below parameters:
../kvm.sh --torture rcuscale --allcpus --duration 10 --kconfig
CONFIG_NR_CPUS=64 \
--bootargs "rcuscale.kfree_rcu_test=1 rcuscale.kfree_nthreads=16
rcuscale.holdoff=20 \
rcuscale.kfree_loops=1 torture.disable_onoff_at_boot" --trust-make
[ 12.054448] kworker/1:1H invoked oom-killer:
gfp_mask=0x2cc0(GFP_KERNEL|__GFP_NOWARN), order=0, oom_score_adj=0
[ 12.055303] CPU: 1 PID: 377 Comm: kworker/1:1H Not tainted 5.11.0-rc3+
#510
[ 12.055416] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.12.0-1 04/01/2014
[ 12.056485] Workqueue: events_highpri fill_page_cache_func
[ 12.056485] Call Trace:
[ 12.056485] dump_stack+0x57/0x6a
[ 12.056485] dump_header+0x4c/0x30a
[ 12.056485] ? del_timer_sync+0x20/0x30
[ 12.056485] out_of_memory.cold.47+0xa/0x7e
[ 12.056485] __alloc_pages_slowpath.constprop.123+0x82f/0xc00
[ 12.056485] __alloc_pages_nodemask+0x289/0x2c0
[ 12.056485] __get_free_pages+0x8/0x30
[