On Thu, Mar 30, 2017 at 05:07:08PM +0200, Jesper Dangaard Brouer wrote:
> On Thu, 30 Mar 2017 14:04:36 +0100
> Mel Gorman <mgor...@techsingularity.net> wrote:
>
> > On Wed, Mar 29, 2017 at 09:44:41PM +0200, Jesper Dangaard Brouer wrote:
> > > > Regardless or using in_irq() (or in combi with in_nmi()) I get the
> > > > following warning below:
> > > >
> > > > [ 0.000000] Kernel command line:
> > > > BOOT_IMAGE=/vmlinuz-4.11.0-rc3-net-next-page-alloc-softirq+
> > > > root=UUID=2e8451ff-6797-49b5-8d3a-eed5a42d7dc9 ro rhgb quiet
> > > > LANG=en_DK.UTF
> > > > -8
> > > > [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> > > > [ 0.000000] ------------[ cut here ]------------
> > > > [ 0.000000] WARNING: CPU: 0 PID: 0 at kernel/softirq.c:161
> > > > __local_bh_enable_ip+0x70/0x90
> > > > [ 0.000000] Modules linked in:
> > > > [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
> > > > 4.11.0-rc3-net-next-page-alloc-softirq+ #235
> > > > [ 0.000000] Hardware name: MSI MS-7984/Z170A GAMING PRO (MS-7984),
> > > > BIOS 1.60 12/16/2015
> > > > [ 0.000000] Call Trace:
> > > > [ 0.000000] dump_stack+0x4f/0x73
> > > > [ 0.000000] __warn+0xcb/0xf0
> > > > [ 0.000000] warn_slowpath_null+0x1d/0x20
> > > > [ 0.000000] __local_bh_enable_ip+0x70/0x90
> > > > [ 0.000000] free_hot_cold_page+0x1a4/0x2f0
> > > > [ 0.000000] __free_pages+0x1f/0x30
> > > > [ 0.000000] __free_pages_bootmem+0xab/0xb8
> > > > [ 0.000000] __free_memory_core+0x79/0x91
> > > > [ 0.000000] free_all_bootmem+0xaa/0x122
> > > > [ 0.000000] mem_init+0x71/0xa4
> > > > [ 0.000000] start_kernel+0x1e5/0x3f1
> > > > [ 0.000000] x86_64_start_reservations+0x2a/0x2c
> > > > [ 0.000000] x86_64_start_kernel+0x178/0x18b
> > > > [ 0.000000] start_cpu+0x14/0x14
> > > > [ 0.000000] ? start_cpu+0x14/0x14
> > > > [ 0.000000] ---[ end trace a57944bec8fc985c ]---
> > > > [ 0.000000] Memory: 32739472K/33439416K available (7624K kernel
> > > > code, 1528K rwdata, 3168K rodata, 1860K init, 2260K bss, 699944K
> > > > reserved, 0K cma-reserved)
> > > >
> > > > And kernel/softirq.c:161 contains:
> > > >
> > > > WARN_ON_ONCE(in_irq() || irqs_disabled());
> > > >
> > > > Thus, I don't think the change in my RFC-patch[1] is safe.
> > > > Of changing[2] to support softirq allocations by replacing
> > > > preempt_disable() with local_bh_disable().
> > > >
> > > > [1] http://lkml.kernel.org/r/20170327143947.4c237...@redhat.com
> > > >
> > > > [2] commit 374ad05ab64d ("mm, page_alloc: only use per-cpu allocator
> > > > for irq-safe requests")
> > > > https://git.kernel.org/torvalds/c/374ad05ab64d
> > >
> > > A patch that avoids the above warning is inlined below, but I'm not
> > > sure if this is best direction. Or we should rather consider reverting
> > > part of commit 374ad05ab64d to avoid the softirq performance regression?
> > >
> >
> > At the moment, I'm not seeing a better alternative. If this works, I
> > think it would still be far superior in terms of performance than a
> > revert.
>
> Started performance benchmarking:
> 163 cycles = current state
> 183 cycles = with BH disable + in_irq
> 218 cycles = with BH disable + in_irq + irqs_disabled
>
> Thus, the performance numbers unfortunately looks bad, once we add the
> test for irqs_disabled(). The slowdown by replacing preempt_disable
> with BH-disable is still a win (we saved 29 cycles before, and loose
> 20, I was expecting regression to be only 10 cycles).
>
This surprises me because I'm not seeing the same severity of problems
with irqs_disabled. Your path is slower than what's currently upstream
but it's still far better than a revert. The softirq column in the
middle is your patch versus a full revert which is the last columnm
4.11.0-rc5 4.11.0-rc5
4.11.0-rc5
vanilla softirq-v2r1
revert-v2r1
Amean alloc-odr0-1 217.00 ( 0.00%) 223.00 ( -2.76%)
280.54 (-29.28%)
Amean alloc-odr0-2 162.23 ( 0.00%) 174.46 ( -7.54%)
210.54 (-29.78%)
Amean alloc-odr0-4 144.15 ( 0.00%) 150.38 ( -4.32%)
182.38 (-26.52%)
Amean alloc-odr0-8 126.00 ( 0.00%) 132.15 ( -4.88%)
282.08 (-123.87%)
Amean alloc-odr0-16 117.00 ( 0.00%) 122.00 ( -4.27%)
253.00 (-116.24%)
Amean alloc-odr0-32 113.00 ( 0.00%) 118.00 ( -4.42%)
145.00 (-28.32%)
Amean alloc-odr0-64 110.77 ( 0.00%) 114.31 ( -3.19%)
143.00 (-29.10%)
Amean alloc-odr0-128 109.00 ( 0.00%) 107.69 ( 1.20%)
179.54 (-64.71%)
Amean alloc-odr0-256 121.00 ( 0.00%) 125.00 ( -3.31%)
232.23 (-91.93%)
Amean alloc-odr0-512 123.46 ( 0.00%) 129.46 ( -4.86%)
148.08 (-19.94%)
Amean alloc-odr0-1024 123.23 ( 0.00%) 128.92 ( -4.62%)
142.46 (-15.61%)
Amean alloc-odr0-2048 125.92 ( 0.00%) 129.62 ( -2.93%)
147.46 (-17.10%)
Amean alloc-odr0-4096 133.85 ( 0.00%) 139.77 ( -4.43%)
155.69 (-16.32%)
Amean alloc-odr0-8192 138.08 ( 0.00%) 142.92 ( -3.51%)
159.00 (-15.15%)
Amean alloc-odr0-16384 133.08 ( 0.00%) 140.08 ( -5.26%)
157.38 (-18.27%)
Amean alloc-odr1-1 390.27 ( 0.00%) 401.53 ( -2.89%)
389.73 ( 0.14%)
Amean alloc-odr1-2 306.33 ( 0.00%) 311.07 ( -1.55%)
304.07 ( 0.74%)
Amean alloc-odr1-4 250.87 ( 0.00%) 258.00 ( -2.84%)
256.53 ( -2.26%)
Amean alloc-odr1-8 221.00 ( 0.00%) 231.07 ( -4.56%)
221.20 ( -0.09%)
Amean alloc-odr1-16 212.07 ( 0.00%) 223.07 ( -5.19%)
208.00 ( 1.92%)
Amean alloc-odr1-32 210.07 ( 0.00%) 215.20 ( -2.44%)
208.20 ( 0.89%)
Amean alloc-odr1-64 197.00 ( 0.00%) 203.00 ( -3.05%)
203.00 ( -3.05%)
Amean alloc-odr1-128 204.07 ( 0.00%) 189.27 ( 7.25%)
200.00 ( 1.99%)
Amean alloc-odr1-256 193.33 ( 0.00%) 190.53 ( 1.45%)
193.80 ( -0.24%)
Amean alloc-odr1-512 180.60 ( 0.00%) 190.33 ( -5.39%)
183.13 ( -1.40%)
Amean alloc-odr1-1024 176.93 ( 0.00%) 182.40 ( -3.09%)
176.33 ( 0.34%)
Amean alloc-odr1-2048 184.60 ( 0.00%) 191.33 ( -3.65%)
180.60 ( 2.17%)
Amean alloc-odr1-4096 184.80 ( 0.00%) 182.60 ( 1.19%)
182.27 ( 1.37%)
Amean alloc-odr1-8192 183.60 ( 0.00%) 180.93 ( 1.45%)
181.07 ( 1.38%)
I revisisted having an irq-safe list but it's excessively complex and
there are significant problems where it's not clear it can be handled
safely so it's not a short-term option.
--
Mel Gorman
SUSE Labs
