Re: [RFC PATCH] powerpc: Disabling MEMORY_HOTPLUG_DEFAULT_ONLINE option for PPC64 arch
On 08/02/2017 05:55 AM, Daniel Henrique Barboza wrote: > > On 08/01/2017 11:39 AM, Daniel Henrique Barboza wrote: >> >> On 08/01/2017 11:05 AM, Nathan Fontenot wrote: >> >>> At this point I don't think we need this patch to disable auto online >>> for ppc64. I would be curious if this is still broken with the latest >>> mainline code though. >> >> If the auto_online feature is already working in upstream 4.13 kernel then I >> don't see >> a reason to apply this patch either. We can leave it as a FYI/reminder of a >> problem >> that was happening in 4.11 and got solved later on. > > I've compiled an upstream kernel with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > and the problem with LMB hotplug is still reproducible. Yes, I was able to re-create the same failure. > > This means that it would be good to have any sort of mechanism (such as the > one this > patch proposes) to remove this option from the pseries kernel until it is > fixed. After looking through the pseries dlpar code I found the issue. The dlpar code assumes the memory is offline after adding it. The call to device_online() then returns non-zero (because the memory is already online) and the dlpar code interprets that as a failure and tries to remove the memory. This then causes the oops we see below because the memory is not offlined prior to the remove call. There is a similar issue with memory remove when the memory is offlined (via sysfs) prior to trying to dlpar remove the memory. The attempt to dlpar remove fails, luckily with just an error message and not an oops. I'll send a patch out to fix this. -Nathan > > Here is the stack trace: > > > Last login: Tue Aug 1 19:26:21 on hvc0 > [danielhb@localhost ~]$ uname -a > Linux localhost.localdomain 4.13.0-rc3+ #1 SMP Tue Aug 1 21:29:14 CDT 2017 > ppc64le ppc64le ppc64le GNU/Linux > [danielhb@localhost ~]$ QEMU 2.9.90 monitor - type 'help' for more information > (qemu) > (qemu) object_add memory-backend-ram,id=ram1,size=1G > (qemu) device_add pc-dimm,id=dimm1,memdev=ram1 > (qemu) [ 617.757086] kernel BUG at mm/memory_hotplug.c:1936! > [ 617.757245] Oops: Exception in kernel mode, sig: 5 [#1] > [ 617.757332] SMP NR_CPUS=1024 > [ 617.757333] NUMA > [ 617.757396] pSeries > [ 617.757479] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 > xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc > ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle > ip6table_raw ip6table_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 > nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw iptable_security > ebtable_filter ebtables ip6table_filter ip6_tables sunrpc ghash_generic > vmx_crypto xfs libcrc32c bochs_drm drm_kms_helper syscopyarea sysfillrect > sysimgblt fb_sys_fops ttm drm virtio_blk ibmvscsi virtio_pci ibmveth > scsi_transport_srp i2c_core virtio_ring virtio crc32c_vpmsum > [ 617.758577] CPU: 6 PID: 5 Comm: kworker/u16:0 Not tainted 4.13.0-rc3+ #1 > [ 617.758741] Workqueue: pseries hotplug workque pseries_hp_work_fn > [ 617.758852] task: c000feb9e780 task.stack: c000fe118000 > [ 617.759047] NIP: c037322c LR: c03731b8 CTR: > > [ 617.759171] REGS: c000fe11b790 TRAP: 0700 Not tainted (4.13.0-rc3+) > [ 617.759275] MSR: 8282b033> [ 617.759284] CR: 22002424 XER: 2000 > [ 617.759450] CFAR: c03731bc SOFTE: 1 > [ 617.759450] GPR00: c03731b8 c000fe11ba10 c141f100 > 0001 > [ 617.759450] GPR04: 01a0 c15dac30 > 8b80 > [ 617.759450] GPR08: 0007 0002 c000e9873480 > 303078302d303030 > [ 617.759450] GPR12: 2200 cfd83c00 c0137f88 > c000fe1f0e40 > [ 617.759450] GPR16: 0001 0004 > 0001 > [ 617.759450] GPR20: c000fc28 0010 c000ebb1c184 > 0010 > [ 617.759450] GPR24: c000fea0 c000f0aec4a0 c000fc28 > > [ 617.759450] GPR28: c1347f08 1000 > 0001 > [ 617.760679] NIP [c037322c] remove_memory+0xfc/0x110 > [ 617.760766] LR [c03731b8] remove_memory+0x88/0x110 > [ 617.760849] Call Trace: > [ 617.760895] [c000fe11ba10] [c03731b8] remove_memory+0x88/0x110 > (unreliable) > [ 617.761032] [c000fe11ba50] [c00cb0e0] dlpar_add_lmb+0x280/0x480 > [ 617.761214] [c000fe11bb30] [c00cc83c] dlpar_memory+0xa5c/0xe50 > [ 617.761319] [c000fe11bbf0] [c00c3b88] > handle_dlpar_errorlog+0xf8/0x160 > [ 617.761443] [c000fe11bc60] [c00c3c84] > pseries_hp_work_fn+0x94/0xa0 > [ 617.761568] [c000fe11bc90] [c012e3e8] > process_one_work+0x248/0x540 > [ 617.761697] [c000fe11bd30] [c012e768] worker_thread+0x88/0x5c0 > [
Re: [RFC PATCH] powerpc: Disabling MEMORY_HOTPLUG_DEFAULT_ONLINE option for PPC64 arch
On 08/01/2017 11:39 AM, Daniel Henrique Barboza wrote: On 08/01/2017 11:05 AM, Nathan Fontenot wrote: At this point I don't think we need this patch to disable auto online for ppc64. I would be curious if this is still broken with the latest mainline code though. If the auto_online feature is already working in upstream 4.13 kernel then I don't see a reason to apply this patch either. We can leave it as a FYI/reminder of a problem that was happening in 4.11 and got solved later on. I've compiled an upstream kernel with CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y and the problem with LMB hotplug is still reproducible. This means that it would be good to have any sort of mechanism (such as the one this patch proposes) to remove this option from the pseries kernel until it is fixed. Here is the stack trace: Last login: Tue Aug 1 19:26:21 on hvc0 [danielhb@localhost ~]$ uname -a Linux localhost.localdomain 4.13.0-rc3+ #1 SMP Tue Aug 1 21:29:14 CDT 2017 ppc64le ppc64le ppc64le GNU/Linux [danielhb@localhost ~]$ QEMU 2.9.90 monitor - type 'help' for more information (qemu) (qemu) object_add memory-backend-ram,id=ram1,size=1G (qemu) device_add pc-dimm,id=dimm1,memdev=ram1 (qemu) [ 617.757086] kernel BUG at mm/memory_hotplug.c:1936! [ 617.757245] Oops: Exception in kernel mode, sig: 5 [#1] [ 617.757332] SMP NR_CPUS=1024 [ 617.757333] NUMA [ 617.757396] pSeries [ 617.757479] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw iptable_security ebtable_filter ebtables ip6table_filter ip6_tables sunrpc ghash_generic vmx_crypto xfs libcrc32c bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_blk ibmvscsi virtio_pci ibmveth scsi_transport_srp i2c_core virtio_ring virtio crc32c_vpmsum [ 617.758577] CPU: 6 PID: 5 Comm: kworker/u16:0 Not tainted 4.13.0-rc3+ #1 [ 617.758741] Workqueue: pseries hotplug workque pseries_hp_work_fn [ 617.758852] task: c000feb9e780 task.stack: c000fe118000 [ 617.759047] NIP: c037322c LR: c03731b8 CTR: [ 617.759171] REGS: c000fe11b790 TRAP: 0700 Not tainted (4.13.0-rc3+) [ 617.759275] MSR: 8282b033[ 617.759284] CR: 22002424 XER: 2000 [ 617.759450] CFAR: c03731bc SOFTE: 1 [ 617.759450] GPR00: c03731b8 c000fe11ba10 c141f100 0001 [ 617.759450] GPR04: 01a0 c15dac30 8b80 [ 617.759450] GPR08: 0007 0002 c000e9873480 303078302d303030 [ 617.759450] GPR12: 2200 cfd83c00 c0137f88 c000fe1f0e40 [ 617.759450] GPR16: 0001 0004 0001 [ 617.759450] GPR20: c000fc28 0010 c000ebb1c184 0010 [ 617.759450] GPR24: c000fea0 c000f0aec4a0 c000fc28 [ 617.759450] GPR28: c1347f08 1000 0001 [ 617.760679] NIP [c037322c] remove_memory+0xfc/0x110 [ 617.760766] LR [c03731b8] remove_memory+0x88/0x110 [ 617.760849] Call Trace: [ 617.760895] [c000fe11ba10] [c03731b8] remove_memory+0x88/0x110 (unreliable) [ 617.761032] [c000fe11ba50] [c00cb0e0] dlpar_add_lmb+0x280/0x480 [ 617.761214] [c000fe11bb30] [c00cc83c] dlpar_memory+0xa5c/0xe50 [ 617.761319] [c000fe11bbf0] [c00c3b88] handle_dlpar_errorlog+0xf8/0x160 [ 617.761443] [c000fe11bc60] [c00c3c84] pseries_hp_work_fn+0x94/0xa0 [ 617.761568] [c000fe11bc90] [c012e3e8] process_one_work+0x248/0x540 [ 617.761697] [c000fe11bd30] [c012e768] worker_thread+0x88/0x5c0 [ 617.761857] [c000fe11bdc0] [c013812c] kthread+0x1ac/0x1c0 [ 617.761962] [c000fe11be30] [c000bc9c] ret_from_kernel_thread+0x5c/0xc0 [ 617.762085] Instruction dump: [ 617.762149] 4be09a9d 6000 4bd95ac5 6000 38210040 e8010010 eb81ffe0 eba1ffe8 [ 617.762274] ebc1fff0 7c0803a6 ebe1fff8 4e800020 <0fe0> [ 617.762403] ---[ end trace feaa62099c5987b9 ]--- Thanks, Daniel Thanks, Daniel -Nathan However, this change alone isn't enough to prevent situations such as [1], where distros can enable the option unaware of the consequences of doing it (e.g. breaking LMB hotplug altogether). Instead of relying on all distros knowing that pseries can't handle CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y after 943db62c316c, this patch changes mm/Kconfig to make the MEMORY_HOTPLUG_DEFAULT_ONLINE config unavailable for the PPC64 arch. [1]
Re: [RFC PATCH] powerpc: Disabling MEMORY_HOTPLUG_DEFAULT_ONLINE option for PPC64 arch
On 08/01/2017 11:05 AM, Nathan Fontenot wrote: On 08/01/2017 04:59 AM, Michael Ellerman wrote: Daniel Henrique Barbozawrites: Commit 943db62c316c ("powerpc/pseries: Revert 'Auto-online hotplugged memory'") reverted the auto-online feature for pseries due to problems with LMB removals not updating the device struct properly. Among other things, this commit made the following change in arch/powerpc/configs/pseries_defconfig: @@ -58,7 +58,6 @@ CONFIG_KEXEC_FILE=y CONFIG_IRQ_ALL_CPUS=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTREMOVE=y -CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y CONFIG_KSM=y The intent was to disable the option in the defconfig of pseries, since after that the code doesn't have this support anymore. It's always polite to Cc the author of a commit you're referring to, so I added Nathan. Noted. Thanks for adding Nathan in the CC. The intention when we merged that fix was that the auto-online code would be "fixed" to mark the device online. I say "fixed" because it wasn't entirely clear if that was the correct behaviour, though it definitely seemed like it should be. I've lost track of where/if the discussion got to on whether the auto-online code should do that or not. Did anything get resolved? I think, though I should go back and test to be sure, that everything works in the latest mainline code. The issue causing this to be a problem was in the original implementation of auto_online support. If you wanted to auto online memory, the code was calling memory_block_change_state(). This worked but did not update the device struct for each of the memory block that was online'ed such that dev->offline == true even after the memory was online. I sent a patch earlier this year (commit dc18d706a436) that corrected this to call device_online() instead of memory_block_change_state(). With this fix (appears to have gone into the 4.11 kernel) it should be possible to use auto_online on power systems. Commit dc18d706a436 was present in the 4.11 kernels that experiences this issue (Fedora 26 and Ubuntu 17.10 in my tests). So I am not entirely sure that we can use auto_online on power systems, at least in those kernels. At this point I don't think we need this patch to disable auto online for ppc64. I would be curious if this is still broken with the latest mainline code though. If the auto_online feature is already working in upstream 4.13 kernel then I don't see a reason to apply this patch either. We can leave it as a FYI/reminder of a problem that was happening in 4.11 and got solved later on. Thanks, Daniel -Nathan However, this change alone isn't enough to prevent situations such as [1], where distros can enable the option unaware of the consequences of doing it (e.g. breaking LMB hotplug altogether). Instead of relying on all distros knowing that pseries can't handle CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y after 943db62c316c, this patch changes mm/Kconfig to make the MEMORY_HOTPLUG_DEFAULT_ONLINE config unavailable for the PPC64 arch. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1476380 Fixes: 943db62c316c ("powerpc/pseries: Revert 'Auto-online hotplugged memory'") Signed-off-by: Daniel Henrique Barboza --- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) I don't own that file, so we at least need an Ack from the mm folks. cheers diff --git a/mm/Kconfig b/mm/Kconfig index 48b1af4..a342c77 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -169,7 +169,7 @@ config MEMORY_HOTPLUG_SPARSE config MEMORY_HOTPLUG_DEFAULT_ONLINE bool "Online the newly added memory blocks by default" default n -depends on MEMORY_HOTPLUG +depends on MEMORY_HOTPLUG && !PPC64 help This option sets the default policy setting for memory hotplug onlining policy (/sys/devices/system/memory/auto_online_blocks) which -- 2.9.4
Re: [RFC PATCH] powerpc: Disabling MEMORY_HOTPLUG_DEFAULT_ONLINE option for PPC64 arch
On 08/01/2017 04:59 AM, Michael Ellerman wrote: > Daniel Henrique Barbozawrites: > >> Commit 943db62c316c ("powerpc/pseries: Revert 'Auto-online >> hotplugged memory'") reverted the auto-online feature for pseries due >> to problems with LMB removals not updating the device struct properly. >> Among other things, this commit made the following change in >> arch/powerpc/configs/pseries_defconfig: >> >> @@ -58,7 +58,6 @@ CONFIG_KEXEC_FILE=y >> CONFIG_IRQ_ALL_CPUS=y >> CONFIG_MEMORY_HOTPLUG=y >> CONFIG_MEMORY_HOTREMOVE=y >> -CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y >> CONFIG_KSM=y >> >> The intent was to disable the option in the defconfig of pseries, since >> after that the code doesn't have this support anymore. > > It's always polite to Cc the author of a commit you're referring to, so > I added Nathan. > > The intention when we merged that fix was that the auto-online code > would be "fixed" to mark the device online. I say "fixed" because it > wasn't entirely clear if that was the correct behaviour, though it > definitely seemed like it should be. > > I've lost track of where/if the discussion got to on whether the > auto-online code should do that or not. Did anything get resolved? I think, though I should go back and test to be sure, that everything works in the latest mainline code. The issue causing this to be a problem was in the original implementation of auto_online support. If you wanted to auto online memory, the code was calling memory_block_change_state(). This worked but did not update the device struct for each of the memory block that was online'ed such that dev->offline == true even after the memory was online. I sent a patch earlier this year (commit dc18d706a436) that corrected this to call device_online() instead of memory_block_change_state(). With this fix (appears to have gone into the 4.11 kernel) it should be possible to use auto_online on power systems. At this point I don't think we need this patch to disable auto online for ppc64. I would be curious if this is still broken with the latest mainline code though. -Nathan > >> However, this change >> alone isn't enough to prevent situations such as [1], where >> distros can enable the option unaware of the consequences of >> doing it (e.g. breaking LMB hotplug altogether). >> >> Instead of relying on all distros knowing that pseries can't handle >> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y after 943db62c316c, this patch >> changes mm/Kconfig to make the MEMORY_HOTPLUG_DEFAULT_ONLINE config >> unavailable for the PPC64 arch. >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1476380 >> >> Fixes: 943db62c316c ("powerpc/pseries: Revert 'Auto-online hotplugged >> memory'") >> Signed-off-by: Daniel Henrique Barboza >> --- >> mm/Kconfig | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) > > I don't own that file, so we at least need an Ack from the mm folks. > > cheers > >> diff --git a/mm/Kconfig b/mm/Kconfig >> index 48b1af4..a342c77 100644 >> --- a/mm/Kconfig >> +++ b/mm/Kconfig >> @@ -169,7 +169,7 @@ config MEMORY_HOTPLUG_SPARSE >> config MEMORY_HOTPLUG_DEFAULT_ONLINE >> bool "Online the newly added memory blocks by default" >> default n >> -depends on MEMORY_HOTPLUG >> +depends on MEMORY_HOTPLUG && !PPC64 >> help >>This option sets the default policy setting for memory hotplug >>onlining policy (/sys/devices/system/memory/auto_online_blocks) which >> -- >> 2.9.4 >
Re: [RFC PATCH] powerpc: Disabling MEMORY_HOTPLUG_DEFAULT_ONLINE option for PPC64 arch
Daniel Henrique Barbozawrites: > Commit 943db62c316c ("powerpc/pseries: Revert 'Auto-online > hotplugged memory'") reverted the auto-online feature for pseries due > to problems with LMB removals not updating the device struct properly. > Among other things, this commit made the following change in > arch/powerpc/configs/pseries_defconfig: > > @@ -58,7 +58,6 @@ CONFIG_KEXEC_FILE=y > CONFIG_IRQ_ALL_CPUS=y > CONFIG_MEMORY_HOTPLUG=y > CONFIG_MEMORY_HOTREMOVE=y > -CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > CONFIG_KSM=y > > The intent was to disable the option in the defconfig of pseries, since > after that the code doesn't have this support anymore. It's always polite to Cc the author of a commit you're referring to, so I added Nathan. The intention when we merged that fix was that the auto-online code would be "fixed" to mark the device online. I say "fixed" because it wasn't entirely clear if that was the correct behaviour, though it definitely seemed like it should be. I've lost track of where/if the discussion got to on whether the auto-online code should do that or not. Did anything get resolved? > However, this change > alone isn't enough to prevent situations such as [1], where > distros can enable the option unaware of the consequences of > doing it (e.g. breaking LMB hotplug altogether). > > Instead of relying on all distros knowing that pseries can't handle > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y after 943db62c316c, this patch > changes mm/Kconfig to make the MEMORY_HOTPLUG_DEFAULT_ONLINE config > unavailable for the PPC64 arch. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1476380 > > Fixes: 943db62c316c ("powerpc/pseries: Revert 'Auto-online hotplugged > memory'") > Signed-off-by: Daniel Henrique Barboza > --- > mm/Kconfig | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) I don't own that file, so we at least need an Ack from the mm folks. cheers > diff --git a/mm/Kconfig b/mm/Kconfig > index 48b1af4..a342c77 100644 > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -169,7 +169,7 @@ config MEMORY_HOTPLUG_SPARSE > config MEMORY_HOTPLUG_DEFAULT_ONLINE > bool "Online the newly added memory blocks by default" > default n > -depends on MEMORY_HOTPLUG > +depends on MEMORY_HOTPLUG && !PPC64 > help > This option sets the default policy setting for memory hotplug > onlining policy (/sys/devices/system/memory/auto_online_blocks) which > -- > 2.9.4
[RFC PATCH] powerpc: Disabling MEMORY_HOTPLUG_DEFAULT_ONLINE option for PPC64 arch
Commit 943db62c316c ("powerpc/pseries: Revert 'Auto-online hotplugged memory'") reverted the auto-online feature for pseries due to problems with LMB removals not updating the device struct properly. Among other things, this commit made the following change in arch/powerpc/configs/pseries_defconfig: @@ -58,7 +58,6 @@ CONFIG_KEXEC_FILE=y CONFIG_IRQ_ALL_CPUS=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTREMOVE=y -CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y CONFIG_KSM=y The intent was to disable the option in the defconfig of pseries, since after that the code doesn't have this support anymore. However, this change alone isn't enough to prevent situations such as [1], where distros can enable the option unaware of the consequences of doing it (e.g. breaking LMB hotplug altogether). Instead of relying on all distros knowing that pseries can't handle CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y after 943db62c316c, this patch changes mm/Kconfig to make the MEMORY_HOTPLUG_DEFAULT_ONLINE config unavailable for the PPC64 arch. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1476380 Fixes: 943db62c316c ("powerpc/pseries: Revert 'Auto-online hotplugged memory'") Signed-off-by: Daniel Henrique Barboza--- mm/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/Kconfig b/mm/Kconfig index 48b1af4..a342c77 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -169,7 +169,7 @@ config MEMORY_HOTPLUG_SPARSE config MEMORY_HOTPLUG_DEFAULT_ONLINE bool "Online the newly added memory blocks by default" default n -depends on MEMORY_HOTPLUG +depends on MEMORY_HOTPLUG && !PPC64 help This option sets the default policy setting for memory hotplug onlining policy (/sys/devices/system/memory/auto_online_blocks) which -- 2.9.4