Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
a...@linux-foundation.org writes: > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to > >http://www.ozlabs.org/~akpm/mmotm/ ... > * mm-sparse-add-a-static-variable-nr_present_sections.patch > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch > * > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch > * > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch This seems to be breaking my powerpc pseries qemu boots. The boot log with some extra debug shows eg: $ make pseries_le_defconfig $ qemu-system-ppc64 -nographic -vga none -M pseries -m 2G -kernel vmlinux ... vmemmap_populate f000..f0004000, node 0 * f000..f100 allocated at c0007e00 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7e00 vmemmap_populate f000..f0008000, node 0 * f000..f100 allocated at c0007d00 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7d00 vmemmap_populate f000..f000c000, node 0 * f000..f100 allocated at c0007c00 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7c00 vmemmap_populate f000..f001, node 0 * f000..f100 allocated at c0007b00 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7b00 vmemmap_populate f000..f0014000, node 0 * f000..f100 allocated at c0007a00 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7a00 vmemmap_populate f000..f0018000, node 0 * f000..f100 allocated at c0007900 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7900 vmemmap_populate f000..f001c000, node 0 * f000..f100 allocated at c0007800 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7800 vmemmap_populate f000..f002, node 0 * f000..f100 allocated at c0007700 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7700 vmemmap_populate f000..f0024000, node 0 * f000..f100 allocated at c0007600 hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys 0x7600 hash__vmemmap_create_mapping: failed -1 Then there's lots of other warnings about bad page states and eventually a NULL deref and we panic(). The problem seems to be that we're calling down into hash__vmemmap_create_mapping() for every call to vmemmap_populate(), whereas previously we would only call hash__vmemmap_create_mapping() once because our vmemmap_populated() would return true. There's actually a comment in sparse_init() that says: * powerpc need to call sparse_init_one_section right after each * sparse_early_mem_map_alloc, so allocate usemap_map at first. So changing that behaviour does seem to be the problem. I assume that comment is talking about the fact that we use pfn_valid() in vmemmap_populated(). I'm not clear on how to fix it though. Any ideas? cheers
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
Hi Michael, On 07/11/18 at 10:49pm, Michael Ellerman wrote: > a...@linux-foundation.org writes: > > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to > > > >http://www.ozlabs.org/~akpm/mmotm/ > ... > > > * mm-sparse-add-a-static-variable-nr_present_sections.patch > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch > > * > > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch > > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch > > * > > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch > > > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch > > This seems to be breaking my powerpc pseries qemu boots. > > The boot log with some extra debug shows eg: > > $ make pseries_le_defconfig > $ qemu-system-ppc64 -nographic -vga none -M pseries -m 2G -kernel vmlinux > vmemmap_populate f000..f0024000, node 0 > * f000..f100 allocated at c0007600 > hash__vmemmap_create_mapping: start 0xf000 size 0x100 phys > 0x7600 > hash__vmemmap_create_mapping: failed -1 > > > > Then there's lots of other warnings about bad page states and eventually > a NULL deref and we panic(). > > > The problem seems to be that we're calling down into > hash__vmemmap_create_mapping() for every call to vmemmap_populate(), > whereas previously we would only call hash__vmemmap_create_mapping() > once because our vmemmap_populated() would return true. > > There's actually a comment in sparse_init() that says: > >* powerpc need to call sparse_init_one_section right after each >* sparse_early_mem_map_alloc, so allocate usemap_map at first. > > So changing that behaviour does seem to be the problem. > > I assume that comment is talking about the fact that we use pfn_valid() > in vmemmap_populated(). > > I'm not clear on how to fix it though. Have you tried reverting that patch and building kernel to test again? Does it work?
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
On Wed, Jul 11, 2018 at 10:49:58PM +1000, Michael Ellerman wrote: > a...@linux-foundation.org writes: > > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to > > > >http://www.ozlabs.org/~akpm/mmotm/ > ... > > > * mm-sparse-add-a-static-variable-nr_present_sections.patch > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch > > * > > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch > > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch > > * > > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch > > > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch > > This seems to be breaking my powerpc pseries qemu boots. > > The boot log with some extra debug shows eg: > > $ make pseries_le_defconfig Could you please share the config? I was not able to find such config in the kernel tree. -- Oscar Salvador SUSE L3
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
I am OK, if this patch is removed from Baoquan's series. But, I would still like to get rid of CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER, I can work on this in my sparse_init re-write series. ppc64 should really fallback safely to small chunks allocs, and if it does not there is some existing bug. Michael please send the config that you used. Thank you, Pavel On Wed, Jul 11, 2018 at 9:37 AM Oscar Salvador wrote: > > On Wed, Jul 11, 2018 at 10:49:58PM +1000, Michael Ellerman wrote: > > a...@linux-foundation.org writes: > > > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to > > > > > >http://www.ozlabs.org/~akpm/mmotm/ > > ... > > > > > * mm-sparse-add-a-static-variable-nr_present_sections.patch > > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch > > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch > > > * > > > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch > > > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch > > > * > > > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch > > > > > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch > > > > This seems to be breaking my powerpc pseries qemu boots. > > > > The boot log with some extra debug shows eg: > > > > $ make pseries_le_defconfig > > Could you please share the config? > I was not able to find such config in the kernel tree. > -- > Oscar Salvador > SUSE L3 >
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
On Wed, 11 Jul 2018 09:55:59 -0400 Pavel Tatashin wrote: > On Wed, Jul 11, 2018 at 9:37 AM Oscar Salvador > wrote: > > > > On Wed, Jul 11, 2018 at 10:49:58PM +1000, Michael Ellerman wrote: > > > a...@linux-foundation.org writes: > > > > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to > > > > > > > >http://www.ozlabs.org/~akpm/mmotm/ > > > ... > > > > > > > * mm-sparse-add-a-static-variable-nr_present_sections.patch > > > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch > > > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch > > > > * > > > > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch > > > > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch > > > > * > > > > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch > > > > > > > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch > > > > > > This seems to be breaking my powerpc pseries qemu boots. > > > > > > The boot log with some extra debug shows eg: > > > > > > $ make pseries_le_defconfig > > > > Could you please share the config? > > I was not able to find such config in the kernel tree. (top-posting repaired so I can reply to your email, add other people and not confuse the heck out of them. Please don't) > I am OK, if this patch is removed from Baoquan's series. But, I would > still like to get rid of CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER, I > can work on this in my sparse_init re-write series. ppc64 should > really fallback safely to small chunks allocs, and if it does not > there is some existing bug. Michael please send the config that you > used. OK, I shall drop mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch for now.
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
Hi Andrew, On Wed, 11 Jul 2018 14:13:44 -0700 Andrew Morton wrote: > > OK, I shall drop > mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch for now. I have dropped it from linux-next today (in case you don't get time). -- Cheers, Stephen Rothwell pgpl2zD9PsNTX.pgp Description: OpenPGP digital signature
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
Baoquan He writes: > On 07/11/18 at 10:49pm, Michael Ellerman wrote: >> a...@linux-foundation.org writes: >> > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to >> > >> >http://www.ozlabs.org/~akpm/mmotm/ >> ... >> >> > * mm-sparse-add-a-static-variable-nr_present_sections.patch >> > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch >> > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch >> > * >> > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch >> > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch >> > * >> > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch >> >> > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch >> >> This seems to be breaking my powerpc pseries qemu boots. ... > > Have you tried reverting that patch and building kernel to test again? > Does it work? Yes. Reverting that patch on top of 98be45067040799a801e6ce52d8bf4659a153893 works as before. cheers
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
Oscar Salvador writes: > On Wed, Jul 11, 2018 at 10:49:58PM +1000, Michael Ellerman wrote: >> a...@linux-foundation.org writes: >> > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to >> > >> >http://www.ozlabs.org/~akpm/mmotm/ >> ... >> >> > * mm-sparse-add-a-static-variable-nr_present_sections.patch >> > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch >> > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch >> > * >> > mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch >> > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch >> > * >> > mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch >> >> > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch >> >> This seems to be breaking my powerpc pseries qemu boots. >> >> The boot log with some extra debug shows eg: >> >> $ make pseries_le_defconfig > > Could you please share the config? > I was not able to find such config in the kernel tree. It's a generated config, see: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/powerpc/Makefile?commit=021c91791a5e7e85c567452f1be3e4c2c6cb6063#n315 It's based on pseries_defconfig, which is an actual file in the tree, but with the endian flipped. You can see the generated result from a recent build here: http://kisskb.ellerman.id.au/kisskb/buildresult/13428803/config/ cheers
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
Oscar Salvador writes: > El dc., 11 jul. 2018 , 15:56, Pavel Tatashin va > escriure: > >> I am OK, if this patch is removed from Baoquan's series. But, I would >> still like to get rid of CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER, I >> can work on this in my sparse_init re-write series. ppc64 should >> really fallback safely to small chunks allocs, and if it does not >> there is some existing bug. Michael please send the config that you >> used. >> >> Thank you, >> Pavel >> On Wed, Jul 11, 2018 at 9:37 AM Oscar Salvador >> wrote: >> > >> > On Wed, Jul 11, 2018 at 10:49:58PM +1000, Michael Ellerman wrote: >> > > a...@linux-foundation.org writes: >> > > > The mm-of-the-moment snapshot 2018-07-10-16-50 has been uploaded to >> > > > >> > > >http://www.ozlabs.org/~akpm/mmotm/ >> > > ... >> > > >> > > > * mm-sparse-add-a-static-variable-nr_present_sections.patch >> > > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing.patch >> > > > * mm-sparsemem-defer-the-ms-section_mem_map-clearing-fix.patch >> > > > * >> mm-sparse-add-a-new-parameter-data_unit_size-for-alloc_usemap_and_memmap.patch >> > > > * mm-sparse-optimize-memmap-allocation-during-sparse_init.patch >> > > > * >> mm-sparse-optimize-memmap-allocation-during-sparse_init-checkpatch-fixes.patch >> > > >> > > > * mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch >> > > >> > > This seems to be breaking my powerpc pseries qemu boots. >> > > >> > > The boot log with some extra debug shows eg: >> > > >> > > $ make pseries_le_defconfig >> > >> > Could you please share the config? >> > I was not able to find such config in the kernel tree. >> >> > I just roughly check, but if I checked the right place, > vmemmap_populated() checks for the section to contain the flags we are > setting in sparse_init_one_section(). Yes. > But with this patch, we populate first everything, and then we call > sparse_init_one_section() in sparse_init(). > As I said I could be mistaken because I just checked the surface. Yeah I think that's correct. This might just be a bug in our code, let me look at it a bit. cheers
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
Hi all, On Thu, 12 Jul 2018 09:47:29 +1000 Stephen Rothwell wrote: > > On Wed, 11 Jul 2018 14:13:44 -0700 Andrew Morton > wrote: > > > > OK, I shall drop > > mm-sparse-remove-config_sparsemem_alloc_mem_map_together.patch for now. > > I have dropped it from linux-next today (in case you don't get time). I am certain I did drop it, but some how it is still in there :-( I will drop it for tomorrow. -- Cheers, Stephen Rothwell pgpyj6MGOa6aW.pgp Description: OpenPGP digital signature
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
> > I just roughly check, but if I checked the right place,
> > vmemmap_populated() checks for the section to contain the flags we are
> > setting in sparse_init_one_section().
>
> Yes.
>
> > But with this patch, we populate first everything, and then we call
> > sparse_init_one_section() in sparse_init().
> > As I said I could be mistaken because I just checked the surface.
>
> Yeah I think that's correct.
>
> This might just be a bug in our code, let me look at it a bit.
I wonder if something like this could make the trick:
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index 51ce091914f9..e281651f50cd 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -177,6 +177,8 @@ static __meminit void vmemmap_list_populate(unsigned long
phys,
vmemmap_list = vmem_back;
}
+static unsigned long last_addr_populated = 0;
+
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int
node,
struct vmem_altmap *altmap)
{
@@ -191,7 +193,7 @@ int __meminit vmemmap_populate(unsigned long start,
unsigned long end, int node,
void *p;
int rc;
- if (vmemmap_populated(start, page_size))
+ if (start + page_size <= last_addr_populated)
continue;
if (altmap)
@@ -212,6 +214,7 @@ int __meminit vmemmap_populate(unsigned long start,
unsigned long end, int node,
__func__, rc);
return -EFAULT;
}
+ last_addr_populated = start + page_size;
}
I know it looks hacky, and chances are that are wrong, but could you give it a
try?
I will try to grab a ppc server and try it out too.
Thanks
--
Oscar Salvador
SUSE L3
Re: Boot failures with "mm/sparse: Remove CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER" on powerpc (was Re: mmotm 2018-07-10-16-50 uploaded)
On Thu, Jul 12, 2018 at 5:50 AM Oscar Salvador wrote: > > > > I just roughly check, but if I checked the right place, > > > vmemmap_populated() checks for the section to contain the flags we are > > > setting in sparse_init_one_section(). > > > > Yes. > > > > > But with this patch, we populate first everything, and then we call > > > sparse_init_one_section() in sparse_init(). > > > As I said I could be mistaken because I just checked the surface. Yes, this is right, sparse_init_one_section() is needed after every populate call on ppc64. I am adding this to my sparse_init re-write, and it actually simplifies code, as it avoids one extra loop, and makes ppc64 to work. Pavel
