On Tue, Sep 30, 2014 at 03:56:24PM +0800, Fengguang Wu wrote: > Hi Joonsoo, > > 0day kernel testing robot got the below dmesg and the first bad commit is > > git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > > commit 36fbfebe776eb5871d61e7a755c9feb1c96cc4aa > Author: Joonsoo Kim <[email protected]> > AuthorDate: Tue Sep 23 11:52:35 2014 +1000 > Commit: Stephen Rothwell <[email protected]> > CommitDate: Tue Sep 23 11:52:35 2014 +1000 > > mm/slab: support slab merge > > Slab merge is good feature to reduce fragmentation. If new creating slab > have similar size and property with exsitent slab, this feature reuse it > rather than creating new one. As a result, objects are packed into fewer > slabs so that fragmentation is reduced. > > Below is result of my testing. > > * After boot, sleep 20; cat /proc/meminfo | grep Slab > > <Before> > Slab: 25136 kB > > <After> > Slab: 24364 kB > > We can save 3% memory used by slab. > > For supporting this feature in SLAB, we need to implement SLAB specific > kmem_cache_flag() and __kmem_cache_alias(), because SLUB implements some > SLUB specific processing related to debug flag and object size change on > these functions. > > Signed-off-by: Joonsoo Kim <[email protected]> > Cc: Christoph Lameter <[email protected]> > Cc: Pekka Enberg <[email protected]> > Cc: David Rientjes <[email protected]> > Signed-off-by: Andrew Morton <[email protected]> > > +----------------------------------------------+------------+------------+---------------+ > | | 8c087489b8 | 36fbfebe77 | > next-20140923 | > +----------------------------------------------+------------+------------+---------------+ > | boot_successes | 60 | 0 | 1 > | > | boot_failures | 0 | 20 | > 314 | > | BUG:unable_to_handle_kernel | 0 | 20 | > 312 | > | Oops | 0 | 20 | > 312 | > | EIP_is_at_kernfs_link_sibling | 0 | 4 | 14 > | > | Kernel_panic-not_syncing:Fatal_exception | 0 | 20 | > 312 | > | backtrace:acpi_bus_scan | 0 | 4 | 14 > | > | backtrace:acpi_scan_init | 0 | 20 | 45 > | > | backtrace:acpi_init | 0 | 20 | 45 > | > | backtrace:kernel_init_freeable | 0 | 20 | > 312 | > | EIP_is_at_kernfs_add_one | 0 | 16 | > 298 | > | backtrace:kobject_add_internal | 0 | 16 | 31 > | > | backtrace:kobject_init_and_add | 0 | 16 | 31 > | > | backtrace:acpi_scan_add_handler_with_hotplug | 0 | 16 | 31 > | > | backtrace:acpi_pci_root_init | 0 | 16 | 31 > | > | backtrace:tty_register_driver | 0 | 0 | > 106 | > | backtrace:pty_init | 0 | 0 | > 106 | > | backtrace:acpi_bus_register_driver | 0 | 0 | 1 > | > | backtrace:acpi_button_driver_init | 0 | 0 | 1 > | > | BUG:kernel_boot_crashed | 0 | 0 | 1 > | > | BUG:kernel_test_crashed | 0 | 0 | 1 > | > | backtrace:subsys_system_register | 0 | 0 | > 160 | > | backtrace:container_dev_init | 0 | 0 | > 160 | > | backtrace:driver_init | 0 | 0 | > 160 | > +----------------------------------------------+------------+------------+---------------+ > > [ 0.463788] ACPI: (supports S0 S5) > [ 0.464003] ACPI: Using IOAPIC for interrupt routing > [ 0.464738] PCI: Using host bridge windows from ACPI; if necessary, use > "pci=nocrs" and report a bug > [ 0.466034] BUG: unable to handle kernel paging request at 00010023 > [ 0.466989] IP: [<c117dcf9>] kernfs_add_one+0x89/0x130 > [ 0.467812] *pdpt = 0000000000000000 *pde = f000ff53f000ff53 > [ 0.468000] Oops: 0002 [#1] SMP > [ 0.468000] Modules linked in: > [ 0.468000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted > 3.17.0-rc6-00089-g36fbfeb #1 > [ 0.468000] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 0.468000] task: d303ec90 ti: d3040000 task.ti: d3040000 > [ 0.468000] EIP: 0060:[<c117dcf9>] EFLAGS: 00010286 CPU: 0 > [ 0.468000] EIP is at kernfs_add_one+0x89/0x130 > [ 0.468000] EAX: 542572cb EBX: 00010003 ECX: 00000008 EDX: 2c8de598 > [ 0.468000] ESI: d311de10 EDI: d311de70 EBP: d3041dd8 ESP: d3041db4 > [ 0.468000] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 > [ 0.468000] CR0: 8005003b CR2: 00010023 CR3: 01a8a000 CR4: 000006f0 > [ 0.468000] Stack: > [ 0.468000] d3006f00 00000202 d311de70 d311de10 d3041dd8 c117dba0 > d311de10 c159a5c0 > [ 0.468000] c1862a00 d3041df0 c117f0f2 00000000 c18629f4 d311de70 > 00000000 d3041e2c > [ 0.468000] c117f8b5 00001000 00000000 c159a5c0 c18629f4 00000000 > 00000001 c1862a00 > [ 0.468000] Call Trace: > [ 0.468000] [<c117dba0>] ? kernfs_new_node+0x30/0x40 > [ 0.468000] [<c117f0f2>] __kernfs_create_file+0x92/0xc0 > [ 0.468000] [<c117f8b5>] sysfs_add_file_mode_ns+0x95/0x190 > [ 0.468000] [<c117f9d7>] sysfs_create_file_ns+0x27/0x40 > [ 0.468000] [<c1252ef6>] kobject_add_internal+0x136/0x2c0 > [ 0.468000] [<c125e360>] ? kvasprintf+0x40/0x50 > [ 0.468000] [<c1252a92>] ? kobject_set_name_vargs+0x42/0x60 > [ 0.468000] [<c12530b5>] kobject_init_and_add+0x35/0x50 > [ 0.468000] [<c12ad04f>] acpi_sysfs_add_hotplug_profile+0x24/0x4a > [ 0.468000] [<c12a7280>] acpi_scan_add_handler_with_hotplug+0x21/0x28 > [ 0.468000] [<c18df524>] acpi_pci_root_init+0x20/0x22 > [ 0.468000] [<c18df0e1>] acpi_scan_init+0x24/0x16d > [ 0.468000] [<c18def73>] acpi_init+0x20c/0x224 > [ 0.468000] [<c18ded67>] ? acpi_sleep_init+0xab/0xab > [ 0.468000] [<c100041e>] do_one_initcall+0x7e/0x1b0 > [ 0.468000] [<c18ded67>] ? acpi_sleep_init+0xab/0xab > [ 0.468000] [<c18b24ba>] ? repair_env_string+0x12/0x54 > [ 0.468000] [<c18b24a8>] ? initcall_blacklist+0x7c/0x7c > [ 0.468000] [<c105e100>] ? parse_args+0x160/0x3f0 > [ 0.468000] [<c18b2bd1>] kernel_init_freeable+0xfc/0x179 > [ 0.468000] [<c156782b>] kernel_init+0xb/0xd0 > [ 0.468000] [<c1574601>] ret_from_kernel_thread+0x21/0x30 > [ 0.468000] [<c1567820>] ? rest_init+0xb0/0xb0 > [ 0.468000] Code: 26 00 83 e1 10 75 5b 8b 46 24 e8 b3 ea ff ff 89 46 38 89 > f0 e8 d9 f9 ff ff 85 c0 89 c3 75 ca 8b 5f 5c 85 db 74 11 e8 97 90 f1 ff <89> > 43 20 89 53 24 89 43 28 89 53 2c b8 c0 2f 85 c1 e8 d1 36 3f > [ 0.468000] EIP: [<c117dcf9>] kernfs_add_one+0x89/0x130 SS:ESP > 0068:d3041db4 > [ 0.468000] CR2: 0000000000010023 > [ 0.468000] ---[ end trace 4fa173691404b63f ]--- > [ 0.468000] Kernel panic - not syncing: Fatal exception > > git bisect start 55f21306900abf9f9d2a087a127ff49c6d388ad2 > 0f33be009b89d2268e94194dc4fd01a7851b6d51 -- > git bisect good 18c13e2d9b75e2760e6520f2fde00401192956f3 # 17:56 20+ > 0 Merge remote-tracking branch 'bluetooth/master' > git bisect good abf79495f38ba66f750566b3f0a8da8dd94b4dc3 # 18:03 20+ > 0 Merge remote-tracking branch 'ftrace/for-next' > git bisect good 0bed22034e26a3c37ee4407fccffa8c095d5e144 # 18:09 20+ > 0 Merge remote-tracking branch 'pinctrl/for-next' > git bisect good 15c9281a15ed7718868d115d4d00619b0b7a2624 # 18:14 20+ > 0 Merge remote-tracking branch 'clk/clk-next' > git bisect good 50939531dea1b913b7fa29f9bbc69feafefd090c # 18:23 20+ > 0 Merge branch 'rd-docs/master' > git bisect good aa881e3c5e87c8aa23519f40554897d56f32b935 # 18:49 20+ > 0 Merge remote-tracking branch 'powerpc-mpe/next' > git bisect bad 81b63d14db32bd7706c955d1e04e65b152b2277a # 18:57 0- > 2 Merge branch 'akpm-current/current' > git bisect bad f313ca82d72066a3c44fd6c66cee57b25de43aa9 # 19:31 0- > 1 introduce-dump_vma-fix > git bisect good 2c5fe9213048c5640b8e46407f5614038c03ad93 # 20:16 20+ > 0 mm: fix kmemcheck.c build errors > git bisect bad 69454f8be7f621ac8c3c6c9763bb70e116988942 # 20:35 0- > 18 block_dev: implement readpages() to optimize sequential read > git bisect bad 66a31d528a1e3d483be2b1c993ec1268412f0074 # 20:40 0- > 1 memory-hotplug-add-sysfs-zones_online_to-attribute-fix-2 > git bisect good 5e8acb68610c077b08cb3f16305aa3cc22e5d2a8 # 21:07 20+ > 0 kernel/kthread.c: partial revert of 81c98869faa5 ("kthread: ensure > locality of task_struct allocations") > git bisect bad 36fbfebe776eb5871d61e7a755c9feb1c96cc4aa # 22:06 0- > 10 mm/slab: support slab merge > git bisect good 11e57381eced875ef5a6fea4005fdf72b6f68eff # 22:19 20+ > 0 mm/slab_common: commonize slab merge logic > git bisect good 8c087489b8a32b9235f7f9417390c62d93aba522 # 22:24 20+ > 0 mm/slab_common: fix build failure if CONFIG_SLUB > # first bad commit: [36fbfebe776eb5871d61e7a755c9feb1c96cc4aa] mm/slab: > support slab merge > git bisect good 8c087489b8a32b9235f7f9417390c62d93aba522 # 22:26 60+ > 0 mm/slab_common: fix build failure if CONFIG_SLUB > git bisect bad 55f21306900abf9f9d2a087a127ff49c6d388ad2 # 22:26 0- > 314 Add linux-next specific files for 20140923 > git bisect good f4cb707e7ad9727a046b463232f2de166e327d3e # 22:32 60+ > 0 Merge tag 'pm+acpi-3.17-rc7' of > git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm > git bisect bad 4d8426f9ac601db2a64fa7be64051d02b9c9fe01 # 22:36 0- > 60 Add linux-next specific files for 20140926 > > > This script may reproduce the error. > > ---------------------------------------------------------------------------- > #!/bin/bash > > kernel=$1 > > kvm=( > qemu-system-x86_64 > -cpu kvm64 > -enable-kvm > -kernel $kernel > -m 320 > -smp 1 > -net nic,vlan=1,model=e1000 > -net user,vlan=1 > -boot order=nc > -no-reboot > -watchdog i6300esb > -rtc base=localtime > -serial stdio > -display none > -monitor null > ) > > append=( > hung_task_panic=1 > earlyprintk=ttyS0,115200 > debug > apic=debug > sysrq_always_enabled > rcupdate.rcu_cpu_stall_timeout=100 > panic=-1 > softlockup_panic=1 > nmi_watchdog=panic > oops=panic > load_ramdisk=2 > prompt_ramdisk=0 > console=ttyS0,115200 > console=tty0 > vga=normal > root=/dev/ram0 > rw > drbd.minor_count=8 > ) > > "${kvm[@]}" --append "${append[*]}" > ----------------------------------------------------------------------------
Thanks Fengguang. I can reproduce this bug in my system with your configuration. Below is fix for this bug. Hello Andrew, Could you take this patch? Thanks. ----------->8------------------ >From e9c884c1f807dc9de2f4cc3b2779a014e5ebb7a1 Mon Sep 17 00:00:00 2001 From: Joonsoo Kim <[email protected]> Date: Thu, 2 Oct 2014 12:09:41 +0900 Subject: [PATCH] mm/slab: adjust object_size in order to fix bug in slab merge Fengguang reported following bug and his bisect result points to this patch ('mm/slab: support slab merge') as root cause. [ 0.466034] BUG: unable to handle kernel paging request at 00010023 [ 0.466989] IP: [<c117dcf9>] kernfs_add_one+0x89/0x130 [ 0.467812] *pdpt = 0000000000000000 *pde = f000ff53f000ff53 [ 0.468000] Oops: 0002 [#1] SMP [ 0.468000] Modules linked in: [ 0.468000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc6-00089-g36fbfeb #1 [ 0.468000] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 0.468000] task: d303ec90 ti: d3040000 task.ti: d3040000 [ 0.468000] EIP: 0060:[<c117dcf9>] EFLAGS: 00010286 CPU: 0 [ 0.468000] EIP is at kernfs_add_one+0x89/0x130 [ 0.468000] EAX: 542572cb EBX: 00010003 ECX: 00000008 EDX: 2c8de598 [ 0.468000] ESI: d311de10 EDI: d311de70 EBP: d3041dd8 ESP: d3041db4 [ 0.468000] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 0.468000] CR0: 8005003b CR2: 00010023 CR3: 01a8a000 CR4: 000006f0 [ 0.468000] Stack: [ 0.468000] d3006f00 00000202 d311de70 d311de10 d3041dd8 c117dba0 d311de10 c159a5c0 [ 0.468000] c1862a00 d3041df0 c117f0f2 00000000 c18629f4 d311de70 00000000 d3041e2c [ 0.468000] c117f8b5 00001000 00000000 c159a5c0 c18629f4 00000000 00000001 c1862a00 [ 0.468000] Call Trace: [ 0.468000] [<c117dba0>] ? kernfs_new_node+0x30/0x40 [ 0.468000] [<c117f0f2>] __kernfs_create_file+0x92/0xc0 [ 0.468000] [<c117f8b5>] sysfs_add_file_mode_ns+0x95/0x190 [ 0.468000] [<c117f9d7>] sysfs_create_file_ns+0x27/0x40 [ 0.468000] [<c1252ef6>] kobject_add_internal+0x136/0x2c0 [ 0.468000] [<c125e360>] ? kvasprintf+0x40/0x50 [ 0.468000] [<c1252a92>] ? kobject_set_name_vargs+0x42/0x60 [ 0.468000] [<c12530b5>] kobject_init_and_add+0x35/0x50 [ 0.468000] [<c12ad04f>] acpi_sysfs_add_hotplug_profile+0x24/0x4a [ 0.468000] [<c12a7280>] acpi_scan_add_handler_with_hotplug+0x21/0x28 [ 0.468000] [<c18df524>] acpi_pci_root_init+0x20/0x22 [ 0.468000] [<c18df0e1>] acpi_scan_init+0x24/0x16d [ 0.468000] [<c18def73>] acpi_init+0x20c/0x224 [ 0.468000] [<c18ded67>] ? acpi_sleep_init+0xab/0xab [ 0.468000] [<c100041e>] do_one_initcall+0x7e/0x1b0 [ 0.468000] [<c18ded67>] ? acpi_sleep_init+0xab/0xab [ 0.468000] [<c18b24ba>] ? repair_env_string+0x12/0x54 [ 0.468000] [<c18b24a8>] ? initcall_blacklist+0x7c/0x7c [ 0.468000] [<c105e100>] ? parse_args+0x160/0x3f0 [ 0.468000] [<c18b2bd1>] kernel_init_freeable+0xfc/0x179 [ 0.468000] [<c156782b>] kernel_init+0xb/0xd0 [ 0.468000] [<c1574601>] ret_from_kernel_thread+0x21/0x30 [ 0.468000] [<c1567820>] ? rest_init+0xb0/0xb0 [snip...] [ 0.468000] EIP: [<c117dcf9>] kernfs_add_one+0x89/0x130 SS:ESP 0068:d3041db4 [ 0.468000] CR2: 0000000000010023 [ 0.468000] ---[ end trace 4fa173691404b63f ]--- [ 0.468000] Kernel panic - not syncing: Fatal exception This error is caused by wrongly initialized object due to slab merge. Size of vm_area_struct is 92 bytes in this configuration, and, for better alignment, this kmem_cache manage memory in 96 bytes unit. But, maybe for performance reason, if user requests zeroing for this object, object is cleared up to 92 bytes. Meanwhile, size of kernfs_node_cache is 96 bytes so that it can be merged with kmem_cache for vm_area_struct. In this situation, if user request zeroing for objects for kernfs_node_cache, object is only cleared up to 92 bytes. So, kernfs_node had odd value on iattr field and this results in de-referencing wrong address bug. To fix this problem, object size is adjusted when merging occurs. After this change, zeroing will be done to complete object so that de-referencing wrong address can't happen. Reported-by: Fengguang Wu <[email protected]> Signed-off-by: Joonsoo Kim <[email protected]> --- mm/slab.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/slab.c b/mm/slab.c index c55b1ec..6307131 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2118,9 +2118,15 @@ __kmem_cache_alias(const char *name, size_t size, size_t align, struct kmem_cache *cachep; cachep = find_mergeable(size, align, flags, name, ctor); - if (cachep) + if (cachep) { cachep->refcount++; + /* + * Adjust the object sizes so that we clear + * the complete object on kzalloc. + */ + cachep->object_size = max_t(int, cachep->object_size, size); + } return cachep; } -- 1.7.9.5 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
