* Sasha Levin <sasha.le...@oracle.com> wrote: > Hi all, > > While fuzzing with trinity inside the latest -next kernel using trinity I've > stumbled on: > > [ 936.784266] divide error: 0000 [#1] PREEMPT SMP KASAN > [ 936.793957] RIP: find_busiest_group (kernel/sched/fair.c:6152 > kernel/sched/fair.c:6223 kernel/sched/fair.c:6341 kernel/sched/fair.c:6603)
Hm, these line numbers don't seem to match up very well with my version of linux-next: 28855005be1d Add linux-next specific files for 20150306 and the Git version info included in the oops seems useless: 4.0.0-rc1-sasha-00044-ge21109a $ git log e21109a fatal: ambiguous argument 'e21109a': unknown revision or path not in the working tree. I think the kernel's SHA1 should be made at least 12 char wide, regardless of the user's gitconfig::core.abbrev settings. Also, latest linux-next is -rc2 based, while your version string says -rc1. > [ 936.829403] load_balance (kernel/sched/fair.c:6857) this does not match up either. > [ 936.829403] run_rebalance_domains (kernel/sched/fair.c:7494 > kernel/sched/fair.c:7659) The line numbers are not even close to anything related: run_rebalance_domains() starts at line 7666 and ends at 7680. Also, why are the offsets into the function missing from the output? Those would allow the rough determination of the crash site, even if debuginfo is crap. I also checked Linus's latest, and they do seem to match up better: affb8172de39 Merge git://git.kernel.org/pub/scm/virt/kvm/kvm and the line number gives: capacity_factor = min_t(unsigned, capacity_factor, DIV_ROUND_CLOSEST(capacity, SCHED_CAPACITY_SCALE)); but that's a division with a constant? Should not trap. So I rebuild a kernel with debug info, pattern matched the disassembly you provided, and that gave me this division: (gdb) list *0xffffffff8107d958 0xffffffff8107d958 is in find_busiest_group (kernel/sched/fair.c:6162). 6157 capacity = group->sgc->capacity; 6158 capacity_orig = group->sgc->capacity_orig; 6159 cpus = group->group_weight; 6160 6161 /* smt := ceil(cpus / capacity), assumes: 1 < smt_capacity < 2 */ 6162 smt = DIV_ROUND_UP(SCHED_CAPACITY_SCALE * cpus, capacity_orig); 6163 capacity_factor = cpus / smt; /* cores */ 6164 6165 capacity_factor = min_t(unsigned, 6166 capacity_factor, DIV_ROUND_CLOSEST(capacity, SCHED_CAPACITY_SCALE)); So this too seems not very plausible: 'capacity_orig' comes straight from group->sgc->capacity_orig, which is: - boot time initialized - sometimes recalculated during CPU hot-plug: not sure how much of that your tests are doing? - but otherwise it's fairly constant and should have crashed your system early on if it was set up wrong unless I missed something that is. > [ 936.829403] __do_softirq (kernel/softirq.c:273 > include/linux/jump_label.h:114 include/trace/events/irq.h:126 > kernel/softirq.c:274) > [ 936.829403] irq_exit (kernel/softirq.c:350 kernel/softirq.c:391) > [ 936.829403] smp_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:918) > [ 936.829403] apic_timer_interrupt (arch/x86/kernel/entry_64.S:958) > [ 936.829403] <EOI> > [ 936.829403] print_context_stack (arch/x86/kernel/dumpstack.c:105) > [ 936.829403] dump_trace (arch/x86/kernel/dumpstack_64.c:244) > [ 936.829403] save_stack_trace (arch/x86/kernel/stacktrace.c:64) > [ 936.829403] __set_page_owner (mm/page_owner.c:72) > [ 936.829403] get_page_from_freelist (include/linux/page_owner.h:26 > mm/page_alloc.c:2176) > [ 936.829403] __alloc_pages_nodemask (mm/page_alloc.c:2844) > [ 936.829403] alloc_pages_vma (mm/mempolicy.c:2007) > [ 936.829403] handle_mm_fault (mm/memory.c:2156 mm/memory.c:3164 > mm/memory.c:3269 mm/memory.c:3298) > [ 936.829403] __do_page_fault (arch/x86/mm/fault.c:1235) > [ 936.829403] trace_do_page_fault (arch/x86/mm/fault.c:1329) > [ 936.829403] do_async_page_fault (arch/x86/kernel/kvm.c:280) > [ 936.829403] async_page_fault (arch/x86/kernel/entry_64.S:1295) So debug info weirdnesses aside, other divisions in find_busiest_group(): sds.avg_load = (SCHED_CAPACITY_SCALE * sds.total_load) / sds.total_capacity; total_capacity ought to be zero only on a totally borked machine (unlikely to boot), or on memory corruption. if calculate_imbalance() got inlined, then: load_above_capacity /= busiest->group_capacity; that too ought to only get corrupted in the most serious cases, we don't recalculate it runtime. So I'm baffled. Some tentative handwaving, pointing away from the scheduler: - Your stack trace is 'weird' not just due to debug info: an async page fault doing allocations, doing a stack trace, interruted by a timer irq, doing scheduler rebalancing... - The (spectacularly misnamed [*] ) CONFIG_PAGE_OWNER=y page lifetime tracing facility got enabled explicitly via the page_owner=on boot parameter, right? Not many people are doing that I suspect. - CONFIG_KASAN=y is enabled in your kernel. New, invasive option, using compiler features that weren't used by kernel code before. - async page faults are virtualization specials: not used much elsewhere. - There's a 'W' taint in your oops. Probably some harmless prior warning? So your crash signature has the combination of 3 'uncommon' kernel features, and a scheduler crash with a relatively constant value that should never be zero and which should crash everywhere. So right now I'd blame the other 3 guys, I wasn't even there that night, officer! Cc:-ed others as well. Thanks, Ingo [*] Please name debugging features accordingly: CONFIG_DEBUG_PAGE_OWNER. Maybe even prefix them with the subsystem: CONFIG_DEBUG_VM_PAGE_OWNER. We already have a nice set of CONFIG_DEBUG_VM* options: CONFIG_DEBUG_VM CONFIG_DEBUG_VM_RB CONFIG_DEBUG_VM_VMACACHE Also, there's no penalty for including a verb, so that people know wth it's doing, at a glance: CONFIG_DEBUG_VM_TRACK_PAGE_OWNER? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/