Re: Kernel 4.1.6 Panic due to slab corruption
On Thu, 10 Sep 2015, Nikolay Borisov wrote: > > echo 1 >santy_checks > > [root@kernighan linux-stable]# cd /sys/kernel/slab/kmalloc-32/ > [root@kernighan kmalloc-32]# echo 1 > sanity_checks > [root@kernighan kmalloc-32]# cat sanity_checks > 1 > > So this works as expected when set by echo. Just for testing I then > tried the following: > [root@kernighan kmalloc-32]# slabinfo -d- kmalloc-32 > kmalloc-32 not empty cannot disable sanity checks > > [root@kernighan kmalloc-32]# echo 0 > sanity_checks > [root@kernighan kmalloc-32]# slabinfo -d- kmalloc-32 > > So turns out slabinfo fails where the raw sys interface succeeds, strange? Weird. slabinfo needs fixing. > > do? Could crash the sysem due to overload of messages. > > Didn't have that much luck with this one: > [root@kernighan kmalloc-32]# dmesg -c > /dev/null > [root@kernighan kmalloc-32]# echo 1 > trace > -bash: echo: write error: Invalid argument Huh? There is no check that I am ware of in the slab code that would return -EINVAL. > > Sanity checking is ok. But I would think you should be fine with enabling > > full debugging on the particular caches of interest. > > I was just thinking that if enabling debug options disables merging this > means it won't be sufficient to enable debugging on kmalloc-32 but > rather before enabling debugging I do need to check which caches were > aliased and enable debugging on those as well, correct? Correct. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On 09/09/2015 05:01 PM, Christoph Lameter wrote: > On Wed, 9 Sep 2015, Nikolay Borisov wrote: > >> [root@kernighan vm]# ./slabinfo -da kmalloc-32 >> Cannot write to dma-kmalloc-32/sanity >> [root@kernighan vm]# ./slabinfo -dF kmalloc-32 >> Cannot write to dma-kmalloc-32/sanity >> [root@kernighan vm]# ./slabinfo -dz kmalloc-32 >> kmalloc-32 not empty cannot enable redzoning >> [root@kernighan vm]# ./slabinfo -dp kmalloc-32 >> kmalloc-32 not empty cannot enable poisoning >> [root@kernighan vm]# ./slabinfo -du kmalloc-32 >> kmalloc-32 not empty cannot enable tracking >> [root@kernighan vm]# ./slabinfo -dt ^kmalloc-32$ >> kmalloc-32 can only enable trace for one slab at a time > > > H.. Whats the problem here? > > christoph@gentwo:/sys/kernel/slab/kmalloc-32$ ls -l > total 0 > -r 1 root root 4096 Sep 9 08:57 aliases > -r 1 root root 4096 Sep 9 08:57 align > -r 1 root root 4096 Sep 9 08:57 alloc_calls > -r 1 root root 4096 Sep 9 08:57 cache_dma > -rw--- 1 root root 4096 Sep 9 08:57 cpu_partial > -r 1 root root 4096 Sep 9 08:57 cpu_slabs > -r 1 root root 4096 Sep 9 08:57 ctor > -r 1 root root 4096 Sep 9 08:57 destroy_by_rcu > -r 1 root root 4096 Sep 9 08:57 free_calls > -r 1 root root 4096 Sep 9 08:57 hwcache_align > -rw--- 1 root root 4096 Sep 9 08:57 min_partial > -r 1 root root 4096 Sep 9 08:57 objects > -r 1 root root 4096 Sep 9 08:57 object_size > -r 1 root root 4096 Sep 9 08:57 objects_partial > -r 1 root root 4096 Sep 9 08:57 objs_per_slab > -rw--- 1 root root 4096 Sep 9 08:57 order > -r 1 root root 4096 Sep 9 08:57 partial > -rw--- 1 root root 4096 Sep 9 08:57 poison > -rw--- 1 root root 4096 Sep 9 08:57 reclaim_account > -rw--- 1 root root 4096 Sep 9 08:57 red_zone > -rw--- 1 root root 4096 Sep 9 08:57 remote_node_defrag_ratio > -r 1 root root 4096 Sep 9 08:57 reserved > -rw--- 1 root root 4096 Sep 9 08:57 sanity_checks > -rw--- 1 root root 4096 Sep 9 08:57 shrink > -r 1 root root 4096 Sep 9 08:57 slabs > -r 1 root root 4096 Sep 9 08:57 slabs_cpu_partial > -r 1 root root 4096 Sep 9 08:57 slab_size > -rw--- 1 root root 4096 Sep 9 08:57 store_user > -r 1 root root 4096 Sep 9 08:57 total_objects > -rw--- 1 root root 4096 Sep 9 08:57 trace > -rw--- 1 root root 4096 Sep 9 08:57 validate > > Try > > echo 1 >santy_checks [root@kernighan linux-stable]# cd /sys/kernel/slab/kmalloc-32/ [root@kernighan kmalloc-32]# echo 1 > sanity_checks [root@kernighan kmalloc-32]# cat sanity_checks 1 So this works as expected when set by echo. Just for testing I then tried the following: [root@kernighan kmalloc-32]# slabinfo -d- kmalloc-32 kmalloc-32 not empty cannot disable sanity checks [root@kernighan kmalloc-32]# echo 0 > sanity_checks [root@kernighan kmalloc-32]# slabinfo -d- kmalloc-32 So turns out slabinfo fails where the raw sys interface succeeds, strange? > > >> >> I did however had success with enabling tracing but couldn't see where >> the output is produced - tried dmesg and the ftrace buffer but nothing >> turned up. > > dmesg is the output channel for tracing. > > What does: > > echo 1 >trace > > do? Could crash the sysem due to overload of messages. Didn't have that much luck with this one: [root@kernighan kmalloc-32]# dmesg -c > /dev/null [root@kernighan kmalloc-32]# echo 1 > trace -bash: echo: write error: Invalid argument > >> But it seems it is not possible to enable any debugging whatsoever, so I >> will restor to doing it at boot time. In this case can you advice which >> options might not result in very high performance degradation - I'm >> thinking of sanity checking and maybe redzoning? > > Sanity checking is ok. But I would think you should be fine with enabling > full debugging on the particular caches of interest. I was just thinking that if enabling debug options disables merging this means it won't be sufficient to enable debugging on kmalloc-32 but rather before enabling debugging I do need to check which caches were aliased and enable debugging on those as well, correct? Regards, Nikolay -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Wed, 9 Sep 2015, Vlastimil Babka wrote: > On 09/09/2015 04:01 PM, Christoph Lameter wrote: > > On Wed, 9 Sep 2015, Nikolay Borisov wrote: > > > > What does: > > > > echo 1 >trace > > > > do? Could crash the sysem due to overload of messages. > > Yes I've seen that happen. Did you consider hooking it to trace_printk() > instead of printk()? The code was there earlier than trace_printk as far as I can tell and the formatting functions are based on printk. Hmmm.. the pr_info in the trace() function could use trace_printk() but that leaves the stack_dump(). -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On 09/09/2015 04:01 PM, Christoph Lameter wrote: On Wed, 9 Sep 2015, Nikolay Borisov wrote: What does: echo 1 >trace do? Could crash the sysem due to overload of messages. Yes I've seen that happen. Did you consider hooking it to trace_printk() instead of printk()? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Wed, 9 Sep 2015, Nikolay Borisov wrote: > [root@kernighan vm]# ./slabinfo -da kmalloc-32 > Cannot write to dma-kmalloc-32/sanity > [root@kernighan vm]# ./slabinfo -dF kmalloc-32 > Cannot write to dma-kmalloc-32/sanity > [root@kernighan vm]# ./slabinfo -dz kmalloc-32 > kmalloc-32 not empty cannot enable redzoning > [root@kernighan vm]# ./slabinfo -dp kmalloc-32 > kmalloc-32 not empty cannot enable poisoning > [root@kernighan vm]# ./slabinfo -du kmalloc-32 > kmalloc-32 not empty cannot enable tracking > [root@kernighan vm]# ./slabinfo -dt ^kmalloc-32$ > kmalloc-32 can only enable trace for one slab at a time H.. Whats the problem here? christoph@gentwo:/sys/kernel/slab/kmalloc-32$ ls -l total 0 -r 1 root root 4096 Sep 9 08:57 aliases -r 1 root root 4096 Sep 9 08:57 align -r 1 root root 4096 Sep 9 08:57 alloc_calls -r 1 root root 4096 Sep 9 08:57 cache_dma -rw--- 1 root root 4096 Sep 9 08:57 cpu_partial -r 1 root root 4096 Sep 9 08:57 cpu_slabs -r 1 root root 4096 Sep 9 08:57 ctor -r 1 root root 4096 Sep 9 08:57 destroy_by_rcu -r 1 root root 4096 Sep 9 08:57 free_calls -r 1 root root 4096 Sep 9 08:57 hwcache_align -rw--- 1 root root 4096 Sep 9 08:57 min_partial -r 1 root root 4096 Sep 9 08:57 objects -r 1 root root 4096 Sep 9 08:57 object_size -r 1 root root 4096 Sep 9 08:57 objects_partial -r 1 root root 4096 Sep 9 08:57 objs_per_slab -rw--- 1 root root 4096 Sep 9 08:57 order -r 1 root root 4096 Sep 9 08:57 partial -rw--- 1 root root 4096 Sep 9 08:57 poison -rw--- 1 root root 4096 Sep 9 08:57 reclaim_account -rw--- 1 root root 4096 Sep 9 08:57 red_zone -rw--- 1 root root 4096 Sep 9 08:57 remote_node_defrag_ratio -r 1 root root 4096 Sep 9 08:57 reserved -rw--- 1 root root 4096 Sep 9 08:57 sanity_checks -rw--- 1 root root 4096 Sep 9 08:57 shrink -r 1 root root 4096 Sep 9 08:57 slabs -r 1 root root 4096 Sep 9 08:57 slabs_cpu_partial -r 1 root root 4096 Sep 9 08:57 slab_size -rw--- 1 root root 4096 Sep 9 08:57 store_user -r 1 root root 4096 Sep 9 08:57 total_objects -rw--- 1 root root 4096 Sep 9 08:57 trace -rw--- 1 root root 4096 Sep 9 08:57 validate Try echo 1 >santy_checks > > I did however had success with enabling tracing but couldn't see where > the output is produced - tried dmesg and the ftrace buffer but nothing > turned up. dmesg is the output channel for tracing. What does: echo 1 >trace do? Could crash the sysem due to overload of messages. > But it seems it is not possible to enable any debugging whatsoever, so I > will restor to doing it at boot time. In this case can you advice which > options might not result in very high performance degradation - I'm > thinking of sanity checking and maybe redzoning? Sanity checking is ok. But I would think you should be fine with enabling full debugging on the particular caches of interest. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On 09/08/2015 06:15 PM, Christoph Lameter wrote: > On Tue, 8 Sep 2015, Nikolay Borisov wrote: > >>> You have read https://www.kernel.org/doc/Documentation/vm/slub.txt? >> >> I've read that I'm also following the merge/nomerge thread on the DM >> mailing list. I guess my understanding is wrong in that if multiple slab >> caches are merged, then it's enough to just instrument the cache to >> which they are being merge in order to have them all instrumented? I >> guess that's not the case, so even though slab caches might be merge >> they are still somehow considered different entities in the kernel? > > Enabling debugging on bootup disables merging. > > If you switch debug options later then its possible to affect all aliases > caches. But then these option changes are only allowed to be used in such > a way as to not affect the object layout. You can switch on sanity checks > and double free checks I believe but nothing else. I tried the following: [root@kernighan vm]# ./slabinfo -da kmalloc-32 Cannot write to dma-kmalloc-32/sanity [root@kernighan vm]# ./slabinfo -dF kmalloc-32 Cannot write to dma-kmalloc-32/sanity [root@kernighan vm]# ./slabinfo -dz kmalloc-32 kmalloc-32 not empty cannot enable redzoning [root@kernighan vm]# ./slabinfo -dp kmalloc-32 kmalloc-32 not empty cannot enable poisoning [root@kernighan vm]# ./slabinfo -du kmalloc-32 kmalloc-32 not empty cannot enable tracking [root@kernighan vm]# ./slabinfo -dt ^kmalloc-32$ kmalloc-32 can only enable trace for one slab at a time I did however had success with enabling tracing but couldn't see where the output is produced - tried dmesg and the ftrace buffer but nothing turned up. But it seems it is not possible to enable any debugging whatsoever, so I will restor to doing it at boot time. In this case can you advice which options might not result in very high performance degradation - I'm thinking of sanity checking and maybe redzoning? > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Tue, 8 Sep 2015, Nikolay Borisov wrote: > > You have read https://www.kernel.org/doc/Documentation/vm/slub.txt? > > I've read that I'm also following the merge/nomerge thread on the DM > mailing list. I guess my understanding is wrong in that if multiple slab > caches are merged, then it's enough to just instrument the cache to > which they are being merge in order to have them all instrumented? I > guess that's not the case, so even though slab caches might be merge > they are still somehow considered different entities in the kernel? Enabling debugging on bootup disables merging. If you switch debug options later then its possible to affect all aliases caches. But then these option changes are only allowed to be used in such a way as to not affect the object layout. You can switch on sanity checks and double free checks I believe but nothing else. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On 09/08/2015 05:27 PM, Christoph Lameter wrote: > On Tue, 8 Sep 2015, Nikolay Borisov wrote: > >> Unfortunately I haven't found a way to reproduce it so the only option >> would be to do this on a live server. However, the performance impact I >> believe is going to be very prohibitive :(. Alternatively what I could >> do is probably leave merging on but enable debugging only for the >> kmalloc-32 slab cache. Do you think this would provide enough >> information to help track the corruption when it happens, without >> impacting performance? > > You have read https://www.kernel.org/doc/Documentation/vm/slub.txt? I've read that I'm also following the merge/nomerge thread on the DM mailing list. I guess my understanding is wrong in that if multiple slab caches are merged, then it's enough to just instrument the cache to which they are being merge in order to have them all instrumented? I guess that's not the case, so even though slab caches might be merge they are still somehow considered different entities in the kernel? > > > The problem now is that merging is on so it could be that the corruption > happens in one of the aliased caches. So maybe only kmalloc-32 wont do > much good. > > Run > > slabinfo -a > > (slabinfo.c is a tool in the kernel tree.) > > to see the list of aliases for kmalloc-32. > > You can also use slabinfo to enable some debugging at runtime. Just > enabling sanity checks may catch something that allows us to track this to > the subsystem. I will experiment with slabinfo. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Tue, 8 Sep 2015, Nikolay Borisov wrote: > Unfortunately I haven't found a way to reproduce it so the only option > would be to do this on a live server. However, the performance impact I > believe is going to be very prohibitive :(. Alternatively what I could > do is probably leave merging on but enable debugging only for the > kmalloc-32 slab cache. Do you think this would provide enough > information to help track the corruption when it happens, without > impacting performance? You have read https://www.kernel.org/doc/Documentation/vm/slub.txt? The problem now is that merging is on so it could be that the corruption happens in one of the aliased caches. So maybe only kmalloc-32 wont do much good. Run slabinfo -a (slabinfo.c is a tool in the kernel tree.) to see the list of aliases for kmalloc-32. You can also use slabinfo to enable some debugging at runtime. Just enabling sanity checks may catch something that allows us to track this to the subsystem. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On 09/08/2015 04:58 PM, Christoph Lameter wrote: > On Mon, 7 Sep 2015, Nikolay Borisov wrote: > >> Did a bit more investigation and it turns out the >> corruption is happening in slab_alloc_node, in the >> 'else' branch when get_freepointer is being called: > > Please reboot the system and specify > > slub_debug > Unfortunately I haven't found a way to reproduce it so the only option would be to do this on a live server. However, the performance impact I believe is going to be very prohibitive :(. Alternatively what I could do is probably leave merging on but enable debugging only for the kmalloc-32 slab cache. Do you think this would provide enough information to help track the corruption when it happens, without impacting performance? > on the kernel command line. This will enable additional diagnostics which > will allow tracking down the issue to the subsystem causing it. > > > Or rebuild with > > CONFIG_SLUB_DEBUG_ON > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Mon, 7 Sep 2015, Nikolay Borisov wrote: > Did a bit more investigation and it turns out the > corruption is happening in slab_alloc_node, in the > 'else' branch when get_freepointer is being called: Please reboot the system and specify slub_debug on the kernel command line. This will enable additional diagnostics which will allow tracking down the issue to the subsystem causing it. Or rebuild with CONFIG_SLUB_DEBUG_ON -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Mon, 07 Sep 2015 11:49:12 +, Holger Hoffstätte wrote: > On Mon, 07 Sep 2015 14:30:49 +0300, Nikolay Borisov wrote: > >> If you have the vmlinux image for the kernel you were running at the >> time, the crash occured, could you post the output of addr2line -f -e >> path/to/vmlinux 8115bd4d to see if it also fails in >> get_freepointer. > > Had to rebuild to get an uncompressed vmlinux, but here it is: > > holger>addr2line -f -e vmlinux 8115bd4d > kmem_cache_alloc > ??:? > > Not sure how much we can learn from this. :} Also for what it's worth the related splatter is all in the same place: holger>zgrep 8115bd4d /var/log/kern.log.0.gz Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:03 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:03 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:03 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:10 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:10 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:10 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:10 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:10 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:10 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 CPU is a 4 core HT i7 (aka 8 vcores), and so I have 8 splats. thanks, Holger -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
On Mon, 07 Sep 2015 14:30:49 +0300, Nikolay Borisov wrote: > If you have the vmlinux image for the kernel you were running at the > time, the crash occured, could you post the output of addr2line -f -e > path/to/vmlinux 8115bd4d to see if it also fails in > get_freepointer. Had to rebuild to get an uncompressed vmlinux, but here it is: holger>addr2line -f -e vmlinux 8115bd4d kmem_cache_alloc ??:? Not sure how much we can learn from this. :} -h -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Kernel 4.1.6 Panic due to slab corruption
Hi, If you have the vmlinux image for the kernel you were running at the time, the crash occured, could you post the output of addr2line -f -e path/to/vmlinux 8115bd4d to see if it also fails in get_freepointer. Regards, Nikolay On 09/07/2015 01:37 PM, Holger Hoffstätte wrote: > On Mon, 07 Sep 2015 11:41:17 +0300, Nikolay Borisov wrote: > >> Hello, >> >> On one of our servers I've observed the a kernel pannic >> happening with the following backtrace: >> >> [654405.527070] BUG: unable to handle kernel paging request at >> 00028001 >> [654405.527076] IP: [] kmem_cache_alloc_node+0x99/0x1e0 >> [654405.527085] PGD 14bef58067 PUD 2ab358067 PMD 0 > > Interesting! I can't offer much help but had a similar panic just the other > day > for no apparent reason while running a bunch of compiles. First time I've seen > this with 4.1.6: > > Sep 5 20:42:02 ragnarok kernel: BUG: unable to handle kernel paging request > at 8800e789b740 > Sep 5 20:42:02 ragnarok kernel: IP: [] > kmem_cache_alloc+0x6d/0x160 > Sep 5 20:42:02 ragnarok kernel: PGD 1aa2067 PUD 61f7fd067 PMD 0 > Sep 5 20:42:02 ragnarok kernel: Oops: [#1] SMP > Sep 5 20:42:02 ragnarok kernel: Modules linked in: auth_rpcgss oid_registry > nfsv4 nfs lockd grace fscache sunrpc autofs4 sch_fq_codel > snd_hda_codec_realtek x86_pkg_temp_thermal coretemp snd_hda_codec_generic > crc32_pclmul crc32c_intel aesni_intel radeon aes_x86_64 glue_helper > snd_hda_codec_hdmi lrw gf128mul ablk_helper cryptd i2c_algo_bit snd_usb_audio > uvcvideo snd_hda_intel drm_kms_helper snd_hda_controller snd_hwdep > videobuf2_vmalloc snd_usbmidi_lib videobuf2_memops snd_hda_codec > videobuf2_core snd_rawmidi i2c_i801 ttm snd_hda_core v4l2_common > snd_seq_device videodev snd_pcm usbhid drm snd_timer r8169 snd i2c_core mii > soundcore parport_pc parport > Sep 5 20:42:02 ragnarok kernel: CPU: 0 PID: 32755 Comm: sh Not tainted 4.1.6 > #1 > Sep 5 20:42:02 ragnarok kernel: Hardware name: Gigabyte Technology Co., Ltd. > P67-DS3-B3/P67-DS3-B3, BIOS F1 05/06/2011 > Sep 5 20:42:02 ragnarok kernel: task: 880569712e20 ti: 8804e4d9 > task.ti: 8804e4d9 > Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] > [] kmem_cache_alloc+0x6d/0x160 > Sep 5 20:42:02 ragnarok kernel: RSP: 0018:8804e4d93d88 EFLAGS: 00010282 > Sep 5 20:42:02 ragnarok kernel: RAX: RBX: 8805e7eacce0 > RCX: 0001f7e8 > Sep 5 20:42:02 ragnarok kernel: RDX: 0001f7e7 RSI: 00d0 > RDI: 00018c70 > Sep 5 20:42:02 ragnarok kernel: RBP: 8804e4d93dc8 R08: 88061f418c70 > R09: > Sep 5 20:42:02 ragnarok kernel: R10: 81748318 R11: ea00139bb500 > R12: 00d0 > Sep 5 20:42:02 ragnarok kernel: R13: 880606890600 R14: 8100d039 > R15: 8800e789b740 > Sep 5 20:42:02 ragnarok kernel: FS: 7f9c1d2f2700() > GS:88061f40() knlGS: > Sep 5 20:42:02 ragnarok kernel: CS: 0010 DS: ES: CR0: > 80050033 > Sep 5 20:42:02 ragnarok kernel: CR2: 8800e789b740 CR3: 0005f68ce000 > CR4: 000406f0 > Sep 5 20:42:02 ragnarok kernel: Stack: > Sep 5 20:42:02 ragnarok kernel: 88061f7e6c00 > 0002 8805e7eacce0 > Sep 5 20:42:02 ragnarok kernel: 880569712e20 01200011 > 8805e7eacce0 880569712e20 > Sep 5 20:42:02 ragnarok kernel: 8804e4d93de8 8100d039 > 7f9c1d2f29d0 > Sep 5 20:42:02 ragnarok kernel: Call Trace: > Sep 5 20:42:02 ragnarok kernel: [] > arch_dup_task_struct+0x69/0x170 > Sep 5 20:42:02 ragnarok kernel: [] > copy_process.part.8+0x14f/0x1760 > Sep 5 20:42:02 ragnarok kernel: [] ? > handle_mm_fault+0xd0f/0x13a0 > Sep 5 20:42:02 ragnarok kernel: [] ? > get_empty_filp+0xd4/0x1c0 > Sep 5 20:42:02 ragnarok kernel: [] ? > recalc_sigpending+0x1f/0x60 > Sep 5 20:42:02 ragnarok kernel: [] do_fork+0xd7/0x370 > Sep 5 20:42:02 ragnarok kernel: [] ? sigprocmask+0x57/0x90 > Sep 5 20:42:02 ragnarok kernel: [] SyS_clone+0x16/0x20 > Sep 5 20:42:02 ragnarok kernel: [] > system_call_fastpath+0x12/0x6a > Sep 5 20:42:02 ragnarok kernel: Code: 65 4c 03 05 ee e3 ea 7e 49 83 78 10 00 > 4d 8b 38 0f 84 b0 00 00 00 4d 85 ff 0f 84 a7 00 00 00 49 63 45 20 48 8d 4a 01 > 49 8b 7d 00 <49> 8b 1c 07 4c 89 f8 65 48 0f c7 0f 0f 94 c0 84 c0 74 b9 49 63 > Sep 5 20:42:02 ragnarok kernel: RIP [] > kmem_cache_alloc+0x6d/0x160 > Sep 5 20:42:02 ragnarok kernel: RSP > Sep 5 20:42:02 ragnarok kernel: CR2: 8800e789b740 > Sep 5 20:42:02 ragnarok kernel: ---[ end trace e4478715791f5752 ]--- > Sep 5 20:42:02 ragnarok kernel: BUG: unable to handle kernel paging request > at 8800e789b740 > Sep 5 20:42:02 ragnarok kernel: IP: [] > kmem_cache_alloc+0x6d/0x160 > Sep 5 20:42:02 ragnarok kernel: PGD 1aa2067 PUD 61f7fd067 PMD 0 > Sep 5 20:42:02 ragnarok kernel: Oops: [#2] SMP > Sep 5 20:42:02 ragnarok k
Re: Kernel 4.1.6 Panic due to slab corruption
Did a bit more investigation and it turns out the corruption is happening in slab_alloc_node, in the 'else' branch when get_freepointer is being called: 0x81182a50 <+144>: movsxd rax,DWORD PTR [r12+0x20] 0x81182a55 <+149>: movrdi,QWORD PTR [r12] 0x81182a59 <+153>: movrbx,QWORD PTR [r13+rax*1+0x0] The problematic line is the +153 offset, running addr2line shows that, this is get_freepointer: addr2line -f -e vmlinux-4.1.6-clouder1 81182a59 get_freepointer /home/projects/linux-stable/mm/slub.c:247 In this case the values of the arguments of this function are completely bogus (or so it seems): 1. RAX is shown to be 0 and rax is supposed to hold the pointer to struct kmem_cache. But curiously there isn't an error for NULL ptr, as well as the check for the return value of slab_pre_alloc_hook would have terminated the function early. 2. The value of r13 (which holds the pointer to the first free object from the freelist) is also bogus: 00028001 I'm a bit puzzled as to why am I not getting a NULL ptr error. But in any case it looks that the per-cpu slub cache freelist has been corrupted. Doing addr2line on the other paging request failures also show that the issue is in the same function - get_freepointer: addr2line -f -e vmlinux-4.1.6-clouder1 811824e5 get_freepointer /home/projects/linux-stable/mm/slub.c:247 Regards, Nikolay On 09/07/2015 11:41 AM, Nikolay Borisov wrote: > Hello, > > On one of our servers I've observed the a kernel pannic > happening with the following backtrace: > > [654405.527070] BUG: unable to handle kernel paging request at > 00028001 > [654405.527076] IP: [] kmem_cache_alloc_node+0x99/0x1e0 > [654405.527085] PGD 14bef58067 PUD 2ab358067 PMD 0 > [654405.527089] Oops: [#11] SMP > [654405.527093] Modules linked in: xt_multiport tcp_diag inet_diag act_police > cls_basic sch_ingress scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 > xt_pkttype xt_state veth openvswitch xt_owner xt_conntrack iptable_filter > iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 > nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ip_tables ib_ipoib rdma_ucm > ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr > ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio libcrc32c > dm_mirror dm_region_hash dm_log iTCO_wdt iTCO_vendor_support sb_edac > edac_core i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core ioatdma dca > ipmi_devintf ipmi_si ipmi_msghandler mpt2sas scsi_transport_sas raid_class > [654405.527145] CPU: 14 PID: 32267 Comm: httpd Tainted: G D L > 4.1.6-clouder1 #1 > [654405.527147] Hardware name: Supermicro > X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0 07/09/2013 > [654405.527149] task: 88139d3b1ec0 ti: 8808eda14000 task.ti: > 8808eda14000 > [654405.527151] RIP: 0010:[] [] > kmem_cache_alloc_node+0x99/0x1e0 > [654405.527155] RSP: 0018:88407fcc3a98 EFLAGS: 00210246 > [654405.527156] RAX: RBX: RCX: > 8814ce9acf80 > [654405.527157] RDX: 837ad864 RSI: 00050200 RDI: > 00018ce0 > [654405.527158] RBP: 88407fcc3af8 R08: 88407fcd8ce0 R09: > a033d990 > [654405.527159] R10: 88058676fdd8 R11: 7b4a R12: > 881fff807ac0 > [654405.527161] R13: 00028001 R14: 0001 R15: > 881fff807ac0 > [654405.527162] FS: () GS:88407fcc(0063) > knlGS:55c832e0 > [654405.527164] CS: 0010 DS: 002b ES: 002b CR0: 80050033 > [654405.527165] CR2: 00028001 CR3: 001467b64000 CR4: > 000406e0 > [654405.527166] Stack: > [654405.527167] > 881ff2d05000 > [654405.527170] 88407fcc3ae8 00050200812b5903 88407fcc3ae8 > 01a2 > [654405.527172] 0001 88058676fc60 88058676fe80 > 1800 > [654405.527175] Call Trace: > [654405.527177] > [654405.527184] [] ovs_flow_stats_update+0x110/0x160 > [openvswitch] > [654405.527189] [] ovs_dp_process_packet+0x64/0xf0 > [openvswitch] > [654405.527193] [] ? netdev_port_receive+0x110/0x110 > [openvswitch] > [654405.527197] [] ? netdev_port_receive+0x110/0x110 > [openvswitch] > [654405.527201] [] ovs_vport_receive+0x85/0xb0 > [openvswitch] > [654405.527207] [] ? blk_mq_free_hctx_request+0x36/0x40 > [654405.527209] [] ? blk_mq_free_request+0x31/0x40 > [654405.527214] [] ? read_tsc+0x9/0x10 > [654405.527220] [] ? ktime_get+0x54/0xc0 > [654405.527225] [] ? put_device+0x17/0x20 > [654405.527227] [] ? tcf_act_police+0x150/0x210 > [act_police] > [654405.527232] [] ? tcf_action_exec+0x51/0xa0 > [654405.527235] [] ? basic_classify+0x75/0xe0 [cls_basic] > [654405.527237] [] ? tc_classify+0x55/0xc0 > [654405.527241] [] netdev_port_receive+0x9d/0x110 > [openvswitch] > [654405.527245] [] netdev_frame_hook+0x34/0x50 > [
Re: Kernel 4.1.6 Panic due to slab corruption
On Mon, 07 Sep 2015 11:41:17 +0300, Nikolay Borisov wrote: > Hello, > > On one of our servers I've observed the a kernel pannic > happening with the following backtrace: > > [654405.527070] BUG: unable to handle kernel paging request at > 00028001 > [654405.527076] IP: [] kmem_cache_alloc_node+0x99/0x1e0 > [654405.527085] PGD 14bef58067 PUD 2ab358067 PMD 0 Interesting! I can't offer much help but had a similar panic just the other day for no apparent reason while running a bunch of compiles. First time I've seen this with 4.1.6: Sep 5 20:42:02 ragnarok kernel: BUG: unable to handle kernel paging request at 8800e789b740 Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: PGD 1aa2067 PUD 61f7fd067 PMD 0 Sep 5 20:42:02 ragnarok kernel: Oops: [#1] SMP Sep 5 20:42:02 ragnarok kernel: Modules linked in: auth_rpcgss oid_registry nfsv4 nfs lockd grace fscache sunrpc autofs4 sch_fq_codel snd_hda_codec_realtek x86_pkg_temp_thermal coretemp snd_hda_codec_generic crc32_pclmul crc32c_intel aesni_intel radeon aes_x86_64 glue_helper snd_hda_codec_hdmi lrw gf128mul ablk_helper cryptd i2c_algo_bit snd_usb_audio uvcvideo snd_hda_intel drm_kms_helper snd_hda_controller snd_hwdep videobuf2_vmalloc snd_usbmidi_lib videobuf2_memops snd_hda_codec videobuf2_core snd_rawmidi i2c_i801 ttm snd_hda_core v4l2_common snd_seq_device videodev snd_pcm usbhid drm snd_timer r8169 snd i2c_core mii soundcore parport_pc parport Sep 5 20:42:02 ragnarok kernel: CPU: 0 PID: 32755 Comm: sh Not tainted 4.1.6 #1 Sep 5 20:42:02 ragnarok kernel: Hardware name: Gigabyte Technology Co., Ltd. P67-DS3-B3/P67-DS3-B3, BIOS F1 05/06/2011 Sep 5 20:42:02 ragnarok kernel: task: 880569712e20 ti: 8804e4d9 task.ti: 8804e4d9 Sep 5 20:42:02 ragnarok kernel: RIP: 0010:[] [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RSP: 0018:8804e4d93d88 EFLAGS: 00010282 Sep 5 20:42:02 ragnarok kernel: RAX: RBX: 8805e7eacce0 RCX: 0001f7e8 Sep 5 20:42:02 ragnarok kernel: RDX: 0001f7e7 RSI: 00d0 RDI: 00018c70 Sep 5 20:42:02 ragnarok kernel: RBP: 8804e4d93dc8 R08: 88061f418c70 R09: Sep 5 20:42:02 ragnarok kernel: R10: 81748318 R11: ea00139bb500 R12: 00d0 Sep 5 20:42:02 ragnarok kernel: R13: 880606890600 R14: 8100d039 R15: 8800e789b740 Sep 5 20:42:02 ragnarok kernel: FS: 7f9c1d2f2700() GS:88061f40() knlGS: Sep 5 20:42:02 ragnarok kernel: CS: 0010 DS: ES: CR0: 80050033 Sep 5 20:42:02 ragnarok kernel: CR2: 8800e789b740 CR3: 0005f68ce000 CR4: 000406f0 Sep 5 20:42:02 ragnarok kernel: Stack: Sep 5 20:42:02 ragnarok kernel: 88061f7e6c00 0002 8805e7eacce0 Sep 5 20:42:02 ragnarok kernel: 880569712e20 01200011 8805e7eacce0 880569712e20 Sep 5 20:42:02 ragnarok kernel: 8804e4d93de8 8100d039 7f9c1d2f29d0 Sep 5 20:42:02 ragnarok kernel: Call Trace: Sep 5 20:42:02 ragnarok kernel: [] arch_dup_task_struct+0x69/0x170 Sep 5 20:42:02 ragnarok kernel: [] copy_process.part.8+0x14f/0x1760 Sep 5 20:42:02 ragnarok kernel: [] ? handle_mm_fault+0xd0f/0x13a0 Sep 5 20:42:02 ragnarok kernel: [] ? get_empty_filp+0xd4/0x1c0 Sep 5 20:42:02 ragnarok kernel: [] ? recalc_sigpending+0x1f/0x60 Sep 5 20:42:02 ragnarok kernel: [] do_fork+0xd7/0x370 Sep 5 20:42:02 ragnarok kernel: [] ? sigprocmask+0x57/0x90 Sep 5 20:42:02 ragnarok kernel: [] SyS_clone+0x16/0x20 Sep 5 20:42:02 ragnarok kernel: [] system_call_fastpath+0x12/0x6a Sep 5 20:42:02 ragnarok kernel: Code: 65 4c 03 05 ee e3 ea 7e 49 83 78 10 00 4d 8b 38 0f 84 b0 00 00 00 4d 85 ff 0f 84 a7 00 00 00 49 63 45 20 48 8d 4a 01 49 8b 7d 00 <49> 8b 1c 07 4c 89 f8 65 48 0f c7 0f 0f 94 c0 84 c0 74 b9 49 63 Sep 5 20:42:02 ragnarok kernel: RIP [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: RSP Sep 5 20:42:02 ragnarok kernel: CR2: 8800e789b740 Sep 5 20:42:02 ragnarok kernel: ---[ end trace e4478715791f5752 ]--- Sep 5 20:42:02 ragnarok kernel: BUG: unable to handle kernel paging request at 8800e789b740 Sep 5 20:42:02 ragnarok kernel: IP: [] kmem_cache_alloc+0x6d/0x160 Sep 5 20:42:02 ragnarok kernel: PGD 1aa2067 PUD 61f7fd067 PMD 0 Sep 5 20:42:02 ragnarok kernel: Oops: [#2] SMP Sep 5 20:42:02 ragnarok kernel: Modules linked in: auth_rpcgss oid_registry nfsv4 nfs lockd grace fscache sunrpc autofs4 sch_fq_codel snd_hda_codec_realtek x86_pkg_temp_thermal coretemp snd_hda_codec_generic crc32_pclmul crc32c_intel aesni_intel radeon aes_x86_64 glue_helper snd_hda_codec_hdmi lrw gf128mul ablk_helper cryptd i2c_algo_bit snd_usb_audio uvcvideo snd_hda_intel drm_kms_helper snd_hda_controller snd_hwdep videobuf2_vmalloc snd_usbmidi_lib videobuf2_memops snd_hda_codec videob
Kernel 4.1.6 Panic due to slab corruption
Hello, On one of our servers I've observed the a kernel pannic happening with the following backtrace: [654405.527070] BUG: unable to handle kernel paging request at 00028001 [654405.527076] IP: [] kmem_cache_alloc_node+0x99/0x1e0 [654405.527085] PGD 14bef58067 PUD 2ab358067 PMD 0 [654405.527089] Oops: [#11] SMP [654405.527093] Modules linked in: xt_multiport tcp_diag inet_diag act_police cls_basic sch_ingress scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 xt_pkttype xt_state veth openvswitch xt_owner xt_conntrack iptable_filter iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio libcrc32c dm_mirror dm_region_hash dm_log iTCO_wdt iTCO_vendor_support sb_edac edac_core i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core ioatdma dca ipmi_devintf ipmi_si ipmi_msghandler mpt2sas scsi_transport_sas raid_class [654405.527145] CPU: 14 PID: 32267 Comm: httpd Tainted: G D L 4.1.6-clouder1 #1 [654405.527147] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0 07/09/2013 [654405.527149] task: 88139d3b1ec0 ti: 8808eda14000 task.ti: 8808eda14000 [654405.527151] RIP: 0010:[] [] kmem_cache_alloc_node+0x99/0x1e0 [654405.527155] RSP: 0018:88407fcc3a98 EFLAGS: 00210246 [654405.527156] RAX: RBX: RCX: 8814ce9acf80 [654405.527157] RDX: 837ad864 RSI: 00050200 RDI: 00018ce0 [654405.527158] RBP: 88407fcc3af8 R08: 88407fcd8ce0 R09: a033d990 [654405.527159] R10: 88058676fdd8 R11: 7b4a R12: 881fff807ac0 [654405.527161] R13: 00028001 R14: 0001 R15: 881fff807ac0 [654405.527162] FS: () GS:88407fcc(0063) knlGS:55c832e0 [654405.527164] CS: 0010 DS: 002b ES: 002b CR0: 80050033 [654405.527165] CR2: 00028001 CR3: 001467b64000 CR4: 000406e0 [654405.527166] Stack: [654405.527167] 881ff2d05000 [654405.527170] 88407fcc3ae8 00050200812b5903 88407fcc3ae8 01a2 [654405.527172] 0001 88058676fc60 88058676fe80 1800 [654405.527175] Call Trace: [654405.527177] [654405.527184] [] ovs_flow_stats_update+0x110/0x160 [openvswitch] [654405.527189] [] ovs_dp_process_packet+0x64/0xf0 [openvswitch] [654405.527193] [] ? netdev_port_receive+0x110/0x110 [openvswitch] [654405.527197] [] ? netdev_port_receive+0x110/0x110 [openvswitch] [654405.527201] [] ovs_vport_receive+0x85/0xb0 [openvswitch] [654405.527207] [] ? blk_mq_free_hctx_request+0x36/0x40 [654405.527209] [] ? blk_mq_free_request+0x31/0x40 [654405.527214] [] ? read_tsc+0x9/0x10 [654405.527220] [] ? ktime_get+0x54/0xc0 [654405.527225] [] ? put_device+0x17/0x20 [654405.527227] [] ? tcf_act_police+0x150/0x210 [act_police] [654405.527232] [] ? tcf_action_exec+0x51/0xa0 [654405.527235] [] ? basic_classify+0x75/0xe0 [cls_basic] [654405.527237] [] ? tc_classify+0x55/0xc0 [654405.527241] [] netdev_port_receive+0x9d/0x110 [openvswitch] [654405.527245] [] netdev_frame_hook+0x34/0x50 [openvswitch] [654405.527250] [] __netif_receive_skb_core+0x206/0x880 [654405.527252] [] __netif_receive_skb+0x27/0x70 [654405.527254] [] process_backlog+0xf1/0x1b0 [654405.527257] [] napi_poll+0xd3/0x1c0 [654405.527259] [] net_rx_action+0x90/0x1c0 [654405.527264] [] __do_softirq+0xfb/0x2a0 [654405.527270] [] do_softirq_own_stack+0x1c/0x30 [654405.527271] [654405.527273] [] do_softirq+0x55/0x60 [654405.527276] [] __local_bh_enable_ip+0x88/0x90 [654405.527279] [] ip_finish_output+0x282/0x490 [654405.527281] [] ip_output+0xab/0xc0 [654405.527283] [] ? ip_finish_output_gso+0x4e0/0x4e0 [654405.527285] [] ip_local_out_sk+0x3b/0x50 [654405.527287] [] ip_queue_xmit+0x14e/0x3c0 [654405.527291] [] tcp_transmit_skb+0x4c2/0x850 [654405.527294] [] tcp_write_xmit+0x19d/0x670 [654405.527298] [] ? copy_user_generic_string+0x31/0x40 [654405.527300] [] __tcp_push_pending_frames+0x32/0xd0 [654405.527302] [] tcp_push+0xf1/0x120 [654405.527304] [] tcp_sendmsg+0x373/0xb60 [654405.527307] [] ? mntput+0x23/0x40 [654405.527310] [] ? path_put+0x22/0x30 [654405.527315] [] inet_sendmsg+0x42/0xb0 [654405.527317] [] ? kmem_cache_alloc+0xee/0x1c0 [654405.527321] [] sock_sendmsg+0x4d/0x60 [654405.527324] [] sock_write_iter+0xb6/0x100 [654405.527328] [] do_iter_readv_writev+0x60/0x90 [654405.527330] [] ? kernel_sendmsg+0x40/0x40 [654405.527332] [] compat_do_readv_writev+0x174/0x1f0 [654405.527337] [] ? rcu_eqs_exit+0x79/0xb0 [654405.527339] [] ? rcu_user_exit+0x13/0x20 [654405.527342] [] compat_SyS_writev+0xc1/0x110 [654405.527346] [] ? context_tracking_user_enter+0x13/0x20 [654405.527349] [] syse
