Re: [PATCH] tracing: fix UAF caused by memory ordering issue
Mark Rutland 于2023年11月14日周二 06:17写道: > Hi, Mark and Steven Thank you so much for the detailed comments. > On Sun, Nov 12, 2023 at 11:00:30PM +0800, Kairui Song wrote: > > From: Kairui Song > > > > Following kernel panic was observed when doing ftrace stress test: > > Can you share some more details: > > * What test specifically are you running? Can you share this so that others > can > try to reproduce the issue? Yes, the panic happened when doing LTP ftrace stress test: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/tracing/ftrace_test/ftrace_stress_test.sh > > * Which machines are you testing on (i.e. which CPU microarchitecture is this > seen with) ? The panic was seen on a ARM64 VM, lscpu output: Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: HiSilicon BIOS Vendor ID: QEMU Model name: Kunpeng-920 BIOS Model name:virt-rhel8.6.0 CPU @ 2.0GHz BIOS CPU family:1 Model: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 4 Stepping: 0x1 BogoMIPS: 200.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm The host machine is a Kunpeng-920 with 4 NUMA nodes and 128 cores. > > * Which compiler are you using? gcc 12.3.1 > > * The log shows this is with v6.1.61+. Can you reproduce this with a mainline > kernel? e.g. v6.6 or v6.7-rc1? It's reproducible with LTS, not tested with mainline, I'll try to reproduce this with the latest mainline. But due to the low reproducibility this may take a while. > > > Unable to handle kernel paging request at virtual address 9699b0f8ece28240 > > Mem abort info: > > ESR = 0x9604 > > EC = 0x25: DABT (current EL), IL = 32 bits > > SET = 0, FnV = 0 > > EA = 0, S1PTW = 0 > > FSC = 0x04: level 0 translation fault > > Data abort info: > > ISV = 0, ISS = 0x0004 > > CM = 0, WnR = 0 > > [9699b0f8ece28240] address between user and kernel address ranges > > Internal error: Oops: 9604 [#1] SMP > > Modules linked in: rpcrdma rdma_cm iw_cm ib_cm ib_core rfkill vfat fat loop > > fuse nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache > > jbd2 sr_mod cdrom crct10dif_ce ghash_ce sha2_ce virtio_gpu virtio_dma_buf > > drm_shmem_helper virtio_blk drm_kms_helper syscopyarea sysfillrect > > sysimgblt fb_sys_fops virtio_console sha256_arm64 sha1_ce drm virtio_scsi > > i2c_core virtio_net net_failover failover virtio_mmio dm_multipath dm_mod > > autofs4 [last unloaded: ipmi_msghandler] > > CPU: 0 PID: 499719 Comm: sh Kdump: loaded Not tainted 6.1.61+ #2 > > Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 > > pstate: 6045 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > > pc : __kmem_cache_alloc_node+0x1dc/0x2e4 > > lr : __kmem_cache_alloc_node+0xac/0x2e4 > > sp : 8ad23aa0 > > x29: 8ad23ab0 x28: 0004052b8000 x27: c513863b > > x26: 0040 x25: c51384f21ca4 x24: > > x23: d615521430b1b1a5 x22: c51386044770 x21: > > x20: 0cc0 x19: c0001200 x18: > > x17: x16: x15: e65e1630 > > x14: 0004 x13: c513863e67a0 x12: c513863af6d8 > > x11: 0001 x10: 8ad23aa0 x9 : c51385058078 > > x8 : 0018 x7 : 0001 x6 : 0010 > > x5 : c09c2280 x4 : c51384f21ca4 x3 : 0040 > > x2 : 9699b0f8ece28240 x1 : c09c2280 x0 : 9699b0f8ece28200 > > Call trace: > > __kmem_cache_alloc_node+0x1dc/0x2e4 > > __kmalloc+0x6c/0x1c0 > > func_add+0x1a4/0x200 > > tracepoint_add_func+0x70/0x230 > > tracepoint_probe_register+0x6c/0xb4 > > trace_event_reg+0x8c/0xa0 > > __ftrace_event_enable_disable+0x17c/0x440 > > __ftrace_set_clr_event_nolock+0xe0/0x150 > > system_enable_write+0xe0/0x114 > > vfs_write+0xd0/0x2dc > > ksys_write+0x78/0x110 > > __arm64_sys_write+0x24/0x30 > > invoke_syscall.constprop.0+0x58/0xf0 > > el0_svc_common.constprop.0+0x54/0x160 > > do_el0_svc+0x2c/0x60 > > el0_svc+0x40/0x1ac > > el0t_64_sync_handler+0xf4/0x120 > > el0t_64_sync+0x19c/0x1a0 > > Code: b9402a63 f9405e77 8b030002 d5384101 (f8636803) > > > > Panic was caused by corrupted freelist pointer. After more debugging, > > I found the root cause is UAF of slab allocated object in ftrace > > introduced by commit eecb91b9f98d ("tracing: Fix memleak due to race > > between current_tracer and trace"), and so far it's only reproducible > > on some ARM64 machines, the UAF and free stack is: > > > > UAF: > > kasan_report+0xa8/0x1bc > > __asan_report_load8_noabort+0x28/0x3c > > print_graph_function_flags+0x524/0x5a0 > > print_graph_function_event+0x28/0x40 > >
Re: [PATCH] tracing: fix UAF caused by memory ordering issue
On Sun, Nov 12, 2023 at 11:00:30PM +0800, Kairui Song wrote: > From: Kairui Song > > Following kernel panic was observed when doing ftrace stress test: Can you share some more details: * What test specifically are you running? Can you share this so that others can try to reproduce the issue? * Which machines are you testing on (i.e. which CPU microarchitecture is this seen with) ? * Which compiler are you using? * The log shows this is with v6.1.61+. Can you reproduce this with a mainline kernel? e.g. v6.6 or v6.7-rc1? > Unable to handle kernel paging request at virtual address 9699b0f8ece28240 > Mem abort info: > ESR = 0x9604 > EC = 0x25: DABT (current EL), IL = 32 bits > SET = 0, FnV = 0 > EA = 0, S1PTW = 0 > FSC = 0x04: level 0 translation fault > Data abort info: > ISV = 0, ISS = 0x0004 > CM = 0, WnR = 0 > [9699b0f8ece28240] address between user and kernel address ranges > Internal error: Oops: 9604 [#1] SMP > Modules linked in: rpcrdma rdma_cm iw_cm ib_cm ib_core rfkill vfat fat loop > fuse nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 > sr_mod cdrom crct10dif_ce ghash_ce sha2_ce virtio_gpu virtio_dma_buf > drm_shmem_helper virtio_blk drm_kms_helper syscopyarea sysfillrect sysimgblt > fb_sys_fops virtio_console sha256_arm64 sha1_ce drm virtio_scsi i2c_core > virtio_net net_failover failover virtio_mmio dm_multipath dm_mod autofs4 > [last unloaded: ipmi_msghandler] > CPU: 0 PID: 499719 Comm: sh Kdump: loaded Not tainted 6.1.61+ #2 > Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 > pstate: 6045 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : __kmem_cache_alloc_node+0x1dc/0x2e4 > lr : __kmem_cache_alloc_node+0xac/0x2e4 > sp : 8ad23aa0 > x29: 8ad23ab0 x28: 0004052b8000 x27: c513863b > x26: 0040 x25: c51384f21ca4 x24: > x23: d615521430b1b1a5 x22: c51386044770 x21: > x20: 0cc0 x19: c0001200 x18: > x17: x16: x15: e65e1630 > x14: 0004 x13: c513863e67a0 x12: c513863af6d8 > x11: 0001 x10: 8ad23aa0 x9 : c51385058078 > x8 : 0018 x7 : 0001 x6 : 0010 > x5 : c09c2280 x4 : c51384f21ca4 x3 : 0040 > x2 : 9699b0f8ece28240 x1 : c09c2280 x0 : 9699b0f8ece28200 > Call trace: > __kmem_cache_alloc_node+0x1dc/0x2e4 > __kmalloc+0x6c/0x1c0 > func_add+0x1a4/0x200 > tracepoint_add_func+0x70/0x230 > tracepoint_probe_register+0x6c/0xb4 > trace_event_reg+0x8c/0xa0 > __ftrace_event_enable_disable+0x17c/0x440 > __ftrace_set_clr_event_nolock+0xe0/0x150 > system_enable_write+0xe0/0x114 > vfs_write+0xd0/0x2dc > ksys_write+0x78/0x110 > __arm64_sys_write+0x24/0x30 > invoke_syscall.constprop.0+0x58/0xf0 > el0_svc_common.constprop.0+0x54/0x160 > do_el0_svc+0x2c/0x60 > el0_svc+0x40/0x1ac > el0t_64_sync_handler+0xf4/0x120 > el0t_64_sync+0x19c/0x1a0 > Code: b9402a63 f9405e77 8b030002 d5384101 (f8636803) > > Panic was caused by corrupted freelist pointer. After more debugging, > I found the root cause is UAF of slab allocated object in ftrace > introduced by commit eecb91b9f98d ("tracing: Fix memleak due to race > between current_tracer and trace"), and so far it's only reproducible > on some ARM64 machines, the UAF and free stack is: > > UAF: > kasan_report+0xa8/0x1bc > __asan_report_load8_noabort+0x28/0x3c > print_graph_function_flags+0x524/0x5a0 > print_graph_function_event+0x28/0x40 > print_trace_line+0x5c4/0x1030 > s_show+0xf0/0x460 > seq_read_iter+0x930/0xf5c > seq_read+0x130/0x1d0 > vfs_read+0x288/0x840 > ksys_read+0x130/0x270 > __arm64_sys_read+0x78/0xac > invoke_syscall.constprop.0+0x90/0x224 > do_el0_svc+0x118/0x3dc > el0_svc+0x54/0x120 > el0t_64_sync_handler+0xf4/0x120 > el0t_64_sync+0x19c/0x1a0 > > Freed by: > kasan_save_free_info+0x38/0x5c > __kasan_slab_free+0xe8/0x154 > slab_free_freelist_hook+0xfc/0x1e0 > __kmem_cache_free+0x138/0x260 > kfree+0xd0/0x1d0 > graph_trace_close+0x60/0x90 > s_start+0x610/0x910 > seq_read_iter+0x274/0xf5c > seq_read+0x130/0x1d0 > vfs_read+0x288/0x840 > ksys_read+0x130/0x270 > __arm64_sys_read+0x78/0xac > invoke_syscall.constprop.0+0x90/0x224 > do_el0_svc+0x118/0x3dc > el0_svc+0x54/0x120 > el0t_64_sync_handler+0xf4/0x120 > el0t_64_sync+0x19c/0x1a0 > > Despite the s_start and s_show being serialized by seq_file mutex, > the tracer struct copy in s_start introduced by the commit mentioned > above is not atomic nor guarenteened to be seen by all CPUs. So > following seneriao is possible (and actually happened): > > CPU 1 CPU 2 > seq_read_iter seq_read_iter > mutex_lock(>lock); > s_start > // iter->trace is graph_trace > iter->trace->close(iter); > graph_trace_close > kfree(data) <- *** data released here *** > //
Re: [PATCH] tracing: fix UAF caused by memory ordering issue
On Sun, 12 Nov 2023 23:00:30 +0800 Kairui Song wrote: > From: Kairui Song > > Following kernel panic was observed when doing ftrace stress test: > > Unable to handle kernel paging request at virtual address 9699b0f8ece28240 > Mem abort info: > ESR = 0x9604 > EC = 0x25: DABT (current EL), IL = 32 bits > SET = 0, FnV = 0 > EA = 0, S1PTW = 0 > FSC = 0x04: level 0 translation fault > Data abort info: > ISV = 0, ISS = 0x0004 > CM = 0, WnR = 0 > [9699b0f8ece28240] address between user and kernel address ranges > Internal error: Oops: 9604 [#1] SMP > Modules linked in: rpcrdma rdma_cm iw_cm ib_cm ib_core rfkill vfat fat loop > fuse nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 > sr_mod cdrom crct10dif_ce ghash_ce sha2_ce virtio_gpu virtio_dma_buf > drm_shmem_helper virtio_blk drm_kms_helper syscopyarea sysfillrect sysimgblt > fb_sys_fops virtio_console sha256_arm64 sha1_ce drm virtio_scsi i2c_core > virtio_net net_failover failover virtio_mmio dm_multipath dm_mod autofs4 > [last unloaded: ipmi_msghandler] > CPU: 0 PID: 499719 Comm: sh Kdump: loaded Not tainted 6.1.61+ #2 > Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 > pstate: 6045 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : __kmem_cache_alloc_node+0x1dc/0x2e4 > lr : __kmem_cache_alloc_node+0xac/0x2e4 > sp : 8ad23aa0 > x29: 8ad23ab0 x28: 0004052b8000 x27: c513863b > x26: 0040 x25: c51384f21ca4 x24: > x23: d615521430b1b1a5 x22: c51386044770 x21: > x20: 0cc0 x19: c0001200 x18: > x17: x16: x15: e65e1630 > x14: 0004 x13: c513863e67a0 x12: c513863af6d8 > x11: 0001 x10: 8ad23aa0 x9 : c51385058078 > x8 : 0018 x7 : 0001 x6 : 0010 > x5 : c09c2280 x4 : c51384f21ca4 x3 : 0040 > x2 : 9699b0f8ece28240 x1 : c09c2280 x0 : 9699b0f8ece28200 > Call trace: > __kmem_cache_alloc_node+0x1dc/0x2e4 > __kmalloc+0x6c/0x1c0 > func_add+0x1a4/0x200 > tracepoint_add_func+0x70/0x230 > tracepoint_probe_register+0x6c/0xb4 > trace_event_reg+0x8c/0xa0 > __ftrace_event_enable_disable+0x17c/0x440 > __ftrace_set_clr_event_nolock+0xe0/0x150 > system_enable_write+0xe0/0x114 > vfs_write+0xd0/0x2dc > ksys_write+0x78/0x110 > __arm64_sys_write+0x24/0x30 > invoke_syscall.constprop.0+0x58/0xf0 > el0_svc_common.constprop.0+0x54/0x160 > do_el0_svc+0x2c/0x60 > el0_svc+0x40/0x1ac > el0t_64_sync_handler+0xf4/0x120 > el0t_64_sync+0x19c/0x1a0 > Code: b9402a63 f9405e77 8b030002 d5384101 (f8636803) > > Panic was caused by corrupted freelist pointer. After more debugging, So you are saying that the above is caused by the free list corruption (hence we can ignore the stack trace) > I found the root cause is UAF of slab allocated object in ftrace > introduced by commit eecb91b9f98d ("tracing: Fix memleak due to race > between current_tracer and trace"), and so far it's only reproducible > on some ARM64 machines, the UAF and free stack is: > > UAF: > kasan_report+0xa8/0x1bc > __asan_report_load8_noabort+0x28/0x3c > print_graph_function_flags+0x524/0x5a0 > print_graph_function_event+0x28/0x40 > print_trace_line+0x5c4/0x1030 > s_show+0xf0/0x460 > seq_read_iter+0x930/0xf5c > seq_read+0x130/0x1d0 > vfs_read+0x288/0x840 > ksys_read+0x130/0x270 > __arm64_sys_read+0x78/0xac > invoke_syscall.constprop.0+0x90/0x224 > do_el0_svc+0x118/0x3dc > el0_svc+0x54/0x120 > el0t_64_sync_handler+0xf4/0x120 > el0t_64_sync+0x19c/0x1a0 > > Freed by: > kasan_save_free_info+0x38/0x5c > __kasan_slab_free+0xe8/0x154 > slab_free_freelist_hook+0xfc/0x1e0 > __kmem_cache_free+0x138/0x260 > kfree+0xd0/0x1d0 > graph_trace_close+0x60/0x90 > s_start+0x610/0x910 > seq_read_iter+0x274/0xf5c > seq_read+0x130/0x1d0 > vfs_read+0x288/0x840 > ksys_read+0x130/0x270 > __arm64_sys_read+0x78/0xac > invoke_syscall.constprop.0+0x90/0x224 > do_el0_svc+0x118/0x3dc > el0_svc+0x54/0x120 > el0t_64_sync_handler+0xf4/0x120 > el0t_64_sync+0x19c/0x1a0 > > Despite the s_start and s_show being serialized by seq_file mutex, > the tracer struct copy in s_start introduced by the commit mentioned > above is not atomic nor guarenteened to be seen by all CPUs. So > following seneriao is possible (and actually happened): What do you mean it is not atomic? The change should be done under a trace_types_lock, right? Doing something under a mutex is suppose to protect it. > > CPU 1 CPU 2 > seq_read_iter seq_read_iter > mutex_lock(>lock); > s_start > // iter->trace is graph_trace > iter->trace->close(iter); > graph_trace_close > kfree(data) <- *** data released here *** > // copy current_trace to iter->trace > // but not synced to CPU 2 > *iter->trace = *tr->current_trace > ... (goes on) >
[PATCH] tracing: fix UAF caused by memory ordering issue
From: Kairui Song Following kernel panic was observed when doing ftrace stress test: Unable to handle kernel paging request at virtual address 9699b0f8ece28240 Mem abort info: ESR = 0x9604 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x04: level 0 translation fault Data abort info: ISV = 0, ISS = 0x0004 CM = 0, WnR = 0 [9699b0f8ece28240] address between user and kernel address ranges Internal error: Oops: 9604 [#1] SMP Modules linked in: rpcrdma rdma_cm iw_cm ib_cm ib_core rfkill vfat fat loop fuse nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom crct10dif_ce ghash_ce sha2_ce virtio_gpu virtio_dma_buf drm_shmem_helper virtio_blk drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops virtio_console sha256_arm64 sha1_ce drm virtio_scsi i2c_core virtio_net net_failover failover virtio_mmio dm_multipath dm_mod autofs4 [last unloaded: ipmi_msghandler] CPU: 0 PID: 499719 Comm: sh Kdump: loaded Not tainted 6.1.61+ #2 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 pstate: 6045 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : __kmem_cache_alloc_node+0x1dc/0x2e4 lr : __kmem_cache_alloc_node+0xac/0x2e4 sp : 8ad23aa0 x29: 8ad23ab0 x28: 0004052b8000 x27: c513863b x26: 0040 x25: c51384f21ca4 x24: x23: d615521430b1b1a5 x22: c51386044770 x21: x20: 0cc0 x19: c0001200 x18: x17: x16: x15: e65e1630 x14: 0004 x13: c513863e67a0 x12: c513863af6d8 x11: 0001 x10: 8ad23aa0 x9 : c51385058078 x8 : 0018 x7 : 0001 x6 : 0010 x5 : c09c2280 x4 : c51384f21ca4 x3 : 0040 x2 : 9699b0f8ece28240 x1 : c09c2280 x0 : 9699b0f8ece28200 Call trace: __kmem_cache_alloc_node+0x1dc/0x2e4 __kmalloc+0x6c/0x1c0 func_add+0x1a4/0x200 tracepoint_add_func+0x70/0x230 tracepoint_probe_register+0x6c/0xb4 trace_event_reg+0x8c/0xa0 __ftrace_event_enable_disable+0x17c/0x440 __ftrace_set_clr_event_nolock+0xe0/0x150 system_enable_write+0xe0/0x114 vfs_write+0xd0/0x2dc ksys_write+0x78/0x110 __arm64_sys_write+0x24/0x30 invoke_syscall.constprop.0+0x58/0xf0 el0_svc_common.constprop.0+0x54/0x160 do_el0_svc+0x2c/0x60 el0_svc+0x40/0x1ac el0t_64_sync_handler+0xf4/0x120 el0t_64_sync+0x19c/0x1a0 Code: b9402a63 f9405e77 8b030002 d5384101 (f8636803) Panic was caused by corrupted freelist pointer. After more debugging, I found the root cause is UAF of slab allocated object in ftrace introduced by commit eecb91b9f98d ("tracing: Fix memleak due to race between current_tracer and trace"), and so far it's only reproducible on some ARM64 machines, the UAF and free stack is: UAF: kasan_report+0xa8/0x1bc __asan_report_load8_noabort+0x28/0x3c print_graph_function_flags+0x524/0x5a0 print_graph_function_event+0x28/0x40 print_trace_line+0x5c4/0x1030 s_show+0xf0/0x460 seq_read_iter+0x930/0xf5c seq_read+0x130/0x1d0 vfs_read+0x288/0x840 ksys_read+0x130/0x270 __arm64_sys_read+0x78/0xac invoke_syscall.constprop.0+0x90/0x224 do_el0_svc+0x118/0x3dc el0_svc+0x54/0x120 el0t_64_sync_handler+0xf4/0x120 el0t_64_sync+0x19c/0x1a0 Freed by: kasan_save_free_info+0x38/0x5c __kasan_slab_free+0xe8/0x154 slab_free_freelist_hook+0xfc/0x1e0 __kmem_cache_free+0x138/0x260 kfree+0xd0/0x1d0 graph_trace_close+0x60/0x90 s_start+0x610/0x910 seq_read_iter+0x274/0xf5c seq_read+0x130/0x1d0 vfs_read+0x288/0x840 ksys_read+0x130/0x270 __arm64_sys_read+0x78/0xac invoke_syscall.constprop.0+0x90/0x224 do_el0_svc+0x118/0x3dc el0_svc+0x54/0x120 el0t_64_sync_handler+0xf4/0x120 el0t_64_sync+0x19c/0x1a0 Despite the s_start and s_show being serialized by seq_file mutex, the tracer struct copy in s_start introduced by the commit mentioned above is not atomic nor guarenteened to be seen by all CPUs. So following seneriao is possible (and actually happened): CPU 1 CPU 2 seq_read_iter seq_read_iter mutex_lock(>lock); s_start // iter->trace is graph_trace iter->trace->close(iter); graph_trace_close kfree(data) <- *** data released here *** // copy current_trace to iter->trace // but not synced to CPU 2 *iter->trace = *tr->current_trace ... (goes on) mutex_unlock(>lock); mutex_lock(>lock); ... (s_start and other work) s_show print_trace_line(iter) // iter->trace is still // old value (graph_trace) iter->trace->print_line() print_graph_function_flags