Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
Hi, Bruce, Dan, This patch took care the panic issue. thanks, -jane On 8/21/19 7:12 AM, J. Bruce Fields wrote: Probably just needs the following. I've been slow to get some bugfixes upstream, sorry--I'll go send a pull request now --b. commit 78e70e780b28 Author: He Zhe Date: Tue Aug 6 17:41:04 2019 +0800 nfsd4: Fix kernel crash when reading proc file reply_cache_stats reply_cache_stats uses wrong parameter as seq file private structure and thus causes the following kernel crash when users read /proc/fs/nfsd/reply_cache_stats BUG: kernel NULL pointer dereference, address: 01f9 PGD 0 P4D 0 Oops: [#3] SMP PTI CPU: 6 PID: 1502 Comm: cat Tainted: G D 5.3.0-rc3+ #1 Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015 RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0 Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00 RSP: 0018:aa520106fe08 EFLAGS: 00010246 RAX: 00cfe1a77123 RBX: RCX: 00291b46 RDX: 00cf RSI: 0006 RDI: 00291b28 RBP: aa520106fe20 R08: 0006 R09: 00cfe17e55dd R10: a424e47c R11: 030b R12: 0001 R13: a424e5697000 R14: 0001 R15: a424e5697000 FS: 7f805735f580() GS:a424f8f8() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 01f9 CR3: 655ce005 CR4: 003606e0 Call Trace: seq_read+0x194/0x3e0 __vfs_read+0x1b/0x40 vfs_read+0x95/0x140 ksys_read+0x61/0xe0 __x64_sys_read+0x1a/0x20 do_syscall_64+0x4d/0x120 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f805728b861 Code: fe ff ff 50 48 8d 3d 86 b4 09 00 e8 79 e0 01 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 d9 19 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54 RSP: 002b:7ffea1ce3c38 EFLAGS: 0246 ORIG_RAX: RAX: ffda RBX: 0002 RCX: 7f805728b861 RDX: 0002 RSI: 7f8057183000 RDI: 0003 RBP: 7f8057183000 R08: 7f8057182010 R09: R10: 0022 R11: 0246 R12: 559a60e8ff10 R13: 0003 R14: 0002 R15: 0002 Modules linked in: CR2: 01f9 ---[ end trace 01613595153f0cba ]--- RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0 Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00 RSP: 0018:aa52004b3e08 EFLAGS: 00010246 RAX: 002bab45a7c6 RBX: RCX: 00291b4c RDX: 002b RSI: 0004 RDI: 00291b28 RBP: aa52004b3e20 R08: 0004 R09: 002bab1c8c7a R10: a424e550 R11: 02a9 R12: 0001 R13: a424e4475000 R14: 0001 R15: a424e4475000 FS: 7f805735f580() GS:a424f8f8() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 01f9 CR3: 655ce005 CR4: 003606e0 Killed Fixes: 3ba75830ce17 ("nfsd4: drc containerization") Signed-off-by: He Zhe Signed-off-by: J. Bruce Fields diff --git a/fs/nfsd/nfscache.c b/fs/nfsd/nfscache.c index 26ad75ae2be0..96352ab7bd81 100644 --- a/fs/nfsd/nfscache.c +++ b/fs/nfsd/nfscache.c @@ -571,7 +571,7 @@ nfsd_cache_append(struct svc_rqst *rqstp, struct kvec *data) */ static int nfsd_reply_cache_stats_show(struct seq_file *m, void *v) { - struct nfsd_net *nn = v; + struct nfsd_net *nn = m->private; seq_printf(m, "max entries: %u\n", nn->max_drc_entries); seq_printf(m, "num entries: %u\n",
Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
Hi, Dan, On 8/20/19 8:48 PM, Dan Williams wrote: On Tue, Aug 20, 2019 at 6:39 PM wrote: Hi, Apology if there is a better channel reporting the issue, if so, please let me know. I just saw below regression in 5.3-rc5 kernel, but not in 5.2-rc7 or earlier kernels. Is the error stable enough to bisect? The error is stable, I haven't tried bisect, thought to report the issue first since it's late in the 5.3 process. Then saw Bruce' email, I'll report soon whether the patch takes care the issue. thanks! -jane
Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
Probably just needs the following. I've been slow to get some bugfixes upstream, sorry--I'll go send a pull request now --b. commit 78e70e780b28 Author: He Zhe Date: Tue Aug 6 17:41:04 2019 +0800 nfsd4: Fix kernel crash when reading proc file reply_cache_stats reply_cache_stats uses wrong parameter as seq file private structure and thus causes the following kernel crash when users read /proc/fs/nfsd/reply_cache_stats BUG: kernel NULL pointer dereference, address: 01f9 PGD 0 P4D 0 Oops: [#3] SMP PTI CPU: 6 PID: 1502 Comm: cat Tainted: G D 5.3.0-rc3+ #1 Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015 RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0 Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00 RSP: 0018:aa520106fe08 EFLAGS: 00010246 RAX: 00cfe1a77123 RBX: RCX: 00291b46 RDX: 00cf RSI: 0006 RDI: 00291b28 RBP: aa520106fe20 R08: 0006 R09: 00cfe17e55dd R10: a424e47c R11: 030b R12: 0001 R13: a424e5697000 R14: 0001 R15: a424e5697000 FS: 7f805735f580() GS:a424f8f8() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 01f9 CR3: 655ce005 CR4: 003606e0 Call Trace: seq_read+0x194/0x3e0 __vfs_read+0x1b/0x40 vfs_read+0x95/0x140 ksys_read+0x61/0xe0 __x64_sys_read+0x1a/0x20 do_syscall_64+0x4d/0x120 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f805728b861 Code: fe ff ff 50 48 8d 3d 86 b4 09 00 e8 79 e0 01 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 d9 19 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54 RSP: 002b:7ffea1ce3c38 EFLAGS: 0246 ORIG_RAX: RAX: ffda RBX: 0002 RCX: 7f805728b861 RDX: 0002 RSI: 7f8057183000 RDI: 0003 RBP: 7f8057183000 R08: 7f8057182010 R09: R10: 0022 R11: 0246 R12: 559a60e8ff10 R13: 0003 R14: 0002 R15: 0002 Modules linked in: CR2: 01f9 ---[ end trace 01613595153f0cba ]--- RIP: 0010:nfsd_reply_cache_stats_show+0x3b/0x2d0 Code: 41 54 49 89 f4 48 89 fe 48 c7 c7 b3 10 33 88 53 bb e8 03 00 00 e8 88 82 d1 ff bf 58 89 41 00 e8 eb c5 85 00 48 83 eb 01 75 f0 <41> 8b 94 24 f8 01 00 00 48 c7 c6 be 10 33 88 4c 89 ef bb e8 03 00 RSP: 0018:aa52004b3e08 EFLAGS: 00010246 RAX: 002bab45a7c6 RBX: RCX: 00291b4c RDX: 002b RSI: 0004 RDI: 00291b28 RBP: aa52004b3e20 R08: 0004 R09: 002bab1c8c7a R10: a424e550 R11: 02a9 R12: 0001 R13: a424e4475000 R14: 0001 R15: a424e4475000 FS: 7f805735f580() GS:a424f8f8() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 01f9 CR3: 655ce005 CR4: 003606e0 Killed Fixes: 3ba75830ce17 ("nfsd4: drc containerization") Signed-off-by: He Zhe Signed-off-by: J. Bruce Fields diff --git a/fs/nfsd/nfscache.c b/fs/nfsd/nfscache.c index 26ad75ae2be0..96352ab7bd81 100644 --- a/fs/nfsd/nfscache.c +++ b/fs/nfsd/nfscache.c @@ -571,7 +571,7 @@ nfsd_cache_append(struct svc_rqst *rqstp, struct kvec *data) */ static int nfsd_reply_cache_stats_show(struct seq_file *m, void *v) { - struct nfsd_net *nn = v; + struct nfsd_net *nn = m->private; seq_printf(m, "max entries: %u\n", nn->max_drc_entries); seq_printf(m, "num entries: %u\n",
Re: kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
On Tue, Aug 20, 2019 at 6:39 PM wrote: > > Hi, > > Apology if there is a better channel reporting the issue, if so, please > let me know. > > I just saw below regression in 5.3-rc5 kernel, but not in 5.2-rc7 or > earlier kernels. Is the error stable enough to bisect?
kernel panic in 5.3-rc5, nfsd_reply_cache_stats_show+0x11
Hi, Apology if there is a better channel reporting the issue, if so, please let me know. I just saw below regression in 5.3-rc5 kernel, but not in 5.2-rc7 or earlier kernels. [ 3533.659787] mce: Uncorrected hardware memory error in user-access at 383e202000 [ 3533.659903] Memory failure: 0x383e202: Sending SIGBUS to read_poison:14493 due to hardware memory corruption [ 3533.679041] Memory failure: 0x383e202: recovery action for dax page: Recovered [ 3564.624934] BUG: kernel NULL pointer dereference, address: 01f9 [ 3564.632707] #PF: supervisor read access in kernel mode [ 3564.638440] #PF: error_code(0x) - not-present page [ 3564.644174] PGD acd7b47067 P4D acd7b47067 PUD acd7aba067 PMD 0 [ 3564.650784] Oops: [#1] SMP NOPTI [ 3564.654869] CPU: 58 PID: 15026 Comm: sosreport Tainted: G M 5.3.0-rc5.master.20190820.ol7.x86_64 #1 [ 3564.666420] Hardware name: Oracle Corporation ORACLE SERVER X8-2L/ASM,MTHRBD,2U, BIOS 52020101 05/07/2019 [ 3564.677112] RIP: 0010:nfsd_reply_cache_stats_show+0x11/0x110 [nfsd] [ 3564.684106] Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 63 47 ec 48 89 e5 5d c3 90 0f 1f 44 00 00 55 31 c0 48 89 e5 41 54 49 89 f4 53 <8b> 96 f8 01 00 00 48 c7 c6 d9 b8 74 c0 48 89 fb e8 9a ae bc f2 41 [ 3564.705062] RSP: 0018:aa140f87fe18 EFLAGS: 00010246 [ 3564.710894] RAX: RBX: 9f7c9b562ca8 RCX: 5c19 [ 3564.718858] RDX: 1000 RSI: 0001 RDI: 9f7c9b562c80 [ 3564.726822] RBP: aa140f87fe28 R08: 9f801fab01a0 R09: 9ed347c06600 [ 3564.734785] R10: 9f801e287000 R11: 9f8012f8d638 R12: 0001 [ 3564.742749] R13: 9f8012f8d600 R14: 9f7c9b562c80 R15: 0001 [ 3564.750712] FS: 7f3cfaa92700() GS:9f801fa8() knlGS: [ 3564.759743] CS: 0010 DS: ES: CR0: 80050033 [ 3564.766156] CR2: 01f9 CR3: 00add1894004 CR4: 007606e0 [ 3564.774120] DR0: DR1: DR2: [ 3564.782084] DR3: DR6: fffe0ff0 DR7: 0400 [ 3564.790050] PKRU: 5554 [ 3564.793068] Call Trace: [ 3564.795800] seq_read+0x13b/0x390 [ 3564.799502] __vfs_read+0x1b/0x40 [ 3564.803202] vfs_read+0x8e/0x140 [ 3564.806794] ksys_read+0x61/0xd0 [ 3564.810394] __x64_sys_read+0x1a/0x20 [ 3564.814484] do_syscall_64+0x60/0x1e0 [ 3564.818572] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 3564.824199] RIP: 0033:0x7f3d163a304d The panic is reproducible every time in 5.3-rc5, here is the steps. On system with Intel DC PMEM, configure at AppDirect NonInterleave mode, # ndctl create-namespace -m devdax { "dev":"namespace1.0", "mode":"devdax", "map":"dev", "size":"124.03 GiB (133.18 GB)", .. "align":2097152, "devices":[ { "chardev":"dax1.0", # ndctl inject-error namespace1.0 -B 16 --count=1 # ./read_poison -x dax1.0 -o 8192 -m 1 Read poison location at (16 * 512 = 8192) About a little under 30sec later, kernel panics. thanks, -jane