在 2015年09月22日 16:40, Qu Wenruo 写道:
在 2015年09月22日 15:34, Stéphane Lesimple 写道:
Le 2015-09-22 03:37, Qu Wenruo a écrit :
Stéphane Lesimple wrote on 2015/09/22 03:30 +0200:
Le 2015-09-20 13:14, Stéphane Lesimple a écrit :
Le 2015-09-20 12:51, Qu Wenruo a écrit :
Would you please use gdb to show the codes of
"btrfs_qgroup_rescan_worker+0x388" ?
(Need kernel debuginfo)
My guess is the following line:(pretty sure, but not 100% sure)
------
/*
* only update status, since the previous part has alreay
updated the
* qgroup info.
*/
trans =trfs_start_transaction(fs_info->quota_root, 1);
<<<<<
if (IS_ERR(trans)) {
err =TR_ERR(trans);
btrfs_err(fs_info,
"fail to start transaction for status
update: %d\n",
err);
goto done;
}
------
The kernel and modules were already compiled with debuginfo.
However for some reason, I couldn't get gdb disassembly of
/proc/kcore
properly
aligned with the source I compiled: the asm code doesn't match the C
code shown
by gdb. In any case, watching the source of this function, this is
the
only place
btrfs_start_transaction is called, so we can be 100% sure it's where
the
crash
happens indeed.
Yep, that's the only caller.
Here is some useful small hint to locate the code, if you are
interestied in kernel development.
# Not sure about whether ubuntu gzipped modules, at least Arch does
# compress it
$ cp <kernel modules dir>/kernel/fs/btrfs/btrfs.ko.gz /tmp/
$ gunzip /tmp/btrfs.ko.gz
$ gdb /tmp/btrfs.ko
# Make sure gdb read all the needed debuginfo
$ gdb list *(btrfs_qgroup_rescan_worker+0x388)
And gdb will find the code position for you.
Quite easy one, only backtrace info is needed.
Ah, thanks for the tips, I was loading whole vmlinux and using
/proc/kcore
as the core info, then adding the module with "add-symbol-file".
But as
we're just looking for the code and not the variables, it was indeed
completely overkill.
(gdb) list *(btrfs_qgroup_rescan_worker+0x388)
0x98068 is in btrfs_qgroup_rescan_worker (fs/btrfs/qgroup.c:2328).
2323
2324 /*
2325 * only update status, since the previous part has
alreay updated the
2326 * qgroup info.
2327 */
2328 trans =trfs_start_transaction(fs_info->quota_root,
1);
2329 if (IS_ERR(trans)) {
2330 err =TR_ERR(trans);
2331 btrfs_err(fs_info,
2332 "fail to start transaction for
status update: %d\n",
So this just confirms what we were already 99% sure of.
Another hint is about how to collect the kernel crash info.
Your netconsole setup would be definitely one good practice.
Another one I use to collect crash info is kdump.
Ubuntu should have a good wiki on it.
I've already come across kdump a few times, but never really look into
it.
To debug the other complicated extend backref bug, it could be of some
use.
So, as a quick summary of this big thread, it seems I've been
hitting
3 bugs, all reproductible :
- kernel BUG on balance (this original thread)
For this, I can't provide much help, as extent backref bug is quite
hard to debug, unless a developer is interested in it and find a
stable way to reproduce it.
Yes, unfortunately as it looks so much like a race condition, I know
I can
reproduce it with my worflow, but it can take between 1 minute
and 12
hours,
so I wouldn't call it a "stable way" to reproduce it
unfortunately :(
Still if any dev is interested in it, I can reproduce it, with a
patched
kernel if needed.
Maybe you are already doing it, you can only compile the btrfs
modules, which will be far more faster than compile the whole kernel,
if and only if the compiled module can be loaded.
Yes, I've compiled this 4.3.0-rc1 in a completely modular form, so
I'll try to
load the modified module and see if the running kernel accepts it. I
have to rmmod
the loaded module first, hence umounting any btrfs fs before that.
Should be able
to do it in a couple hours.
I'll delete again all my snapshots and run my script. Should be easy
to trigger
the (hopefully worked-around) bug again.
Well, I didn't trigger this exact bug, but another one, not less severe
though, as it also crashed the system:
[92098.841309] general protection fault: 0000 [#1] SMP
[92098.841338] Modules linked in: ...
[92098.841814] CPU: 1 PID: 24655 Comm: kworker/u4:12 Not tainted
4.3.0-rc1 #1
[92098.841834] Hardware name: ASUS All Series/H87I-PLUS, BIOS 1005
01/06/2014
[92098.841868] Workqueue: btrfs-qgroup-rescan
btrfs_qgroup_rescan_helper
[btrfs]
[92098.841889] task: ffff8800b6cc4100 ti: ffff8800a3dc8000 task.ti:
ffff8800a3dc8000
[92098.841910] RIP: 0010:[<ffffffff813ae6c6>] [<ffffffff813ae6c6>]
memcpy_erms+0x6/0x10
[92098.841935] RSP: 0018:ffff8800a3dcbcc8 EFLAGS: 00010207
[92098.841950] RAX: ffff8800a3dcbd67 RBX: 0000000000000009 RCX:
0000000000000009
[92098.841970] RDX: 0000000000000009 RSI: 0005080000000000 RDI:
ffff8800a3dcbd67
[92098.841989] RBP: ffff8800a3dcbd00 R08: 0000000000019c60 R09:
ffff88011fb19c60
[92098.842009] R10: ffffea0003006480 R11: 0000000001000000 R12:
ffff8800b76c32c0
[92098.842028] R13: 0000160000000000 R14: ffff8800a3dcbd70 R15:
0000000000000009
[92098.842048] FS: 0000000000000000(0000) GS:ffff88011fb00000(0000)
knlGS:0000000000000000
[92098.842070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92098.842086] CR2: 00007fe1f2bd8000 CR3: 0000000001c10000 CR4:
00000000000406e0
[92098.842105] Stack:
[92098.842111] ffffffffc035a5d8 ffffffffc0396d00 000000000000028b
0000000000000000
[92098.842212] 0000cc6c00000000 ffff8800b76c3200 0000160000000000
ffff8800a3dcbdc0
[92098.842237] ffffffffc039af3d ffff8800c7196dc8 ffff8800c7196e08
ffff8800c7196da0
[92098.842261] Call Trace:
[92098.842277] [<ffffffffc035a5d8>] ? read_extent_buffer+0xb8/0x110
[btrfs]
[92098.842304] [<ffffffffc0396d00>] ? btrfs_find_all_roots+0x60/0x70
[btrfs]
[92098.842329] [<ffffffffc039af3d>]
btrfs_qgroup_rescan_worker+0x28d/0x5a0 [btrfs]
Would you please show the code of it?
This one seems to be another stupid bug I made when rewriting the
framework.
Maybe I forgot to reinit some variants or I'm screwing memory...
(gdb) list *(btrfs_qgroup_rescan_worker+0x28d)
0x97f6d is in btrfs_qgroup_rescan_worker (fs/btrfs/ctree.h:2760).
2755
2756 static inline void btrfs_disk_key_to_cpu(struct btrfs_key *cpu,
2757 struct btrfs_disk_key
*disk)
2758 {
2759 cpu->offset =e64_to_cpu(disk->offset);
2760 cpu->type =isk->type;
2761 cpu->objectid =e64_to_cpu(disk->objectid);
2762 }
2763
2764 static inline void btrfs_cpu_key_to_disk(struct btrfs_disk_key
*disk,
(gdb)
Does it makes sense ?
So it seems that the memory of cpu key is being screwed up...
The code is be specific thin inline function, so what about other stack?
Like btrfs_qgroup_rescan_helper+0x12?
Thanks,
Qu
Oh, I forgot that you can just change the number of
btrfs_qgroup_rescan_worker+0x28d to smaller value.
Try +0x280 for example, which will revert to 14 bytes asm code back,
which may jump out of the inline function range, and may give you a good
hint.
Or gdb may have a better mode for inline function, but I don't know...
Thanks,
Qu
[92098.842351] [<ffffffff810a1a0d>] ?
ttwu_do_activate.constprop.90+0x5d/0x70
[92098.842377] [<ffffffffc03674e0>] normal_work_helper+0xc0/0x270
[btrfs]
[92098.842401] [<ffffffffc03678a2>]
btrfs_qgroup_rescan_helper+0x12/0x20 [btrfs]
[92098.842421] [<ffffffff8109127e>] process_one_work+0x14e/0x3d0
[92098.842438] [<ffffffff8109192a>] worker_thread+0x11a/0x470
[92098.842454] [<ffffffff81091810>] ? rescuer_thread+0x310/0x310
[92098.842471] [<ffffffff81097059>] kthread+0xc9/0xe0
[92098.842485] [<ffffffff81096f90>] ? kthread_park+0x60/0x60
[92098.842502] [<ffffffff817aac4f>] ret_from_fork+0x3f/0x70
[92098.842517] [<ffffffff81096f90>] ? kthread_park+0x60/0x60
[92098.842532] Code: ff eb eb 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48
c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48
89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38
[92098.842658] RIP [<ffffffff813ae6c6>] memcpy_erms+0x6/0x10
[92098.842675] RSP <ffff8800a3dcbcc8>
[92098.849594] ---[ end trace 9d5fb7931a3ec713 ]---
I would definitely say that rescans should be avoided on current
kernels
as the possibility that it'll bring the system down shouldn't be
ignored.
It confirms that this code really needs a rewrite !
Regards,
--
To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
