Le 2015-09-20 03:22, Qu Wenruo a écrit :
The mentioned steps are as follows :
0) Rsync data from the next ext4 "snapshot" to the subvolume
1) Do 'sync; btrfs qgroup show -prce --raw' and save the output
<==
2) Create the needed readonly snapshot on btrfs
3) Do 'sync; btrfs qgroup show -prce --raw' and save the output
<==
4) Avoid doing IO if possible until step 6)
5) Do 'btrfs quota rescan -w' and save it <==
6) Do 'sync; btrfs qgroup show -prce --raw' and save the output
<==
The resulting files are available here:
http://speed47.net/tmp2/qgroup.tar.gz
The run2 is the more complete one, during run1 the machine crashed
even faster.
It's interesting to note, however, that it seems to have crashed the
same way and at the same step in the process.
Your data really helps a lot!!
And the good news is, the qgroup accouting part is working as expected.
Although I only checked about 1/3/6 of about 5 snaps, they are all OK.
I can make a script to cross check them, but from the last few result,
I think qgroup works fine.
I'm more confident about the minus number, which should be a result of
deleted subvolume, and the real problem is, such qgroup is not handled
well with qgroup rescan.
I agree with your analysis, this matches what I observed.
I'll try to add a hot fix for such case if needed.
But right now, I don't have a good idea for it until Mark's work of
rescan subtree.
Maybe I can add a new option for btrfs-progs to automatically remove
the qgroup and trigger a rescan?
Until this is properly fixed in the kernel code, and this is good news
to
know Mark and you are working on it, this would be a good workaround
yes!
[ 5738.172879] Call Trace:
[ 5738.172887] [<ffffffffc031565b>] btrfs_start_transaction+0x1b/0x20
[btrfs]
[ 5738.172896] [<ffffffffc0378038>]
btrfs_qgroup_rescan_worker+0x388/0x5a0 [btrfs]
Your netconsole backtrace is also of greate value.
This one implies that, my rework also caused some stupid bug.
(Yeah, I always make such bugs) or some existing unexposed rescan bug.
Would you please use gdb to show the codes of
"btrfs_qgroup_rescan_worker+0x388" ?
(Need kernel debuginfo)
My guess is the following line:(pretty sure, but not 100% sure)
------
/*
* only update status, since the previous part has alreay
updated the
* qgroup info.
*/
trans = btrfs_start_transaction(fs_info->quota_root, 1); <<<<<
if (IS_ERR(trans)) {
err = PTR_ERR(trans);
btrfs_err(fs_info,
"fail to start transaction for status update:
%d\n",
err);
goto done;
}
------
The kernel and modules were already compiled with debuginfo.
However for some reason, I couldn't get gdb disassembly of /proc/kcore
properly
aligned with the source I compiled: the asm code doesn't match the C
code shown
by gdb. In any case, watching the source of this function, this is the
only place
btrfs_start_transaction is called, so we can be 100% sure it's where the
crash
happens indeed.
But that means, at rescan time, fs_info->quota_root is still NULL,
which is quite wired.
I can add extra check to avoid such NULL pointer for now, but it's
better to review the existing rescan workflow, as I think there is
some race for it to init quota_root.
You can also try the following hotfix patch to see if it works:
http://pastebin.com/966GQXPk
My concern is, this may cause qgroup rescan to exit without updating
its accounting info...
So still need your help.
Or I can use your reproducer script to test it next Monday.
Compiling with your patch, just amended of a little printk to know if
the execution
flow enters the added if condition. Will let you know about the results.
But I'm pretty sure I can get that (u64)-1 value again by deleting
snapshots. Shall I ? Or do you have something else for me to run
before that ?
You have already done a great job in helping maturing qgroups.
The minus number and 0 excl is somewhat expected for deleted snapshots.
Good news is, 1) it doesn't affect valid(non-orphan) qgroup.
2) Mark is already working on it.
I'll try to add a btrfs-progs hotfix for you to delete and rescan
qgroups to avoid such problem.
That would be good !
So, as a quick summary of this big thread, it seems I've been hitting
3 bugs, all reproductible :
- kernel BUG on balance (this original thread)
For this, I can't provide much help, as extent backref bug is quite
hard to debug, unless a developer is interested in it and find a
stable way to reproduce it.
Yes, unfortunately as it looks so much like a race condition, I know I
can
reproduce it with my worflow, but it can take between 1 minute and 12
hours,
so I wouldn't call it a "stable way" to reproduce it unfortunately :(
Still if any dev is interested in it, I can reproduce it, with a patched
kernel if needed.
The rest two are explained or have hot fix mentioned above.
And thanks for that, will keep you posted.
--
Stéphane.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
