On 2018/8/12 上午8:30, Qu Wenruo wrote: > > > On 2018/8/12 上午5:10, Dan Merillat wrote: >> 19 hours later, still going extremely slowly and taking longer and >> longer for progress made. Main symptom is the mount process is >> spinning at 100% CPU, interspersed with btrfs-transaction spinning at >> 100% CPU. >> So far it's racked up 14h45m of CPU time on mount and an additional >> 3h40m on btrfs-transaction. >> >> The current drop key changes every 10-15 minutes when I check it via >> inspect-internal, so some progress is slowly being made. >> >> I built the kernel with ftrace to see what's going on internally, this >> is the pattern I'm seeing: >> > [snip] > > It looks pretty like qgroup, but too many noise. > The pin point trace event would btrfs_find_all_roots(). > >> >> Repeats indefinitely. btrace shows basically zero activity on the >> array while it spins, with the occasional burst when mount & >> btrfs-transaction swap off. >> >> To recap the chain of events leading up to this: >> 11TB Array got completely full and started fragmenting badly. >> Ran bedup and it found 600gb of duplicate files that it offline-shared. >> Reboot for unrelated reasons > > 11T, with highly deduped usage is really the worst scenario case for qgroup. > Qgroup is not really good at handle hight reflinked files, nor balance. > When they combines, it goes worse. > >> Enabled quota on all subvolumes to try to track where the new data is >> coming from >> Tried to balance metadata due to transaction CPU spikes >> Force-rebooted after the array was completely lagged out. >> >> Now attempting to mount it RW. Readonly works, but RW has taken well >> over 24 hours at this point. > > I'll add a new rescue subcommand, 'btrfs rescue disable-quota' for you > to disable quota offline.
Patch set (from my work mailbox), titled "[PATCH] btrfs-progs: rescue: Add ability to disable quota offline". Can also be fetched from github: https://github.com/adam900710/btrfs-progs/tree/quota_disable Usage is: # btrfs rescue disable-quota <device> Tested locally, it would just toggle the ON/OFF flag for quota, so the modification should be minimal. Thanks, Qu > > Thanks, > Qu > >
signature.asc
Description: OpenPGP digital signature
