On Fri, Dec 01, 2017 at 05:15:55PM +0100, Tomasz Pala wrote: > Hello, > > I got a problem with btrfs running out of space (not THE > Internet-wide, well known issues with interpretation). > > The problem is: something eats the space while not running anything that > justifies this. There were 18 GB free space available, suddenly it > dropped to 8 GB and then to 63 MB during one night. I recovered 1 GB > with rebalance -dusage=5 -musage=5 (or sth about), but it is being eaten > right now, just as I'm writing this e-mail: > > /dev/sda2 64G 63G 452M 100% / > /dev/sda2 64G 63G 365M 100% / > /dev/sda2 64G 63G 316M 100% / > /dev/sda2 64G 63G 287M 100% / > /dev/sda2 64G 63G 268M 100% / > /dev/sda2 64G 63G 239M 100% / > /dev/sda2 64G 63G 230M 100% / > /dev/sda2 64G 63G 182M 100% / > /dev/sda2 64G 63G 163M 100% / > /dev/sda2 64G 64G 153M 100% / > /dev/sda2 64G 64G 143M 100% / > /dev/sda2 64G 64G 96M 100% / > /dev/sda2 64G 64G 88M 100% / > /dev/sda2 64G 64G 57M 100% / > /dev/sda2 64G 64G 25M 100% / > > while my rough calculations show, that there should be at least 10 GB of > free space. After enabling quotas it is somehow confirmed: > > # btrfs qgroup sh --sort=excl / > qgroupid rfer excl > -------- ---- ---- > 0/5 16.00KiB 16.00KiB > [30 snapshots with about 100 MiB excl] > 0/333 24.53GiB 305.79MiB > 0/298 13.44GiB 312.74MiB > 0/327 23.79GiB 427.13MiB > 0/331 23.93GiB 930.51MiB > 0/260 12.25GiB 3.22GiB > 0/312 19.70GiB 4.56GiB > 0/388 28.75GiB 7.15GiB > 0/291 30.60GiB 9.01GiB <- this is the running one > > This is about 30 GB total excl (didn't find a switch to sum this up). I > know I can't just add 'excl' to get usage, so tried to pinpoint the > exact files that occupy space in 0/388 exclusively (this is the last > snapshots taken, all of the snapshots are created from the running fs).
The thing I'd first go looking for here is some rogue process writing lots of data. I've had something like this happen to me before, a few times. First, I'd look for large files with "du -ms /* | sort -n", then work down into the tree until you find them. If that doesn't show up anything unusually large, then lsof to look for open but deleted files (orphans) which are still being written to by some process. This is very likely _not_ to be a btrfs problem, but instead some runaway process writing lots of crap very fast. Log files are probably the most plausible location, but not the only one. > Now, the weird part for me is exclusive data count: > > # btrfs sub sh ./snapshot-171125 > [...] > Subvolume ID: 388 > # btrfs fi du -s ./snapshot-171125 > Total Exclusive Set shared Filename > 21.50GiB 63.35MiB 20.77GiB snapshot-171125 > > > How is that possible? This doesn't even remotely relate to 7.15 GiB > from qgroup.~The same amount differs in total: 28.75-21.50=7.25 GiB. > And the same happens with other snapshots, much more exclusive data > shown in qgroup than actually found in files. So if not files, where > is that space wasted? Metadata? Personally, I'd trust qgroups' output about as far as I could spit Belgium(*). Hugo. (*) No offence indended to Belgium. -- Hugo Mills | I used to live in hope, but I got evicted. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
signature.asc
Description: Digital signature
