Re: "no space left on device" from tar on ppc64le
Hi Rich, Le 18/06/2019 à 13:19, Chris Murphy a écrit : > On Tue, Jun 18, 2019 at 4:23 PM Rich Turner wrote: >> >> tar: ./lib/modules/4.4.73-7-default/kernel/drivers/md/faulty.ko: Cannot >> open: No space left on device > > If this really is a 4.4.73 based kernel, I expect the report is out of > scope for this list. There have been 109 subsequent stable releases of > the 4.4 kernel since. There nearly 3000 commits between 4.4 and 5.1. I would also recommend using a newer kernel. I'm the one who reported the problem back in October, and I had to make new SD cards 2 weeks ago. I used the exact same script on kernel 5.1.x and the problem did not come up again, ie I was able to untar without the hack to slow down write speed. Thanks to Btrfs developpers! Best regards, -- Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527
Re: "no space left on device" from tar on ppc64le
On Tue, Jun 18, 2019 at 4:23 PM Rich Turner wrote: > > tar: ./lib/modules/4.4.73-7-default/kernel/drivers/md/faulty.ko: Cannot open: > No space left on device If this really is a 4.4.73 based kernel, I expect the report is out of scope for this list. There have been 109 subsequent stable releases of the 4.4 kernel since. There nearly 3000 commits between 4.4 and 5.1. Ideally you'd retest with 5.2-rc5 since that's the current mainline that this upstream development list is actively working on fixing bugs in. But I'd suggest at the oldest, 4.19.52 or whatever most recent version has Btrfs backports. If it does reproduce with a recent kernel, I suggest remounting with option enospc_debug to include additional information; and also include the output from $ sudo ./btrfs-debugfs -b / mntpoint The btrfs-debugfs script can be found in upstream btrfs-progs, I'm not sure if it's typically packaged by distributions. https://github.com/kdave/btrfs-progs > PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3" In the case you can't test with something recent, then I suggest reporting it to this distribution's support. That's the way it's supposed to work. -- Chris Murphy
Re: No space left on device when doing "mkdir"
It did it again: shrapnel share # touch test.txt touch: cannot touch 'test.txt': No space left on device shrapnel share # df -h Filesystem Size Used Avail Use% Mounted on /dev/root35G 19G 15G 56% / devtmpfs 10M 0 10M 0% /dev tmpfs 3.2G 1.2M 3.2G 1% /run shm 16G 0 16G 0% /dev/shm cgroup_root 10M 0 10M 0% /sys/fs/cgroup /dev/sdb 35T 22T 14T 62% /home/exports shrapnel share # grep -IR . /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/ /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/flags:2 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/raid1/used_bytes:3997696 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/raid1/total_bytes:33554432 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/disk_total:67108864 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_may_use:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_readonly:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_used:3997696 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_reserved:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/disk_used:7995392 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/total_bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/total_bytes:33554432 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/flags:4 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/raid1/used_bytes:66595684352 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/raid1/total_bytes:280246616064 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_pinned:835584 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/disk_total:560493232128 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_may_use:2014478974976 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_readonly:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_used:66595684352 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_reserved:16384 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/disk_used:133191368704 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/total_bytes_pinned:1048576 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/total_bytes:280246616064 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/global_rsv_size:536870912 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/flags:1 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/raid1/used_bytes:23249396273152 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/raid1/total_bytes:23320598675456 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_pinned:1835008 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/disk_total:46641197350912 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_may_use:262144 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_readonly:1769472 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_used:23249396273152 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_reserved:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/disk_used:46498792546304 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/total_bytes_pinned:2097152 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/total_bytes:23320598675456 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/global_rsv_reserved:536018944 On Fri, Apr 28, 2017 at 8:56 AM, Gerard Saraber wrote: > Dmarc is off, here's the output of the allocations: it's working > correctly right now, I'll update when it does it again. > > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/flags:2 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/raid1/used_bytes:3948544 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/raid1/total_bytes:33554432 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_pinned:0 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/disk_total:67108864 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_may_use:0 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_readonly:0 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_used:3948544 > /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_reserved:0 > /sys/fs/btrfs/
Re: No space left on device when doing "mkdir"
Dmarc is off, here's the output of the allocations: it's working correctly right now, I'll update when it does it again. /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/flags:2 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/raid1/used_bytes:3948544 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/raid1/total_bytes:33554432 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/disk_total:67108864 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_may_use:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_readonly:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_used:3948544 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/bytes_reserved:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/disk_used:7897088 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/total_bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/system/total_bytes:33554432 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/flags:4 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/raid1/used_bytes:65864957952 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/raid1/total_bytes:83751862272 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/disk_total:167503724544 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_may_use:739508224 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_readonly:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_used:65864957952 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/bytes_reserved:1835008 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/disk_used:131729915904 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/total_bytes_pinned:1884160 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/metadata/total_bytes:83751862272 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/global_rsv_size:536870912 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/flags:1 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/raid1/used_bytes:23029876707328 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/raid1/total_bytes:23175643529216 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/disk_total:46351287058432 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_may_use:36474880 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_readonly:1703936 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_used:23029876707328 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/bytes_reserved:15003648 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/disk_used:46059753414656 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/total_bytes_pinned:0 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/data/total_bytes:23175643529216 /sys/fs/btrfs/7af2e65c-3935-4e0d-aa63-9ef6be991cb9/allocation/global_rsv_reserved:536870912 On Thu, Apr 27, 2017 at 6:35 PM, Chris Murphy wrote: > On Thu, Apr 27, 2017 at 10:46 AM, Gerard Saraber wrote: >> After a reboot, I found this in the logs: >> [ 322.510152] BTRFS info (device sdm): The free space cache file >> (36114966511616) is invalid. skip it >> [ 488.702570] btrfs_printk: 847 callbacks suppressed >> >> >> >> On Thu, Apr 27, 2017 at 10:18 AM, Gerard Saraber wrote: >>> no snapshots and no qgroups, just a straight up large volume. >>> >>> shrapnel gerard-store # btrfs fi df /home/exports >>> Data, RAID1: total=20.93TiB, used=20.86TiB >>> System, RAID1: total=32.00MiB, used=3.73MiB >>> Metadata, RAID1: total=79.00GiB, used=61.10GiB >>> GlobalReserve, single: total=512.00MiB, used=544.00KiB >>> >>> shrapnel gerard-store # btrfs filesystem usage /home/exports >>> Overall: >>> Device size: 69.13TiB >>> Device allocated: 42.01TiB >>> Device unallocated: 27.13TiB >>> Device missing: 0.00B >>> Used: 41.84TiB >>> Free (estimated): 13.63TiB (min: 13.63TiB) >>> Data ratio: 2.00 >>> Metadata ratio: 2.00 >>> Global reserve: 512.00MiB (used: 1.52MiB) >>> >>> On Thu, Apr 27, 2017 at 9:07 AM, Roman Mamedov wrote: On Thu, 27 Apr 2017 08:52:30 -0500 Gerard Saraber wrote: > I could just reboot the sy
Re: No space left on device when doing "mkdir"
On Thu, Apr 27, 2017 at 10:46 AM, Gerard Saraber wrote: > After a reboot, I found this in the logs: > [ 322.510152] BTRFS info (device sdm): The free space cache file > (36114966511616) is invalid. skip it > [ 488.702570] btrfs_printk: 847 callbacks suppressed > > > > On Thu, Apr 27, 2017 at 10:18 AM, Gerard Saraber wrote: >> no snapshots and no qgroups, just a straight up large volume. >> >> shrapnel gerard-store # btrfs fi df /home/exports >> Data, RAID1: total=20.93TiB, used=20.86TiB >> System, RAID1: total=32.00MiB, used=3.73MiB >> Metadata, RAID1: total=79.00GiB, used=61.10GiB >> GlobalReserve, single: total=512.00MiB, used=544.00KiB >> >> shrapnel gerard-store # btrfs filesystem usage /home/exports >> Overall: >> Device size: 69.13TiB >> Device allocated: 42.01TiB >> Device unallocated: 27.13TiB >> Device missing: 0.00B >> Used: 41.84TiB >> Free (estimated): 13.63TiB (min: 13.63TiB) >> Data ratio: 2.00 >> Metadata ratio: 2.00 >> Global reserve: 512.00MiB (used: 1.52MiB) >> >> On Thu, Apr 27, 2017 at 9:07 AM, Roman Mamedov wrote: >>> On Thu, 27 Apr 2017 08:52:30 -0500 >>> Gerard Saraber wrote: >>> I could just reboot the system and be fine for a week or so, but is there any way to diagnose this? >>> >>> `btrfs fi df` for a start. >>> >>> Also obligatory questions: do you have a lot of snapshots, and do you use >>> qgroups? >>> A dev might find this helpful $ grep -IR . /sys/fs/btrfs/usevolumeUUIDhere/allocation/ Also note that a lot of people on Btrfs aren't getting Gerard's emails, because anyone using gmail and some other agents see it as spam because of DMARC failure. Basically rarcoa.com is configured to tell mail senders to fail to (re)send emails, they can only be sent from raroa.com. Anyway, I think this is supposed to be fixed in mailing list servers, they need to strip these headers and insert their own rather than leaving them intact only later to get rejected due to honoring the header's stated policy. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device when doing "mkdir"
After a reboot, I found this in the logs: [ 322.510152] BTRFS info (device sdm): The free space cache file (36114966511616) is invalid. skip it [ 488.702570] btrfs_printk: 847 callbacks suppressed On Thu, Apr 27, 2017 at 10:18 AM, Gerard Saraber wrote: > no snapshots and no qgroups, just a straight up large volume. > > shrapnel gerard-store # btrfs fi df /home/exports > Data, RAID1: total=20.93TiB, used=20.86TiB > System, RAID1: total=32.00MiB, used=3.73MiB > Metadata, RAID1: total=79.00GiB, used=61.10GiB > GlobalReserve, single: total=512.00MiB, used=544.00KiB > > shrapnel gerard-store # btrfs filesystem usage /home/exports > Overall: > Device size: 69.13TiB > Device allocated: 42.01TiB > Device unallocated: 27.13TiB > Device missing: 0.00B > Used: 41.84TiB > Free (estimated): 13.63TiB (min: 13.63TiB) > Data ratio: 2.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 1.52MiB) > > On Thu, Apr 27, 2017 at 9:07 AM, Roman Mamedov wrote: >> On Thu, 27 Apr 2017 08:52:30 -0500 >> Gerard Saraber wrote: >> >>> I could just reboot the system and be fine for a week or so, but is >>> there any way to diagnose this? >> >> `btrfs fi df` for a start. >> >> Also obligatory questions: do you have a lot of snapshots, and do you use >> qgroups? >> >> -- >> With respect, >> Roman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device when doing "mkdir"
no snapshots and no qgroups, just a straight up large volume. shrapnel gerard-store # btrfs fi df /home/exports Data, RAID1: total=20.93TiB, used=20.86TiB System, RAID1: total=32.00MiB, used=3.73MiB Metadata, RAID1: total=79.00GiB, used=61.10GiB GlobalReserve, single: total=512.00MiB, used=544.00KiB shrapnel gerard-store # btrfs filesystem usage /home/exports Overall: Device size: 69.13TiB Device allocated: 42.01TiB Device unallocated: 27.13TiB Device missing: 0.00B Used: 41.84TiB Free (estimated): 13.63TiB (min: 13.63TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 1.52MiB) On Thu, Apr 27, 2017 at 9:07 AM, Roman Mamedov wrote: > On Thu, 27 Apr 2017 08:52:30 -0500 > Gerard Saraber wrote: > >> I could just reboot the system and be fine for a week or so, but is >> there any way to diagnose this? > > `btrfs fi df` for a start. > > Also obligatory questions: do you have a lot of snapshots, and do you use > qgroups? > > -- > With respect, > Roman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device when doing "mkdir"
On Thu, 27 Apr 2017 08:52:30 -0500 Gerard Saraber wrote: > I could just reboot the system and be fine for a week or so, but is > there any way to diagnose this? `btrfs fi df` for a start. Also obligatory questions: do you have a lot of snapshots, and do you use qgroups? -- With respect, Roman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On Tue, Aug 9, 2016 at 7:16 AM, Austin S. Hemmelgarn wrote: > On 2016-08-09 05:50, MegaBrutal wrote: >> >> 2016-06-03 14:43 GMT+02:00 Austin S. Hemmelgarn : >>> >>> >>> Also, since you're on a new enough kernel, try 'lazytime' in the mount >>> options as well, this defers all on-disk timestamp updates for up to 24 >>> hours or until the inode gets written out anyway, but keeps the updated info >>> in memory. The only downside to this is that mtimes might not be correct >>> after an unclean shutdown, but most software will have no issues with this. >>> >> >> Hi all, >> >> Sorry for reviving this old thread, and probably it's not the best >> place to ask about this... but I added the "noatime" option in fstab, >> restarted the system, and now I think I should try "lazytime" too (as >> I like the idea to have proper atimes with delayed writing to disk). >> So now I'd just like to test the "lazytime" mount option without >> restart. >> >> I remounted the file system like this: >> >> mount -o remount,lazytime / >> >> But now the FS still has the "noatime" mount option, which I guess >> renders "lazytime" ineffective. I thought they are supposed to be >> mutually exclusive, so I'm actually surprised that I can have both >> mount options at the same time. > > No, lazytime also affects mtime handling, not just atime, so they aren't > mutually exclusive, and it does make sense to have both. >> >> >> Now my mount looks like this: >> >> /dev/mapper/centrevg-rootlv on / type btrfs >> (rw,noatime,lazytime,space_cache,subvolid=257,subvol=/@) >> >> I also tried to explicitly add "atime" to negate "noatime" (man mount >> says "atime" is the option to disable "noatime"), like this: >> >> mount -o remount,atime,lazytime / >> >> But the "noatime" option still stays. Why? Is it a BTRFS specific >> issue, or does it reside in another layer? >> >> By the way, is it valid to mount BTRFS subvolumes with different atime >> policies? Then how do child subvolumes behave? > > I'm not entirely certain (I have my kernel patched so noatime is the > default, and rarely if ever use anything else, so I don't have much in the > way of experience with this), but my guess would be that it can't be done, > and that changing atime handling on remount isn't really handled. I do know > that adding lazytime on remount doesn't always work, but that kind of makes > sense, since it causes significant changes in how mtimes and atimes are > handled internally. > TLDR: try 'mount -o remount,strictatime / && mount -o remount,relatime /' This seems strange, but on my unpatched system: $ uname -r 4.7.0 $ mount -o loop,noatime,lazytime /var/tmp/test.dd /mnt $ grep ^/dev/loop0 /proc/mounts /dev/loop0 /mnt btrfs rw,lazytime,noatime,space_cache,subvolid=5,subvol=/ 0 0 $ mount -o remount,relatime /mnt $ grep ^/dev/loop0 /proc/mounts /dev/loop0 /mnt btrfs rw,lazytime,noatime,space_cache,subvolid=5,subvol=/ 0 0 ^^^ No change to noatime option $ mount -o remount,strictatime /mnt $ grep ^/dev/loop0 /proc/mounts /dev/loop0 /mnt btrfs rw,lazytime,space_cache,subvolid=5,subvol=/ 0 0 ^^^ that updated it... $ mount -o remount,relatime /mnt $ grep ^/dev/loop0 /proc/mounts /dev/loop0 /mnt btrfs rw,lazytime,relatime,space_cache,subvolid=5,subvol=/ 0 0 ^^^ now it changes ~ Noah -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On 2016-08-09 05:50, MegaBrutal wrote: 2016-06-03 14:43 GMT+02:00 Austin S. Hemmelgarn : Also, since you're on a new enough kernel, try 'lazytime' in the mount options as well, this defers all on-disk timestamp updates for up to 24 hours or until the inode gets written out anyway, but keeps the updated info in memory. The only downside to this is that mtimes might not be correct after an unclean shutdown, but most software will have no issues with this. Hi all, Sorry for reviving this old thread, and probably it's not the best place to ask about this... but I added the "noatime" option in fstab, restarted the system, and now I think I should try "lazytime" too (as I like the idea to have proper atimes with delayed writing to disk). So now I'd just like to test the "lazytime" mount option without restart. I remounted the file system like this: mount -o remount,lazytime / But now the FS still has the "noatime" mount option, which I guess renders "lazytime" ineffective. I thought they are supposed to be mutually exclusive, so I'm actually surprised that I can have both mount options at the same time. No, lazytime also affects mtime handling, not just atime, so they aren't mutually exclusive, and it does make sense to have both. Now my mount looks like this: /dev/mapper/centrevg-rootlv on / type btrfs (rw,noatime,lazytime,space_cache,subvolid=257,subvol=/@) I also tried to explicitly add "atime" to negate "noatime" (man mount says "atime" is the option to disable "noatime"), like this: mount -o remount,atime,lazytime / But the "noatime" option still stays. Why? Is it a BTRFS specific issue, or does it reside in another layer? By the way, is it valid to mount BTRFS subvolumes with different atime policies? Then how do child subvolumes behave? I'm not entirely certain (I have my kernel patched so noatime is the default, and rarely if ever use anything else, so I don't have much in the way of experience with this), but my guess would be that it can't be done, and that changing atime handling on remount isn't really handled. I do know that adding lazytime on remount doesn't always work, but that kind of makes sense, since it causes significant changes in how mtimes and atimes are handled internally. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
2016-06-03 14:43 GMT+02:00 Austin S. Hemmelgarn : > > Also, since you're on a new enough kernel, try 'lazytime' in the mount > options as well, this defers all on-disk timestamp updates for up to 24 hours > or until the inode gets written out anyway, but keeps the updated info in > memory. The only downside to this is that mtimes might not be correct after > an unclean shutdown, but most software will have no issues with this. > Hi all, Sorry for reviving this old thread, and probably it's not the best place to ask about this... but I added the "noatime" option in fstab, restarted the system, and now I think I should try "lazytime" too (as I like the idea to have proper atimes with delayed writing to disk). So now I'd just like to test the "lazytime" mount option without restart. I remounted the file system like this: mount -o remount,lazytime / But now the FS still has the "noatime" mount option, which I guess renders "lazytime" ineffective. I thought they are supposed to be mutually exclusive, so I'm actually surprised that I can have both mount options at the same time. Now my mount looks like this: /dev/mapper/centrevg-rootlv on / type btrfs (rw,noatime,lazytime,space_cache,subvolid=257,subvol=/@) I also tried to explicitly add "atime" to negate "noatime" (man mount says "atime" is the option to disable "noatime"), like this: mount -o remount,atime,lazytime / But the "noatime" option still stays. Why? Is it a BTRFS specific issue, or does it reside in another layer? By the way, is it valid to mount BTRFS subvolumes with different atime policies? Then how do child subvolumes behave? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On Sat, Jun 04, 2016 at 09:27:13AM +0300, Andrei Borzenkov wrote: > 02.06.2016 15:56, Austin S. Hemmelgarn пишет: > > > > In your particular situation, what's happened is that you have all the > > space allocated to chunks, but have free space within those chunks. > > Balance never puts data in existing chunks, and you can't allocate any > > new chunks, so you can't run a balance. However, because of that free > > space in the chunks, you can still use the filesystem itself for > > 'regular' filesystem operations. > > > > How balance decides where to put data from chunks it frees? I.e. let's > say I have one free data chunk and 10 chunks filled to 10%. Will "btrfs > ba start -dusage=10" pack data from all 10 chunks into single one, this > freeing 10 chunks for further processing? Yes, it will. Andrei's assertion is, I'm afraid, incorrect. Hugo. -- Hugo Mills | There's many a slip 'twixt wicket-keeper and gully. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 | signature.asc Description: Digital signature
Re: "No space left on device" and balance doesn't work
02.06.2016 15:56, Austin S. Hemmelgarn пишет: > > In your particular situation, what's happened is that you have all the > space allocated to chunks, but have free space within those chunks. > Balance never puts data in existing chunks, and you can't allocate any > new chunks, so you can't run a balance. However, because of that free > space in the chunks, you can still use the filesystem itself for > 'regular' filesystem operations. > How balance decides where to put data from chunks it frees? I.e. let's say I have one free data chunk and 10 chunks filled to 10%. Will "btrfs ba start -dusage=10" pack data from all 10 chunks into single one, this freeing 10 chunks for further processing? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On 2016-06-02 18:45, Henk Slager wrote: On Thu, Jun 2, 2016 at 3:55 PM, MegaBrutal wrote: 2016-06-02 0:22 GMT+02:00 Henk Slager : What is the kernel version used? Is the fs on a mechanical disk or SSD? What are the mount options? How old is the fs? Linux 4.4.0-22-generic (Ubuntu 16.04). Mechanical disks in LVM. Mount: /dev/mapper/centrevg-rootlv on / type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/@) I don't know how to retrieve the exact FS age, but it was created in 2014 August. Snapshots (their names encode their creation dates): ID 908 gen 487349 top level 5 path @-snapshot-2016050301 ... ID 937 gen 521829 top level 5 path @-snapshot-2016060201 Removing old snapshots is the most feasible solution, but I can also increase the FS size. It's easy since it's in LVM, and there is plenty of space in the volume group. Probably I should rewrite my alert script to check btrfs fi show instead of plain df. Yes I think that makes sense, to decide on chunk-level. You can see how big the chunks are with the linked show_usage.py program, most of 33 should be 1GiB as already very well explained by Austin. The setup looks all pretty normal and btrfs should be able to handle it, but unfortunately your fs is a typical example that one currently needs to monitor/tune a btrfs fs for its 'health' in order to keep it running longterm. You might want to change mount option relatime to noatime, so that you have less writes to metadata chunks. It should lower the scattering inside the metadata chunks. Also, since you're on a new enough kernel, try 'lazytime' in the mount options as well, this defers all on-disk timestamp updates for up to 24 hours or until the inode gets written out anyway, but keeps the updated info in memory. The only downside to this is that mtimes might not be correct after an unclean shutdown, but most software will have no issues with this. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On Fri, Jun 03, 2016 at 12:45:51AM +0200, Henk Slager wrote: > The setup looks all pretty normal and btrfs should be able to handle > it, but unfortunately your fs is a typical example that one currently > needs to monitor/tune a btrfs fs for its 'health' in order to keep it > running longterm. What kind of work is being done to address this major usability issue? What is the timeframe for a fix? Greetings Marc -- - Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany| lose things."Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On Thu, Jun 2, 2016 at 3:55 PM, MegaBrutal wrote: > 2016-06-02 0:22 GMT+02:00 Henk Slager : >> What is the kernel version used? >> Is the fs on a mechanical disk or SSD? >> What are the mount options? >> How old is the fs? > > Linux 4.4.0-22-generic (Ubuntu 16.04). > Mechanical disks in LVM. > Mount: /dev/mapper/centrevg-rootlv on / type btrfs > (rw,relatime,space_cache,subvolid=257,subvol=/@) > I don't know how to retrieve the exact FS age, but it was created in > 2014 August. > > Snapshots (their names encode their creation dates): > > ID 908 gen 487349 top level 5 path @-snapshot-2016050301 ... > ID 937 gen 521829 top level 5 path @-snapshot-2016060201 > > Removing old snapshots is the most feasible solution, but I can also > increase the FS size. It's easy since it's in LVM, and there is plenty > of space in the volume group. > > Probably I should rewrite my alert script to check btrfs fi show > instead of plain df. Yes I think that makes sense, to decide on chunk-level. You can see how big the chunks are with the linked show_usage.py program, most of 33 should be 1GiB as already very well explained by Austin. The setup looks all pretty normal and btrfs should be able to handle it, but unfortunately your fs is a typical example that one currently needs to monitor/tune a btrfs fs for its 'health' in order to keep it running longterm. You might want to change mount option relatime to noatime, so that you have less writes to metadata chunks. It should lower the scattering inside the metadata chunks. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
2016-06-02 0:22 GMT+02:00 Henk Slager : > What is the kernel version used? > Is the fs on a mechanical disk or SSD? > What are the mount options? > How old is the fs? Linux 4.4.0-22-generic (Ubuntu 16.04). Mechanical disks in LVM. Mount: /dev/mapper/centrevg-rootlv on / type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/@) I don't know how to retrieve the exact FS age, but it was created in 2014 August. Snapshots (their names encode their creation dates): ID 908 gen 487349 top level 5 path @-snapshot-2016050301 ID 909 gen 488849 top level 5 path @-snapshot-2016050401 ID 910 gen 490313 top level 5 path @-snapshot-2016050501 ID 911 gen 491763 top level 5 path @-snapshot-2016050601 ID 912 gen 493399 top level 5 path @-snapshot-2016050702 ID 913 gen 494996 top level 5 path @-snapshot-2016050802 ID 914 gen 496495 top level 5 path @-snapshot-2016050902 ID 915 gen 498094 top level 5 path @-snapshot-2016051005 ID 916 gen 499688 top level 5 path @-snapshot-2016051102 ID 917 gen 501308 top level 5 path @-snapshot-2016051201 ID 918 gen 503375 top level 5 path @-snapshot-2016051402 ID 919 gen 504356 top level 5 path @-snapshot-2016051501 ID 920 gen 505890 top level 5 path @-snapshot-2016051601 ID 921 gen 506901 top level 5 path @-snapshot-2016051701 ID 922 gen 507313 top level 5 path @-snapshot-2016051802 ID 923 gen 507712 top level 5 path @-snapshot-2016051901 ID 924 gen 508057 top level 5 path @-snapshot-2016052001 ID 925 gen 508882 top level 5 path @-snapshot-2016052101 ID 926 gen 509241 top level 5 path @-snapshot-2016052201 ID 927 gen 509618 top level 5 path @-snapshot-2016052301 ID 928 gen 510277 top level 5 path @-snapshot-2016052402 ID 929 gen 511357 top level 5 path @-snapshot-2016052502 ID 930 gen 512125 top level 5 path @-snapshot-2016052602 ID 931 gen 513292 top level 5 path @-snapshot-2016052701 ID 932 gen 515766 top level 5 path @-snapshot-2016052802 ID 933 gen 517349 top level 5 path @-snapshot-2016052904 ID 934 gen 519004 top level 5 path @-snapshot-2016053002 ID 935 gen 519500 top level 5 path @-snapshot-2016053102 ID 936 gen 519847 top level 5 path @-snapshot-2016060101 ID 937 gen 521829 top level 5 path @-snapshot-2016060201 Removing old snapshots is the most feasible solution, but I can also increase the FS size. It's easy since it's in LVM, and there is plenty of space in the volume group. Probably I should rewrite my alert script to check btrfs fi show instead of plain df. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
On 2016-06-01 14:30, MegaBrutal wrote: Hi all, I have a 20 GB file system and df says I have about 2,6 GB free space, yet I can't do anything on the file system because I get "No space left on device" errors. I read that balance may help to remedy the situation, but it actually doesn't. Some data about the FS: root@ReThinkCentre:~# df -h / FájlrendszerMéret Fogl. Szab. Fo.% Csatol. pont /dev/mapper/centrevg-rootlv 20G 18G 2,6G 88% / root@ReThinkCentre:~# btrfs fi show / Label: 'RootFS' uuid: 3f002b8d-8a1f-41df-ad05-e3c91d7603fb Total devices 1 FS bytes used 15.42GiB devid1 size 20.00GiB used 20.00GiB path /dev/mapper/centrevg-rootlv root@ReThinkCentre:~# btrfs fi df / Data, single: total=16.69GiB, used=14.14GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.62GiB, used=1.28GiB GlobalReserve, single: total=352.00MiB, used=0.00B root@ReThinkCentre:~# btrfs version btrfs-progs v4.4 This happens when I try to balance: root@ReThinkCentre:~# btrfs fi balance start -dusage=66 / Done, had to relocate 0 out of 33 chunks root@ReThinkCentre:~# btrfs fi balance start -dusage=67 / ERROR: error during balancing '/': No space left on device There may be more info in syslog - try dmesg | tail "dmesg | tail" does not show anything related to this. It is important to note that the file system currently has 32 snapshots of / at the moment, and snapshots taking up all the free space is a plausible explanation. Maybe deleting some of the oldest snapshots or just increasing the file system would help the situation. However, I'm still interested, if the file system is full, why does df show there is free space, and how could I show the situation without having the mentioned options? I actually have an alert set up which triggers when the FS usage reaches 90%, so then I know I have to delete some old snapshots. It worked so far, I cleaned the snapshots at 90%, FS usage fell back, everyone was happy. But now the alert didn't even trigger because the FS is at 88% usage, so it shouldn't be full yet. The first thing that needs to be understood is that df has been pretty much unchanged since it was introduced in the 70's (IIRC, it was in at least SVR4, possibly earlier UNIX versions too). Back then, it was pretty easy to say what percentage of space was used and how much is left. Back then, a filesystem only allocated one set of blocks for a file, and it didn't need extra space for updates, and the file took up exactly as much space as it's size on disk (usually, it can get kind of complicated based on a number of factors). In addition, traditional UFS had a fixed size metadata area for the inodes, which simplified computations even more. In BTRFS though, almost all of these assumptions which the original interface made aren't guaranteed. Now, the biggest difference though is in how BTRFS allocates space. BTRFS uses a two tier allocation system. First, you have high-level allocations of what are usually referred to as chunks, and then it allocates blocks within those chunks. The balance operation operates at the chunk level, whereas things like defragmentation operate at the block level. For performance reasons, BTRFS usually has separate chunks for metadata and data. Data chunks are usually 1GB, and metadata chunks are usually 256MB, although both can vary in size based on the size of the filesystem. Figuring out the exact size gets tricky on a live filesystem, but if your filesystem is between 16G and 64G, you're pretty much guaranteed to have chunks which are the default size. Now, because of the segregation of data and metadata, and how chunk allocation works, it's possible to end up in a situation where you technically have free space, but you can't actually do anything with it. This is because most file operations on BTRFS require at least a few blocks of metadata space so that the COW updates can happen. You luckily don't appear to be quite to that point. For compatibility reasons, we have to report _something_ through df. We can't however report many of the situational things about the state of the FS itself (for example, if you have all the possible chunks allocated, no space in data chunks, but free space in metadata chunks, it's possible to create a lot of very small files, but creating a big one will fail). As a result of this, what we report through df is technically absolutely correct (in your case, you _do_ technically have 2.6G of free space), but is also absolutely useless for any kind of management decision. In your particular situation, what's happened is that you have all the space allocated to chunks, but have free space within those chunks. Balance never puts data in existing chunks, and you can't allocate any new chunks, so you can't run a balance. However, because of that free space in the chunks, you can still use the filesystem itself for 'regular' filesystem operations.
Re: "No space left on device" and balance doesn't work
On Wed, Jun 1, 2016 at 11:06 PM, MegaBrutal wrote: > Hi Peter, > > I tried. I either get "Done, had to relocate 0 out of 33 chunks" or > "ERROR: error during balancing '/': No space left on device", and > nothing changes. > > > 2016-06-01 22:29 GMT+02:00 Peter Becker : >> try this: >> >> btrfs fi balance start -musage=0 / >> btrfs fi balance start -dusage=0 / >> >> btrfs fi balance start -musage=1 / >> btrfs fi balance start -dusage=1 / >> >> btrfs fi balance start -musage=5 / >> btrfs fi balance start -musage=10 / >> btrfs fi balance start -musage=20 / >> >> >> btrfs fi balance start -dusage=5 / >> btrfs fi balance start -dusage=10 / >> btrfs fi balance start -dusage=20 / >> >> >> 2016-06-01 20:30 GMT+02:00 MegaBrutal : >>> Hi all, >>> >>> I have a 20 GB file system and df says I have about 2,6 GB free space, >>> yet I can't do anything on the file system because I get "No space >>> left on device" errors. I read that balance may help to remedy the >>> situation, but it actually doesn't. >>> >>> >>> Some data about the FS: >>> >>> >>> root@ReThinkCentre:~# df -h / >>> FájlrendszerMéret Fogl. Szab. Fo.% Csatol. pont >>> /dev/mapper/centrevg-rootlv 20G 18G 2,6G 88% / >>> >>> root@ReThinkCentre:~# btrfs fi show / >>> Label: 'RootFS' uuid: 3f002b8d-8a1f-41df-ad05-e3c91d7603fb >>> Total devices 1 FS bytes used 15.42GiB >>> devid1 size 20.00GiB used 20.00GiB path >>> /dev/mapper/centrevg-rootlv The device is completely filled with chunks (size and used are the same) and none of the chunks is empty so a balance won't work at all. A way to get out of this situation is to add a temporary extra device (e.g. 8GB USB stick or a loop device on some larger USB disk) to the fs and then do the various balance operations. Removing as much as possible snapshots will ease the balance mostly, depending how old the snapshots are. Once you see that the total amount of space used by chunks is (a few GiB) less then 19GiB, you can remove the temporary extra device from the fs again. It then is still possible that you run into the same situation again; This is a longterm bug/problem. A brand new kernel might help, with ENOSPC patches included. What is the kernel version used? Is the fs on a mechanical disk or SSD? What are the mount options? How old is the fs? You might want to run this phython script, so you get an idea of what the chunks fill-level is https://github.com/knorrie/btrfs-heatmap/blob/master/show_usage.py Also you could mount the fs with enospc_debug, and see what is reported in dmesg >>> root@ReThinkCentre:~# btrfs fi df / >>> Data, single: total=16.69GiB, used=14.14GiB >>> System, DUP: total=32.00MiB, used=16.00KiB >>> Metadata, DUP: total=1.62GiB, used=1.28GiB >>> GlobalReserve, single: total=352.00MiB, used=0.00B >>> >>> root@ReThinkCentre:~# btrfs version >>> btrfs-progs v4.4 >>> >>> >>> This happens when I try to balance: >>> >>> root@ReThinkCentre:~# btrfs fi balance start -dusage=66 / >>> Done, had to relocate 0 out of 33 chunks >>> root@ReThinkCentre:~# btrfs fi balance start -dusage=67 / >>> ERROR: error during balancing '/': No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> >>> "dmesg | tail" does not show anything related to this. >>> >>> It is important to note that the file system currently has 32 >>> snapshots of / at the moment, and snapshots taking up all the free >>> space is a plausible explanation. Maybe deleting some of the oldest >>> snapshots or just increasing the file system would help the situation. >>> However, I'm still interested, if the file system is full, why does df >>> show there is free space, and how could I show the situation without >>> having the mentioned options? I actually have an alert set up which >>> triggers when the FS usage reaches 90%, so then I know I have to >>> delete some old snapshots. It worked so far, I cleaned the snapshots >>> at 90%, FS usage fell back, everyone was happy. But now the alert >>> didn't even trigger because the FS is at 88% usage, so it shouldn't be >>> full yet. >>> >>> >>> Best regards and kecske, >>> MegaBrutal >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majord...@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
Hi Peter, I tried. I either get "Done, had to relocate 0 out of 33 chunks" or "ERROR: error during balancing '/': No space left on device", and nothing changes. 2016-06-01 22:29 GMT+02:00 Peter Becker : > try this: > > btrfs fi balance start -musage=0 / > btrfs fi balance start -dusage=0 / > > btrfs fi balance start -musage=1 / > btrfs fi balance start -dusage=1 / > > btrfs fi balance start -musage=5 / > btrfs fi balance start -musage=10 / > btrfs fi balance start -musage=20 / > > > btrfs fi balance start -dusage=5 / > btrfs fi balance start -dusage=10 / > btrfs fi balance start -dusage=20 / > > > 2016-06-01 20:30 GMT+02:00 MegaBrutal : >> Hi all, >> >> I have a 20 GB file system and df says I have about 2,6 GB free space, >> yet I can't do anything on the file system because I get "No space >> left on device" errors. I read that balance may help to remedy the >> situation, but it actually doesn't. >> >> >> Some data about the FS: >> >> >> root@ReThinkCentre:~# df -h / >> FájlrendszerMéret Fogl. Szab. Fo.% Csatol. pont >> /dev/mapper/centrevg-rootlv 20G 18G 2,6G 88% / >> >> root@ReThinkCentre:~# btrfs fi show / >> Label: 'RootFS' uuid: 3f002b8d-8a1f-41df-ad05-e3c91d7603fb >> Total devices 1 FS bytes used 15.42GiB >> devid1 size 20.00GiB used 20.00GiB path >> /dev/mapper/centrevg-rootlv >> >> root@ReThinkCentre:~# btrfs fi df / >> Data, single: total=16.69GiB, used=14.14GiB >> System, DUP: total=32.00MiB, used=16.00KiB >> Metadata, DUP: total=1.62GiB, used=1.28GiB >> GlobalReserve, single: total=352.00MiB, used=0.00B >> >> root@ReThinkCentre:~# btrfs version >> btrfs-progs v4.4 >> >> >> This happens when I try to balance: >> >> root@ReThinkCentre:~# btrfs fi balance start -dusage=66 / >> Done, had to relocate 0 out of 33 chunks >> root@ReThinkCentre:~# btrfs fi balance start -dusage=67 / >> ERROR: error during balancing '/': No space left on device >> There may be more info in syslog - try dmesg | tail >> >> >> "dmesg | tail" does not show anything related to this. >> >> It is important to note that the file system currently has 32 >> snapshots of / at the moment, and snapshots taking up all the free >> space is a plausible explanation. Maybe deleting some of the oldest >> snapshots or just increasing the file system would help the situation. >> However, I'm still interested, if the file system is full, why does df >> show there is free space, and how could I show the situation without >> having the mentioned options? I actually have an alert set up which >> triggers when the FS usage reaches 90%, so then I know I have to >> delete some old snapshots. It worked so far, I cleaned the snapshots >> at 90%, FS usage fell back, everyone was happy. But now the alert >> didn't even trigger because the FS is at 88% usage, so it shouldn't be >> full yet. >> >> >> Best regards and kecske, >> MegaBrutal >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" and balance doesn't work
try this: btrfs fi balance start -musage=0 / btrfs fi balance start -dusage=0 / btrfs fi balance start -musage=1 / btrfs fi balance start -dusage=1 / btrfs fi balance start -musage=5 / btrfs fi balance start -musage=10 / btrfs fi balance start -musage=20 / btrfs fi balance start -dusage=5 / btrfs fi balance start -dusage=10 / btrfs fi balance start -dusage=20 / 2016-06-01 20:30 GMT+02:00 MegaBrutal : > Hi all, > > I have a 20 GB file system and df says I have about 2,6 GB free space, > yet I can't do anything on the file system because I get "No space > left on device" errors. I read that balance may help to remedy the > situation, but it actually doesn't. > > > Some data about the FS: > > > root@ReThinkCentre:~# df -h / > FájlrendszerMéret Fogl. Szab. Fo.% Csatol. pont > /dev/mapper/centrevg-rootlv 20G 18G 2,6G 88% / > > root@ReThinkCentre:~# btrfs fi show / > Label: 'RootFS' uuid: 3f002b8d-8a1f-41df-ad05-e3c91d7603fb > Total devices 1 FS bytes used 15.42GiB > devid1 size 20.00GiB used 20.00GiB path > /dev/mapper/centrevg-rootlv > > root@ReThinkCentre:~# btrfs fi df / > Data, single: total=16.69GiB, used=14.14GiB > System, DUP: total=32.00MiB, used=16.00KiB > Metadata, DUP: total=1.62GiB, used=1.28GiB > GlobalReserve, single: total=352.00MiB, used=0.00B > > root@ReThinkCentre:~# btrfs version > btrfs-progs v4.4 > > > This happens when I try to balance: > > root@ReThinkCentre:~# btrfs fi balance start -dusage=66 / > Done, had to relocate 0 out of 33 chunks > root@ReThinkCentre:~# btrfs fi balance start -dusage=67 / > ERROR: error during balancing '/': No space left on device > There may be more info in syslog - try dmesg | tail > > > "dmesg | tail" does not show anything related to this. > > It is important to note that the file system currently has 32 > snapshots of / at the moment, and snapshots taking up all the free > space is a plausible explanation. Maybe deleting some of the oldest > snapshots or just increasing the file system would help the situation. > However, I'm still interested, if the file system is full, why does df > show there is free space, and how could I show the situation without > having the mentioned options? I actually have an alert set up which > triggers when the FS usage reaches 90%, so then I know I have to > delete some old snapshots. It worked so far, I cleaned the snapshots > at 90%, FS usage fell back, everyone was happy. But now the alert > didn't even trigger because the FS is at 88% usage, so it shouldn't be > full yet. > > > Best regards and kecske, > MegaBrutal > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" during retroactive compression with btrfs filesystem defragment
Thank you too for the enlightenment. Not just now but so many times in the past (just the compilation of your list interventions is a wiki in its own right). Me too, I've been meaning to create a wiki account for quite some time (but I was partly intimidated by the formality of the request :-) )... Finally, with your encouragement, I did create one as soon as I got back from work and rushed to my first edit/correction. Only to discover that someone else got there first! Speed of light. This is what will always leave me with my mouth open in vibrant FOSS communities... -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" during retroactive compression with btrfs filesystem defragment
George Eleftheriou posted on Mon, 07 Apr 2014 12:34:27 +0200 as excerpted: > Browsing the btrfs wiki for a relevant warning I just found this one: > > Caveat: Before Linux 3.9, which adds snapshot-aware defragmentation, > defragmenting a file which had a COW copy (either a snapshot copy or one > made with cp --reflink or bcp) would produce two unrelated files. If you > defragmented a subvolume that had a snapshot, you would roughly double > the disk usage, as the snapshot files were no longer COW images of the > originals. > > Which comes close but not quite, since the issue happened during > compression and not plain defragmentation and since the kernel I was > running was 3.13.8 Unfortunately, that bit of the wiki isn't current. 3.14 disabled snapshot-aware defrag once again and I believe the disabling was pulled into stable as well, as the algorithm used simply didn't scale well at all, and people with lots of snapshots (as with snapper) were seeing defrag go hours, even days, with little apparent progress at all! So it was disabled, allowing people to at least have a /somewhat/ useable defrag on the currently mounted snapshot, even if it was at the expense of increasing space usage due to being snapshot unaware once again. The idea was to rewrite the algorithm to something that scaled rather better, and then reenable snapshot-awareness, but I've seen nothing posted as to the progress on that. If you have a kernel git tree, it's... commit 8101c8dbf6243ba517aab58d69bf1bc37d8b7b9c , merged in commit 878a876b2e10888afe53766dcca33f723ae20edc , which is described as v3.14-rc1-13-g878a876, so just after 3.14-rc1. So anyway, that's very likely what you were seeing, not only the compression thing, which might or might not work that way as well. Meanwhile, however, your post did help me make the connection. Someone else had posted similar results, tho I think without compression involved but I *DO* know he had lots of snapshots, and I didn't make the logical connection with snapshot unaware defrag there, so the fact that he had to delete the snapshots to do the defrag was there but unexplained. But your post prodded my thinking and now I realize what happened there as well. Thanks. =:^) I've been meaning to get myself a wiki account so I can fix such things myself, but I tend to work much better in the familiar environment of lists and newsgroups and I've not done so, so as of now the task of editing the wiki for such updates must fall to others. I'm sure others reading the wiki would be grateful if you took the time to do that update, now that you have the knowledge to do so. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
On Thu, Feb 27, 2014 at 02:16:07AM +0200, Marcus Sundman wrote: > On 25.02.2014 22:30, Josef Bacik wrote: > >On 02/25/2014 03:27 PM, Marcus Sundman wrote: > >>On 25.02.2014 22:19, Hugo Mills wrote: > >>>On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: > 370GB of 410GB used isn't really "fine", it's over 90% usage. > > That said, I'd be interested to know why btrfs fi show > /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G > used... > >>>This is an FAQ... > >>> > >>>btrfs fi show tells you how much is allocated out of the > >>>available pool on each disk. btrfs fi df then shows how much of > >>>that allocated space (in each category) is used. > >>What is the difference between the "used 371.11GB" and the "used > >>412.54GB" displayed by "btrfs fi show"? > >> > >>>The problem here is also in the FAQ: the metadata is close to > >>>full -- typically something like 500-750 MiB of headroom is > >>>needed in metadata. The FS can't allocate more metadata because > >>>it's allocated everything already (total=used in btrfs fi show), > >>>so the solution is to do a filtered balance: > >>> > >>>btrfs balance start -dusage=5 /mountpoint > >>Of course that was the first thing I tried, and it didn't help *at* > >>*all*: > >> > >The -dusage= is a means to an end, so if that doesn't work try > >a larger number, up to 100. Really once you pass 50 and it's not > >working then it's time to just do a balance. The next thing is to use > >compression (too late for this option really) or add another disk. > > So it relocates some chunks. What will that do? Does it mean I can > now use the remaining 45 GB? Or will it run out of "disk space" > again after using a gig or two? What it's done is moved some of the data around. One effect of this process is that when a balance operation balances a chunk, it deallocates that space. Therefore you should now see that the "used" value for the device in btrfs fi show is lower than the "total" value. Also, the "total" value for data in btrfs fi df should now be lower than it was before. This means that the next time the FS needs some metadata space (probably real soon now, because it was running out of it earlier), it now has uncommitted space to allocate to metadata. > If it's the allocated metadata space that is the problem then how > can I pre-allocate more of it so it won't run out of it? It's possible that when filling up the remaining 45 GiB of space the metadata will need to expand again, and you will need to run a balance to do the same job again. Unless you have lots and lots of small files or small fragments (or lots more snapshots than you've been making to date), I'd say it's probably unlikely, though. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Well, sir, the floor is yours. But remember, the --- roof is ours! signature.asc Description: Digital signature
Re: No space left on device (again)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/26/2014 07:16 PM, Marcus Sundman wrote: > On 25.02.2014 22:30, Josef Bacik wrote: >> On 02/25/2014 03:27 PM, Marcus Sundman wrote: >>> On 25.02.2014 22:19, Hugo Mills wrote: On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: > 370GB of 410GB used isn't really "fine", it's over 90% > usage. > > That said, I'd be interested to know why btrfs fi show > /dev/sda3 shows 412.54G used, but btrfs fi df /home shows > 379G used... This is an FAQ... btrfs fi show tells you how much is allocated out of the available pool on each disk. btrfs fi df then shows how much of that allocated space (in each category) is used. >>> What is the difference between the "used 371.11GB" and the >>> "used 412.54GB" displayed by "btrfs fi show"? >>> The problem here is also in the FAQ: the metadata is close to full -- typically something like 500-750 MiB of headroom is needed in metadata. The FS can't allocate more metadata because it's allocated everything already (total=used in btrfs fi show), so the solution is to do a filtered balance: btrfs balance start -dusage=5 /mountpoint >>> Of course that was the first thing I tried, and it didn't help >>> *at* *all*: >>> >> The -dusage= is a means to an end, so if that doesn't >> work try a larger number, up to 100. Really once you pass 50 and >> it's not working then it's time to just do a balance. The next >> thing is to use compression (too late for this option really) or >> add another disk. > > So it relocates some chunks. What will that do? Does it mean I can > now use the remaining 45 GB? Or will it run out of "disk space" > again after using a gig or two? If it's the allocated metadata > space that is the problem then how can I pre-allocate more of it so > it won't run out of it? > > It will allocate more as it sees fit, there is a metadata_ratio=N option which lets you force it to allocate a metadata chunk for every N data chunk allocations, but that shouldn't be needed in this case. What does btrfs fi df and btrfs show look like now? Thanks, Josef -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJTDpIMAAoJEANb+wAKly3BfKUQAK/WzvL/SENlUhruUoL7ADid tM6ivq2BZOwmlw+CzOIzjrwv9gwRe+UAfjKjGstueLoVY+lHNYaCC5OI94OOWDPo SyLJdJhcA3FYYBlyz7EC6gG4fNFyJg6egshoyMmWJuny5ivfnltdQhLLz2IQIHek Ud9ElN+SMvCeQmGZxdKg5yc2oIgBdc5xGfegtfuCqFkdhu+BZTbcXFsOD4Pjnsg0 Mtw8H5YBeXcFBV34I8F6l+O3AGjDl8jF/cFuCEbRTJQFkvpKhoHMw+O2AWykDQqc xw3H1YghGeY1fqN2geyYSYVGVOGxWeO2kju7Itom8Ph5AinMPMf8lpe97nChp2hr D3T1QwhkmMD9T1O02hvF9C8E46q2iyjOrPxPU8z3LsRTKaNXNzJ+u1P2ac7kIQSk x2stB9u0Qluut+5twzLQuefzoCNf/2RtAlL2cPXyq6ikLHBSNkebqbGBD2IpTsa7 5TsXNHWbIc0maDdXrJ0BYJ7obEaU/2nBCkSA3DaGBupTtC3vwSWAoAEw8/JtVeQr ARXSIPblXZqwrGyJyvtNmC8zjDGAD93H/xr0+oCOzHqPIr7NvTT9vZKfENmblsfm vx8wmgbSBzWp9W9YpUO5X0ZjLestPzOziIesCrMJ2yKBUEUtafiFdbT5D7e9dn8x 7Z2MZcYyk5H7QF65+IWt =8iSO -END PGP SIGNATURE- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
On 25.02.2014 22:30, Josef Bacik wrote: On 02/25/2014 03:27 PM, Marcus Sundman wrote: On 25.02.2014 22:19, Hugo Mills wrote: On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: 370GB of 410GB used isn't really "fine", it's over 90% usage. That said, I'd be interested to know why btrfs fi show /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G used... This is an FAQ... btrfs fi show tells you how much is allocated out of the available pool on each disk. btrfs fi df then shows how much of that allocated space (in each category) is used. What is the difference between the "used 371.11GB" and the "used 412.54GB" displayed by "btrfs fi show"? The problem here is also in the FAQ: the metadata is close to full -- typically something like 500-750 MiB of headroom is needed in metadata. The FS can't allocate more metadata because it's allocated everything already (total=used in btrfs fi show), so the solution is to do a filtered balance: btrfs balance start -dusage=5 /mountpoint Of course that was the first thing I tried, and it didn't help *at* *all*: The -dusage= is a means to an end, so if that doesn't work try a larger number, up to 100. Really once you pass 50 and it's not working then it's time to just do a balance. The next thing is to use compression (too late for this option really) or add another disk. So it relocates some chunks. What will that do? Does it mean I can now use the remaining 45 GB? Or will it run out of "disk space" again after using a gig or two? If it's the allocated metadata space that is the problem then how can I pre-allocate more of it so it won't run out of it? - Marcus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
Josef Bacik wrote (ao): > The -dusage= is a means to an end, so if that doesn't work try > a larger number, up to 100. Really once you pass 50 and it's not > working then it's time to just do a balance. The next thing is to use > compression (too late for this option really) or add another disk. btrfs filesystem defragment -vrc /home/ would apply compression to all files, no? Sander -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
On Tue, Feb 25, 2014 at 10:27:58PM +0200, Marcus Sundman wrote: > On 25.02.2014 22:19, Hugo Mills wrote: > >On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: > >>370GB of 410GB used isn't really "fine", it's over 90% usage. > >> > >>That said, I'd be interested to know why btrfs fi show /dev/sda3 > >>shows 412.54G used, but btrfs fi df /home shows 379G used... > >This is an FAQ... > > > >btrfs fi show tells you how much is allocated out of the available > >pool on each disk. btrfs fi df then shows how much of that allocated > >space (in each category) is used. > > What is the difference between the "used 371.11GB" and the "used > 412.54GB" displayed by "btrfs fi show"? The first and smaller value is (in this case) the sum of all the "used" values in the output of btrfs fi df. The second is the total amount of raw disk space that has been allocated. (Yes, this is a bit of a mess -- there's been discussions recently about improving the output). > >The problem here is also in the FAQ: the metadata is close to full > >-- typically something like 500-750 MiB of headroom is needed in > >metadata. The FS can't allocate more metadata because it's allocated > >everything already (total=used in btrfs fi show), so the solution is > >to do a filtered balance: > > > >btrfs balance start -dusage=5 /mountpoint > > Of course that was the first thing I tried, and it didn't help *at* *all*: > > ># btrfs filesystem balance start -dusage=5 /home > >Done, had to relocate 0 out of 415 chunks > ># > > ... and it really didn't free anything. OK, so crank up the number to 10 and try again. If that still moves nothing, then increase again until it frees up something. With an FS at 90% full or so, particularly if it's seen heavy use, the chances are reasonable that all of the chunks are over 5% full, so putting the value up will eventually get to the point of moving at least one chunk, which should be all that's needed at this point. Hugo. > >>On 02/25/2014 11:49 AM, Marcus Sundman wrote: > >>>Hi > >>> > >>>I get "No space left on device" and it is unclear why: > >>> > # df -h|grep sda3 > /dev/sda3 413G 368G 45G 90% /home > # btrfs filesystem show /dev/sda3 > Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60 > Total devices 1 FS bytes used 371.11GB > devid1 size 412.54GB used 412.54GB path /dev/sda3 > > Btrfs v0.20-rc1 > # btrfs filesystem df /home > Data: total=410.52GB, used=369.61GB > System: total=4.00MB, used=64.00KB > Metadata: total=2.01GB, used=1.50GB > # > >>>So, 'data' and 'metadata' seem to be fine(?), but 'system' is a > >>>bit low. Is that it? If so, can I do something about it? Or should > >>>I look somewhere else? > >>> > >>>I really wish I could get a warning before running out of disk > >>>space, instead of everything breaking suddenly when there seems to > >>>be lots and lots of space left. > >>> > >>>- Marcus > >>> > -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Jazz is the sort of music where no-one plays anything the --- same way once. signature.asc Description: Digital signature
Re: No space left on device (again)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/25/2014 03:27 PM, Marcus Sundman wrote: > On 25.02.2014 22:19, Hugo Mills wrote: >> On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: >>> 370GB of 410GB used isn't really "fine", it's over 90% usage. >>> >>> That said, I'd be interested to know why btrfs fi show >>> /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G >>> used... >> This is an FAQ... >> >> btrfs fi show tells you how much is allocated out of the >> available pool on each disk. btrfs fi df then shows how much of >> that allocated space (in each category) is used. > > What is the difference between the "used 371.11GB" and the "used > 412.54GB" displayed by "btrfs fi show"? > >> The problem here is also in the FAQ: the metadata is close to >> full -- typically something like 500-750 MiB of headroom is >> needed in metadata. The FS can't allocate more metadata because >> it's allocated everything already (total=used in btrfs fi show), >> so the solution is to do a filtered balance: >> >> btrfs balance start -dusage=5 /mountpoint > > Of course that was the first thing I tried, and it didn't help *at* > *all*: > The -dusage= is a means to an end, so if that doesn't work try a larger number, up to 100. Really once you pass 50 and it's not working then it's time to just do a balance. The next thing is to use compression (too late for this option really) or add another disk. Thanks, Josef -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJTDP1mAAoJEANb+wAKly3BQooQAMjRfSc0fJcrVFhNqAXvuxKO f9ALYcENk7mVEMiyPg9v/7oiHW+FSQuJxaJ+yPlLTvepFmh3VE00t4PHsTCheAld KdIoP9KzvCrmPSH5YxBg9Tmk60lonYnGfCsggK3kgRP8JIHUcBRKzlFKAc3p8vzY UEEYLS0AYXvBAPdY3bUwhB5txMDvNbetew8ak5nUgZaC1J4rWe7LL6ccv845hhf5 OHcKLu3MjR6ejW9qDK9RC2ryjCY+vDCUMO4VPpma3OZ6OHA3g7niaXYqBEvht7O0 yNeGQA54O1goIFo9knPG4XE7WurlF69n+knQA7dsqjux9bIXHVClCCwsrUpXJIY3 98KigtPVz737j1dNqaeQdn6xl6kwRpj2lRhWuPBZ1AnJNWr1cLK2jmWOuFmyrdi8 vTNJJvUlpwGnnUHdXu8WjMCSNgXX9pIXDZIC/f22wjZFyY23yfSYOTCXTEOlpH8b b4R7VnZ6QfxGgJnqhHPWBE7FRaekI2MLfT6aYVL2sNvVPZoMo0wzjnxshwp2+8dK C0j2sNsoRdKPlGECYuh2eG40JQG2JKnZr2bMSwDI40GvDcQmyG7gD+jouOtei/vv KUDDfhNGnv0SdqmAk/UC4Cl1dDD9Y4Z0kAuAd7iTJhAcLszteeGqiDvdvtup5o7i HZcFmZX6EPBZXKrIcnzJ =+Oe0 -END PGP SIGNATURE- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
Try btrfs filesystem balance start -dusage=15 /home, and gradually increase it until you see it relocate at least one chunk. On Tue, Feb 25, 2014 at 2:27 PM, Marcus Sundman wrote: > On 25.02.2014 22:19, Hugo Mills wrote: >> >> On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: >>> >>> 370GB of 410GB used isn't really "fine", it's over 90% usage. >>> >>> That said, I'd be interested to know why btrfs fi show /dev/sda3 >>> shows 412.54G used, but btrfs fi df /home shows 379G used... >> >> This is an FAQ... >> >> btrfs fi show tells you how much is allocated out of the available >> pool on each disk. btrfs fi df then shows how much of that allocated >> space (in each category) is used. > > > What is the difference between the "used 371.11GB" and the "used 412.54GB" > displayed by "btrfs fi show"? > > >> The problem here is also in the FAQ: the metadata is close to full >> -- typically something like 500-750 MiB of headroom is needed in >> metadata. The FS can't allocate more metadata because it's allocated >> everything already (total=used in btrfs fi show), so the solution is >> to do a filtered balance: >> >> btrfs balance start -dusage=5 /mountpoint > > > Of course that was the first thing I tried, and it didn't help *at* *all*: > >> # btrfs filesystem balance start -dusage=5 /home >> Done, had to relocate 0 out of 415 chunks >> # > > > ... and it really didn't free anything. > > >>> On 02/25/2014 11:49 AM, Marcus Sundman wrote: Hi I get "No space left on device" and it is unclear why: > # df -h|grep sda3 > /dev/sda3 413G 368G 45G 90% /home > # btrfs filesystem show /dev/sda3 > Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60 > Total devices 1 FS bytes used 371.11GB > devid1 size 412.54GB used 412.54GB path /dev/sda3 > > Btrfs v0.20-rc1 > # btrfs filesystem df /home > Data: total=410.52GB, used=369.61GB > System: total=4.00MB, used=64.00KB > Metadata: total=2.01GB, used=1.50GB > # So, 'data' and 'metadata' seem to be fine(?), but 'system' is a bit low. Is that it? If so, can I do something about it? Or should I look somewhere else? I really wish I could get a warning before running out of disk space, instead of everything breaking suddenly when there seems to be lots and lots of space left. - Marcus > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
On 25.02.2014 22:19, Hugo Mills wrote: On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: 370GB of 410GB used isn't really "fine", it's over 90% usage. That said, I'd be interested to know why btrfs fi show /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G used... This is an FAQ... btrfs fi show tells you how much is allocated out of the available pool on each disk. btrfs fi df then shows how much of that allocated space (in each category) is used. What is the difference between the "used 371.11GB" and the "used 412.54GB" displayed by "btrfs fi show"? The problem here is also in the FAQ: the metadata is close to full -- typically something like 500-750 MiB of headroom is needed in metadata. The FS can't allocate more metadata because it's allocated everything already (total=used in btrfs fi show), so the solution is to do a filtered balance: btrfs balance start -dusage=5 /mountpoint Of course that was the first thing I tried, and it didn't help *at* *all*: # btrfs filesystem balance start -dusage=5 /home Done, had to relocate 0 out of 415 chunks # ... and it really didn't free anything. On 02/25/2014 11:49 AM, Marcus Sundman wrote: Hi I get "No space left on device" and it is unclear why: # df -h|grep sda3 /dev/sda3 413G 368G 45G 90% /home # btrfs filesystem show /dev/sda3 Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60 Total devices 1 FS bytes used 371.11GB devid1 size 412.54GB used 412.54GB path /dev/sda3 Btrfs v0.20-rc1 # btrfs filesystem df /home Data: total=410.52GB, used=369.61GB System: total=4.00MB, used=64.00KB Metadata: total=2.01GB, used=1.50GB # So, 'data' and 'metadata' seem to be fine(?), but 'system' is a bit low. Is that it? If so, can I do something about it? Or should I look somewhere else? I really wish I could get a warning before running out of disk space, instead of everything breaking suddenly when there seems to be lots and lots of space left. - Marcus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote: > 370GB of 410GB used isn't really "fine", it's over 90% usage. > > That said, I'd be interested to know why btrfs fi show /dev/sda3 > shows 412.54G used, but btrfs fi df /home shows 379G used... This is an FAQ... btrfs fi show tells you how much is allocated out of the available pool on each disk. btrfs fi df then shows how much of that allocated space (in each category) is used. The problem here is also in the FAQ: the metadata is close to full -- typically something like 500-750 MiB of headroom is needed in metadata. The FS can't allocate more metadata because it's allocated everything already (total=used in btrfs fi show), so the solution is to do a filtered balance: btrfs balance start -dusage=5 /mountpoint Hugo. > > On 02/25/2014 11:49 AM, Marcus Sundman wrote: > >Hi > > > >I get "No space left on device" and it is unclear why: > > > >># df -h|grep sda3 > >>/dev/sda3 413G 368G 45G 90% /home > >># btrfs filesystem show /dev/sda3 > >>Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60 > >>Total devices 1 FS bytes used 371.11GB > >>devid1 size 412.54GB used 412.54GB path /dev/sda3 > >> > >>Btrfs v0.20-rc1 > >># btrfs filesystem df /home > >>Data: total=410.52GB, used=369.61GB > >>System: total=4.00MB, used=64.00KB > >>Metadata: total=2.01GB, used=1.50GB > >># > > > >So, 'data' and 'metadata' seem to be fine(?), but 'system' is a > >bit low. Is that it? If so, can I do something about it? Or should > >I look somewhere else? > > > >I really wish I could get a warning before running out of disk > >space, instead of everything breaking suddenly when there seems to > >be lots and lots of space left. > > > >- Marcus > > -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Jazz is the sort of music where no-one plays anything the --- same way once. signature.asc Description: Digital signature
Re: No space left on device (again)
On 25.02.2014 20:05, Jim Salter wrote: 370GB of 410GB used isn't really "fine", it's over 90% usage. Still, 45 gigs should be free there. If those 45 gigs aren't really there then it shouldn't say they are there, imho. That said, I'd be interested to know why btrfs fi show /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G used... The former also says "bytes used 371.11GB" and there's not *that* much difference between "371.11" and "369.61" gigs. The devid line seems to always say "412.54GB used 412.54GB" no matter how much is actually in use. I have no idea what these numbers actually mean. On 02/25/2014 11:49 AM, Marcus Sundman wrote: Hi I get "No space left on device" and it is unclear why: # df -h|grep sda3 /dev/sda3 413G 368G 45G 90% /home # btrfs filesystem show /dev/sda3 Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60 Total devices 1 FS bytes used 371.11GB devid1 size 412.54GB used 412.54GB path /dev/sda3 Btrfs v0.20-rc1 # btrfs filesystem df /home Data: total=410.52GB, used=369.61GB System: total=4.00MB, used=64.00KB Metadata: total=2.01GB, used=1.50GB # So, 'data' and 'metadata' seem to be fine(?), but 'system' is a bit low. Is that it? If so, can I do something about it? Or should I look somewhere else? I really wish I could get a warning before running out of disk space, instead of everything breaking suddenly when there seems to be lots and lots of space left. - Marcus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (again)
370GB of 410GB used isn't really "fine", it's over 90% usage. That said, I'd be interested to know why btrfs fi show /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G used... On 02/25/2014 11:49 AM, Marcus Sundman wrote: Hi I get "No space left on device" and it is unclear why: # df -h|grep sda3 /dev/sda3 413G 368G 45G 90% /home # btrfs filesystem show /dev/sda3 Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60 Total devices 1 FS bytes used 371.11GB devid1 size 412.54GB used 412.54GB path /dev/sda3 Btrfs v0.20-rc1 # btrfs filesystem df /home Data: total=410.52GB, used=369.61GB System: total=4.00MB, used=64.00KB Metadata: total=2.01GB, used=1.50GB # So, 'data' and 'metadata' seem to be fine(?), but 'system' is a bit low. Is that it? If so, can I do something about it? Or should I look somewhere else? I really wish I could get a warning before running out of disk space, instead of everything breaking suddenly when there seems to be lots and lots of space left. - Marcus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device
On Wed, Feb 12, 2014 at 11:45:34AM +0100, Jakob Truelsen wrote: > Hi and thanks for the quick reply. Have remounted the filesystem with > enospc_debug, and run the rebalance you suggested, with the trance > below. So perhaps the next step is for me to figure out how to take a > metadata image and send it to josef (perhaps with a box of tissues) You can get the metadata image with: btrfs-image -c9 -t4 /dev/sda image-file-to-create It will probably be a couple of hundred megabytes in size (your metadata size, compressed). It will contain filenames and xattrs, so if those are sensitive, you may want to use -s to hide that information. After that, probably the best place to make sure it's not lost is to open an issue on bugzilla.kernel.org, making sure that you set the Component to btrfs. Hugo. > /Jakob > > > [jakobt@soda ~]$ mount > ... > /dev/sda on /data type btrfs (rw,relatime,nospace_cache,enospc_debug) > > [jakobt@soda ~]$ sudo btrfs balance start -dusage=0 /data > ERROR: error during balancing '/data' - No space left on device > There may be more info in syslog - try dmesg | tail > > [jakobt@soda ~]$ touch /data/jakobt/monkey > touch: cannot touch ‘/data/jakobt/monkey’: No space left on device > > [jakobt@soda ~]$ dmesg | tail -n3 > [1117530.870965] btrfs: device label Data devid 1 transid 47091 /dev/sda > [1117573.087580] btrfs: 426 enospc errors during balance > [1117642.002437] btrfs: 426 enospc errors during balance > On Wed, Feb 12, 2014 at 11:26 AM, Hugo Mills wrote: > > On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote: > >> Hello. I am experiencing "No space left on device" with a btrfs file > >> system, yet I cannot seem to find any exhausted resource. Could some > >> resource I do not know about be exhausted, or is this caused by > >> something else. Below is a trace of information that might be usefull, > >> please let me know if there is anything I can do to resolve the issue. > >> > >> /Jakob > >> > >> [jakobt@soda ~]$ uname -a > >> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014 > >> x86_64 GNU/Linux > > > >Were you using this kernel when the problem happened? > > > >> [jakobt@soda ~]$ btrfs --version > >> Btrfs v3.12 > >> > >> [jakobt@soda ~]$ mount > >> ... > >> /dev/sda on /data type btrfs (rw,relatime,nospace_cache) > >> > >> [jakobt@soda ~]$ df /data > >> Filesystem 1K-blocks Used Available Use% Mounted on > >> /dev/sdb2 76594224 49247368 23433028 68% / > >> > >> [jakobt@soda ~]$ btrfs filesystem df /data > >> Data, single: total=1.82TiB, used=518.04GiB > >> System, DUP: total=8.00MiB, used=204.00KiB > >> System, single: total=4.00MiB, used=0.00 > >> Metadata, DUP: total=1.00GiB, used=767.70MiB > > > >^^^ This is your problem, most likely in conjunction with all the > > space on the device being allocated. Being a copy-on-write filesystem, > > btrfs needs free space to make any update. If it doesn't have that > > free space, you get "No space left on device". You typically need > > somewhere around 0.5-1 GiB of headroom in metadata for normal > > operation, so I'm surprised you got this far. :) > > > >The FS should normally allocate more metadata space as it needs it, > > but because (I think) your data allocation has taken up all the > > available space on the device, there's no way for it to add more. > > > >> Metadata, single: total=8.00MiB, used=0.00 > > > >> [jakobt@soda ~]$ touch /data/jakobt/hat > >> touch: cannot touch ‘/data/jakobt/hat’: No space left on device > >> > >> [jakobt@soda ~]$ sudo btrfs fi balance start /data > >> ERROR: error during balancing '/data' - No space left on device > >> There may be more info in syslog - try dmesg | tail > > > >Try: > > > > btrfs balance start -dusage=0 /data > > > > which should go looking for entirely unused block groups and reclaim > > those. (If you don't use the -dusage= parameter, it will try to > > balance everything, which takes a long time). > > > >> [jakobt@soda ~]$ dmesg | grep tail -n 2 > >> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda > >> [1113507.641752] btrfs: 1866 enospc errors during balance > > > >Although, that said... it looks like it's tried every block group > > and failed with each one, so my suggestion above may not work in this > > instance. > > > >Let us know what happens with the balance command above anyway > > (dmesg output is useful information at this point). If that doesn't > > help, then we'll probably need to take a metadata image and throw it > > in josef's direction, where he will start crying at having to deal > > with enospc problems again. :) > > > >Hugo. > > -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- If you're not part of the solution, you're part --- of the precipiate. signature.as
Re: No space left on device
Hi and thanks for the quick reply. Have remounted the filesystem with enospc_debug, and run the rebalance you suggested, with the trance below. So perhaps the next step is for me to figure out how to take a metadata image and send it to josef (perhaps with a box of tissues) /Jakob [jakobt@soda ~]$ mount ... /dev/sda on /data type btrfs (rw,relatime,nospace_cache,enospc_debug) [jakobt@soda ~]$ sudo btrfs balance start -dusage=0 /data ERROR: error during balancing '/data' - No space left on device There may be more info in syslog - try dmesg | tail [jakobt@soda ~]$ touch /data/jakobt/monkey touch: cannot touch ‘/data/jakobt/monkey’: No space left on device [jakobt@soda ~]$ dmesg | tail -n3 [1117530.870965] btrfs: device label Data devid 1 transid 47091 /dev/sda [1117573.087580] btrfs: 426 enospc errors during balance [1117642.002437] btrfs: 426 enospc errors during balance On Wed, Feb 12, 2014 at 11:26 AM, Hugo Mills wrote: > On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote: >> Hello. I am experiencing "No space left on device" with a btrfs file >> system, yet I cannot seem to find any exhausted resource. Could some >> resource I do not know about be exhausted, or is this caused by >> something else. Below is a trace of information that might be usefull, >> please let me know if there is anything I can do to resolve the issue. >> >> /Jakob >> >> [jakobt@soda ~]$ uname -a >> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014 >> x86_64 GNU/Linux > >Were you using this kernel when the problem happened? > >> [jakobt@soda ~]$ btrfs --version >> Btrfs v3.12 >> >> [jakobt@soda ~]$ mount >> ... >> /dev/sda on /data type btrfs (rw,relatime,nospace_cache) >> >> [jakobt@soda ~]$ df /data >> Filesystem 1K-blocks Used Available Use% Mounted on >> /dev/sdb2 76594224 49247368 23433028 68% / >> >> [jakobt@soda ~]$ btrfs filesystem df /data >> Data, single: total=1.82TiB, used=518.04GiB >> System, DUP: total=8.00MiB, used=204.00KiB >> System, single: total=4.00MiB, used=0.00 >> Metadata, DUP: total=1.00GiB, used=767.70MiB > >^^^ This is your problem, most likely in conjunction with all the > space on the device being allocated. Being a copy-on-write filesystem, > btrfs needs free space to make any update. If it doesn't have that > free space, you get "No space left on device". You typically need > somewhere around 0.5-1 GiB of headroom in metadata for normal > operation, so I'm surprised you got this far. :) > >The FS should normally allocate more metadata space as it needs it, > but because (I think) your data allocation has taken up all the > available space on the device, there's no way for it to add more. > >> Metadata, single: total=8.00MiB, used=0.00 > >> [jakobt@soda ~]$ touch /data/jakobt/hat >> touch: cannot touch ‘/data/jakobt/hat’: No space left on device >> >> [jakobt@soda ~]$ sudo btrfs fi balance start /data >> ERROR: error during balancing '/data' - No space left on device >> There may be more info in syslog - try dmesg | tail > >Try: > > btrfs balance start -dusage=0 /data > > which should go looking for entirely unused block groups and reclaim > those. (If you don't use the -dusage= parameter, it will try to > balance everything, which takes a long time). > >> [jakobt@soda ~]$ dmesg | grep tail -n 2 >> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda >> [1113507.641752] btrfs: 1866 enospc errors during balance > >Although, that said... it looks like it's tried every block group > and failed with each one, so my suggestion above may not work in this > instance. > >Let us know what happens with the balance command above anyway > (dmesg output is useful information at this point). If that doesn't > help, then we'll probably need to take a metadata image and throw it > in josef's direction, where he will start crying at having to deal > with enospc problems again. :) > >Hugo. > > -- > === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === > PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk > --- If you're not part of the solution, you're part --- >of the precipiate. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device
On 12/02/2014 09:51, Jakob Truelsen wrote: Hello. I am experiencing "No space left on device" with a btrfs file system, yet I cannot seem to find any exhausted resource. Could some resource I do not know about be exhausted, or is this caused by something else. Below is a trace of information that might be usefull, please let me know if there is anything I can do to resolve the issue. /Jakob [jakobt@soda ~]$ uname -a Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014 x86_64 GNU/Linux [jakobt@soda ~]$ btrfs --version Btrfs v3.12 [jakobt@soda ~]$ mount ... /dev/sda on /data type btrfs (rw,relatime,nospace_cache) [jakobt@soda ~]$ df /data Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb2 76594224 49247368 23433028 68% / [jakobt@soda ~]$ btrfs filesystem df /data Data, single: total=1.82TiB, used=518.04GiB System, DUP: total=8.00MiB, used=204.00KiB System, single: total=4.00MiB, used=0.00 Metadata, DUP: total=1.00GiB, used=767.70MiB Metadata, single: total=8.00MiB, used=0.00 [jakobt@soda ~]$ touch /data/jakobt/hat touch: cannot touch ‘/data/jakobt/hat’: No space left on device [jakobt@soda ~]$ sudo btrfs fi balance start /data ERROR: error during balancing '/data' - No space left on device There may be more info in syslog - try dmesg | tail [jakobt@soda ~]$ dmesg | grep tail -n 2 [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda [1113507.641752] btrfs: 1866 enospc errors during balance [jakobt@soda ~]$ sudo umount /data [jakobt@soda ~]$ sudo btrfsck /dev/sda ... cache and super generation don't match, space cache will be invalidated .. found 172181366447 bytes used err is 0 total csum bytes: 418841160 total tree bytes: 805187584 total fs tree bytes: 247164928 total extent tree bytes: 26329088 btree space waste bytes: 164771401 file data blocks allocated: 561564688384 referenced 511759908864 Btrfs v3.12 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Jacob, Have you tried balancing just the data/metadata chunks only? Regards, Leonidas -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device
On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote: > Hello. I am experiencing "No space left on device" with a btrfs file > system, yet I cannot seem to find any exhausted resource. Could some > resource I do not know about be exhausted, or is this caused by > something else. Below is a trace of information that might be usefull, > please let me know if there is anything I can do to resolve the issue. > > /Jakob > > [jakobt@soda ~]$ uname -a > Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014 > x86_64 GNU/Linux Were you using this kernel when the problem happened? > [jakobt@soda ~]$ btrfs --version > Btrfs v3.12 > > [jakobt@soda ~]$ mount > ... > /dev/sda on /data type btrfs (rw,relatime,nospace_cache) > > [jakobt@soda ~]$ df /data > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/sdb2 76594224 49247368 23433028 68% / > > [jakobt@soda ~]$ btrfs filesystem df /data > Data, single: total=1.82TiB, used=518.04GiB > System, DUP: total=8.00MiB, used=204.00KiB > System, single: total=4.00MiB, used=0.00 > Metadata, DUP: total=1.00GiB, used=767.70MiB ^^^ This is your problem, most likely in conjunction with all the space on the device being allocated. Being a copy-on-write filesystem, btrfs needs free space to make any update. If it doesn't have that free space, you get "No space left on device". You typically need somewhere around 0.5-1 GiB of headroom in metadata for normal operation, so I'm surprised you got this far. :) The FS should normally allocate more metadata space as it needs it, but because (I think) your data allocation has taken up all the available space on the device, there's no way for it to add more. > Metadata, single: total=8.00MiB, used=0.00 > [jakobt@soda ~]$ touch /data/jakobt/hat > touch: cannot touch ‘/data/jakobt/hat’: No space left on device > > [jakobt@soda ~]$ sudo btrfs fi balance start /data > ERROR: error during balancing '/data' - No space left on device > There may be more info in syslog - try dmesg | tail Try: btrfs balance start -dusage=0 /data which should go looking for entirely unused block groups and reclaim those. (If you don't use the -dusage= parameter, it will try to balance everything, which takes a long time). > [jakobt@soda ~]$ dmesg | grep tail -n 2 > [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda > [1113507.641752] btrfs: 1866 enospc errors during balance Although, that said... it looks like it's tried every block group and failed with each one, so my suggestion above may not work in this instance. Let us know what happens with the balance command above anyway (dmesg output is useful information at this point). If that doesn't help, then we'll probably need to take a metadata image and throw it in josef's direction, where he will start crying at having to deal with enospc problems again. :) Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- If you're not part of the solution, you're part --- of the precipiate. signature.asc Description: Digital signature
Re: "No space left on device"
Hugo Mills posted on Thu, 14 Nov 2013 21:00:56 + as excerpted: >> Is there a formula to calculate how much space btrfs _might_ need? > > Not really. I'd expect to need something in the range 250-1500 GiB of > headroom, depending on the size of the filesystem (and on the size of > the metadata). As a somewhat more concrete answer... While recently doing a bit of research on something else, I came across comments that on a large enough filesystem, data chunks default to 1 GiB, while metadata chunks default to 256 MiB. And we know that data mode defaults to SINGLE, while metadata mode defaults to DUP. So on a default single-device btrfs of several gigs plus, assuming the files being manipulated are under 1 GiB size, keeping an unallocated space reserve of 1.5 GiB should be reasonable. That's enough unallocated space to allocate one more 1 GiB data chunk, plus one more 256 MiB metadata chunk, doubled to a half GiB due to DUP mode. Obviously in the single-mode-metadata case, the metadata requirement would be only a single copy, so 256 MiB for it, 1.25 GiB total unallocated, minimum. btrfs filesystem show is the command used to see what your allocated space for a filesystem looks like, per device. However, it doesn't note UNALLOCATED space, only size and used (aka allocated), so an admin must do the math to figure unallocated. If the files being manipulated are over a gig in size, round up to the nearest whole GiB for the data and add another half GiB to cover the quarter-gig DUP metadata case. If the filesystem is under a gig in size, btrfs defaults to mixed data+metadata, with chunks of 256 MiB if there's space but apparently rather more flexibility in ordered to better utilize all available space. At such "small" sizes[1], full allocation with no more to allocate being common, but one does hope people using such sized filesystems have a good idea what will be going on them, and they won't / need/ to allocate further chunks after the initial filesystem population. And quite in contrast to the multi-TB filesystems, rebalancing such a filesystem in ordered to recover lost space should be relatively fast even on spinning rust. For filesystems of 1 GiB up to say 10 GiB, it's a more open question, altho at that size, there's still a rather good chance that the sysadmin has a reasonably good idea what's going on the filesystem and has planned accordingly, with some "reasonable" level of over-allocation for future- proofing and plan fuzziness, and rebalances should still occur in reasonable time as well, so it shouldn't be a /huge/ problem unless the admin simply isn't tracking the situation. The multi-device situation is another dimension vector. Apparently, except for single mode, btrfs at this point only ever allocates in pairs (plus raid5/6 checksum chunks if applicable, and pairs of pairs in raid10 mode), regardless of the number of devices available, which does simplify calculations to some degree. Btrfs' multi-device default (for >1 GiB per device sizes, anyway) is single data, raid1 metadata. So to reserve space for one chunk of either type, we'd need at least 1 GiB unallocated on ONE device to allow at least one single-mode data chunk allocation, PLUS at least 256 MiB unallocated on each of TWO devices to cover at least one raid1-mode metadata chunk allocation. Thus, with two devices, we'd require at least 1.25 GiB free/unallocated on one device (1 GiB data chunk plus one copy of the 256 MiB metadata chunk), 256 MiB on the other (the second copy of the metadata). For a three+ device filesystem, that would work, OR 256 MiB on each of two (for the raid1 metadata), 1 GiB on a third (for the data). For raid1 data the 1 GiB data chunks must have two copies, each on its own device, and the above multi-device default scenario would modify accordingly: 2-device-case: 1.25 GiB minimum unallocated on each device (one copy each for a data and a metadata chunk). 3-device-case: That OR 1.25/1.0/.25 GiB. 4-device-plus-case: Either of those or 1.0/1.0/.25/.25 GiB. For single metadata plus default single data, we're back to the 1.25 GiB total case, in two separate chunks of 1 GiB and 256 MiB, either on separate devices or the same device. I haven't personally played with the raid0 case as it doesn't fit my use- case, but the wiki documentation suggests that it still allocates chunks only in pairs, striping the data/metadata across the pair. So we're looking at a minimum 1 GiB on each of two separate devices for a raid0 data chunk allocation (which would then allow two gigs of data), a minimum of 256 MiB on each of two separate devices for a raid0 metadata chunk allocation (which would hold a half-gig of metadata). Permutations are, as they say "left as an exercise for the reader." =:^) Apparently raid10 mode is pairs of pairs, so allocates in sets of four. Metadata: 256 MiB on each of four separate devices, 512 MiB metadata capacity.
Re: "No space left on device"
On Thu, Nov 14, 2013 at 07:54:21PM +, Leonidas Spyropoulos wrote: > Hello, > > I've been following this list for years and I see during various situations > this message coming up. Some times it's a genuine problem that there is > actually not enough space. In other cases it's some by-product of something > else. I have seen this error personality on a broken system ( which I never > figured out what had happened). > I know this is still experimental but I just want to make sure my > expectations are not really out on sync with the others. > > As an end user when I see an error like this the first thing I will do is > check the space (using 'df' command) [1]. If I see more that 7% I usually > think it's OK, (depends on the size of partition as well). The problem is that there's two kinds of space (data and metadata), and either could run out. The FS won't, currently, attempt to reallocate between one and the other -- hence the recommendation for a filtered balance to fix it (one of the side-effects of a balance is to be able to free up unused or little-used chunks of allocation). > - Is this unreasonable in btrfs filesystem? Is there a formula to calculate > how much space btrfs _might_ need? Not really. I'd expect to need something in the range 250-1500 GiB of headroom, depending on the size of the filesystem (and on the size of the metadata). > - It's probably not your job but can df reports correct sizes for btrfs? > I've seen some threads on trying to show the actual space occupied by data > and/or metadata with btrfs command. Can we expect this someone to be > incorporated into df command? Sadly, no. The POSIX API for df doesn't contain enough information to give an accurate representation of the space on the FS. > - In cases that btrfs reports this error but there's something else that's > causing it, can we expect better error handling from btrfs so the end user > is pointed to the correct direction? Again, we're constrained by the POSIX API here -- we only have ENOSPC to represent the error condition. I suspect the best we could do (possibly) is print something in dmesg, where users don't tend to look anyway. There's a future project on the books to make the FS attempt to recover unused or little-used chunks, which should reduce the odds of the ENOSPC showing up in an "unexpected" way (i.e. running out of metadata). Hugo. > [1]: one could argue that an end user should use the btrfs commands instead > but let's leave that for now. > > Apologies if these have already been answered or are already on roadmap. > > Thanks in advanced, your comments are appreciated. > > Kind regards, > Leonidas > -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- emacs: Eats Memory and Crashes. --- signature.asc Description: Digital signature
Re: No space left on device, problem
On Mon, Oct 28, 2013 at 8:05 PM, Josef Bacik wrote: > On Sun, Oct 27, 2013 at 09:50:37AM +0100, Igor M wrote: >> On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski wrote: >> >> Still no messages. Parameter seems to be active as >> >> /sys/module/printk/parameters/ignore_loglevel is Y, but there are no >> >> messages in log files or dmesg. Maybe I need to turn on some kernel >> >> debugging option and recompile kernel ? >> >> Also I should mention that cca 230G+ data was copied before this error >> >> started to occur. >> > >> > I think I saw a similar issue before. >> > >> > Can you try using rsync with "--bwlimit XY" option to copy the files? >> > >> > The option will limit the speed, in kB, at which the file is being >> > copied; it will work even when source and destination files are on a >> > local machine. >> > >> >> Also I run strace cp -a .. >> ... >> read(3, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 >> write(4, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 >> read(3, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = 65536 >> write(4, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = -1 ENOSPC (No >> space left on device) >> >> Last two write calls take a lot more time, and then last one returns >> ENOSPC. But if this write is retryed, then it succeeds. >> I tried with midnight commander and when error occurs, if I Retry >> operation then it finishes copying this file until error occurs again >> at next file. >> >> With --bwlimit it seems to be better, lower the speed later the error >> occurs, and if it's slow enough copy is successfull. >> But now I'm not sure anymore. I copied a few files with bwlimit, and >> now sudenly error doesn't occur anymore, even with no bwlimit. >> I'll do some more tests. > > I just sent a patch to the list > > [PATCH] Btrfs: make sure the delalloc workers actually flush compressed writes > > Can you run this patch and see if it makes a difference for your test? > Thanks, > > Josef I'll try with this patch. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Mon, Oct 28, 2013 at 6:57 PM, Chris Murphy wrote: > > On Oct 28, 2013, at 1:40 AM, Igor M wrote: >> >> dmesg: http://pastebin.com/t2H1QYye > > > You've got a warning related to pcie bridge on boot, with a trace that > follows. I don't know if this could be related to some problems. > > [0.325976] [ cut here ] > [0.326086] WARNING: CPU: 5 PID: 1 at drivers/pci/search.c:46 > pci_find_upstream_pcie_bridge+0x50/0x70() > [0.326263] Modules linked in: > [0.326412] CPU: 5 PID: 1 Comm: swapper/0 Not tainted 3.11.6 #1 > [0.326520] Hardware name: System manufacturer System Product Name/P8H77-V > LE, BIOS 0601 06/06/2012 > [0.326695] 0009 81449f86 > > [0.327032] 810514b1 88040b031898 88040b031800 > 88040b031898 > [0.327367] 8000 77ff8000 812191a0 > 0004 > [0.327704] Call Trace: > [0.327819] [] ? dump_stack+0x41/0x51 > [0.327931] [] ? warn_slowpath_common+0x81/0xb0 > [0.328040] [] ? pci_find_upstream_pcie_bridge+0x50/0x70 > [0.328151] [] ? intel_iommu_add_device+0x43/0x210 > [0.328261] [] ? bus_set_iommu+0x60/0x60 > [0.328370] [] ? add_iommu_group+0x2c/0x60 > [0.328481] [] ? bus_for_each_dev+0x4d/0x80 > [0.328591] [] ? bus_set_iommu+0x4a/0x60 > [0.328701] [] ? intel_iommu_init+0xb20/0xc45 > [0.328812] [] ? unpack_to_rootfs+0x24b/0x25b > [0.328922] [] ? pci_iommu_init+0xe/0x37 > [0.329031] [] ? memblock_find_dma_reserve+0x148/0x148 > [0.329142] [] ? do_one_initcall+0x102/0x150 > [0.329252] [] ? kernel_init_freeable+0xfd/0x18e > [0.329362] [] ? do_early_param+0x83/0x83 > [0.329471] [] ? rest_init+0x70/0x70 > [0.329579] [] ? kernel_init+0x9/0xe0 > [0.329688] [] ? ret_from_fork+0x7c/0xb0 > [0.329797] [] ? rest_init+0x70/0x70 > [0.329907] ---[ end trace 0946f959337cff8b ]--- > > > There are also numerous ACPI errors. > > ACPI Error: [DSSP] Namespace lookup failure > > check this: > http://forums.gentoo.org/viewtopic-t-960476-start-0.html > > > Anyway, I don't see any read or write failures for any of the drives which is > what I was kinda expecting. > > >> >> dest disk: http://pastebin.com/ez9jALS2 > > This is a new drive with only 71 power on hours yet I'm seeing this: > > • 0x0009 2 27 Transition from drive PhyRdy to drive > PhyNRdy > • 0x000a 2 27 Device-to-host register FISes sent due to a > COMRESET > > > > That's unexpected but I don't know that it's releated. The dmesg doesn't > report any phy issues with the drive. Maybe check syslog or journalctl with a > case insensitive search for phy and see if you find anything. > > > > > Chris Murphy > Drive should be ok. About pcie_bridge warning, I'm not sure how to solve, otherwise this computer is stable. There is no error if compression is turned off or non-compressible file is copied. I'll try on different computer and see if the same will happen. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 09:50:37AM +0100, Igor M wrote: > On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski wrote: > >> Still no messages. Parameter seems to be active as > >> /sys/module/printk/parameters/ignore_loglevel is Y, but there are no > >> messages in log files or dmesg. Maybe I need to turn on some kernel > >> debugging option and recompile kernel ? > >> Also I should mention that cca 230G+ data was copied before this error > >> started to occur. > > > > I think I saw a similar issue before. > > > > Can you try using rsync with "--bwlimit XY" option to copy the files? > > > > The option will limit the speed, in kB, at which the file is being > > copied; it will work even when source and destination files are on a > > local machine. > > > > Also I run strace cp -a .. > ... > read(3, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 > write(4, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 > read(3, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = 65536 > write(4, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = -1 ENOSPC (No > space left on device) > > Last two write calls take a lot more time, and then last one returns > ENOSPC. But if this write is retryed, then it succeeds. > I tried with midnight commander and when error occurs, if I Retry > operation then it finishes copying this file until error occurs again > at next file. > > With --bwlimit it seems to be better, lower the speed later the error > occurs, and if it's slow enough copy is successfull. > But now I'm not sure anymore. I copied a few files with bwlimit, and > now sudenly error doesn't occur anymore, even with no bwlimit. > I'll do some more tests. I just sent a patch to the list [PATCH] Btrfs: make sure the delalloc workers actually flush compressed writes Can you run this patch and see if it makes a difference for your test? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 28, 2013, at 1:40 AM, Igor M wrote: > > dmesg: http://pastebin.com/t2H1QYye You've got a warning related to pcie bridge on boot, with a trace that follows. I don't know if this could be related to some problems. [0.325976] [ cut here ] [0.326086] WARNING: CPU: 5 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x50/0x70() [0.326263] Modules linked in: [0.326412] CPU: 5 PID: 1 Comm: swapper/0 Not tainted 3.11.6 #1 [0.326520] Hardware name: System manufacturer System Product Name/P8H77-V LE, BIOS 0601 06/06/2012 [0.326695] 0009 81449f86 [0.327032] 810514b1 88040b031898 88040b031800 88040b031898 [0.327367] 8000 77ff8000 812191a0 0004 [0.327704] Call Trace: [0.327819] [] ? dump_stack+0x41/0x51 [0.327931] [] ? warn_slowpath_common+0x81/0xb0 [0.328040] [] ? pci_find_upstream_pcie_bridge+0x50/0x70 [0.328151] [] ? intel_iommu_add_device+0x43/0x210 [0.328261] [] ? bus_set_iommu+0x60/0x60 [0.328370] [] ? add_iommu_group+0x2c/0x60 [0.328481] [] ? bus_for_each_dev+0x4d/0x80 [0.328591] [] ? bus_set_iommu+0x4a/0x60 [0.328701] [] ? intel_iommu_init+0xb20/0xc45 [0.328812] [] ? unpack_to_rootfs+0x24b/0x25b [0.328922] [] ? pci_iommu_init+0xe/0x37 [0.329031] [] ? memblock_find_dma_reserve+0x148/0x148 [0.329142] [] ? do_one_initcall+0x102/0x150 [0.329252] [] ? kernel_init_freeable+0xfd/0x18e [0.329362] [] ? do_early_param+0x83/0x83 [0.329471] [] ? rest_init+0x70/0x70 [0.329579] [] ? kernel_init+0x9/0xe0 [0.329688] [] ? ret_from_fork+0x7c/0xb0 [0.329797] [] ? rest_init+0x70/0x70 [0.329907] ---[ end trace 0946f959337cff8b ]--- There are also numerous ACPI errors. ACPI Error: [DSSP] Namespace lookup failure check this: http://forums.gentoo.org/viewtopic-t-960476-start-0.html Anyway, I don't see any read or write failures for any of the drives which is what I was kinda expecting. > > dest disk: http://pastebin.com/ez9jALS2 This is a new drive with only 71 power on hours yet I'm seeing this: • 0x0009 2 27 Transition from drive PhyRdy to drive PhyNRdy • 0x000a 2 27 Device-to-host register FISes sent due to a COMRESET That's unexpected but I don't know that it's releated. The dmesg doesn't report any phy issues with the drive. Maybe check syslog or journalctl with a case insensitive search for phy and see if you find anything. Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Mon, Oct 28, 2013 at 12:27 AM, Chris Murphy wrote: > > On Oct 27, 2013, at 4:46 PM, Chris Murphy wrote: > >> >> On Oct 27, 2013, at 3:53 PM, Igor M wrote: >> >>> I made some more tests. Disk is 3TB, first cca 225GB is copied without >>> errors. >>> Then errors 'No space left on device' begins. >> >> Post the full entire dmesg somewhere please. pastebin.com is one option. > > > And on list or pasted, the output for the disk from: > > smartctl -x /dev/sdX > dmesg: http://pastebin.com/t2H1QYye source disk: http://pastebin.com/JqKxkxKr dest disk: http://pastebin.com/ez9jALS2 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 27, 2013, at 4:46 PM, Chris Murphy wrote: > > On Oct 27, 2013, at 3:53 PM, Igor M wrote: > >> I made some more tests. Disk is 3TB, first cca 225GB is copied without >> errors. >> Then errors 'No space left on device' begins. > > Post the full entire dmesg somewhere please. pastebin.com is one option. And on list or pasted, the output for the disk from: smartctl -x /dev/sdX Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 27, 2013, at 3:53 PM, Igor M wrote: > I made some more tests. Disk is 3TB, first cca 225GB is copied without errors. > Then errors 'No space left on device' begins. Post the full entire dmesg somewhere please. pastebin.com is one option. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
I made some more tests. Disk is 3TB, first cca 225GB is copied without errors. Then errors 'No space left on device' begins. Now if I use rsync with '--bwlimit' option no error occurs or if I choose 'Retry' in Midnight Commander then continues and after a while another error occurs and again 'Retry' and so on. I also noticed something else. Just before this error occurs, write() call takes a lot longer, I also see that progress stops. If I do 'btrfs fi df ..' at this moment: Data: total=114.01GB, used=112.00GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=1.00GB, used=452.93MB Metadata: total=8.00MB, used=0.00 then error is reported, again 'btrfs fi df ..' Data: total=114.01GB, used=113.00GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=1.00GB, used=456.98MB Metadata: total=8.00MB, used=0.00 and then 'Retry' and it goes on, until another error. Always before error, copying stops and used Data and Metadata changes. Maybe it's something with allocating metadata. I don't know. This goes on for cca 25G-30G and from now on no errors anymore. After this 1.3TB was copied without errors. But some of data was on rather slow disk, so maybe that's why no more errors. On Sun, Oct 27, 2013 at 9:50 AM, Igor M wrote: > On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski wrote: >>> Still no messages. Parameter seems to be active as >>> /sys/module/printk/parameters/ignore_loglevel is Y, but there are no >>> messages in log files or dmesg. Maybe I need to turn on some kernel >>> debugging option and recompile kernel ? >>> Also I should mention that cca 230G+ data was copied before this error >>> started to occur. >> >> I think I saw a similar issue before. >> >> Can you try using rsync with "--bwlimit XY" option to copy the files? >> >> The option will limit the speed, in kB, at which the file is being >> copied; it will work even when source and destination files are on a >> local machine. >> > > Also I run strace cp -a .. > ... > read(3, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 > write(4, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 > read(3, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = 65536 > write(4, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = -1 ENOSPC (No > space left on device) > > Last two write calls take a lot more time, and then last one returns > ENOSPC. But if this write is retryed, then it succeeds. > I tried with midnight commander and when error occurs, if I Retry > operation then it finishes copying this file until error occurs again > at next file. > > With --bwlimit it seems to be better, lower the speed later the error > occurs, and if it's slow enough copy is successfull. > But now I'm not sure anymore. I copied a few files with bwlimit, and > now sudenly error doesn't occur anymore, even with no bwlimit. > I'll do some more tests. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 9:56 AM, Brendan Hide wrote: > On 2013/10/27 10:50 AM, Igor M wrote: >> >> On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski >> wrote: Still no messages. Parameter seems to be active as /sys/module/printk/parameters/ignore_loglevel is Y, but there are no messages in log files or dmesg. Maybe I need to turn on some kernel debugging option and recompile kernel ? Also I should mention that cca 230G+ data was copied before this error started to occur. >>> >>> I think I saw a similar issue before. >>> >>> Can you try using rsync with "--bwlimit XY" option to copy the files? >>> >>> The option will limit the speed, in kB, at which the file is being >>> copied; it will work even when source and destination files are on a >>> local machine. >>> >> Also I run strace cp -a .. >> ... >> read(3, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 >> write(4, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 >> read(3, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = 65536 >> write(4, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = -1 ENOSPC (No >> space left on device) >> >> Last two write calls take a lot more time, and then last one returns >> ENOSPC. But if this write is retryed, then it succeeds. >> I tried with midnight commander and when error occurs, if I Retry >> operation then it finishes copying this file until error occurs again >> at next file. >> >> With --bwlimit it seems to be better, lower the speed later the error >> occurs, and if it's slow enough copy is successfull. >> But now I'm not sure anymore. I copied a few files with bwlimit, and >> now sudenly error doesn't occur anymore, even with no bwlimit. >> I'll do some more tests. >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > This sounds to me like the problem is related to read performance causing a > bork. This would explain why bwlimit helps, as well as why cp works the > second time around (since it is cached). > cp doesn't work second time. cp always fails. If last write() call is retryed it works, I think this midnight commander do, if you choose 'retry'. It doesn't copy from begining it continues where it left. I also tried from different disk, same result. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On 2013/10/27 10:50 AM, Igor M wrote: On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski wrote: Still no messages. Parameter seems to be active as /sys/module/printk/parameters/ignore_loglevel is Y, but there are no messages in log files or dmesg. Maybe I need to turn on some kernel debugging option and recompile kernel ? Also I should mention that cca 230G+ data was copied before this error started to occur. I think I saw a similar issue before. Can you try using rsync with "--bwlimit XY" option to copy the files? The option will limit the speed, in kB, at which the file is being copied; it will work even when source and destination files are on a local machine. Also I run strace cp -a .. ... read(3, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 write(4, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 read(3, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = 65536 write(4, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = -1 ENOSPC (No space left on device) Last two write calls take a lot more time, and then last one returns ENOSPC. But if this write is retryed, then it succeeds. I tried with midnight commander and when error occurs, if I Retry operation then it finishes copying this file until error occurs again at next file. With --bwlimit it seems to be better, lower the speed later the error occurs, and if it's slow enough copy is successfull. But now I'm not sure anymore. I copied a few files with bwlimit, and now sudenly error doesn't occur anymore, even with no bwlimit. I'll do some more tests. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html This sounds to me like the problem is related to read performance causing a bork. This would explain why bwlimit helps, as well as why cp works the second time around (since it is cached). -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 2:00 AM, Tomasz Chmielewski wrote: >> Still no messages. Parameter seems to be active as >> /sys/module/printk/parameters/ignore_loglevel is Y, but there are no >> messages in log files or dmesg. Maybe I need to turn on some kernel >> debugging option and recompile kernel ? >> Also I should mention that cca 230G+ data was copied before this error >> started to occur. > > I think I saw a similar issue before. > > Can you try using rsync with "--bwlimit XY" option to copy the files? > > The option will limit the speed, in kB, at which the file is being > copied; it will work even when source and destination files are on a > local machine. > Also I run strace cp -a .. ... read(3, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 write(4, "350348f07$0$24520$c3e8da3$fb4835"..., 65536) = 65536 read(3, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = 65536 write(4, "62.76C52BF412E849CB86D4FF3898B94"..., 65536) = -1 ENOSPC (No space left on device) Last two write calls take a lot more time, and then last one returns ENOSPC. But if this write is retryed, then it succeeds. I tried with midnight commander and when error occurs, if I Retry operation then it finishes copying this file until error occurs again at next file. With --bwlimit it seems to be better, lower the speed later the error occurs, and if it's slow enough copy is successfull. But now I'm not sure anymore. I copied a few files with bwlimit, and now sudenly error doesn't occur anymore, even with no bwlimit. I'll do some more tests. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
> Still no messages. Parameter seems to be active as > /sys/module/printk/parameters/ignore_loglevel is Y, but there are no > messages in log files or dmesg. Maybe I need to turn on some kernel > debugging option and recompile kernel ? > Also I should mention that cca 230G+ data was copied before this error > started to occur. I think I saw a similar issue before. Can you try using rsync with "--bwlimit XY" option to copy the files? The option will limit the speed, in kB, at which the file is being copied; it will work even when source and destination files are on a local machine. -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
Didn't see before. btrfs progs were compiled today form git. # btrfs version Btrfs v0.20-rc1-358-g194aa4a -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 12:22 AM, Igor M wrote: > On Sun, Oct 27, 2013 at 12:17 AM, Chris Murphy > wrote: >> >> On Oct 26, 2013, at 3:53 PM, Igor M wrote: >> >>> >>> I even added enospc_debug mount option, still no messages. >> >> If it were kernel enospc, you should have messages in dmesg. >> >> What version of btrfs progs when making the btrfs volume? >> >>> >>> cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No >>> space left on device >> >> Reboot with kernel parameter ignore_loglevel and retry the copy, and see if >> you now have anything in dmesg at the time of the copy. >> >> > > Some more info. This files are MySQL database files. Tables contains > only text so they compress well. > But, if I copy some uncompressable file, for example some video than > everything is ok, no error message, copy is successfull. > I also tried mounting without compression (without compress-force=lzo) > and in this case no error is reported. > So it seems it's something with compression ? > > I'll try rebooting with ignore_loglevel parameter. > >>> >>> It's the same error if I try to copy for ex. with midnight commander. >> >> I have the same kernel version, the same mount options, and use the same cp >> -a on a 5.3GB file and cannot reproduce your results. >> >> >> Chris Murphy Still no messages. Parameter seems to be active as /sys/module/printk/parameters/ignore_loglevel is Y, but there are no messages in log files or dmesg. Maybe I need to turn on some kernel debugging option and recompile kernel ? Also I should mention that cca 230G+ data was copied before this error started to occur. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sun, Oct 27, 2013 at 12:17 AM, Chris Murphy wrote: > > On Oct 26, 2013, at 3:53 PM, Igor M wrote: > >> >> I even added enospc_debug mount option, still no messages. > > If it were kernel enospc, you should have messages in dmesg. > > What version of btrfs progs when making the btrfs volume? > >> >> cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No >> space left on device > > Reboot with kernel parameter ignore_loglevel and retry the copy, and see if > you now have anything in dmesg at the time of the copy. > > Some more info. This files are MySQL database files. Tables contains only text so they compress well. But, if I copy some uncompressable file, for example some video than everything is ok, no error message, copy is successfull. I also tried mounting without compression (without compress-force=lzo) and in this case no error is reported. So it seems it's something with compression ? I'll try rebooting with ignore_loglevel parameter. >> >> It's the same error if I try to copy for ex. with midnight commander. > > I have the same kernel version, the same mount options, and use the same cp > -a on a 5.3GB file and cannot reproduce your results. > > > Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 26, 2013, at 3:53 PM, Igor M wrote: > > I even added enospc_debug mount option, still no messages. If it were kernel enospc, you should have messages in dmesg. What version of btrfs progs when making the btrfs volume? > > cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No > space left on device Reboot with kernel parameter ignore_loglevel and retry the copy, and see if you now have anything in dmesg at the time of the copy. > > It's the same error if I try to copy for ex. with midnight commander. I have the same kernel version, the same mount options, and use the same cp -a on a 5.3GB file and cannot reproduce your results. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Sat, Oct 26, 2013 at 11:35 PM, Chris Murphy wrote: > > On Oct 26, 2013, at 1:46 PM, Igor M wrote: >> >> # mount -t btrfs -o compress=lzo,compress-force=lzo / > > Why do you have two compression mount options? You need to pick one of these. I removed one. I was thinking both were needed. > >> What to do ? > > Are there any kernel messages reported by dmesg at the time the copy starts > and fails? What's the exact copy command you're using? No messages. Just whem mounting: device fsid c0bfcb22-8b7c-4936-afcd- 7acdf58f1d6c devid 1 transid 622 /dev/sdb btrfs: force lzo compression btrfs: disk space caching is enabled I even added enospc_debug mount option, still no messages. I'm using simple cp command: # cp -a /mnt/old_hd/data/gbdata/* /usr/local/mysql/data/gbdata/ or for one file # cp -a /mnt/old_hd/data/gbdata/parts_0016.MYD /usr/local/mysql/data/gbdata/ cp: writing ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0016.MYD’: No space left on device It's the same error if I try to copy for ex. with midnight commander. On Sat, Oct 26, 2013 at 11:35 PM, Chris Murphy wrote: > > On Oct 26, 2013, at 1:46 PM, Igor M wrote: >> >> # mount -t btrfs -o compress=lzo,compress-force=lzo / > > Why do you have two compression mount options? You need to pick one of these. > >> What to do ? > > Are there any kernel messages reported by dmesg at the time the copy starts > and fails? What's the exact copy command you're using? > > > Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
On Oct 26, 2013, at 1:46 PM, Igor M wrote: > > # mount -t btrfs -o compress=lzo,compress-force=lzo / Why do you have two compression mount options? You need to pick one of these. > What to do ? Are there any kernel messages reported by dmesg at the time the copy starts and fails? What's the exact copy command you're using? Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device, problem
Some more info, exact error message is: cp: writing ‘/usr/local/mysql/data/gbdata/parts_0015.MYI’: No space left on device cp: failed to extend ‘/usr/local/mysql/data/gbdata/parts_0015.MYI’: No space left on device Files are 2.7G - 7.7G big. On Sat, Oct 26, 2013 at 9:46 PM, Igor M wrote: > Hello, > > I just upgraded kernel to 3.11.6 added new disk and created btrfs: > > # mkfs.btrfs /dev/sdb > # mount -t btrfs -o compress=lzo,compress-force=lzo /dev/sdb > /usr/local/mysql/data > > I started copying files from old disk and then I got 'No space left on > device', but there is a lot of space. > > # df -h > FilesystemSize Used Avail Use% Mounted on > /dev/sdb 2.8T 110G 2.7T 4% /usr/local/mysql/data > > # btrfs fi show > Label: none uuid: c0bfcb22-8b7c-4936-afcd-7acdf58f1d6c > Total devices 1 FS bytes used 108.68GB > devid1 size 2.73TB used 113.04GB path /dev/sdb > > # btrfs fi df /usr/local/mysql/data > Data: total=111.01GB, used=108.25GB > System, DUP: total=8.00MB, used=20.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=1.00GB, used=441.91MB > Metadata: total=8.00MB, used=0.00 > > I tried balance from FAQ: > > # btrfs fi balance start -dusage=5 /usr/local/mysql/data > Done, had to relocate 4 out of 117 chunks > > > But it doesn't help. When I try to copy file there is still 'No space > left on device'. > > What to do ? > > > Regards, > > Igor -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device with 3.10-rc2
Am Samstag, 25. Mai 2013, 19:36:03 schrieb Martin Steigerwald: > Hi! > > Now I got it myself what I read again and again on this mailinglist: During > apt-get upgrade I get no space left on device. > > But there is: > > merkaba:~> df -hT / > DateisystemTyp Größe Benutzt Verf. Verw% Eingehängt auf > /dev/mapper/merkaba-debian btrfs 19G 14G 4,6G 76% / > > > merkaba:[…]> ./btrfs fi df / > Disk size:18.62GB > Disk allocated: 18.62GB > Disk unallocated:0.00 > Used: 13.79GB > Free (Estimated): 4.84GB (Max: 4.84GB, min: 4.84GB) > Data to disk ratio: 100 % > merkaba:[…]> ./btrfs fi disk-usage / […] > I think I will let it run a rebalance. Not working as well: > > merkaba:~> btrfs fi ba / > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > merkaba:~#19> dmesg | tail > […] > [72914.023312] btrfs: relocating block group 19348324352 flags 1 > [72914.131856] btrfs: 24 enospc errors during balance […] > How do I get this filesystem to an operational state again despite > reformatting? btrfs balance start worked a bit farther after apt-get clean and completed after removing the one and only snapshot from 2013-05-01. Still I consider this a bug. Mount options: martin@merkaba:~> grep "/ .*btrfs" /proc/mounts /dev/mapper/merkaba-debian / btrfs rw,noatime,compress=lzo,ssd,space_cache 0 0 State after removing snapshot and balance: merkaba[…]> ./btrfs fi sh failed to read /dev/sr0 Label: 'home' uuid: […] Total devices 1 FS bytes used 197.65GB devid1 size 223.52GB used 219.04GB path /dev/dm-2 Label: 'debian' uuid: […] Total devices 1 FS bytes used 10.21GB devid1 size 18.62GB used 11.00GB path /dev/dm-0 Btrfs v0.19-367-g711b24a merkaba:[…]> ./btrfs fi df / Disk size:18.62GB Disk allocated: 11.00GB Disk unallocated: 7.62GB Used: 10.22GB Free (Estimated): 8.41GB (Max: 8.41GB, min: 4.59GB) Data to disk ratio: 100 % merkaba:[…]> ./btrfs fi disk-usage / Data,Single: Size:10.00GB, Used:9.61GB /dev/mapper/merkaba-debian 10.00GB Metadata,Single: Size:1.00GB, Used:602.45MB /dev/mapper/merkaba-debian 1.00GB System,Single: Size:4.00MB, Used:4.00KB /dev/mapper/merkaba-debian 4.00MB Unallocated: /dev/mapper/merkaba-debian 7.62GB merkaba:[…]> ./btrfs dev disk-usage / /dev/mapper/merkaba-debian 18.62GB Data,Single: 10.00GB Metadata,Single: 1.00GB System,Single:4.00MB Unallocated: 7.62GB Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" although df reports only 55% in use
On Tue, Mar 26, 2013 at 05:28:23PM -0600, Clemens Eisserer wrote: > Hi, > > I am using a btrfs loopback mounted file with lzo-compression on > Linux-3.7.9, and I ran into "No space left on device" messages, > although df reports only 55% of space is used: > > # touch testfile > touch: cannot touch `testfile': No space left on device > > # df > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/loop0 32768000 17383924 14649172 55% /home/ce/anditest > > # btrfs filesystem df . > Data: total=28.22GB, used=14.25GB > System, DUP: total=8.00MB, used=12.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=1.50GB, used=1.16GB > Metadata: total=8.00MB, used=0.00 > > Any ideas what is going wrong here? > Can I see btrfs fi show and try using btrfs-next and see if the problem is already fixed. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" although df reports only 55% in use
Hi Hugo, >> # btrfs filesystem df . >> Data: total=28.22GB, used=14.25GB >> System, DUP: total=8.00MB, used=12.00KB >> System: total=4.00MB, used=0.00 >> Metadata, DUP: total=1.50GB, used=1.16GB > >Your metadata is close to full -- we need quite a lot of working > space to CoW into. You (probably) have the full disk allocated to > chunks, and too much is allocated to data; not enough to metadata. You > need o balance, with a filter for the unused data chunks. See the > section in the FAQ on the wiki about full filesystems. (Sorry for not > finding the link, I'm on a restricted connection right now) I'll re-run the test on a filesystem created with -M (metadata and normal data combined). As far as I understand it shouldn't run in the same problem. > Almost 400MB are not sufficient to do simple touch? What is it doing? I have to admit that thought occurred to me too ;) Regards, Clemens -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" although df reports only 55% in use
On 03/27/2013 10:18 AM, Hugo Mills wrote: On Wed, Mar 27, 2013 at 12:28:23AM +0100, Clemens Eisserer wrote: I am using a btrfs loopback mounted file with lzo-compression on Linux-3.7.9, and I ran into "No space left on device" messages, although df reports only 55% of space is used: # touch testfile touch: cannot touch `testfile': No space left on device # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/loop0 32768000 17383924 14649172 55% /home/ce/anditest # btrfs filesystem df . Data: total=28.22GB, used=14.25GB System, DUP: total=8.00MB, used=12.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=1.50GB, used=1.16GB Your metadata is close to full -- we need quite a lot of working Almost 400MB are not sufficient to do simple touch? What is it doing? Cheers, Bernd -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: "No space left on device" although df reports only 55% in use
On Wed, Mar 27, 2013 at 12:28:23AM +0100, Clemens Eisserer wrote: > I am using a btrfs loopback mounted file with lzo-compression on > Linux-3.7.9, and I ran into "No space left on device" messages, > although df reports only 55% of space is used: > > # touch testfile > touch: cannot touch `testfile': No space left on device > > # df > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/loop0 32768000 17383924 14649172 55% /home/ce/anditest > > # btrfs filesystem df . > Data: total=28.22GB, used=14.25GB > System, DUP: total=8.00MB, used=12.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=1.50GB, used=1.16GB Your metadata is close to full -- we need quite a lot of working space to CoW into. You (probably) have the full disk allocated to chunks, and too much is allocated to data; not enough to metadata. You need o balance, with a filter for the unused data chunks. See the section in the FAQ on the wiki about full filesystems. (Sorry for not finding the link, I'm on a restricted connection right now) Hugo. > Metadata: total=8.00MB, used=0.00 > > Any ideas what is going wrong here? > > Thank you in advance, Clemens -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Gentlemen! You can't fight here! This is the War Room! --- signature.asc Description: Digital signature
Re: "No space left on device" although df reports only 55% in use
Hi again, I wonder if this is the intended behaviour or some known issue? Otherwise I could provide a tool written by a friend of mine which can trigger this issue within a few minutes on a a fresh btrfs partition. Its basically a file-system aging tester, which replays some real-world logs taken from NFS file servers. Regards, Clemens -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
HI, Am 26.03.2013 20:38, schrieb Josef Bacik: On Tue, Mar 26, 2013 at 01:22:20PM -0600, Stefan Priebe wrote: Hi, but when i transfer big files i see now this one: [20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds. [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20368.895140] rsync D 8160f580 0 14911 1 0x [20368.895148] 8801ca63fc78 0086 8800c28f8198 88022394f800 [20368.895158] 8801ca63ffd8 8801ca63ffd8 8801ca63ffd8 00012c40 [20368.895163] 81a11440 8801c9d36340 8801ca63fc88 8801cefce130 [20368.895169] Call Trace: [20368.895180] [] schedule+0x24/0x70 [20368.895207] [] wait_current_trans.isra.32+0x95/0x100 [btrfs] [20368.895214] [] ? add_wait_queue+0x60/0x60 [20368.895236] [] start_transaction.part.33+0x13d/0x4d0 [btrfs] [20368.895252] [] ? inode_permission+0x13/0x50 [20368.895271] [] start_transaction+0x24/0x30 [btrfs] [20368.895287] [] btrfs_start_transaction+0x13/0x20 [btrfs] [20368.895302] [] __unlink_start_trans+0x70/0x460 [btrfs] [20368.895307] [] ? check_acl+0x5a/0x122 [20368.895312] [] ? ns_capable+0x30/0x60 [20368.895317] [] ? generic_permission+0xbd/0x110 [20368.895336] [] btrfs_unlink+0x32/0xc0 [btrfs] [20368.895341] [] vfs_unlink.part.61+0x6d/0xd0 [20368.895345] [] vfs_unlink+0x37/0x50 [20368.895349] [] do_unlinkat+0x19b/0x240 [20368.895354] [] sys_unlink+0x11/0x20 [20368.895359] [] system_call_fastpath+0x16/0x1b Speed is just 100kb/s instead of 100MB/s. Hrm I wonder if 512 is too small for your case, can you add this patch to the pile and see what dmesg says when you are having these problems? Thanks, No idea why but i can't reproduce... ;-( Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 01:22:20PM -0600, Stefan Priebe wrote:
> Hi,
>
> but when i transfer big files i see now this one:
> [20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
> [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [20368.895140] rsync D 8160f580 0 14911 1
> 0x
> [20368.895148] 8801ca63fc78 0086 8800c28f8198
> 88022394f800
> [20368.895158] 8801ca63ffd8 8801ca63ffd8 8801ca63ffd8
> 00012c40
> [20368.895163] 81a11440 8801c9d36340 8801ca63fc88
> 8801cefce130
> [20368.895169] Call Trace:
> [20368.895180] [] schedule+0x24/0x70
> [20368.895207] []
> wait_current_trans.isra.32+0x95/0x100 [btrfs]
> [20368.895214] [] ? add_wait_queue+0x60/0x60
> [20368.895236] []
> start_transaction.part.33+0x13d/0x4d0 [btrfs]
> [20368.895252] [] ? inode_permission+0x13/0x50
> [20368.895271] [] start_transaction+0x24/0x30 [btrfs]
> [20368.895287] [] btrfs_start_transaction+0x13/0x20
> [btrfs]
> [20368.895302] [] __unlink_start_trans+0x70/0x460 [btrfs]
> [20368.895307] [] ? check_acl+0x5a/0x122
> [20368.895312] [] ? ns_capable+0x30/0x60
> [20368.895317] [] ? generic_permission+0xbd/0x110
> [20368.895336] [] btrfs_unlink+0x32/0xc0 [btrfs]
> [20368.895341] [] vfs_unlink.part.61+0x6d/0xd0
> [20368.895345] [] vfs_unlink+0x37/0x50
> [20368.895349] [] do_unlinkat+0x19b/0x240
> [20368.895354] [] sys_unlink+0x11/0x20
> [20368.895359] [] system_call_fastpath+0x16/0x1b
>
> Speed is just 100kb/s instead of 100MB/s.
>
Hrm I wonder if 512 is too small for your case, can you add this patch to the
pile and see what dmesg says when you are having these problems? Thanks,
Josef
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 50767bb..d19c9f6 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -31,6 +31,7 @@
#include "inode-map.h"
#include "volumes.h"
#include "dev-replace.h"
+#include "math.h"
#define BTRFS_ROOT_TRANS_TAG 0
@@ -576,10 +577,19 @@ void btrfs_throttle(struct btrfs_root *root)
static int should_end_transaction(struct btrfs_trans_handle *trans,
struct btrfs_root *root)
{
- int ret;
+ struct btrfs_block_rsv *block_rsv = &root->fs_info->global_block_rsv;
+ u64 num_bytes = 0;
+ int ret = 1;
- ret = btrfs_block_rsv_check(root, &root->fs_info->global_block_rsv, 5);
- return ret ? 1 : 0;
+ spin_lock(&block_rsv->lock);
+ num_bytes = div_factor(block_rsv->size, 5);
+ if (block_rsv->reserved >= num_bytes)
+ ret = 0;
+ else
+ printk(KERN_ERR "we're pretty low, setting blocked, reserved
%Lu, size %Lu, num %Lu\n",
+ block_rsv->reserved, block_rsv->size, num_bytes);
+ spin_unlock(&block_rsv->lock);
+ return ret;
}
int btrfs_should_end_transaction(struct btrfs_trans_handle *trans,
--
1.7.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi, but when i transfer big files i see now this one: [20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds. [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20368.895140] rsync D 8160f580 0 14911 1 0x [20368.895148] 8801ca63fc78 0086 8800c28f8198 88022394f800 [20368.895158] 8801ca63ffd8 8801ca63ffd8 8801ca63ffd8 00012c40 [20368.895163] 81a11440 8801c9d36340 8801ca63fc88 8801cefce130 [20368.895169] Call Trace: [20368.895180] [] schedule+0x24/0x70 [20368.895207] [] wait_current_trans.isra.32+0x95/0x100 [btrfs] [20368.895214] [] ? add_wait_queue+0x60/0x60 [20368.895236] [] start_transaction.part.33+0x13d/0x4d0 [btrfs] [20368.895252] [] ? inode_permission+0x13/0x50 [20368.895271] [] start_transaction+0x24/0x30 [btrfs] [20368.895287] [] btrfs_start_transaction+0x13/0x20 [btrfs] [20368.895302] [] __unlink_start_trans+0x70/0x460 [btrfs] [20368.895307] [] ? check_acl+0x5a/0x122 [20368.895312] [] ? ns_capable+0x30/0x60 [20368.895317] [] ? generic_permission+0xbd/0x110 [20368.895336] [] btrfs_unlink+0x32/0xc0 [btrfs] [20368.895341] [] vfs_unlink.part.61+0x6d/0xd0 [20368.895345] [] vfs_unlink+0x37/0x50 [20368.895349] [] do_unlinkat+0x19b/0x240 [20368.895354] [] sys_unlink+0x11/0x20 [20368.895359] [] system_call_fastpath+0x16/0x1b Speed is just 100kb/s instead of 100MB/s. Stefan Am 26.03.2013 20:16, schrieb Josef Bacik: On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe wrote: Hi Josef, Am 26.03.2013 18:45, schrieb Josef Bacik: Am 26.03.2013 16:25, schrieb Josef Bacik: On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote: Hi, Am 26.03.2013 15:44, schrieb Josef Bacik: Am 26.03.2013 13:53, schrieb Josef Bacik: no - it's just mounted with mount -o noatime :~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks, It still does not fix the problem. The rsync output looks like this so it does not work for file a but then continues on c d e, ... sync -av --progress /backup/ /mnt/ sending incremental file list .etc_openvpn/ipp.txt 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/ .log/UcliEvt.log 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61%7.35MB/s0:00:01 the dmesg output looks like this: [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0, reserved=49152, may_use=6438453248, readonly=65536 Ok so then this is probably it, let me know if it helps. Thanks, OK it now has copied a lot of files (170) without an error all were very small. Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks, Output is now: [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438354944, readonly=65536 Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks, Yes - this works now. Which of the patches can i drop? Do i just need the last one? Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5? So I'll put together all of the patches that actually need to go up for this and post them, but basically its the mutex patch, the last patch I sent you and the one that adjusts the reservations for rename and delete. Thank
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe wrote: > Hi Josef, > > Am 26.03.2013 18:45, schrieb Josef Bacik: > >> Am 26.03.2013 16:25, schrieb Josef Bacik: > >>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG > >>> wrote: > Hi, > Am 26.03.2013 15:44, schrieb Josef Bacik: > Am 26.03.2013 13:53, schrieb Josef Bacik: > no - it's just mounted with mount -o noatime > > :~# cat /proc/mounts | grep btrfs > /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 > > >>> > >>> Ok I think I see what's going on. Can you try this patch and see if > >>> it fixes > >>> it? Thanks, > >> > >> It still does not fix the problem. > >> > >> The rsync output looks like this so it does not work for file a but > >> then > >> continues on c d e, ... > >> > >> sync -av --progress /backup/ /mnt/ > >> sending incremental file list > >> .etc_openvpn/ipp.txt > >>229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) > >> .etc_openvpn/openvpn-status.log > >>360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) > >> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> > >> ".etc_openvpn/ipp.txt": No space left on device (28) > >> .log/ > >> .log/UcliEvt.log > >> 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) > >> .log/auth.log > >> 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) > >> .log/auth.log.1 > >> 19431424 61%7.35MB/s0:00:01 > >> > >> the dmesg output looks like this: > >> [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush > >> 2, flush_state 7 dumping space info > >> [ 551.323694] space_info 4 has 6439526400 free, is full > >> [ 551.323696] space_info total=25748307968, used=19308666880, > >> pinned=0, > >> reserved=49152, may_use=6438453248, readonly=65536 > >> > > > > Ok so then this is probably it, let me know if it helps. Thanks, > > OK it now has copied a lot of files (170) without an error all were very > small. > > >>> > >>> Welp progress is good. Throw this into the mix and go again, it's just > >>> adding > >>> some more debugging so I can make sure I'm going down the right rabbit > >>> hole. > >>> Thanks, > >> > >> Output is now: > >> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush > >> 2, flush_state 7 dumping space info > >> [ 9587.527392] dumping block rsv 2, size 0 reserved 0 > >> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 > >> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 > >> [ 9587.646958] space_info 4 has 6439428096 free, is full > >> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, > >> reserved=45056, may_use=6438453248, readonly=65536 > >> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush > >> 2, flush_state 7 dumping space info > >> [ 9587.727000] dumping block rsv 2, size 0 reserved 0 > >> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 > >> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 > >> [ 9587.839935] space_info 4 has 6439428096 free, is full > >> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, > >> reserved=45056, may_use=6438354944, readonly=65536 > >> > > > > Well then that looks like I was going down the wrong rabbit hole. This > > should > > fix you up, for real this time ;). Thanks, > > Yes - this works now. Which of the patches can i drop? Do i just need > the last one? > Is it safe to add another 18TB raid via converting it to btrfs raid0? > Will the fix be part of 3.9-rc5? > So I'll put together all of the patches that actually need to go up for this and post them, but basically its the mutex patch, the last patch I sent you and the one that adjusts the reservations for rename and delete. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Josef, Am 26.03.2013 18:45, schrieb Josef Bacik: Am 26.03.2013 16:25, schrieb Josef Bacik: On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote: Hi, Am 26.03.2013 15:44, schrieb Josef Bacik: Am 26.03.2013 13:53, schrieb Josef Bacik: no - it's just mounted with mount -o noatime :~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks, It still does not fix the problem. The rsync output looks like this so it does not work for file a but then continues on c d e, ... sync -av --progress /backup/ /mnt/ sending incremental file list .etc_openvpn/ipp.txt 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/ .log/UcliEvt.log 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61%7.35MB/s0:00:01 the dmesg output looks like this: [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0, reserved=49152, may_use=6438453248, readonly=65536 Ok so then this is probably it, let me know if it helps. Thanks, OK it now has copied a lot of files (170) without an error all were very small. Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks, Output is now: [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438354944, readonly=65536 Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks, Yes - this works now. Which of the patches can i drop? Do i just need the last one? Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5? Thanks and greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 10:19:19AM -0600, Stefan Priebe wrote: > HI, > > > Am 26.03.2013 16:25, schrieb Josef Bacik: > > On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG > > wrote: > >> Hi, > >> Am 26.03.2013 15:44, schrieb Josef Bacik: > >> Am 26.03.2013 13:53, schrieb Josef Bacik: > >> no - it's just mounted with mount -o noatime > >> > >> :~# cat /proc/mounts | grep btrfs > >> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 > >> > > > > Ok I think I see what's going on. Can you try this patch and see if it > > fixes > > it? Thanks, > > It still does not fix the problem. > > The rsync output looks like this so it does not work for file a but then > continues on c d e, ... > > sync -av --progress /backup/ /mnt/ > sending incremental file list > .etc_openvpn/ipp.txt > 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) > .etc_openvpn/openvpn-status.log > 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) > rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> > ".etc_openvpn/ipp.txt": No space left on device (28) > .log/ > .log/UcliEvt.log > 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) > .log/auth.log > 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) > .log/auth.log.1 > 19431424 61%7.35MB/s0:00:01 > > the dmesg output looks like this: > [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 551.323694] space_info 4 has 6439526400 free, is full > [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0, > reserved=49152, may_use=6438453248, readonly=65536 > > >>> > >>> Ok so then this is probably it, let me know if it helps. Thanks, > >> > >> OK it now has copied a lot of files (170) without an error all were very > >> small. > >> > > > > Welp progress is good. Throw this into the mix and go again, it's just > > adding > > some more debugging so I can make sure I'm going down the right rabbit hole. > > Thanks, > > Output is now: > [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 9587.527392] dumping block rsv 2, size 0 reserved 0 > [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 > [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 > [ 9587.646958] space_info 4 has 6439428096 free, is full > [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, > reserved=45056, may_use=6438453248, readonly=65536 > [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 9587.727000] dumping block rsv 2, size 0 reserved 0 > [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 > [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 > [ 9587.839935] space_info 4 has 6439428096 free, is full > [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, > reserved=45056, may_use=6438354944, readonly=65536 > Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks, Josef diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 1cf810a..ac415cf7 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -4471,7 +4471,12 @@ static void update_global_block_rsv(struct btrfs_fs_info *fs_info) spin_lock(&sinfo->lock); spin_lock(&block_rsv->lock); - block_rsv->size = num_bytes; + /* +* Limit the global block rsv to 512mb, we have infrastructure in place +* to throttle reservations if we start getting low on global block rsv +* space. +*/ + block_rsv->size = min_t(u64, num_bytes, 512 * 1024 * 1024); num_bytes = sinfo->bytes_used + sinfo->bytes_pinned + sinfo->bytes_reserved + sinfo->bytes_readonly + -- 1.7.7.6 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
HI, Am 26.03.2013 16:25, schrieb Josef Bacik: On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote: Hi, Am 26.03.2013 15:44, schrieb Josef Bacik: Am 26.03.2013 13:53, schrieb Josef Bacik: no - it's just mounted with mount -o noatime :~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks, It still does not fix the problem. The rsync output looks like this so it does not work for file a but then continues on c d e, ... sync -av --progress /backup/ /mnt/ sending incremental file list .etc_openvpn/ipp.txt 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/ .log/UcliEvt.log 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61%7.35MB/s0:00:01 the dmesg output looks like this: [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0, reserved=49152, may_use=6438453248, readonly=65536 Ok so then this is probably it, let me know if it helps. Thanks, OK it now has copied a lot of files (170) without an error all were very small. Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks, Output is now: [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438354944, readonly=65536 Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi,
> Am 26.03.2013 15:44, schrieb Josef Bacik:
> Am 26.03.2013 13:53, schrieb Josef Bacik:
> no - it's just mounted with mount -o noatime
>
> :~# cat /proc/mounts | grep btrfs
> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>
> >>>
> >>> Ok I think I see what's going on. Can you try this patch and see if it
> >>> fixes
> >>> it? Thanks,
> >>
> >> It still does not fix the problem.
> >>
> >> The rsync output looks like this so it does not work for file a but then
> >> continues on c d e, ...
> >>
> >> sync -av --progress /backup/ /mnt/
> >> sending incremental file list
> >> .etc_openvpn/ipp.txt
> >> 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196)
> >> .etc_openvpn/openvpn-status.log
> >> 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196)
> >> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >> ".etc_openvpn/ipp.txt": No space left on device (28)
> >> .log/
> >> .log/UcliEvt.log
> >> 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700)
> >> .log/auth.log
> >> 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700)
> >> .log/auth.log.1
> >> 19431424 61%7.35MB/s0:00:01
> >>
> >> the dmesg output looks like this:
> >> [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7 dumping space info
> >> [ 551.323694] space_info 4 has 6439526400 free, is full
> >> [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> >> reserved=49152, may_use=6438453248, readonly=65536
> >>
> >
> > Ok so then this is probably it, let me know if it helps. Thanks,
>
> OK it now has copied a lot of files (170) without an error all were very
> small.
>
Welp progress is good. Throw this into the mix and go again, it's just adding
some more debugging so I can make sure I'm going down the right rabbit hole.
Thanks,
Josef
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 84e8909..1cf810a 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4026,6 +4026,15 @@ static int flush_space(struct btrfs_root *root,
return ret;
}
+
+static void dump_block_rsv(struct btrfs_block_rsv *block_rsv)
+{
+ spin_lock(&block_rsv->lock);
+ printk(KERN_ERR "dumping block rsv %u, size %Lu reserved %Lu\n",
+ block_rsv->type, block_rsv->size, block_rsv->reserved);
+ spin_unlock(&block_rsv->lock);
+}
+
/**
* reserve_metadata_bytes - try to reserve bytes from the block_rsv's space
* @root - the root we're allocating for
@@ -4179,6 +4188,9 @@ out:
"flush %d, flush_state %d dumping space
info\n", block_rsv->type,
block_rsv->size, block_rsv->reserved, flush,
flush_state);
spin_unlock(&block_rsv->lock);
+ dump_block_rsv(&root->fs_info->delalloc_block_rsv);
+ dump_block_rsv(&root->fs_info->delayed_block_rsv);
+ dump_block_rsv(&root->fs_info->global_block_rsv);
dump_space_info(space_info, 0, 0);
}
--
1.7.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi, Am 26.03.2013 15:44, schrieb Josef Bacik: Am 26.03.2013 13:53, schrieb Josef Bacik: no - it's just mounted with mount -o noatime :~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 >>> >>> Ok I think I see what's going on. Can you try this patch and see if it >>> fixes >>> it? Thanks, >> >> It still does not fix the problem. >> >> The rsync output looks like this so it does not work for file a but then >> continues on c d e, ... >> >> sync -av --progress /backup/ /mnt/ >> sending incremental file list >> .etc_openvpn/ipp.txt >> 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) >> .etc_openvpn/openvpn-status.log >> 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) >> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> >> ".etc_openvpn/ipp.txt": No space left on device (28) >> .log/ >> .log/UcliEvt.log >> 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) >> .log/auth.log >> 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) >> .log/auth.log.1 >> 19431424 61%7.35MB/s0:00:01 >> >> the dmesg output looks like this: >> [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush >> 2, flush_state 7 dumping space info >> [ 551.323694] space_info 4 has 6439526400 free, is full >> [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0, >> reserved=49152, may_use=6438453248, readonly=65536 >> > > Ok so then this is probably it, let me know if it helps. Thanks, OK it now has copied a lot of files (170) without an error all were very small. Then this again: rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.vhost_net.mod.GrhWet" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/vhost_net.mod": No space left on device (28) rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rhine.mod.X2Ofhz" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rhine.mod": No space left on device (28) rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rng.mod.RAyxkF" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rng.mod": No space left on device (28) rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.virtio.mod.w7INoL" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/virtio.mod": No space left on device (28)^C [ 5012.468988] space_info total=25748307968, used=19308773376, pinned=0, reserved=69632, may_use=6438354944, readonly=65536 [ 5012.472981] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 5012.472982] space_info 4 has 6439399424 free, is full [ 5012.472982] space_info total=25748307968, used=19308773376, pinned=0, reserved=69632, may_use=6438354944, readonly=65536 [ 5012.476974] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 5012.476975] space_info 4 has 6439399424 free, is full [ 5012.476976] space_info total=25748307968, used=19308773376, pinned=0, reserved=69632, may_use=6438354944, readonly=65536 [ 5012.480968] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 5012.480969] space_info 4 has 6439399424 free, is full Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 07:49:16AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
>
>
> Am 26.03.2013 14:30, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG
> > wrote:
> >> Hi Josef,
> >>
> >> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> Hi,
>
> output here:
> [ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7 dumping space info
> [ 590.548280] space_info 4 has 6439292928 free, is full
> [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0,
> reserved=32768, may_use=6438354944, readonly=65536
> [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7 dumping space info
> [ 590.552264] space_info 4 has 6439284736 free, is full
> [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0,
> reserved=40960, may_use=6438354944, readonly=65536
> [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7 dumping space info
> [ 590.556258] space_info 4 has 6439284736 free, is full
> [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0,
> reserved=40960, may_use=6438354944, readonly=65536
> [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7 dumping space info
> [ 591.147373] space_info 4 has 6439235584 free, is full
> [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0,
> reserved=90112, may_use=6438354944, readonly=65536
> [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7 dumping space info
> [ 595.562390] space_info 4 has 6439120896 free, is full
> [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,
> reserved=73728, may_use=6438297600, readonly=65536
>
> >>>
> >>> Weird, we have all the flushing stuff set and yet there is still a whole
> >>> lot of
> >>> outstanding reservations. Do you have compression enabled? Thanks,
> >>
> >> no - it's just mounted with mount -o noatime
> >>
> >> :~# cat /proc/mounts | grep btrfs
> >> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>
> >
> > Ok I think I see what's going on. Can you try this patch and see if it
> > fixes
> > it? Thanks,
>
> It still does not fix the problem.
>
> The rsync output looks like this so it does not work for file a but then
> continues on c d e, ...
>
> sync -av --progress /backup/ /mnt/
> sending incremental file list
> .etc_openvpn/ipp.txt
> 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196)
> .etc_openvpn/openvpn-status.log
> 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196)
> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> ".etc_openvpn/ipp.txt": No space left on device (28)
> .log/
> .log/UcliEvt.log
> 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700)
> .log/auth.log
> 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700)
> .log/auth.log.1
> 19431424 61%7.35MB/s0:00:01
>
> the dmesg output looks like this:
> [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7 dumping space info
> [ 551.323694] space_info 4 has 6439526400 free, is full
> [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> reserved=49152, may_use=6438453248, readonly=65536
>
Ok so then this is probably it, let me know if it helps. Thanks,
Josef
diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c
index dc08d77..005c45d 100644
--- a/fs/btrfs/ordered-data.c
+++ b/fs/btrfs/ordered-data.c
@@ -557,6 +557,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root,
int delay_iput)
INIT_LIST_HEAD(&splice);
INIT_LIST_HEAD(&works);
+ mutex_lock(&root->fs_info->ordered_operations_mutex);
spin_lock(&root->fs_info->ordered_extent_lock);
list_splice_init(&root->fs_info->ordered_extents, &splice);
while (!list_empty(&splice)) {
@@ -600,6 +601,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root,
int delay_iput)
cond_resched();
}
+ mutex_unlock(&root->fs_info->ordered_operations_mutex);
}
/*
--
1.7.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Josef, Am 26.03.2013 14:30, schrieb Josef Bacik: > On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG wrote: >> Hi Josef, >> >> Am 26.03.2013 13:53, schrieb Josef Bacik: >>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote: Hi, output here: [ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0, reserved=73728, may_use=6438297600, readonly=65536 >>> >>> Weird, we have all the flushing stuff set and yet there is still a whole >>> lot of >>> outstanding reservations. Do you have compression enabled? Thanks, >> >> no - it's just mounted with mount -o noatime >> >> :~# cat /proc/mounts | grep btrfs >> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 >> > > Ok I think I see what's going on. Can you try this patch and see if it fixes > it? Thanks, It still does not fix the problem. The rsync output looks like this so it does not work for file a but then continues on c d e, ... sync -av --progress /backup/ /mnt/ sending incremental file list .etc_openvpn/ipp.txt 229 100%3.99kB/s0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100%6.28kB/s0:00:00 (xfer#3, to-check=1007/1196) rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/ .log/UcliEvt.log 104188 100% 147.67kB/s0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100%2.97MB/s0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61%7.35MB/s0:00:01 the dmesg output looks like this: [ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0, reserved=49152, may_use=6438453248, readonly=65536 Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
>
> Am 26.03.2013 13:53, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> >> Hi,
> >>
> >> output here:
> >> [ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7 dumping space info
> >> [ 590.548280] space_info 4 has 6439292928 free, is full
> >> [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0,
> >> reserved=32768, may_use=6438354944, readonly=65536
> >> [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7 dumping space info
> >> [ 590.552264] space_info 4 has 6439284736 free, is full
> >> [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0,
> >> reserved=40960, may_use=6438354944, readonly=65536
> >> [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7 dumping space info
> >> [ 590.556258] space_info 4 has 6439284736 free, is full
> >> [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0,
> >> reserved=40960, may_use=6438354944, readonly=65536
> >> [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7 dumping space info
> >> [ 591.147373] space_info 4 has 6439235584 free, is full
> >> [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0,
> >> reserved=90112, may_use=6438354944, readonly=65536
> >> [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7 dumping space info
> >> [ 595.562390] space_info 4 has 6439120896 free, is full
> >> [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,
> >> reserved=73728, may_use=6438297600, readonly=65536
> >>
> >
> > Weird, we have all the flushing stuff set and yet there is still a whole
> > lot of
> > outstanding reservations. Do you have compression enabled? Thanks,
>
> no - it's just mounted with mount -o noatime
>
> :~# cat /proc/mounts | grep btrfs
> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>
Ok I think I see what's going on. Can you try this patch and see if it fixes
it? Thanks,
Josef
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index bf6433f..84e8909 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3803,6 +3803,19 @@ static int can_overcommit(struct btrfs_root *root,
return 0;
}
+static int btrfs_try_writeback(struct super_block *sb, unsigned long nr_pages,
+ enum wb_reason reason)
+{
+ if (!writeback_in_progress(sb->s_bdi) &&
+ down_read_trylock(&sb->s_umount)) {
+ writeback_inodes_sb_nr(sb, nr_pages, reason);
+ up_read(&sb->s_umount);
+ return 1;
+ }
+
+ return 0;
+}
+
void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
unsigned long nr_pages)
{
@@ -3810,8 +3823,7 @@ void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
int started;
/* If we can not start writeback, just sync all the delalloc file. */
- started = try_to_writeback_inodes_sb_nr(sb, nr_pages,
- WB_REASON_FS_FREE_SPACE);
+ started = btrfs_try_writeback(sb, nr_pages, WB_REASON_FS_FREE_SPACE);
if (!started) {
/*
* We needn't worry the filesystem going from r/w to r/o though
--
1.7.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Josef, Am 26.03.2013 13:53, schrieb Josef Bacik: > On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote: >> Hi, >> >> output here: >> [ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush >> 2, flush_state 7 dumping space info >> [ 590.548280] space_info 4 has 6439292928 free, is full >> [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, >> reserved=32768, may_use=6438354944, readonly=65536 >> [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush >> 2, flush_state 7 dumping space info >> [ 590.552264] space_info 4 has 6439284736 free, is full >> [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, >> reserved=40960, may_use=6438354944, readonly=65536 >> [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush >> 2, flush_state 7 dumping space info >> [ 590.556258] space_info 4 has 6439284736 free, is full >> [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, >> reserved=40960, may_use=6438354944, readonly=65536 >> [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush >> 2, flush_state 7 dumping space info >> [ 591.147373] space_info 4 has 6439235584 free, is full >> [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, >> reserved=90112, may_use=6438354944, readonly=65536 >> [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush >> 2, flush_state 7 dumping space info >> [ 595.562390] space_info 4 has 6439120896 free, is full >> [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0, >> reserved=73728, may_use=6438297600, readonly=65536 >> > > Weird, we have all the flushing stuff set and yet there is still a whole lot > of > outstanding reservations. Do you have compression enabled? Thanks, no - it's just mounted with mount -o noatime :~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0 Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote: > Hi, > > output here: > [ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 590.548280] space_info 4 has 6439292928 free, is full > [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, > reserved=32768, may_use=6438354944, readonly=65536 > [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 590.552264] space_info 4 has 6439284736 free, is full > [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, > reserved=40960, may_use=6438354944, readonly=65536 > [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 590.556258] space_info 4 has 6439284736 free, is full > [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, > reserved=40960, may_use=6438354944, readonly=65536 > [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 591.147373] space_info 4 has 6439235584 free, is full > [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, > reserved=90112, may_use=6438354944, readonly=65536 > [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush > 2, flush_state 7 dumping space info > [ 595.562390] space_info 4 has 6439120896 free, is full > [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0, > reserved=73728, may_use=6438297600, readonly=65536 > Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7 dumping space info
[ 590.548280] space_info 4 has 6439292928 free, is full
[ 590.548283] space_info total=25748307968, used=19308916736, pinned=0,
reserved=32768, may_use=6438354944, readonly=65536
[ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7 dumping space info
[ 590.552264] space_info 4 has 6439284736 free, is full
[ 590.552267] space_info total=25748307968, used=19308916736, pinned=0,
reserved=40960, may_use=6438354944, readonly=65536
[ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7 dumping space info
[ 590.556258] space_info 4 has 6439284736 free, is full
[ 590.556261] space_info total=25748307968, used=19308916736, pinned=0,
reserved=40960, may_use=6438354944, readonly=65536
[ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7 dumping space info
[ 591.147373] space_info 4 has 6439235584 free, is full
[ 591.147375] space_info total=25748307968, used=19308916736, pinned=0,
reserved=90112, may_use=6438354944, readonly=65536
[ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7 dumping space info
[ 595.562390] space_info 4 has 6439120896 free, is full
[ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,
reserved=73728, may_use=6438297600, readonly=65536
Am 25.03.2013 21:14, schrieb Josef Bacik:
On Fri, Mar 22, 2013 at 02:55:07PM -0600, Stefan Priebe wrote:
Hi Jsoef,
thanks!
Ok I don't think the thing I just fixed will make any difference for you so
here's another debug patch, just apply it on top of what I've already sent you
and re-run and give me the dmesg again. Thanks,
Josef
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index aabaea6..bf6433f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4162,7 +4162,11 @@ out:
}
if (flushing) {
if (ret == -ENOSPC) {
- printk(KERN_ERR "returning enospc, dumping space
info\n");
+ spin_lock(&block_rsv->lock);
+ printk(KERN_ERR "returning enospc, space_info %u, size %Lu
reserved %Lu, "
+ "flush %d, flush_state %d dumping space info\n",
block_rsv->type,
+ block_rsv->size, block_rsv->reserved, flush,
flush_state);
+ spin_unlock(&block_rsv->lock);
dump_space_info(space_info, 0, 0);
}
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Fri, Mar 22, 2013 at 02:55:07PM -0600, Stefan Priebe wrote:
> Hi Jsoef,
>
> thanks!
Ok I don't think the thing I just fixed will make any difference for you so
here's another debug patch, just apply it on top of what I've already sent you
and re-run and give me the dmesg again. Thanks,
Josef
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index aabaea6..bf6433f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4162,7 +4162,11 @@ out:
}
if (flushing) {
if (ret == -ENOSPC) {
- printk(KERN_ERR "returning enospc, dumping space
info\n");
+ spin_lock(&block_rsv->lock);
+ printk(KERN_ERR "returning enospc, space_info %u, size
%Lu reserved %Lu, "
+ "flush %d, flush_state %d dumping space
info\n", block_rsv->type,
+ block_rsv->size, block_rsv->reserved, flush,
flush_state);
+ spin_unlock(&block_rsv->lock);
dump_space_info(space_info, 0, 0);
}
--
1.7.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Jsoef, thanks! Am 22.03.2013 21:49, schrieb Josef Bacik: On Fri, Mar 22, 2013 at 01:10:05PM -0600, Stefan Priebe wrote: Hi Josef, Am 22.03.2013 16:54, schrieb Josef Bacik: On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote: Hi Josef, Am 22.03.2013 14:53, schrieb Josef Bacik: On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote: Hi Chris, Which kernel are you running? -chris vanilla 3.8.3. Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel? any ideas what i can check? 3.9-rc3 gives me same results. Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks, Great! Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback. Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks, Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some. Ok so I've been working on another ENOSPC related problem that I think is affecting you as well. So I'm going to nail that down and see if that helps you too, if not we'll poke around your problem some more and try and work out what's happening. I'll get back to this fresh Monday morning. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Fri, Mar 22, 2013 at 01:10:05PM -0600, Stefan Priebe wrote: > Hi Josef, > Am 22.03.2013 16:54, schrieb Josef Bacik: > > On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG > > wrote: > >> Hi Josef, > >> Am 22.03.2013 14:53, schrieb Josef Bacik: > >>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG > >>> wrote: > Hi Chris, > > > Which kernel are you running? > > > > -chris > > vanilla 3.8.3. > >>> > >>> Ok, with the 3.9 merge window Josef changed how we do the > >>> reservations. > >>> Are you able to try a slightly more experimental kernel? > > any ideas what i can check? 3.9-rc3 gives me same results. > > >>> > >>> Sorry Stefan I'm almost done with what I'm working on and then I'll work > >>> up a > >>> patch for you to run so I can narrow down what's going on. Thanks, > >> > >> Great! > >> > >> Thanks - just wanted to know that it's not my fault. I'm happy to test > >> the patch and provide feedback. > >> > > Ok I think we are way over-reserving for rename, can you give this patch a > > whirl > > and see what happens? If it still fails can you capture dmesg and reply > > with > > that so I can see what's going on. Thanks, > > Thanks for the patch. I was able to copy some files and i'm still able > put it copy some then fails on some then copies some then fails on some. > Ok so I've been working on another ENOSPC related problem that I think is affecting you as well. So I'm going to nail that down and see if that helps you too, if not we'll poke around your problem some more and try and work out what's happening. I'll get back to this fresh Monday morning. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Josef, Am 22.03.2013 16:54, schrieb Josef Bacik: On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote: Hi Josef, Am 22.03.2013 14:53, schrieb Josef Bacik: On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote: Hi Chris, Which kernel are you running? -chris vanilla 3.8.3. Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel? any ideas what i can check? 3.9-rc3 gives me same results. Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks, Great! Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback. Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks, Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some. Rsync output: .software/kernel/linux-3.9-rc3/drivers/rtc/rtc-wm8350.c 12156 100% 11.17kB/s0:00:01 (xfer#21395, to-check=1008/52625) .software/kernel/linux-3.9-rc3/drivers/rtc/rtc-x1205.c 16460 100% 15.05kB/s0:00:01 (xfer#21396, to-check=1007/52625) .software/kernel/linux-3.9-rc3/drivers/rtc/systohc.c 1223 100%1.12kB/s0:00:01 (xfer#21397, to-check=1006/52625) .software/kernel/linux-3.9-rc3/drivers/s390/ .software/kernel/linux-3.9-rc3/drivers/s390/Makefile 144 100%0.13kB/s0:00:01 (xfer#21398, to-check=1052/52672) .software/kernel/linux-3.9-rc3/drivers/s390/block/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/block" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/char/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/char" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/cio/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/cio" failed: No space left on device (28) rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.88pm8607.c.YBi0Rz" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/crypto/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/crypto" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/kvm/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/kvm" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/net/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/net" failed: No space left on device (28) rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Kconfig.0lsfuE" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Makefile.C9wI7I" failed: No space left on device (28) .software/kernel/linux-3.9-rc3/drivers/s390/scsi/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/scsi" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/sbus/ .software/kernel/linux-3.9-rc3/drivers/sbus/Makefile 70 100%0.06kB/s0:00:01 (xfer#21399, to-check=1003/52824) .software/kernel/linux-3.9-rc3/drivers/sbus/char/ rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/sbus/char" failed: No space left on device (28) rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.aat2870-regulator.c.BGDwPN" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/scsi/ .software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.ko.cmd 208 100%0.16kB/s0:00:01 (xfer#21400, to-check=1407/53242) .software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.mod.o.cmd 27987 100% 21.54kB/s0:00:01 (xfer#21401, to-check=1406/53242) .software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.o.cmd 43340 100% 32.63kB/s0:00:01 (xfer#21402, to-check=1405/53242) .software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.ko.cmd 204 100% 15.32kB/s0:00:00 (xfer#21403, to-check=1404/53242) .software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.mod.o.cmd 27975 100%1.91MB/s0:00:00 (xfer#21404, to-check=1403/53242) .software/
Re: No space left on device (28)
On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
> Am 22.03.2013 14:53, schrieb Josef Bacik:
> > On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG
> > wrote:
> >> Hi Chris,
> >>
> >>> Which kernel are you running?
> >>>
> >>> -chris
> >>
> >> vanilla 3.8.3.
> >
> > Ok, with the 3.9 merge window Josef changed how we do the reservations.
> > Are you able to try a slightly more experimental kernel?
> >>
> >> any ideas what i can check? 3.9-rc3 gives me same results.
> >>
> >
> > Sorry Stefan I'm almost done with what I'm working on and then I'll work up
> > a
> > patch for you to run so I can narrow down what's going on. Thanks,
>
> Great!
>
> Thanks - just wanted to know that it's not my fault. I'm happy to test
> the patch and provide feedback.
>
Ok I think we are way over-reserving for rename, can you give this patch a whirl
and see what happens? If it still fails can you capture dmesg and reply with
that so I can see what's going on. Thanks,
Josef
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 9ac2eca..aabaea6 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4161,6 +4161,11 @@ out:
ret = 0;
}
if (flushing) {
+ if (ret == -ENOSPC) {
+ printk(KERN_ERR "returning enospc, dumping space
info\n");
+ dump_space_info(space_info, 0, 0);
+ }
+
spin_lock(&space_info->lock);
space_info->flush = 0;
wake_up_all(&space_info->wait);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index ca1b767..3980ae7 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3679,11 +3679,9 @@ static struct btrfs_trans_handle
*__unlink_start_trans(struct inode *dir,
* 1 for the dir item
* 1 for the dir index
* 1 for the inode ref
-* 1 for the inode ref in the tree log
-* 2 for the dir entries in the log
* 1 for the inode
*/
- trans = btrfs_start_transaction(root, 8);
+ trans = btrfs_start_transaction(root, 5);
if (!IS_ERR(trans) || PTR_ERR(trans) != -ENOSPC)
return trans;
@@ -8127,7 +8125,7 @@ static int btrfs_rename(struct inode *old_dir, struct
dentry *old_dentry,
* inodes. So 5 * 2 is 10, plus 1 for the new link, so 11 total items
* should cover the worst case number of items we'll modify.
*/
- trans = btrfs_start_transaction(root, 20);
+ trans = btrfs_start_transaction(root, 11);
if (IS_ERR(trans)) {
ret = PTR_ERR(trans);
goto out_notrans;
--
1.7.7.6
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Josef, Am 22.03.2013 14:53, schrieb Josef Bacik: > On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote: >> Hi Chris, >> >>> Which kernel are you running? >>> >>> -chris >> >> vanilla 3.8.3. > > Ok, with the 3.9 merge window Josef changed how we do the reservations. > Are you able to try a slightly more experimental kernel? >> >> any ideas what i can check? 3.9-rc3 gives me same results. >> > > Sorry Stefan I'm almost done with what I'm working on and then I'll work up a > patch for you to run so I can narrow down what's going on. Thanks, Great! Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback. Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote: > Hi Chris, > > > Which kernel are you running? > > > > -chris > > vanilla 3.8.3. > >>> > >>> Ok, with the 3.9 merge window Josef changed how we do the reservations. > >>> Are you able to try a slightly more experimental kernel? > > any ideas what i can check? 3.9-rc3 gives me same results. > Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Chris, > Which kernel are you running? > > -chris vanilla 3.8.3. >>> >>> Ok, with the 3.9 merge window Josef changed how we do the reservations. >>> Are you able to try a slightly more experimental kernel? any ideas what i can check? 3.9-rc3 gives me same results. Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi, Am 22.03.2013 07:41, schrieb cwillu:> On Fri, Mar 22, 2013 at 12:39 AM, Stefan Priebe - Profihost AG > wrote: >> Already tried with value 5 did not help ;-( and it also happens with >> plain cp copying a 15gb file and aborts at about 80% > > You tried -musage=5? Your original email said -dusage=5. sorry i missed the difference - but it does not work: ~# btrfs fi balance start -musage=5 /mnt/ ERROR: error during balancing '/mnt/' - No space left on device There may be more info in syslog - try dmesg | tail ~# dmesg|tail [ 1183.019367] device fsid fc23c4a8-a351-4c91-96a2-ee6abeffe59a devid 1 transid 6224 /dev/mapper/raid54tb1 [ 1183.019860] btrfs: disk space caching is enabled [42719.915781] btrfs: relocating block group 4194304 flags 4 [42727.916141] btrfs: 5 enospc errors during balance Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Fri, Mar 22, 2013 at 12:39 AM, Stefan Priebe - Profihost AG wrote: > Already tried with value 5 did not help ;-( and it also happens with plain cp > copying a 15gb file and aborts at about 80% You tried -musage=5? Your original email said -dusage=5. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Already tried with value 5 did not help ;-( and it also happens with plain cp
copying a 15gb file and aborts at about 80%
Am 22.03.2013 um 07:24 schrieb cwillu :
> On Fri, Mar 22, 2013 at 12:13 AM, Roman Mamedov wrote:
>> On Thu, 21 Mar 2013 20:42:28 +0100
>> Stefan Priebe wrote:
>>
>> I might be wrong here, but doesn't this
>>
>>> rsync: rename
>>> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/""
>>> ->
>>> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
>>
>> ...try to move a file from
>>
>> "/mnt/.software/"
>>
>> to
>>
>> ".software/"
>>
>> (relative to current dir)??
>
> No; that's rsync giving the full path, and then the target path
> relative to the command it was given. The filename itself
> (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to
> temporarily, so it can mv it over the original in an atomic fashion...
>
> Stefan: ...which means that the actual copy succeeded, which suggests
> that this is more of a metadata enospc thing.
>
> You might try btrfs balance start -musage=5 (instead of -dusage), and
> if that doesn't report any chunks balanced, try a high number until it
> does.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
No this is fine it's rsync style same happens with cp. Am 22.03.2013 um 07:13 schrieb Roman Mamedov : > On Thu, 21 Mar 2013 20:42:28 +0100 > Stefan Priebe wrote: > > I might be wrong here, but doesn't this > >> rsync: rename >> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" >> >> -> >> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": > > ...try to move a file from > > "/mnt/.software/" > > to > > ".software/" > > (relative to current dir)?? > > Maybe you end up trying to rsync your array to a small root partition or > something? > >> No space left on device (28) >> >> # btrfs filesystem df /mnt/ >> Data: total=18.14TB, used=15.05TB >> System, DUP: total=8.00MB, used=1.94MB >> System: total=4.00MB, used=0.00 >> Metadata, DUP: total=23.98GB, used=17.98GB >> Metadata: total=8.00MB, used=0.00 >> >> Stefan >> >> Am 21.03.2013 20:02, schrieb Chris Mason: >>> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59) Hi Chris, Am 21.03.2013 um 19:00 schrieb Chris Mason : > Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) >> Hi, >> >> while copying a 15GB file to my btrfs volume i'm getting "No space left >> on device". >> >> Some information: >> >> # btrfs filesystem df /mnt/ >> Data: total=18.14TB, used=15.05TB >> System, DUP: total=8.00MB, used=1.94MB >> System: total=4.00MB, used=0.00 >> Metadata, DUP: total=23.98GB, used=17.97GB >> Metadata: total=8.00MB, used=0.00 >> >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt >> >> # btrfs fi show >> Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a >>Total devices 1 FS bytes used 15.07TB >>devid1 size 18.19TB used 18.19TB path /dev/dm-3 >> >> # btrfs fi balance start -dusage=5 /mnt/ >> Done, had to relocate 0 out of 18604 chunks >> >> What wrong here? What is about the last 3TB? > > Which kernel are you running? > > -chris vanilla 3.8.3. >>> >>> Ok, with the 3.9 merge window Josef changed how we do the reservations. >>> Are you able to try a slightly more experimental kernel? >>> >>> -chris >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- > With respect, > Roman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Fri, Mar 22, 2013 at 12:13 AM, Roman Mamedov wrote:
> On Thu, 21 Mar 2013 20:42:28 +0100
> Stefan Priebe wrote:
>
> I might be wrong here, but doesn't this
>
>> rsync: rename
>> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/""
>> ->
>> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
>
> ...try to move a file from
>
> "/mnt/.software/"
>
> to
>
> ".software/"
>
> (relative to current dir)??
No; that's rsync giving the full path, and then the target path
relative to the command it was given. The filename itself
(".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to
temporarily, so it can mv it over the original in an atomic fashion...
Stefan: ...which means that the actual copy succeeded, which suggests
that this is more of a metadata enospc thing.
You might try btrfs balance start -musage=5 (instead of -dusage), and
if that doesn't report any chunks balanced, try a high number until it
does.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
On Thu, 21 Mar 2013 20:42:28 +0100 Stefan Priebe wrote: I might be wrong here, but doesn't this > rsync: rename > "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" > > -> > ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": ...try to move a file from "/mnt/.software/" to ".software/" (relative to current dir)?? Maybe you end up trying to rsync your array to a small root partition or something? > No space left on device (28) > > # btrfs filesystem df /mnt/ > Data: total=18.14TB, used=15.05TB > System, DUP: total=8.00MB, used=1.94MB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=23.98GB, used=17.98GB > Metadata: total=8.00MB, used=0.00 > > Stefan > > Am 21.03.2013 20:02, schrieb Chris Mason: > > Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59) > >> Hi Chris, > >> > >> Am 21.03.2013 um 19:00 schrieb Chris Mason : > >> > >>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) > Hi, > > while copying a 15GB file to my btrfs volume i'm getting "No space left > on device". > > Some information: > > # btrfs filesystem df /mnt/ > Data: total=18.14TB, used=15.05TB > System, DUP: total=8.00MB, used=1.94MB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=23.98GB, used=17.97GB > Metadata: total=8.00MB, used=0.00 > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt > > # btrfs fi show > Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a > Total devices 1 FS bytes used 15.07TB > devid1 size 18.19TB used 18.19TB path /dev/dm-3 > > # btrfs fi balance start -dusage=5 /mnt/ > Done, had to relocate 0 out of 18604 chunks > > What wrong here? What is about the last 3TB? > >>> > >>> Which kernel are you running? > >>> > >>> -chris > >> > >> vanilla 3.8.3. > > > > Ok, with the 3.9 merge window Josef changed how we do the reservations. > > Are you able to try a slightly more experimental kernel? > > > > -chris > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- With respect, Roman signature.asc Description: PGP signature
Re: No space left on device (28)
Should i try to downgrade? Greets, Stefan Am 21.03.2013 um 20:42 schrieb Stefan Priebe : > Yes, i've now upgraded to 3.9-rc3 same result. > > rsync: rename > "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" > -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": > No space left on device (28) > > # btrfs filesystem df /mnt/ > Data: total=18.14TB, used=15.05TB > System, DUP: total=8.00MB, used=1.94MB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=23.98GB, used=17.98GB > Metadata: total=8.00MB, used=0.00 > > Stefan > > Am 21.03.2013 20:02, schrieb Chris Mason: >> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59) >>> Hi Chris, >>> >>> Am 21.03.2013 um 19:00 schrieb Chris Mason : >>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) > Hi, > > while copying a 15GB file to my btrfs volume i'm getting "No space left > on device". > > Some information: > > # btrfs filesystem df /mnt/ > Data: total=18.14TB, used=15.05TB > System, DUP: total=8.00MB, used=1.94MB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=23.98GB, used=17.97GB > Metadata: total=8.00MB, used=0.00 > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt > > # btrfs fi show > Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a >Total devices 1 FS bytes used 15.07TB >devid1 size 18.19TB used 18.19TB path /dev/dm-3 > > # btrfs fi balance start -dusage=5 /mnt/ > Done, had to relocate 0 out of 18604 chunks > > What wrong here? What is about the last 3TB? Which kernel are you running? -chris >>> >>> vanilla 3.8.3. >> >> Ok, with the 3.9 merge window Josef changed how we do the reservations. >> Are you able to try a slightly more experimental kernel? >> >> -chris >> -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Yes, i've now upgraded to 3.9-rc3 same result. rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": No space left on device (28) # btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.98GB Metadata: total=8.00MB, used=0.00 Stefan Am 21.03.2013 20:02, schrieb Chris Mason: Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59) Hi Chris, Am 21.03.2013 um 19:00 schrieb Chris Mason : Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) Hi, while copying a 15GB file to my btrfs volume i'm getting "No space left on device". Some information: # btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00 # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt # btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid1 size 18.19TB used 18.19TB path /dev/dm-3 # btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunks What wrong here? What is about the last 3TB? Which kernel are you running? -chris vanilla 3.8.3. Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59) > Hi Chris, > > Am 21.03.2013 um 19:00 schrieb Chris Mason : > > > Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) > >> Hi, > >> > >> while copying a 15GB file to my btrfs volume i'm getting "No space left > >> on device". > >> > >> Some information: > >> > >> # btrfs filesystem df /mnt/ > >> Data: total=18.14TB, used=15.05TB > >> System, DUP: total=8.00MB, used=1.94MB > >> System: total=4.00MB, used=0.00 > >> Metadata, DUP: total=23.98GB, used=17.97GB > >> Metadata: total=8.00MB, used=0.00 > >> > >> # df -h > >> Filesystem Size Used Avail Use% Mounted on > >> /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt > >> > >> # btrfs fi show > >> Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a > >>Total devices 1 FS bytes used 15.07TB > >>devid1 size 18.19TB used 18.19TB path /dev/dm-3 > >> > >> # btrfs fi balance start -dusage=5 /mnt/ > >> Done, had to relocate 0 out of 18604 chunks > >> > >> What wrong here? What is about the last 3TB? > > > > Which kernel are you running? > > > > -chris > > vanilla 3.8.3. Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Hi Chris, Am 21.03.2013 um 19:00 schrieb Chris Mason : > Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) >> Hi, >> >> while copying a 15GB file to my btrfs volume i'm getting "No space left >> on device". >> >> Some information: >> >> # btrfs filesystem df /mnt/ >> Data: total=18.14TB, used=15.05TB >> System, DUP: total=8.00MB, used=1.94MB >> System: total=4.00MB, used=0.00 >> Metadata, DUP: total=23.98GB, used=17.97GB >> Metadata: total=8.00MB, used=0.00 >> >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt >> >> # btrfs fi show >> Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a >>Total devices 1 FS bytes used 15.07TB >>devid1 size 18.19TB used 18.19TB path /dev/dm-3 >> >> # btrfs fi balance start -dusage=5 /mnt/ >> Done, had to relocate 0 out of 18604 chunks >> >> What wrong here? What is about the last 3TB? > > Which kernel are you running? > > -chris vanilla 3.8.3. Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: No space left on device (28)
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04) > Hi, > > while copying a 15GB file to my btrfs volume i'm getting "No space left > on device". > > Some information: > > # btrfs filesystem df /mnt/ > Data: total=18.14TB, used=15.05TB > System, DUP: total=8.00MB, used=1.94MB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=23.98GB, used=17.97GB > Metadata: total=8.00MB, used=0.00 > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt > > # btrfs fi show > Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a > Total devices 1 FS bytes used 15.07TB > devid1 size 18.19TB used 18.19TB path /dev/dm-3 > > # btrfs fi balance start -dusage=5 /mnt/ > Done, had to relocate 0 out of 18604 chunks > > What wrong here? What is about the last 3TB? Which kernel are you running? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: no space left on device.
On 11/02/2012 04:54 PM, Kyle Gates wrote: So I have ended up in a state where I can't delete files with rm. the error I get is no space on device. however I'm not even close to empty. /dev/sdb1 38G 27G 9.5G 75% there is about 800k files/dirs in this filesystem extra strange is that I can in another directory create and delete files. So I tried pretty much all I could google my way to but problem persisted. So I decided to do a backup and a format. But when the backup was done I tried one more time and now it was possible to delete the directory and all content? using the 3.5 kernel in ubuntu 12.10. Is this a known issue ? is it fixed in later kernels? fsck /btrfs scrub and kernel log. nothing indicate any problem of any kind. First let's see the output of: btrfs fi df /mountpoint You're probably way over allocated in metadata so a balance should help: btrfs bal start -m /mountpoint or omit the -m option to run a full balance. I had to use the disk for work so I could not be stuck in that situation and thus had to nuke the disk. maybe I can recreate the state I put back mostly the same data again minus some stuff I did not really need. So under what circumstance can this actually happen ? why could I remove/add files in one directory and not another? its the same partition. And also the filesystem was just a few days old and no snapshots or anything, well I do use lzo compression but other than that default values was used to create it. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
