Re: ENOSPC errors during balance
Am Tue, 22 Jul 2014 03:26:39 + (UTC) schrieb Duncan 1i5t5.dun...@cox.net: Marc Joliet posted on Tue, 22 Jul 2014 01:30:22 +0200 as excerpted: And now that the background deletion of the old snapshots is done, the file system ended up at: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=219.00GiB, used=140.13GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.50GiB, used=2.40GiB unknown, single: total=512.00MiB, used=0.00 I don't know how reliable du is for this, but I used it to estimate how much used data I should expect, and I get 138 GiB. That means that the snapshots yield about 2 GiB overhead, which is very reasonable, I think. Obviously I'll be starting a full balance now. [snip total/used discussion] No, you misunderstand: read my email three steps above yours (from the 21. at 15:22). I am wondering about why the disk usage ballooned to 200 GiB in the first place. -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
On 20/07/14 14:59, Duncan wrote: Marc Joliet posted on Sun, 20 Jul 2014 12:22:33 +0200 as excerpted: On the other hand, the wiki [0] says that defragmentation (and balancing) is optional, and the only reason stated for doing either is because they will have impact on performance. Yes. That's what threw off the other guy as well. He decided to skip it for the same reason. If I had a wiki account I'd change it, but for whatever reason I tend to be far more comfortable writing list replies, sometimes repeatedly, than writing anything on the web, which I tend to treat as read-only. So I've never gotten a wiki account and thus haven't changed it, and apparently the other guy with the problem and anyone else that knows hasn't changed it either, so the conversion page still continues to underemphasize the importance of completing the conversion steps, including the defrag, in proper order. I've inserted information specific to this in the wiki. Others with wiki accounts, feel free to review: https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3#Before_first_use -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during balance
Am Sun, 20 Jul 2014 21:44:40 +0200 schrieb Marc Joliet mar...@gmx.de: [...] What I did: - delete the single largest file on the file system, a 12 GB VM image, along with all subvolumes that contained it - rsync it over again [...] I want to point out at this point, though, that doing those two steps freed a disproportionate amount of space. The image file is only 12 GB, and it hadn't changed in any of the snapshots (I haven't used this VM since June), so that subvolume delete -c snapshots returned after a few seconds. Yet deleting it seems to have freed up twice as much. You can see this from the filesystem df output: before, used was at 229.04 GiB, and after deleting it and copying it back (and after a day's worth of backups) went down to 218 GiB. Does anyone have any idea how this happened? Actually, now I remember something that is probably related: when I first moved to my current backup scheme last week, I first copied the data from the last rsnapshot based backup with cp --reflink to the new backup location, but forgot to use -a. I interrupted it and ran cp -a -u --reflink, but it had already copied a lot, and I was too impatient to start over; after all, the data hadn't changed. Then, when rsync (with --inplace) ran for the first time, all of these files with wrong permissions and different time stamps were copied over, but for some reason, the space used increased *greatly*; *much* more than I would expect from changed metadata. The total size of the file system data should be around 142 GB (+ snapshots), but, well, it's more than 1.5 times as much. Perhaps cp --reflink treats hard links differently than expected? I would have expected the data pointed to by the hard link to have been referenced, but maybe something else happened? -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
Am Mon, 21 Jul 2014 15:22:16 +0200 schrieb Marc Joliet mar...@gmx.de: Am Sun, 20 Jul 2014 21:44:40 +0200 schrieb Marc Joliet mar...@gmx.de: [...] What I did: - delete the single largest file on the file system, a 12 GB VM image, along with all subvolumes that contained it - rsync it over again [...] I want to point out at this point, though, that doing those two steps freed a disproportionate amount of space. The image file is only 12 GB, and it hadn't changed in any of the snapshots (I haven't used this VM since June), so that subvolume delete -c snapshots returned after a few seconds. Yet deleting it seems to have freed up twice as much. You can see this from the filesystem df output: before, used was at 229.04 GiB, and after deleting it and copying it back (and after a day's worth of backups) went down to 218 GiB. Does anyone have any idea how this happened? Actually, now I remember something that is probably related: when I first moved to my current backup scheme last week, I first copied the data from the last rsnapshot based backup with cp --reflink to the new backup location, but forgot to use -a. I interrupted it and ran cp -a -u --reflink, but it had already copied a lot, and I was too impatient to start over; after all, the data hadn't changed. Then, when rsync (with --inplace) ran for the first time, all of these files with wrong permissions and different time stamps were copied over, but for some reason, the space used increased *greatly*; *much* more than I would expect from changed metadata. The total size of the file system data should be around 142 GB (+ snapshots), but, well, it's more than 1.5 times as much. Perhaps cp --reflink treats hard links differently than expected? I would have expected the data pointed to by the hard link to have been referenced, but maybe something else happened? Hah, OK, apparently when my daily backup removed the oldest daily snapshot, it freed up whatever was taking up so much space, so as of now the file system uses only 169.14 GiB (from 218). Weird. -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
Am Tue, 22 Jul 2014 00:30:57 +0200 schrieb Marc Joliet mar...@gmx.de: Am Mon, 21 Jul 2014 15:22:16 +0200 schrieb Marc Joliet mar...@gmx.de: Am Sun, 20 Jul 2014 21:44:40 +0200 schrieb Marc Joliet mar...@gmx.de: [...] What I did: - delete the single largest file on the file system, a 12 GB VM image, along with all subvolumes that contained it - rsync it over again [...] I want to point out at this point, though, that doing those two steps freed a disproportionate amount of space. The image file is only 12 GB, and it hadn't changed in any of the snapshots (I haven't used this VM since June), so that subvolume delete -c snapshots returned after a few seconds. Yet deleting it seems to have freed up twice as much. You can see this from the filesystem df output: before, used was at 229.04 GiB, and after deleting it and copying it back (and after a day's worth of backups) went down to 218 GiB. Does anyone have any idea how this happened? Actually, now I remember something that is probably related: when I first moved to my current backup scheme last week, I first copied the data from the last rsnapshot based backup with cp --reflink to the new backup location, but forgot to use -a. I interrupted it and ran cp -a -u --reflink, but it had already copied a lot, and I was too impatient to start over; after all, the data hadn't changed. Then, when rsync (with --inplace) ran for the first time, all of these files with wrong permissions and different time stamps were copied over, but for some reason, the space used increased *greatly*; *much* more than I would expect from changed metadata. The total size of the file system data should be around 142 GB (+ snapshots), but, well, it's more than 1.5 times as much. Perhaps cp --reflink treats hard links differently than expected? I would have expected the data pointed to by the hard link to have been referenced, but maybe something else happened? Hah, OK, apparently when my daily backup removed the oldest daily snapshot, it freed up whatever was taking up so much space, so as of now the file system uses only 169.14 GiB (from 218). Weird. And now that the background deletion of the old snapshots is done, the file system ended up at: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=219.00GiB, used=140.13GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.50GiB, used=2.40GiB unknown, single: total=512.00MiB, used=0.00 I don't know how reliable du is for this, but I used it to estimate how much used data I should expect, and I get 138 GiB. That means that the snapshots yield about 2 GiB overhead, which is very reasonable, I think. Obviously I'll be starting a full balance now. I still think this whole... thing is very odd, hopefully somebody can shed some light on it for me (maybe it's obvious, but I don't see it). -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
Marc Joliet posted on Tue, 22 Jul 2014 01:30:22 +0200 as excerpted: And now that the background deletion of the old snapshots is done, the file system ended up at: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=219.00GiB, used=140.13GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.50GiB, used=2.40GiB unknown, single: total=512.00MiB, used=0.00 I don't know how reliable du is for this, but I used it to estimate how much used data I should expect, and I get 138 GiB. That means that the snapshots yield about 2 GiB overhead, which is very reasonable, I think. Obviously I'll be starting a full balance now. FWIW, the balance should reduce the data total quite a bit, to 141-ish GiB (might be 142 or 145, but it should definitely come down from 219 GiB), because the spread between total and used is relatively high, now, and balance is what's used to bring that back down. Metadata total will probably come down a bit as well, to 3.00 GiB or so. What's going on there is this: Btrfs allocates and deallocates data and metadata in two stages. First it allocates chunks, 1 GiB in size for data, 256 MiB in size for metadata, but because metadata is dup by default it allocates two chunks so half a GiB at a time, there. Then the actual file data and metadata can be written into the pre-allocated chunks, filling them up. As they near full, more chunks will be allocated from the unallocated pool as necessary. But on file deletion, btrfs only automatically handles the file data/metadata level; it doesn't (yet) automatically deallocate the chunks, nor can it change the allocation from say a data chunk to a metadata chunk. So when a chunk is allocated, it stays allocated. That's the spread you see in btrfs filesystem df, between total and used, for each chunk type. The way to recover those allocated but unused chunks to the unallocated pool, so they can be reallocated between data and metadata as necessary, is with a balance. That balance, therefore, should reduce the spread seen in the above between total and used. Meanwhile, btrfs filesystem df shows the spread between allocated and used for each type, but what about unallocated? Simple. Btrfs filesystem show lists total filesystem size as well as allocated usage for each device. (The total line is something else, I recommend ignoring it as it's simply confusing. Only pay attention to the individual device lines.) Thus, to get a proper picture of the space usage status on a btrfs filesystem, you must have both the btrfs filesystem show and btrfs filesystem df output for that filesystem, show to tell you how much of the total space is chunk-allocated for each device, df to tell you what those allocations are, and how much of the chunk-allocated space is actually used, for each allocation type. It's wise to keep track of the show output in particular, and when the spread between used (allocated) and total for each device gets low, under a few GiB, check btrfs fi df and see what's using that space unnecessarily and then do a balance to recover it, if possible. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during balance
Am Sat, 19 Jul 2014 19:11:00 -0600 schrieb Chris Murphy li...@colorremedies.com: I'm seeing this also in the 2nd dmesg: [ 249.893310] BTRFS error (device sdg2): free space inode generation (0) did not match free space cache generation (26286) So you could try umounting the volume. And doing a one time mount with the clear_cache mount option. Give it some time to rebuild the space cache. After that you could umount again, and mount with enospc_debug and try to reproduce the enospc with another balance and see if dmesg contains more information this time. OK, I did that, and the new dmesg is attached. Also, some outputs again, first filesystem df (that total surge at the end sure is consistent): # btrfs filesystem df /mnt Data, single: total=237.00GiB, used=229.67GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.50GiB, used=3.49GiB unknown, single: total=512.00MiB, used=0.00 And here what I described in my initial post, the output of balance status immediately after the error (turns out my memory was correct): btrfs filesystem balance status /mnt Balance on '/mnt' is running 0 out of about 0 chunks balanced (0 considered), -nan% left (Also, this is with Gentoo kernel 3.15.6 now.) -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup dmesg4.log.xz Description: application/xz signature.asc Description: PGP signature
Re: ENOSPC errors during balance
Am Sat, 19 Jul 2014 18:53:03 -0600 schrieb Chris Murphy li...@colorremedies.com: On Jul 19, 2014, at 2:58 PM, Marc Joliet mar...@gmx.de wrote: Am Sat, 19 Jul 2014 22:10:51 +0200 schrieb Marc Joliet mar...@gmx.de: [...] Another random idea: the number of errors decreased the second time I ran balance (from 4 to 2), I could run another full balance and see if it keeps decreasing. Well, this time there were still 2 ENOSPC errors. But I can show the df output after such an ENOSPC error, to illustrate what I meant with the sudden surge in total usage: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=236.00GiB, used=229.04GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.00GiB, used=3.20GiB unknown, single: total=512.00MiB, used=0.00 And then after running a balance and (almost) immediately cancelling: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=230.00GiB, used=229.04GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.00GiB, used=3.20GiB unknown, single: total=512.00MiB, used=0.00 I think it's a bit weird. Two options: a. Keep using the file system, with judicious backups, if a dev wants more info they'll reply to the thread; b. Migrate the data to a new file system, first capture the file system with btrfs-image in case a dev wants more info and you've since blown away the filesystem, and then move it to a new btrfs fs. I'd use send/receive for this to preserve subvolumes and snapshots. OK, I'll keep that in mind. I'll keep running the file system for now, just in case it's a run-time error (i.e., a bug in the balance code, and not a problem with the file system itself). If it gets trashed on its own, or I move to a new file system, I'll be sure to follow the steps you outlined. Chris Murphy Thanks -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
Am Sun, 20 Jul 2014 02:39:27 + (UTC) schrieb Duncan 1i5t5.dun...@cox.net: Chris Murphy posted on Sat, 19 Jul 2014 11:38:08 -0600 as excerpted: I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc errors during balance but it seems informational rather than either a warning or problem. I'd treat ext4-btrfs converted file systems to be something of an odd duck, in that it's uncommon, therefore isn't getting as much testing and extra caution is a good idea. Make frequent backups. Expanding on that a bit... Balance simply rewrites chunks, combining where possible and possibly converting to a different layout (single/dup/raid0/1/10/5/6[1]) in the process. The most common reason for enospc during balance is of course all space allocated to chunks, with various workarounds for that if it happens, but that doesn't seem to be what was happening to you (Mark J./OP). Based on very similar issues reported by another ext4 - btrfs converter and the discussion on that thread, here's what I think happened: First a critical question for you as it's a critical piece of this scenario that you didn't mention in your summary. The wiki page on ext4 - btrfs conversion suggests deleting the ext2_saved subvolume and then doing a full defrag and rebalance. You're attempting a full rebalance, but have you yet deleted ext2_saved and did you do the defrag before attempting the rebalance? I'm guessing not, as was the case with the other user that reported this issue. Here's what apparently happened in his case and how we fixed it: Ah, I actually did, in fact. I only implicitly said it, though. Here's what I wrote: After converting the backup partition about a week ago, following the wiki entry on ext4 conversion, I eventually ran a full balance [...] The wiki says to run a full balance (and defragment before that, but that was slw, so I didn't do it), *after* deleting the ext4 file system image. So the full balance was right after doing that :) . The problem is that btrfs data chunks are 1 GiB each. Thus, the maximum size of a btrfs extent is 1 GiB. But ext4 doesn't have an arbitrary limitation on extent size, and for files over a GiB in size, ext4 extents can /also/ be over a GiB in size. That results in two potential issues at balance time. First, btrfs treats the ext2_saved subvolume as a read-only snapshot and won't touch it, thus keeping the ext* data intact in case the user wishes to rollback to ext*. I don't think btrfs touches that data during a balance either, as it really couldn't do so /safely/ without incorporating all of the ext* code into btrfs. I'm not sure how it expresses that situation, but it's quite possible that btrfs treats it as enospc. Second, for files that had ext4 extents greater than a GiB, balance will naturally enospc, because even the biggest possible btrfs extent, a full 1 GiB data chunk, is too small to hold the existing file extent. Of course this only happens on filesystems converted from ext*, because natively btrfs has no way to make an extent larger than a GiB, so it won't run into the problem if it was created natively instead of converted from ext*. Once the ext2_saved subvolume/snapshot is deleted, defragging should cure the problem as it rewrites those files to btrfs-native chunks, normally defragging but in this case fragging to the 1 GiB btrfs-native data-chunk- size extent size. Hmm, well, I didn't defragment because it would have taken *forever* to go through all those hardlinks, plus my experience is that ext* doesn't fragment much at all, so I skipped that step. But I certainly have files over 1GB in size. On the other hand, the wiki [0] says that defragmentation (and balancing) is optional, and the only reason stated for doing either is because they will have impact on performance. Alternatively, and this is what the other guy did, one can find all the files from the original ext*fs over a GiB in size, and move them off- filesystem and back AFAIK he had several gigs of spare RAM and no files larger than that, so he used tmpfs as the temporary storage location, which is memory so the only I/O is that on the btrfs in question. By doing that he deleted the existing files on btrfs and recreated them, naturally splitting the extents on data-chunk-boundaries as btrfs normally does, in the recreation. If you had deleted the ext2_saved subvolume/snapshot and done the defrag already, that explanation doesn't work as-is, but I'd still consider it an artifact from the conversion, and try the alternative move-off- filesystem-temporarily method. I'll try this and see, but I think I have more files 1GB than would account for this error (which comes towards the end of the balance when only a few chunks are left). I'll see what find /mnt -type f -size +1G finds :) . If you don't have any files over a GiB in size, then I don't know...
Re: ENOSPC errors during balance
Am Sun, 20 Jul 2014 12:22:33 +0200 schrieb Marc Joliet mar...@gmx.de: [...] I'll try this and see, but I think I have more files 1GB than would account for this error (which comes towards the end of the balance when only a few chunks are left). I'll see what find /mnt -type f -size +1G finds :) . Now that I think about it, though, it sounds like it could explain the sudden surge in total data size: for one very big file, several chunks/extents are created, but the data cannot be copied from the original ext4 extent. So far, the above find command has only found a few handful of files (plus all the reflinks in the snapshots), much to my surprise. It still has one subvolume to go through, though. And just for completeness, that same find command didn't find any files on /, which I also converted from ext4, and for which a full balance completed successfully. So maybe this is in the right direction, but I'll wait and see what Chris Murphy (or anyone else) might find in my latest dmesg output. -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
Marc Joliet posted on Sun, 20 Jul 2014 12:22:33 +0200 as excerpted: On the other hand, the wiki [0] says that defragmentation (and balancing) is optional, and the only reason stated for doing either is because they will have impact on performance. Yes. That's what threw off the other guy as well. He decided to skip it for the same reason. If I had a wiki account I'd change it, but for whatever reason I tend to be far more comfortable writing list replies, sometimes repeatedly, than writing anything on the web, which I tend to treat as read-only. So I've never gotten a wiki account and thus haven't changed it, and apparently the other guy with the problem and anyone else that knows hasn't changed it either, so the conversion page still continues to underemphasize the importance of completing the conversion steps, including the defrag, in proper order. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during balance
Marc Joliet posted on Sun, 20 Jul 2014 21:44:40 +0200 as excerpted: Am Sun, 20 Jul 2014 13:40:54 +0200 schrieb Marc Joliet mar...@gmx.de: Am Sun, 20 Jul 2014 12:22:33 +0200 schrieb Marc Joliet mar...@gmx.de: [...] I'll try this and see, but I think I have more files 1GB than would account for this error (which comes towards the end of the balance when only a few chunks are left). I'll see what find /mnt -type f -size +1G finds :) . Note that it's extent's over 1 GiB on the converted former ext4, not necessarily files over 1 GiB. You may have files over a GiB that were already broken into extents that are all less than a GiB, and btrfs would be able to deal with them fine. It's only when a single extent ended up larger than a GiB on the former ext4 that btrfs can't deal with it. Now that I think about it, though, it sounds like it could explain the sudden surge in total data size: for one very big file, several chunks/extents are created, but the data cannot be copied from the original ext4 extent. I hadn't thought about that effect, but good deductive reasoning. =:^) Well, turns out that was it! What I did: - delete the single largest file on the file system, a 12 GB VM image, along with all subvolumes that contained it - rsync it over again - start a full balance This time, the balance finished successfully :-) . Good to read! We're now two for two on this technique working around this problem! =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during balance
The 2nd dmesg (didn't look at the 1st), has many instances like this; [96241.882138] ata2.00: exception Emask 0x1 SAct 0x7ffe0fff SErr 0x0 action 0x6 frozen [96241.882139] ata2.00: Ata error. fis:0x21 [96241.882142] ata2.00: failed command: READ FPDMA QUEUED [96241.882148] ata2.00: cmd 60/08:00:68:0a:2d/00:00:18:00:00/40 tag 0 ncq 4096 in res 41/00:58:40:5c:2c/00:00:18:00:00/40 Emask 0x1 (device error) I'm not sure what this error is, it acts like an unrecoverable read error but I'm not seeing UNC reported. It looks like ata 2.00 is sdb, which is a member of a btrfs raid10 volume. So this isn't related to your sdg2 and enospc error, it's a different problem. I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc errors during balance but it seems informational rather than either a warning or problem. I'd treat ext4-btrfs converted file systems to be something of an odd duck, in that it's uncommon, therefore isn't getting as much testing and extra caution is a good idea. Make frequent backups. Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during balance
Am Sat, 19 Jul 2014 22:10:51 +0200 schrieb Marc Joliet mar...@gmx.de: [...] Another random idea: the number of errors decreased the second time I ran balance (from 4 to 2), I could run another full balance and see if it keeps decreasing. Well, this time there were still 2 ENOSPC errors. But I can show the df output after such an ENOSPC error, to illustrate what I meant with the sudden surge in total usage: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=236.00GiB, used=229.04GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.00GiB, used=3.20GiB unknown, single: total=512.00MiB, used=0.00 And then after running a balance and (almost) immediately cancelling: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=230.00GiB, used=229.04GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.00GiB, used=3.20GiB unknown, single: total=512.00MiB, used=0.00 -- Marc Joliet -- People who think they know everything really annoy those of us who know we don't - Bjarne Stroustrup signature.asc Description: PGP signature
Re: ENOSPC errors during balance
On Sat, Jul 19, 2014 at 11:38:08AM -0600, Chris Murphy wrote: [96241.882138] ata2.00: exception Emask 0x1 SAct 0x7ffe0fff SErr 0x0 action 0x6 frozen [96241.882139] ata2.00: Ata error. fis:0x21 [96241.882142] ata2.00: failed command: READ FPDMA QUEUED [96241.882148] ata2.00: cmd 60/08:00:68:0a:2d/00:00:18:00:00/40 tag 0 ncq 4096 in res 41/00:58:40:5c:2c/00:00:18:00:00/40 Emask 0x1 (device error) Afair those are somehow related to NCQ. Piotr Szymaniak. -- Komnata audiencyjna zlotego Bruna przerastala wszystko, co dotad widzialem. Musial zatrudnic dziesiatki programistow i kreatorow, by stworzyc tak wyuzdane i wysmakowane wnetrze. Dzwieki, barwy, ksztalty i zapachy wywolywaly erekcje. -- Marcin Przybylek, Gamedec: Syndrom Adelheima signature.asc Description: Digital signature
Re: ENOSPC errors during balance
On Jul 19, 2014, at 2:58 PM, Marc Joliet mar...@gmx.de wrote: Am Sat, 19 Jul 2014 22:10:51 +0200 schrieb Marc Joliet mar...@gmx.de: [...] Another random idea: the number of errors decreased the second time I ran balance (from 4 to 2), I could run another full balance and see if it keeps decreasing. Well, this time there were still 2 ENOSPC errors. But I can show the df output after such an ENOSPC error, to illustrate what I meant with the sudden surge in total usage: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=236.00GiB, used=229.04GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.00GiB, used=3.20GiB unknown, single: total=512.00MiB, used=0.00 And then after running a balance and (almost) immediately cancelling: # btrfs filesystem df /run/media/marcec/MARCEC_BACKUP Data, single: total=230.00GiB, used=229.04GiB System, DUP: total=32.00MiB, used=36.00KiB Metadata, DUP: total=4.00GiB, used=3.20GiB unknown, single: total=512.00MiB, used=0.00 I think it's a bit weird. Two options: a. Keep using the file system, with judicious backups, if a dev wants more info they'll reply to the thread; b. Migrate the data to a new file system, first capture the file system with btrfs-image in case a dev wants more info and you've since blown away the filesystem, and then move it to a new btrfs fs. I'd use send/receive for this to preserve subvolumes and snapshots. Chris Murphy-- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ENOSPC errors during balance
Chris Murphy posted on Sat, 19 Jul 2014 11:38:08 -0600 as excerpted: I'm not sure of the reason for the BTRFS info (device sdg2): 2 enospc errors during balance but it seems informational rather than either a warning or problem. I'd treat ext4-btrfs converted file systems to be something of an odd duck, in that it's uncommon, therefore isn't getting as much testing and extra caution is a good idea. Make frequent backups. Expanding on that a bit... Balance simply rewrites chunks, combining where possible and possibly converting to a different layout (single/dup/raid0/1/10/5/6[1]) in the process. The most common reason for enospc during balance is of course all space allocated to chunks, with various workarounds for that if it happens, but that doesn't seem to be what was happening to you (Mark J./OP). Based on very similar issues reported by another ext4 - btrfs converter and the discussion on that thread, here's what I think happened: First a critical question for you as it's a critical piece of this scenario that you didn't mention in your summary. The wiki page on ext4 - btrfs conversion suggests deleting the ext2_saved subvolume and then doing a full defrag and rebalance. You're attempting a full rebalance, but have you yet deleted ext2_saved and did you do the defrag before attempting the rebalance? I'm guessing not, as was the case with the other user that reported this issue. Here's what apparently happened in his case and how we fixed it: The problem is that btrfs data chunks are 1 GiB each. Thus, the maximum size of a btrfs extent is 1 GiB. But ext4 doesn't have an arbitrary limitation on extent size, and for files over a GiB in size, ext4 extents can /also/ be over a GiB in size. That results in two potential issues at balance time. First, btrfs treats the ext2_saved subvolume as a read-only snapshot and won't touch it, thus keeping the ext* data intact in case the user wishes to rollback to ext*. I don't think btrfs touches that data during a balance either, as it really couldn't do so /safely/ without incorporating all of the ext* code into btrfs. I'm not sure how it expresses that situation, but it's quite possible that btrfs treats it as enospc. Second, for files that had ext4 extents greater than a GiB, balance will naturally enospc, because even the biggest possible btrfs extent, a full 1 GiB data chunk, is too small to hold the existing file extent. Of course this only happens on filesystems converted from ext*, because natively btrfs has no way to make an extent larger than a GiB, so it won't run into the problem if it was created natively instead of converted from ext*. Once the ext2_saved subvolume/snapshot is deleted, defragging should cure the problem as it rewrites those files to btrfs-native chunks, normally defragging but in this case fragging to the 1 GiB btrfs-native data-chunk- size extent size. Alternatively, and this is what the other guy did, one can find all the files from the original ext*fs over a GiB in size, and move them off- filesystem and back AFAIK he had several gigs of spare RAM and no files larger than that, so he used tmpfs as the temporary storage location, which is memory so the only I/O is that on the btrfs in question. By doing that he deleted the existing files on btrfs and recreated them, naturally splitting the extents on data-chunk-boundaries as btrfs normally does, in the recreation. If you had deleted the ext2_saved subvolume/snapshot and done the defrag already, that explanation doesn't work as-is, but I'd still consider it an artifact from the conversion, and try the alternative move-off- filesystem-temporarily method. If you don't have any files over a GiB in size, then I don't know... perhaps it's some other bug. --- [1] Raid5/6 support not yet complete. Operational code is there but recovery code is still incomplete. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
