Re: What to do about snapshot-aware defrag
Martin m_bt...@ml1.co.uk writes: The *ONLY* application that I know of that uses atime is Mutt and then *only* for mbox files!... However, users, such as myself :), can be interested in when a certain file has been last accessed. With snapshots I can even get an idea of all the times the file has been accessed. *And go KISS and move on faster* better? Well, it in uncertain to me if it truly is better that btrfs would after that point no longer truly even support atime, if using it results in blowing up snapshot sizes. They might at that point even consider just using LVM2 snapshots (shudder) ;). -- _ / __// /__ __ http://www.modeemi.fi/~flux/\ \ / /_ / // // /\ \/ /\ / /_/ /_/ \___/ /_/\_\@modeemi.fi \/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On 04/06/14 10:19, Erkki Seppala wrote: Martin m_bt...@ml1.co.uk writes: The *ONLY* application that I know of that uses atime is Mutt and then *only* for mbox files!... However, users, such as myself :), can be interested in when a certain file has been last accessed. With snapshots I can even get an idea of all the times the file has been accessed. *And go KISS and move on faster* better? Well, it in uncertain to me if it truly is better that btrfs would after that point no longer truly even support atime, if using it results in blowing up snapshot sizes. They might at that point even consider just using LVM2 snapshots (shudder) ;). Not quite... My emphasis is: 1: Go KISS for the defrag and accept that any atime use will render the defrag ineffective. Give a note that the noatime mount option should be used. 2: Consider using noatime as a /default/ being as there are no known 'must-use' use cases. Those users still wanting atime can add that as a mount option with the note that atime use reduces the snapshot defrag effectiveness. (The for/against atime is a good subject for another thread!) Go fast KISS! Regards, Martin -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On Jun 4, 2014, at 7:15 AM, Martin m_bt...@ml1.co.uk wrote: Consider using noatime as a /default/ being as there are no known 'must-use' use cases. The quote I'm finding on the interwebs is POSIX “requires that operating systems maintain file system metadata that records when each file was last accessed. I'm not sure if upstream kernel projects aim for LSB (and thus POSIX) compliance by default and let distros opt out; or the opposite. Those users still wanting atime can add that as a mount option with the note that atime use reduces the snapshot defrag effectiveness. I can imagine some optimizations for Btrfs that are easier than other file systems, like a way to point metadata chunks to specific devices, for example metadata to persistent memory, while the data goes to conventional hard drives. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On 02/06/14 14:22, Josef Bacik wrote: On 05/30/2014 06:00 PM, Martin wrote: OK... I'll jump in... On 30/05/14 21:43, Josef Bacik wrote: Hello, TL;DR: I want to only do snapshot-aware defrag on inodes in snapshots that haven't changed since the snapshot was taken. Yay or nay (with a reason why for nay) [...] === Summary and what I need === Option 1: Only relink inodes that haven't changed since the snapshot was taken. [...] Obvious way to go for fast KISS. One question: Will option one mean that we always need to mount with noatime or read-only to allow snapshot defragging to do anything? Yeah atime would screw this up, I hadn't thought of that. With that being the case I think the only option is to keep the old behavior, we don't want to screw up stuff like this just because users used a backup program on their snapshot and didn't use noatime. Thanks, Not so fast into non-KISS! The *ONLY* application that I know of that uses atime is Mutt and then *only* for mbox files!... NOTHING else uses atime as far as I know. We already have most distros enabling reltime by default as a just-in-case... Can we not have noatime as the default for btrfs? Also widely note that default in the man page and wiki and with why?... *And go KISS and move on faster* better? Myself, I still use Mutt sometimes, but no mbox, and all my filesystems have been noatime for many years now with good positive results. (Both home and work servers.) Regards, Martin -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On 06/01/2014 11:07 PM, Mitch Harder wrote: On Sat, May 31, 2014 at 6:51 PM, Brendan Hide bren...@swiftspirit.co.za wrote: On 2014/05/31 12:00 AM, Martin wrote: OK... I'll jump in... On 30/05/14 21:43, Josef Bacik wrote: [snip] Option 1: Only relink inodes that haven't changed since the snapshot was taken. Pros: -Faster -Simpler -Less duplicated code, uses existing functions for tricky operations so less likely to introduce weird bugs. Cons: -Could possibly lost some of the snapshot-awareness of the defrag. If you just touch a file we would not do the relinking and you'd end up with twice the space usage. [...] Obvious way to go for fast KISS. I second this - KISS is better. Would in-band dedupe resolve the issue with losing the snapshot-awareness of the defrag? I figure that if someone absolutely wants everything deduped efficiently they'd put in the necessary resources (memory/dedicated SSD/etc) to have in-band dedupe work well. One question: Will option one mean that we always need to mount with noatime or read-only to allow snapshot defragging to do anything? When snapshot-aware defrag first came out, I was convinced it was a must-have capability for nearly everybody using btrfs. But, the more I look at my work load and common practices with btrfs, the more I am wondering just how often snapshot-aware defrag was actually doing something for me. I use a lot of snapshots. But for the most part, once I touch a file in my current subvolume, the whole file needs to be COW-ed from it's previous version. The whole file doesn't need to be cow'ed, just the part that you touch. So old snapshot-aware defrag probably would have saved you quite a bit of space. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On 05/30/2014 06:00 PM, Martin wrote: OK... I'll jump in... On 30/05/14 21:43, Josef Bacik wrote: Hello, TL;DR: I want to only do snapshot-aware defrag on inodes in snapshots that haven't changed since the snapshot was taken. Yay or nay (with a reason why for nay) [...] === Summary and what I need === Option 1: Only relink inodes that haven't changed since the snapshot was taken. Pros: -Faster -Simpler -Less duplicated code, uses existing functions for tricky operations so less likely to introduce weird bugs. Cons: -Could possibly lost some of the snapshot-awareness of the defrag. If you just touch a file we would not do the relinking and you'd end up with twice the space usage. [...] Obvious way to go for fast KISS. One question: Will option one mean that we always need to mount with noatime or read-only to allow snapshot defragging to do anything? Yeah atime would screw this up, I hadn't thought of that. With that being the case I think the only option is to keep the old behavior, we don't want to screw up stuff like this just because users used a backup program on their snapshot and didn't use noatime. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On Sat, May 31, 2014 at 6:51 PM, Brendan Hide bren...@swiftspirit.co.za wrote: On 2014/05/31 12:00 AM, Martin wrote: OK... I'll jump in... On 30/05/14 21:43, Josef Bacik wrote: [snip] Option 1: Only relink inodes that haven't changed since the snapshot was taken. Pros: -Faster -Simpler -Less duplicated code, uses existing functions for tricky operations so less likely to introduce weird bugs. Cons: -Could possibly lost some of the snapshot-awareness of the defrag. If you just touch a file we would not do the relinking and you'd end up with twice the space usage. [...] Obvious way to go for fast KISS. I second this - KISS is better. Would in-band dedupe resolve the issue with losing the snapshot-awareness of the defrag? I figure that if someone absolutely wants everything deduped efficiently they'd put in the necessary resources (memory/dedicated SSD/etc) to have in-band dedupe work well. One question: Will option one mean that we always need to mount with noatime or read-only to allow snapshot defragging to do anything? When snapshot-aware defrag first came out, I was convinced it was a must-have capability for nearly everybody using btrfs. But, the more I look at my work load and common practices with btrfs, the more I am wondering just how often snapshot-aware defrag was actually doing something for me. I use a lot of snapshots. But for the most part, once I touch a file in my current subvolume, the whole file needs to be COW-ed from it's previous version. Now that we have a working sysfs, I wonder if we could implement some counters to track how often snapshot-aware defrag would have run. I might be surprised at how much it was doing. --- Regards, Mitch Harder -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
On 2014/05/31 12:00 AM, Martin wrote: OK... I'll jump in... On 30/05/14 21:43, Josef Bacik wrote: [snip] Option 1: Only relink inodes that haven't changed since the snapshot was taken. Pros: -Faster -Simpler -Less duplicated code, uses existing functions for tricky operations so less likely to introduce weird bugs. Cons: -Could possibly lost some of the snapshot-awareness of the defrag. If you just touch a file we would not do the relinking and you'd end up with twice the space usage. [...] Obvious way to go for fast KISS. I second this - KISS is better. Would in-band dedupe resolve the issue with losing the snapshot-awareness of the defrag? I figure that if someone absolutely wants everything deduped efficiently they'd put in the necessary resources (memory/dedicated SSD/etc) to have in-band dedupe work well. One question: Will option one mean that we always need to mount with noatime or read-only to allow snapshot defragging to do anything? That is a very good question. I very rarely have mounts without noatime - and usually only because I hadn't thought of it. Regards, Martin -- __ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: What to do about snapshot-aware defrag
Brendan Hide posted on Sun, 01 Jun 2014 01:51:49 +0200 as excerpted: Will option one mean that we always need to mount with noatime or read-only to allow snapshot defragging to do anything? That is a very good question. I very rarely have mounts without noatime - and usually only because I hadn't thought of it. Heh, a couple months ago I got tired of having to add noatime to all my standard mounts, and decided to patch my kernel to noatime (instead of relatime) by default. I can if I need to still use relatime or strictatime, but if they aren't listed, noatime gets added by default, now. =:^) The only remaining problem is that it's not a full default, as noatime still shows up in /proc/self/mounts, but avoiding that would have complexified the patch, and not being an actual coder (and DEFINITELY not a kernel coder), I decided I best leave well enough alone, patching only what I had to to avoid having relatime if I omitted the parameter, and only what I was reasonably sure I could do without screwing things up, based on my limited sysadmin level reading of the sources. It would sure be nice if there were a kernel config option for that default, which would naturally still default to relatime, which is I'm sure where most of the distros not already patching it to noatime would leave it, as well. Meanwhile, agreed, a good question that does indeed remain! =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
