[zfs-discuss] Automatic removal of old snapshots
Hi, I'm building a new ZFS fileserver for our lab and I'd like to have these features: - take a snapshot of users' home directories every N minutes (N is 5 or 10) - remove all old snapshots, keep just these: - all snapshots made during last H hours (H=24) - keep one snapshot per day (e.g. made at 16:00 hrs) for the last D days (D=7) - keep one snapshot per week for the last W weeks (W=4) - keep the last Y monthly snapshots (Y=12) In other words, every user would have (H * 60/N) + D + W + Y snapshots in their home directories (N=5 = 311 snapshots). Before re-inventing the wheel, does anyone have any nice shell script to do this kind of thing (to be executed from cron)? We have only few users and quite static home directories so this kind of snapshot amount should be fine... Martti ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
Before re-inventing the wheel, does anyone have any nice shell script to do this kind of thing (to be executed from cron)? http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_10 http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_11 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote: Before re-inventing the wheel, does anyone have any nice shell script to do this kind of thing (to be executed from cron)? http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_10 http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_11 Storage Checkpoints in Veritas software has this feature (removing the oldest checkpoint in case of 100% filesystem usage) by default. Why not add such option to ZFS ? Regards Przemyslaw Bak (przemol) -- http://przemol.blogspot.com/ -- Znajdź mieszkanie w Twoim regionie! kliknij http://link.interia.pl/f1f19 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
On Thu, 2008-09-25 at 12:07 +0200, [EMAIL PROTECTED] wrote: On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote: Storage Checkpoints in Veritas software has this feature (removing the oldest checkpoint in case of 100% filesystem usage) by default. Why not add such option to ZFS ? - because we shouldn't assume a user's intentions - some would consider automatic destruction of data (even historical data) a p1 bug. nv_100 will have the ZFS Automatic Snapshot service in it, disabled by default[1] The GUI side of this, (Time Slider) has exactly this functionality - it'll start deleting snapshots that it has taken when the filesystem reaches a certain threshold. More details at: http://opensolaris.org/os/community/arc/caselog/2008/571/mail cheers, tim [1] actually while I'm here, quick poll - the schedules for retaining snapshots currently are: - Frequent snapshots, taken every 15 minutes, keeping the 4 most recent - Hourly snapshots taken once every hour, keeping 24 - Daily snapshots taken once every 24 hours, keeping 7 - Weekly snapshots taken once every 7 days, keeping 4 - Monthly snapshots taken on the first day of every month, keeping 12 I'm arguing that we should be keeping 31 daily snapshots, and not just 7. There's some overlap with the weekly ones yes, but the extra granularity over the past month could be useful, anyone else have an opinion? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
On Thu, Sep 25, 2008 at 11:30:04AM +0100, Tim Foster wrote: On Thu, 2008-09-25 at 12:07 +0200, [EMAIL PROTECTED] wrote: On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote: Storage Checkpoints in Veritas software has this feature (removing the oldest checkpoint in case of 100% filesystem usage) by default. Why not add such option to ZFS ? - because we shouldn't assume a user's intentions - some would consider automatic destruction of data (even historical data) a p1 bug. Tim, nobody is going to assume user's intentions. Just give us snapshot-related property which we can set to on/off and everybody can setup zfs according to his/her needs. Regards Przemyslaw Bak (przemol) -- http://przemol.blogspot.com/ -- Znajdź mieszkanie w Twoim regionie! kliknij http://link.interia.pl/f1f19 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
On Thu, 2008-09-25 at 12:52 +0200, [EMAIL PROTECTED] wrote: nobody is going to assume user's intentions. Just give us snapshot-related property which we can set to on/off and everybody can setup zfs according to his/her needs. Then that'll be there in nv_100. Enjoy! cheers, tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
[EMAIL PROTECTED] wrote on 09/25/2008 05:30:04 AM: On Thu, 2008-09-25 at 12:07 +0200, [EMAIL PROTECTED] wrote: On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote: Storage Checkpoints in Veritas software has this feature (removing the oldest checkpoint in case of 100% filesystem usage) by default. Why not add such option to ZFS ? - because we shouldn't assume a user's intentions - some would consider automatic destruction of data (even historical data) a p1 bug. nv_100 will have the ZFS Automatic Snapshot service in it, disabled by default[1] The GUI side of this, (Time Slider) has exactly this functionality - it'll start deleting snapshots that it has taken when the filesystem reaches a certain threshold. More details at: http://opensolaris.org/os/community/arc/caselog/2008/571/mail cheers, tim [1] actually while I'm here, quick poll - the schedules for retaining snapshots currently are: - Frequent snapshots, taken every 15 minutes, keeping the 4 most recent - Hourly snapshots taken once every hour, keeping 24 - Daily snapshots taken once every 24 hours, keeping 7 - Weekly snapshots taken once every 7 days, keeping 4 - Monthly snapshots taken on the first day of every month, keeping 12 I'm arguing that we should be keeping 31 daily snapshots, and not just 7. There's some overlap with the weekly ones yes, but the extra granularity over the past month could be useful, anyone else have an opinion? Tim, That snap schedule seems reasonable to me. Relate to the cleanup part of the doc linked, do you know the rational for killing off the most recent (15 minute and hourly) snaps vs the oldest (monthly) first? To me, it seems like the biggest bang for the buck (snapshot held data) would be killing off (oldest) monthly, weekly, hourly, daily, hourly, 15 in that order. Also I guess user case in my mind would leave a desktop user more likely to need access to a few minutes, hours or days ago then 12 months ago. -Wade ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
On Thu, 2008-09-25 at 10:19 -0500, [EMAIL PROTECTED] wrote: That snap schedule seems reasonable to me. Relate to the cleanup part of the doc linked, do you know the rational for killing off the most recent (15 minute and hourly) snaps vs the oldest (monthly) first? It's a tough call (which thankfully I didn't have to make). I'm not sure there was a rationale, other than the guys who were implementing it messing about on their desktops finding out where /they/ tended to get the most savings. Really, it all depends on how volatile your filesystems are, where the best place to retrieve data from. I suspect it'll take more real-user testing to determine what's the best balance between data availability and disk space. cheers, tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
[EMAIL PROTECTED] wrote on 09/25/2008 10:34:41 AM: On Thu, 2008-09-25 at 10:19 -0500, [EMAIL PROTECTED] wrote: That snap schedule seems reasonable to me. Relate to the cleanup part of the doc linked, do you know the rational for killing off the most recent (15 minute and hourly) snaps vs the oldest (monthly) first? It's a tough call (which thankfully I didn't have to make). I'm not sure there was a rationale, other than the guys who were implementing it messing about on their desktops finding out where /they/ tended to get the most savings. Really, it all depends on how volatile your filesystems are, where the best place to retrieve data from. I suspect it'll take more real-user testing to determine what's the best balance between data availability and disk space. Thanks Tim. It seems like a very cool project. Maybe there will be more push to show more block born/dead/held stats from a public ZFS API when stuff like this gets rolling. That should allow more of an educated guess as to what snapshots are holding what data. -Wade ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
Wade, that order. Also I guess user case in my mind would leave a desktop user more likely to need access to a few minutes, hours or days ago then 12 months ago. You are guessing that, but I am a desktop user who'd rather like the contrary. I think Tim has already stated that he would not want to hardcode any assumptions as to which snapshots one would want to delete first, and I think he is very right in doing so. Nils ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
Hi Wade, We considered a number of approaches including just deleting oldest snapshots first and progressing through to the newest snapshots. When you consider the default snapshot schedules we are going to use, the model is that snapshots get thinned out over time. So in situations were disk space runs low, we stay consistent with this model but accelerate the aging process and thin out snapshots quicker, instead of just chopping the end off. The exception to this is that frequent snapshots will only get deleted as a very last resort. The reason for this is that frequent snapshots typically consume very little space, and they will get deleted within an hour or so anyway. And more importantly, the most common type of error is human error rather than hardware failure, such as accidentally deleting or overwriting a file or corrupting it. These kind of errors are usually noticed instantly or very soon after the event so retaining the frequent snapshots will hopefully provide some reasonable protection against this most common type of error. Cheers, Niall. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Automatic removal of old snapshots
tf == Tim Foster [EMAIL PROTECTED] writes: tf anyone else have an opinion? keep the number of snapshots small until the performacne problems with booting/importing/scrubbing while having lots of snapshots are resolved. pgp4Qi9Dyk7O4.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
