Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
On Tue, Apr 27, 2010 at 1:15 PM, Francois Tigeot ftig...@wolfpond.org wrote: I think sysutils/rsnapshot does what you want. It uses hard links to simulate snapshots on classic filesystems and manages different ranges of snapshots. You can specify how much snapshots you want to keep for each range: [rsnapshot.conf] intervalhourly 6 intervaldaily 7 intervalweekly 4 intervalmonthly 12 The hammer utility would be much better if it implemented some similar mechanism IMHO. Hi Francois, I am familiar with rsnapshot and have been using it/backuppc but I guess hammer snapshot is much faster and that rsnapshot when the directory has many files. Also when there are many small changes to many files spread over the day rsnapshot will store all the changed file as they are induvidually and will take more space if I take a snapshot every 5 mins. Thanks for the tip :-) --Siju
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
On Tue, Apr 27, 2010 at 7:26 AM, Matthew Dillon dil...@apollo.backplane.com wrote: hammer cleanup only removes snapshots over X days old. It can't distinguish between fine-grained and coarse-grained snapshots that you explicitly tell hammer to make. You would have to remove those yourself (if you want to expire them before the X days) using hammer snaprm. You can script it fairly easily by setting the comment field for each snapshot you take, then filtering out the list based on that. See the manual page. Yes the comment field was in my mind. I thought of entering the output of #date -j -f %a %b %d %T %Z %Y `date` +%s to the comments field of the snapshots. Then the script will prune snapshots that have the commentfield 172800 less than the current value of the date by seconds from epoch. But then I thought it is easier to rsync from the master pfs in disk1 after the 5 mins snapshots to another master pfs in disk2 and let periodic take daily snapshots of the pfs in disk1 for now :-) Thanks for the idea! --Siju
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
Hi George, On Wed, Apr 28, 2010 at 03:40:08PM +0530, Siju George wrote: On Tue, Apr 27, 2010 at 1:15 PM, Francois Tigeot ftig...@wolfpond.org wrote: I think sysutils/rsnapshot does what you want. It uses hard links to simulate snapshots on classic filesystems and manages different ranges of snapshots. You can specify how much snapshots you want to keep for each range: [rsnapshot.conf] intervaldaily 7 intervalmonthly 12 The hammer utility would be much better if it implemented some similar mechanism IMHO. I am familiar with rsnapshot and have been using it/backuppc but I guess hammer snapshot is much faster and that rsnapshot when the directory has many files. Also when there are many small changes to many files spread over the day rsnapshot will store all the changed file as they are induvidually and will take more space if I take a snapshot every 5 mins. Yeah, I'm in full agreement with you. I think I was a bit misanderstood: Hammer performance is *much better* than rsnapshot (obviously) but there is no way to easily tell it to keep its snapshot distribution in different intervals for archiving purposes. This is the one thing rsnapshot does extremely well; I was just trying to say this is a good idea we should steal :-) -- Francois Tigeot
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
On Wed, Apr 28, 2010 at 4:10 PM, Francois Tigeot ftig...@wolfpond.org wrote: Yeah, I'm in full agreement with you. I think I was a bit misanderstood: Hammer performance is *much better* than rsnapshot (obviously) but there is no way to easily tell it to keep its snapshot distribution in different intervals for archiving purposes. This is the one thing rsnapshot does extremely well; I was just trying to say this is a good idea we should steal :-) hope Matt will agree ;-) --Siju
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
: Yeah, I'm in full agreement with you. : : I think I was a bit misanderstood: Hammer performance is *much better* than : rsnapshot (obviously) but there is no way to easily tell it to keep its : snapshot distribution in different intervals for archiving purposes. : This is the one thing rsnapshot does extremely well; I was just trying to say : this is a good idea we should steal :-) : : :hope Matt will agree ;-) : :--Siju A more complex snapshot retention scheme would be a nice project for someone. The hammer config file is just a block of text so it is possible to add pretty much whatever we want to it, with accompanying programming. If someone wants to take on this project I think it could be integrated into the hammer cleanup code fairly easily by having the hammer cleanup field classify the snapshots it makes itself using the comment field, and then parsing that field to figure out cleanup intervals. This is the kind of format I would recommend for the config entry: snapshots 5m 5d snapshots 1d 60d snapshots 1w 2y e.g. the above would mean: Keep 5-minute snapshots for 5 days, daily snapshots for 60 days, and weekly snapshots for two years. (NOTE: This might not actually be viable, it is just an example, as you would have to have quite a bit of storage to keep weekly's for 2-years). hammer could track the snapshots independently, even if they overlap, by using the comment field to distinguish them. It could then use the comment field when doing the prune to figure out which snapshots line is applicable. -Matt Matthew Dillon dil...@backplane.com
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
On Mon, Apr 26, 2010 at 06:56:50PM -0700, Matthew Dillon wrote: :All I need is to figure out how to remove the 5mins snapshots that :gets mirrored on the slave older than two days with out removing the :daily snapshots. : :But I am a bit confused now since I dont see snapshots actually :removed after a hammer cleanup. :I will send the details with a new subject : :--Siju hammer cleanup only removes snapshots over X days old. It can't distinguish between fine-grained and coarse-grained snapshots that you explicitly tell hammer to make. You would have to remove those yourself (if you want to expire them before the X days) using hammer snaprm. You can script it fairly easily by setting the comment field for each snapshot you take, then filtering out the list based on that. See the manual page. I think sysutils/rsnapshot does what you want. It uses hard links to simulate snapshots on classic filesystems and manages different ranges of snapshots. You can specify how much snapshots you want to keep for each range: [rsnapshot.conf] intervalhourly 6 intervaldaily 7 intervalweekly 4 intervalmonthly 12 The hammer utility would be much better if it implemented some similar mechanism IMHO. -- Francois Tigeot
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
On Sat, Apr 24, 2010 at 11:16 PM, Matthew Dillon dil...@apollo.backplane.com wrote: So, here's an example. Lets say you have: /samba_export Your samba export hierarchy /hammer Your hammer filesystem /hammer/pfs/blah Some PFS in the hammer filesystem And now lets asy you want to make snapshots from PFS(s) in the hammer filesystem available to the export. This would work: mkdir /samba_export/fubar mount_null /hammer/pfs /samba_export/fubar If I do this then all the pfses under /hammer/pfs will be exported using samba I guess you meant mount_null /hammer/pfs/blah /samba_export/fubar ? That is what I did for the Master pfs /Backup1 - Hammer File System /Backup1/www-5mbak- Samba export Directory and mount /Backup1/pfs/www-hot /Backup1/www-5mbak/www-hotnullrw 0 0 But the problem is that a slave PFs cannot be mounted. So My problem is what I do with the slave PFS :-( You may have other issues... tracking the snapshots could present a problem since hammer cleanup only allows one snapshot directory to be specified. So you might have to come up with a little script to maintain a second set of snapsoft softlinks with the proper samba path which mirrors the master set which hammer cleanup maintains. Otherwise you will wind up with a lot of stale snapshot softlinks. I Initially planned to take the 5 mins snapshots on the Master andDaily snapshots on the slave. So I can export by Samba 2 shares. One having the links to the 5 mins backup( for 2 days ) and other having the links to the Daily backup ( 2000 days or old ) All I need is to figure out how to remove the 5mins snapshots that gets mirrored on the slave older than two days with out removing the daily snapshots. But I am a bit confused now since I dont see snapshots actually removed after a hammer cleanup. I will send the details with a new subject thanks --Siju
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
:All I need is to figure out how to remove the 5mins snapshots that :gets mirrored on the slave older than two days with out removing the :daily snapshots. : :But I am a bit confused now since I dont see snapshots actually :removed after a hammer cleanup. :I will send the details with a new subject : :thanks : :--Siju hammer cleanup only removes snapshots over X days old. It can't distinguish between fine-grained and coarse-grained snapshots that you explicitly tell hammer to make. You would have to remove those yourself (if you want to expire them before the X days) using hammer snaprm. You can script it fairly easily by setting the comment field for each snapshot you take, then filtering out the list based on that. See the manual page. -Matt Matthew Dillon dil...@backplane.com
Re: Snapshots ordering on slave and pfs according to freequency for snapshot management
I've been thinking about the issue of mounting snapshots and I have an idea. In order for a slave snapshot link to work it must reference the @@transactionid:pfsid portion of the softlink relative to the base HAMMER filesystem the PFS was created in. You can do this by creating a dummy subdirectory in the base HAMMER filesystem and then using a nullfs mount to mount that underneath your samba export hierarchy. Then you can have snapshot softlinks that push in through there. You do not have to rename or move the PFS itself. So, here's an example. Lets say you have: /samba_export Your samba export hierarchy /hammer Your hammer filesystem /hammer/pfs/blahSome PFS in the hammer filesystem And now lets asy you want to make snapshots from PFS(s) in the hammer filesystem available to the export. This would work: mkdir /samba_export/fubar mount_null /hammer/pfs /samba_export/fubar Similarly you can adjust or create snapshot softlinks that are relative to the samba_export path's null mount and store them somewhere in /samba_export. In this case you might want to use a different directory: mkdir /samba_export/fubar mkdir /hammer/samba_pfs mount_null /hammer/samba_pfs /samba_export/fubar And then create the specially formatted hammer PFS softlinks in /hammer/samba_pfs: cd /hammer/samba_pfs ln -s @@PFS1 blah (where the PFS number is formatted as @@PFS%05d). There is absolutely nothing preventing you from creating as many @@PFS%05d softlinks as you want, all pointing to the same PFS. The only requirement is that they be created inside the HAMMER filesystem the PFSs were created in. You may have other issues... tracking the snapshots could present a problem since hammer cleanup only allows one snapshot directory to be specified. So you might have to come up with a little script to maintain a second set of snapsoft softlinks with the proper samba path which mirrors the master set which hammer cleanup maintains. Otherwise you will wind up with a lot of stale snapshot softlinks. -Matt
Snapshots ordering on slave and pfs according to freequency for snapshot management
On Fri, Apr 23, 2010 at 2:22 AM, Matthew Dillon dil...@apollo.backplane.com wrote: :For a MS Windows user it will look like the actual pic below. : :http://picasaweb.google.com/sgeorge.ml/OpenBSDDesktop#5462824618389365522 : :Clicking on the snapshot Directory they can get the snapshot contents :-) I am totally amazed that it actually works. Wow, very cool. Talk about instant gratification! It didn't work for the slave pfs because /Backup2/www-5mbak/www-hot was a symlink to /Backup2/pfs/www-hot/ When you share /Backup2/www-5mbak/ using samba you will see an icon for www-hot and all the snapshots but you cannot access them because they are not directly under the directory sahred using samba so I did the following Destroyed the pfs /Backup2/pfs/www-hot/ and created it in /Backup2/pfs/www-5mbak/www-hot/ dfly-bkpsrv# ls -l /Backup2/www-5mbak/www-hot lrwxr-xr-x 1 root wheel 31 Apr 23 14:28 /Backup2/www-5mbak/www-hot - /Backup2/pfs/www-5mbak/www-hot/ and shared /Backup2/pfs/www-5mbak/ using samba. And set the snashot directory to it snapshots=/Backup2/pfs/www-5mbak But the problem I face is that. mirror-stream mirrors all the 5 minutes snapshots from the Master. On the slave pfs my config is dfly-bkpsrv# hammer config /Backup2/www-5mbak/www-hot/ snapshots 1d 2000d prune 1d 5m rebalance 1d 5m reblock 1d 5m recopy3d 10m for the daily snapshots with a prune-min=2d So it seems the 5 min snapshots will never get removed during cleanup. So I guess I will have to configure the 5 mins snapshots on the slave and the daily snapshots on the Master to adjust things? Thanks --Siju
