Re: [zfs-discuss] scripting incremental replication data streams
Hi Edward, My own personal view on this is that the simplest option is the best. In your script, create a new snapshot using one of 2 names. Let's call them SNAPSEND_A and SNAPSEND_B. You can decide which one by checking which currently exists. As manual setup, on the first run, create SNAPSEND_A and send it to your target. This can, obviously, be done incrementally from your last replication/last common snapshot. Now in your script, you would: * Check your source dataset for the existence of SNAPSEND_A and SNAPSEND_B. Let's assume this is the first run after manual setup, so SNAPSEND_A will exist. * Create SNAPSEND_B. Replicate this over to your receiving dataset. * Remove SNAPSEND_A on both sides. This will leave all intermediate snapshots. Next run, it will create SNAPSEND_A again, and remove B when finished. Hope this helps. Karl ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
> I send a replication data stream from one host to another. (and receive). Have you looked at snapsend? http://labs.omniti.com/labs/tools/browser/trunk/snapsend.sh We've used this for a while in production for our backups and replication. We adapted it a bit so it runs every minute without overlapping (through a daemon process) and sends email alerts when errors occur. We'll be updating it in the next week or so some more. We want the process to run on the remote host instead of the primary and have it save x days of snaps for backups as well. This way it will save the last 60 minutes of snapshots (which are sent every minute or less) as well as one snapshot per day for 30 days. All snapshots are created on the primary host and pulled to the remote host to avoid issues. If you are interested we can push the code to github once we test it. -Chris ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
On Wed, Sep 12, 2012 at 7:16 PM, Ian Collins wrote: > On 09/13/12 10:23 AM, Timothy Coalson wrote: > >> Unless i'm missing something, they didn't solve the "matching snapshots" >> thing yet, from their site: >> >> "To Do: >> >> Additional error handling for mismatched snapshots (last destination snap >> no longer exists on the source) walk backwards through the remote snaps >> until a common snapshot is found and destroy non-matching remote snapshots" >> >> > That's what I do as party of my "destroy snapshots not on the source" > check. Over many years of managing various distributed systems, I've > discovered the apparently simple tends to get complex! I tricked the auto-snapshot service into doing that bit for me on the receiving end, so I can just use zfs send -I, however due to when it decides to do the cleanup for each category (it won't clean up any old daily snapshots unless the auto-snapshot property is true when it is time to take a daily snap), adding it to cron requires more care than I'd like. Since I am only replicating between two hosts, though, I can deal with it. I have been tempted to write a cron script to replace the auto-snap service, to remove its naming classes, cleanup oddities, etc, but haven't had a good reason to yet. Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
On 09/13/12 10:23 AM, Timothy Coalson wrote: Unless i'm missing something, they didn't solve the "matching snapshots" thing yet, from their site: "To Do: Additional error handling for mismatched snapshots (last destination snap no longer exists on the source) walk backwards through the remote snaps until a common snapshot is found and destroy non-matching remote snapshots" That's what I do as party of my "destroy snapshots not on the source" check. Over many years of managing various distributed systems, I've discovered the apparently simple tends to get complex! -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
Unless i'm missing something, they didn't solve the "matching snapshots" thing yet, from their site: "To Do: Additional error handling for mismatched snapshots (last destination snap no longer exists on the source) walk backwards through the remote snaps until a common snapshot is found and destroy non-matching remote snapshots" As more evidence, the script doesn't contain any for loops, and has comments that indicate it just uses the latest snapshot on the target. It also doesn't appear that the script uses "zfs hold" anywhere, so I don't think it will stop the auto-snapshots from disappearing on the source (i'm also not sure how the auto-snap service will respond to a failure to destroy a held snapshot, i've had it go into maintenance mode on me for just changing the timezone). Why are you trying to solve this harder problem, rather than using your own snapshots that the auto-snapshot service won't destroy? If you make a snapshot that doesn't start with zfs-auto-snap, then use it (them) as the endpoint(s) for the transfer, adding holds if desired, this entire mess of matching snapshots that may or may not still exist just goes away. Tim On Wed, Sep 12, 2012 at 5:05 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) < opensolarisisdeadlongliveopensola...@nedharvey.com> wrote: > > From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > > boun...@opensolaris.org] On Behalf Of Edward Ned Harvey > > > > Question #2: What's the best way to find the latest matching snap on > both > > the source and destination? At present, it seems, I'll have to build a > list of > > sender snaps, and a list of receiver snaps, and parse and search them, > till I > > find the latest one that exists in both. For shell scripting, this is > very non- > > trivial. > > Someone replied to me off-list and said: > > Try http://blog.infrageeks.com/auto-replicate/ > > Does everything you're looking for in a portable shell script. > > The auto-backup adds in zfs holds > > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Edward Ned Harvey > > Question #2: What's the best way to find the latest matching snap on both > the source and destination? At present, it seems, I'll have to build a list > of > sender snaps, and a list of receiver snaps, and parse and search them, till I > find the latest one that exists in both. For shell scripting, this is very > non- > trivial. Someone replied to me off-list and said: Try http://blog.infrageeks.com/auto-replicate/ Does everything you're looking for in a portable shell script. The auto-backup adds in zfs holds ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
> From: Richard Elling [mailto:richard.ell...@gmail.com] > > Question #2: What's the best way to find the latest matching snap on both > the source and destination? At present, it seems, I'll have to build a list > of > sender snaps, and a list of receiver snaps, and parse and search them, till I > find the latest one that exists in both. For shell scripting, this is very > non- > trivial. > > Actually, it is quite easy. You will notice that "zfs list -t snapshot" shows > the > list in > creation time order. Actually, I already knew that. But at the time of initial send, the latest snap is most likely a "frequent," which will most likely not exist at a later time to be the base of the incremental. Which means, during the incremental, an arbitrary number of the latest snaps on both the sender and receiver probably need to be ignored. In other words, I can't just use a fixed tail or awk command... And I can't alphabetic sort... In shell programming land (unless I want to python or something) I'll have to nest a for-loop in a for-loop. export latestmatch="" export sendersnaps=`ssh otherhost "zfs list -t snapshot | grep $FILESYSTEM" | sed 's/ .*//'"` export receiversnaps=`zfs list -t snapshot | grep $FILESYSTEM | sed 's/ .*//'` for sendersnap in $sendersnaps ; do for receiversnap in $receiversnaps ; do if [ "$sendersnap" = "$receiversnap" ] ; then export latestmatch = $sendersnap fi done done if [ -z "$latestmatch" ] ; then echo "No matching snaps, can't send incremental." # Do a full send, or abort else echo "Doing incremental" ssh otherhost "zfs send -I $latestmatch ... etc etc... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
On 09/13/12 07:44 AM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: I send a replication data stream from one host to another. (and receive). I discovered that after receiving, I need to remove the auto-snapshot property on the receiving side, and set the readonly property on the receiving side, to prevent accidental changes (including auto-snapshots.) Question #1:Actually, do I need to remove the auto-snapshot on the receiving side?Or is it sufficient to simply set the readonly property?Will the readonly property prevent auto-snapshots from occurring? So then, sometime later, I want to send an incremental replication stream.I need to name an incremental source snap on the sending side...which needs to be the latest matching snap that exists on both sides. Question #2:What's the best way to find the latest matching snap on both the source and destination?At present, it seems, I'll have to build a list of sender snaps, and a list of receiver snaps, and parse and search them, till I find the latest one that exists in both.For shell scripting, this is very non-trivial. That's pretty much how I do it. Get the two (sorted) sets of snapshots, remove those that only exist on the remote end (ageing) and send those that only exist locally. The first incremental pair will be the last common snapshot and the first unique local snapshot. I haven't tried this in a script, but it's quite straightforward in C++ using the standard library set container and algorithms. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
When I wrote a script for this, I used separate snapshots, with a different naming convention, to use as the endpoints for the incremental send. With this, it becomes easier: find the newest snapshot with that naming convention on the sending side, and check that it exists on the receiving side. This way, you don't have to deal with the latest frequent/hourly on the target side having been removed from the source side since the last backup. Tim On Wed, Sep 12, 2012 at 2:44 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) < opensolarisisdeadlongliveopensola...@nedharvey.com> wrote: > I send a replication data stream from one host to another. (and receive). > > > I discovered that after receiving, I need to remove the auto-snapshot > property on the receiving side, and set the readonly property on the > receiving side, to prevent accidental changes (including auto-snapshots.)* > *** > > ** ** > > Question #1: Actually, do I need to remove the auto-snapshot on the > receiving side? Or is it sufficient to simply set the readonly property? > Will the readonly property prevent auto-snapshots from occurring? > > ** ** > > So then, sometime later, I want to send an incremental replication stream. > I need to name an incremental source snap on the sending side... which > needs to be the latest matching snap that exists on both sides. > > ** ** > > Question #2: What's the best way to find the latest matching snap on > both the source and destination? At present, it seems, I'll have to > build a list of sender snaps, and a list of receiver snaps, and parse and > search them, till I find the latest one that exists in both. For shell > scripting, this is very non-trivial. > > ** ** > > Thanks... > > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] scripting incremental replication data streams
On Sep 12, 2012, at 12:44 PM, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: > I send a replication data stream from one host to another. (and receive). > I discovered that after receiving, I need to remove the auto-snapshot > property on the receiving side, and set the readonly property on the > receiving side, to prevent accidental changes (including auto-snapshots.) > > Question #1: Actually, do I need to remove the auto-snapshot on the > receiving side? Yes > Or is it sufficient to simply set the readonly property? No > Will the readonly property prevent auto-snapshots from occurring? No > > So then, sometime later, I want to send an incremental replication stream. I > need to name an incremental source snap on the sending side... which needs > to be the latest matching snap that exists on both sides. > > Question #2: What's the best way to find the latest matching snap on both > the source and destination? At present, it seems, I'll have to build a list > of sender snaps, and a list of receiver snaps, and parse and search them, > till I find the latest one that exists in both. For shell scripting, this is > very non-trivial. Actually, it is quite easy. You will notice that "zfs list -t snapshot" shows the list in creation time order. If you are more paranoid, you can get the snapshot's creation time from the "creation" property. For convenience, "zfs get -p creation ..." will return the time as a number. Something like this: for i in $(zfs list -t snapshot -H -o name); do echo $(zfs get -p -H -o value creation $i) $i; done | sort -n -- richard -- illumos Day & ZFS Day, Oct 1-2, 2012 San Fransisco www.zfsday.com richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] scripting incremental replication data streams
I send a replication data stream from one host to another. (and receive). I discovered that after receiving, I need to remove the auto-snapshot property on the receiving side, and set the readonly property on the receiving side, to prevent accidental changes (including auto-snapshots.) Question #1: Actually, do I need to remove the auto-snapshot on the receiving side? Or is it sufficient to simply set the readonly property? Will the readonly property prevent auto-snapshots from occurring? So then, sometime later, I want to send an incremental replication stream. I need to name an incremental source snap on the sending side... which needs to be the latest matching snap that exists on both sides. Question #2: What's the best way to find the latest matching snap on both the source and destination? At present, it seems, I'll have to build a list of sender snaps, and a list of receiver snaps, and parse and search them, till I find the latest one that exists in both. For shell scripting, this is very non-trivial. Thanks... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Scripting
hi most mordern server has separate ILOM that support IPMLtool that can talk to HDD what is your server? does it has separate remote management port? On 8/10/2011 8:36 AM, Lanky Doodle wrote: Hiya, Now I have figured out how to read disks using dd to make LEDs blink, I want to write a little script that iterates through all drives, dd's them with a few thousand counts, stop, then dd's them again with another few thousand counts, so I end up with maybe 5 blinks. I don't want somebody to write something for me, I'd like to be pointed in the right direction so I can build one myself :) Thanks <>___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Scripting
Hiya, Now I have figured out how to read disks using dd to make LEDs blink, I want to write a little script that iterates through all drives, dd's them with a few thousand counts, stop, then dd's them again with another few thousand counts, so I end up with maybe 5 blinks. I don't want somebody to write something for me, I'd like to be pointed in the right direction so I can build one myself :) Thanks -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Scripting zfs send / receive
Hi Clive King has a nice blog entry showing this in action http://blogs.sun.com/clive/entry/replication_using_zfs with associated script at: http://blogs.sun.com/clive/resource/zfs_repl.ksh Which I think answers most of your questions. Enda Ross wrote: > Hey folks, > > Is anybody able to help a Solaris scripting newbie with this? I want to put > together an automatic script to take snapshots on one system and send them > across to another. I've shown the manual process works, but only have a very > basic idea about how I'm going to automate this. > > My current thinking is that I want to put together a cron job that will work > along these lines: > > - Run every 15 mins > - take a new snapshot of the pool > - send the snapshot to the remote system with zfs send / receive and ssh. > (am I right in thinking I can get ssh to work with no password if I create a > public/private key pair? http://www.go2linux.org/ssh-login-using-no-password) > - send an e-mail alert if zfs send / receive fails for any reason (with the > text of the failure message) > - send an e-mail alert if zfs send / receive takes longer than 15 minutes and > clashes with the next attempt > - delete the oldest snapshot on both systems if the send / receive worked > > Can anybody think of any potential problems I may have missed? > > Bearing in mind I've next to no experience in bash scripting, how does the > following look? > > ** > #!/bin/bash > > # Prepare variables for e-mail alerts > SUBJECT="zfs send / receive error" > EMAIL="[EMAIL PROTECTED]" > > NEWSNAP="build filesystem + snapshot name here" > RESULTS=$(/usr/sbin/zfs snapshot $NEWSNAP) > # how do I check for a snapshot failure here? Just look for non blank > $RESULTS? > if $RESULTS; then ># send e-mail >/bin/mail -s $SUBJECT $EMAIL $RESULTS >exit > fi > > PREVIOUSSNAP="build filesystem + snapshot name here" > RESULTS=$(/usr/sbin/zfs send -i $NEWSNAP $PREVIOUSSNAP | ssh -l *user* > *remote-system* /usr/sbin/zfs receive *filesystem*) > # again, how do I check for error messages here? Do I just look for a blank > $RESULTS to indicate success? > if $RESULTS ok; then >OBSOLETESNAP="build filesystem + name here" >zfs destroy $OBSOLETESNAP >ssh -l *user* *remote-system* /usr/sbin/zfs destroy $OBSOLETESNAP > else ># send e-mail with error message >/bin/mail -s $SUBJECT $EMAIL $RESULTS > fi > ** > > One concern I have is what happens if the send / receive takes longer than 15 > minutes. Do I need to check that manually, or will the script cope with this > already? Can anybody confirm that it will behave as I am hoping in that the > script will take the next snapshot, but the send / receive will fail and > generate an e-mail alert? > > thanks, > > Ross > -- > This message posted from opensolaris.org > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Scripting zfs send / receive
Hi Mertol, Yes, I'm using zfs send -i to just send the changes rather than the whole thing. I'll have a think about your suggestion for deleting snapshots too, that does sound like a good idea. Unfortunately I won't be able to synchronise any applications with this script. It's backing up a filestore used by VMware ESX, so could be holding any number of machines. The aim of this is purely to give me a crash consistent backup of those virtual machines just in case. We have other software in place for our regular backups, this is our belt & braces disaster recovery copy :). Ross > Date: Fri, 26 Sep 2008 12:53:06 +0300> From: [EMAIL PROTECTED]> Subject: RE: > [zfs-discuss] Scripting zfs send / receive> To: [EMAIL PROTECTED]> > Hi Ross > ;> > I am no expert in scripting but I was a software engineer once :)> It's > good to desing the script to be able to tolerate errors. > > Instead of > sending the snapshot I'd recommending sending difference of snap> shots. > > Also instead of deleting the oldest snapshot I recommend deleting all but> > newest X number of snapshots. (incase for some reason script is unable to> > delete a snap shot, it will clean them in the next run, as it will always> > leave a fix number of snap shots alive) > > Also you may want your script > talk to application running on the Fs before> and after snapshot to make the > snapshot consistent. > > My 2 cents...> > Best regards> Mertol> > > > Mertol > Ozyoney > Storage Practice - Sales Manager> > Sun Microsystems, TR> Istanbul > TR> Phone +902123352200> Mobile +905339310752> Fax +90212335> Email > [EMAIL PROTECTED]> > > > -Original Message-> From: [EMAIL PROTECTED]> > [mailto:[EMAIL PROTECTED] On Behalf Of Ross> Sent: Friday, September 26, 2008 > 12:43 PM> To: zfs-discuss@opensolaris.org> Subject: [zfs-discuss] Scripting > zfs send / receive> > Hey folks,> > Is anybody able to help a Solaris > scripting newbie with this? I want to put> together an automatic script to > take snapshots on one system and send them> across to another. I've shown the > manual process works, but only have a very> basic idea about how I'm going to > automate this.> > My current thinking is that I want to put together a cron > job that will work> along these lines:> > - Run every 15 mins> - take a new > snapshot of the pool> - send the snapshot to the remote system with zfs send > / receive and ssh.> (am I right in thinking I can get ssh to work with no > password if I create a> public/private key pair?> > http://www.go2linux.org/ssh-login-using-no-password)> - send an e-mail alert > if zfs send / receive fails for any reason (with the> text of the failure > message)> - send an e-mail alert if zfs send / receive takes longer than 15 > minutes> and clashes with the next attempt> - delete the oldest snapshot on > both systems if the send / receive worked> > Can anybody think of any > potential problems I may have missed? > > Bearing in mind I've next to no > experience in bash scripting, how does the> following look?> > > **> > #!/bin/bash> > # Prepare variables for e-mail alerts> SUBJECT="zfs send / > receive error"> EMAIL="[EMAIL PROTECTED]"> > NEWSNAP="build filesystem + > snapshot name here"> RESULTS=$(/usr/sbin/zfs snapshot $NEWSNAP)> # how do I > check for a snapshot failure here? Just look for non blank> $RESULTS?> if > $RESULTS; then> # send e-mail> /bin/mail -s $SUBJECT $EMAIL $RESULTS> exit> > fi> > PREVIOUSSNAP="build filesystem + snapshot name here"> > RESULTS=$(/usr/sbin/zfs send -i $NEWSNAP $PREVIOUSSNAP | ssh -l *user*> > *remote-system* /usr/sbin/zfs receive *filesystem*)> # again, how do I check > for error messages here? Do I just look for a blank> $RESULTS to indicate > success?> if $RESULTS ok; then> OBSOLETESNAP="build filesystem + name here"> > zfs destroy $OBSOLETESNAP> ssh -l *user* *remote-system* /usr/sbin/zfs > destroy $OBSOLETESNAP> else > # send e-mail with error message> /bin/mail -s > $SUBJECT $EMAIL $RESULTS> fi> > **> > One > concern I have is what happens if the send / receive takes longer than> 15 > minutes. Do I need to check that manually, or will the script cope with> this > already? Can anybody confirm that it will behave as I am hoping in that> the >
[zfs-discuss] Scripting zfs send / receive
Hey folks, Is anybody able to help a Solaris scripting newbie with this? I want to put together an automatic script to take snapshots on one system and send them across to another. I've shown the manual process works, but only have a very basic idea about how I'm going to automate this. My current thinking is that I want to put together a cron job that will work along these lines: - Run every 15 mins - take a new snapshot of the pool - send the snapshot to the remote system with zfs send / receive and ssh. (am I right in thinking I can get ssh to work with no password if I create a public/private key pair? http://www.go2linux.org/ssh-login-using-no-password) - send an e-mail alert if zfs send / receive fails for any reason (with the text of the failure message) - send an e-mail alert if zfs send / receive takes longer than 15 minutes and clashes with the next attempt - delete the oldest snapshot on both systems if the send / receive worked Can anybody think of any potential problems I may have missed? Bearing in mind I've next to no experience in bash scripting, how does the following look? ** #!/bin/bash # Prepare variables for e-mail alerts SUBJECT="zfs send / receive error" EMAIL="[EMAIL PROTECTED]" NEWSNAP="build filesystem + snapshot name here" RESULTS=$(/usr/sbin/zfs snapshot $NEWSNAP) # how do I check for a snapshot failure here? Just look for non blank $RESULTS? if $RESULTS; then # send e-mail /bin/mail -s $SUBJECT $EMAIL $RESULTS exit fi PREVIOUSSNAP="build filesystem + snapshot name here" RESULTS=$(/usr/sbin/zfs send -i $NEWSNAP $PREVIOUSSNAP | ssh -l *user* *remote-system* /usr/sbin/zfs receive *filesystem*) # again, how do I check for error messages here? Do I just look for a blank $RESULTS to indicate success? if $RESULTS ok; then OBSOLETESNAP="build filesystem + name here" zfs destroy $OBSOLETESNAP ssh -l *user* *remote-system* /usr/sbin/zfs destroy $OBSOLETESNAP else # send e-mail with error message /bin/mail -s $SUBJECT $EMAIL $RESULTS fi ** One concern I have is what happens if the send / receive takes longer than 15 minutes. Do I need to check that manually, or will the script cope with this already? Can anybody confirm that it will behave as I am hoping in that the script will take the next snapshot, but the send / receive will fail and generate an e-mail alert? thanks, Ross -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
