Re: Confusion about including and excluding files in tarsnap.conf versus the command line
Hi Graham, Thanks for your reply, and my humble apologies for the ridiculous length of time in responding. I blame Christmas. Have finally got back to this and I see the mistake in my script that meant the output file wasn't even being created. Totally my fault. Sorry. Craig On Sat, 2018-11-24 at 12:30 -0800, Graham Percival wrote: > On Fri, Nov 23, 2018 at 05:33:03PM -0800, Craig Hartnett wrote: > > Hi Graham, > > > > Thanks for your reply. Sorry for not getting back to you sooner. > > No problem. I'm glad it worked out! > > > The cron job has been happily running all week, so things are good. > > However, the script at https://www.tarsnap.com/tips.html#receive-mail > > refers to an output file that isn't actually created, so should I be > > specifying the output file name on the command line or somewhere else? > > Doesn't seem like it's needed though, as cron sends along the output > > anyway. > > I'm not certain that I follow. That script creates a (temporary) output file, > dumps info into it[1], emails it to you, then deletes the output file. > > Hmm, the script would be easier to read if I had a > > # Clean up > rm $tarsnap_output_filename > > comment. > > [1] I just noticed that it only sends info if you have "print-stats" in a > config file; otherwise a successful backup will send an empty email with the > subject "Tarsnap backup success". I think that's the right behaviour to have, > but I should add a note about print-stats. > > Cheers, > - Graham
Re: Confusion about including and excluding files in tarsnap.conf versus the command line
On Fri, Nov 23, 2018 at 05:33:03PM -0800, Craig Hartnett wrote: > Hi Graham, > > Thanks for your reply. Sorry for not getting back to you sooner. No problem. I'm glad it worked out! > The cron job has been happily running all week, so things are good. > However, the script at https://www.tarsnap.com/tips.html#receive-mail > refers to an output file that isn't actually created, so should I be > specifying the output file name on the command line or somewhere else? > Doesn't seem like it's needed though, as cron sends along the output > anyway. I'm not certain that I follow. That script creates a (temporary) output file, dumps info into it[1], emails it to you, then deletes the output file. Hmm, the script would be easier to read if I had a # Clean up rm $tarsnap_output_filename comment. [1] I just noticed that it only sends info if you have "print-stats" in a config file; otherwise a successful backup will send an empty email with the subject "Tarsnap backup success". I think that's the right behaviour to have, but I should add a note about print-stats. Cheers, - Graham
Re: Confusion about including and excluding files in tarsnap.conf versus the command line
On Sun, Nov 11, 2018 at 06:02:31PM -0800, Craig Hartnett wrote: > But I have to ask now, and it's something I think would be very useful > to include on the "Getting started" page, how long should the first > back-up take? I realise that's asking how long is a piece of string, but > my first back-up has been going for a little over 28 hours now. > According to my dry run I should have about 85 GB of compressed data to > upload. I'm on a (Shaw) cable connection ... not their fastest, but I > suspect the connection isn't the bottleneck. I'm pretty certain that your first upload will be limited by your upload speed. 85 GB is a lot, and cable ISPs often have a lower upload speed than download. If your Shaw cable offers 150 kb/s of sustained upload, then I think it will take 6.5 days. Let's do this in MB: 85000 MB / 0.150 MB/s = 57 seconds 57 seconds / 60 s/m / 60 m/h = 157.4 hours If you have sustained upload of 300 kb/s, that would only be 78.7 hours, or 3 days. Which, come to think of it, looks close to the time you estimated. (Also, note that some ISPs offer a higher "burst" upload rate, but throttle it down to a lower level if lasts for longer than some amount of time.) > Interrupting the process with "killall -SIGUSR1 tarsnap" shows me that > there is progress, but I have no idea if I'm 10% in or 90% in. Right. I've been working on displaying more info (if requested by the user), but nothing's made it in yet. Cheers, - Graham
Re: Confusion about including and excluding files in tarsnap.conf versus the command line
Hi Graham, On Thu, 2018-11-08 at 18:07 -0800, Graham Percival wrote: > Welcome! Thanks, and thanks for your reply. > Careful there! "--exclude" only accepts a single value. OK, that was clear in the config file, but not in the man page. > "tarsnap -c -f foo" doesn't work, even if we have an "include bar" > in the config file). Yes, I noticed. > Please delete the "include" lines, then use this command-line: > /usr/local/bin/tarsnap -c \ > -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \ > /media/USER/PATH \ > /home/USER\ > /etc \ > /opt OK, thanks, that worked. > For more information, please see: > http://www.tarsnap.com/selecting-files.html > including the warnings about trying it with --dry-run before doing a real > backup. I think that in your case, you don't need "include" at all. Yup, I did read that page. Thanks. I did a dry run with your new command above, including with the verbose parameter, and was happy with the paths that were included. But I have to ask now, and it's something I think would be very useful to include on the "Getting started" page, how long should the first back-up take? I realise that's asking how long is a piece of string, but my first back-up has been going for a little over 28 hours now. According to my dry run I should have about 85 GB of compressed data to upload. I'm on a (Shaw) cable connection ... not their fastest, but I suspect the connection isn't the bottleneck. Interrupting the process with "killall -SIGUSR1 tarsnap" shows me that there is progress, but I have no idea if I'm 10% in or 90% in. The activity in my account shows that I've been charged most recently for about 31 GB of storage, so that seems to suggest that after 28 hours I am only about 36% done (21/85). That means this back-up will take until Wednesday morning! Am I right? Probably wouldn't have changed my course of action, to be honest (except that I might have started the back-up on Friday afternoon), but would have been useful to know up front. Craig
Re: Confusion about including and excluding files in tarsnap.conf versus the command line
On Thu, Nov 08, 2018 at 07:57:42PM -0800, Michael Sierchio wrote: >envoyé par un téléphone mobile avec «autocorrect» démoniaque >On Thu, Nov 8, 2018, 18:07 Graham Percival <[1]gperc...@tarsnap.com >wrote: > > Careful there! "--exclude" only accepts a single value. > >Except it takes globbed names, right? Yes, provided that the shell doesn't expand them. (so put the globbed name inside quotes) For example: $ ls double-zeros* double-zeros-bsd.tar double-zeros-gnu.tar $ tarsnap -c --dry-run -v --include double-zeros* double-zeros-* a double-zeros-bsd.tar $ tarsnap -c --dry-run -v --include "double-zeros*" double-zeros-* a double-zeros-bsd.tar a double-zeros-gnu.tar $ I'll add a warning about this to selecting-files.html. Cheers, - Graham
Re: Confusion about including and excluding files in tarsnap.conf versus the command line
On Thu, Nov 08, 2018 at 05:20:12PM -0800, Craig Hartnett wrote: > Just signed up Welcome! > (e.g., tarsnap --dry-run --print-stats --humanize-numbers > -c /media/USER/PATH > --exclude /media/USER/PATH/.Trash-1000 /media/USER/PATH/lost > +found /media/USER/PATH/OTHER-EXCLUDED-DIRECTORY) Careful there! "--exclude" only accepts a single value. If you want to exclude multiple values on the command-line (which is not recommended; I would definitely put them in your config file), you would need: --exclude /media/USER/PATH/.Trash-1000 --exclude /media/USER/PATH/lost+found --exclude /media/USER/PATH/OTHER-EXCLUDED-DIRECTORY > So here I am today and now wanting to run my first real back-up and I'm > stuck on the same problem. The information in step 1A at > http://www.tarsnap.com/simple-usage.html still seems to require me to > put my list of included and excluded paths on the command line. Why is > this? Why doesn't tarsnap just get the list from tarsnap.conf? Hmm, I'll tweak simple-usage.html somewhat. tarsnap does indeed get the values from tarsnap.conf, with three exceptions: -c: tarsnap doesn't know which mode (create/extract/test/etc.), so we need to give this on the command-line -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)": since the archive name changes every day, we need to give it here. /MY/DATADIR: we need to give the "root" directory to include (running "tarsnap -c -f foo" doesn't work, even if we have an "include bar" in the config file). > exclude /media/USER/PATH/.Trash-1000 > exclude /media/USER/PATH/lost+found > exclude /media/USER/PATH/_hold > > include /media/USER/PATH > include /home/USER > include /etc > include /opt Please delete the "include" lines, then use this command-line: /usr/local/bin/tarsnap -c \ -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \ /media/USER/PATH\ /home/USER \ /etc\ /opt (apologies for the confusing name of "--include"... it should really be "--include-only". Unfortunately the name came from the 1980s-era tar(1) program, is still in use nowadays, and we don't want to break backwards compatibility with the expected behaviour of tar(1).) For more information, please see: http://www.tarsnap.com/selecting-files.html including the warnings about trying it with --dry-run before doing a real backup. I think that in your case, you don't need "include" at all. Cheers, - Graham PS: I almost recommended that you kept the "include" lines and tried: /usr/local/bin/tarsnap -c \ -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \ / However, when I was just about to send the email, I noticed: - The process of recursing into subdirectories is also affected. Adding this to the config file: include /home and then attempting to back up / results in nothing being backed up. http://www.tarsnap.com/selecting-files.html So clearly I already made that mistake a few years ago when I was writing that page! - Graham
