Re: [DNG] Backup revisited - the rsync time machine

[Hendrik Boom]

On Mon, Feb 05, 2018 at 09:08:59AM -1000, Joel Roth wrote:
> On Mon, Feb 05, 2018 at 09:15:33AM -0500, Hendrik Boom wrote:
> > On Fri, Feb 02, 2018 at 05:32:36PM -1000, Joel Roth wrote:
> > > Dear list,
> > > 
> > > For years I'd used a couple of rsync scripts for backup,
> > > usually just full snapshots.
> > > 
> > > I knew there is an option using hardlinks that behaves like
> > > the Mac Time Machine app, giving cheap incremental backups.
> > > 
> > > https://blog.interlinked.org/tutorials/rsync_time_machine.html
> > > 
> ()
> > > probably someone has done it better...
> > > 
> > 
> > There's rdiff-backup, which uses an efficient algorithm to identify what 
> > has changed and transmit the diffs over the network.  It also keeps a 
> > history of old backups on the backup drive, so you can restore as of a 
> > previous date.
> 
> I used rdiff-backup years ago. ISTR it litters the directory tree
> with index files.

It has index files on the backup, right?  They are for storing things 
like reverse diffs so it can reconstruct old backups.  I've never  
noticed it litterring the actual working file system -- just the 
backups.

-- hendrik
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] Backup revisited - the rsync time machine

[Joel Roth]

On Mon, Feb 05, 2018 at 09:15:33AM -0500, Hendrik Boom wrote:
> On Fri, Feb 02, 2018 at 05:32:36PM -1000, Joel Roth wrote:
> > Dear list,
> > 
> > For years I'd used a couple of rsync scripts for backup,
> > usually just full snapshots.
> > 
> > I knew there is an option using hardlinks that behaves like
> > the Mac Time Machine app, giving cheap incremental backups.
> > 
> > https://blog.interlinked.org/tutorials/rsync_time_machine.html
> > 
()
> > probably someone has done it better...
> > 
> 
> There's rdiff-backup, which uses an efficient algorithm to identify what 
> has changed and transmit the diffs over the network.  It also keeps a 
> history of old backups on the backup drive, so you can restore as of a 
> previous date.

I used rdiff-backup years ago. ISTR it litters the directory tree
with index files.

cheers,
 
> And the files themselves are readable on the backup drive, as ng as you 
> don't have them compressed or encrypted.
> 
> -- hendrik
> ___
> Dng mailing list
> Dng@lists.dyne.org
> https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

-- 
Joel Roth
  

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] Backup revisited - the rsync time machine

[viverna]

il devuanizzato Joel Roth  il 03-02-18 04:32:36 ha scritto:
> Dear list,
> 
> For years I'd used a couple of rsync scripts for backup,
> usually just full snapshots.
> 
> I knew there is an option using hardlinks that behaves like
> the Mac Time Machine app, giving cheap incremental backups.
You may try rsnapshot.
It is based on rsync and use hard links whenever possible.

-- 
viverna
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] Backup revisited - the rsync time machine

[Erik Christiansen]

On 02.02.18 17:32, Joel Roth wrote:
> Probably you all have something much better, but for the 
> sake of discussion, and will post my humble offering. 

Not better, but a data point for anyone else backing up to a flash drive,
which is convenient for the off-site backup. Rsync compares checksums
between source and destination, reprocessing a single write fail, and
erroring on a second. BUT, if the flash drive has bit drop-outs after
time, then that isn't detectable by rsync. Files not updated in a
subsequent backup remain corrupted, and any updated are simply
overwritten, and liable to repeated bit loss.

OK, I've only ever had one flash drive which started to lose bits (in 
6% of files before I detected it), but my backup script now does a
diff -qr on the backup after the rsync. It's a bit time consuming, but
you don't have to watch it.

Erik
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

[DNG] Backup revisited - the rsync time machine

[Joel Roth]

Dear list,

For years I'd used a couple of rsync scripts for backup,
usually just full snapshots.

I knew there is an option using hardlinks that behaves like
the Mac Time Machine app, giving cheap incremental backups.

https://blog.interlinked.org/tutorials/rsync_time_machine.html

And now I fool around with it myself. 

Since I have map UUIDs to mount points in /etc/fstab,
I can put the full paths in the backup script 
and simply run it without parameters to get a 
date-and-time-stamped directory containing a full backup.

Probably you all have something much better, but for the 
sake of discussion, and will post my humble offering. 

The clever part of the code is using a symlink
.../backups/current/ to provide rsync with the --link-dest
argument, the tree of files available for hardlinking during
next backup pass. 

Also, the one-file-system argument to rsync lets me backup
the root directory without pulling in other mounts. 

The current script doesn't support copying over a network,
but can be easily achieved by consulting online resources
(left as an exercise to the reader.)

Obviously, you will need to configure it. Note that the directories
excluded from backup are created in the last step.

probably someone has done it better...


# rsync-time

#!/bin/sh
DATE=`date "+%Y.%m.%d-%H:%M:%S"`
BACKUP_ROOT=/mnt/matte/backups
SOURCE_DIR=/
BACKUP_DIR="$BACKUP_ROOT/back-$DATE"
HARDLINK_ROOT=$BACKUP_ROOT/current
HOME_=home/jill
LOG=/var/log/backup.log
EXCLUDE_HOME="\
--exclude $HOME_/.mozilla/firefox/n5c5mmhh.default/cache2  \
--exclude $HOME_/.cache/mozilla/firefox/n5c5mmhh.default/cache2 \
--exclude $HOME_/.opera/cache4 \
--exclude $HOME_/Desktop \
--exclude $HOME_/Downloads \
--exclude $HOME_/.cpan \
--exclude $HOME_/.cpanm \
--exclude $HOME_/.cpanplus \
--exclude $HOME_/perl5 "
EXCLUDE_SYS="\
--exclude /dev \
--exclude /proc \
--exclude /sys  \
--exclude /tmp \
--exclude /var/cache/apt/archives \
--exclude /var/run  "
CMD="rsync -avxP $EXCLUDE_HOME $EXCLUDE_SYS --link-dest=$HARDLINK_ROOT 
$SOURCE_DIR $BACKUP_DIR"
echo $CMD
$CMD
rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
echo "$DATE - Starting hardlink backup to $BACKUP_ROOT" >> $LOG
echo $CMD >> $LOG
echo "Completed" >> $LOG
rm -f $HARDLINK_ROOT
ln -s $BACKUP_DIR $HARDLINK_ROOT
cd $BACKUP_DIR
for dir in dev proc sys  tmp var/run var/cache/apt/archives ; do mkdir $dir; 
done
cd -

# end

-- 
Joel Roth
  

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng