Re: Suggestions for multilevel backup of single machine?

[Glyn Astill]

--- On Wed, 15/4/09, Johannes Wiedersich  
wrote:

> From: Johannes Wiedersich 
> Subject: Re: Suggestions for multilevel backup of single machine?
> To: debian-user@lists.debian.org
> Date: Wednesday, 15 April, 2009, 3:32 PM
> Andrei Popescu wrote:
> > On Sat,11.Apr.09, 13:47:08, James Youngman wrote:
> >  
> >> (2) It would be useful to have a historic backup
> capability too (e.g.
> >> the way the filesystem looked yesterday, last
> week, last month and a
> >> year ago), at least for filesystems like /home.
> > 
> > etckeeper. Admittedly, it was created for /etc (it
> hooks into dpkg and 
> > records all changes), but it should be usable also for
> /home. You could 
> > also use git directly if you don't care about file
> permissions.
> 
> Probably overkill. It should work fine for text data,
> however, I don't
> think it would scale well for binary data. I'm not sure
> it would handle
> things like a changed exif comment gracefully. In my
> experience git
> becomes sluggish for large repositories of GBs worth of
> data and I have
> hundreds of GB of data.
> 
> I use a custom rsync-script that keeps hard links of all
> unchanged
> files. Occasionally, I delete old backups and I think
> it's not
> straightforward to achieve that with git.

How about rsnapshot?





--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Johannes Wiedersich]

Andrei Popescu wrote:
> On Sat,11.Apr.09, 13:47:08, James Youngman wrote:
>  
>> (2) It would be useful to have a historic backup capability too (e.g.
>> the way the filesystem looked yesterday, last week, last month and a
>> year ago), at least for filesystems like /home.
> 
> etckeeper. Admittedly, it was created for /etc (it hooks into dpkg and 
> records all changes), but it should be usable also for /home. You could 
> also use git directly if you don't care about file permissions.

Probably overkill. It should work fine for text data, however, I don't
think it would scale well for binary data. I'm not sure it would handle
things like a changed exif comment gracefully. In my experience git
becomes sluggish for large repositories of GBs worth of data and I have
hundreds of GB of data.

I use a custom rsync-script that keeps hard links of all unchanged
files. Occasionally, I delete old backups and I think it's not
straightforward to achieve that with git.

Regards,

Johannes


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Adam Hardy]

Rick Thomas on 13/04/09 22:19, wrote:

It does exactly what you are asking for and it does it automatically.

And it's a debian package:


$ aptitude show rsnapshot
Package: rsnapshot
State: installed
Automatically installed: no
Version: 1.2.9-1
Priority: optional
Section: utils
Maintainer: Simon Boulet 
Uncompressed Size: 457k
Depends: perl, rsync, logrotate
Recommends: ssh
Description: local and remote filesystem snapshot utility
 rsnapshot is a filesystem snapshot utility. It can take incremental 
snapshots of local and remote filesystems for any number of
 machines. rsnapshot makes extensive use of hard links, so disk space 
is only used when absolutely necessary.


 Homepage: http://www.rsnapshot.org/

Tags: admin::backup, implemented-in::perl, implemented-in::shell, 
role::program, use::storing, works-with::file


How does rsnapshot compare to faubackup? I'm using faubackup for a year or so 
now and it seems to work fine.


It's also in debian packages and does pretty much exactly the same as rsnapshot:

Description: Backup System using a Filesystem for Storage
 This Program uses a filesystem on a hard drive for incremental
 and full backups.
 All Backups can easily be accessed by standard filesystem tools
 (ls, find, grep, cp, ...)
 .
 Later Backups to the same filesystem will automatically be
 incremental, as unchanged files are only hard-linked with
 the existing version of the file.
Tag: admin::backup, hardware::storage, interface::commandline, 
interface::daemon, role::program, scope::utility, use::storing



--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Rick Thomas]

On Apr 11, 2009, at 9:44 AM, Tapani Tarvainen wrote:

On Sat, Apr 11, 2009 at 01:47:08PM +0100, James Youngman  
(j...@gnu.org) wrote:



(2) It would be useful to have a historic backup capability too (e.g.
the way the filesystem looked yesterday, last week, last month and a
year ago), at least for filesystems like /home.



What are good solutions for doing (2)?


I'm using rsnapshot, and given what you told of your setup,
I think you'd be happy with it, too.


I've been using rsnapshot for a couple of years now (4 "hourly"/day, 7  
"daily"/week, 4 "weekly"/month, 12 "monthly"/year, 10 "yearly", and a  
single "epoch" taken when the system was originally set up.)


It's quite robust and has saved my bacon a couple of times.

It does exactly what you are asking for and it does it automatically.

And it's a debian package:


$ aptitude show rsnapshot
Package: rsnapshot
State: installed
Automatically installed: no
Version: 1.2.9-1
Priority: optional
Section: utils
Maintainer: Simon Boulet 
Uncompressed Size: 457k
Depends: perl, rsync, logrotate
Recommends: ssh
Description: local and remote filesystem snapshot utility
 rsnapshot is a filesystem snapshot utility. It can take incremental  
snapshots of local and remote filesystems for any number of
 machines. rsnapshot makes extensive use of hard links, so disk  
space is only used when absolutely necessary.


 Homepage: http://www.rsnapshot.org/

Tags: admin::backup, implemented-in::perl, implemented-in::shell,  
role::program, use::storing, works-with::file




--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Mark Neyhart]

James Youngman wrote:

> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.
> 
I am currently using Disk Dirvish  which does
an excellent job of keeping multiple backup copies.  It uses the
--link option of rsync to reduce the amount of disk space required.




-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Mark Neyhart]

James Youngman wrote:

> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.


I am currently using Disk Dirvish which does an excellent job of keeping
multiple backup copies.  It uses the --link option of rsync to reduce the amount
of disk space required.  You can check it out at http://www.dirvish.org


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Henri Salo]

On Sat, Apr 11, 2009 at 01:47:08PM +0100, James Youngman wrote:
> Here's my current backup arrangement:
> 
> Data is stored in filesystems on LVM volumes over RAID1.  While RAID1
> presents some protection from disk failure, it gives no protection
> against data corruption due to flaky hardware or data loss caused by
> fire or theft.
> 
> Therefore I have an offsite backup arrangement.   This consists of two
> rsync backups.  One backup goes to a local disk (different disk
> manufacturer, different disk controller) and the other rsync backup is
> to a disk at work.  This works a bit but the outgoing bandwidth on my
> cable connection is low (about 0.3 Mbps).  If I make a large change to
> the machine (e.g. dist-upgrade) I physically swap the home and work
> backup disks (this is the main reason for keeping the local backup
> too).  This at least allows me to place an upper limit on the amount
> of data I would lose in the case of (e.g.) a fire.
> 
> However, there are two respects in which I think some improvement
> would be useful:
> 
> (1) Quite a lot of the files on my system are files I never expect to
> change again.  I plan to write a few scripts which will tell me if a
> file that hadn't been modified in, say, two years was in fact recently
> modified.  This could give me early warning that the disk controller
> has gone berserk (again).
> 
> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.
> 
> What are good solutions for doing (2)?   (Please only recommend
> software you're using yourself :)
> 
> Thanks,
> James.

I use duplicity 

---
Henri Salo


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Cameron Hutchison]

James Youngman  writes:

>(2) It would be useful to have a historic backup capability too (e.g.
>the way the filesystem looked yesterday, last week, last month and a
>year ago), at least for filesystems like /home.

>What are good solutions for doing (2)?   (Please only recommend
>software you're using yourself :)

I do this using rsync with the --link-dest option. The idea is that you
sync A to B with reference to C. If a file in A and C are the same, a
hard link is created in B instead of a copy of the file. This means the
only new files in B are files that have changed in A relative to C.

When I start the backup I create a directory for the new backup, named
with the date of the backup. If a symlink "latest" exists, I run rsync
with the argument "--link-dest=.../latest". The new backup will be
performed relative to the latest backup. When rsync is complete, I
update the "latest" link to point to the just-performed backup.

I periodically purge old backups, leaving behind backups ending in 01.
This leaves one backup per month (e.g. 20090301, 20090201, etc) for a
longer term historical backup.

I have a script that does this for me which you are welcome to. Just
email me.


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Jeff Chimene]

On 12/23/-28158 12:59 PM, James Youngman wrote:

Here's my current backup arrangement:

Data is stored in filesystems on LVM volumes over RAID1.  While RAID1
presents some protection from disk failure, it gives no protection
against data corruption due to flaky hardware or data loss caused by
fire or theft.

Therefore I have an offsite backup arrangement.   This consists of two
rsync backups.  One backup goes to a local disk (different disk
manufacturer, different disk controller) and the other rsync backup is
to a disk at work.  This works a bit but the outgoing bandwidth on my
cable connection is low (about 0.3 Mbps).  If I make a large change to
the machine (e.g. dist-upgrade) I physically swap the home and work
backup disks (this is the main reason for keeping the local backup
too).  This at least allows me to place an upper limit on the amount
of data I would lose in the case of (e.g.) a fire.

However, there are two respects in which I think some improvement
would be useful:

(1) Quite a lot of the files on my system are files I never expect to
change again.  I plan to write a few scripts which will tell me if a
file that hadn't been modified in, say, two years was in fact recently
modified.  This could give me early warning that the disk controller
has gone berserk (again).
   


Isn't this what the file alteration monitor is for?


(2) It would be useful to have a historic backup capability too (e.g.
the way the filesystem looked yesterday, last week, last month and a
year ago), at least for filesystems like /home.
   
I think it's extreme going back farther than a month. A 
grandfather/father/son scheme tailored to your requirements should be 
more than sufficient. But, it's your system...

What are good solutions for doing (2)?   (Please only recommend
software you're using yourself :)

   
I like backup-manager for SOHO backups. I've been through duplicity, 
backup-ninja and possibly others. Once I resolved A REALLY ANNOYING BUG, 
it works very well. I, too, do offsite backups (via scp). backup-manager 
fails gracefully (I changed the sshd port thus defeating 
backup-manager's ability to successfully connect via scp. This only left 
the backup on the local machine waiting to be copied to the remote 
machine once I updated the configuration). I also like that the email 
logs contain enough, but not too much info.


On the downside, I don't think backup-manager has a calendar of 
sufficient duration to handle your "last month, year ago" requirement. 
You may have to solve that via cron.


You should be prepared to try several of the available backup packages 
and decide for yourself. You're obviously the type who likes to be 
"fer-sure, fer-sure".


Cheers,


--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[owens]

>
>
>
> Original Message 
>From: rac...@makeworld.com
>To: debian-user@lists.debian.org
>Subject: Re: Suggestions for multilevel backup of single machine?
>Date: Sat, 11 Apr 2009 11:05:30 -0500
>
>>On Sat, 11 Apr 2009 13:47:08 +0100
>>James Youngman  wrote:
>>
>>> (2) It would be useful to have a historic backup capability too
>(e.g.
>>> the way the filesystem looked yesterday, last week, last month and
>a
>>> year ago), at least for filesystems like /home.
>>> 
>>> What are good solutions for doing (2)?   (Please only recommend
>>> software you're using yourself :)
>>> 
>>> Thanks,
>>> James.
>>> 
>>> 
>>
>>I myself use BackupPC (both at home and at work). 
>>To me, it's easy to setup, easy to use, and does everything I want
>it
>>to (including historical).
>>
>>
>>
>>
>>-- 
>>Best regards,
>>
>>Chris
>>
>>()  ascii ribbon campaign - against html e-mail
>>/\  www.asciiribbon.org   - against proprietary attachments
>>
>>  "There's no place like 127.0.0.1"
>>
I'm also a Backuppc user.  It even works for my wife's v...@#ta machine
Larry
>>
>>-- 
>>To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
>>with a subject of "unsubscribe". Trouble? Contact listmas...@lists.d
>ebian.org
>>
>>




--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Chris]

On Sat, 11 Apr 2009 13:47:08 +0100
James Youngman  wrote:

> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.
> 
> What are good solutions for doing (2)?   (Please only recommend
> software you're using yourself :)
> 
> Thanks,
> James.
> 
> 

I myself use BackupPC (both at home and at work). 
To me, it's easy to setup, easy to use, and does everything I want it
to (including historical).




-- 
Best regards,

Chris

()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments

"There's no place like 127.0.0.1"


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Andrei Popescu]

On Sat,11.Apr.09, 13:47:08, James Youngman wrote:
 
> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.

etckeeper. Admittedly, it was created for /etc (it hooks into dpkg and 
records all changes), but it should be usable also for /home. You could 
also use git directly if you don't care about file permissions.

Regards,
Andrei
-- 
If you can't explain it simply, you don't understand it well enough.
(Albert Einstein)


signature.asc
Description: Digital signature

Re: Suggestions for multilevel backup of single machine?

[Chris Burkhardt]

James Youngman wrote:
> [...]
> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.
> 
> What are good solutions for doing (2)?   (Please only recommend
> software you're using yourself :)

I use rdiff-backup for incremental backups of /home:

http://www.gnu.org/savannah-checkouts/non-gnu/rdiff-backup/

- Chris B


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Suggestions for multilevel backup of single machine?

[Tapani Tarvainen]

On Sat, Apr 11, 2009 at 01:47:08PM +0100, James Youngman (j...@gnu.org) wrote:

> (2) It would be useful to have a historic backup capability too (e.g.
> the way the filesystem looked yesterday, last week, last month and a
> year ago), at least for filesystems like /home.

> What are good solutions for doing (2)?

I'm using rsnapshot, and given what you told of your setup,
I think you'd be happy with it, too.

-- 
Tapani Tarvainen


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Suggestions for multilevel backup of single machine?

[James Youngman]

Here's my current backup arrangement:

Data is stored in filesystems on LVM volumes over RAID1.  While RAID1
presents some protection from disk failure, it gives no protection
against data corruption due to flaky hardware or data loss caused by
fire or theft.

Therefore I have an offsite backup arrangement.   This consists of two
rsync backups.  One backup goes to a local disk (different disk
manufacturer, different disk controller) and the other rsync backup is
to a disk at work.  This works a bit but the outgoing bandwidth on my
cable connection is low (about 0.3 Mbps).  If I make a large change to
the machine (e.g. dist-upgrade) I physically swap the home and work
backup disks (this is the main reason for keeping the local backup
too).  This at least allows me to place an upper limit on the amount
of data I would lose in the case of (e.g.) a fire.

However, there are two respects in which I think some improvement
would be useful:

(1) Quite a lot of the files on my system are files I never expect to
change again.  I plan to write a few scripts which will tell me if a
file that hadn't been modified in, say, two years was in fact recently
modified.  This could give me early warning that the disk controller
has gone berserk (again).

(2) It would be useful to have a historic backup capability too (e.g.
the way the filesystem looked yesterday, last week, last month and a
year ago), at least for filesystems like /home.

What are good solutions for doing (2)?   (Please only recommend
software you're using yourself :)

Thanks,
James.


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org