Re: backup strategies
On Mon, Oct 31, 2005 at 02:37:32PM -0500, Paul Mather wrote: Not quite: NetBSD also features softupdates and also supports snapshots (though I don't know how stable it is, as I've never tried it on my NetBSD system). The snapshot interface under NetBSD is different from That's also good to know. Thanks for the information. -- Csaba Henk My sense of humour is often too subtle to cope with getting smileyd. Please don't take it personal. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
Thanks for all the tips and answers, I will consider the mentioned
alternatives.
Yet I have one more question...
On Sun, Oct 30, 2005 at 01:22:35PM -0600, Eric Schuele wrote:
dump(8) will create a snapshot of a live filesystem, dump the snapshot
and then remove the snapshot, if given the correct flags ('-L').
Can even a full bakcup done safely on a live filesystem by dump -L?
As dump(8) says when explaining the -L flag:
To obtain a consistent dump image, dump takes a snapshot of the file
system in the .snap directory in the root of the file system being
dumped and then does a dump of the snapshot.
I don't see how the temporary snapshot can improve the
reliability/consistency/correctness of the dump. Could someone explain
this?
--
Csaba Henk
My sense of humour is often too subtle to cope with getting smileyd.
Please don't take it personal.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
Csaba Henk wrote:
Thanks for all the tips and answers, I will consider the mentioned
alternatives.
Yet I have one more question...
On Sun, Oct 30, 2005 at 01:22:35PM -0600, Eric Schuele wrote:
dump(8) will create a snapshot of a live filesystem, dump the snapshot
and then remove the snapshot, if given the correct flags ('-L').
Can even a full bakcup done safely on a live filesystem by dump -L?
As dump(8) says when explaining the -L flag:
To obtain a consistent dump image, dump takes a snapshot of the file
system in the .snap directory in the root of the file system being
dumped and then does a dump of the snapshot.
I don't see how the temporary snapshot can improve the
reliability/consistency/correctness of the dump. Could someone explain
this?
How do snapshots work and how do they provide the consistency necessary
for a dump?
(Well, I'm probably not the best one to answer this, but I'll take a swing.)
Basically, when taking a snapshot... any new activity on the filesystem
in question is suspended. All presently executing syscalls are allowed
to finish. The filesystem is synchronized as if unmounting, and then a
snapshot file is created. At this point filesystem activity is resumed.
The real magic is in the snapshot file. The file is a snapshot of the
filesystem metadata at the time the snapshot was taken. From this point
on... as long as the snapshot exists on your machine... it tracks the
changes that occur within the filesystem from the time you took the
snapshot to the present. What this means is the snapshot file holds
pointers to the data on your filesystem. As that data changes (since
the snapshot), the old data gets claimed by the snapshot... and the new
data is written to disk. Same with deleting data... the deleted data
gets claimed by the snapshot and the filesystem looses it. Any
unchanged data is simply referred to (in the snapshot file) as pointers
to the existing data on the actual filesystem, and any new data need not
be mentioned in the snapshot file.
So the snapshot can be taken very quickly... and requires a bit of
maintenance by the OS during its lifetime. But if you create it, dump
it and remove it. The OS need not worry about it for long.
SoftUpdates are required on the filesystem.
HTH.
--
Regards,
Eric
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
On Mon, Oct 31, 2005 at 09:40:16AM -0600, Eric Schuele wrote: How do snapshots work and how do they provide the consistency necessary for a dump? [...] SoftUpdates are required on the filesystem. This sounds beautiful. I am amazed. I knew of softupdates, but they were always a shady corner of my understanding of BSD. So live fs dumping is based on the great hackery of softupdates. Fine, but in this case... shouldn't the man page make a mention of it? It just says that -L is ignored in case of unmounted/ro mounted fs-s, or if there is no proper .snap dir. But it doesn't say that it will be ignored if softupdates is not turned on... Going a bit off: which OS-es provide this live snapshot dumping capability? FreeBSD? FreeBSD = 5.x ? *BSD ? Maybe something else? (AFAIK, softupdates is supported also in other members of the BSD family, yet the NetBSD dump manpage didn't have such a -L flag...) -- Csaba Henk My sense of humour is often too subtle to cope with getting smileyd. Please don't take it personal. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
Csaba Henk wrote: On Mon, Oct 31, 2005 at 09:40:16AM -0600, Eric Schuele wrote: How do snapshots work and how do they provide the consistency necessary for a dump? [...] SoftUpdates are required on the filesystem. This sounds beautiful. I am amazed. I knew of softupdates, but they were always a shady corner of my understanding of BSD. So live fs dumping is based on the great hackery of softupdates. Fine, but in this case... shouldn't the man page make a mention of it? The online manual mentions it in 16.13. Wouldn't hurt for it to be in the man page as well. It just says that -L is ignored in case of unmounted/ro mounted fs-s, or if there is no proper .snap dir. But it doesn't say that it will be ignored if softupdates is not turned on... Going a bit off: which OS-es provide this live snapshot dumping capability? FreeBSD? FreeBSD = 5.x ? *BSD ? Maybe something else? AFAIK FreeBSD 5.0+. Other *BSD as well, i believe... but someone else would have to answer that. (AFAIK, softupdates is supported also in other members of the BSD family, yet the NetBSD dump manpage didn't have such a -L flag...) -- Regards, Eric ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
On Mon, Oct 31, 2005 at 10:32:02AM -0600, Eric Schuele wrote: The online manual mentions it in 16.13. Wouldn't hurt for it to be in the man page as well. Oh, yeah, thanks. This makes things clear. I missed this somehow. AFAIK FreeBSD 5.0+. Other *BSD as well, i believe... but someone else would have to answer that. (AFAIK, softupdates is supported also in other members of the BSD family, yet the NetBSD dump manpage didn't have such a -L flag...) OK, as I understand now, softupdates might be available for other BSD-s, but snapshotting is not a free bonus which comes with softupdates, but a new innovation based on that... and is a true FBSD innovation. So I'd guess it's still a unique thing. -- Csaba Henk My sense of humour is often too subtle to cope with getting smileyd. Please don't take it personal. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
On Mon, 31 Oct 2005 18:59:21 +0100, Csaba Henk [EMAIL PROTECTED] wrote: On Mon, Oct 31, 2005 at 10:32:02AM -0600, Eric Schuele wrote: The online manual mentions it in 16.13. Wouldn't hurt for it to be in the man page as well. Oh, yeah, thanks. This makes things clear. I missed this somehow. AFAIK FreeBSD 5.0+. Other *BSD as well, i believe... but someone else would have to answer that. (AFAIK, softupdates is supported also in other members of the BSD family, yet the NetBSD dump manpage didn't have such a -L flag...) OK, as I understand now, softupdates might be available for other BSD-s, but snapshotting is not a free bonus which comes with softupdates, but a new innovation based on that... and is a true FBSD innovation. So I'd guess it's still a unique thing. Not quite: NetBSD also features softupdates and also supports snapshots (though I don't know how stable it is, as I've never tried it on my NetBSD system). The snapshot interface under NetBSD is different from that on FreeBSD: you can create a snapshot device that can be used to snapshot a file system. (You can then mount or dump the snapshot device to get a consistent image/backup of the filesystem being snapshotted.) The main difference appears to be you are not limited to snapshots residing on the same file system of which you are taking a snapshot, which could be handy for near-full active filesystems. See fss(4) and fssconfig(8) man pages under NetBSD for details. The other thing to note about FreeBSD snapshots that I don't think has been mentioned is that they are only supported on UFS2 filesystems, meaning they are unavailable under FreeBSD 4.x and earlier (or on older filesystems created by those older versions of FreeBSD). I've been using snapshots under FreeBSD 5 onwards, pretty much since the feature became available, and have only ever had a problem once (due to a race condition in the snapshot code, it was surmised), and that was with a nightly automated network backup of snapshots of all filesystems on a system with high disk I/O. I find it to be a really valuable feature. Cheers, Paul. -- e-mail: [EMAIL PROTECTED] Without music to decorate it, time is just a bunch of boring production deadlines or dates by which bills must be paid. --- Frank Vincent Zappa ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
In the last episode (Oct 31), Paul Mather said: The other thing to note about FreeBSD snapshots that I don't think has been mentioned is that they are only supported on UFS2 filesystems, meaning they are unavailable under FreeBSD 4.x and earlier (or on older filesystems created by those older versions of FreeBSD). Snapshots work just fine on UFS1 filesystems; you just need to be running 5.x or newer. -- Dan Nelson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
On Mon, 2005-10-31 at 13:53 -0600, Dan Nelson wrote: In the last episode (Oct 31), Paul Mather said: The other thing to note about FreeBSD snapshots that I don't think has been mentioned is that they are only supported on UFS2 filesystems, meaning they are unavailable under FreeBSD 4.x and earlier (or on older filesystems created by those older versions of FreeBSD). Snapshots work just fine on UFS1 filesystems; you just need to be running 5.x or newer. You are correct, sir; ignore me---I must have been thinking of native extended attributes support... (Unfortunately, this still means you can't take snapshots on 4.x and earlier, though, which is a shame because I find the feature very handy.) Cheers, Paul. -- e-mail: [EMAIL PROTECTED] Without music to decorate it, time is just a bunch of boring production deadlines or dates by which bills must be paid. --- Frank Vincent Zappa ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
Hi On 10/30/05, Csaba Henk [EMAIL PROTECTED] wrote: Hi! We plan to set up a backup server. While the basic backup procedure is clear -- use some archiving utility like dump, tar, or cpio and send data to the backup server via ssh or a network mount -- there are many details which are unclear for me. The two biggest problems are: 1) What parts are to be backed up? If I backup the whole system, the backup disk will get full soon. You could say it's not necessary, and that only the valueable data should be backed up (and not those parts which are easy to re-create by means of a new installation). But, say, someone breaks into the machince. How could I reliably find out the Achilles heel she used to get in if I don't have a complete system backup? Or if she has a backdoor left behind? Depends on what the risk you trying to mitigate with backup. Think of the problems and how you would get around them. There are file consistency utils you can run to see if root-kits etc have been installed. 2) How to schedule backups? I guess services should stop for the backup period as the backup could be unreliable or inconsistent if disk/file writes were going on during backup. It sounds as if I should drop to single user mode. Or is there a less drastic approach? And if I dropped to single user mode, I would lose control over the box for that period, as the box is accessed via ssh and sshd is also stopped in single user mode -- this sounds scary... With FreeBSD 5.x and later you can snapshop the filesystem then use a special 'dump' to backup that snapshot to the backup machine. have a look at amanda and bacula for how they handle this and do some research on different backup strategies and their risks and benfits wrt to Unix systems - theres lots out there.. -- Martin TYA. -- Csaba Henk My sense of humour is often too subtle to cope with getting smileyd. Please don't take it personal. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
Martin Hepworth wrote:
Hi
On 10/30/05, Csaba Henk [EMAIL PROTECTED] wrote:
Hi!
We plan to set up a backup server.
While the basic backup procedure is clear -- use some archiving utility
like dump, tar, or cpio and send data to the backup server via ssh or a
network mount -- there are many details which are unclear for me.
The two biggest problems are:
1) What parts are to be backed up? If I backup the whole system, the
backup disk will get full soon. You could say it's not necessary, and
that only the valueable data should be backed up (and not those parts
which are easy to re-create by means of a new installation). But, say,
someone breaks into the machince. How could I reliably find out the
Achilles heel she used to get in if I don't have a complete system
backup? Or if she has a backdoor left behind?
Depends on what the risk you trying to mitigate with backup. Think of the
problems and how you would get around them. There are file consistency utils
you can run to see if root-kits etc have been installed.
2) How to schedule backups? I guess services should stop for the backup
period as the backup could be unreliable or inconsistent if disk/file
writes were going on during backup. It sounds as if I should drop to
single user mode. Or is there a less drastic approach? And if I dropped
to single user mode, I would lose control over the box for that period,
as the box is accessed via ssh and sshd is also stopped in single user
mode -- this sounds scary...
With FreeBSD 5.x and later you can snapshop the filesystem then use a
special 'dump' to backup that snapshot to the backup machine.
dump(8) will create a snapshot of a live filesystem, dump the snapshot
and then remove the snapshot, if given the correct flags ('-L').
have a look at amanda and bacula for how they handle this and do some
research on different backup strategies and their risks and benfits wrt to
Unix systems - theres lots out there..
--
Martin
TYA.
--
Csaba Henk
My sense of humour is often too subtle to cope with getting smileyd.
Please don't take it personal.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to
[EMAIL PROTECTED]
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
--
Regards,
Eric
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
On Sun, 30 Oct 2005 14:49:02 +0100 Csaba Henk [EMAIL PROTECTED] wrote: We plan to set up a backup server. -- cut -- 1) What parts are to be backed up? If I backup the whole system, the backup disk will get full soon. incremental backups via a script called from cron sounds good, you might consider trying rdiff-backup http://www.nongnu.org/rdiff-backup/ http://www.nongnu.org/rdiff-backup/examples.html -- grtjs, albi gpg-key: lynx -dump http://scii.nl/~albi/gpg.asc | gpg --import ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: backup strategies
BackupPC http://www.google.com/search?hl=enlr=newwindow=1q=backuppc+freebsd On 10/31/05, albi [EMAIL PROTECTED] wrote: On Sun, 30 Oct 2005 14:49:02 +0100 Csaba Henk [EMAIL PROTECTED] wrote: We plan to set up a backup server. -- cut -- 1) What parts are to be backed up? If I backup the whole system, the backup disk will get full soon. incremental backups via a script called from cron sounds good, you might consider trying rdiff-backup http://www.nongnu.org/rdiff-backup/ http://www.nongnu.org/rdiff-backup/examples.html -- grtjs, albi gpg-key: lynx -dump http://scii.nl/~albi/gpg.asc | gpg --import ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- Seek ye first the kingdom of God and all these things shall be added unto you. Winelfred G. Pasamba Adventist University of the Philippines Computer Science Department, AUP Online Information System ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
