Re: [BackupPC-users] Full vs. Incremental (was: Backup through slow line?)
What about with multiple full backups, say your full backup is 50gig and you do a full backup once a week, will mean each full backup will use 50gigs of space? Or does the pool do some other linking between the data contained in each full backup? On Thu, Sep 4, 2008 at 5:05 PM, Christian Völker [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yohoo! | I keep repeating this, so you might all be bored, but still: No, definetly not bored. For this it looks like it's too complicated :-\ | When you factor pooling into the equasion, this means that the difference | between rsync full and incremental backups is smaller than for tar/smb | backups, which is yet far smaller than for tape backups for instance. Or, the | other way around, for tape backups the difference is obvious, for tar/smb you | can still see it clearly, while for rsync backups you have to look through a | microscope. But it's still the same difference, and it's still there. | I like that explanation! :-) IMHO it should go to the Wiki. I would prefer to have a more detailled explanation for this. As I want to understand what's going on it's not helpfull to compare things instead of giving examples. So WHAT it the difference between an rsync full and incremental backup? What CAN be the microscopic differences? Let's say I have a running Linux server which I back up through rsync with default settings and I already have one successful full backup. There are now two parts to look at: 1. The backup process itself 2. The stored backup So what are the differences now between a full and incremental one? Greetings Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFIv4jB0XNIYlAXmzsRAqaWAKCxdKDyFTKxnmsabPhTZ43bmDLq8gCffvEg xUxCQEZWTW/kIR+n1Dx9BNc= =lCCj -END PGP SIGNATURE- - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full vs. Incremental (was: Backup through slow line?)
Hi, dan wrote on 2008-09-05 21:26:25 -0600 [Re: [BackupPC-users] Full vs. Incremental (was: Backup through slow line?)]: the real difference between an rsync full and incremental in backuppc are the 3 following 1) a full has rsync ignore matching mtimes and checksum all files while and incremental trusts matching mtimes 2)backuppc has a different keep/delete schedule for fulls vs incrementals. 3)an incremental backup in backuppc only holds changed files and and backup is filled by the previous full backup (make that backup of next lower level instead of full backup; also note that you are using the term filled contrary to its meaning within BackupPC) while a full backup has no dependancy on another backup. there are no other differences. ... except 4) a full backup is based on the previous backup of any level (including possibly a partial backup) and an incremental is based on the last (successful) backup of the next lower level. Most of this has been mentioned in this thread before. 5) Failed full backups are saved as partials, failed incrementals aren't. does that explain it?? That depends on how you define explain. It doesn't detail the consequences, but we've had that, and reducing it to the technical differences does add another point of view. Regards, Holger - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full vs. Incremental (was: Backup through slow line?)
Hi,
Stephen Vaughan wrote on 2008-09-06 19:11:08 +1000 [Re: [BackupPC-users] Full
vs. Incremental (was: Backup through slow line?)]:
What about with multiple full backups, say your full backup is 50gig and you
do a full backup once a week, will mean each full backup will use 50gigs of
space? Or does the pool do some other linking between the data contained in
each full backup?
pooling also works for full backups. Why shouldn't it?
Remember: if you've got a file containing just the word foo (or - more
usefully - any other content) and store it in multiple locations, on different
machines, in different shares, multiply within a share, and take as many
backups of all of those machines and shares as you want, you will still only
have one file in the pool to which all of those occurrences are linked -
providing maximum link count permits. Only when you reach $Conf{HardLinkMax} -
31999 by default - occurrences, will a copy of the pool file be created for
the next $Conf{HardLinkMax} occurrences. For reference, 31999 weeks is over
613 years, so if you have no duplication within your data or across servers,
you probably won't be around to see that happen ;-).
What will *not* be shared between full (or incremental) backups is the
directory structure for the pc/backupnum trees. The directory tree as such
is identical in a full and an incremental backup, but a full backup contains
directory entries for all existing files, an incremental only for changed
files. A directory with 1000 entries occupies more disk space than an empty
directory. This means that a full backup will in fact use more space than an
incremental - typically only slightly more, but if you have very short files
with very long names, that might be different. Compare the output of 'df' and
'df -i' (on the systems you are backing up, not the pool file system!) or the
BackupPC stats to get a rough idea of your average file size before or after
compression, considering or not considering pooling (actually, you're
interested in the size of compressed new files). Depending on file system and
mount options, storage allocation for files and directories usually happens
in multiples of file system blocks (usually something like 1KB or 4KB), which
is also true for the pool, so even an empty directory in an incremental may
take up 4KB of disk space (because it actually contains entries for '.' and
'..' - empty *files* should *not* take up disk space except for the directory
entry).
To summarize, it depends on your setup, and you probably don't need to worry
about it :-).
Regards,
Holger
-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full vs. Incremental
Stephen Vaughan wrote: What about with multiple full backups, say your full backup is 50gig and you do a full backup once a week, will mean each full backup will use 50gigs of space? Or does the pool do some other linking between the data contained in each full backup? All files with exactly the same content are pooled together regardless of the source of the other matches. The actual size requirement for a new full is somewhat unpredictable. If you have large files that change even slightly (growing logs, unix mailboxes, database dumps, etc.), you'll get a new copy in the pool where unchanged files will be linked. The pooling check happens after the transfer, though. The comparison for changes to transfer with the rsync method works against the filenames in the prior full or full+incremental if you use incremental levels. A growing logfile, for example, would have rsync send only the difference while the backuppc side reconstructs a full copy merged with the old content, then this new file would be added to the pool. -- Les Mikesell [EMAIL PROTECTED] - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full vs. Incremental (was: Backup through slow line?)
the real difference between an rsync full and incremental in backuppc are the 3 following 1) a full has rsync ignore matching mtimes and checksum all files while and incremental trusts matching mtimes 2)backuppc has a different keep/delete schedule for fulls vs incrementals. 3)an incremental backup in backuppc only holds changed files and and backup is filled by the previous full backup while a full backup has no dependancy on another backup. there are no other differences. does that explain it?? On Thu, Sep 4, 2008 at 1:05 AM, Christian Völker [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yohoo! | I keep repeating this, so you might all be bored, but still: No, definetly not bored. For this it looks like it's too complicated :-\ | When you factor pooling into the equasion, this means that the difference | between rsync full and incremental backups is smaller than for tar/smb | backups, which is yet far smaller than for tape backups for instance. Or, the | other way around, for tape backups the difference is obvious, for tar/smb you | can still see it clearly, while for rsync backups you have to look through a | microscope. But it's still the same difference, and it's still there. | I like that explanation! :-) IMHO it should go to the Wiki. I would prefer to have a more detailled explanation for this. As I want to understand what's going on it's not helpfull to compare things instead of giving examples. So WHAT it the difference between an rsync full and incremental backup? What CAN be the microscopic differences? Let's say I have a running Linux server which I back up through rsync with default settings and I already have one successful full backup. There are now two parts to look at: 1. The backup process itself 2. The stored backup So what are the differences now between a full and incremental one? Greetings Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFIv4jB0XNIYlAXmzsRAqaWAKCxdKDyFTKxnmsabPhTZ43bmDLq8gCffvEg xUxCQEZWTW/kIR+n1Dx9BNc= =lCCj -END PGP SIGNATURE- - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Full vs. Incremental (was: Backup through slow line?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yohoo! | I keep repeating this, so you might all be bored, but still: No, definetly not bored. For this it looks like it's too complicated :-\ | When you factor pooling into the equasion, this means that the difference | between rsync full and incremental backups is smaller than for tar/smb | backups, which is yet far smaller than for tape backups for instance. Or, the | other way around, for tape backups the difference is obvious, for tar/smb you | can still see it clearly, while for rsync backups you have to look through a | microscope. But it's still the same difference, and it's still there. | I like that explanation! :-) IMHO it should go to the Wiki. I would prefer to have a more detailled explanation for this. As I want to understand what's going on it's not helpfull to compare things instead of giving examples. So WHAT it the difference between an rsync full and incremental backup? What CAN be the microscopic differences? Let's say I have a running Linux server which I back up through rsync with default settings and I already have one successful full backup. There are now two parts to look at: 1. The backup process itself 2. The stored backup So what are the differences now between a full and incremental one? Greetings Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFIv4jB0XNIYlAXmzsRAqaWAKCxdKDyFTKxnmsabPhTZ43bmDLq8gCffvEg xUxCQEZWTW/kIR+n1Dx9BNc= =lCCj -END PGP SIGNATURE- - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full vs. Incremental
Christian, Here is a good post on Full vs. Incremental from a year ago. In particular, check out the two threads that Holger links to. http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg06101.html -Rob Christian Völker wrote: Yohoo! | I keep repeating this, so you might all be bored, but still: No, definetly not bored. For this it looks like it's too complicated :-\ | When you factor pooling into the equasion, this means that the difference | between rsync full and incremental backups is smaller than for tar/smb | backups, which is yet far smaller than for tape backups for instance. Or, the | other way around, for tape backups the difference is obvious, for tar/smb you | can still see it clearly, while for rsync backups you have to look through a | microscope. But it's still the same difference, and it's still there. | I like that explanation! :-) IMHO it should go to the Wiki. I would prefer to have a more detailled explanation for this. As I want to understand what's going on it's not helpfull to compare things instead of giving examples. So WHAT it the difference between an rsync full and incremental backup? What CAN be the microscopic differences? Let's say I have a running Linux server which I back up through rsync with default settings and I already have one successful full backup. There are now two parts to look at: 1. The backup process itself 2. The stored backup So what are the differences now between a full and incremental one? Greetings Christian - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the addressee, any disclosure, reproduction, copying, distribution, or other dissemination or use of this transmission in error please notify the sender immediately and then delete this e-mail. E-mail transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard copy version. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Full VS Incremental
I need to call on my trusty BackupPC server to do a near bare-metal recovery of a server. I've got the OS loaded well enough to interface with BackupPC. What I need to know now is this: If I pick the most recent incremental install, will the data be filled to include all the files from the previous full backup, or do I need to do the full, then each incremental since? -- C-ya, Mark Artificial Intelligence usually beats real stupidity. --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnkkid=103432bid=230486dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full VS Incremental
I need to call on my trusty BackupPC server to do a near bare-metal recovery of a server. I've got the OS loaded well enough to interface with BackupPC. What I need to know now is this: If I pick the most recent incremental install, will the data be filled to include all the files from the previous full backup, or do I need to do the full, then each incremental since? it will be filled. no need for two restores. paul =- paul fox, [EMAIL PROTECTED] (arlington, ma, where it's 35.8 degrees) --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnkkid=103432bid=230486dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Full VS Incremental
Thanks, that's what I was hoping to find out. C-ya, Mark Duddley DoRight's Horses name was Horse. Paul Fox wrote: I need to call on my trusty BackupPC server to do a near bare-metal recovery of a server. I've got the OS loaded well enough to interface with BackupPC. What I need to know now is this: If I pick the most recent incremental install, will the data be filled to include all the files from the previous full backup, or do I need to do the full, then each incremental since? it will be filled. no need for two restores. paul =- paul fox, [EMAIL PROTECTED] (arlington, ma, where it's 35.8 degrees) --- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnkkid=103432bid=230486dat=121642 ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
