Re: [BackupPC-users] Pooling doesn't work
Hi, thanks for replying and explaining. This made some things clearer for me. Holger Parplies schrieb: > Hi again, > > I was just going to add that your subject is incorrect, but I see that you > seem to be having a second issue. Sorry for replying a bit hastily, but your > wording does make it rather easy to draw incorrect conclusions (or rather miss > essential points). I'm sorry. My English isn't as that good as it should be. > > Patric Hafner wrote on 2009-10-08 16:29:54 +0200 [[BackupPC-users] Pooling > doesn't work]: >> [...] >> My problem is, that during an incremental backup nearly every file is >> marked as "create", so nearly every file will be downloaded again. >> About 20% are marked as "pool". > > Note that "pool" also means "downloaded again". Not downloaded due to rsync > savings is "same" (in a full backup) or the file simply not appearing in the > log (in an incremental backup). Okay, I understand. > >> But those files marked as "create" haven't changed since the last run, >> timestamps are still the same. For example the whole /etc directory will >> be downloaded every day. And I can surely say that nothing changed there. > > Timestamps are not the only indication of change. It is *possible* to modify a > file without changing the timestamp (e.g. resetting it after the change). But > that is probably not what is happening here. Okay, but why assumes rsync that one day after performing a full backup, that some files changed which a definitively not changed. I did the following to exclude changes: I setup a new linux client created some directories in /home/. After that I started a full backup. One day later I performed an incremental backup. The log says that the whole /etc and /home directory is set to create. Why this behavior? I guess pooling works, and my main problem is: Too many files are marked unexplainable as create. > > It would appear that pooling is only *partially* working (which is confusing > in itself). You couldn't have files marked "pool" if there was no pooling at > all. I would *guess* that you have probably incorrectly changed $TopDir after > having made some backups. You probably have tons of "link failed ..." errors > in your log files. New files are not added to the pool, so only files already > present from your first backups would be found there, though linking would not > work for them, either. > > Again, for anything more than educated guesses about what might be going > wrong, we need details about your setup. > > - What version, what installation method, what OS, which paths, which > filesystem(s), how partitioned? BackupPC 3.1.0 that comes with Debian Lenny. One root partition (ext3, /, normal SATA drive), one data partition (ext3, /data, external RAID 5 array). > - What did you change recently? Move $TopDir? How? See > > http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Change_archive_directory > for details on what you should or should not have done. I created a soft-link during installation: /var/lib/backuppc -> /data/backuppc/. No backup was made before creating this soft-link. > - Is there anything suspicious in your log files ($LogDir/LOG and > $TopDir/pc/hostname/XferLOG.NN.z)? Everything fine. No "link failed" errors. > >> This results an an extensive amount of traffic, which is unacceptable. > > Err, no. It results in an excessive amount of storage being used. Traffic is > independant of storage. If pooling was working correctly, you could still have > the same amount of traffic, but everything should be marked "pool" and stored > only once. Conversely, if rsync transfers were working correctly, you would > save traffic, but that does not imply that pooling would work. True, for this > one host unchanged files would be re-used, but they would not be matched up > against independent copies of identical content (from the same host or > different hosts). Thanks for explaining. >> i'm running BackupPC 3.1.0 with Debian Lenny. BackupPC is responsible >> for about 5 Clients which are backupped over rsync/ssh. >> >> My problem is, that during an incremental backup nearly every file is >> marked as "create", so nearly every file will be downloaded again. >> About 20% are marked as "pool". >> But those files marked as "create" haven't changed since the last run, >> timestamps are still the same. For example the whole /etc directory will >> be downloaded every day. And I can surely say that nothing changed there. >> [...] >> Does anyone has an idea? This would be great. > > yes, yo
Re: [BackupPC-users] Pooling doesn't work
Hi again, I was just going to add that your subject is incorrect, but I see that you seem to be having a second issue. Sorry for replying a bit hastily, but your wording does make it rather easy to draw incorrect conclusions (or rather miss essential points). Patric Hafner wrote on 2009-10-08 16:29:54 +0200 [[BackupPC-users] Pooling doesn't work]: > [...] > My problem is, that during an incremental backup nearly every file is > marked as "create", so nearly every file will be downloaded again. > About 20% are marked as "pool". Note that "pool" also means "downloaded again". Not downloaded due to rsync savings is "same" (in a full backup) or the file simply not appearing in the log (in an incremental backup). > But those files marked as "create" haven't changed since the last run, > timestamps are still the same. For example the whole /etc directory will > be downloaded every day. And I can surely say that nothing changed there. Timestamps are not the only indication of change. It is *possible* to modify a file without changing the timestamp (e.g. resetting it after the change). But that is probably not what is happening here. It would appear that pooling is only *partially* working (which is confusing in itself). You couldn't have files marked "pool" if there was no pooling at all. I would *guess* that you have probably incorrectly changed $TopDir after having made some backups. You probably have tons of "link failed ..." errors in your log files. New files are not added to the pool, so only files already present from your first backups would be found there, though linking would not work for them, either. Again, for anything more than educated guesses about what might be going wrong, we need details about your setup. - What version, what installation method, what OS, which paths, which filesystem(s), how partitioned? - What did you change recently? Move $TopDir? How? See http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Change_archive_directory for details on what you should or should not have done. - Is there anything suspicious in your log files ($LogDir/LOG and $TopDir/pc/hostname/XferLOG.NN.z)? > This results an an extensive amount of traffic, which is unacceptable. Err, no. It results in an excessive amount of storage being used. Traffic is independant of storage. If pooling was working correctly, you could still have the same amount of traffic, but everything should be marked "pool" and stored only once. Conversely, if rsync transfers were working correctly, you would save traffic, but that does not imply that pooling would work. True, for this one host unchanged files would be re-used, but they would not be matched up against independent copies of identical content (from the same host or different hosts). You need to fix both issues, and they are independant of each other. Regards, Holger -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Pooling doesn't work
Hi, Patric Hafner wrote on 2009-10-08 16:29:54 +0200 [[BackupPC-users] Pooling doesn't work]: > i'm running BackupPC 3.1.0 with Debian Lenny. BackupPC is responsible > for about 5 Clients which are backupped over rsync/ssh. > > My problem is, that during an incremental backup nearly every file is > marked as "create", so nearly every file will be downloaded again. > About 20% are marked as "pool". > But those files marked as "create" haven't changed since the last run, > timestamps are still the same. For example the whole /etc directory will > be downloaded every day. And I can surely say that nothing changed there. > [...] > Does anyone has an idea? This would be great. yes, you are probably incorrectly using incremental backups, but since you don't say anything about your configuration, we can only guess. Level 1 incremental backups download everything that has changed since the last full backup. Presuming your last full was long ago, or you have modified your configuration since then (e.g. changed from a test backup of, say, /lib, to a full backup of all of your root file system), you will be downloading everything changed or added since the last full backup with every incremental. Run a full backup and see if the following incrementals behave better. If so, send us some details about your configuration (esp. full and incremental backup scheduling settings) to let us help you adjust your schedule. In short: you *need* regular full backups. Regards, Holger -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Pooling doesn't work
Hello everybody, i'm running BackupPC 3.1.0 with Debian Lenny. BackupPC is responsible for about 5 Clients which are backupped over rsync/ssh. My problem is, that during an incremental backup nearly every file is marked as "create", so nearly every file will be downloaded again. About 20% are marked as "pool". But those files marked as "create" haven't changed since the last run, timestamps are still the same. For example the whole /etc directory will be downloaded every day. And I can surely say that nothing changed there. This results an an extensive amount of traffic, which is unacceptable. For example: My old, handwritten rsync-based solution found around 50MB of changed files, but BackupPC 3000MB. An example File Size/Count Reuse Summary Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 35 incr996 3560.5 1.69710 178.9 402 3381.7 Does anyone has an idea? This would be great. Big thanks for your help. Best regards, Patric -- Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
