Re: [BackupPC-users] Disk space used far higher than reported pool size
Craig O'Brien cobr...@fishman.com wrote on 10/29/2013 08:21:11 PM: I'm not sure how I can go about determining if a particular backup is using the pool or just storing the files in the PC folder. What's the best way to check if a given backup set is represented in the pool or not? Would knowing the size of all the pc folders help narrow it down? Nope. I'm not sure if this is the best way to check the hard linking, but here's a test I thought might be helpful. I did this command to see if a common file in these backups are pointing to the same inodes. You want to look for files in the pc directory that have only one hardlink. These files are not in the pool and need to either be deleted or connected to files in the pool. Jeff Kosowsky gave you a command to list files with only one hardlink. Adam Goryachev gave you a good du command to find out how much space is being taken by the pc directory *after* counting the files in the pool separately. That command will take a long time to run, but it will give you a pretty clear idea of where the space is being consumed. I would not stake my life on this, but I would bet a pretty substantial amount of money: you did something to break the pooling. Most likely by copying backups around. This undid the hardlinks and left you with individual copies of the files. Or punt completely: rebuild the BackupPC server and start over. You could do almost as well by confirming that your latest backups *are* hardlinking properly and then deleting all of the old backups except maybe a copy or two. I would not delete the copies by hand, but rather change the configuration to only keep 1 full and 1 incremental. It might be a good idea to make some archives to make sure you have a good copy somewhere. In any case, once BackupPC has deleted all of the old backups, go into your pc directories and make sure that there is indeed only the backups listed in the GUI in the folder structure. Then, change the incremental and full keep counts back to what they should be and allow it to rebuild. The only other thing that I can think of is that you did something wrong with archiving and accidentally archived data somewhere within the BackupPC tree. In my case, I archive to a removable hard drive and sometimes the drive is not mounted when the archive runs. The archives are then put on the backup drive (because that's where the removable drive is mounted). That's tricky because you can't see the files when the drive *is* mounted (which is the vast majority of the time). I have to unmount the drive and then I can see terabytes of archive data that should have been written to a removable drive. I don't know if that might be part of your problem. But it's the only other thing I can think of. Tim Massey Out of the Box Solutions, Inc. Creative IT Solutions Made Simple! http://www.OutOfTheBoxSolutions.com tmas...@obscorp.com 22108 Harper Ave. St. Clair Shores, MI 48080 Office: (800)750-4OBS (4627) Cell: (586)945-8796 -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
On 31/10/13 00:04, Timothy J Massey wrote: The only other thing that I can think of is that you did something wrong with archiving and accidentally archived data somewhere within the BackupPC tree. In my case, I archive to a removable hard drive and sometimes the drive is not mounted when the archive runs. The archives are then put on the backup drive (because that's where the removable drive is mounted). That's tricky because you can't see the files when the drive *is* mounted (which is the vast majority of the time). I have to unmount the drive and then I can see terabytes of archive data that should have been written to a removable drive. I don't know if that might be part of your problem. But it's the only other thing I can think of. Not really relevant to this thread, but I have in the past added a empty file to each of the removable drives, then test if the file exists before creating the archives. If the drive isn't mounted, the file won't exist. Thus preventing that issue. I'm sure you've probably considered this previously already :) Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
Adam Goryachev mailingli...@websitemanagers.com.au wrote on 10/30/2013 09:18:59 AM: Not really relevant to this thread, but I have in the past added a empty file to each of the removable drives, then test if the file exists before creating the archives. If the drive isn't mounted, the file won't exist. Thus preventing that issue. I'm sure you've probably considered this previously already :) Thank you for the suggestion! My thought was to parse the output of df /path/to/drive and confirm that it was mounted correctly. (I already do that in the scripts I use to mount the removable drive and re-format it.) If it happened more than once or twice a year, I probably would! :) Your way certainly works, too. Because it's the root of the removable drive, I could simply look for lost+found, too! :) Tim Massey Out of the Box Solutions, Inc. Creative IT Solutions Made Simple! http://www.OutOfTheBoxSolutions.com tmas...@obscorp.com 22108 Harper Ave. St. Clair Shores, MI 48080 Office: (800)750-4OBS (4627) Cell: (586)945-8796 -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
On 2013-10-29 17:53, Craig O'Brien wrote: On the General Server Information page, it says Pool is 2922.42GB comprising 6061942 files and 4369 directories, but our pool file system which contains nothing but backuppc and is 11 TB in size is 100% full. Did you forget to exclude the path to TopDir (usually /var/lib/backuppc) from the backup of the BackupPC server itself? I've seen that before. Heck, I've DONE that before. Regards, Tyler -- The intellectual is constantly betrayed by his vanity. Godlike he blandly assumes that he can express everything in words; whereas the things one loves, lives, and dies for are not, in the last analysis completely expressible in words. -- Anne Morrow Lindbergh -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
I'm fairly sure:
du -sm /backup/pool /backup/cpool /backup/pc/*
It should count all the data under pool and cpool, and there should be
minimal space used for the pc folders (because it counts the space for the
first time the inode is seen)
I'm trying that now. I'll report back when it finishes.
To delete a host, hit the Delete button. For Add, Delete, and
configuration copy, changes don't take effect until you select Save. None
of the deleted host's backups will be removed, so if you accidently delete
a host, simply re-add it.
* To completely remove a host's backups, you need to manually remove the
files below /var/lib/backuppc/pc/HOST *
This is how I've done it when I've removed a host. I would delete the
/backup/pc/host directory and remove the entry from /etc/BackupPC/hosts
file.
I would not stake my life on this, but I would bet a pretty substantial
amount of money: you did something to break the pooling. Most likely by
copying backups around. This undid the hardlinks and left you with
individual copies of the files.
I don't doubt the pooling is probably broken but I haven't moved any
backups around. For what it's worth I before I switched all the pc's to use
rsync instead of smb a couple months ago my pool file system was sitting at
30%. I don't know if that's relevant but it does seem odd that my problems
seem to have started with that.
Or punt completely: rebuild the BackupPC server and start over.
You could do almost as well by confirming that your latest backups *are*
hardlinking properly and then deleting all of the old backups except maybe
a copy or two.
I would not delete the copies by hand, but rather change the
configuration to only keep 1 full and 1 incremental.
It might be a good idea to make some archives to make sure you have a
good copy somewhere. In any case, once BackupPC has deleted all of the old
backups,
go into your pc directories and make sure that there is indeed only the
backups listed in the GUI in the folder structure.
Then, change the incremental and full keep counts back to what they
should be and allow it to rebuild.
I'll probably have to do that. At this point I'm just trying to add to the
knowledge base and figure out how it went wrong so it doesn't just happen
again.
My thought was to parse the output of df /path/to/drive and confirm
that it was mounted correctly.
Just in case it helps at all:
bash-4.1$ df -h /backup
FilesystemSize Used Avail Use% Mounted on
/dev/sda1 11T 9.7T 28G 100% /backup
bash-4.1$
Did you forget to exclude the path to TopDir (usually /var/lib/backuppc)
from the backup of the BackupPC server itself? I've seen that before.
Heck,
I've DONE that before.
I don't have the server backing itself up.
Here's my config file (with #comment lines removed) just in case that helps
at all.
-
$Conf{ServerHost} = 'localhost';
$Conf{ServerPort} = -1;
$Conf{ServerMesgSecret} = '';
$Conf{MyPath} = '/bin';
$Conf{UmaskMode} = 23;
$Conf{WakeupSchedule} = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23];
$Conf{MaxBackups} = 4;
$Conf{MaxUserBackups} = 4;
$Conf{MaxPendingCmds} = 15;
$Conf{CmdQueueNice} = 10;
$Conf{MaxBackupPCNightlyJobs} = 4;
$Conf{BackupPCNightlyPeriod} = 1;
$Conf{MaxOldLogFiles} = 14;
$Conf{DfPath} = '/bin/df';
$Conf{DfCmd} = '$dfPath $topDir';
$Conf{SplitPath} = '/usr/bin/split';
$Conf{ParPath} = undef;
$Conf{CatPath} = '/bin/cat';
$Conf{GzipPath} = '/bin/gzip';
$Conf{Bzip2Path} = '/usr/bin/bzip2';
$Conf{DfMaxUsagePct} = 95;
$Conf{TrashCleanSleepSec} = 300;
$Conf{DHCPAddressRanges} = [];
$Conf{BackupPCUser} = 'backuppc';
$Conf{TopDir} = '/var/lib/BackupPC/';
$Conf{ConfDir} = '/etc/BackupPC/';
$Conf{LogDir} = '/var/log/BackupPC';
$Conf{InstallDir} = '/usr/share/BackupPC';
$Conf{CgiDir} = '/usr/share/BackupPC/sbin/';
$Conf{BackupPCUserVerify} = '1';
$Conf{HardLinkMax} = 31999;
$Conf{PerlModuleLoad} = undef;
$Conf{ServerInitdPath} = undef;
$Conf{ServerInitdStartCmd} = '';
$Conf{FullPeriod} = 90;
$Conf{IncrPeriod} = 7;
$Conf{FullKeepCnt} = [ 4 ];
$Conf{FullKeepCntMin} = 1;
$Conf{FullAgeMax} = 3000;
$Conf{IncrKeepCnt} = 13;
$Conf{IncrKeepCntMin} = 1;
$Conf{IncrAgeMax} = 45;
$Conf{IncrLevels} = [ 1 ];
$Conf{BackupsDisable} = 0;
$Conf{PartialAgeMax} = 3;
$Conf{IncrFill} = '0';
$Conf{RestoreInfoKeepCnt} = 5;
$Conf{ArchiveInfoKeepCnt} = 5;
$Conf{BackupFilesOnly} = {};
$Conf{BackupFilesExclude} = {};
$Conf{BlackoutBadPingLimit} = 3;
$Conf{BlackoutGoodCnt} = 7;
$Conf{BlackoutPeriods} = [
{
'hourEnd' = '19.5',
'weekDays' = [
1,
2,
3,
4,
5
],
'hourBegin' = 7
}
];
$Conf{BackupZeroFilesIsFatal} = '1';
$Conf{XferMethod} = 'rsyncd';
$Conf{XferLogLevel} = 1;
$Conf{ClientCharset} = '';
$Conf{ClientCharsetLegacy} = 'iso-8859-1';
$Conf{SmbShareName} = [
'C$'
];
$Conf{SmbShareUserName} = '';
$Conf{SmbSharePasswd} = '';
$Conf{SmbClientPath} = '/usr/bin/smbclient';
Re: [BackupPC-users] Disk space used far higher than reported pool size
Hi, I'll reply here, because I think the issue is visible here. Craig O'Brien wrote on 2013-10-29 15:30:46 -0400 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: The topdir is /var/lib/BackupPC which is a link to /backup I believe that may be your problem. I'm not sure why, but I vaguely recall reports of linking problems related to softlinking the pool directory (though I'm not absolutely sure about the exact circumstances, hence vaguely recall ;-), though *in theory* it should work. I'd always prefer mounting the partition to /var/lib/backuppc instead of the softlink if you don't have a *very* good reason for requiring the pool to be mounted elsewhere. Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find large amounts of BackupPC_link got error -4 when calling MakeFileLink messages in your log files. Also note that at least for *rsync backups* files will be hardlinked to identical copies in previous backups even if pooling isn't working. Hope that helps, and wish I'd found the time yesterday, Regards, Holger -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
On Wed, Oct 30, 2013 at 10:12 AM, Holger Parplies wb...@parplies.de wrote: Craig O'Brien wrote on 2013-10-29 15:30:46 -0400 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: The topdir is /var/lib/BackupPC which is a link to /backup I believe that may be your problem. I'm not sure why, but I vaguely recall reports of linking problems related to softlinking the pool directory (though I'm not absolutely sure about the exact circumstances, hence vaguely recall ;-), though *in theory* it should work. I'd always prefer mounting the partition to /var/lib/backuppc instead of the softlink if you don't have a *very* good reason for requiring the pool to be mounted elsewhere. Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find large amounts of BackupPC_link got error -4 when calling MakeFileLink messages in your log files. If you can find those errors in the logs for files that still exist you might try doing the same link with the ln command to see if it gives a better diagnostic for why it fails. It could be something quirky like a small limit to the number of hardlinks allowed in your target filesystem, or something you are missing about permissions/ownership (nfsv4 can be weird). -- Les Mikesell lesmikes...@gmail.com -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
Just to add two things ... Holger Parplies wrote on 2013-10-30 16:12:02 +0100 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: [...] Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find large amounts of BackupPC_link got error -4 when calling MakeFileLink messages in your log files. That would be the main server log file ($topDir/log/LOG and the rotated older copies), I guess. Also note that at least for *rsync backups* files will be hardlinked to identical copies in previous backups even if pooling isn't working. Since you have just written that you are, in fact, using rsync, I should add that this will make recovery more difficult, since unchanged files will continue to be just linked to the version in the reference backup. This file should normally already be linked to the pool, and BackupPC makes no further effort to check and fix that. This means that if you delete all but a few recent backups after fixing the root cause of the problem, only newly changed files will be added to the pool, while unchanged files will remain outside the pool. This may or may not be a problem for you. If it is, you will need to fix pooling for (some) existing backups, which is Hard(tm) (i.e. costly in terms of CPU power). Jeffrey, I think we need a script to check pooling? My (still unfinished) BackupPC_copyPool can generate a (huge) list of files, which can be sort(1)ed by inode number. Parsing that should easily reveal anything not correctly linked in an acceptable time frame (of course *generating* the list takes one traversal of all pool and pc directories, but the rest would be fast enough). Does that help, or have you already got something more suited? Are you interested or should I be? ;-) Regards, Holger -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
On Wed, Oct 30, 2013 at 10:48 AM, Holger Parplies wb...@parplies.de wrote: Also note that at least for *rsync backups* files will be hardlinked to identical copies in previous backups even if pooling isn't working. Since you have just written that you are, in fact, using rsync, I should add that this will make recovery more difficult, since unchanged files will continue to be just linked to the version in the reference backup. This file should normally already be linked to the pool, and BackupPC makes no further effort to check and fix that. This means that if you delete all but a few recent backups after fixing the root cause of the problem, only newly changed files will be added to the pool, while unchanged files will remain outside the pool. This may or may not be a problem for you. If it is, you will need to fix pooling for (some) existing backups, which is Hard(tm) (i.e. costly in terms of CPU power). If you can free up some space to work, you could try renaming the pc/host directories one at a time to hold a copy that you could access in an emergency and let it start over with a fresh full run where all files would be new and linked to the pool. Once you have sufficient history in the new tree, you can delete the old one that you renamed. This should eventually clean things up as long as you don't continue to have link errors. Alternatively, if you don't need more than the last backup for one or more targets, you could archive it with the archive host setup or running Backuppc_tarCreate on to some other media, delete the host and add it back to get a clean start. -- Les Mikesell lesmikes...@gmail.com -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
Hi, Les Mikesell wrote on 2013-10-30 11:28:26 -0500 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: On Wed, Oct 30, 2013 at 10:48 AM, Holger Parplies wb...@parplies.de wrote: Also note that at least for *rsync backups* files will be hardlinked to identical copies in previous backups even if pooling isn't working. [...] This means that if you delete all but a few recent backups after fixing the root cause of the problem, only newly changed files will be added to the pool, while unchanged files will remain outside the pool. This may or may not be a problem for you. If it is, you will need to fix pooling for (some) existing backups, which is Hard(tm) (i.e. costly in terms of CPU power). I intentionally did not go into detail, because we have not yet confirmed that this is indeed the problem, and if so, what the requirements of the OP are. If you can free up some space to work, you could try renaming the pc/host directories one at a time to hold a copy that you could access in an emergency and let it start over with a fresh full run where all files would be new and linked to the pool. Correct, but with an already almost full pool file system you *will* run into problems, because even unchanged files will now require a new copy. Some space might be a bit of an understatement :). [...] This should eventually clean things up as long as you don't continue to have link errors. Yes, as it's basically an extension of start off fresh with the addition of keep old history around in parallel. The notable thing is that you need to *make sure* you have eliminated the problem for there to be any point in starting over. Aside from that, I would think it might be worth the effort of determining whether all hosts are affected or not (though I can't really see why there should be a difference between hosts). If some aren't, you could at least keep their history. Regards, Holger -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd full backup
Hi, Adam Goryachev wrote on 2013-10-29 15:29:42 +1100 [Re: [BackupPC-users] rsyncd full backup]: On 29/10/13 15:14, Sharuzzaman Ahmat Raslan wrote: [...] On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan sharuzza...@gmail.com mailto:sharuzza...@gmail.com wrote: [...] Initially, the backup transport is SMB, but recently, I noticed a lot of machine backup (full and incremental) is not able to complete in 8 hours, due to large number of file, and big file size. Last week, I installed DeltaCopy (rsycnd server for Windows) on one machine, and change the backup transport to rysncd. The backup runs well. But today, I noticed, when BackupPC is running a full backup on the machine that have rsyncd, it still takes 8 hours to do full backup. [...] Rsync will only transfer the changed data, but in full runs the contents of the files are read at both ends and compared with block checksums, so it takes some time. [...] In essence, if I enable |--checksum-seed=32761 | then the rsync full backup will be faster? Yes, the third full backup after you enable that option will be faster *IF* the slow speed is due to the backup server needing to decompress the file and check the content. let me stress that again: don't expect a speedup on the *first* full backup after you enable that option. In my limited opinion (I haven't compared speeds because I don't have any issues with slow backups), the *second* full backup should be faster, as you have pre-existing full backups, i.e. the next full can add the checksums. In any case, the *third* full backup should hopefully be faster :-). In the case that your backup client has really slow disk, then there is nothing you can do, except maybe modify backuppc for full backups to not send the ignore-times option to rsync (ie, every backup is an incremental). Or, of course, upgrade the client to improve performance. Actually, it is worth noting that aside from a possible speed improvement the switch from smb to rsync(d) gives you far more precise *incremental* backups, so it might be an option to increase FullPeriod. This may transfer more data (because the delta is always relative to the reference backup - normally the previous full backup - and not to the previous incremental backup), but you can always explore the IncrLevels setting. So, while you might not speed up the full runs, you might get away with doing them less often. I would not recommend patching the ignore-times option away altogether. But Adams point is correct: you need to find out where the problem is, before you can fix it. While you might be able to find the problem by trying out fixes, that might not be the most efficient way :-). Regards, Holger -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
On 31/10/13 07:51, Holger Parplies wrote: Yes, as it's basically an extension of start off fresh with the addition of keep old history around in parallel. The notable thing is that you need to *make sure* you have eliminated the problem for there to be any point in starting over. Aside from that, I would think it might be worth the effort of determining whether all hosts are affected or not (though I can't really see why there should be a difference between hosts). If some aren't, you could at least keep their history. I suspect at least some hosts OR some backups are correct, or else OP wouldn't have anything in the pool. If you find the problem affects all hosts (the du command discussed previously will tell you that), then you might want to look at one individual host like this: du -sm /backup/pool /backup/cpool /backup/pc/host1/* This should be a *lot* quicker than the previous du command, and also should show minimal disk usage for each backup for host1. It is quicker because you are only looking at the set of files for the pool, plus one host. PS, at this stage, you may want to look at the recent thread regarding disk caches, and caching directory entries instead of file contents. It might help with all the directory based searches you are doing to find the problem. Long term you may (or not) want to keep the settings. Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
Holger Parplies wrote at about 16:48:11 +0100 on Wednesday, October 30, 2013: Jeffrey, I think we need a script to check pooling? My (still unfinished) BackupPC_copyPool can generate a (huge) list of files, which can be sort(1)ed by inode number. Parsing that should easily reveal anything not correctly linked in an acceptable time frame (of course *generating* the list takes one traversal of all pool and pc directories, but the rest would be fast enough). Does that help, or have you already got something more suited? Are you interested or should I be? ;-) I have code that will do this in 2 related ways: 1. Run my routine BackupPC_copyPcPool.pl with the --fixlinks|-f option which will fix missing (or invalid) pc-to-pool links on the fly as the routine crawls the pc tree (after creating an in-memory hash cache of the pool inodes) 3. Run my routine BackupPC_copyPcPool.pl to generate a list of the non-zero length, non-linked files (other than backupInfo which is the only non-zero length file in the pc tree that should not be linked to the pool) which lie in the pc tree. The routine always creates this list since these files would need to be transferred manually if not linked to the pool. Then pipe this list of unlinked files to my routine BackupPc_relinkPcFiles.pl to fix each of the non-linked files Note that BackupPC_copyPcPool works by caching the inode numbers of the pool/cpool entries in a hash which allows for quick lookup and checking of whether a pc tree file is linked to a valid pool file. Note that the above methods take care of the cases when pc tree files are: 1. Unlinked to anything else (nlinks =1) 2. Linked to other pc files (but not to the pool) Note that both methods properly either make a new link in the pool or delete the existing pc file and link it to a pre-existing pool entry depending on whether or not a pool entry already exists. -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd full backup
Hi Holger, Based on short session of troubleshooting, I believe the machine actually suffer from low I/O speed to the disk. Average read is about 3 MB/s, which I considered slow for a SATA disk in IDE emulation. I'm planning to suggest to the customer to have a RAID 1 setup to increase the I/O speed. I'm looking at possibilities to speed things up by not having to change the overall setup. Thank you for providing new insights to me regarding rsync. Glad to learn new things :) Thanks. On Thu, Oct 31, 2013 at 5:15 AM, Holger Parplies wb...@parplies.de wrote: Hi, Adam Goryachev wrote on 2013-10-29 15:29:42 +1100 [Re: [BackupPC-users] rsyncd full backup]: On 29/10/13 15:14, Sharuzzaman Ahmat Raslan wrote: [...] On Tue, Oct 29, 2013 at 11:33 AM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: On Mon, Oct 28, 2013 at 10:08 PM, Sharuzzaman Ahmat Raslan sharuzza...@gmail.com mailto:sharuzza...@gmail.com wrote: [...] Initially, the backup transport is SMB, but recently, I noticed a lot of machine backup (full and incremental) is not able to complete in 8 hours, due to large number of file, and big file size. Last week, I installed DeltaCopy (rsycnd server for Windows) on one machine, and change the backup transport to rysncd. The backup runs well. But today, I noticed, when BackupPC is running a full backup on the machine that have rsyncd, it still takes 8 hours to do full backup. [...] Rsync will only transfer the changed data, but in full runs the contents of the files are read at both ends and compared with block checksums, so it takes some time. [...] In essence, if I enable |--checksum-seed=32761 | then the rsync full backup will be faster? Yes, the third full backup after you enable that option will be faster *IF* the slow speed is due to the backup server needing to decompress the file and check the content. let me stress that again: don't expect a speedup on the *first* full backup after you enable that option. In my limited opinion (I haven't compared speeds because I don't have any issues with slow backups), the *second* full backup should be faster, as you have pre-existing full backups, i.e. the next full can add the checksums. In any case, the *third* full backup should hopefully be faster :-). In the case that your backup client has really slow disk, then there is nothing you can do, except maybe modify backuppc for full backups to not send the ignore-times option to rsync (ie, every backup is an incremental). Or, of course, upgrade the client to improve performance. Actually, it is worth noting that aside from a possible speed improvement the switch from smb to rsync(d) gives you far more precise *incremental* backups, so it might be an option to increase FullPeriod. This may transfer more data (because the delta is always relative to the reference backup - normally the previous full backup - and not to the previous incremental backup), but you can always explore the IncrLevels setting. So, while you might not speed up the full runs, you might get away with doing them less often. I would not recommend patching the ignore-times option away altogether. But Adams point is correct: you need to find out where the problem is, before you can fix it. While you might be able to find the problem by trying out fixes, that might not be the most efficient way :-). Regards, Holger -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Sharuzzaman Ahmat Raslan -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Disk space used far higher than reported pool size
Hi, Adam Goryachev wrote on 2013-10-31 09:04:48 +1100 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: On 31/10/13 07:51, Holger Parplies wrote: [...] Aside from that, I would think it might be worth the effort of determining whether all hosts are affected or not (though I can't really see why there should be a difference between hosts). If some aren't, you could at least keep their history. I suspect at least some hosts OR some backups are correct, or else OP wouldn't have anything in the pool. as I understand it, the backups from before the change from smb to rsyncd are linked into the pool. Since the change, some or all are not. Whether the change of XferMethod has anything to do with the problem or whether it coincidentally happened at about the same point in time remains to be seen. I still suspect the link to $topDir as cause, and BackupPC_link is independent of the XferMethod used (so a change in XferMethod shouldn't have any influence). [...] you might want to look at one individual host like this: du -sm /backup/pool /backup/cpool /backup/pc/host1/* This should be a *lot* quicker than the previous du command, and also should show minimal disk usage for each backup for host1. It is quicker because you are only looking at the set of files for the pool, plus one host. Just keep in mind that *incrementals* might be small even if not linked to pool files. Oh, and there is still another method that is *orders of magnitude* faster: look into the log file(s), or even at the *size* of the log files. If it happens every day, for each host, it shouldn't be hard to find. You can even write a Perl one-liner to show you which hosts it happens for (give me a sample log line and I will). If the log files show nothing, we're back to finding the problem, but I doubt that. You can't break pooling by copying, as was suggested. Yes, you get independent copies of files, and they might stay independent, but changed files should get pooled again, and your file system usage wouldn't continue growing in such a way as it seems to be. If pooling is currently broken, there's a reason for that, and there should be log messages indicating problems. PS, at this stage, you may want to look at the recent thread regarding disk caches, and caching directory entries instead of file contents. It might help with all the directory based searches you are doing to find the problem. Long term you may (or not) want to keep the settings. Yes, but remember that for a similarly sized pool it used up about 32 GB of 96 GB available memory. If you can do your investigation on a reasonably idle system (i.e. not running backups, without long pauses), you should get all the benefits of caching your amount of memory allows without any tuning. And even tuning won't let you hold 32 GB of file system metadata in 4 GB of memory :-). It all depends on file count and hardware memory configuration. Regards, Holger -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd full backup
On 31/10/13 13:06, Sharuzzaman Ahmat Raslan wrote: Hi Holger, Based on short session of troubleshooting, I believe the machine actually suffer from low I/O speed to the disk. Average read is about 3 MB/s, which I considered slow for a SATA disk in IDE emulation. Is that under load, or while idle? If it is under load, then it might be expected, remember throughput is very bad for HD when you have random load due to seek times. If it is idle and has that performance level, then there is something wrong. Even old IDE disks could do at least 30 to 50MB/s for large contiguous reads. I'm planning to suggest to the customer to have a RAID 1 setup to increase the I/O speed. I'm looking at possibilities to speed things up by not having to change the overall setup. While RAID1 will assist in reliability and is one strategy to reduce downtime/data loss (but it isn't a backup), it also is not going to improve performance. With RAID1 you still need to write to both disks, and while it is theoretically possible to balance reads across both disks, it likely won't do that well without a proper hardware raid controller. Personally, my suggestion would be to consider using a SSD, since you are using such an old drive, probably you don't need a lot of space, so a 120GB SSD might be suitable. An SSD will handle random IO significantly better than any one or two drive system, with much higher transfer rates as well (there is no penalty for seek times with SSD). Again, personally, I've used a couple of systems with 5 x 480GB Intel 520s SSD in RAID5, and they have been working really well (except they were difficult to actually get stock of them most of this year, and I hear they are now replaced by a new model). Regards, Adam -- Adam Goryachev Website Managers P: +61 2 8304 a...@websitemanagers.com.au F: +61 2 8304 0001 www.websitemanagers.com.au -- Adam Goryachev Website Managers www.websitemanagers.com.au -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd full backup
On 31/10/13 13:56, Sharuzzaman Ahmat Raslan wrote: Hi Adam, The low I/O is when the machine is under load. Thank you for suggesting to use SSD. I have been thinking about that as well, but currently, the storage of BackupPC is using a 1TB disk, with about 80% utilization. Changing to 1TB SSD might be a little bit overkill on the customer's budget :) Sure, 2 x 480GB SSD in linear RAID is still relatively expensive :) though it certainly is a huge performance improvement. BTW, FYI, I get 2.5GB/s read and 1.5GB/s write performance from my RAID5... Maybe I should look at bcache for Linux :) https://lwn.net/Articles/497024/ http://bcache.evilpiepirate.org/ I've seen that also, but I'm not sure it is a good (stable) solution for real use (at least, I'm not prepared to use that for a server yet, your tolerance might be different). In addition, it probably won't help the backup work load, since you need to read the entire disk, and the entire disk won't fit into the cache Regards, Adam -- Adam Goryachev Website Managers P: +61 2 8304 a...@websitemanagers.com.au F: +61 2 8304 0001 www.websitemanagers.com.au -- Adam Goryachev Website Managers www.websitemanagers.com.au -- Android is increasing in popularity, but the open development platform that developers love is also attractive to malware creators. Download this white paper to learn more about secure code signing practices that can help keep Android apps secure. http://pubads.g.doubleclick.net/gampad/clk?id=65839951iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
