[BackupPC-users] BackupPC hangs
Hi, I backup a huge QNAS Storage with rsync over ssh and it always hangs with 100% load. I tried to find the problem with lsof: BackupPC_ 1842 backuppc3w REG 253,3 2490 285576417 /var/lib/backuppc/pc/data/LOG.102016 BackupPC_ 1842 backuppc4w REG 253,3 4447002 285576424 /var/lib/backuppc/pc/data/XferLOG.z BackupPC_ 1842 backuppc5w REG 253,3 3130697 2324 /var/lib/backuppc/pc/data/NewFileList BackupPC_ 1842 backuppc6r REG 253,330938 1330743 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185 BackupPC_ 1842 backuppc7r REG 253,377320 419524 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0 BackupPC_ 1842 backuppc8u unix 0x880064e68dc0 0t0 997783 socket BackupPC_ 1842 backuppc9r REG 253,3 171056 419525 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1 BackupPC_ 1842 backuppc 10u unix 0x880064e69780 0t0 997109 socket BackupPC_ 1842 backuppc 11r REG 253,372265 419526 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2 BackupPC_ 1842 backuppc 12r REG 253,372310 419527 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_3 BackupPC_ 1842 backuppc 13r REG 253,372292 419538 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_4 BackupPC_ 1842 backuppc 14r REG 253,3 109717 419539 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_5 BackupPC_ 1842 backuppc 15r REG 253,3 150976 419540 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_6 BackupPC_ 1842 backuppc 16r REG 253,396457 419541 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_7 BackupPC_ 1842 backuppc 17r REG 253,396459 419542 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_8 BackupPC_ 1842 backuppc 18r REG 253,396699 419563 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_9 BackupPC_ 1842 backuppc 19r REG 253,397760 419564 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_10 BackupPC_ 1842 backuppc 20r REG 253,399200 419565 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_11 BackupPC_ 1842 backuppc 21r REG 253,3 101227 419566 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_12 BackupPC_ 1842 backuppc 22r REG 253,3 102926 419567 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_13 BackupPC_ 1842 backuppc 23r REG 253,3 103426 419568 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_14 BackupPC_ 1842 backuppc 24r REG 253,3 102631 419569 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_15 BackupPC_ 1842 backuppc 25r REG 253,3 101767 419570 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_16 BackupPC_ 1842 backuppc 26r REG 253,3 100944 419571 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_17 BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18 BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587 /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33 Nothing special in the logs. It counts the files up and up and then restarts with that. I should delete my pool and start new but I don't want to do that. :/ Any ideas? Thanks! Markus -- Markus Hirschmann, Dipl.-Inf. Univ. Veitshöchheimer Straße 1, 97080 Würzburg Tel: +49 931 8092864Mobil: +49 179 5408429 m...@durtro.de Steuernummer 257/229/10802 -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC hangs
Hi,
Markus Hirschmann wrote on 2016-10-10 17:54:45 +0200 [[BackupPC-users] BackupPC
hangs]:
>
> I backup a huge QNAS Storage with rsync over ssh and it always hangs
> with 100% load. I tried to find the problem with lsof:
>
> [...]
> BackupPC_ 1842 backuppc6r REG 253,330938 1330743
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_185
185 ... wow ...
> BackupPC_ 1842 backuppc7r REG 253,377320 419524
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0
> BackupPC_ 1842 backuppc9r REG 253,3 171056 419525
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_1
> BackupPC_ 1842 backuppc 11r REG 253,372265 419526
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_2
[and so on]
You've got a rather long hash chain here (185 candidates for a pool match).
For the current file in question, BackupPC needs to compare the contents with
all candidates (or at least those that haven't been ruled out yet ... see
the missing numbers further down:
> BackupPC_ 1842 backuppc 27r REG 253,3 100064 419572
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_18
> BackupPC_ 1842 backuppc 28r REG 253,3 113544 419587
> /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_33
). I'm guessing the file is rather large, so the comparison takes quite long.
Note also that BackupPC won't take all candidates into account at once, due
to the limit on concurrently open files, though I'm not sure what the strategy
is (it might mean the comparison needs to be repeated).
Hash collisions happen (mainly) for files with identical *length* and identical
first and last 128 KiB chunk within the first 1 MiB of data (if I remember
correctly; the details aren't important, so I won't check). Maybe you're
backing up a large database with changes just in the wrong places? Log files
would tend to grow, i.e. change length (and therefore hash), so they usually
won't cause trouble in form of hash chains.
Hint: 'ls -ali /var/lib/backuppc/cpool/d/6/d/d6dfb50e7648054a9c2f83537f956c2c_0'
(or any other file in the chain) for a quick glance at (compressed) length and
number of links (an extremely large number of copies would also tend to make
hash chains longer than they need to be as well as make comparisons tedious -
your $Conf{HardLinkMax} isn't by chance set to a ridiculously small value, is
it?). You might also want to 'BackupPC_zcat !$ | wc -c' (please expand the
!$ ;-) for an exact uncompressed file size. The inode number is included in
the ls example, in case you feel like doing a 'find $TopDir/pc -inum ...' for
locating the file(s) in question.
> [...] I should delete my pool and start new but I don't want to do that. :/
I'm not sure that would help much. If your backup strategy continues to backup
this content, you'll probably run into the same situation again in the future.
If you can and want to remove the file from (some) previous backups, search
the list for ... err ... something like BackupPC_delete - a script to safely
delete individual files from backups. BackupPC_nightly should take care of
cleaning up (i.e. chain renumbering), *BUT* *PLEASE don't run it manually*!
Of course, if you don't need to backup the file, simply excluding it would
also fix things, as the comparison wouldn't happen in the future.
Hope that helps.
Regards,
Holger
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
