Re: [BackupPC-users] Bizarre form of cpool corruption.
On Mon, Dec 06, 2010 at 11:33:15PM -0800, Craig Barratt wrote: > Robin, > > > Nope; I'm a big tab completion user, so I would have seen it. ;) > > > > $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc* > > 255523133 -rw-r- 3 backuppc backuppc 27 Nov 22 19:33 > > /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > > Hmmm. I'm not familiar with Jeffrey's zLib code. What happens > when you compute the digest of this pool file? Ie: > > perl /tmp/bpctest.pl /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc $ perl /tmp/bpctest.pl /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc 15c0e4b08058ef3704b8fc24887e2bcc /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > Here is another script that computes the BackupPC digest. You > might need to modify the library paths. $ sudo -u backuppc perl /tmp/bpctest2.pl /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt 15c0e4b08058ef3704b8fc24887e2bcc /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt So, yeah, that's really it. They're both really there, and that's the right md5sum, and both the pool file and the original file have more than 1 hardlink count, and there's no inode match. No idea how this happened, but I have a lot of them. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
Robin,
> Nope; I'm a big tab completion user, so I would have seen it. ;)
>
> $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc*
> 255523133 -rw-r- 3 backuppc backuppc 27 Nov 22 19:33
> /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc
Hmmm. I'm not familiar with Jeffrey's zLib code. What happens
when you compute the digest of this pool file? Ie:
perl /tmp/bpctest.pl /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc
Here is another script that computes the BackupPC digest. You might need
to modify the library paths.
Craig
##
#!/bin/perl
use strict;
use lib "/usr/local/BackupPC/lib";
use BackupPC::Lib;
use BackupPC::FileZIO;
die("BackupPC::Lib->new failed\n") if ( !(my $bpc = BackupPC::Lib->new) );
while ( @ARGV ) {
my $md5 = Digest::MD5->new;
my($data, $fileSize, $d, $r, $fh);
if ( !defined($fh = BackupPC::FileZIO->open($ARGV[0], 0, 1)) ) {
print(STDERR "$0: can't open $ARGV[0]\n");
exit;
}
$fh->read(\$data, 1048576);
$fileSize += length($data);
while ( ($r = $fh->read(\$d, 1048576)) > 0 ) {
$fileSize += $r;
}
$fh->close();
my $digest = $bpc->Buffer2MD5($md5, $fileSize, \$data);
print("$digest $ARGV[0]\n");
shift @ARGV;
}
--
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
On Mon, Dec 06, 2010 at 10:46:12PM -0800, Craig Barratt wrote: > Robin, > > > ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > > > > *BUT*. Not linked. > > > > $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > > 255523133 -rw-r- 3 backuppc backuppc 27 Nov 22 19:33 > > /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > > 2376493624 -rw-r- 3 backuppc backuppc 27 Nov 24 09:59 > > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > > You need to check all files with that digest (since there could be > collisions): > > $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc* > > I would expect one of them will have an inode of 2376493624. Nope; I'm a big tab completion user, so I would have seen it. ;) $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc* 255523133 -rw-r- 3 backuppc backuppc 27 Nov 22 19:33 /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc $ > I'm happy to help you look at the original problem with > BackupPC_tarPCCopy if you can't make progress using Jeffrey's > scripts. I honestly don't know how this happened, but the next time I use BackupPC_tarPCCopy I'll see if it gets weird. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
Robin, > ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > > *BUT*. Not linked. > > $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > 255523133 -rw-r- 3 backuppc backuppc 27 Nov 22 19:33 > /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc > 2376493624 -rw-r- 3 backuppc backuppc 27 Nov 24 09:59 > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt You need to check all files with that digest (since there could be collisions): $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc* I would expect one of them will have an inode of 2376493624. I'm happy to help you look at the original problem with BackupPC_tarPCCopy if you can't make progress using Jeffrey's scripts. Craig -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Another jLib/fixLinks issue.
Robin,
> That's a bunch of files with *thirty two thousand* hard links.
> Apparently that's a limit of some kind.
Yes, see $Conf{HardLinkMax}. Certain file systems allow more, but
several file systems have a limit of 32,000 (strangely not 2^15-1).
Craig
--
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] jLib bug with md5sum stuff
On Mon, Dec 06, 2010 at 12:32:01PM -0800, Robin Lee Powell wrote:
>
> substr outside of string at /usr/local/lib64/BackupPC/jLib.pm line 162.
> Use of uninitialized value in concatenation (.) or string at
> /usr/local/lib64/BackupPC/jLib.pm line 162.
>
> The line(s) in question:
>
> $datalast = substr($data[($i-1)%2], $rsize, _128KB-$rsize)
> . substr($data[$i%2], 0 ,$rsize);
>
> in zFile2MD5
>
> I have no idea how that could happen. -_- Will help debug any way
> requested.
Had to figure it out sooner rather than later. :)
It happens when a file is > 128KiB but < 256KiB, so only one buffer
is full. Fix:
--- /tmp/jLib.pm2010-12-07 05:47:42.086646699 +
+++ /usr/local/lib64/BackupPC/jLib.pm 2010-12-07 05:46:55.108231972 +
@@ -159,8 +159,8 @@
while ( ($rsize = $fh->read(\$data[(++$i)%2], _128KB)) == _128KB
&& ($totsize += $rsize) < _1MB) {}
$totsize +=$rsize if $rsize < _128KB; # Add back in partial read
- $datalast = ( $data[($i-1)%2] ? substr($data[($i-1)%2], $rsize,
_128KB-$rsize) : '' )
- . ( $data[$i%2] ? substr($data[$i%2], 0 ,$rsize) : '' );
+ $datalast = substr($data[($i-1)%2], $rsize, _128KB-$rsize)
+ . substr($data[$i%2], 0 ,$rsize);
}
$filesize = $totsize if $totsize < _1MB; #Already know the size because
read it all
if ($filesize == 0) { # Try to find size from attrib file
-Robin
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
--
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
On Mon, Dec 06, 2010 at 01:17:43PM -0800, Robin Lee Powell wrote:
>
> So, yeah. More than one link, matches something in the pool, but
> not actually linked to it. Isn't that *awesome*? ;'(
>
> I very much want BackupPC_fixLinks to deal with this, and I'm
> trying to modify it to do that now.
Seems to be working; here's the diff. Feel free to drop the print
statements. :)
For all I know this will eat your dog; I have no idea what else I
broke. I *do* know that it should be a flag, because I expect that
checksumming *everything* takes a very, very long time.
-Robin
--- /usr/local/bin/BackupPC_fixLinks2010-12-05 10:38:42.028375381 +
+++ /tmp/BackupPC_fixLinks 2010-12-06 22:53:00.174783302 +
@@ -358,6 +358,7 @@
sub BadOrMissingLinks {
my $matchpath = $_[0];
(my $matchname = $matchpath) =~ s|^$pc/*||; # Delete leading path
directories (up to machine)
+ print "file: $matchpath\n";
my $rettype;
my $matchtype;
@@ -368,7 +369,7 @@
warnerr "Can't stat: $matchpath\n";
return;
}
-if ($nlinkM == 1 && $sizeM > 0) { # Non-zero file with no link to pool
+if ($sizeM > 0) { # Non-zero file with no link to pool
my $matchbyte = firstbyte($matchpath);
my $comparflg = 'x'; # Default if no link to pool
my $matchtype = "NewFile"; # Default if no link to pool
@@ -386,6 +387,11 @@
my $md5sumpath = my $md5sumpathbase = $bpc->MD52Path($md5sum,
0, $thepooldir);
my $i;
for ($i=-1; -f $md5sumpath ; $md5sumpath = $md5sumpathbase .
'_' . ++$i) {
+ my $md5sumpathinode = (stat($md5sumpath))[1];
+ # This is actually a correct and matching link; do
nothing
+ if( $md5sumpathinode == $inoM ) {
+ return -1;
+ }
#Again start at the root, try to find best match in pool...
if ((my $cmpresult = compare_files ($matchpath,
$md5sumpath, $cmprsslvl)) > 0) { #match found
@@ -405,6 +411,7 @@
$matchtype = "NewLink";
$totnewlinks++;
$rettype=2; #NewLink
+ print "NEW LINK files: `/bin/ls -li
$matchpath $md5sumpath` \n";
goto match_return;
} #Otherwise, continue to move up the chain looking
for a pool match...
}
@@ -420,6 +427,7 @@
$md5sumhash{$fullmd5sum} = $md5sum;
$rettype=3; #NewFile-x
}
+ print "NO MATCH files: `/bin/ls -li
$matchpath $md5sumpath` \n";
match_return:
@MatchA = ($matchname, $inoM, $md5sum, $matchtype, $thepool,
${comparflg}.${matchbyte}.${md5sumbyte}, $nlinkM, $sizeM);
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
--
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Another jLib/fixLinks issue.
On Mon, Dec 06, 2010 at 03:40:04PM -0800, Robin Lee Powell wrote: > This is *fascinating*. > > >From the actually-fixing-stuff part of the run, I get: > > ERROR: > "tm50-s00292__nfs/68/f%2f/fshared/fthepoint/fsite_images/f0042/f4097/fMULTI_medium.jpg" > - Too many links if added to "59c43b51dbdd9031ba54971e359cdcec" > > to which I say "lolwut?" and investigate. > > $ ls -li /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec* > 2159521202 -rw-r- 31999 backuppc backuppc 76046 Oct 7 08:29 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec > 2670969865 -rw-r- 31999 backuppc backuppc 76046 Oct 16 15:15 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_0 > 79561977 -rw-r- 31999 backuppc backuppc 76046 Oct 22 22:07 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_1 > 156369809 -rw-r- 31999 backuppc backuppc 76046 Oct 31 09:06 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_2 > 3389777838 -rw-r- 31999 backuppc backuppc 76046 Nov 7 09:10 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_3 > 106188559 -rw-r- 31999 backuppc backuppc 76046 Nov 13 15:10 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_4 > 247044591 -rw-r- 31999 backuppc backuppc 76046 Nov 19 17:20 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_5 > 293083240 -rw-r- 31999 backuppc backuppc 76046 Nov 26 06:14 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_6 > 513555136 -rw-r- 31999 backuppc backuppc 76046 Dec 1 19:37 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_7 > 52908307 -rw-r- 7767 backuppc backuppc 76046 Dec 4 10:37 > /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_8 > $ ls -li > /backups/pc/tm50-s00292__nfs/68/f%2f/fshared/fthepoint/fsite_images/f0042/f4097/fMULTI_medium.jpg > 374791856 -rw-r- 1 backuppc backuppc 76046 Dec 4 08:03 > /backups/pc/tm50-s00292__nfs/68/f%2f/fshared/fthepoint/fsite_images/f0042/f4097/fMULTI_medium.jpg > > That's a bunch of files with *thirty two thousand* hard links. > Apparently that's a limit of some kind. BackupPC handles this by > adding new copies, a hack that BackupPC_fixLinks is apparently > unaware of. I have many pages of these errors, btw; thousands, I think. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Another jLib/fixLinks issue.
This is *fascinating*. >From the actually-fixing-stuff part of the run, I get: ERROR: "tm50-s00292__nfs/68/f%2f/fshared/fthepoint/fsite_images/f0042/f4097/fMULTI_medium.jpg" - Too many links if added to "59c43b51dbdd9031ba54971e359cdcec" to which I say "lolwut?" and investigate. $ ls -li /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec* 2159521202 -rw-r- 31999 backuppc backuppc 76046 Oct 7 08:29 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec 2670969865 -rw-r- 31999 backuppc backuppc 76046 Oct 16 15:15 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_0 79561977 -rw-r- 31999 backuppc backuppc 76046 Oct 22 22:07 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_1 156369809 -rw-r- 31999 backuppc backuppc 76046 Oct 31 09:06 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_2 3389777838 -rw-r- 31999 backuppc backuppc 76046 Nov 7 09:10 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_3 106188559 -rw-r- 31999 backuppc backuppc 76046 Nov 13 15:10 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_4 247044591 -rw-r- 31999 backuppc backuppc 76046 Nov 19 17:20 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_5 293083240 -rw-r- 31999 backuppc backuppc 76046 Nov 26 06:14 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_6 513555136 -rw-r- 31999 backuppc backuppc 76046 Dec 1 19:37 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_7 52908307 -rw-r- 7767 backuppc backuppc 76046 Dec 4 10:37 /backups/cpool/5/9/c/59c43b51dbdd9031ba54971e359cdcec_8 $ ls -li /backups/pc/tm50-s00292__nfs/68/f%2f/fshared/fthepoint/fsite_images/f0042/f4097/fMULTI_medium.jpg 374791856 -rw-r- 1 backuppc backuppc 76046 Dec 4 08:03 /backups/pc/tm50-s00292__nfs/68/f%2f/fshared/fthepoint/fsite_images/f0042/f4097/fMULTI_medium.jpg That's a bunch of files with *thirty two thousand* hard links. Apparently that's a limit of some kind. BackupPC handles this by adding new copies, a hack that BackupPC_fixLinks is apparently unaware of. Did I mention that my installation is large? :D -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
On Mon, Dec 06, 2010 at 03:48:04PM -0500, Jeffrey J. Kosowsky wrote: > Robin Lee Powell wrote at about 10:37:52 -0800 on Monday, December 6, 2010: > > > > So I'm writing a script to transfer a client from one host to > > another, using tarPCCopy, and I'm getting messages like this: > > > > Can't find > foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > in pool, will copy file > > > > which is fascinating because the first column in ls -l is *3*. -_- > > > > The tarPCCopy tar file therefore ends up becoming really large > > (hundreds of gibibytes) with files that already exist in the pool, > > presumably. > > > > I've tried running md5sum on that file; can't find that in the pool. > > I've tried BackupPC_zcat | md5sum; can't find that in the pool. > > Well the 'md5sum' used in pool naming is only a partial file md5sum. > I wrote (and posted) a routine to calculate and optionally test for > existence of the md5sum pool name corresponding to any pc tree > file. I will attach a copy to the end of this post. > > > BackupPC_fixLinks, from the wiki, doesn't see the problem at all, > > which I'd *very* much like to fix. > > First check to make sure there really is a problem with the pool... > Then, we need to figure out whether there is a problem with tarcopy or > with my program BackupPC_fixLinks etc. $ ls -l /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt -rw-r- 3 backuppc backuppc 27 Nov 24 09:59 /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt $ perl /tmp/bpctest.pl /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt 15c0e4b08058ef3704b8fc24887e2bcc /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt $ ls -l /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc -rw-r- 3 backuppc backuppc 27 Nov 22 19:33 /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt *BUT*. Not linked. $ ls -li /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt 255523133 -rw-r- 3 backuppc backuppc 27 Nov 22 19:33 /backups/cpool/1/5/c/15c0e4b08058ef3704b8fc24887e2bcc 2376493624 -rw-r- 3 backuppc backuppc 27 Nov 24 09:59 /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt Isn't that fascinating, boys and girls? Let's check another. $ ls -l /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7809/fposter/fposter_234517.zip -rw-r- 3 backuppc backuppc 8510861 Nov 24 09:14 /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7809/fposter/fposter_234517.zip $ perl /tmp/bpctest.pl /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7809/fposter/fposter_234517.zip 42a13e7f5875b2d8ff79ae54e2cb41a9 /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7809/fposter/fposter_234517.zip $ ls -l /backups/cpool/4/2/a/42a13e7f5875b2d8ff79ae54e2cb41a9 -rw-r- 3 backuppc backuppc 8510861 Nov 22 18:53 /backups/cpool/4/2/a/42a13e7f5875b2d8ff79ae54e2cb41a9 $ ls -li /backups/cpool/4/2/a/42a13e7f5875b2d8ff79ae54e2cb41a9 /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7809/fposter/fposter_234517.zip 3696447185 -rw-r- 3 backuppc backuppc 8510861 Nov 22 18:53 /backups/cpool/4/2/a/42a13e7f5875b2d8ff79ae54e2cb41a9 145635130 -rw-r- 3 backuppc backuppc 8510861 Nov 24 09:14 /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7809/fposter/fposter_234517.zip So, yeah. More than one link, matches something in the pool, but not actually linked to it. Isn't that *awesome*? ;'( I very much want BackupPC_fixLinks to deal with this, and I'm trying to modify it to do that now. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned witho
Re: [BackupPC-users] Feature request: "finish what you're doing and then stop"
Dne 6.12.2010 20:02, Robin Lee Powell napsal(a): > > It would be really nice to be able to tell the backuppc server to > finish all current backups without queuing any others, and then > stop/exit completely. > > I know I can sort-of do this by disabling backups for each host, but > that's a really big pain, and from the reading of the queuing system > I did a couple of weeks ago, I think this would be pretty easy to > add. > > -Robin Hi, I opened a similar topic a few weeks ago here http://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg20221.html . The proposed solution - BackupsDisable to 2 on the main config level in config.pl and BackupPC_serverMesg server reload has been a complete success for my needs. Of course a built-in solution would be certainly welcome :) Regards, Pavel. -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] jLib.pm question
On 12/6/10 2:43 PM, Robin Lee Powell wrote: > >> I'm not sure what you mean... the program should work with any >> number of links. If there is a missing use case, please explain so >> I can fix it. > > I have files with 3+ links that, according to BackupPC_tarPCCopy, > aren't actually linked into the pool. See my other post. I think files backed up with rsync would make the links within a host tree for repeated runs even if the pool copy had been deleted accidentally. Or if the pc tree had been copied from some other location by some method that lost the pool link. > As a random side comment, it's amazing how much interesting stuff > I'm finding. :) I wonder if our BackupPC install is now the > largest in the world; just under 25 TiB across 5 hosts and thousands > of clients. I suspect I'm hitting so much stuff because no-one has > tried to productionalize it at this scale before. Could be a record. -- Les Mikesell lesmikes...@gmail.com -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
On Mon, Dec 06, 2010 at 03:48:04PM -0500, Jeffrey J. Kosowsky wrote: > Robin Lee Powell wrote at about 10:37:52 -0800 on Monday, December 6, 2010: > > > > So I'm writing a script to transfer a client from one host to > > another, using tarPCCopy, and I'm getting messages like this: > > > > Can't find > foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > in pool, will copy file > > > > which is fascinating because the first column in ls -l is *3*. -_- > > > > The tarPCCopy tar file therefore ends up becoming really large > > (hundreds of gibibytes) with files that already exist in the pool, > > presumably. > > > > I've tried running md5sum on that file; can't find that in the pool. > > I've tried BackupPC_zcat | md5sum; can't find that in the pool. > > Well the 'md5sum' used in pool naming is only a partial file md5sum. > I wrote (and posted) a routine to calculate and optionally test for > existence of the md5sum pool name corresponding to any pc tree > file. I will attach a copy to the end of this post. > > > BackupPC_fixLinks, from the wiki, doesn't see the problem at all, > > which I'd *very* much like to fix. > > First check to make sure there really is a problem with the pool... > Then, we need to figure out whether there is a problem with tarcopy or > with my program BackupPC_fixLinks etc. Before I go off testing that, I just want to mention that you're my hero. :D Next to the actual authors of BackupPC, of course. I've made so much use of your scripts it's not even funny; BackupPC is great but it breaks down when you're trying to juggle super-large clients (I have *one BackupPC client* with 2.3TiB in the pool) around between various BackupPC hosts, and other crazy large-scale crap. Your scripts have been invaluable. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
Robin Lee Powell wrote at about 10:37:52 -0800 on Monday, December 6, 2010:
>
> So I'm writing a script to transfer a client from one host to
> another, using tarPCCopy, and I'm getting messages like this:
>
> Can't find
> foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt
> in pool, will copy file
>
> which is fascinating because the first column in ls -l is *3*. -_-
>
> The tarPCCopy tar file therefore ends up becoming really large
> (hundreds of gibibytes) with files that already exist in the pool,
> presumably.
>
> I've tried running md5sum on that file; can't find that in the pool.
> I've tried BackupPC_zcat | md5sum; can't find that in the pool.
Well the 'md5sum' used in pool naming is only a partial file md5sum.
I wrote (and posted) a routine to calculate and optionally test for
existence of the md5sum pool name corresponding to any pc tree
file. I will attach a copy to the end of this post.
> BackupPC_fixLinks, from the wiki, doesn't see the problem at all,
> which I'd *very* much like to fix.
First check to make sure there really is a problem with the pool...
Then, we need to figure out whether there is a problem with tarcopy or
with my program BackupPC_fixLinks etc.
#!/usr/bin/perl
#= -*-perl-*-
#
# BackupPC_zfile2md5.pl: calculate and optionally verify the BackupPC-style
#partial md5sum of any file compressed by BackupPC
#
# DESCRIPTION
# This program allows you to calculate the partial md5sum
# corresponding to the cpool path for any file that uses
# BackupPC-style compression whether or not that file is actually
# stored or linked to the cpool. Optionally, if the file is a cpool
# entry or is linked to the cpool, you can add the '-k' flag to
# verify whether the corresponding cpool path is consistent with the
# actual md5sum of the file.
#
# Multiple files or directories can be given on the command line,
# allowing you to calculate (and optionally verify) the md5sum for
# multiple files or multiple trees of files. The script also does
# path globbing using standard shell globbing conventions.
#
# Paths are assumed to be either absolute or relative to the current
# directory unless, the options -C, -c, or -p are given in which
# case the paths are understood to be a cpool file name (without
# path), a path relative to the cpool, or a path relative to the
# pc directory, respectively.
# AUTHOR
# Jeff Kosowsky
#
# COPYRIGHT
# Copyright (C) 2009 Jeff Kosowsky
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
#
#
# Version 1.0, released Dec 2009
#
#
use strict;
use warnings;
use Getopt::Std;
use lib "/usr/share/BackupPC/lib";
use BackupPC::FileZIO;
use BackupPC::Lib;
use BackupPC::jLib;
use Cwd 'abs_path';
use File::Find;
use File::Glob ':glob';
die("BackupPC::Lib->new failed\n") if ( !(my $bpc = BackupPC::Lib->new("", "",
"", 1)) ); #No user check
%Conf = $bpc->Conf(); #Global variable defined in jLib.pm (do not use 'my')
my %opts;
if ( !getopts("Ccpka", \%opts) || @ARGV < 1
|| (defined($opts{C}) + defined($opts{c}) + defined($opts{p}) > 1)) {
print STDERR < 0 ? "cpool" : "pool";
my $md5 = Digest::MD5->new;
my @zpathlist;
foreach (@ARGV) {
if($opts{C}) {
@zpathlist = (@zpathlist, bsd_glob($bpc->MD52Path($_,
$compress)));
} elsif($opts{c}) {
@zpathlist = (@zpathlist, bsd_glob($bpc->TopDir() . "cpool/" .
$_));
} elsif($opts{p}) {
@zpathlist = (@zpathlist, bsd_glob($bpc->TopDir() . "pc/" .
$_));
} else {
@zpathlist = (@zpathlist, bsd_glob(abs_path($_)));
}
}
die "No valid paths...\n" unless @zpathlist;
foreach my $zpath (@zpathlist) {
$zpath =~ s#/+#/#g; #Remove extra slashes
$zpath =~ s#/\.(/|$)#/#g; #Remove extra /.
unless(-e $zpath) {
warn "'$zpath' is not an existing file or directory path...\n";
next;
}
find(\&ch
Re: [BackupPC-users] jLib.pm question
On Mon, Dec 06, 2010 at 03:33:52PM -0500, Jeffrey J. Kosowsky wrote:
> Robin Lee Powell wrote at about 11:28:56 -0800 on Monday, December 6, 2010:
> >
> > I've locally modified BackupPC_fixLinks to check files even if
> > they have more than one link.
>
> I'm not sure what you mean... the program should work with any
> number of links. If there is a missing use case, please explain so
> I can fix it.
I have files with 3+ links that, according to BackupPC_tarPCCopy,
aren't actually linked into the pool. See my other post.
> >I'm getting a lot of errors like this:
> >
> > Can't read size of
> /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/flogs/fapplication.log-20100830.gz
> from attrib file so calculating manually
> >
> > Would it be possible for jLib to fix that, so it doesn't happen
> > again on subsequent runs?
>
> It's been a while since I wrote the program... but I think that
> not being able to read the size is a true error case and I would
> wonder why the attrib file is not including the filesize entry. As
> far as I recall, every file backed up should be listed in the
> attrib file and the size is one of the basic features that should
> always be recorded.
Yeah, these backups are broken somehow; see above.
> Are your attrib files corrupted?
I don't know? What should they look like?
> Can you look at the attrib file and see what might be wrong and/or
> different about that file?
/me finds BackupPC_attribPrint :)
I'm afraid it looks like a bug in your code. :(
Example:
Can't read size of
/backups/pc/customer--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/fbubblevision/freports/fproduct_sales/fproduct_sales_2010-04-05_2334.zip
from attrib file so calculating manually
Attrib file:
$ ls -l
/backups/pc/customer--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/fbubblevision/freports/fproduct_sales/attrib
-rw-r- 12 backuppc backuppc 86 Oct 12 18:04
/backups/pc/customer--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/fbubblevision/freports/fproduct_sales/attrib
Contents thereof:
$ sudo -u backuppc BackupPC_attribPrint
/backups/pc/customer--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/fbubblevision/freports/fproduct_sales/attrib
$VAR1 = {
'product_sales_2010-04-05_2333.zip' => {
'uid' => 1000,
'mtime' => 1270535656,
'mode' => 33188,
'size' => 3527103,
'sizeDiv4GB' => 0,
'type' => 0,
'gid' => 415,
'sizeMod4GB' => 3527103
},
'product_sales_2010-04-05_2330.zip' => {
'uid' => 1000,
'mtime' => 1270535436,
'mode' => 33188,
'size' => 3527103,
'sizeDiv4GB' => 0,
'type' => 0,
'gid' => 415,
'sizeMod4GB' => 3527103
},
'product_sales_2010-04-05_2325.zip' => {
'uid' => 1000,
'mtime' => 1270535162,
'mode' => 33188,
'size' => 3527103,
'sizeDiv4GB' => 0,
'type' => 0,
'gid' => 415,
'sizeMod4GB' => 3527103
},
'product_sales_2010-04-05_2334.zip' => {
'uid' => 1000,
'mtime' => 1270535712,
'mode' => 33188,
'size' => 3527103,
'sizeDiv4GB' => 0,
'type' => 0,
'gid' => 415,
'sizeMod4GB' => 3527103
}
};
As you can see, fproduct_sales_2010-04-05_2334.zip *does* have an
associated size.
As a random side comment, it's amazing how much interesting stuff
I'm finding. :) I wonder if our BackupPC install is now the
largest in the world; just under 25 TiB across 5 hosts and thousands
of clients. I suspect I'm hitting so much stuff because no-one has
tried to productionalize it at this scale before.
-Robin
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
--
What happens now with your Lotus Notes apps - do you make another costly
upgrade, or settle for being marooned without product support? Time to move
off Lotus Notes and onto the cloud with Force.com, apps are easier to build,
use, and manage than apps on traditional platforms. Sign up for the Lotus
Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] jLib.pm question
Robin Lee Powell wrote at about 11:28:56 -0800 on Monday, December 6, 2010: > > I've locally modified BackupPC_fixLinks to check files even if they > have more than one link. I'm not sure what you mean... the program should work with any number of links. If there is a missing use case, please explain so I can fix it. >I'm getting a lot of errors like this: > > Can't read size of > /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/flogs/fapplication.log-20100830.gz > from attrib file so calculating manually > > Would it be possible for jLib to fix that, so it doesn't happen > again on subsequent runs? It's been a while since I wrote the program... but I think that not being able to read the size is a true error case and I would wonder why the attrib file is not including the filesize entry. As far as I recall, every file backed up should be listed in the attrib file and the size is one of the basic features that should always be recorded. Are your attrib files corrupted? Can you look at the attrib file and see what might be wrong and/or different about that file? If those are 'normal', perhaps there is something wrong with my code... > Or is that something that should be: fixed? I'm not really clear on > what the problem is. I actually wrote the warning more as belt-and-suspenders since that case should only appear if your attrib file is corrupted (unless I am not understanding something about BackupPC attrib files) -- to tell the truth, I don't believe I ever encountered this error on *real* data. I don't think it would be a good idea to add to BackuPC_fixLinks to fix bad attrib files. The "beauty" of this program is that it is all about detecting and fixing bad *links* per its name. One could easily write another program to check attrib files for integrity and fix where broken (Actually, size is probably the only piece of redundant information since it is also calculable from the actual file). Note also that BackupPC_fixLinks wasn't designed to detect (let alone) fix inconsistent attrib content and indeed it only even looks at the attrib file for size information as a shortcut for files ~>1MB so that it doesn't have to decompress the cpool. Let me know if this answers your question or if I misunderstood what you were requesting... -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] jLib bug with md5sum stuff
substr outside of string at /usr/local/lib64/BackupPC/jLib.pm line 162. Use of uninitialized value in concatenation (.) or string at /usr/local/lib64/BackupPC/jLib.pm line 162. The line(s) in question: $datalast = substr($data[($i-1)%2], $rsize, _128KB-$rsize) . substr($data[$i%2], 0 ,$rsize); in zFile2MD5 I have no idea how that could happen. -_- Will help debug any way requested. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Bizarre form of cpool corruption.
On Mon, Dec 06, 2010 at 10:37:52AM -0800, Robin Lee Powell wrote: > > So I'm writing a script to transfer a client from one host to > another, using tarPCCopy, and I'm getting messages like this: > > Can't find > foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt > in pool, will copy file > > which is fascinating because the first column in ls -l is *3*. -_- > > The tarPCCopy tar file therefore ends up becoming really large > (hundreds of gibibytes) with files that already exist in the pool, > presumably. > > I've tried running md5sum on that file; can't find that in the pool. > I've tried BackupPC_zcat | md5sum; can't find that in the pool. I see that that's not how BackupPC md5sums work. > BackupPC_fixLinks, from the wiki, doesn't see the problem at all, > which I'd *very* much like to fix. I've locally modified my copy to check all files. Trying it out now. I'd still like to know what the hell happened here. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] jLib.pm question
I've locally modified BackupPC_fixLinks to check files even if they have more than one link. I'm getting a lot of errors like this: Can't read size of /backups/pc/foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/flogs/fapplication.log-20100830.gz from attrib file so calculating manually Would it be possible for jLib to fix that, so it doesn't happen again on subsequent runs? Or is that something that should be fixed? I'm not really clear on what the problem is. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Feature request: "finish what you're doing and then stop"
It would be really nice to be able to tell the backuppc server to finish all current backups without queuing any others, and then stop/exit completely. I know I can sort-of do this by disabling backups for each host, but that's a really big pain, and from the reading of the queuing system I did a couple of weeks ago, I think this would be pretty easy to add. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Don't use rsync -H!! (was Re: copying the pool to a new filesystem?!)
On Thu, Dec 02, 2010 at 01:27:17PM +0100, Oliver Freyd wrote: > Hello, > > I'm a happy user of BackupPC since a few years, > running an old installation of backuppc that was created > on some version of SuSE linux, then ported over to debian lenny. > > The pool is a reiserfs3 on LVM, about 300GB size, but with a lot of > hardlinks... > Now I'm trying to put the pool onto a new filesystem, so I created an > XFS on a striped RAID0 of 3 disks (to speed up copying), and use > rsync -aHv to copy everything including the hardlinks. > The cpool itself took about a day, and now it is running for 6 days and > maybe it has done 70% of the work. BTW, a copy with dd takes about 2 hours. > > I've tried to do this with BackupPC_TarPCCopy, but it does not seem to > be any faster. IME it's *much* faster that way; you do BackupPC_TarPCCopy, and then rsync the cpoll *without -H*. It shouldn't take any longer than the actual data transfer time itself. I've moved user backups of a TiB and larger in a day or two this way. If you try to copy BackupPC data with rsync -H, you're doing it wrong. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Bizarre form of cpool corruption.
So I'm writing a script to transfer a client from one host to another, using tarPCCopy, and I'm getting messages like this: Can't find foo--tm50-e00145--tm50-s00339---shared/47/f%2f/fshared/ffoo/fpurchase_order_assets/fbatch_7813/f7105620_done.txt in pool, will copy file which is fascinating because the first column in ls -l is *3*. -_- The tarPCCopy tar file therefore ends up becoming really large (hundreds of gibibytes) with files that already exist in the pool, presumably. I've tried running md5sum on that file; can't find that in the pool. I've tried BackupPC_zcat | md5sum; can't find that in the pool. BackupPC_fixLinks, from the wiki, doesn't see the problem at all, which I'd *very* much like to fix. I don't have the room to copy a duplicate of this pool like this. It's not happening with every file, but it's happening enough to be a real problem. What's going on here? How can I fix it? -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Sanity check re: pooling.
On Mon, 2010-12-06 at 10:47 -0500, Ken D'Ambrosio wrote: > Hi! I've got two servers, each around a TB, that I'm backing up with > BackupPC. (No, not for "real" backups, but to be able to not have to > recall tapes when someone deletes a file. Darn users.) I'm planning on > merging the two Windows servers into one Linux box serving via Samba. > Assuming pooling works the way I think it does, there shouldn't be any > significant increase in needed disk space, should there? > > Thanks! > > -Ken > > I think you are right, in the medium term. In the short term, if the host is different then everything will be new and will be transferred over to the backup machine. Once there, it will be checked against the pool and each file will be replaced with a hard link to the pooled version from the original server, therefore you'll eventually get all the space back to what it was with two servers. Ed # Attention: This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary, confidential and/or privileged information and should not be copied, disclosed, distributed, retained or used by any other party. If you are not an intended recipient please notify the sender immediately and delete this e-mail (including attachments and copies). The statements and opinions expressed in this e-mail are those of the author and do not necessarily reflect those of the Royal Marsden NHS Foundation Trust. The Trust does not take any responsibility for the statements and opinions of the author. Website: http://www.royalmarsden.nhs.uk # -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Sanity check re: pooling.
Hi! I've got two servers, each around a TB, that I'm backing up with BackupPC. (No, not for "real" backups, but to be able to not have to recall tapes when someone deletes a file. Darn users.) I'm planning on merging the two Windows servers into one Linux box serving via Samba. Assuming pooling works the way I think it does, there shouldn't be any significant increase in needed disk space, should there? Thanks! -Ken -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- What happens now with your Lotus Notes apps - do you make another costly upgrade, or settle for being marooned without product support? Time to move off Lotus Notes and onto the cloud with Force.com, apps are easier to build, use, and manage than apps on traditional platforms. Sign up for the Lotus Notes Migration Kit to learn more. http://p.sf.net/sfu/salesforce-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
