Re: [BackupPC-users] BUG in backuppc md5sum calculation for root attrib files (WAS: cpool md5sum errors with certain attrib files)
Jeffrey J. Kosowsky wrote at about 03:33:11 -0500 on Tuesday, December 29, 2009:
> Jeffrey J. Kosowsky wrote at about 11:50:04 -0500 on Tuesday, December 22,
> 2009:
> > In my neuroses, I ran a perl script that recursed through the cpool
> > and checked whether the md5sum of each stored file corresponded to its
> > location in the pool (note when I say md5sum I mean the special notion
> > of md5sum defined in BackupPC::Lib.pm)
> >
> > 1. Out of a total of 855,584 pool entries, I found a total of 35 errors.
> >
> > 2. Interestingly, all 35 of these errors corresponded to 'attrib' files.
> >
> > 3. Perhaps even more interestingly, all but two of the attrib files
> >were at the top level -- i.e., $TopDir/pc///attrib
> >(this represents 33 out of a total of 87 backups)
> >
> > 4. None of the attrib files appear corrupted when I examine them using
> >BackupPC_attribPrint
> >
> > So what could possibly be causing the md5sum to be wrong just on a
> > small subset of my pool files?
> >
> > Why are these errors exclusively limited to attrib files of which
> > almost all are top-level attrib files (even though they constitute a
> > tiny fraction of total attrib files)?
> >
> > - Disk corruption or hardware errors seem unlikely due to the specific
> > nature of these errors and the fact that the file data itself seems
> > intact
> >
> > Of course, I could easily write a routine to "fix" these errors, but I
> > just don't understand what is wrong here. I suppose the errors aren't
> > particularly dangerous in that the only potential issue they could
> > cause would be some missed opportunities for pool de-duplication of
> > stored attrib files. But there shouldn't be wrong pool md5sums...
> >
>
> OK. I think I found a way to reproduce this.
>
> The md5sum for the root level attrib (i.e., the attrib file at the
> level of pc/machine/attrib) is wrong if:
> 1. There are at least 2 shares
> 2. The attrib entries for each of the shares has changed since the
>last backup (e.g., if the share directory has it's mtime modified
>
> Try the following on a machine with >=2 shares
> 1. Touch one of the share directories (to change the mtime)
> 2. Run a backup
> 3. Run another backup immediately afterwards (or more specifically
>without changing any of the attrib entries for each of the shares)
> 4. Look at:
>diff machine/n/attrib machine/n+1/attrib
> ==> no diffs
>ls -i machine/n/attrib machine/n+1/attrib
> ==> different i-nodes
> 5. The *2nd* attrib is stored in the correct md5sum cpool entry; the
> first one is not.
>
OK. I found the bug. NOTE THIS BUG AFFECTS EVERYBODY WHO IS BACKING UP
MORE THAN ONE SHARE -- i.e., everybody backing up more than one file
has such pool errors though they should not effect data integrity,
only pooling efficiency.
The problem is that after each share is backed up, the base attrib
file ($TopDir/machine/n/attrib) is written to NewFileList.
Note that the logger notes this with:
attribWrite(dir=) -> /var/lib/BackupPC//pc/machine/new//attrib
When you have multiple shares then multiple versions of the base
attrib file appear in NewList.
Then when BackupPC_link is called, it goes through NewList in order so
it ends up linking the attrib file using the first instance of the
md5sum in the list. But this of course is the wrong md5sum since it
only includes the attrib info for the first share.
When subsequent entries for the base attrib are looped through, the
routine MakeFileLink does nothing since the following test fails:
} elsif ( $newFile && -f $name && (stat($name))[3] == 1 ) {
This all explains why:
1. The error only occurs when backing up more than one share
2. If there are m shares, then the error only occurs on the first
(m-1) backups where there have been no changes to the share
directories. That is because on each subsequent backup, one fewer
intermediate base attrib is added to NewFileList since each backup
adds an intermediate version to the cpool. So finally, after 'm-1'
only the actual final base attrib is added to NewFileList and can
then be linked properly to the cpool.
Here are some thoughts on approaches to fixing the bug:
1. You could 'reverse' the iteration through the Newfile
list but this may not be easy since currently it is read in one-by-one
on-the-fly as follows:
if ( open(NEW, "<", "$Dir/NewFileList.$Backups[$num]{num}") ) {
binmode(NEW);
while ( ) {
chomp;
next if ( !/(\w+) (\d+) (.*)/ );
LinkNewFile($1, $2, "$CurrDumpDir/$3");
}
close(NEW);
}
2. An alternative approach would be to eliminate the test
(stat($name))[3] == 1 )
noted above which would allow later links to overwrite earlier
ones. However, this may have other unforeseen ill effects
3. A third approach would be to only wri
Re: [BackupPC-users] Windows bare metal restore
On 12/29 12:51 , l...@consolejunky.net wrote: > add this to the list of possible Windows imaging issues: > > http://anandtech.com/storage/showdoc.aspx?i=3691 Thanks for the link, that's an informative article. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsyncd via ssh-redirected port
Les Mikesell schrieb:
> Guido Schmidt wrote:
>> So, what could cause BackupPC not to connect to the tunnel?
>>
>
> I may have missed something in the thread, but did you change the ping
> command
> to something that would succeed?
>
Yes. The host is publicly available, I just put it's real name into
$Conf{PingCmd} = '$pingPath -c 1 -w 3 host.example.com';
(Have a look at my first posting.)
Guido
--
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BUG? manual run of BackupPC_dump doesn't recognize"ClientNameAlias" - NEVER MIND!!!!
Jeffrey J. Kosowsky wrote at about 01:44:13 -0500 on Tuesday, December 29, 2009:
> In the file /etc/BackupPC/pc/machinealias.pl, I have set:
>$Conf{ClientNameAlias} = mymachine
> When I run from my shell
> /usr/share/BackupPC/bin/BackupPC_dump -d -v -f mymachinealias
> I get:
> cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 mymachinealias
> cmdSystemOrEval: finished: got output ping: unknown host mymachinealias
>
> The backup however works fine when run from the web interface.
>
> It seems like running BackupPC_dump doesn't use ClientNameAlias.
> This seems to be a bug or at the very least an undesirable feature ;)
>
>
My bad... the problem was the '-d' flag which I have always added by
habit which signals that the backup uses a dhcp address...
It works fine without the '-d' flag...
--
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] Any ETA for official Version 3.2.0 release?
Version 3.1.0 is getting a bit long in the tooth (2+ years old) with a number of important bugfixes added since then. Also, 3.2.0beta0 has been out for almost 8 months and based on the traffic on this mailing list seems to be rock stable. The reason I ask, is that it would be nice to get the latest version into the official distros and some distros seem to shy away from upgrading to beta releases... -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backing up a laptop via DHCP
> My home network is basically a windows Workgroup and I was planning on > access my laptops via smb. From my backuppc server I can see both laptops > on the workgroup. > > > > Any help, guidance would be much appreciated. > The easiest thing I guess would be to: First: install Samba to be the WINS server for your network. (wins support = yes) check 7.3.3: http://oreilly.com/catalog/samba/chapter/book/ch07_03.html (It's only a really short step from here to have samba also be the fileserver for your home network) 2d: Install DHCP3-server. There are a lot of examples but the essential is giving the wins server address with the dhcp lease. That way your windows desktops will always register their netbios name with the samba wins server and backuppc can query it as mentioned with the nmblookup command. As a side effect network browsing in windows will also speed up. If you don't want to install and configure this, you can always set the wins server manually in the advanced network settings in windows. Maybe it is possible to have Backuppc check with dns instead of netbios but that's still harder to configure in my opinion. Greetings, Koen Linders -- This SF.Net email is sponsored by the Verizon Developer Community Take advantage of Verizon's best-in-class app development support A streamlined, 14 day to market process makes app distribution fast and easy Join now and get one step closer to millions of Verizon customers http://p.sf.net/sfu/verizon-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] BUG in backuppc md5sum calculation for root attrib files (WAS: cpool md5sum errors with certain attrib files)
Jeffrey J. Kosowsky wrote at about 11:50:04 -0500 on Tuesday, December 22, 2009: > In my neuroses, I ran a perl script that recursed through the cpool > and checked whether the md5sum of each stored file corresponded to its > location in the pool (note when I say md5sum I mean the special notion > of md5sum defined in BackupPC::Lib.pm) > > 1. Out of a total of 855,584 pool entries, I found a total of 35 errors. > > 2. Interestingly, all 35 of these errors corresponded to 'attrib' files. > > 3. Perhaps even more interestingly, all but two of the attrib files >were at the top level -- i.e., $TopDir/pc///attrib >(this represents 33 out of a total of 87 backups) > > 4. None of the attrib files appear corrupted when I examine them using >BackupPC_attribPrint > > So what could possibly be causing the md5sum to be wrong just on a > small subset of my pool files? > > Why are these errors exclusively limited to attrib files of which > almost all are top-level attrib files (even though they constitute a > tiny fraction of total attrib files)? > > - Disk corruption or hardware errors seem unlikely due to the specific > nature of these errors and the fact that the file data itself seems > intact > > Of course, I could easily write a routine to "fix" these errors, but I > just don't understand what is wrong here. I suppose the errors aren't > particularly dangerous in that the only potential issue they could > cause would be some missed opportunities for pool de-duplication of > stored attrib files. But there shouldn't be wrong pool md5sums... > OK. I think I found a way to reproduce this. The md5sum for the root level attrib (i.e., the attrib file at the level of pc/machine/attrib) is wrong if: 1. There are at least 2 shares 2. The attrib entries for each of the shares has changed since the last backup (e.g., if the share directory has it's mtime modified Try the following on a machine with >=2 shares 1. Touch one of the share directories (to change the mtime) 2. Run a backup 3. Run another backup immediately afterwards (or more specifically without changing any of the attrib entries for each of the shares) 4. Look at: diff machine/n/attrib machine/n+1/attrib ==> no diffs ls -i machine/n/attrib machine/n+1/attrib ==> different i-nodes 5. The *2nd* attrib is stored in the correct md5sum cpool entry; the first one is not. To explore this, you can use the following perl script I wrote: In particular, try something like BackupPC_zfile2md5 -p -k "machine/*/attrib" (note the script is really just a nice wrapper around the routine zFile2MD5 which is part of my jLib.pm module that can be found on the wikki) #!/usr/bin/perl #= -*-perl-*- # # BackupPC_zfile2md5.pl: calculate and optionally verify the BackupPC-style #partial md5sum of any file compressed by BackupPC # # DESCRIPTION # This program allows you to calculate the partial md5sum # corresponding to the cpool path for any file that uses # BackupPC-style compression whether or not that file is actually # stored or linked to the cpool. Optionally, if the file is a cpool # entry or is linked to the cpool, you can add the '-k' flag to # verify whether the corresponding cpool path is consistent with the # actual md5sum of the file. # # Multiple files or directories can be given on the command line, # allowing you to calculate (and optionally verify) the md5sum for # multiple files or multiple trees of files. The script also does # path globbing using standard shell globbing conventions. # # Paths are assumed to be either absolute or relative to the current # directory unless, the options -C, -c, or -p are given in which # case the paths are understood to be a cpool file name (without # path), a path relative to the cpool, or a path relative to the # pc directory, respectively. # AUTHOR # Jeff Kosowsky # # COPYRIGHT # Copyright (C) 2009 Jeff Kosowsky # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA # # # # Version 1.0, released Dec 2009
