Re: [BackupPC-users] Exclude list being ignored?
On 10 October 2012 23:10, Ian P. Christian wrote: > > Running: /usr/bin/ssh -q -x -l root my.hostname.com /usr/bin/rsync > --server --sender --numeric-ids --perms --owner --group -D --links > --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 > --ignore-times . / > > Sent exclude: /proc > Sent exclude: /sys > > I've just noticed on this output there is a 'Sent exclude' - I was expecting to see this in the rsync arguments. Perhaps it is working after all! How does this exclude get processed if it's not in the rsync command args? Thanks -- Blog: http://pookey.co.uk/blog Follow me on twitter: http://twitter.com/ipchristian -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On Thu, Oct 11, 2012 at 4:30 AM, Ian P. Christian wrote: > >> >> Running: /usr/bin/ssh -q -x -l root my.hostname.com /usr/bin/rsync >> --server --sender --numeric-ids --perms --owner --group -D --links >> --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 >> --ignore-times . / > > >> >> Sent exclude: /proc >> Sent exclude: /sys >> > > I've just noticed on this output there is a 'Sent exclude' - I was expecting > to see this in the rsync arguments. Perhaps it is working after all! How > does this exclude get processed if it's not in the rsync command args? > That looked normal to me - are they not being excluded in the backup results? -- Les Mikesell lesmikes...@gmail.com -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On 11 October 2012 18:00, Les Mikesell wrote: > That looked normal to me - are they not being excluded in the backup > results? > During the effort to get this working, I changed a few things before posting, and it does appear to work. I just assumed it wasn't after not seeing it excluded from the rsync. If it's not being excluded from the rsync, where is it being excluded? If it's done only on the backuppc server side, does this not mean I'm still reading all the files on the client with rsync? Thanks, -- Blog: http://pookey.co.uk/blog Follow me on twitter: http://twitter.com/ipchristian -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On Thu, Oct 11, 2012 at 12:11 PM, Ian P. Christian wrote: > > > On 11 October 2012 18:00, Les Mikesell wrote: >> >> That looked normal to me - are they not being excluded in the backup >> results? > > > During the effort to get this working, I changed a few things before > posting, and it does appear to work. I just assumed it wasn't after not > seeing it excluded from the rsync. > > If it's not being excluded from the rsync, where is it being excluded? If > it's done only on the backuppc server side, does this not mean I'm still > reading all the files on the client with rsync? > The details are passed over the rsync connection protocol instead of the command line - as they would have to be if you were using rsyncd and chatting with remote rsync in daemon mode instead of passing a command line to sshd. -- Les Mikesell lesmikes...@gmail.com -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Steven writes: > We are running backuppc 2.1.2 on a debian 4 host, backing up other > debian hosts. Rsync version is 2.6.9 protocol version 29 on server and > clients. > > I've set up backuppc on other systems and never run into this issue, but > whenever we run backups the $Conf{BackupFilesExclude} setting is > completely ignored and everything is backed up. Here's out backuppc config: I'm a little suspicious of whether the sudo will cause a problem with shell-meta characters - I'm not sure if an extra shell is run by ssh in this case. Try this: - Try removing the '/var/lib/vmware/Virtual Machines...' excludes (the ones with meta characters) - Try replacing sudo with a full path: $Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host sudo...'; In each case, you can run the backup manually with the -v flag to see what rsync command is run, and then hit ^C to abort after you see whether the excludes are working or not. Please post the rsync command and whether it worked or not. Craig - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
I changed the exclude list to: $Conf{BackupFilesExclude} = { '*' => [ '/proc', '/sys', '/tmp', '/var/tmp', '/usr/tmp', '/mnt', '/media', '/auto', '/var/run/acpid.socket', '/var/run/dbus/system_bus_socket', '/var/lib/backuppc/cpool', '/var/lib/backuppc/pc', '/var/lib/backuppc/pool', '/var/lib/backuppc/trash', #'/var/lib/vmware/Virtual Machines/*/*.vmdk', #'/var/lib/vmware/Virtual Machines/*/*.vmem', #'/var/lib/vmware/Virtual Machines/*/*.vmsd', #'/var/lib/vmware/Virtual Machines/*/*.vmsn', #'/var/lib/vmware/Virtual Machines/*/*.vmss', '/backupdata' ] }; And the rsync command to: $Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host /usr/bin/sudo $rsyncPath $argList+'; and ran the backup again. It did not work, sadly. Output: - Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group --links --times --block-size=2048 --recursive -D --bwlimit=200 --ignore-times . / Xfer PIDs are now 13052 xferPids 13052 Got remote protocol 29 Negotiated protocol version 26 Remote[2]: file has vanished: "/proc/2/exe" Remote[2]: file has vanished: "/proc/2/task/2/exe" Remote[2]: file has vanished: "/proc/3/exe" Remote[2]: file has vanished: "/proc/3/task/3/exe" Remote[2]: file has vanished: "/proc/4/exe" Remote[2]: file has vanished: "/proc/4/task/4/exe" Remote[2]: file has vanished: "/proc/5/exe" Remote[2]: file has vanished: "/proc/5/task/5/exe" Remote[2]: file has vanished: "/proc/6/exe" Remote[2]: file has vanished: "/proc/6/task/6/exe" Remote[2]: file has vanished: "/proc/7/exe" Remote[2]: file has vanished: "/proc/7/task/7/exe" Remote[2]: file has vanished: "/proc/11/exe" Remote[2]: file has vanished: "/proc/11/task/11/exe" Remote[2]: file has vanished: "/proc/14/exe" Remote[2]: file has vanished: "/proc/14/task/14/exe" Remote[2]: file has vanished: "/proc/16/exe" Remote[2]: file has vanished: "/proc/16/task/16/exe" Remote[2]: file has vanished: "/proc/92/exe" Remote[2]: file has vanished: "/proc/92/task/92/exe" Remote[2]: file has vanished: "/proc/93/exe" Remote[2]: file has vanished: "/proc/93/task/93/exe" Remote[2]: file has vanished: "/proc/94/exe" Remote[2]: file has vanished: "/proc/94/task/94/exe" Remote[2]: file has vanished: "/proc/95/exe" Remote[2]: file has vanished: "/proc/95/task/95/exe" Remote[2]: file has vanished: "/proc/601/exe" Remote[2]: file has vanished: "/proc/601/task/601/exe" Remote[2]: file has vanished: "/proc/602/exe" Remote[2]: file has vanished: "/proc/602/task/602/exe" Remote[2]: file has vanished: "/proc/746/exe" Remote[2]: file has vanished: "/proc/746/task/746/exe" Remote[2]: file has vanished: "/proc/747/exe" Remote[2]: file has vanished: "/proc/747/task/747/exe" Remote[2]: file has vanished: "/proc/748/exe" Remote[2]: file has vanished: "/proc/748/task/748/exe" Remote[2]: file has vanished: "/proc/749/exe" Remote[2]: file has vanished: "/proc/749/task/749/exe" Remote[2]: file has vanished: "/proc/750/exe" Remote[2]: file has vanished: "/proc/750/task/750/exe" Remote[2]: file has vanished: "/proc/751/exe" Remote[2]: file has vanished: "/proc/751/task/751/exe" Remote[2]: file has vanished: "/proc/995/exe" Remote[2]: file has vanished: "/proc/995/task/995/exe" Remote[2]: file has vanished: "/proc/999/exe" Remote[2]: file has vanished: "/proc/999/task/999/exe" Remote[2]: file has vanished: "/proc/1003/exe" Remote[2]: file has vanished: "/proc/1003/task/1003/exe" Remote[2]: file has vanished: "/proc/1032/exe" Remote[2]: file has vanished: "/proc/1032/task/1032/exe" Remote[2]: file has vanished: "/proc/1092/exe" Remote[2]: file has vanished: "/proc/1092/task/1092/exe" Remote[2]: file has vanished: "/proc/1673/exe" Remote[2]: file has vanished: "/proc/1673/task/1673/exe" Remote[2]: file has vanished: "/proc/1934/exe" Remote[2]: file has vanished: "/proc/1934/task/1934/exe" Remote[2]: file has vanished: "/proc/2819/exe" Remote[2]: file has vanished: "/proc/2819/task/2819/exe" Xfer PIDs are now 13052,13055 xferPids 13052,13055 create d 755 0/04096 . create d 755 0/04096 .vmware create d 755 0/04096 backupdata same 644 0/01853 backupdata/backups create d 755 0/04096 backupdata/cache create d 755 0/04096 backupdata/cache/apt create d 755 0/0 24576 backupdata/cache/apt/archives same 444 0/0 179206 backupdata/cache/apt/archives/adduser_3.102 _all.deb create 444 0/0 197976 backupdata/cache/apt/archives/apt-utils_0.6 .46.4-0.1_amd64.deb Parent read EOF from child: fatal error! Done: 0 files, 0 bytes Got fatal error during xfer
Re: [BackupPC-users] Exclude list being ignored?
Steven writes: > Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com > /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms > --owner --group --links --times --block-size=2048 --recursive -D > --bwlimit=200 --ignore-times . / Yes, you can see none of the excludes make it into the command. But you knew that. The most likely explanation is there is a per-client config.pl that is overriding the setting. Try editing $Conf{BackupFilesExclude} and $Conf{RsyncClientCmd} in the per-client config.pl (make $Conf{RsyncClientCmd} something different to the main config file to make sure the file is correctly being parsed). Then check the actual command being run. Craig - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
mark k wrote: > Here is how I have mine setup, note the "/' =>" at the top, backing > up 30 + systems this way with nor errors. > > $Conf{BackupFilesExclude} = { > '/' => [ > '/proc/*', > '/sys/*', > '/var/run/*', > '/dev/*', If you use '/' as the key, then these excludes only apply to machines where you backup / (which may be all of your machines). Using '*' means: use these excludes for all machines that don't have any explicit excludes attached to them, which would have the same effect if you backup / on all your machines. We use the following with rsync over SSH and it works just fine: $Conf{BackupFilesExclude} = { '*' => [ '/proc', '/var/named/run-root/proc', '/sys', '/mnt' ] }; Nils Breunese. - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Here is how I have mine setup, note the "/' =>" at the top, backing up 30 + systems this way with nor errors. $Conf{BackupFilesExclude} = { '/' => [ '/proc/*', '/sys/*', '/var/run/*', '/dev/*', - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Craig Barratt wrote: > Steven writes: > > >> Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com >> /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms >> --owner --group --links --times --block-size=2048 --recursive -D >> --bwlimit=200 --ignore-times . / >> > > Yes, you can see none of the excludes make it into the command. > But you knew that. > > The most likely explanation is there is a per-client config.pl that > is overriding the setting. Try editing $Conf{BackupFilesExclude} > and $Conf{RsyncClientCmd} in the per-client config.pl (make > $Conf{RsyncClientCmd} something different to the main config > file to make sure the file is correctly being parsed). Then > check the actual command being run. > > Craig > Yes, I know that seems like the likely answer, but unfortunately it isn't. This happens on all of the hosts, regardless of whether or not there are per host configuration files. As an update, it works fine if I update the RsyncArgs directly, like so: $Conf{RsyncArgs} = [ # # Do not edit these! # '--numeric-ids', '--perms', '--owner', '--group', '--links', '--times', '--block-size=2048', '--recursive', # # If you are using a patched client rsync that supports the # --checksum-seed option (see http://backuppc.sourceforge.net), # then uncomment this to enabled rsync checksum cachcing # #'--checksum-seed=32761', # # Add additional arguments here # '-D', '--bwlimit=200', '--exclude', '/proc', '--exclude', '/sys', '--exclude', '/tmp', '--exclude', '/var/tmp', '--exclude', '/usr/tmp', '--exclude', '/mnt', '--exclude', '/media', '--exclude', '/auto', '--exclude', '/var/run/acpid.socket', '--exclude', '/var/run/dbus/system_bus_socket', '--exclude', '/var/lib/backuppc/cpool', '--exclude', '/var/lib/backuppc/pc', '--exclude', '/var/lib/backuppc/pool', '--exclude', '/var/lib/backuppc/trash', '--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmdk', '--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmem', '--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmsd', '--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmsn', '--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmss', '--exclude', '/backupdata', ]; -- Puryear Information Technology, LLC Baton Rouge, LA * 225-706-8414 http://www.puryear-it.com Visit http://www.puryear-it.com/pubs/ebooks/ to download your free copies of: "Best Practices for Managing Linux and UNIX Servers" "Spam Fighting and Email Security in the 21st Century" - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Steven writes: > Yes, I know that seems like the likely answer, but unfortunately it > isn't. This happens on all of the hosts, regardless of whether or not > there are per host configuration files. Well, if you are confident the likely explanation is wrong, I'd recommend adding some debug statements to the code to figure out why the value isn't being read. Assuming you are running 3.x, in the ConfigDataRead function in lib/BackupPC/Storage/Text.pm add the two print statements: sub ConfigDataRead { my($s, $host) = @_; my($ret, $mesg, $config, @configs); # # TODO: add lock # my $conf = {}; my $configPath = $s->ConfigPath($host); push(@configs, $configPath) if ( -f $configPath ); foreach $config ( @configs ) { %Conf = (); if ( !defined($ret = do $config) && ($! || $@) ) { $mesg = "Couldn't open $config: $!" if ( $! ); $mesg = "Couldn't execute $config: $@" if ( $@ ); $mesg =~ s/[\n\r]+//; return ($mesg, $conf); } %$conf = ( %$conf, %Conf ); print(STDERR "Read config file $config;\nBackupFilesExclude is now:"); # < add print(STDERR Dumper($conf->{BackupFilesExclude})); # < add } Then run su - backuppc BackupPC_dump -f -v HOSTNAME and you should see it read each config file and the resulting BackupFilesExclude value. Hit ^C after that. Craig - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/