Re: [BackupPC-users] Exclude list being ignored?
On 10 October 2012 23:10, Ian P. Christian wrote: > > Running: /usr/bin/ssh -q -x -l root my.hostname.com /usr/bin/rsync > --server --sender --numeric-ids --perms --owner --group -D --links > --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 > --ignore-times . / > > Sent exclude: /proc > Sent exclude: /sys > > I've just noticed on this output there is a 'Sent exclude' - I was expecting to see this in the rsync arguments. Perhaps it is working after all! How does this exclude get processed if it's not in the rsync command args? Thanks -- Blog: http://pookey.co.uk/blog Follow me on twitter: http://twitter.com/ipchristian -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On Thu, Oct 11, 2012 at 4:30 AM, Ian P. Christian wrote: > >> >> Running: /usr/bin/ssh -q -x -l root my.hostname.com /usr/bin/rsync >> --server --sender --numeric-ids --perms --owner --group -D --links >> --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 >> --ignore-times . / > > >> >> Sent exclude: /proc >> Sent exclude: /sys >> > > I've just noticed on this output there is a 'Sent exclude' - I was expecting > to see this in the rsync arguments. Perhaps it is working after all! How > does this exclude get processed if it's not in the rsync command args? > That looked normal to me - are they not being excluded in the backup results? -- Les Mikesell lesmikes...@gmail.com -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On 11 October 2012 18:00, Les Mikesell wrote: > That looked normal to me - are they not being excluded in the backup > results? > During the effort to get this working, I changed a few things before posting, and it does appear to work. I just assumed it wasn't after not seeing it excluded from the rsync. If it's not being excluded from the rsync, where is it being excluded? If it's done only on the backuppc server side, does this not mean I'm still reading all the files on the client with rsync? Thanks, -- Blog: http://pookey.co.uk/blog Follow me on twitter: http://twitter.com/ipchristian -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On Thu, Oct 11, 2012 at 12:11 PM, Ian P. Christian wrote: > > > On 11 October 2012 18:00, Les Mikesell wrote: >> >> That looked normal to me - are they not being excluded in the backup >> results? > > > During the effort to get this working, I changed a few things before > posting, and it does appear to work. I just assumed it wasn't after not > seeing it excluded from the rsync. > > If it's not being excluded from the rsync, where is it being excluded? If > it's done only on the backuppc server side, does this not mean I'm still > reading all the files on the client with rsync? > The details are passed over the rsync connection protocol instead of the command line - as they would have to be if you were using rsyncd and chatting with remote rsync in daemon mode instead of passing a command line to sshd. -- Les Mikesell lesmikes...@gmail.com -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Steven writes:
> We are running backuppc 2.1.2 on a debian 4 host, backing up other
> debian hosts. Rsync version is 2.6.9 protocol version 29 on server and
> clients.
>
> I've set up backuppc on other systems and never run into this issue, but
> whenever we run backups the $Conf{BackupFilesExclude} setting is
> completely ignored and everything is backed up. Here's out backuppc config:
I'm a little suspicious of whether the sudo will cause a problem
with shell-meta characters - I'm not sure if an extra shell is
run by ssh in this case.
Try this:
- Try removing the '/var/lib/vmware/Virtual Machines...' excludes (the
ones with meta characters)
- Try replacing sudo with a full path:
$Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host sudo...';
In each case, you can run the backup manually with the -v flag to
see what rsync command is run, and then hit ^C to abort after you
see whether the excludes are working or not. Please post the rsync
command and whether it worked or not.
Craig
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
I changed the exclude list to:
$Conf{BackupFilesExclude} = {
'*' => [
'/proc',
'/sys',
'/tmp',
'/var/tmp',
'/usr/tmp',
'/mnt',
'/media',
'/auto',
'/var/run/acpid.socket',
'/var/run/dbus/system_bus_socket',
'/var/lib/backuppc/cpool',
'/var/lib/backuppc/pc',
'/var/lib/backuppc/pool',
'/var/lib/backuppc/trash',
#'/var/lib/vmware/Virtual Machines/*/*.vmdk',
#'/var/lib/vmware/Virtual Machines/*/*.vmem',
#'/var/lib/vmware/Virtual Machines/*/*.vmsd',
#'/var/lib/vmware/Virtual Machines/*/*.vmsn',
#'/var/lib/vmware/Virtual Machines/*/*.vmss',
'/backupdata'
]
};
And the rsync command to:
$Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host /usr/bin/sudo
$rsyncPath $argList+';
and ran the backup again. It did not work, sadly. Output:
-
Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com
/usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
--owner --group --links --times --block-size=2048 --recursive -D
--bwlimit=200 --ignore-times . /
Xfer PIDs are now 13052
xferPids 13052
Got remote protocol 29
Negotiated protocol version 26
Remote[2]: file has vanished: "/proc/2/exe"
Remote[2]: file has vanished: "/proc/2/task/2/exe"
Remote[2]: file has vanished: "/proc/3/exe"
Remote[2]: file has vanished: "/proc/3/task/3/exe"
Remote[2]: file has vanished: "/proc/4/exe"
Remote[2]: file has vanished: "/proc/4/task/4/exe"
Remote[2]: file has vanished: "/proc/5/exe"
Remote[2]: file has vanished: "/proc/5/task/5/exe"
Remote[2]: file has vanished: "/proc/6/exe"
Remote[2]: file has vanished: "/proc/6/task/6/exe"
Remote[2]: file has vanished: "/proc/7/exe"
Remote[2]: file has vanished: "/proc/7/task/7/exe"
Remote[2]: file has vanished: "/proc/11/exe"
Remote[2]: file has vanished: "/proc/11/task/11/exe"
Remote[2]: file has vanished: "/proc/14/exe"
Remote[2]: file has vanished: "/proc/14/task/14/exe"
Remote[2]: file has vanished: "/proc/16/exe"
Remote[2]: file has vanished: "/proc/16/task/16/exe"
Remote[2]: file has vanished: "/proc/92/exe"
Remote[2]: file has vanished: "/proc/92/task/92/exe"
Remote[2]: file has vanished: "/proc/93/exe"
Remote[2]: file has vanished: "/proc/93/task/93/exe"
Remote[2]: file has vanished: "/proc/94/exe"
Remote[2]: file has vanished: "/proc/94/task/94/exe"
Remote[2]: file has vanished: "/proc/95/exe"
Remote[2]: file has vanished: "/proc/95/task/95/exe"
Remote[2]: file has vanished: "/proc/601/exe"
Remote[2]: file has vanished: "/proc/601/task/601/exe"
Remote[2]: file has vanished: "/proc/602/exe"
Remote[2]: file has vanished: "/proc/602/task/602/exe"
Remote[2]: file has vanished: "/proc/746/exe"
Remote[2]: file has vanished: "/proc/746/task/746/exe"
Remote[2]: file has vanished: "/proc/747/exe"
Remote[2]: file has vanished: "/proc/747/task/747/exe"
Remote[2]: file has vanished: "/proc/748/exe"
Remote[2]: file has vanished: "/proc/748/task/748/exe"
Remote[2]: file has vanished: "/proc/749/exe"
Remote[2]: file has vanished: "/proc/749/task/749/exe"
Remote[2]: file has vanished: "/proc/750/exe"
Remote[2]: file has vanished: "/proc/750/task/750/exe"
Remote[2]: file has vanished: "/proc/751/exe"
Remote[2]: file has vanished: "/proc/751/task/751/exe"
Remote[2]: file has vanished: "/proc/995/exe"
Remote[2]: file has vanished: "/proc/995/task/995/exe"
Remote[2]: file has vanished: "/proc/999/exe"
Remote[2]: file has vanished: "/proc/999/task/999/exe"
Remote[2]: file has vanished: "/proc/1003/exe"
Remote[2]: file has vanished: "/proc/1003/task/1003/exe"
Remote[2]: file has vanished: "/proc/1032/exe"
Remote[2]: file has vanished: "/proc/1032/task/1032/exe"
Remote[2]: file has vanished: "/proc/1092/exe"
Remote[2]: file has vanished: "/proc/1092/task/1092/exe"
Remote[2]: file has vanished: "/proc/1673/exe"
Remote[2]: file has vanished: "/proc/1673/task/1673/exe"
Remote[2]: file has vanished: "/proc/1934/exe"
Remote[2]: file has vanished: "/proc/1934/task/1934/exe"
Remote[2]: file has vanished: "/proc/2819/exe"
Remote[2]: file has vanished: "/proc/2819/task/2819/exe"
Xfer PIDs are now 13052,13055
xferPids 13052,13055
create d 755 0/04096 .
create d 755 0/04096 .vmware
create d 755 0/04096 backupdata
same 644 0/01853 backupdata/backups
create d 755 0/04096 backupdata/cache
create d 755 0/04096 backupdata/cache/apt
create d 755 0/0 24576 backupdata/cache/apt/archives
same 444 0/0 179206
backupdata/cache/apt/archives/adduser_3.102
_all.deb
create 444 0/0 197976
backupdata/cache/apt/archives/apt-utils_0.6
.46.4-0.1_amd64.deb
Parent read EOF from child: fatal error!
Done: 0 files, 0 bytes
Got fatal error during xfer
Re: [BackupPC-users] Exclude list being ignored?
Steven writes:
> Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com
> /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
> --owner --group --links --times --block-size=2048 --recursive -D
> --bwlimit=200 --ignore-times . /
Yes, you can see none of the excludes make it into the command.
But you knew that.
The most likely explanation is there is a per-client config.pl that
is overriding the setting. Try editing $Conf{BackupFilesExclude}
and $Conf{RsyncClientCmd} in the per-client config.pl (make
$Conf{RsyncClientCmd} something different to the main config
file to make sure the file is correctly being parsed). Then
check the actual command being run.
Craig
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
mark k wrote:
> Here is how I have mine setup, note the "/' =>" at the top, backing
> up 30 + systems this way with nor errors.
>
> $Conf{BackupFilesExclude} = {
> '/' => [
> '/proc/*',
> '/sys/*',
> '/var/run/*',
> '/dev/*',
If you use '/' as the key, then these excludes only apply to machines
where you backup / (which may be all of your machines). Using '*'
means: use these excludes for all machines that don't have any
explicit excludes attached to them, which would have the same effect
if you backup / on all your machines.
We use the following with rsync over SSH and it works just fine:
$Conf{BackupFilesExclude} = {
'*' => [
'/proc',
'/var/named/run-root/proc',
'/sys',
'/mnt'
]
};
Nils Breunese.
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Here is how I have mine setup, note the "/' =>" at the top, backing up 30 +
systems this way with nor errors.
$Conf{BackupFilesExclude} = {
'/' => [
'/proc/*',
'/sys/*',
'/var/run/*',
'/dev/*',
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Craig Barratt wrote:
> Steven writes:
>
>
>> Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com
>> /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
>> --owner --group --links --times --block-size=2048 --recursive -D
>> --bwlimit=200 --ignore-times . /
>>
>
> Yes, you can see none of the excludes make it into the command.
> But you knew that.
>
> The most likely explanation is there is a per-client config.pl that
> is overriding the setting. Try editing $Conf{BackupFilesExclude}
> and $Conf{RsyncClientCmd} in the per-client config.pl (make
> $Conf{RsyncClientCmd} something different to the main config
> file to make sure the file is correctly being parsed). Then
> check the actual command being run.
>
> Craig
>
Yes, I know that seems like the likely answer, but unfortunately it
isn't. This happens on all of the hosts, regardless of whether or not
there are per host configuration files.
As an update, it works fine if I update the RsyncArgs directly, like so:
$Conf{RsyncArgs} = [
#
# Do not edit these!
#
'--numeric-ids',
'--perms',
'--owner',
'--group',
'--links',
'--times',
'--block-size=2048',
'--recursive',
#
# If you are using a patched client rsync that supports the
# --checksum-seed option (see http://backuppc.sourceforge.net),
# then uncomment this to enabled rsync checksum cachcing
#
#'--checksum-seed=32761',
#
# Add additional arguments here
#
'-D',
'--bwlimit=200',
'--exclude', '/proc',
'--exclude', '/sys',
'--exclude', '/tmp',
'--exclude', '/var/tmp',
'--exclude', '/usr/tmp',
'--exclude', '/mnt',
'--exclude', '/media',
'--exclude', '/auto',
'--exclude', '/var/run/acpid.socket',
'--exclude', '/var/run/dbus/system_bus_socket',
'--exclude', '/var/lib/backuppc/cpool',
'--exclude', '/var/lib/backuppc/pc',
'--exclude', '/var/lib/backuppc/pool',
'--exclude', '/var/lib/backuppc/trash',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmdk',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmem',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmsd',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmsn',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmss',
'--exclude', '/backupdata',
];
--
Puryear Information Technology, LLC
Baton Rouge, LA * 225-706-8414
http://www.puryear-it.com
Visit http://www.puryear-it.com/pubs/ebooks/ to download your free
copies of:
"Best Practices for Managing Linux and UNIX Servers"
"Spam Fighting and Email Security in the 21st Century"
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Steven writes:
> Yes, I know that seems like the likely answer, but unfortunately it
> isn't. This happens on all of the hosts, regardless of whether or not
> there are per host configuration files.
Well, if you are confident the likely explanation is wrong, I'd
recommend adding some debug statements to the code to figure
out why the value isn't being read.
Assuming you are running 3.x, in the ConfigDataRead function in
lib/BackupPC/Storage/Text.pm add the two print statements:
sub ConfigDataRead
{
my($s, $host) = @_;
my($ret, $mesg, $config, @configs);
#
# TODO: add lock
#
my $conf = {};
my $configPath = $s->ConfigPath($host);
push(@configs, $configPath) if ( -f $configPath );
foreach $config ( @configs ) {
%Conf = ();
if ( !defined($ret = do $config) && ($! || $@) ) {
$mesg = "Couldn't open $config: $!" if ( $! );
$mesg = "Couldn't execute $config: $@" if ( $@ );
$mesg =~ s/[\n\r]+//;
return ($mesg, $conf);
}
%$conf = ( %$conf, %Conf );
print(STDERR "Read config file $config;\nBackupFilesExclude is
now:"); # < add
print(STDERR Dumper($conf->{BackupFilesExclude}));
# < add
}
Then run
su - backuppc
BackupPC_dump -f -v HOSTNAME
and you should see it read each config file and the resulting BackupFilesExclude
value. Hit ^C after that.
Craig
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
