Hi, Mark,
I went back and read your Jan. 16th posting. Sounds like a very ambitious
piece of code... Some issues:
-- Because the --include option of rsync (or the include option of an
rsync module) is not exclusive, like $Conf{BackupFilesOnly} is, we'd have
to work with --exclude (as you said), listing everything we didn't want for
each set for a share, make each of those a separate module, and write them
to rsyncd.conf. This doesn't need to happen from BackupPC (although it
would be nice). It can run perhaps as a cron job on the client, or with
DumpPreUserCmd.
-- Maybe a simplified one would do as a proof of concept. I'm thinking it
would be easier to only balance number of files, not taking into account
sizes, memory availability on server and client, etc. And it would run in
two passes, one to list and chunk directories in sets, and a second one to
consolidate subtrees.
It seems doable, but certainly not trivial.
Assuming we have that rsyncd.conf nicely constructed, we'd then have to
find out the list of modules that applies to the share to be backed up.
Something like
rsync <host>::
would return, say,
shareA_set1
shareA_set2
shareB_set1
shareB_set2
and then it would have to be filtered for the relevant modules (assuming
they are named sensibly). This could be done at DumpPreUserCmd too. So
let's say we are backing up share B: the relevant modules are shareB_set1
and shareB_set2.
Next, we need to call rsync with that filtered list of modules. Could it be
something like this?:
$Conf{XferMethod} = 'rsyncd';
$Conf{DumpPreUserCmd} = <magical command that constructs rsync.conf,
and sets a variable>;
$Conf{RsyncShareName} = ['shareB_set1', 'shareB_set2'];
In particular, I don't know how BackupPC connects to an rsync daemon on the
client (I assume it doesn't use the $Conf{RsyncClientCmd} because the
comments there say "This setting only matters if $Conf{XferMethod} =
'rsync'"). And can the DumpPre... set a config variable, i.e.,
$Conf{RsyncShareName};?
Or would something like this work, which should be executed when the config
file is read? I'm assuming BackupPC rereads/require's the per-pc config
file before every dump:
$Conf{XferMethod} = 'rsyncd';
$Conf{DumpPreUserCmd} = <regular command not related to rsyncd>;
sub get_modules {
my($host, $share) = @_;
[...] # Magic happens here, and @modules is defined.
# This sub could conceivably also construct the rsync.conf
on the client.
return @modules;
}
$Conf{RsyncShareName} = [ get_modules($host, $share) ];
And one more question: after all this, is each module backed up
independently or are all used at once? The concern here, after all, is
memory. If each module was done independently (whether in one or multiple
connections) then we'd be limiting memory usage, but if not, we are back
where we were...
Too many questions, sorry; enough for today.
Bernardo
[EMAIL PROTECTED]
Sent by: To: [EMAIL
PROTECTED]
[EMAIL PROTECTED] cc:
backuppc-users@lists.sourceforge.net
ceforge.net bcc:
Subject: Re:
[BackupPC-users] New user, few questions
01/25/2006 10:51 AM
Please respond to backuppc
In the message dated: Tue, 24 Jan 2006 19:16:26 EST,
The pithy ruminations from [EMAIL PROTECTED] on
<Re: [BackupPC-users] New user, few questions> were:
=>
=> Dan,
=>
=>
=> > Alternately, just break your backup set into two (or more) pieces.
=>
=> Ah, I haven't tried that yet... There are a couple of reasons why:
=>
=> -- even though I'm backing up only one machine, I have one "client"
per
=> backed up dir ("share"?, I'm using rsync), because I want to invoke the
=> creation of a single LVM snapshot at a time with $Conf{DumpPreUserCmd}
(I
=> don't want more than one snapshot simultaneously on the client, at least
=> for now, for performance and space reasons).
=>
=> -- subdirectories below each backed up dir can change, so I can't use
a
=> static list, lest I be forced to maintain it (I'm lazy, I know).
=>
=> But now you got me thinking... there might be ways to do it. For
example, I
=> know that in the home partition there is one user that takes as much
space
=> as everybody else together. So that would be a good candidate to break
that
=> backup set into two nearly-equal-size pieces.
=>
This sounds a great deal like what I proposed on Jan 16, describing a
suggestion to enhance backuppc to dynamically create backup sets of small
sizes.
I saw much the same problems as you, with long back times, and the burden
of
maintaining different backup sets in the face of changing disk usage.
Check the archive for the post entitled "rsync dynamic backup set
partitioning
enhancement proposal".
=> This is what I have currently:
=>
=> $Conf{RsyncShareName} = '/home_snapshot';
=>
=> Would something like this work?:
=>
=> $Conf{RsyncShareName} = '/home_snapshot';
=>
=> $Conf{BackupFilesOnly} = {
=> '/home_snapshot' => [ '/<large_user_home_dir>' ],
=> '/home_snapshot' => [ some grep-like operation that lists all home
=> directories, but excludes /<large_user_home_dir> ],
=> };
=>
=>
=> Because the config files are Perl scripts, the second list can be
generated
=> dynamically. I'd have to ssh to the client, ls /home_snapshot, filter
=> <large_user...> out from that directory listing... looks doable... Argh!
Exactly. I think this should be built into backuppc...
=> that wouldn't fly, because two hash entries with the same key means the
=> second one would overwrite the first one. Hmm, but something along those
Not a problem, if you consider the "hash" to be the full path.
=> lines... Is it possible to have arrays of arrays?:
Yes, perl conceptually supports arrays of arrays. I don't know if that
would
work w/in backuppc as it is now.
=>
=> $Conf{BackupFilesOnly} = {
=> '/home_snapshot' => [
=> [ '/<large_user_home_dir>' ],
=> [ some grep-like operation that lists all
home
=> directories, but excludes /<large_user_home_dir> ]
=> ],
=> };
=>
=>
=> In which case I could just as well do this:
=>
=> $Conf{RsyncShareName} = [
=> [ '/home_snapshot/<large_user_home_dir>' ],
=> [ /home_snapshot prefixed to each element of the filtered list of
=> home dirs ]
=> ];
=>
=>
=>
=> I bet there is some much simpler way of doing this, and I just have the
=> blinders on.
Yes. Use the "exclude" options in the client-side rsyncd.conf. For example,
my
backuppc server connects to a number of targets (in rsync terminology, a
"module") on the clients. The clients then define what directories are
included/
excluded in each target. In theory, the rsyncd.conf file could be build
dynamically. Here's a snippet of an rsyncd.conf on a backup client:
[justroot]
path = /
exclude = boot cdrom home opt tmp usr usr/local var win98 proc sys
mnt/*/*
[usr]
path = /usr
exclude = usr/local/
[usr_local]
path = /usr/local
The backuppc server simply connects to the client modules "justroot",
"usr",
"usr_local", and so on, without maintaining the exclude lists on the
server.
Each of these targets, corresponding to mount points, could easily be
generated
dynamically.
=>
=>
=> > cat5e should work fine for gigabit (it is the specified cable for
=> > 1000baseT). You don't need cat6.
=>
=> Thanks for correcting me on cat 5e vs cat6. That's good news for me.
=>
=>
=> Bernardo Rechea
=>
Mark
----
Mark Bergman
[EMAIL PROTECTED]
Seeking a Unix/Linux sysadmin position local to Philadelphia or via
telecommuting
http://wwwkeys.pgp.net:11371/pks/lookup?op=get&search=bergman%40merctech.com
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log
files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
