Re: [BackupPC-users] Splitting up large directories.
On Tue, May 18, 2010 at 09:30:43PM +, John Rouillard wrote:
> On Tue, May 18, 2010 at 02:04:46PM -0700, Robin Lee Powell wrote:
> > A customer we're backing up has a directory with ~500 subdirs
> > and hundreds of GiB of data. We're using BackupPC in rsync+ssh
> > mode.
> >
> > As a first pass at breaking that up, I made a bunch of seperate
> > host entries like /A/*0, /A/*1, ... (all the dirs have numeric
> > names).
> >
> > That seems to select the right files, but it doesn't work
> > because BackupPC ends up running a bunch of them at once,
> > hammering that customer's machine.
>
> You can set it up so only a few of those hosts will run at the
> same time using
>
> $Conf{UserCmdCheckStatus} = 1;
>
> and a $Conf{DumpPreUserCmd}/$Conf{DumpPostUserCmd} that know the
> host names and implement a counting semaphore to make sure only
> some number of them are running at the same time. I posted a
> longer sketch of how I limit the number of parallel backups to
> remote sites in the archives some time ago.
That's a fantastic idea! I don't even need to do anything
complicated; just use "lockfile /tmp/backuppc" OSLT, since I only
care about not overloading single hosts.
Thanks!
-Robin
--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Splitting up large directories.
$Conf{BackupPCNightlyPeriod} has no effect on how much of a client's
directory tree is backed up on each backup run; that's for something
totally different.
bandwidth isn't the problem; disk access is.
-Robin
On Tue, May 18, 2010 at 04:26:00PM -0500, Kameleon wrote:
> Why not just make backuppc take a week or so to traverse the
> entire pool using $Conf{BackupPCNightlyPeriod} = 7 or similar?
> Also you can use the rsync option to limit bandwidth so it won't
> kill their outbound connections using the --bwlimit=XX where XX is
> the speed in kB/sec to limit to.
>
> On Tue, May 18, 2010 at 4:04 PM, Robin Lee Powell <
> rlpow...@digitalkingdom.org> wrote:
>
> >
> > A customer we're backing up has a directory with ~500 subdirs and
> > hundreds of GiB of data. We're using BackupPC in rsync+ssh mode.
> >
> > As a first pass at breaking that up, I made a bunch of seperate host
> > entries like /A/*0, /A/*1, ... (all the dirs have numeric names).
> >
> > That seems to select the right files, but it doesn't work because
> > BackupPC ends up running a bunch of them at once, hammering that
> > customer's machine.
> >
> > I could make those into share names, but I'm worried about running
> > out of command-line argument space in that case; that is, that
> > "rsync /A/*0" will at some point in the future expand to a hundred
> > or more directories and break.
> >
> > What I want to do is have a bunch of shares like [ "/A", "/A", ...],
> > and have something like:
> >
> > $Conf{BackupFilesOnly} = {
> > "/A" => "/A/*0",
> > "/A" => "/A/*1",
> > "/A" => "/A/*2",
> > }
> >
> > But obviously that's not going to work.
> >
> > Does anyone have any other way to handle this?
> >
> > -Robin
> >
> > --
> > http://singinst.org/ : Our last, best hope for a fantastic future.
> > Lojban (http://www.lojban.org/): The language in which "this parrot
> > is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
> > is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
> >
> >
> > --
> >
> > ___
> > BackupPC-users mailing list
> > BackupPC-users@lists.sourceforge.net
> > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > Wiki:http://backuppc.wiki.sourceforge.net
> > Project: http://backuppc.sourceforge.net/
> >
> --
>
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Splitting up large directories.
On Tue, May 18, 2010 at 02:04:46PM -0700, Robin Lee Powell wrote:
> A customer we're backing up has a directory with ~500 subdirs and
> hundreds of GiB of data. We're using BackupPC in rsync+ssh mode.
>
> As a first pass at breaking that up, I made a bunch of seperate host
> entries like /A/*0, /A/*1, ... (all the dirs have numeric names).
>
> That seems to select the right files, but it doesn't work because
> BackupPC ends up running a bunch of them at once, hammering that
> customer's machine.
You can set it up so only a few of those hosts will run at the same
time using
$Conf{UserCmdCheckStatus} = 1;
and a $Conf{DumpPreUserCmd}/$Conf{DumpPostUserCmd} that know the host
names and implement a counting semaphore to make sure only some number
of them are running at the same time. I posted a longer sketch of how
I limit the number of parallel backups to remote sites in the archives
some time ago.
> I could make those into share names, but I'm worried about running
> out of command-line argument space in that case; that is, that
> "rsync /A/*0" will at some point in the future expand to a hundred
> or more directories and break.
>
> What I want to do is have a bunch of shares like [ "/A", "/A", ...],
> and have something like:
>
> $Conf{BackupFilesOnly} = {
> "/A" => "/A/*0",
> "/A" => "/A/*1",
> "/A" => "/A/*2",
> }
>
> But obviously that's not going to work.
>
> Does anyone have any other way to handle this?
How about a set of symbolic links on the host:
/backup_slices/A0
/backup_slices/A1
/backup_slices/A2
/backup_slices/A3
/backup_slices/A4
/backup_slices/A5
...
which all point to /A. Then a spec:
$Conf{BackupFilesOnly} = {
"/backup_slices/A0/." => "/*0",
"/backup_slices/A1/." => "/*1",
"/backup_slices/A2/." => "/*2",
...
}
Not pretty, but I think it will work as the trailing . should cause
rsync to dereference the symbolic link, and the resulting rsync will
start in /A.
--
-- rouilj
John Rouillard System Administrator
Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111
--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Splitting up large directories.
Why not just make backuppc take a week or so to traverse the entire pool
using $Conf{BackupPCNightlyPeriod} = 7 or similar? Also you can use the
rsync option to limit bandwidth so it won't kill their outbound connections
using the --bwlimit=XX where XX is the speed in kB/sec to limit to.
On Tue, May 18, 2010 at 4:04 PM, Robin Lee Powell <
rlpow...@digitalkingdom.org> wrote:
>
> A customer we're backing up has a directory with ~500 subdirs and
> hundreds of GiB of data. We're using BackupPC in rsync+ssh mode.
>
> As a first pass at breaking that up, I made a bunch of seperate host
> entries like /A/*0, /A/*1, ... (all the dirs have numeric names).
>
> That seems to select the right files, but it doesn't work because
> BackupPC ends up running a bunch of them at once, hammering that
> customer's machine.
>
> I could make those into share names, but I'm worried about running
> out of command-line argument space in that case; that is, that
> "rsync /A/*0" will at some point in the future expand to a hundred
> or more directories and break.
>
> What I want to do is have a bunch of shares like [ "/A", "/A", ...],
> and have something like:
>
> $Conf{BackupFilesOnly} = {
> "/A" => "/A/*0",
> "/A" => "/A/*1",
> "/A" => "/A/*2",
> }
>
> But obviously that's not going to work.
>
> Does anyone have any other way to handle this?
>
> -Robin
>
> --
> http://singinst.org/ : Our last, best hope for a fantastic future.
> Lojban (http://www.lojban.org/): The language in which "this parrot
> is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
> is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
>
>
> --
>
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
