On Tue, May 18, 2010 at 02:04:46PM -0700, Robin Lee Powell wrote: > A customer we're backing up has a directory with ~500 subdirs and > hundreds of GiB of data. We're using BackupPC in rsync+ssh mode. > > As a first pass at breaking that up, I made a bunch of seperate host > entries like /A/*0, /A/*1, ... (all the dirs have numeric names). > > That seems to select the right files, but it doesn't work because > BackupPC ends up running a bunch of them at once, hammering that > customer's machine.
You can set it up so only a few of those hosts will run at the same time using $Conf{UserCmdCheckStatus} = 1; and a $Conf{DumpPreUserCmd}/$Conf{DumpPostUserCmd} that know the host names and implement a counting semaphore to make sure only some number of them are running at the same time. I posted a longer sketch of how I limit the number of parallel backups to remote sites in the archives some time ago. > I could make those into share names, but I'm worried about running > out of command-line argument space in that case; that is, that > "rsync /A/*0" will at some point in the future expand to a hundred > or more directories and break. > > What I want to do is have a bunch of shares like [ "/A", "/A", ...], > and have something like: > > $Conf{BackupFilesOnly} = { > "/A" => "/A/*0", > "/A" => "/A/*1", > "/A" => "/A/*2", > } > > But obviously that's not going to work. > > Does anyone have any other way to handle this? How about a set of symbolic links on the host: /backup_slices/A0 /backup_slices/A1 /backup_slices/A2 /backup_slices/A3 /backup_slices/A4 /backup_slices/A5 ... which all point to /A. Then a spec: $Conf{BackupFilesOnly} = { "/backup_slices/A0/." => "/*0", "/backup_slices/A1/." => "/*1", "/backup_slices/A2/." => "/*2", ... } Not pretty, but I think it will work as the trailing . should cause rsync to dereference the symbolic link, and the resulting rsync will start in /A. -- -- rouilj John Rouillard System Administrator Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111 ------------------------------------------------------------------------------ _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/