We have a CMS that basically stores user data in a fs structure such as 
/users/a/b/abraham/.  Whenever a user edits one of their own files,  the 
webapp will touch a file in a specific location such as 
/activeUsers/abraham.  We use a predump script that quickly generates a 
list of recently active user's paths such as:
./users/a/b/abraham
./users/b/r/brien

With tar, the "full" backup is straightforward (and time-consuming).  
The incrementals, however, are fed the "-T" option for the filelist 
(very fast).  We quickly realized that our incrementals are not 
"filled", presumably because tar is not creating the empty directory 
tree for the rest of the filesystem (backupc thinks those directories 
were deleted).  We've really thrown a monkey wrench into the backup 
semantics since  not only are the incrementals based off the last full 
(--newer), but the list (-T) will only include paths that had updates 
since the last incremental (which could be hours).

SO, now we are installing backuppc 3.0beta for the multilevel 
incrementals feature (very cool!!).  We are hoping that we can use rsync 
instead of tar so that the incrementals will be 'filled in" and such 
(not to mention deleted/renamed files are detected).  One problem is 
there is no "RsyncFullArgs" and "RsyncIncArgs" with which we can trick 
backuppc the same way we trick tar.  Are we being nuts here?  If this 
even possible with Rsync?

Our "end goal" is to be able to take a Monthly Full, and then 
incrementals every 2 hours (360 incrementals).  By using the hints it is 
very fast. (~10 mins per backup).  Our motivation is the fact that we 
have 8 million+ files and regular tar/rsync fulls or incrementals 
literally take 15+ hours (and growing) to complete simply due to the 
traversal of all these trees and checking stats on these small files 
(generates unacceptable iowait as well).

Here is our tar config that works besides the lack of filled incs.

$Conf{TarClientCmd} = '$tarPath -c -v -f - -C /mnt/nfs'
                           . ' --totals';

$Conf{TarClientRestoreCmd} = '$tarPath -x -v -f - -C /mnt/nfs'  #change 
dir to where nfs data is mounted
                           . ' --totals';

$Conf{TarFullArgs} = './users';  #backup entire users folder

$Conf{TarIncrArgs} = '--newer=$incrDate -T 
/var/lib/backuppc-meta/backuplist.txt   #backup only files newer than 
last full, but ALSO only in the backuplist.txt (webapp generated)
';
$Conf{TarShareName} = '/';        #this is arbitrary

$Conf{DumpPreUserCmd} = 
'/usr/local/bin/predump.sh';                        #generates 
backuplist.txt
$Conf{DumpPostUserCmd} = '/usr/local/bin/postdump.sh $xferOK';      
#removes backuplist.txt if the xfer was OK


Thanks for any advice!!

brien

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/

Reply via email to