We have a CMS that basically stores user data in a fs structure such as
/users/a/b/abraham/. Whenever a user edits one of their own files, the
webapp will touch a file in a specific location such as
/activeUsers/abraham. We use a predump script that quickly generates a
list of recently active user's paths such as:
./users/a/b/abraham
./users/b/r/brien
With tar, the "full" backup is straightforward (and time-consuming).
The incrementals, however, are fed the "-T" option for the filelist
(very fast). We quickly realized that our incrementals are not
"filled", presumably because tar is not creating the empty directory
tree for the rest of the filesystem (backupc thinks those directories
were deleted). We've really thrown a monkey wrench into the backup
semantics since not only are the incrementals based off the last full
(--newer), but the list (-T) will only include paths that had updates
since the last incremental (which could be hours).
SO, now we are installing backuppc 3.0beta for the multilevel
incrementals feature (very cool!!). We are hoping that we can use rsync
instead of tar so that the incrementals will be 'filled in" and such
(not to mention deleted/renamed files are detected). One problem is
there is no "RsyncFullArgs" and "RsyncIncArgs" with which we can trick
backuppc the same way we trick tar. Are we being nuts here? If this
even possible with Rsync?
Our "end goal" is to be able to take a Monthly Full, and then
incrementals every 2 hours (360 incrementals). By using the hints it is
very fast. (~10 mins per backup). Our motivation is the fact that we
have 8 million+ files and regular tar/rsync fulls or incrementals
literally take 15+ hours (and growing) to complete simply due to the
traversal of all these trees and checking stats on these small files
(generates unacceptable iowait as well).
Here is our tar config that works besides the lack of filled incs.
$Conf{TarClientCmd} = '$tarPath -c -v -f - -C /mnt/nfs'
. ' --totals';
$Conf{TarClientRestoreCmd} = '$tarPath -x -v -f - -C /mnt/nfs' #change
dir to where nfs data is mounted
. ' --totals';
$Conf{TarFullArgs} = './users'; #backup entire users folder
$Conf{TarIncrArgs} = '--newer=$incrDate -T
/var/lib/backuppc-meta/backuplist.txt #backup only files newer than
last full, but ALSO only in the backuplist.txt (webapp generated)
';
$Conf{TarShareName} = '/'; #this is arbitrary
$Conf{DumpPreUserCmd} =
'/usr/local/bin/predump.sh'; #generates
backuplist.txt
$Conf{DumpPostUserCmd} = '/usr/local/bin/postdump.sh $xferOK';
#removes backuplist.txt if the xfer was OK
Thanks for any advice!!
brien
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
