On Fri, 2007-08-24 at 09:52 +1200, Darryl Dixon - Winterhouse Consulting wrote: > > Hi > > > > I have a file system that contains millions of small files. Since I > > backup it everyday with rsync using slow WAN link, I think it will be > > nice that if rsync can do this: > > > > An option that let rsync only check with remote rsync daemon about local > > files that has last modification time newer than one day ago (so is > > modified since yesterday backup). This can greatly reduce the WAN > > traffic. > > > > Is this doable with current rsync? > > > > Hi Ming, List, > > I thought I'd reply as I have used rsync in a similar scenario (~1TB of 13 > million files in two filesystems backed up offsite). > > There are a couple of approaches that will do what you want - what OS are > you using? (Windows, Solaris, Linux ...?). One is to run 'find -mtime -1 >
Linux, > my_files.list' and then use the rsync --files-from=my_files.list to send > only the new files. Running find can be time consuming(!), but effectively > that's what you'd be doing with an 'rsync -mtime -1' option anyway. rsync has such option? i do not know. > Another option (and this is the one that I used) is to audit filesystem > events as they are happenining, and keep a live list of all modified files > all the time. This list can then be fed to rsync with the --files-from > option. > > On Solaris this can be achieved with the BSM module and NFS logging (if > you're running and NFS server). On Linux I heavily modified pyinotify > (http://pyinotify.sourceforge.net) to achieve the same result. The outcome > is that every 5 minutes during the day I ship all the files changed in the > previous 5 minutes offsite to the backup server. This works perfectly - > and the volume of change is about 30,000 new files per day! this sounds cool. i will look into this inotify. my last look at inotify give me an impression that it can not scalable enough to observe a file system with multiple million files. maybe i am seriously wrong. just curious, how you deal with file/directory that were deleted or renamed. > > Email me direct if you want more details :) > > regards, > Darryl Dixon > Winterhouse Consulting Ltd > http://www.winterhouseconsulting.com > [EMAIL PROTECTED] > -- Ming Zhang @#$%^ purging memory... (*!% http://blackmagic02881.wordpress.com/ http://www.linkedin.com/in/blackmagic02881 -------------------------------------------- -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html