Hi, On Friday 17 February 2006 07:53, Craig Barratt wrote: > David Brown writes: > > I've been using backuppc for several days, and I really like the concept > > behind it. The web interface is very helpful. However, I'm having a > > very hard time figuring out what to store the backup filesystem on. > > > > I've tried both XFS and ReiserFS, and both have utterly abysmal > > performance in the backup tree. The problem has to do with the > > hardlinked trees. > > > > - Most filesystems optimize directories by using inodes that are stored > > near one another for files in the same directory. This allows access > > to files in the same directory to be localized on the disk.
I've tried a lot of filesystems with backuppc and I've run across the same things you have. I've stuck with reiserfs (version 3) because it was the "least of all evils" (that's quite a literal translation from a Dutch proverb, I hope you understand what I mean). Ext3 bogged down completely when the amount of files started to get larger (no dir_index), JFS was also slow and had a memory leak when I tried it, XFS worked ok, but then I had trouble with my hardware raid and had to rebuild the filesystem using the xfs repair tools and that just didn't work. No such experience yet with reiser, so I stuck with that. That being said: as you are still testing if I understand your mail correctly, could you do me a favor and do a test with ext3 with dir_index and -T news? Dir_index doesn't provide you with an advantage in the general case (if I read the benchmarks published all over the internet correctly), but it may work here. I sadly don't have enough spare hardware to build a serious testmachine. I would much rather use ext3 if I can than a "special" filesystem, for all kinds of reasons. I would also like to see how reiser 4 performs, but as far as I know, that's still in a state of flux (and still not added to the standard kernel source), so I'm a bit reluctant to let it have control of my backups. But if someone has experience with it on backuppc, please tell me about it :) > > - BackupPC creates the files in the backup directory, and then > > hardlinks them, by hash, into the pool. This means that each of the > > entries in a pool directory has an inode (and data) on a diverse part of > > the disk. Just statting the files in a pool directory is very slow. 'du' > > of the pool directory takes several hours on any filesystem I've tried it > > on. I don't think ext3 with dir_index will be a "miracle fs", but I'm rather curious how it behaves in this situation. > > - Other than the first backup directory, the backup directories aren't > > much better, since most of the files are hardlinks back to the pool. > > You're exactly right. A major performance limitation of BackupPC > is that backup directories tend to have widely dispersed inodes. > Yes, just stat()ing files in a single directory involves lots of > disk seeks. > > A custom BackupPCd client is being developed, and once it is > ready I'm curious to see if sorting readdir contents by inode > number on the server will help the performance. It worked wonders for the nightly runs. I used to run the version made by someone who ordered the files by inode and that worked fine. I only stopped using it because I had to tinker with it every time backuppc was upgraded and the newer versions of backuppc don't have to process the whole (c)pool in one go. So I let backuppc only do a small portion each night, which solves my problem just the same. Regards, Guus Houtzager ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
