David Rees schrieb: > On Mon, Feb 25, 2008 at 1:23 AM, Tomasz Chmielewski <[EMAIL PROTECTED]> wrote: >> Unfortunately, it doesn't scale very well in terms of performance - you >> may see this thread on linux-fsdevel list for more info: >> http://marc.info/?t=120333985100003&r=2&w=4 > > What version of BackupPC? 3.1.0 does the inode sorting in the nightly > process as described in the thread if you have IO::Dirent installed > (check that's installed, too).
Yes, I have 3.1.0. But before, when the system was smaller, it was 3.0.0, and perhaps earlier versions of BackupPC, too. But I didn't have IO::Dirent installed, thanks for the hint. Let's hope the list of directories in "trash" will keep decreasing now. Right now, I have almost 100 directories there, and it is growing each day a bit. Although - with IO::Dirent "wa" is now 100% almost all the time, and the system feels much slower. Hm. Let's hope it's coincidence, and assume the system was just committing a big write... >> The main problem seems to be hard disk seeks caused by a great amount of >> hardlinks. I.e., removing anything from the drive takes ages. > > The only way you can really reduce the cost of all these seeks (unless > you can find a filesystem better than ext3 which handles this type of > situation better) is to try to limit head movement on the spindles. > This would mean partitioning your disks and using a smaller portion of > the disk for the BackupPC data partition. Or adding more spindles with > higher rotation speeds and lower seek latencies. Yeah, or SSD disks which don't have seek problem. Or even better, use RAM-based storage devices with built-in battery backup - not only you have no seek problems, they are fast, too (and very, very expensive). Unfortunately, changing hardware is not an option at this moment. > I did find it odd that you said after dropping the caches deletes ran > fairly quickly for a while, then slowed down. This seems to point to > some sort of kernel bug. No, actually, everything is fine here. To remove anything, kernel has to read directory structures etc. first. Then, it can "remove" lots of files really fast without committing anything to the disk (yet) - everything stays in memory. Once kernel feels the pressure on memory, it will start committing the changes (file removals) to the disk. This is where seek problem leads to poor performance: write commits compete with reads, so there is lots of seek. I managed to get around this a bit by increasing commit= ext3 mount option, and also, setting these values (and putting 3 GB of RAM into the machine): echo 50 > /proc/sys/vm/dirty_ratio echo 50 > /proc/sys/vm/dirty_background_ratio echo 6000 > /proc/sys/vm/dirty_writeback_centisecs echo 6000 > /proc/sys/vm/dirty_expire_centisecs I guess enabling write cache would help a lot, as then, the storage could commit the changes out of order (less seek) - but I don't trust it much. > Some people have said that perhaps reiserfs or xfs may perform better > than ext3 for this type of workload, but I don't know of any real > benchmarks made with a BackupPC type workload which is fairly unusual. > > I assume that you've already mounted the filesystem with noatime? Yep. When the filesystem was still much slower, I didn't have this option enabled. After enabling it, all nightly tasks and file removals were done 20-50% faster, so it was noticeable gain. -- Tomasz Chmielewski http://wpkg.org ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ BackupPC-users mailing list [email protected] List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
