On Thu, Mar 20, 2014 at 12:21 PM, Timothy J Massey <tmas...@obscorp.com> wrote: > > > > > You've used BackupPC before, Les, right? ;-) > > BackupPC prefers pool reads over writes when possible, and it typically > > accesses large amounts of data almost randomly. Caching metadata will help, > > caching data likely won't. > > > I've given up explaining this to Mr. Mikesell. I've posted info to the list > several times showing the performance on BackupPC servers that had 512MB > (Megabytes) of RAM. Zero swapping, and something like 300MB used for > caching. I have then upgraded them to 4GB of RAM (8 times as much RAM!) and > saw a whopping 15 or so minutes savings on a backup that took 10 to 12 hours. > (In fact, the only real reason I use 4GB minimum across the board now is > that I ran into a problem where a fsck wouldn't complete without more RAM.)
4GB is still a tiny amount of RAM these days. Try 20+. You basically want to cache all of your pool directory/inode data plus the pc backup trees being traversed by rsync. And without threshing it out with the rsync copy of the remote trees that are being processed. > And I haven't seen that RAID-5 has been that much of an issue. As you > mention, BackupPC is *READ* heavy so the RMW penalty doesn't hurt much. My > experience with both RAID-5 *AND* RAID-6 (even software RAID-6!) is just fine. Sure, they work. But they force all of the disk heads to seek every time unless you have a large number of disks, and every short write (and most of them are with all of the directory/inode operations happening), is going to cost an extra disk revolution time for the read/modify/write step. And that 'read heavy' assessment is somewhat optimistic if you have any large files that are modified in place. There you get the worst case of alternating small reads/writes in different places as the changes delivered by rsync are merged into a copy of the old file. > Of course, I have a minimum of 4 drives in a RAID array (6 minimum for > RAID-6), so I'm usually bottlenecked somewhere else anyway, such as a single > 1Gb link. It's not hard to mange writing 70MB/s of data! Seriously? You have rsync pegging a 1Gb link for long periods of time anytime but the first run? Do you have something creating a lot of huge new files? My times are mostly constrained by the read time on the targets with a relatively small amount of data transfer. -- Les Mikesell lesmikes...@gmail.com ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/