Re: [BackupPC-users] Troubleshooting a slow backup
On Thu, Sep 27, 2007 at 04:07:21PM -0700, David Rees wrote: > On 9/27/07, Dan Pritts <[EMAIL PROTECTED]> wrote: > > So I've been of the opinion (not backed up by experimental data) that > > a concatenation (what linux md driver calls LINEAR; similar effects can > > be realized with LVM) of two RAID1's would be better for BackupPC than > > a RAID10. > > > > My rationale for this is that in RAID10, the disks are generally > > seeking to the same spot, unless you have a write that doesn't span > > across multiple raid stripes. This certainly happens, but i suspect > > most writes span multiple stripes. > > > > i guess this really depends on the RAID stripe size, bigger would be better. > > Looking at my average file size on one of my backuppc servers, it > appears to be about 50KB. With a typical stripe size being 64KB, that > would seem to indicate that your average write will fit on one stripe, > so that may hurt your argument. I'm not sure why i wrote that; the thing that i typically think about with backuppc is all seeking required due to the extensive use of hard links, and i'm trying to minimize that. Certainly all the hard links that backuppc does are pretty much guaranteed to be tiny writes, and if i'm right that is a huge portion of the I/O load. What follows is pretty much stream-of-conciousness and I don't have the time to edit it; i've spent too much time on this already. Sorry for that but perhaps you'll find it interesting. on the other hand what ARE the odds of your 50KB file fitting in a 64KB stripe? It would have to *start* within the first 14KB of the stripe; so it happens roughly 25% the time if this start location is random. The block size of your filesystem probably comes into play here, but with a typical 4KB block size there is enough distribution in that 64KB stripe that it probably is close to random. (without thinking too hard) So only a quarter of those 50KB files would fit in a single stripe. Hmm. how many disk ops are required for a single file write? Hmm, off the top of my head & after reading some about good old ext2: - directory modification to create the file, and journal write of the directory mod (how many actual journal writes?) - modify the inode to note blocks in use, owner, timestamp, etc. (for a big file, you might have indirect blocks also) - modify used-block data structure - actual write of the data blocks; assume contiguous - modify superblock's block-free count (a separate operation from used-block data structure in ext2; not sure about other filesystems) Little seeking is required for the inode, block structure, & data blocks. But i bet this group is likely to span multiple stripes. If you're lucky the directory is close by & fits into this category. These are nearby, related writes; slower than a single write but probably very fast on command-queuing disks. (not sure how much slower on non-command-queuing PATA disks, but perhaps significantly) This group of writes, while close together, is pretty much guaranteed to be spanning multiple stripes. So let's consider it a big single write. the superblock free block count & journal writes are probably pretty small. Those probably each fit within a stripe. *but* they involve probably a significant seek from the data area. So let's consider these two small writes. So an average file *probably* involves a big single write and two small single writes. In the case of a hard link to an existing file, you have two small writes, one to the directory and one to the inode to update the link count. These are likely nowhere particularly near one another; so likely half the time in RAID10 they go to different disks. In a concatenation, on a reasonably full filesystem, i bet this is similar. I'm sure that there is something inaccurate in what I wrote above. So at the end of this what i come up with is that as you say below, it's awfully complicated, with lots of big and lots of small writes, and neither of us has probably considered all of the implications. > Additionally, if we look at the big picture where we are writing out a > bunch of files, these are pretty much guaranteed to be scattered all > over the disk with your typical filesystem. Definitely, on a full disk. In which case concatenation is as good as RAID10. > Even a fresh filesystem > will scatter new files over the disk to help avoid fragmentation > issues should you decide to grow the file later. Didn't know that. Is it typically truly randomly/evenly spaced? Or will it tend to start out at the front and work its way to the back, leaving spaces in the middle? Any quick articles you can suggest (not that i don't believe you, just interested in more info)? > Now throw in the number of backups you are doing and you end up with > too many variables to consider before assuming that a linear array > will outperform a striped array. So, you don't know either :) > For random reads all over the di
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/27/07, Dan Pritts <[EMAIL PROTECTED]> wrote: > So I've been of the opinion (not backed up by experimental data) that > a concatenation (what linux md driver calls LINEAR; similar effects can > be realized with LVM) of two RAID1's would be better for BackupPC than > a RAID10. > > My rationale for this is that in RAID10, the disks are generally > seeking to the same spot, unless you have a write that doesn't span > across multiple raid stripes. This certainly happens, but i suspect > most writes span multiple stripes. > > i guess this really depends on the RAID stripe size, bigger would be better. Looking at my average file size on one of my backuppc servers, it appears to be about 50KB. With a typical stripe size being 64KB, that would seem to indicate that your average write will fit on one stripe, so that may hurt your argument. Additionally, if we look at the big picture where we are writing out a bunch of files, these are pretty much guaranteed to be scattered all over the disk with your typical filesystem. Even a fresh filesystem will scatter new files over the disk to help avoid fragmentation issues should you decide to grow the file later. Now throw in the number of backups you are doing and you end up with too many variables to consider before assuming that a linear array will outperform a striped array. For random reads all over the disk, the performance should be similar for small files but large file reads should be up to twice as fast. Throw in multiple readers and the difference will narrow. > > Stay away from RAID5 unless you have a good > > controller with a battery backed cache. > > Even then, performance won't be great, especially on random small writes > (look up the "RAID5 write hole" and "read-modify-write" to understand why). But wait, I thought you said that the average write under backuppc load would be larger than a stripe? So which is it? ;-) -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On Wed, Sep 26, 2007 at 11:17:54AM -0700, David Rees wrote: > Your BackupPC server's disk is completely maxed out. Looks like it is > doing a lot of seeking. To get more throughput, you'll need more disk > spindles. RAID1 will improve random read IO performance, but you'll > need RAID10 w/4 disks which should get you 2-4x read performance and > 2x write performance. So I've been of the opinion (not backed up by experimental data) that a concatenation (what linux md driver calls LINEAR; similar effects can be realized with LVM) of two RAID1's would be better for BackupPC than a RAID10. My rationale for this is that in RAID10, the disks are generally seeking to the same spot, unless you have a write that doesn't span across multiple raid stripes. This certainly happens, but i suspect most writes span multiple stripes. i guess this really depends on the RAID stripe size, bigger would be better. IF i'm right and your disk heads are seeking more or less in tandem to one another, then you don't really buy much random i/o write performance by going to RAID10. On the other hand, with the concatenation, over time, as the disks fill up, you are likely to have the disk heads seeking totally independently of one another so you should get more random IOPS. You don't have any control over which spindles get which I/Os, however, so it could easily be imbalanced, especially at the beginning before the disks fill up. database guys always say "raid10 raid10 raid10". Presumably, backuppc's random I/O is similar to a big database. So maybe I'm just wrong and there's something better about RAID10 that I don't understand. On the other hand, if they're just comparing to raid5, well, yeah, raid10 is better. David, do you have any data comparing the two scenarios? Do you see flaws in my logic? > Stay away from RAID5 unless you have a good > controller with a battery backed cache. Even then, performance won't be great, especially on random small writes (look up the "RAID5 write hole" and "read-modify-write" to understand why). danno - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/26/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > David Rees wrote: > > Your machine looks fine to me. Your backuppc data partition is a single > > disk? > > My servers disk it 6 250G IDE drives arranged in a RAID5 with 1 Hot > Spare. The Controller is a 3Ware Escalade 7506-8 Controller. OK, there are some settings which can especially help with 3ware controllers. What is the output of the following files (use cat)? /sys/block/sd*/queue/max_sectors_kb /sys/block/sd*/queue/nr_requests /sys/block/sd*/device/queue_depth Specifically, the queue depth is often much larger than nr_requests, so setting nr_requests to something about twice as big as the queue_depth helps. On a 3ware card, the queue_depth should be about 254, so trying setting nr_requests to 512 like this (assuming sda is your raid array): echo 512 > /sys/block/sda/queue/nr_requests Also setting max_sectors_kb to 64 can help with 3ware raid 5 arrays: echo 64 > /sys/block/sda/queue/max_sectors_kb If the system gets sluggish under high IO load, you can also switch to the deadline IO scheduler: echo deadline > /sys/block/sda/queue/scheduler More info/reference: http://www.3ware.com/KB/article.aspx?id=11050 > > As the others suggested, mounting the backuppc data partition with > > noatime can help. I usually mount all my filesystems with noatime. > > Mounting the backuppc data partition with data=ordered option may > > help, too (the default is data=writeback). > > I'll add data=ordered now Ugh, data=ordered is the default, you should try data=writeback (had it backwards). http://linuxmafia.com/faq/Filesystems/journaled-filesystems.html > I'll try the checksum seed with a test backup now.. Typically, the next backup won't get much faster, IIRC it takes 2-3 backups to realize the full performance from using checksum seed. I don't know if the actual value of checksum seed matters, but 32761 is typically used as suggested in the default config.pl file. -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
I am testing with --checksum-seed=0 .. before I waste several hours, is 0 a useful number? Thanks in advance, Tony Nelson Starpoint Solutions David Rees wrote: On 9/26/07, Tony Nelson <[EMAIL PROTECTED]> wrote: Well, due to a power failure, I was put in the lovely position of a corrupted ReiserFS tree. I ran reiserfsck, which took 4 days to complete and just couldn't bring myself to trust stability of the disk. Given the lack of interest/maintainers in ReiserFS, I wouldn't recommend it to anyone. It still seems very slow to me. I don't know if I should attribute it to the fact that everything it is doing is a full backup or not. I've attached the output from vmstat on the BackupPC server. The server is currently running 3 full backups. Your BackupPC server's disk is completely maxed out. Looks like it is doing a lot of seeking. To get more throughput, you'll need more disk spindles. RAID1 will improve random read IO performance, but you'll need RAID10 w/4 disks which should get you 2-4x read performance and 2x write performance. Stay away from RAID5 unless you have a good controller with a battery backed cache. You can see from the vmstat output that the server is doing a bit of reading and some writing. Given the low throughput (2-3MB/s) and high IO wait, you can see that the disk is spending most of it's time doing lots of small IO all over the disk which is typical of BackupPC disk load. I'm going to investigate the currently installed software on this machine just to see if I can find any issues. Your machine looks fine to me. Your backuppc data partition is a single disk? As the others suggested, mounting the backuppc data partition with noatime can help. I usually mount all my filesystems with noatime. Mounting the backuppc data partition with data=ordered option may help, too (the default is data=writeback). I also typically avoid running more than 2 concurrent backups at a time, but it's really a matter of balancing server CPU/disk/network utilization to find out how many you can run. For a typical single/dual CPU system with a single disk or RAID1, backing up machines on the LAN, 2 concurrent backups is enough. Did you also try enabling the checksum-seed option? What about upgrading to backuppc 3.1beta and installing the IO::Direct perl module? 3.1beta also has a few other performance improvements. -Dave begin:vcard fn:Tony Nelson n:Nelson;Tony org:Starpoint Solutions adr;dom:2nd Floor;;115 Broadyway;New York;NY;10006 email;internet:[EMAIL PROTECTED] title:Dir of IT Operation x-mozilla-html:FALSE url:http://www.starpoint.com version:2.1 end:vcard - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
David Rees wrote: On 9/26/07, Tony Nelson <[EMAIL PROTECTED]> wrote: Well, due to a power failure, I was put in the lovely position of a corrupted ReiserFS tree. I ran reiserfsck, which took 4 days to complete and just couldn't bring myself to trust stability of the disk. Given the lack of interest/maintainers in ReiserFS, I wouldn't recommend it to anyone. I definitely won't be using it anymore. It still seems very slow to me. I don't know if I should attribute it to the fact that everything it is doing is a full backup or not. I've attached the output from vmstat on the BackupPC server. The server is currently running 3 full backups. Your BackupPC server's disk is completely maxed out. Looks like it is doing a lot of seeking. To get more throughput, you'll need more disk spindles. RAID1 will improve random read IO performance, but you'll need RAID10 w/4 disks which should get you 2-4x read performance and 2x write performance. Stay away from RAID5 unless you have a good controller with a battery backed cache. You can see from the vmstat output that the server is doing a bit of reading and some writing. Given the low throughput (2-3MB/s) and high IO wait, you can see that the disk is spending most of it's time doing lots of small IO all over the disk which is typical of BackupPC disk load. I'm going to investigate the currently installed software on this machine just to see if I can find any issues. Your machine looks fine to me. Your backuppc data partition is a single disk? My servers disk it 6 250G IDE drives arranged in a RAID5 with 1 Hot Spare. The Controller is a 3Ware Escalade 7506-8 Controller. As the others suggested, mounting the backuppc data partition with noatime can help. I usually mount all my filesystems with noatime. Mounting the backuppc data partition with data=ordered option may help, too (the default is data=writeback). I was using noatime /dev/sda1 /var/lib/backuppc ext3noatime 1 2 I'll add data=ordered now I also typically avoid running more than 2 concurrent backups at a time, but it's really a matter of balancing server CPU/disk/network utilization to find out how many you can run. For a typical single/dual CPU system with a single disk or RAID1, backing up machines on the LAN, 2 concurrent backups is enough. Did you also try enabling the checksum-seed option? What about upgrading to backuppc 3.1beta and installing the IO::Direct perl module? 3.1beta also has a few other performance improvements. I'll try the checksum seed with a test backup now.. -Dave begin:vcard fn:Tony Nelson n:Nelson;Tony org:Starpoint Solutions adr;dom:2nd Floor;;115 Broadyway;New York;NY;10006 email;internet:[EMAIL PROTECTED] title:Dir of IT Operation x-mozilla-html:FALSE url:http://www.starpoint.com version:2.1 end:vcard - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/26/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > Well, due to a power failure, I was put in the lovely position of a > corrupted ReiserFS tree. I ran reiserfsck, which took 4 days to > complete and just couldn't bring myself to trust stability of the disk. Given the lack of interest/maintainers in ReiserFS, I wouldn't recommend it to anyone. > It still seems very slow to me. I don't know if I should attribute it > to the fact that everything it is doing is a full backup or not. > > I've attached the output from vmstat on the BackupPC server. The server > is currently running 3 full backups. Your BackupPC server's disk is completely maxed out. Looks like it is doing a lot of seeking. To get more throughput, you'll need more disk spindles. RAID1 will improve random read IO performance, but you'll need RAID10 w/4 disks which should get you 2-4x read performance and 2x write performance. Stay away from RAID5 unless you have a good controller with a battery backed cache. You can see from the vmstat output that the server is doing a bit of reading and some writing. Given the low throughput (2-3MB/s) and high IO wait, you can see that the disk is spending most of it's time doing lots of small IO all over the disk which is typical of BackupPC disk load. > I'm going to investigate the currently installed software on this > machine just to see if I can find any issues. Your machine looks fine to me. Your backuppc data partition is a single disk? As the others suggested, mounting the backuppc data partition with noatime can help. I usually mount all my filesystems with noatime. Mounting the backuppc data partition with data=ordered option may help, too (the default is data=writeback). I also typically avoid running more than 2 concurrent backups at a time, but it's really a matter of balancing server CPU/disk/network utilization to find out how many you can run. For a typical single/dual CPU system with a single disk or RAID1, backing up machines on the LAN, 2 concurrent backups is enough. Did you also try enabling the checksum-seed option? What about upgrading to backuppc 3.1beta and installing the IO::Direct perl module? 3.1beta also has a few other performance improvements. -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On Wed, 26 Sep 2007 10:28:31 -0400 Tony Nelson <[EMAIL PROTECTED]> wrote: > Well, due to a power failure, I was put in the lovely position of a > corrupted ReiserFS tree. I ran reiserfsck, which took 4 days to > complete and just couldn't bring myself to trust stability of the > disk. > > So, I backed up all of my BackupPC config information, and > reformatted the disk as ext3. > > Restored all my config, and fired up BackupPC. > > It still seems very slow to me. I don't know if I should attribute > it to the fact that everything it is doing is a full backup or not. > > I've attached the output from vmstat on the BackupPC server. The > server is currently running 3 full backups. > > I'm going to investigate the currently installed software on this > machine just to see if I can find any issues. > Change your mount command to include the no atime option like so: /dev/sda4 /data ext3 noatime There is no need for the system to update the access time (e.g. do a write to disk) everytime it looks at a file. This can make a big difference. I found that running 3ware cards, I can only handle 3 concurrent backups in my situation. Anything more really bogs down the system. I think this is pretty system and load specific so you will have to do testing with your system and load to find out what works best for you. cheers, ski -- "When we try to pick out anything by itself, we find it connected to the entire universe"John Muir Chris "Ski" Kacoroski, [EMAIL PROTECTED], 206-501-9803 - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 09/26 10:28 , Tony Nelson wrote: > It still seems very slow to me. I don't know if I should attribute it > to the fact that everything it is doing is a full backup or not. > > I've attached the output from vmstat on the BackupPC server. The server > is currently running 3 full backups. I've found that more than 2 backups simultaneously tends to bog everything down with Disk I/O. I don't know what the baseline for your system is like, so I can't judge much based on that vmstat output. I'd suggest running 2 backups simultaneously, and see if the disk I/O is greater, lesser, or similar. Then try it with 1 backup. There are also some journaling options you can tweak to get better ext3 performance, (at the risk of increased ease of corruption) and you might even consider moving the journal to another device. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On Thursday 20 September 2007 01:49:01 David Rees wrote: > On 9/19/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > > I looked into the checksum-seed option for rsync and it appears to a > > patch that I don't have. I am using Gentoo and just installed rsync > > from Portage. Has that patch every made it into the rsync upstream? > > Checksum seed support was added in rsync 2.6.3 (30 Sep 2004) so > any halfway recent distro should support it. I can't imagine that > Gentoo doesn't. Yes, Gentoo version of rsync does support checksum-seed option. :) Best regards, Janne Pikkarainen - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/19/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > Attached are the files you requested. The BackupPC server was running 2 > long running backups when I took these. In addition to the screenshots > you requested, I added a screenshot from the web console of BackupPC. >From your screenshots, the web server CPU/disk utilization is not an issue, vmstat is showing it to be near idle. Your backup server is showing about 22% waiting on disk IO, how many processors and what type of disk/filesystem is your backuppc data partition on? If your backuppc data partition isn't mounted with noatime, that's one of the first things I'd do (but isn't likely to make a huge difference). It also appears that you aren't using the checksum-seed rsync option, you should enable that as well. > I attempted counting the number of files being backed up by selecting > data from a SQL server that contains information about most of the > files. The most reasonable ballpark I can give you is between 275k and > 300k files. If a it's better, I'll run a 'find /ha -type f | wc -l' > during the off hours tonight, unless you can suggest a better command. Not too many files. How much new data is being backed up each day? You can get file counts and sizes of full/incr backups by looking at the hosts's summary page under "File Size/Count Reuse Summary" -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/19/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > I looked into the checksum-seed option for rsync and it appears to a > patch that I don't have. I am using Gentoo and just installed rsync > from Portage. Has that patch every made it into the rsync upstream? Checksum seed support was added in rsync 2.6.3 (30 Sep 2004) so any halfway recent distro should support it. I can't imagine that Gentoo doesn't. -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/19/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > David Rees wrote: > > Your backup server is showing about 22% waiting on disk IO, how many > > processors and what type of disk/filesystem is your backuppc data > > partition on? > > The system has 2 dual core Xeon processors. That's what I expected given the slightly less than 25% disk wait, one of the processors is just sitting there for disk IO most of the time. > The filesystem is a 1.5TB Hardware controlled RAID Array made up of > 250GB IDE (not SATA) drives, in a Penguin Computing Relion 430 server. What kind of RAID? (1, 5, 10, etc) > The filesystem is a ReiserFS (M5 hash if memory serves) filesystem, > mounted as: > > /dev/sda1 /var/lib/backuppc reiserfs noatime 1 2 I know some people like using reiserfs for some reason, but I seem to recall having slow IO performance over lots of small files a while back using it. I use ext3 on all my backuppc servers myself. > I'm not aware of the checksum-seed option to rsync, but I'll look it up > and add it tonight, and see how the backup goes. The check-sum seed option will definitely reduce the IO load on the server by some amount. This was added in rsync 2.6.3 (30 Sep 2004) so any halfway recent distro should support it. > It turns out my guess was way wrong. > > New > 3086 Files > 1572.8MB Data If you're getting that much new data a day, you should try to run full backups more frequently, or upgrade to BackupPC 3 which supports multi-level rsync incrementals. > I also see looking at that page, I haven't had a good full backup in > quite some time. I manually schedule this particular backup out of cron > to ensure that it starts at the same time every day and it looks like > the full from the past couple of weekends failed. I wonder if that is > affecting performance. That is definitely affecting performance since for each incremental backup, you need to transfer over all the data that has changed since the last full backup (unless you are using multi-level rsync incrementals in BackupPC 3). Also available in BackupPC 3.1 beta, you can install the IO::Dirent perl module on the server which should also improve backuppc performance during trash clean operations. -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
Dave, I looked into the checksum-seed option for rsync and it appears to a patch that I don't have. I am using Gentoo and just installed rsync from Portage. Has that patch every made it into the rsync upstream? Thanks Tony Tony Nelson wrote: David Rees wrote: On 9/19/07, Tony Nelson <[EMAIL PROTECTED]> wrote: Attached are the files you requested. The BackupPC server was running 2 long running backups when I took these. In addition to the screenshots you requested, I added a screenshot from the web console of BackupPC. From your screenshots, the web server CPU/disk utilization is not an issue, vmstat is showing it to be near idle. Your backup server is showing about 22% waiting on disk IO, how many processors and what type of disk/filesystem is your backuppc data partition on? The system has 2 dual core Xeon processors. CPU0: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09 (repeated of course for CPU[1..3]) The filesystem is a 1.5TB Hardware controlled RAID Array made up of 250GB IDE (not SATA) drives, in a Penguin Computing Relion 430 server. If your backuppc data partition isn't mounted with noatime, that's one of the first things I'd do (but isn't likely to make a huge difference). It also appears that you aren't using the checksum-seed rsync option, you should enable that as well. The filesystem is a ReiserFS (M5 hash if memory serves) filesystem, mounted as: /dev/sda1 /var/lib/backuppc reiserfs noatime 1 2 I'm not aware of the checksum-seed option to rsync, but I'll look it up and add it tonight, and see how the backup goes. I attempted counting the number of files being backed up by selecting data from a SQL server that contains information about most of the files. The most reasonable ballpark I can give you is between 275k and 300k files. If a it's better, I'll run a 'find /ha -type f | wc -l' during the off hours tonight, unless you can suggest a better command. Not too many files. How much new data is being backed up each day? You can get file counts and sizes of full/incr backups by looking at the hosts's summary page under "File Size/Count Reuse Summary" It turns out my guess was way wrong. Totals 842487 Files 24267.8MB Data 0.52MB/s Transfer Existing 841356 Files 22695.2MB Data New 3086 Files 1572.8MB Data I also see looking at that page, I haven't had a good full backup in quite some time. I manually schedule this particular backup out of cron to ensure that it starts at the same time every day and it looks like the full from the past couple of weekends failed. I wonder if that is affecting performance. Thanks for your help Tony - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ begin:vcard fn:Tony Nelson n:Nelson;Tony org:Starpoint Solutions adr;dom:2nd Floor;;115 Broadyway;New York;NY;10006 email;internet:[EMAIL PROTECTED] title:Dir of IT Operation x-mozilla-html:FALSE url:http://www.starpoint.com version:2.1 end:vcard - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
David Rees wrote: On 9/19/07, Tony Nelson <[EMAIL PROTECTED]> wrote: Attached are the files you requested. The BackupPC server was running 2 long running backups when I took these. In addition to the screenshots you requested, I added a screenshot from the web console of BackupPC. From your screenshots, the web server CPU/disk utilization is not an issue, vmstat is showing it to be near idle. Your backup server is showing about 22% waiting on disk IO, how many processors and what type of disk/filesystem is your backuppc data partition on? The system has 2 dual core Xeon processors. CPU0: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09 (repeated of course for CPU[1..3]) The filesystem is a 1.5TB Hardware controlled RAID Array made up of 250GB IDE (not SATA) drives, in a Penguin Computing Relion 430 server. If your backuppc data partition isn't mounted with noatime, that's one of the first things I'd do (but isn't likely to make a huge difference). It also appears that you aren't using the checksum-seed rsync option, you should enable that as well. The filesystem is a ReiserFS (M5 hash if memory serves) filesystem, mounted as: /dev/sda1 /var/lib/backuppc reiserfs noatime 1 2 I'm not aware of the checksum-seed option to rsync, but I'll look it up and add it tonight, and see how the backup goes. I attempted counting the number of files being backed up by selecting data from a SQL server that contains information about most of the files. The most reasonable ballpark I can give you is between 275k and 300k files. If a it's better, I'll run a 'find /ha -type f | wc -l' during the off hours tonight, unless you can suggest a better command. Not too many files. How much new data is being backed up each day? You can get file counts and sizes of full/incr backups by looking at the hosts's summary page under "File Size/Count Reuse Summary" It turns out my guess was way wrong. Totals 842487 Files 24267.8MB Data 0.52MB/s Transfer Existing 841356 Files 22695.2MB Data New 3086 Files 1572.8MB Data I also see looking at that page, I haven't had a good full backup in quite some time. I manually schedule this particular backup out of cron to ensure that it starts at the same time every day and it looks like the full from the past couple of weekends failed. I wonder if that is affecting performance. Thanks for your help Tony begin:vcard fn:Tony Nelson n:Nelson;Tony org:Starpoint Solutions adr;dom:2nd Floor;;115 Broadyway;New York;NY;10006 email;internet:[EMAIL PROTECTED] title:Dir of IT Operation x-mozilla-html:FALSE url:http://www.starpoint.com version:2.1 end:vcard - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 9/18/07, Tony Nelson <[EMAIL PROTECTED]> wrote: > What I would like to do is figure out the best way of determining if the > source of the slowness is the target server, the backuppc server or a > network bottleneck that I just can't imagine. Fire up `top` and `vmstat 3` on each machine while the backup is running, preferably when it's the only backup running and nothing else is going on. Grab a representative screenshot from each and post them up, it should quickly show where the bottleneck is. How many files total are being backed up? -Dave - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Troubleshooting a slow backup
On 09/18 10:45 , Tony Nelson wrote: > I have been using BackupPC very successfully for several years at > several sites now. I'm presently having an issue with a few backups > running very slowly and I'm looking for help troubleshooting the issue. Have you tried doing the backup with tar? don't be surprised if tar is 2x-3x faster than rsync, I'd consider that normal when taking a fresh backup. if it's 4x-5x faster, there may be an inefficiency somewhere. have you tried the backup with regular rsync, to a fresh directory (and then re-do it a day later), and measured how long that takes? -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
