Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
The box isn't used for anything else..I can keep an eye on it tonight and see whats causing the load in those first few hours.. On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
going by the graphs it seems to be an IO issue, has anyone tried running raid 10 with backuppc? On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Stephen Vaughan wrote: The box isn't used for anything else..I can keep an eye on it tonight and see whats causing the load in those first few hours.. On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. The load and waiting on I/O is definitly suspicious. Maybe something like a cronjob running updatedb for locate ? /etc/updatedb.conf holds the paths that should be excluded if you don't want to turn it off completely. As they are both doing a lot of disk activity it will slow down rsync. I suggest having a good look at syslog looking for CRON and/or check: /etc/crontab /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /var/spool/cron/ -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
iostat is also showing iowait fluctuating between 50-75 % -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
I definitely don't have updated installed on the box. I'm monitoring it the moment and it seems BackupPC_nightly is running while performing a backup of 2 servers, it must run run backuppc_nightly for the first few hours every night. Output from top, CPU usage is in the 9th column. 12269 backuppc 20 0 117m 114m 2364 D 55 2.9 2:47.40 BackupPC_dump 12270 backuppc 20 0 226m 216m 2356 D 35 5.5 1:22.66 BackupPC_dump 13538 backuppc 20 0 185m 176m 1324 D 29 4.5 0:25.90 BackupPC_dump 12264 backuppc 20 0 6 8076 2060 D 10 0.2 0:22.06 BackupPC_nightl 12262 backuppc 20 0 11136 8092 2060 D5 0.2 0:25.46 BackupPC_nightl 12267 backuppc 20 0 11160 8112 2060 D4 0.2 0:23.40 BackupPC_nightl 13109 backuppc 20 0 106m 101m 1324 S3 2.6 0:52.04 BackupPC_dump 12268 backuppc 20 0 11024 8088 2060 D2 0.2 0:23.84 BackupPC_nightl 12261 backuppc 20 0 11148 8120 2072 D1 0.2 0:24.46 BackupPC_nightl 12265 backuppc 20 0 11136 8088 2060 D1 0.2 0:23.90 BackupPC_nightl 10 root 15 -5 000 S1 0.0 38:30.83 ksoftirqd/2 182 root 15 -5 000 S1 0.0 16:12.28 kblockd/0 351 root 15 -5 000 S1 0.0 1296:21 kswapd0 12263 backuppc 20 0 11132 8112 2060 D1 0.2 0:21.20 BackupPC_nightl 12266 backuppc 20 0 11024 8084 2060 D1 0.2 0:24.10 BackupPC_nightl On Tue, Jul 7, 2009 at 6:20 PM, Leen Besselink l...@consolejunky.netwrote: Stephen Vaughan wrote: The box isn't used for anything else..I can keep an eye on it tonight and see whats causing the load in those first few hours.. On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. The load and waiting on I/O is definitly suspicious. Maybe something like a cronjob running updatedb for locate ? /etc/updatedb.conf holds the paths that should be excluded if you don't want to turn it off completely. As they are both doing a lot of disk activity it will slow down rsync. I suggest having a good look at syslog looking for CRON and/or check: /etc/crontab /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /var/spool/cron/ -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen Vaughan wrote: I definitely don't have updated installed on the box. I'm monitoring it the moment and it seems BackupPC_nightly is running while performing a backup of 2 servers, it must run run backuppc_nightly for the first few hours every night. That is definitely the first thing you should change, set the nightly job to run at 10am outside the backup window. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpTMVIACgkQGyoxogrTyiWIjACgic2MncLhz9i5Zn88xCAAHkQ4 qF8An1nz2Aed2TSAjDRrCSJlFV3oryIY =+TIB -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
yep ok I've just done that. On Tue, Jul 7, 2009 at 9:28 PM, Adam Goryachev mailingli...@websitemanagers.com.au wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen Vaughan wrote: I definitely don't have updated installed on the box. I'm monitoring it the moment and it seems BackupPC_nightly is running while performing a backup of 2 servers, it must run run backuppc_nightly for the first few hours every night. That is definitely the first thing you should change, set the nightly job to run at 10am outside the backup window. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpTMVIACgkQGyoxogrTyiWIjACgic2MncLhz9i5Zn88xCAAHkQ4 qF8An1nz2Aed2TSAjDRrCSJlFV3oryIY =+TIB -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Hi,
On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote:
The only thing that seems slightly strange in the graphs is the load average
going to 12 as the backups start and staying there a couple of hours.
Normally
that's the average number of 'other' processes that are waiting for CPU but
otherwise runnable (i.e. not themselves blocked on i/o).
I used to think that, but in fact processes that are blocked in disk
i/o (the ones in D state) do count in load average. So the load
average of 12 in this case probably means processes writing to the
disk.
From another e-mail showing several (12?) processes BackupPC_dump and
BackupPC_nightly in D state, those should be the reason why this box
stays with a load average of 12 during the night.
From those observations, it seems to me that the bottleneck in this
case is disk I/O.
Stephen, it seems strange to me that there are 8 BackupPC_nightly
processes running, have you increased $Conf{MaxBackupPCNightlyJobs}
from 2 to 8?
I would suggest you start by setting $Conf{MaxBackupPCNightlyJobs}
back to 2 or even to 1. If you set it to 1 and it can't finish its job
in 24h, then increase $Conf{BackupPCNightlyPeriod} to 2 or 4 so that
only 1/2 or 1/4 of the pool is processed each night.
HTH,
Filipe
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Filipe Brandenburger wrote:
On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote:
The only thing that seems slightly strange in the graphs is the load average
going to 12 as the backups start and staying there a couple of hours.
Normally
that's the average number of 'other' processes that are waiting for CPU but
otherwise runnable (i.e. not themselves blocked on i/o).
I used to think that, but in fact processes that are blocked in disk
i/o (the ones in D state) do count in load average. So the load
average of 12 in this case probably means processes writing to the
disk.
That must be a Linux quirk (bug?) but it does explain some numbers I've
seen. Regardless, there shouldn't be that many things running.
Stephen, it seems strange to me that there are 8 BackupPC_nightly
processes running, have you increased $Conf{MaxBackupPCNightlyJobs}
from 2 to 8?
I would suggest you start by setting $Conf{MaxBackupPCNightlyJobs}
back to 2 or even to 1. If you set it to 1 and it can't finish its job
in 24h, then increase $Conf{BackupPCNightlyPeriod} to 2 or 4 so that
only 1/2 or 1/4 of the pool is processed each night.
More importantly, BackupPC_nightly shouldn't overlap with backup jobs if
possible.
--
Les Mikesell
lesmikes...@gmail.com
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Hi, Les Mikesell wrote on 2009-07-07 12:17:56 -0500 [Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server]: Filipe Brandenburger wrote: On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote: The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). I used to think that, but in fact processes that are blocked in disk i/o (the ones in D state) do count in load average. So the load average of 12 in this case probably means processes writing to the disk. That must be a Linux quirk (bug?) but it does explain some numbers I've seen. if it is, it's inherited at least from SunOS (and HP-UX, if I remember correctly). I haven't been using Solaris for quite a while, so I can't say if the load on NFS clients still goes up when NFS servers go down. SunOS 5.10 w(1) documents the load to mean average number of jobs in the run queue, which should *not* include processes waiting for I/O. Probably a Solaris quirk (bug?) though. Both ways of defining load make sense. Processes waiting for short term disk I/O are using resources (and would probably be running if the disk was simply faster). NFS I/O is not necessarily short term, but that's a different matter. Linux uptime(1) documents what system load means on Linux. Wherever it matters, you won't be looking at a single figure to measure your system's state anyway. Regardless, there shouldn't be that many things running. Yes, running BackupPC_nightly outside the backup window (as has already been agreed upon) is definitely important. Regards, Holger -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Holger Parplies wrote: Les Mikesell wrote on 2009-07-07 12:17:56 -0500 [Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server]: Filipe Brandenburger wrote: On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote: The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). I used to think that, but in fact processes that are blocked in disk i/o (the ones in D state) do count in load average. So the load average of 12 in this case probably means processes writing to the disk. That must be a Linux quirk (bug?) but it does explain some numbers I've seen. if it is, it's inherited at least from SunOS (and HP-UX, if I remember correctly). I haven't been using Solaris for quite a while, so I can't say if the load on NFS clients still goes up when NFS servers go down. SunOS 5.10 w(1) documents the load to mean average number of jobs in the run queue, which should *not* include processes waiting for I/O. Probably a Solaris quirk (bug?) though. Both ways of defining load make sense. Processes waiting for short term disk I/O are using resources (and would probably be running if the disk was simply faster). NFS I/O is not necessarily short term, but that's a different matter. It doesn't make sense to me to consider a process runnable when it is waiting for a hardware operation to complete - the scheduler should be ignoring it. I suppose if the disk in question is an IDE that the CPU has to micro-manage it might make sense to blame the application for the CPU use even if the kernel is doing it. Linux uptime(1) documents what system load means on Linux. Wherever it matters, you won't be looking at a single figure to measure your system's state anyway. Yes, the load average in mostly just useful to tell you if a faster CPU would help, but it isn't even good for that if it counts things that couldn't use the CPU anyway. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
I have been using backuppc for quite some time. Here are some of the thinks I have found. 1) backup method is super important. If your backup machine is in near the servers it is going to backup, then consider NOT using rsync. The disadvantage is that you will hammer your network but this can be overcome with a coupe gigabit network cards and a basic gigabit switch creating a backup only network. I suggest this because backuppc is IO bound and the way that rsync checks files is more intensive than doing a tar backup. Using a direct file copy method will still check mod times but will not scan the file list rsync does. You will get to take better advantage of sequential reads and writes rather than random reads and writes. 2) A 2Ghz Dual Core CPU with 4GB of ram seams to be about the peak of efficiency. Other factors will limit backup speed long before you fill up 4GB of ram or use all of the CPU. 3) hard disk performance is the #1 thing. RAM only helps compensate so much by giving extra cache memory. Get a real RAID card. software RAID is great for some things but the time it takes for data to travel accross the system bus twice (to cpu and from cpu) is much greater than the time it takes to go to the storage controllers CPU and back. Get a controller than has 128MB or more of cache on board and supports whatever RAID mode you want. 4) RAID1. Skip RAID5, do RAID1. This data is important and the speed that you back it up is important. A few extra dollar on some redundant hard disks is a bargain. If you need more than 2 drives, consider RAID1 pairs with RAID0 on top. 5) Hot spare. When a drive fails, you want the rebuild to start immediately and get done. That is a window for data loss. The only way around this without moving to a parity based RAID5 or 6 is to do triple drive redundancy(RAID1 with 3 disks) 6) secondary backup server. You can mirror you whole array with rsync though it will be slow and use a ton of RAM. What I have done on smaller setups is to actually put my RAID1 device md0 in a RAID1 of md1 that is made of md0 and a AoE disk on another server. You can do some scripts to add the AoE disk to the array until it is completely synced up and then remove it. If you dont remove it, then the md1 raid device will be slow as it will stop to sync up the compontent devices every so often. You can use this same setup to sync to an external USB drive by adding it to the array for a sync and removing it afterwards. AoE = ATA over Ethernet and it has great sequential reads and write on ethernet. I have had 93% efficient transfers which means that I can get up to 116MB/s on Gigabit as there is no TCP/IP overhead, just straight ethernet frames. A RAID1 rebuild is sequential but there is some CPU overhead in either the system CPU or the RAID controller. I have managed over 80MB/s rebuild speed on a PCI-Express based server on Intel Pro Gigabit cards and a Cisco catalyst switch with AoE. Expect that to half if you use lesser hardware. AoE only works on local ethernet as ethernet frames do not travel beyond a managed switched ethernet, which means that you cannot router AoE over the internet or over a VPN. -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
dan wrote: 1) backup method is super important. If your backup machine is in near the servers it is going to backup, then consider NOT using rsync. Or, if you like the mirroring ability of rsync but don't want or need to use the special rsync algorithm, use rsync with the '-W' option. Jon -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
I started out with a Dell PowerEdge with a 2.00GHz Intel CPU and 512MB of RAM, backing up to a software RAID 5 of four commodity SATA drives. I later doubled RAM (1 GB total) and the speed increase was at least double. I don't know if additional RAM would increase its performance further, but I'd recommend no less than 1 gig. On Sun, Jul 5, 2009 at 22:11, Stephen Vaughan stephenvaug...@gmail.comwrote: Hi all, In everyone's opinion, which resource(s) does backuppc rely on the most? cpu, memory or disk? -- Best Regards, Stephen -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- [EOM] -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Stephen Vaughan wrote at about 15:11:43 +1000 on Monday, July 6, 2009: Hi all, In everyone's opinion, which resource(s) does backuppc rely on the most? cpu, memory or disk? None of the above. I find network bandwidth typically most rate limiting. -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Hi, In everyone's opinion, which resource(s) does backuppc rely on the most? cpu, memory or disk? i've used / configured backuppc servers; minimal configuration was: - mini-itx with 1Ghz VIA CPU; - 1 GB ram; When possible OS (Debian) installed in - DOM (disk on module); When possible, more SATA channels with: - PCI Sata controller; When possible: - RAID1 disk configuration; With minimal configuration, medium network bandwidth usage (form BackupPC host status panel) was less than 5 Mb/s. I see that higher CPU Ghz guarantee better value. Regards M -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
okay network is one I left out, but I don't find that to be a problem. It rarely goes above 30mbit. Our setup is a single box w/ Xeon dual core 2.4ghz x 2, 2x300gig 10k scsi (raid 1) drives and 4gig memory, Debian 5.0, gigabit ethernet. We struggle to backup 4 servers each night, 453gigs / 7,807,318 files. Backup window is 9pm to 9am, and we just fit inside that each night. We do incrementals 6 nights a week and 1 full per week. I'm thinking of upgrading the box to raid5 array with 15k scsi drives, more memory and a quad core cpu. It's difficult to know what is enough though.. On Mon, Jul 6, 2009 at 4:27 PM, Jeffrey J. Kosowsky backu...@kosowsky.orgwrote: Stephen Vaughan wrote at about 15:11:43 +1000 on Monday, July 6, 2009: Hi all, In everyone's opinion, which resource(s) does backuppc rely on the most? cpu, memory or disk? None of the above. I find network bandwidth typically most rate limiting. -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen Vaughan wrote: okay network is one I left out, but I don't find that to be a problem. It rarely goes above 30mbit. Our setup is a single box w/ Xeon dual core 2.4ghz x 2, 2x300gig 10k scsi (raid 1) drives and 4gig memory, Debian 5.0, gigabit ethernet. We struggle to backup 4 servers each night, 453gigs / 7,807,318 files. Backup window is 9pm to 9am, and we just fit inside that each night. We do incrementals 6 nights a week and 1 full per week. I'm thinking of upgrading the box to raid5 array with 15k scsi drives, more memory and a quad core cpu. It's difficult to know what is enough though.. Given that you have a current system, you should monitor the various characteristics and once you have the information you will be able to clearly decide what/where to spend money on upgrades. Of course, if you upgrade *everything* then you will probably improve things... Upgrading RAM will improve the space for cache, depending on the number of files, and backup method (rsync/rsyncd) might help with that also. Upgrading disk storage to faster drives, more spindles, etc, will assist with IO, perhaps a better quality RAID card with cache should also improve write speeds... Upgrading the CPU from dual core to quad core, with the same cache/clock speed/etc, will improve performance if you are doing compression, calculating checksums, etc. Of course, you can spend a million dollars on upgrading the backup server, only to find that it still isn't any faster, because the disk IO on the client is maxed out, or the client is swapping to disk, etc... So, I really would suggest doing some careful measuring before you decide what to do next. One thing that can very broadly help determine if the bottleneck is the client or the backup server is to do one backup at a time. If this runs at the same speed as doing 4 at a time, then the bottleneck is on the client. Also, I found that limiting my under-spec backup server to 2 con-current backups at a time actually improved the performance, and allowed backups to complete faster compared to doing 4 at a time. Finally, unless a large amount of data is likely to be common between your four clients, it may be easier/simpler/etc to just split the load, have two backup servers, and each backup server backs up two clients. Perhaps others could comment on good ways to determine where a bottleneck is? Certainly checking to see how much swap space is used during the backups is a simple method to suggest you need more RAM. I assume there are values from vmstat which could indicate CPU is running out of grunt as well - what numbers are best to look at from here? Finally, for IO, I think vmstat will show blocks in/out from disk, but I don't think that is the best method. I think there is a tool iostat but have never used it... Hope you get more than 0.02c from all that, because it took me longer to type it up :) Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpR6noACgkQGyoxogrTyiWEyQCfen7AKLmryb6OaAc4NmVM5eeL 2WoAniU6BEFo+w8yBZwGbY34Q5IcHgb+ =S+M3 -END PGP SIGNATURE- -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Stephen Vaughan wrote: okay network is one I left out, but I don't find that to be a problem. It rarely goes above 30mbit. Our setup is a single box w/ Xeon dual core 2.4ghz x 2, 2x300gig 10k scsi (raid 1) drives and 4gig memory, Debian 5.0, gigabit ethernet. We struggle to backup 4 servers each night, 453gigs / 7,807,318 files. Backup window is 9pm to 9am, and we just fit inside that each night. We do incrementals 6 nights a week and 1 full per week. Have you done the usual efficiency-related things to the filesystem (mount with noatime,nodiratime,data=writeback, etc.), and made sure that the archive is excluded from the nightly updatedb runs? Also, sharing the filesystem with any app that might do an fsync() (mail programs, databases, etc.) is bad at least on ext3 because it waits to sync the entire filesystem buffer to disk instead of just the associated file data. -- Les Mikesell lesmikes...@gmail.com -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I think nodiratime is actually a subset of noatime, so that shouldn't make any difference, the data=writeback may help though... I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I don't recall, but I think you might need to wait for a couple of full backups before this will have an impact I've attached some graphs which might shed some light.. -- I'm not sure what they all mean, but if you do, then you could decide what needs to be upgraded. Also, as mentioned previously, you will need to look at similar graphs for each client to ensure they are not the limiting factor. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpSpmIACgkQGyoxogrTyiXpSQCfc5+2/Ta21swd5LHmxUzJDQgT GUoAn2MecWLA5MJGoNOXUuAUnWgJEyzP =WHmU -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Hardware considerations for building dedicated backuppc server
Hi all, In everyone's opinion, which resource(s) does backuppc rely on the most? cpu, memory or disk? -- Best Regards, Stephen -- ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
