Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
The box isn't used for anything else..I can keep an eye on it tonight and see whats causing the load in those first few hours.. On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] MySQL dump backup example scripts request
Admiral Beotch пишет: I use a cron job that backups my databases on a daily/weekly basis and then depend on backuppc to grab those static files - the backup of mysql is not dependant on the DumpPreUserCmd in backuppc. The command I have cron'd is: mysqldump --opt --all-databases -u root | gzip -9 /backups/mySQL-AllDB.sql.gz I think, that is bad idea to run mysqldump and pipe its to gzip. because mysql stay locked while gzip working -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
going by the graphs it seems to be an IO issue, has anyone tried running raid 10 with backuppc? On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Stephen Vaughan wrote: The box isn't used for anything else..I can keep an eye on it tonight and see whats causing the load in those first few hours.. On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. The load and waiting on I/O is definitly suspicious. Maybe something like a cronjob running updatedb for locate ? /etc/updatedb.conf holds the paths that should be excluded if you don't want to turn it off completely. As they are both doing a lot of disk activity it will slow down rsync. I suggest having a good look at syslog looking for CRON and/or check: /etc/crontab /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /var/spool/cron/ -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
iostat is also showing iowait fluctuating between 50-75 % -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
I definitely don't have updated installed on the box. I'm monitoring it the moment and it seems BackupPC_nightly is running while performing a backup of 2 servers, it must run run backuppc_nightly for the first few hours every night. Output from top, CPU usage is in the 9th column. 12269 backuppc 20 0 117m 114m 2364 D 55 2.9 2:47.40 BackupPC_dump 12270 backuppc 20 0 226m 216m 2356 D 35 5.5 1:22.66 BackupPC_dump 13538 backuppc 20 0 185m 176m 1324 D 29 4.5 0:25.90 BackupPC_dump 12264 backuppc 20 0 6 8076 2060 D 10 0.2 0:22.06 BackupPC_nightl 12262 backuppc 20 0 11136 8092 2060 D5 0.2 0:25.46 BackupPC_nightl 12267 backuppc 20 0 11160 8112 2060 D4 0.2 0:23.40 BackupPC_nightl 13109 backuppc 20 0 106m 101m 1324 S3 2.6 0:52.04 BackupPC_dump 12268 backuppc 20 0 11024 8088 2060 D2 0.2 0:23.84 BackupPC_nightl 12261 backuppc 20 0 11148 8120 2072 D1 0.2 0:24.46 BackupPC_nightl 12265 backuppc 20 0 11136 8088 2060 D1 0.2 0:23.90 BackupPC_nightl 10 root 15 -5 000 S1 0.0 38:30.83 ksoftirqd/2 182 root 15 -5 000 S1 0.0 16:12.28 kblockd/0 351 root 15 -5 000 S1 0.0 1296:21 kswapd0 12263 backuppc 20 0 11132 8112 2060 D1 0.2 0:21.20 BackupPC_nightl 12266 backuppc 20 0 11024 8084 2060 D1 0.2 0:24.10 BackupPC_nightl On Tue, Jul 7, 2009 at 6:20 PM, Leen Besselink l...@consolejunky.netwrote: Stephen Vaughan wrote: The box isn't used for anything else..I can keep an eye on it tonight and see whats causing the load in those first few hours.. On Tue, Jul 7, 2009 at 1:57 PM, Les Mikesell lesmikes...@gmail.com mailto:lesmikes...@gmail.com wrote: Stephen Vaughan wrote: I'm only running 2 concurrent backups. The disk is/was mounted with noatime,data=journal. I've changed this to noatime,nodiratime,data=writeback I've also just enabled the checksum-seed option in rsyncargs. I have munin running on the box, it looks as though there is a bit of iowait on the cpu, iostat doesn't look too bad to me, but I might be reading it wrong.. I've attached some graphs which might shed some light.. The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). Do you have other things running that should be active at this time? Idle daemons wouldn't count because they should be blocked waiting for input. The load and waiting on I/O is definitly suspicious. Maybe something like a cronjob running updatedb for locate ? /etc/updatedb.conf holds the paths that should be excluded if you don't want to turn it off completely. As they are both doing a lot of disk activity it will slow down rsync. I suggest having a good look at syslog looking for CRON and/or check: /etc/crontab /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/ /var/spool/cron/ -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen Vaughan wrote: I definitely don't have updated installed on the box. I'm monitoring it the moment and it seems BackupPC_nightly is running while performing a backup of 2 servers, it must run run backuppc_nightly for the first few hours every night. That is definitely the first thing you should change, set the nightly job to run at 10am outside the backup window. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpTMVIACgkQGyoxogrTyiWIjACgic2MncLhz9i5Zn88xCAAHkQ4 qF8An1nz2Aed2TSAjDRrCSJlFV3oryIY =+TIB -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
yep ok I've just done that. On Tue, Jul 7, 2009 at 9:28 PM, Adam Goryachev mailingli...@websitemanagers.com.au wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephen Vaughan wrote: I definitely don't have updated installed on the box. I'm monitoring it the moment and it seems BackupPC_nightly is running while performing a backup of 2 servers, it must run run backuppc_nightly for the first few hours every night. That is definitely the first thing you should change, set the nightly job to run at 10am outside the backup window. Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpTMVIACgkQGyoxogrTyiWIjACgic2MncLhz9i5Zn88xCAAHkQ4 qF8An1nz2Aed2TSAjDRrCSJlFV3oryIY =+TIB -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Best Regards, Stephen -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] MySQL dump backup example scripts request
Good call! I didnt notice this because I'm only backing up 60M of data, which took about 90 seconds. A significantly larger database would have been killed with this. By separating the commands, everything completes within 30 seconds - and that's even with an additional PGP command! Thanks! -- https://ssd.eff.org http://video.google.com/videoplay?docid=8167533318153586646 http://www.vlrc.org/articles/160.html On Mon, Jul 6, 2009 at 23:46, Boltris Alexandr ua2...@gmail.com wrote: I think, that is bad idea to run mysqldump and pipe its to gzip. because mysql stay locked while gzip working -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Hi,
On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote:
The only thing that seems slightly strange in the graphs is the load average
going to 12 as the backups start and staying there a couple of hours.
Normally
that's the average number of 'other' processes that are waiting for CPU but
otherwise runnable (i.e. not themselves blocked on i/o).
I used to think that, but in fact processes that are blocked in disk
i/o (the ones in D state) do count in load average. So the load
average of 12 in this case probably means processes writing to the
disk.
From another e-mail showing several (12?) processes BackupPC_dump and
BackupPC_nightly in D state, those should be the reason why this box
stays with a load average of 12 during the night.
From those observations, it seems to me that the bottleneck in this
case is disk I/O.
Stephen, it seems strange to me that there are 8 BackupPC_nightly
processes running, have you increased $Conf{MaxBackupPCNightlyJobs}
from 2 to 8?
I would suggest you start by setting $Conf{MaxBackupPCNightlyJobs}
back to 2 or even to 1. If you set it to 1 and it can't finish its job
in 24h, then increase $Conf{BackupPCNightlyPeriod} to 2 or 4 so that
only 1/2 or 1/4 of the pool is processed each night.
HTH,
Filipe
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] Multiple backuppc server
Hi All,
We've started to setup a large multiple server backuppc environment, and wanted
a few thoughts/ideas/advice.
We've got a large 2TB nas at the back of it with gig connectivity.
Filesystem is LVM on top of OCFS2 so we have multiple front-end servers with
read/write.
Backuppc on each host is setup with relevant different hostnames and setup
separate logdirectories. The actual top/backup location is shared on the main
nas store.
So
$Conf{TopDir} = /backups/backuppc/
$Conf{ConfDir} = '/etc/backuppc';
$Conf{LogDir} = '/backups/backup02';
$Conf{InstallDir} = '/usr/share/backuppc';
$Conf{CgiDir} = '/usr/share/backuppc/cgi-bin';
Each server has its own list of hosts in /etc/backuppc/hosts as that's how I'm
splitting the job queues. i.e. a host only exists in one server hosts file (at
present either backup01 or backup02).
Can anyone see any pitfalls with this? The only strange thing I've noticed is
with the trashClean process, it seems to be trying to clean things that the
other server is creating/working on and failing with Can't read
/var/lib/backuppc/trash//home/blah/thing/file: No such file or directory.
It doesn't seem a major thing so I'm ignoring it for now!
Anyone see any pitfalls/problems with what I'm doing here?
Cheers!
Andy
This message (and any associated files) is intended only for the use of the
intended recipient(s) and may contain information that is confidential, subject
to copyright or constitutes a trade secret. If you have received this message
in error, please contact Onyx Group immediately by replying to the message and
deleting it from your computer. Any content included in this mail does not
necessarily reflect the views of The Onyx Group.
Onyx Group Ltd, Registered in England Registration Number: 5682619, VAT Number:
884255494
Registered Office: Aurora Court, Barton Road, Middlesbrough, TS2 1RY.
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Multiple backuppc server
Hi, On Tue, Jul 7, 2009 at 08:50, Andy Browna...@onyx.net wrote: Anyone see any pitfalls/problems with what I'm doing here? Yes, IMHO you are introducing complexity where it is not needed. What are you trying to accomplish? Fault-tolerance? In that case you should probably have an active/passive deployment where the backup host is only used when the master fails. Load-balancing for compressing which is CPU intensive? In that case you should probably split your storage in two 1TB LUNs, export one of them to each host and run BackupPC over a regular ext3 (or XFS) filesystem. OCFS or any other cluster filesystem will only introduce overhead and management complexity, and from your description it seems you are not sharing files between hosts anyway. HTH, Filipe -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] https for zimbra
It's not up, we should get it up and running. -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] https for zimbra
On Tue, Jul 7, 2009 at 06:12, Andrew Libby ali...@xforty.com wrote: It's not up, we should get it up and running. What? -- [EOM] -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Filipe Brandenburger wrote:
On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote:
The only thing that seems slightly strange in the graphs is the load average
going to 12 as the backups start and staying there a couple of hours.
Normally
that's the average number of 'other' processes that are waiting for CPU but
otherwise runnable (i.e. not themselves blocked on i/o).
I used to think that, but in fact processes that are blocked in disk
i/o (the ones in D state) do count in load average. So the load
average of 12 in this case probably means processes writing to the
disk.
That must be a Linux quirk (bug?) but it does explain some numbers I've
seen. Regardless, there shouldn't be that many things running.
Stephen, it seems strange to me that there are 8 BackupPC_nightly
processes running, have you increased $Conf{MaxBackupPCNightlyJobs}
from 2 to 8?
I would suggest you start by setting $Conf{MaxBackupPCNightlyJobs}
back to 2 or even to 1. If you set it to 1 and it can't finish its job
in 24h, then increase $Conf{BackupPCNightlyPeriod} to 2 or 4 so that
only 1/2 or 1/4 of the pool is processed each night.
More importantly, BackupPC_nightly shouldn't overlap with backup jobs if
possible.
--
Les Mikesell
lesmikes...@gmail.com
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/blackberry
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Hi, Les Mikesell wrote on 2009-07-07 12:17:56 -0500 [Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server]: Filipe Brandenburger wrote: On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote: The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). I used to think that, but in fact processes that are blocked in disk i/o (the ones in D state) do count in load average. So the load average of 12 in this case probably means processes writing to the disk. That must be a Linux quirk (bug?) but it does explain some numbers I've seen. if it is, it's inherited at least from SunOS (and HP-UX, if I remember correctly). I haven't been using Solaris for quite a while, so I can't say if the load on NFS clients still goes up when NFS servers go down. SunOS 5.10 w(1) documents the load to mean average number of jobs in the run queue, which should *not* include processes waiting for I/O. Probably a Solaris quirk (bug?) though. Both ways of defining load make sense. Processes waiting for short term disk I/O are using resources (and would probably be running if the disk was simply faster). NFS I/O is not necessarily short term, but that's a different matter. Linux uptime(1) documents what system load means on Linux. Wherever it matters, you won't be looking at a single figure to measure your system's state anyway. Regardless, there shouldn't be that many things running. Yes, running BackupPC_nightly outside the backup window (as has already been agreed upon) is definitely important. Regards, Holger -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Users managing their backups themself?
Hello, I added the htpasswd users with passwords, and they each have their hosts configured, but when they log in they get a page with a blank report and no possible options, not even starting their own backups. I even tried to check all possible checkbox in the cgi configuration to be sure I didn't miss anything. Phil Les Mikesell wrote: error403 wrote: Hi, Is there a way to give each user that logs into the backuppc interface to start a full/incremental backup by themself? Like that, if a user is about to make a big change to its computer he can make a backup before doing anything wrong. Using backuppc 3.0 on Ubuntu, backups are done locally using tar over sshfs. That should be the default if you have specified 'owners' in the hosts file and configured web logins and passwords for the same names. Each login other than the administrators should only see the hosts they 'own'. If you've made everything look like one machine instead of accessing the hosts directly, you'll have to make hosts entries that use the alias feature to handle them in the normal way. -- Les Mikesell lesmikesell at gmail.com -- ___ BackupPC-users mailing list BackupPC-users at lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ +-- |This was sent by krunchyf...@videotron.ca via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
Holger Parplies wrote: Les Mikesell wrote on 2009-07-07 12:17:56 -0500 [Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server]: Filipe Brandenburger wrote: On Mon, Jul 6, 2009 at 23:57, Les Mikeselllesmikes...@gmail.com wrote: The only thing that seems slightly strange in the graphs is the load average going to 12 as the backups start and staying there a couple of hours. Normally that's the average number of 'other' processes that are waiting for CPU but otherwise runnable (i.e. not themselves blocked on i/o). I used to think that, but in fact processes that are blocked in disk i/o (the ones in D state) do count in load average. So the load average of 12 in this case probably means processes writing to the disk. That must be a Linux quirk (bug?) but it does explain some numbers I've seen. if it is, it's inherited at least from SunOS (and HP-UX, if I remember correctly). I haven't been using Solaris for quite a while, so I can't say if the load on NFS clients still goes up when NFS servers go down. SunOS 5.10 w(1) documents the load to mean average number of jobs in the run queue, which should *not* include processes waiting for I/O. Probably a Solaris quirk (bug?) though. Both ways of defining load make sense. Processes waiting for short term disk I/O are using resources (and would probably be running if the disk was simply faster). NFS I/O is not necessarily short term, but that's a different matter. It doesn't make sense to me to consider a process runnable when it is waiting for a hardware operation to complete - the scheduler should be ignoring it. I suppose if the disk in question is an IDE that the CPU has to micro-manage it might make sense to blame the application for the CPU use even if the kernel is doing it. Linux uptime(1) documents what system load means on Linux. Wherever it matters, you won't be looking at a single figure to measure your system's state anyway. Yes, the load average in mostly just useful to tell you if a faster CPU would help, but it isn't even good for that if it counts things that couldn't use the CPU anyway. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Users managing their backups themself?
error403 wrote: Hello, I added the htpasswd users with passwords, and they each have their hosts configured, but when they log in they get a page with a blank report and no possible options, not even starting their own backups. I even tried to check all possible checkbox in the cgi configuration to be sure I didn't miss anything. If you edit the hosts file, does the name in the 'user' column exactly match the web login name (upper/lower case included)? That is what is supposed to determine the hosts they see. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] OS X failing to back up
Ted To wrote: Hi, I'm not sure what the problem is but after reinstalling OS X on my wife's laptop, a backup has never been successfully completed. I did not change any of the configuration files and am using xtar as suggested at: http://wiki.nerdylorrin.net/wiki/Wiki.jsp?page=BackupPC . I put the error log of at: http://pastebin.com/m1009bc48 . I'm running an oldish version of backuppc (2.1.2 on a dapper drake box). Any help will be appreciated. Based on these bits from the error log... Running: /usr/bin/ssh -q -x -n -l root snoopy /usr/bin/env LC_ALL=C /usr/bin/nice -n 20 /usr/bin/xtar -c -v -f - -C /Users --totals --exclude=./louise/Documents/Parallels . Xfer PIDs are now 8958,8957 [SNIP] Connection to snoopy closed by remote host. [SNIP] Backup aborted (lost network connection during backup) Running: /usr/bin/ssh -q -x -n -l root snoopy /usr/bin/env LC_ALL=C /usr/bin/nice -n 20 /usr/bin/xtar -c -v -f - -C /Users --totals --exclude=./louise/Documents/Parallels . Xfer PIDs are now 24340,24339 [SNIP] Read from remote host snoopy: No route to host ...you seem to have a network connectivity/reliability problem. Thanks, Ted To Chris -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] BackupPC and File::RsyncP and Akamai
I'm attempting to back up some files off Akamai Netstorage (a remote fileshare similar to EC2), which has a rsync-over-ssh interface. They support protocol version 29, and rsync 2.6.6. It appears that File::RsyncP only supports protocol version 28, and something is going wonky. Here's the output of the log: full backup started for directory / Running: /usr/bin/ssh -C -q -i /var/lib/backuppc/.ssh/id_dsa_akamai -x -l sshacs host /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group --links --times --block-size=2048 --recursive --ignore-times . / Xfer PIDs are now 23993 Got remote protocol 29 Negotiated protocol version 28 Remote[98]: n junk junk appears to be a list of the files, with ugly binary breaks between -- I'm guessing it's the raw dump of the protocol list as rsync sends it. I suspect a problem on Akamai's implementation with the way that it negotiates protocol 28. First of all - is there a version of File::RsyncP that supports protocol version 29? If not - is it possible to configure BackupPC to use rsync itself instead of using the Perl modules? Or is that a hard requirement for BackupPC? Appreciate any thoughts - thanks! -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC and File::RsyncP and Akamai
On 07/07 03:20 , Nate Carlson wrote: First of all - is there a version of File::RsyncP that supports protocol version 29? doesn't look like there's a public release of it, if it does exist. If not - is it possible to configure BackupPC to use rsync itself instead of using the Perl modules? Or is that a hard requirement for BackupPC? No. Craig can answer this in more detail; but it does not look like it's possible to do what you want, with BackupPC. Perhaps rsnapshot would be more appropriate to what you need? http://rsnapshot.org/ Alternatively, dump your Akamized files to some computer somewhere, and let BackupPC backup that. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC and File::RsyncP and Akamai
Nate Carlson wrote: I'm attempting to back up some files off Akamai Netstorage (a remote fileshare similar to EC2), which has a rsync-over-ssh interface. They support protocol version 29, and rsync 2.6.6. It appears that File::RsyncP only supports protocol version 28, and something is going wonky. Here's the output of the log: full backup started for directory / Running: /usr/bin/ssh -C -q -i /var/lib/backuppc/.ssh/id_dsa_akamai -x -l sshacs host /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group --links --times --block-size=2048 --recursive --ignore-times . / Xfer PIDs are now 23993 Got remote protocol 29 Negotiated protocol version 28 Remote[98]: n junk junk appears to be a list of the files, with ugly binary breaks between -- I'm guessing it's the raw dump of the protocol list as rsync sends it. I suspect a problem on Akamai's implementation with the way that it negotiates protocol 28. First of all - is there a version of File::RsyncP that supports protocol version 29? If not - is it possible to configure BackupPC to use rsync itself instead of using the Perl modules? Or is that a hard requirement for BackupPC? Appreciate any thoughts - thanks! Considering how cheap disk space is, the easy approach is probably to update a full copy locally with a stock rsync, then let backuppc back that up to keep a history. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC and File::RsyncP and Akamai
Hi, Carl Wilhelm Soderstrom wrote on 2009-07-07 15:43:30 -0500 [Re: [BackupPC-users] BackupPC and File::RsyncP and Akamai]: On 07/07 03:20 , Nate Carlson wrote: First of all - is there a version of File::RsyncP that supports protocol version 29? doesn't look like there's a public release of it, if it does exist. I don't think there is, and I doubt there ever will be - if Craig updates File::RsyncP, then I'd guess he'll update it to protocol version 30. While rsync is supposed to be downward compatible to lower protocol versions, I don't know if File::RsyncP is. If not - is it possible to configure BackupPC to use rsync itself instead of using the Perl modules? No, it is not possible. The whole purpose of File::RsyncP is to make rsync compatible with the BackupPC pool storage format - stock rsync doesn't understand pooling, the compressed file format, or file name mangling. tar and smbclient output a tar stream to stdout, where BackupPC can pick it up and create files in the pool. rsync obviously cannot do that. It needs access to the local files in order to compare file lists and speed up transfers. Since these files aren't stored verbatim on disk, you need glue code for accessing them pretty much inside the rsync protocol implementation. As I understand it, File::RsyncP allows any storage backend (implemented in Perl) to be integrated without modifying the rsync code. That suggests that updating it to a new rsync version would probably mean re-implementing it (but I'm just guessing). Does creating a local copy with rsync 3.0 work? What if you force the protocol to 28 (--protocol=28)? Are you sure the output is generated by rsync and not (partly) by some rc files? Can you log in with ssh and run anything except rsync? Perhaps 'man rsync'? :) Maybe you can switch off the output somehow ... you could try things like --out-format='' or --quiet if you can't find anything more appropriate (and I'd test with command line rsync first, not with BackupPC). Regards, Holger -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
I have been using backuppc for quite some time. Here are some of the thinks I have found. 1) backup method is super important. If your backup machine is in near the servers it is going to backup, then consider NOT using rsync. The disadvantage is that you will hammer your network but this can be overcome with a coupe gigabit network cards and a basic gigabit switch creating a backup only network. I suggest this because backuppc is IO bound and the way that rsync checks files is more intensive than doing a tar backup. Using a direct file copy method will still check mod times but will not scan the file list rsync does. You will get to take better advantage of sequential reads and writes rather than random reads and writes. 2) A 2Ghz Dual Core CPU with 4GB of ram seams to be about the peak of efficiency. Other factors will limit backup speed long before you fill up 4GB of ram or use all of the CPU. 3) hard disk performance is the #1 thing. RAM only helps compensate so much by giving extra cache memory. Get a real RAID card. software RAID is great for some things but the time it takes for data to travel accross the system bus twice (to cpu and from cpu) is much greater than the time it takes to go to the storage controllers CPU and back. Get a controller than has 128MB or more of cache on board and supports whatever RAID mode you want. 4) RAID1. Skip RAID5, do RAID1. This data is important and the speed that you back it up is important. A few extra dollar on some redundant hard disks is a bargain. If you need more than 2 drives, consider RAID1 pairs with RAID0 on top. 5) Hot spare. When a drive fails, you want the rebuild to start immediately and get done. That is a window for data loss. The only way around this without moving to a parity based RAID5 or 6 is to do triple drive redundancy(RAID1 with 3 disks) 6) secondary backup server. You can mirror you whole array with rsync though it will be slow and use a ton of RAM. What I have done on smaller setups is to actually put my RAID1 device md0 in a RAID1 of md1 that is made of md0 and a AoE disk on another server. You can do some scripts to add the AoE disk to the array until it is completely synced up and then remove it. If you dont remove it, then the md1 raid device will be slow as it will stop to sync up the compontent devices every so often. You can use this same setup to sync to an external USB drive by adding it to the array for a sync and removing it afterwards. AoE = ATA over Ethernet and it has great sequential reads and write on ethernet. I have had 93% efficient transfers which means that I can get up to 116MB/s on Gigabit as there is no TCP/IP overhead, just straight ethernet frames. A RAID1 rebuild is sequential but there is some CPU overhead in either the system CPU or the RAID controller. I have managed over 80MB/s rebuild speed on a PCI-Express based server on Intel Pro Gigabit cards and a Cisco catalyst switch with AoE. Expect that to half if you use lesser hardware. AoE only works on local ethernet as ethernet frames do not travel beyond a managed switched ethernet, which means that you cannot router AoE over the internet or over a VPN. -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] [OT] Linux load values (was: Re: Hardware considerations for building dedicated backuppc server)
Hi, Les Mikesell wrote on 2009-07-07 13:45:11 -0500 [Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server]: It doesn't make sense to me to consider a process runnable when it is waiting for a hardware operation to complete - I never said a process in state D is in state R. I was talking about the load average, as were you in the thread before. the scheduler should be ignoring it. It is. I suppose if the disk in question is an IDE that the CPU has to micro-manage it might make sense to blame the application for the CPU use even if the kernel is doing it. Are you talking about PIO? You really should get your system to use UDMA. ;-) No, my point was, the load average is an attempt to fit the state of a system into one single number (which, as we've agreed, is only good for getting a quick impression, nothing more). I'm sure I don't have to tell you that a lot of disk activity will make a system more unresponsive than a lot of processes competing for the CPU. So why just ignore that fact in the single number? We don't have to look far for a practical example. The original poster had a load average of 12 on his system, which fit in with his observation that his machine was heavily loaded, as you might say. The single number indicated that something was probably not as it should be. Without taking D processes into account, the load average might have been, say, 0.5. Would that have done the real situation more justice? Wasn't it exactly the load average, computed the way it is, that pointed at where to look for the problem? In what state are processes that are waiting for a page fault to complete? They're obviously active (as in wanting to run), but not runnable. You'd argue that a faster CPU wouldn't help, because they can't run anyway. I'd argue that they're part of what is going on (or trying to) on the system. Let's just disagree on that, ok? Yes, the load average in mostly just useful to tell you if a faster CPU would help, but it isn't even good for that if it counts things that couldn't use the CPU anyway. I never looked at the load average that way. Where I need to make that decision, it wouldn't work that way in either case, but I can see that it could for some people. I'm not sure that was the original intention of whoever thought up the load average, though. Regards, Holger -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Hardware considerations for building dedicated backuppc server
dan wrote: 1) backup method is super important. If your backup machine is in near the servers it is going to backup, then consider NOT using rsync. Or, if you like the mirroring ability of rsync but don't want or need to use the special rsync algorithm, use rsync with the '-W' option. Jon -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
