Re: [BackupPC-users] usb slow for random access? (was Re: Using rsync for blockdevice-level synchronisation of BackupPC pools)
Hi Dan, On Fri, Sep 11, 2009 at 01:40:02PM -0400, Dan Pritts wrote: I'd say: Replace that USB 2.0 disk by something else like something connected via Firewire or eSATA. USB 2.0 is very, very slow, especially for random access. do you have empirical results that show this? I did not do benchmarks. It's just my personal experience that I've yet to see an USB-attached disk which feels fast. Remember: Disks do not speak USB, they are adressed via IDE or SATA. So, if you use USB, you get an additional translation layer. Apart from that it looks like USB is not optimized for fast transfer and low latency. SATA et al are designed for adressing hard disks, they don't care about input devices etc. So there is less overhead. Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Setting up a new BackupPC server
James Ward wrote: I'm setting up a new BackupPC server as my current one has gotten full. This system has 2G RAM, quad Intel(R) Xeon(TM) CPU 3.20GHz and a 3ware 6.5T array. I believe the array is currently RAID5 with no hot spare. From what I'm reading, RAID5 is a no-no as is ext3? What is the best way to set up the RAID array for BackupPC? In my personal experience, on our large arrays we use RAID5 with XFS filesystem. -- Jeremy Mann jer...@biochem.uthscsa.edu University of Texas Health Science Center Bioinformatics Core Facility http://www.bioinformatics.uthscsa.edu Phone: (210) 567-2672 -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Setting up a new BackupPC server
On Mon, Sep 14, 2009 at 11:42:42AM -0700, James Ward wrote: I'm setting up a new BackupPC server as my current one has gotten full. This system has 2G RAM, quad Intel(R) Xeon(TM) CPU 3.20GHz and a 3ware 6.5T array. I believe the array is currently RAID5 with no hot spare. From what I'm reading, RAID5 is a no-no as is ext3? What is the best way to set up the RAID array for BackupPC? The array for BackupPC should be tuned for access time, mostly. Therfore: As much disk heads as possible, disks as fast as possible. If you want to use few big disks, go for RAID-10 since it does not suffer the write-performance-impact of RAID-5. If you aim for more disks (like 10), you might want to use RAID-5 or -6. I've got a RAID-5 with only 3 disks and I'm regretting it a lot (just didn't find the time to buy that 4th disk and migrate to RAID-10 - I'll report if I did the move and what the results were). Also, ext3 doesn't seem to be a good choice. XFS and ZFS are recommended. (But ZFS is Solaris-only and needs lots of memory and CPU power itself.) Oh, BTW, consider upgrading RAM. It's very, very cheap these days and helps a lot for having the OS cache metadata (directories etc.). HTH, Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Two Hosts mutual exclusive?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi there, is there a chance to prevent two (or more...) hosts (or groups) being backed up at the same time? I have a couple of virtual machines residing on the same physical host. Sthey shouldn't be backed up together. Using the ping timeout doesn't work as there is enough bandwith on the network. But I don't want to reduce the storage bandwith with two (or more) backups at the same time. I know, I can set the maximum number of backups to 1 thus getting the result I want. But this would be valid for the independent other machines, too. So for those where I don't mind them being backed up at the same time. Any ideas? Or feature request for the next version? Greetings Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org/ iD8DBQFKrpQQ0XNIYlAXmzsRAs//AJ9e74JMRYGCrA/XSveOlH2D1DpaCACeL9L8 7cH5W+v9rbvf6ost+wBoK2A= =Vog3 -END PGP SIGNATURE- -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Setting up a new BackupPC server
I forgot to mention there are 16 disks in the big array. So you'd recommend RAID5 or 6? On Sep 14, 2009, at 11:58 AM, Tino Schwarze wrote: On Mon, Sep 14, 2009 at 11:42:42AM -0700, James Ward wrote: I'm setting up a new BackupPC server as my current one has gotten full. This system has 2G RAM, quad Intel(R) Xeon(TM) CPU 3.20GHz and a 3ware 6.5T array. I believe the array is currently RAID5 with no hot spare. From what I'm reading, RAID5 is a no-no as is ext3? What is the best way to set up the RAID array for BackupPC? The array for BackupPC should be tuned for access time, mostly. Therfore: As much disk heads as possible, disks as fast as possible. If you want to use few big disks, go for RAID-10 since it does not suffer the write-performance-impact of RAID-5. If you aim for more disks (like 10), you might want to use RAID-5 or -6. I've got a RAID-5 with only 3 disks and I'm regretting it a lot (just didn't find the time to buy that 4th disk and migrate to RAID-10 - I'll report if I did the move and what the results were). Also, ext3 doesn't seem to be a good choice. XFS and ZFS are recommended. (But ZFS is Solaris-only and needs lots of memory and CPU power itself.) Oh, BTW, consider upgrading RAM. It's very, very cheap these days and helps a lot for having the OS cache metadata (directories etc.). HTH, Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ Ward... James Ward Tekco Management Group, LLC jew...@torzo.com 520-290-0190x268 ICQ: 201663408 smime.p7s Description: S/MIME cryptographic signature -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Access Deny / Authorization Required
Hi, My Backuppc Server was working properly, but suddenly it started to deny my username. I tried to re-configure the password using the command htpasswd and nothing... Have anyone seen this problem before? BR, Simone -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] usb slow for random access? (was Re: Using rsync for blockdevice-level synchronisation of BackupPC pools)
USB is slower because a) there is an additional protocol translation to/from USB b) USB chipsets must hand off data to the CPU for processing which causes each piece of data to have additional latency going through the CPU once as raw USB packets to be translated by the driver and then again by whatever app is processing that data. SATA/SAS/IDE all have DMA so they can dump the usable data to memory and the CPU can process it once from there. c)because USB packets (for storage devices) are fairly simple packets to decode, its the mhz that matter as its how fast the packet can be pushed through. Improving controller design can only have a marginal impact on performance unless a high speed controller is used specifically for storage devices(i dont believe there are any on the market). d)USB devices rely on a driver to process the raw USB packets into scsi/ide/ata packets. SATA/IDE controllers require a driver only to read packets already in the appropriate format. More processing is done in the driver and software tends to have more latency that hardware. to break that down to a sign phrase. USB requires multiple levels of data processing to get the data delivered to the OS while specialize storage interfaces do most of the work in a hardware chip before handing data to the OS. On Mon, Sep 14, 2009 at 5:23 AM, Tino Schwarze backuppc.li...@tisc.dewrote: Hi Dan, On Fri, Sep 11, 2009 at 01:40:02PM -0400, Dan Pritts wrote: I'd say: Replace that USB 2.0 disk by something else like something connected via Firewire or eSATA. USB 2.0 is very, very slow, especially for random access. do you have empirical results that show this? I did not do benchmarks. It's just my personal experience that I've yet to see an USB-attached disk which feels fast. Remember: Disks do not speak USB, they are adressed via IDE or SATA. So, if you use USB, you get an additional translation layer. Apart from that it looks like USB is not optimized for fast transfer and low latency. SATA et al are designed for adressing hard disks, they don't care about input devices etc. So there is less overhead. Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.craniosacralzentrum.de -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Two Hosts mutual exclusive?
Christian Völker wrote: is there a chance to prevent two (or more...) hosts (or groups) being backed up at the same time? I have a couple of virtual machines residing on the same physical host. Sthey shouldn't be backed up together. Using the ping timeout doesn't work as there is enough bandwith on the network. But I don't want to reduce the storage bandwith with two (or more) backups at the same time. I know, I can set the maximum number of backups to 1 thus getting the result I want. But this would be valid for the independent other machines, too. So for those where I don't mind them being backed up at the same time. Any ideas? Or feature request for the next version? You can set the BlackoutPeriods so the allowed times don't overlap. Or it might be good enough to force a run at a time you'd like and let the near 24hr scheduling keep the cycle. -- Les Mikesell lesmikes...@gmail.com -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Setting up a new BackupPC server
I forgot to mention there are 16 disks in the big array. So you'd recommend RAID5 or 6? I would always recommend NOT using RAID5 or 6. I believe that those two raid levels were designed for a time when disk storage was expensive. by your numbers (6.5TB in raid 5 with 16 drives) My math says you have 500GB drives. I would highly suggest RAID10. There are a few ways to handle this but here is my recommendation. 8 raid1 arrays giving 8x500GB pairs. then raid0 the 8 pairs for 4TB. Alternatively, A hot spare or 2 is a lifesaver. use 6 raid1 pairs stacked in raid0 for 3TB and 2 hot spare disks. I am assuming you have 16 drive trays. That would give you 14 spindles, 7 of which are active and in a raid0 you will get a very nice improvement in I/O performance. You can also break down the raid1 in difference chunks like 4 disk raid1 sets and raid0 the 3 arrays but you will have less active heads (only 3) as all other heads will be writing redundant info. This is slightly safer because you can tolerate 2 drive failure simultaneously while in the first scenario you can tolerate 1 and then the array will immediately go into rebuild. As far as filesystem. I really like EXT3 because it is very very reliable but it is not the fastest thing out there. It also has very a long FSCK process which is not fun on a multi-TB array. XFS is a very good performer for backuppc and it is current, stable, and maintained and likely the best choice. ZFS is platform dependant and hungry for resources BUT is very very fast when tuned properly and can handle your volume management. It also does raidz/raidz2 faster than many hardware cards and is less impacted by parity calculations than standard raid5/6. ZFS is a beast of a filesystem and you need to balance the pros vs the cons. I suggest XFS. Pretty good performance and pretty much none of ZFS's negatives(resource hungry, solaris only) good luck -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Access Deny / Authorization Required
Hi, On Mon, Sep 14, 2009 at 15:17, Simone S. Santiago sim...@magistrainfo.com.br wrote: My Backuppc Server was working properly, but suddenly it started to deny my username. I tried to re-configure the password using the command htpasswd and nothing... Have anyone seen this problem before? It's really hard to diagnose the problem considering the information you are giving... I suggest you check your Apache logs, the location of which will depend on which Linux distribution you are using, to determine what might be going wrong with your authentication. If you are using Fedora/CentOS/RHEL the logs should be at /var/log/httpd/access_log and /var/log/httpd/error_log, you might try to run the tail -f /var/log/httpd/*_log command and then try to login to your server to see what will be logged there while you do that. If that still does not help you solve your problem, then please post back to the list with more information, such as which Linux distribution and version are you using, which version of BackupPC you have installed, a snippet of the relevant Apache configuration, a ls -l of the file with the passwords (to check for permission issues), what you found on the Apache logs, if there is anything weird in /var/log/messages or any other relevant logs, etc. HTH, Filipe -- Come build with us! The BlackBerryreg; Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9#45;12, 2009. Register now#33; http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Client with dynamic IP, backups cut off at 24h
Hi, schmoove wrote on 2009-09-14 06:02:16 -0400 [[BackupPC-users] Client with dynamic IP, backups cut off at 24h]: [...] the backups fail and soon start over from scratch. I have noticed this happening for 5 days when somehow backuppc managed to overcome this 24h problem by itself and was able to resume failed backups all of a sudden. Is this just coincidence, or why do I see these two different behaviors? failing full rsync(d) backups result in a partial backup which can be resumed (i.e. restarted, taking the already transferred data into account). Failing incremental backups (or full backups with other XferMethods) are discarded. Are your incremental backups really taking more than 24 hours, or did your full backup simply take 5 days to complete (being interrupted and restarted four times)? Can you trust backups that seem to regularly take so long? If this doesn't apply, you need to give more details on your configuration and setup. If it does, you probably do, anyway. Anything I can do to tweak backuppc to always resume incomplete backups? Only do full backups? That is probably not a solution. You seem to have a basic problem which needs solving (too much regularly changing data over too little bandwidth or wrong choice of XferMethod). Regards, Holger -- Come build with us! The BlackBerryreg; Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9#45;12, 2009. Register now#33; http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Setting up a new BackupPC server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 James Ward wrote: I'm setting up a new BackupPC server as my current one has gotten full. This system has 2G RAM, quad Intel(R) Xeon(TM) CPU 3.20GHz and a 3ware As someone else mentioned, I would strongly suggest upgrading the RAM to at least 4G if not 8G... This will be very helpful when using rsync[d] for backups to multiple targets, especially if some have a large number of small files. Also very useful as cache for the filesystem... Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkqu3agACgkQGyoxogrTyiWtsgCfWNZUNGJGNcAOtxaX6D+3Pam6 2jsAni0AqzO2n8IUQFVLF5DUQ9BNT4ym =EeNW -END PGP SIGNATURE- -- Come build with us! The BlackBerryreg; Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9#45;12, 2009. Register now#33; http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Client with dynamic IP, backups cut off at 24h
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Holger Parplies wrote: Hi, schmoove wrote on 2009-09-14 06:02:16 -0400 [[BackupPC-users] Client with dynamic IP, backups cut off at 24h]: [...] the backups fail and soon start over from scratch. I have noticed this happening for 5 days when somehow backuppc managed to overcome this 24h problem by itself and was able to resume failed backups all of a sudden. Is this just coincidence, or why do I see these two different behaviors? failing full rsync(d) backups result in a partial backup which can be resumed (i.e. restarted, taking the already transferred data into account). Failing incremental backups (or full backups with other XferMethods) are discarded. Are your incremental backups really taking more than 24 hours, or did your full backup simply take 5 days to complete (being interrupted and restarted four times)? Can you trust backups that seem to regularly take so long? If this doesn't apply, you need to give more details on your configuration and setup. If it does, you probably do, anyway. Anything I can do to tweak backuppc to always resume incomplete backups? Only do full backups? That is probably not a solution. You seem to have a basic problem which needs solving (too much regularly changing data over too little bandwidth or wrong choice of XferMethod). Regards, Holger Or perhaps it was a one-off (never only happens once, but rare event) where someone changed a large amount of data at once. This would cause the failing incremental backups, and eventually a full backup with the help of partials might complete. The big problem is where you have a single large file which takes more than 24 hours to backup, because the partially transferred file will be deleted and start again in each backup attempt I'd suggest you review your logs and find out what happened, and then see if you think it will be a regular problem, and if it needs to be solved... Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkqu5jwACgkQGyoxogrTyiXQWgCfdcRlunTqKWbJLLmZX7ZLSlGg DVkAoLIAPLAKC39+1Q/6zO9UuQYT9wFp =dOM3 -END PGP SIGNATURE- -- Come build with us! The BlackBerryreg; Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9#45;12, 2009. Register now#33; http://p.sf.net/sfu/devconf ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
