On Apr 18, 2008, at 11:32 AM, Tony Schreiner wrote: > And yes, I struggle with what needs be backed up. The users > (bioinformatics research) can generate a couple of 100 GB of data > every day, some of it very large files, some of it hectathousands of > small files, some of which needs to be saved, some of which does not. > There is no easy way for me to predict yes or no.
How much memory does the server have? Just a guess, but especially if you are using rsync, you could be running into problems with the size of the "list" that rsync has to maintain in memory, and rsync could spend all of its time paging. "top" for e.g. should give you some idea. I would recommend using 64-bit OS and max out the memory if possible, that should help unless something else is going on. Also the BackupPC server(s) should be dedicated for that purpose and not running other jobs (my prior experience with bioinformatics is they like to squeeze CPU cycles from any box they can get their hands on). Second, take a look at aggregation and backplane speed on the network switches, maybe links are getting saturated somewhere, or you need to do some trunking, make sure your link speeds are really Gb/s from end to end, try some large file transfers to see how much throughput you really get from point to point. Take a look at the network interface stats e.g. from ifconfig to see if any of the numbers look awry, double check duplex settings and for MTU consistency, any routers or firewalls in the path. I am sure you already know all that, but once in awhile I know myself I forget to check the things that are quick and easy to check and would be easy to fix, very little time wasted if that turns out not to be relevant. If possible, you may want to have a parallel "storage network" connected to extra NIC card in each server to get maximum bandwidth for backups--if you make that secure and separate from the regular LAN, then you don't need to e.g. tunnel through SSH, that will save you some overhead if you are encrypting right now. I take it for granted that you are using some type of hardware RAID or a SAN to get better I/O throughput on the server, you would pretty much have to be doing that for your capacity. Jonathan ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/