Re: [BackupPC-users] Backups tooo slow over WAN - how to split them
On 03/15 03:44 , Linux Punk wrote: You could just keep the one machine entry, but use the exclude options to only back up a small portion of the machine. Once that completes gradually remove directories from the excluded list until the full backup completes fine. This assumes that most of your data is static and that any files that change can be easily backed up in a daily window. This is what I've done in the past when making the first backup of a really big machine over a really slow link. Sometimes it takes a few weeks (!) to get the data moved at first (unless users don't mind their link being crushed during the day, which they usually do). -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backups tooo slow over WAN - how to split them
On 3/15/2011 3:49 PM, David Herring wrote: I'm trying to backup windows servers with approx 3 partitions of 100G each over a WAN link. This takes 'days' to run and never successfully completes. I'm using rsyncd on the windows machines - and the backup is to a ubuntu server. So, how do you split the backups from a single machine - do you have to create multiple host entries for each machine ? Does this make restore too painful ? Is there anything else I should be doing ? MTU size ? You can split the backups by directories. Just create separate backup hosts that each backup some of the directories. Give each host slightly different names and use the ClientNameAlias setting to point them back to the proper host name. Keep in mind that this may result in the separate backups running simultaneously. To prevent this you can use the DumpPreUserCmd and DumpPostUserCmd to set and check for lock files. -- Bowie -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backups tooo slow over WAN - how to split them
On 3/15/2011 3:23 PM, David Herring wrote: New to this list - not sure if my first post worked - so am resending I'm trying to backup windows servers with approx 3 partitions of 100G each over a WAN link. I'm using rsyncd on the windows machines - and the backup is to a ubuntu server. This takes 'days' to run and never successfully complete - normally I have to stop the backuppc (via /etc/init.d/backuppc stop) and this loses the already downloaded data If you are running rsync/rsyncd you should get a partial backup when a full fails and the next run should use that as the rsync base. Do you have the latest version installed? So, how do you split the backups from a single machine - so each can run within a reasonable (say 4 hour) timeframe ? Do you have to create multiple host entries for each machine ? Does this make restore too painful ? Is the whole issue about number of files / rsyncd performance You can make a host entry for each partition (or each rsyncd module, which wouldn't have to be partition boundaries), with ClientAlias settings pointing to the same actual host. This will let each be scheduled separately. Is there anything else I should be doing ? MTU size ? MTU size might come into play if you are using a VPN that doesn't handle it gracefully. If you use openvpn, enabling lzo compression might help. Some people work out some other way to get the initial backup (move the server to the target LAN or vice versa, copy to an external disk and mount in a local machine with a ClientAlias point there for the first run, etc.). After your first full completes the rest should be much faster. -- Les Mikesell lesmikes...@gmail.com -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backups tooo slow over WAN - how to split them
Bowie Bailey wrote at about 16:39:59 -0400 on Tuesday, March 15, 2011: On 3/15/2011 3:49 PM, David Herring wrote: I'm trying to backup windows servers with approx 3 partitions of 100G each over a WAN link. This takes 'days' to run and never successfully completes. I'm using rsyncd on the windows machines - and the backup is to a ubuntu server. So, how do you split the backups from a single machine - do you have to create multiple host entries for each machine ? Does this make restore too painful ? Is there anything else I should be doing ? MTU size ? You can split the backups by directories. Just create separate backup hosts that each backup some of the directories. Give each host slightly different names and use the ClientNameAlias setting to point them back to the proper host name. Keep in mind that this may result in the separate backups running simultaneously. To prevent this you can use the DumpPreUserCmd and DumpPostUserCmd to set and check for lock files. Bowie's suggestion is spot on. But I'm still not sure why backup over WAN should take 'days' even for 300GB and even if this were the first time through or even if the files changed heavily. Assuming you have even low-end machines and a 100Mbps Internet, you should be able to get O(5-10MBps) speeds which should complete your backups in well under a day. Perhaps you have some large files that are causing rsyncd timeouts? (you might want to check the 'timout' parameter setting in rsync). Also, look at the client and server logs to see why the backups don't complete. -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Backups tooo slow over WAN - how to split them
You could just keep the one machine entry, but use the exclude options to only back up a small portion of the machine. Once that completes gradually remove directories from the excluded list until the full backup completes fine. This assumes that most of your data is static and that any files that change can be easily backed up in a daily window. Brian On Tue, Mar 15, 2011 at 1:49 PM, David Herring d...@netfm.org wrote: I'm trying to backup windows servers with approx 3 partitions of 100G each over a WAN link. This takes 'days' to run and never successfully completes. I'm using rsyncd on the windows machines - and the backup is to a ubuntu server. So, how do you split the backups from a single machine - do you have to create multiple host entries for each machine ? Does this make restore too painful ? Is there anything else I should be doing ? MTU size ? Any help great-fully received, Dave -- David Herring -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
