Re: [BackupPC-users] Migrating to bigger storage - real world experience.
Hi, In a similar situation, we use afio. It is like cpio but much more efficient. Yves John Pettitt wrote: Notes on migrating to bigger storage. Two weeks ago I asked about migrating to my BackupPC pool to bigger storage. I got a number of responses and after some experimentation reached the following conclusions: Suggestions: 1) dd the filesystem then expand it on the new storage. It's fast with a good network you can max disk. Copying on the same machine I was bus limited at ~40MB/sec. This is probably the best approach *if* you can expand your filesystems. However it turns out that growfs on FreeBSD is less than reliable with very large filesystems (in my case it refused to grow a 600GB filesystem to 1TB exiting with an error about seeking to a negative block number) Your experience may vary depending on OS but you are strongly advised to test it first. 2) cp, pax, tar, rsync et al. All of the file copy programs have severe limitations when dealing with BackupPC pools. The initial copy file by file is slow (~ 1/3 to 1/4 of the dd copy speed) but the subsequent creation of the hard linked backup trees for each client is painfuly slow. I aborted my copy after three days with less than 25% of the files linked. 3) Dump/restore – this has the potential to work well *if* you have a lot of memory in the machine – the restore process on my machine ran out of memory (I only have 1GB in the box) 4) Don’t bother. This is the approach I finally chose – I decided to just create a new server and let it start backing up hosts and at the same time turn off the old server but keep that data until I have a cycle with at least two full backups for each host. A hybrid approach using this and pax/cp/tar should also be possible copying only the pool. Turning off the nightly cleanup jobs and running a full backup to create new backup tress linked to the pool then once that has run re-enabling nightly clean up. Other data points – The Box is an old slow Celeron 2.93 Ghz box with 1GB ram and a highpoint raid card with 6 WD250 IDE drives running FreeBSD 6.2. BackupPC version is 3.0beta3. Backup pool is ~ 300GB and contains uses 9 million inodes. File systems are ufs2 with soft-updates enabled. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
I allways move my pools using super complex procedure stop backuppc cp -a start backuppc :-D On Thu, 21 Dec 2006 09:25:31 -0500 (EST), Stephen Joyce wrote I've had good results moving storage pools between RAID devices (for maintenance) using xfsdump and xfsrestore. I'd recommend that you investigate the dump/restore commands for your filesystem (I've learned to avoid ext for anything over ~1TB but YMMV). For locally attached devices, the rates were competitive with all other methods I tried. You can use dump/restore commands over the net from one server to another, but beware of the overhead of your transfer protocol (if you're using a ssh tunnel choose your cipher wisely to reduce overhead). Hope this helps. Cheers, Stephen -- Stephen Joyce Systems AdministratorP A N I C Physics Astronomy Department Physics Astronomy University of North Carolina at Chapel Hill Network Infrastructure voice: (919) 962-7214and Computing fax: (919) 962-0480 http://www.panic.unc.edu (3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead. -- RFC 1925 - Fundamental truths of networking On Thu, 21 Dec 2006, Mike Sauer wrote: I've has success using PAX to copy ~72GB of data. pax -r -w /old/path /new/path It took quite a while, far longer than DD would have, but personally prefer copying on a file level to a device level. The tail end copy process in an intense IO seek for to match hard lines. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ -- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ === El que de amigos carece es porque no los merece. === - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
On Wed, Dec 27, 2006 at 11:04:06AM +0100, Diaz Rodriguez, Eduardo wrote: I allways move my pools using super complex procedure stop backuppc cp -a start backuppc Be sure that cp is a GNU cp or at least handles hardlinks correctly - otherwise you'll multiple your storage requirements. Bye, Tino. -- www.quantenfeuerwerk.de www.spiritualdesign-chemnitz.de www.lebensraum11.de - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
Only check that the size in the new file system has the same size that the old system. are this correct? On Wed, 27 Dec 2006 11:10:03 +0100, Tino Schwarze wrote On Wed, Dec 27, 2006 at 11:04:06AM +0100, Diaz Rodriguez, Eduardo wrote: I allways move my pools using super complex procedure stop backuppc cp -a start backuppc Be sure that cp is a GNU cp or at least handles hardlinks correctly - otherwise you'll multiple your storage requirements. Bye, Tino. -- www.quantenfeuerwerk.de www.spiritualdesign-chemnitz.de www.lebensraum11.de - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ === El que de amigos carece es porque no los merece. === - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
On Wed, Dec 27, 2006 at 12:42:38PM +0100, Diaz Rodriguez, Eduardo wrote: Only check that the size in the new file system has the same size that the old system. are this correct? Yes, you would see a significant increase in used space on the new file system if the copying went wrong and didn't preserve hardlinks. HTH, Tino. -- www.quantenfeuerwerk.de www.spiritualdesign-chemnitz.de www.lebensraum11.de - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
I've had good results moving storage pools between RAID devices (for maintenance) using xfsdump and xfsrestore. I'd recommend that you investigate the dump/restore commands for your filesystem (I've learned to avoid ext for anything over ~1TB but YMMV). For locally attached devices, the rates were competitive with all other methods I tried. You can use dump/restore commands over the net from one server to another, but beware of the overhead of your transfer protocol (if you're using a ssh tunnel choose your cipher wisely to reduce overhead). Hope this helps. Cheers, Stephen -- Stephen Joyce Systems AdministratorP A N I C Physics Astronomy Department Physics Astronomy University of North Carolina at Chapel Hill Network Infrastructure voice: (919) 962-7214and Computing fax: (919) 962-0480 http://www.panic.unc.edu (3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead. -- RFC 1925 - Fundamental truths of networking On Thu, 21 Dec 2006, Mike Sauer wrote: I've has success using PAX to copy ~72GB of data. pax -r -w /old/path /new/path It took quite a while, far longer than DD would have, but personally prefer copying on a file level to a device level. The tail end copy process in an intense IO seek for to match hard lines. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ -- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
On 12/20/06, John Pettitt [EMAIL PROTECTED] wrote: I'm about to migrate my BackupPC partition to a new raid controller (more space and more spindles) - my current thinking is to use dump/restore - has anybody done this - what issues did you encounter? I've used tar over ssh which worked well, you could also use tar over netcat, but haven't tried that. Not as fast as dd, but not bad. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Migrating to bigger storage
On Wed, 2006-12-20 at 15:25 -0600, Carl Wilhelm Soderstrom wrote: dd bypasses those memory problems. if you want to move it across the network, pipe it through netcat. I moved 200GB of backuppc pool across the network that way in about 12 hours. once on the new box, I just used the reiserfs resize utility to make the filesystem large enough to take advantage of all the space. dd is most probably the only efficient way to copy a backuppc pool to another device. Anything else is just too slow. Having said that, has anyone tried it on a 500 GB + pool? How about 1 TB? Just curious... Regards, Ranbir -- Kanwar Ranbir Sandhu Linux 2.6.18-1.2849.fc6 i686 GNU/Linux 23:36:17 up 4 days, 15:31, 1 user, load average: 0.79, 0.70, 0.63 - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
