Re: [gentoo-user] cloning drivers that are not the same size
Bob Sanders wrote: How about benchmarks? Has anyone seen benchmarks of dump vs. partimage vs. tar vs. rsync vs. cp? That would be interesting. Why? The task is to move the data from one partition to a new disk/partition. Getting it reliably done, in a repeatable, sane, manner is more important than speed. Using something like dump/xfsdump, the limiting factor is drive i/o or the disk channel depending upon the setup. Also, a journaling filesystem will impose a certain amount of overhead and disk writes are going to be the bottleneck. The only question is - which keeps the target's disk buffer full? The max transfer rate can be calculated from hard drive sustained sequential write performance, the max speed, minus the overhead of the file system. The assumptions are dma is used, drives are on different controller channels, and memory is sufficient. Bob Why? Just a simple thought exercise :-). The drive i/o being the limiting factor seems like a good assumption. The only thing I can think of that might cause one tool perform better than another would be sequential vs. nonsequential writes. Something like partimage might be able to do it with more sequential writes. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
Ow Mun Heng wrote: Normally when I want to populate a new dir structure (which is empty to begin with) I do instead of issuing the command cp -a /some/area/mp3 /other/area/mp3-store use tar lcf - . | (cd /other/area/mp3-store; tar -xpvf - ) tar lcf - /path/to/file | (cd /other/area/mp3-store; tar -xpvf - ) tar -czf - /directory/to/copy | ssh systemB tar -xzvf - -C /tmp Thanks, that clarifies it for me. I wasn't aware that cp and rsync were inferior to tar in this respect but I suppose a simple benchmark would prove it. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
Neil Bothwick wrote: On Wed, 15 Jun 2005 16:07:35 -0700, Zac Medico wrote: The dd program knows nothing about partions but it will work with the associated device file. I would use fdisk, mke2fs, and rsync. I'd go with rsync too. It may well be slower than tar, but it can be interrupted and resumed. You can use dd for this, I have done so. As long as the new partitions are larger than the old ones, you'll end up with a small filesystem on a large partition, which you can then resize with the appropriate tool for the filesystem. I wouldn't chance dd'ing a whole drive unless they are identical, you could cause problems with the partition table. You should also note that dd is by far the slowest way of doing this, partimage is a much better tool for cloning partitions. Yeah, partimage is nice the way that it skips empty blocks. Can it clone directly from one partition to another or is an intermediate file required? I've found that squashfs is an iteresting option for doing backups/cloning because the compressed filesystem can also be reused to make a livecd or livedvd. For normal cloning, I mount the squashfs and use rsync to copy the files into a fresh ext3 partition. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
I've found that squashfs is an iteresting option for doing backups/cloning because the compressed filesystem can also be reused to make a livecd or livedvd. For normal cloning, I mount the squashfs and use rsync to copy the files into a fresh ext3 partition. This is interesting.. How is it done? Can you elaborate more?? -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 16:26:07 up 5:45, 4 users, load average: 0.63, 0.66, 0.67 -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
On Thu, 2005-06-16 at 08:46 +0100, Neil Bothwick wrote: On Wed, 15 Jun 2005 16:07:35 -0700, Zac Medico wrote: The dd program knows nothing about partions but it will work with the associated device file. I would use fdisk, mke2fs, and rsync. I'd go with rsync too. It may well be slower than tar, but it can be interrupted and resumed. As mentioned, go with tar if the destination tree is empty. If you have to abort midway, there's still an option to run rsync on it. So you still do save some time. But note that rsync _may_ take up more CPU resource than tar. You should also note that dd is by far the slowest way of doing this, partimage is a much better tool for cloning partitions. But one thing about partimage, I tried it once and going from a 10GB partition in partimage to 20GB partition in the new drive, the new drive only still saw itself as 10GB. I had to use partimage to resize and extend it to make it recognise itself as 20GB again. I don't know what happened and it was a long time ago. -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 16:26:39 up 5:45, 4 users, load average: 1.55, 0.89, 0.74 -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
Ow Mun Heng wrote: I've found that squashfs is an iteresting option for doing backups/cloning because the compressed filesystem can also be reused to make a livecd or livedvd. For normal cloning, I mount the squashfs and use rsync to copy the files into a fresh ext3 partition. This is interesting.. How is it done? Can you elaborate more?? Install squashfs-tools and use mksquashfs to make compressed filesystems (similar to tar archives but mountable). The standard gentoo-sources are already patched with squashfs so you only have to enable it in the kernel config. To do online backups of my desktop system I have a script that backs up my whole root filesystem with the exception of some files (especially some sensitive ones in /etc). I do separate backups of /etc. Occasionally I use my squashfs backups to create livedvds. I use a genkernel initrd that is patched to use unionfs for copy on write functionality. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
On Thu, 16 Jun 2005 16:29:10 +0800, Ow Mun Heng wrote: You should also note that dd is by far the slowest way of doing this, partimage is a much better tool for cloning partitions. But one thing about partimage, I tried it once and going from a 10GB partition in partimage to 20GB partition in the new drive, the new drive only still saw itself as 10GB. I had to use partimage to resize and extend it to make it recognise itself as 20GB again. partimage actually copies the filesystem. Put a 10GB filesystem on a 20GB partition and it will still be a 10GB filesystem. you just need to resize it. -- Neil Bothwick Access denied--nah nah na nah nah! pgp32r9BgaccU.pgp Description: PGP signature
Re: [gentoo-user] cloning drivers that are not the same size
On Thu, 16 Jun 2005 01:12:30 -0700, Zac Medico wrote: Yeah, partimage is nice the way that it skips empty blocks. Can it clone directly from one partition to another or is an intermediate file required? It appears to want a file, but I imagine you could use /dev/stdin and / dev/stdout as the files and pie one invocation into the other. Something like partimage -z0 -o -b -d -V0 save /dev/hda1 /dev/stdout | \ partimage -b restore /dev/hdc1 /dev/stdin As the target drive is empty, there's nothing to lose by trying. -- Neil Bothwick Woody, I said TUCK the children in bed! --Mia Farrow pgpILwecnEVxw.pgp Description: PGP signature
Re: [gentoo-user] cloning drivers that are not the same size
On Thu, 16 Jun 2005 01:44:56 -0700, Zac Medico wrote: To do online backups of my desktop system I have a script that backs up my whole root filesystem with the exception of some files (especially some sensitive ones in /etc). I do separate backups of /etc. Occasionally I use my squashfs backups to create livedvds. I do something similar. I use rdiff-backup to create hourly or daily backups of various directories. Then a weekly cron job creates squashfs files of these backups, which I write to a home made live DVD that contains everything I need (I hope) to restore a system. -- Neil Bothwick BASIC: Bill's Attempt to Seize Industry Control pgpSAT1jFI003.pgp Description: PGP signature
Re: [gentoo-user] cloning drivers that are not the same size
On Thu, 16 Jun 2005 13:11:33 +0800 Ow Mun Heng [EMAIL PROTECTED] wrote: Actually I would use fdisk/mke2fs and tar rather than rsync since it's much faster that way. I'd suggest another option - use xfs and xfsdump and xfsrestore. At the bottom of the xfsdump man page there are examples of ways to dump out the file system. The way I moved my /home from a small disk to a larger one was - fdisk/cfdisk the new drive mkfs.xfs /dev/sda (it was attached via a USB to ide adapter) mkdir /d2 mount /dev/sda1 /d2 xfsdump - /home | xfsrestore - /d2/ There is a similar dump/restore for the ext2 filesystem - app-arch/dump. Bob - - -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
Bob Sanders wrote: On Thu, 16 Jun 2005 13:11:33 +0800 Ow Mun Heng [EMAIL PROTECTED] wrote: Actually I would use fdisk/mke2fs and tar rather than rsync since it's much faster that way. I'd suggest another option - use xfs and xfsdump and xfsrestore. At the bottom of the xfsdump man page there are examples of ways to dump out the file system. The way I moved my /home from a small disk to a larger one was - fdisk/cfdisk the new drive mkfs.xfs /dev/sda (it was attached via a USB to ide adapter) mkdir /d2 mount /dev/sda1 /d2 xfsdump - /home | xfsrestore - /d2/ There is a similar dump/restore for the ext2 filesystem - app-arch/dump. Bob - - This method looks interesting. I found a quote from Linux Torvalds saying dump can misbehave if there are dirty buffers. Has anyone experienced that? http://www.geoffholden.com/content/presentations/Backups/ How about benchmarks? Has anyone seen benchmarks of dump vs. partimage vs. tar vs. rsync vs. cp? That would be interesting. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
On Thu, 16 Jun 2005 18:23:30 -0700 Zac Medico [EMAIL PROTECTED] wrote: Bob Sanders wrote: This method looks interesting. I found a quote from Linux Torvalds saying dump can misbehave if there are dirty buffers. Has anyone experienced that? I haven't used dump in five or more years, but things to keep in mind - This technique runs things through memory, thus a quiet system is needed. No ripping cds in the background, compiling, letting batch jobs run, etc. For xfsdump, networking needs to up enough to set hostname. Don't ask, it's always been that way. It's best to run a repair on the new disk after completion - unmount it, then a file system check. For xfs, run xfs_repair. http://www.geoffholden.com/content/presentations/Backups/ How about benchmarks? Has anyone seen benchmarks of dump vs. partimage vs. tar vs. rsync vs. cp? That would be interesting. Why? The task is to move the data from one partition to a new disk/partition. Getting it reliably done, in a repeatable, sane, manner is more important than speed. Using something like dump/xfsdump, the limiting factor is drive i/o or the disk channel depending upon the setup. Also, a journaling filesystem will impose a certain amount of overhead and disk writes are going to be the bottleneck. The only question is - which keeps the target's disk buffer full? The max transfer rate can be calculated from hard drive sustained sequential write performance, the max speed, minus the overhead of the file system. The assumptions are dma is used, drives are on different controller channels, and memory is sufficient. Bob -- - -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
Mark Knecht wrote: Hi, I'm not at all clear from reading man dd whether it will work for drives that are not the same size? For instance my current working drive in onePundit-R is 8GB and has 3 partitions - boot, root and swap. I'd like to copy these partitions to a new 80GB drive for use in another Pundit-R. 1) Can dd be used to copy partitions? (It seems so - just checking) 2) If yes above, then do I need to make identical sized partitions on the target drive before copying, or does dd create the partition? (I hope it doesn't actually) After the clone I'll rename the machine and give it a different IP address but I assume that everything else is 100% identical. The extra drive space will be used for some other purpose and does not have to be part of root. Thanks, Mark The dd program knows nothing about partions but it will work with the associated device file. I would use fdisk, mke2fs, and rsync. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
On Wed, 2005-06-15 at 18:57 -0700, Mark Knecht wrote: On 6/15/05, Zac Medico [EMAIL PROTECTED] wrote: Mark Knecht wrote: Hi, I'm not at all clear from reading man dd whether it will work for drives that are not the same size? For instance my current working drive in onePundit-R is 8GB and has 3 partitions - boot, root and swap. I'd like to copy these partitions to a new 80GB drive for use in another Pundit-R. 1) Can dd be used to copy partitions? (It seems so - just checking) 2) If yes above, then do I need to make identical sized partitions on the target drive before copying, or does dd create the partition? (I hope it doesn't actually) The dd program knows nothing about partions but it will work with the associated device file. I would use fdisk, mke2fs, and rsync. Zac OK, cool. Thanks. I've wanted a reason to learn a bit about rsync anyway. Actually I would use fdisk/mke2fs and tar rather than rsync since it's much faster that way. (tar uses block-by-block copy and rsync should be using char-by-char?) -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 13:10:26 up 2:29, 5 users, load average: 1.23, 1.21, 0.74 -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] cloning drivers that are not the same size
Ow Mun Heng wrote: On Wed, 2005-06-15 at 18:57 -0700, Mark Knecht wrote: On 6/15/05, Zac Medico [EMAIL PROTECTED] wrote: Mark Knecht wrote: Hi, I'm not at all clear from reading man dd whether it will work for drives that are not the same size? For instance my current working drive in onePundit-R is 8GB and has 3 partitions - boot, root and swap. I'd like to copy these partitions to a new 80GB drive for use in another Pundit-R. 1) Can dd be used to copy partitions? (It seems so - just checking) 2) If yes above, then do I need to make identical sized partitions on the target drive before copying, or does dd create the partition? (I hope it doesn't actually) The dd program knows nothing about partions but it will work with the associated device file. I would use fdisk, mke2fs, and rsync. Zac OK, cool. Thanks. I've wanted a reason to learn a bit about rsync anyway. Actually I would use fdisk/mke2fs and tar rather than rsync since it's much faster that way. (tar uses block-by-block copy and rsync should be using char-by-char?) Tar to copy files? I'm sorry, you lost me there :-). A possible block by block way that I know of would be to use dd and ext2resize. Zac -- gentoo-user@gentoo.org mailing list
