Re: Cloning disks with dd and netcat
Giles Nunn said: > Hi all, > > I tried the dd route to do exactly the same thing. I wanted to recreate > a server or a variation of it quickly and easily. Eventually I gave up > and used systemimager instead. It is quick and simple. It is based on > rsync and it is in woody. I have it working using network boot and it > takes ~3 minutes to boot, partition and install a complete base server - > fully automatically. I am still playing with it as I want to script the > autoconfig of multiple copies of a base server, but it works brilliantly > for a simple clone as it is. Why reinvent fire Systemimager works beautifully. I use it to deply to a bunch of lab computers, completelly hands off via a bootable cd. And also use to backup/restore my servers when and if an upgrade or system goes bad. I use the deb files from systemimager.org... -- --Luke CS Sysadmin, Montana State University-Bozeman -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
On Tue, 2004-07-27 18:42:09 +0200, Leonardo Boselli <[EMAIL PROTECTED]> wrote in message <[EMAIL PROTECTED]>: > Il 27 Jul 2004 alle 18:18 Jan-Benedict Glaw immise in rete > >> Try that with > >> a formerly booting NT system on a NTFS filesystem:) just copy the > >> "root"... by root i say /dev/hda , the raw partition. I worked fine > >> for me many times. > > > > Aren't MS-DOS' io.sys and msdos.sys expected to be in specific areas? > > only one of the two, in recent version, anyway as long the disk has the > same number of sectors and heads they will end on the same place Will they? How's that guaranteed when using tar/cpio or any other file-based backup method? MfG, JBG -- Jan-Benedict Glaw [EMAIL PROTECTED]. +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf| Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); signature.asc Description: Digital signature
Re: Cloning disks with dd and netcat
Hi all,
I tried the dd route to do exactly the same thing. I wanted to recreate
a server or a variation of it quickly and easily. Eventually I gave up
and used systemimager instead. It is quick and simple. It is based on
rsync and it is in woody. I have it working using network boot and it
takes ~3 minutes to boot, partition and install a complete base server -
fully automatically. I am still playing with it as I want to script the
autoconfig of multiple copies of a base server, but it works brilliantly
for a simple clone as it is.
HTH
Giles
On Tue, 2004-07-27 at 11:12, David Ross wrote:
> Hi
>
> I'm having problems cloning a hard drive. What I want to try do is set
> aside a server on our internal network to store a whole bunch of server
> images. So instead of manually installing a new server and copying
> configs across, we just extract the appropriate image onto a harddrive.
>
> At the moment I've got an image on the image server but when extracting
> it to the new disk I am getting an error. I created the image by doing
> this:
>
> dd if=/dev/zero of=empty.tmp bs=1024
> count=FREEBLOCKSCOUNT
> rm empty.tmp
>
> This cleans up the disk space for each partition apparently for better
> compression, but I'm not sure about the 2nd line and FREEBLOCKSCOUNT.
> After this, which took a while, I issued the following from the image
> server which creates the image:
>
> nc -v -w 120 -p -l < /dev/null > image.gz
>
> Then from the machine we want to clone, I booted with a knoppix cd then
> set it up on the network then did this:
>
> dd if=/dev/had bs=512 | gzip -c | nc -v -w 60
>
> After a few hours, I ended up with image.gz sitting on the image server.
>
> Now, I've got a new 20Gb ready, slotted it into a new PC and booted with
> trusty knoppix. I then configured her on the int network with a DHCP
> assigned IP address.
>
> I then tried to cat the image from the image server across the network
> using netcat like the doc said. This is what I did:
>
> imgserver:/# cat image.gz | nc -v -w 120 -p -l
> listening on [any] ...
>
> Then I did this on the new pc with knoppix booted:
>
> tty1[/]# nc -v -w 60 XXX.XXX.XXX.XXX < /dev/null | gzip -dc | dd
> of=/dev/hda bs=512
> imgserver.whatever.co.za [XXX.XXX.XXX.XXX] (?) open
>
> Now it *should* be accepting the image through port and extracting
> it to /dev/had but after a few seconds the error:
>
> connect to [XXX.XXX.XXX.XXX] from fw.whatever.co.za [YYY.YYY.YYY.YYY]
> 1026
> too many output retries : Broken pipe
> imgserver:/#
>
> comes up on the image server and then this:
>
> hda: read_intr: status=0x59 { DriveReady SeekComplete DataRequest Error
> }
> hda: read_intr: error=0x40 { UncorrectableError }, LBAsect=19528,
> sector=19528
> end_request: I/O error, dev 03:00 (hda), sector 19528
> dd: writing `/dev/hda': Input/output error
> 19529+0 records in
> 19528+0 records out
> 9998336 bytes transferred in 9.226423 seconds (1083663 bytes/sec)
> too many output retries : Broken pipe
> [EMAIL PROTECTED]/]#
>
> comes up on the new pc. Obviously the first thing I did was swap the
> harddrive just in case the one in the new pc is faulty, but I get the
> same error. I also tried to use zcat but no luck.
>
> The docs I've been using can be found at http://wyae.de/docs/img_dd.php
>
> Please let me know if I need to supply more info on this or if there is
> anything I've left out. Thanks for your time and effort in advance!
>
> Thanks
> Dave
--
Giles Nunn - ISP Officer
Carms ICT Development Centre
+44 1267 228277
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
Il 27 Jul 2004 alle 18:18 Jan-Benedict Glaw immise in rete >> Try that with >> a formerly booting NT system on a NTFS filesystem:) just copy the >> "root"... by root i say /dev/hda , the raw partition. I worked fine >> for me many times. > > Aren't MS-DOS' io.sys and msdos.sys expected to be in specific areas? only one of the two, in recent version, anyway as long the disk has the same number of sectors and heads they will end on the same place -- Leonardo Boselli Nucleo Informatico e Telematico del Dipartimento Ingegneria Civile Universita` di Firenze , V. S. Marta 3 - I-50139 Firenze tel +39 0554796431 cell +39 3488605348 fax +39 055495333 http://www.dicea.unifi.it/~leo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
On Tue, 2004-07-27 17:52:25 +0200, Leonardo Boselli <[EMAIL PROTECTED]> wrote in message <[EMAIL PROTECTED]>: > Il 27 Jul 2004 alle 17:42 Jan-Benedict Glaw immise in rete > > > I use knoppix to make a cpio image of the 'root filesystem' I'll be > > > imaging. (eg mounted / /usr /home and /var). Then with another > > > script > > Try that with a formerly booting NT system on a NTFS filesystem:) > > just copy the "root"... by root i say /dev/hda , the raw partition. > I worked fine for me many times. Aren't MS-DOS' io.sys and msdos.sys expected to be in specific areas? MfG, JBG -- Jan-Benedict Glaw [EMAIL PROTECTED]. +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf| Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); signature.asc Description: Digital signature
Re: Cloning disks with dd and netcat
Il 27 Jul 2004 alle 17:42 Jan-Benedict Glaw immise in rete > > I use knoppix to make a cpio image of the 'root filesystem' I'll be > > imaging. (eg mounted / /usr /home and /var). Then with another > > script > Try that with a formerly booting NT system on a NTFS filesystem:) just copy the "root"... by root i say /dev/hda , the raw partition. I worked fine for me many times. -- Leonardo Boselli Nucleo Informatico e Telematico del Dipartimento Ingegneria Civile Universita` di Firenze , V. S. Marta 3 - I-50139 Firenze tel +39 0554796431 cell +39 3488605348 fax +39 055495333 http://www.dicea.unifi.it/~leo -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
On Tue, 27 Jul 2004 11:15:54 EDT, "George Georgalis" writes: >I don't think your system will work though. because you are writing the >mbr and the partition table with the fs image, you would have to have >the _exact_ same disk which is less common then you may expect. If the new disk is _larger or equal_ in size it'll work. Granted, you might lose a bit of space if it's larger. cheers, &rw -- / Ing. Robert Waldner | Security Engineer | CoreTec IT-Security \ \ <[EMAIL PROTECTED]> | T +43 1 503 72 73 | F +43 1 503 72 73 x99 / pgptq7u2791Om.pgp Description: PGP signature
Re: Cloning disks with dd and netcat
On Tue, 2004-07-27 11:15:54 -0400, George Georgalis <[EMAIL PROTECTED]> wrote in message <[EMAIL PROTECTED]>: > On Tue, Jul 27, 2004 at 04:09:04PM +0200, Volker Tanger wrote: > yes knoppix 2 will save time, you can "su -" from x as well. ...or just switch away from X11 down to one of the text consoles. > I don't think your system will work though. because you are writing the > mbr and the partition table with the fs image, you would have to have > the _exact_ same disk which is less common then you may expect. CHS values are mostly meaningless today, so for all "modern" software, it's mostly okay working on an image that was ripped off a different disk, as long as the source disk is smaller than the target:) Of course, you loose the additional size of your new disk, if it's larger. Possibly one can "fix" that by adding additional partitions, but I've never ever tried that, to be honest:) > I use knoppix to make a cpio image of the 'root filesystem' I'll be > imaging. (eg mounted / /usr /home and /var). Then with another script Try that with a formerly booting NT system on a NTFS filesystem:) > under knoppix I partition the disk per the application, wget the cpio > with http/https (maybe with passwd) to stdout and unzip the cpio > image to the filesystem. Do a similar procedure to put the right > kernel/modules on the target, complete with vmlinuz simlink. and run a > bootloader. That'd work for Unix systems, and if you used some hacked tar, that could even work with ACLs. But you'll face a hard time to try to boot DOS or Windows afterwards (even Linux wouldn't boot, as long as you didn't write a new boot sektor for it). Basically, what is what we want to achieve (normally)? * Prepare a full crash-recovery backup for a machine, while a cold spare box (or at least a HDD of same or larger size) is available. A dd-like backup is a cool thing for that, mostly independant of the operating system. * Same as above, but with Linux (or similar) as OS. A small sfdisk input script and tar-like backup may be a lot faster than the above, additionally allowing you to easily resize partitions. However, you've got to take care about booting the box by re-installing the bootloader. * Simple data backup. Just use tar/cpio/whatever. Possibly, utilities that know about the filesystem (dump, ...) may even be faster than accessing all the single files with tar-alikes. > It is a time consuming to get setup but the process is designed to be > portable, fast and maintainable. On a fast network the image can be done > in 5 to 15 minutes. Right, but leaves you with the problem of making Non-Linux systems bootable:) MfG, JBG -- Jan-Benedict Glaw [EMAIL PROTECTED]. +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf| Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); signature.asc Description: Digital signature
Re: Cloning disks with dd and netcat
On Tue, Jul 27, 2004 at 04:09:04PM +0200, Volker Tanger wrote:
>Boot in text mode ("knoppix 2") or Ctrl-Alt-1 from X11 into console. Try
>again then.
yes knoppix 2 will save time, you can "su -" from x as well.
your problem though is with the fstab knoppix creates, wait you're not
mounting the partition, you're dding it. trust me it's the fstab. change
the options to something you are familiar with, like "defaults" and
you'll be able to write the mbr and partitions.
I don't think your system will work though. because you are writing the
mbr and the partition table with the fs image, you would have to have
the _exact_ same disk which is less common then you may expect.
I use knoppix to make a cpio image of the 'root filesystem' I'll be
imaging. (eg mounted / /usr /home and /var). Then with another script
under knoppix I partition the disk per the application, wget the cpio
with http/https (maybe with passwd) to stdout and unzip the cpio
image to the filesystem. Do a similar procedure to put the right
kernel/modules on the target, complete with vmlinuz simlink. and run a
bootloader.
It is a time consuming to get setup but the process is designed to be
portable, fast and maintainable. On a fast network the image can be done
in 5 to 15 minutes.
// George
--
George Georgalis, Architect and administrator, Linux services. IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:[EMAIL PROTECTED]
Key fingerprint = 5415 2738 61CF 6AE1 E9A7 9EF0 0186 503B 9831 1631
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
Greetings!
> > Do you have any kind of BIOS-configurable write/virus protection
> > for that harddisc switched off?
>
> BIOS is ignored nicely once the kernel switched on VM and went into
> protected more...
Yes, I know - but I've encountered hardware where the "100% IDE"
controller could be switched into read-only mode EVEN FOR NON-BIOS
operation. Granted, it was a jumper back then (probably breaker plus
pullup/pulldown for R/W signal line), but that could be done with some
CMOS/Flash setting today, too.
It just strook me odd that root could not write even the first few
bits...
*ahem*
Stop. Different idea.
@David Ross: you wrote you booted from Knoppix. I hope you did use plain
text mode? If you used the X11/KDE desktop you're usually logged in as
"knoppix" or whatever plain/non-root user. And of course you're not
allowed to (write) access the raw device as ordinary user...
Boot in text mode ("knoppix 2") or Ctrl-Alt-1 from X11 into console. Try
again then.
If this does not solve the problem, we'll have to search on.
Bye
Volker Tanger
ITK Security
PS: I've updated my docs accordingly - that's an easily overlooked
stuble block.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
On Tue, 2004-07-27 13:13:22 +0200, Volker Tanger <[EMAIL PROTECTED]> wrote in message <[EMAIL PROTECTED]>: > On Tue, 27 Jul 2004 12:12:33 +0200 "David Ross" <[EMAIL PROTECTED]> > wrote: > [...] > > Obviously the first thing I did was swap the > > harddrive just in case the one in the new pc is faulty, but I get the > > same error. > Do you have any kind of BIOS-configurable write/virus protection > for that harddisc switched off? BIOS is ignored nicely once the kernel switched on VM and went into protected more... MfG, JBG -- Jan-Benedict Glaw [EMAIL PROTECTED]. +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf| Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); signature.asc Description: Digital signature
RE: Cloning disks with dd and netcat
Thanks for the replys so far guys. Yeah I suppose I could RSYNC it across like Volker said but yeah, I'm trying to do this in the simplest way, ie no messing around with the bootloader or installing a base system first. I'm trying again now with a new image...etc Thanks again, I'll let you guys know how it goes. Dave -Original Message- From: Robert Waldner [mailto:[EMAIL PROTECTED] Sent: 27 July 2004 01:35 To: [EMAIL PROTECTED] Subject: Re: Cloning disks with dd and netcat On Tue, 27 Jul 2004 13:13:22 +0200, "Volker Tanger" writes: >What happens if you do the partitioning manually and image the >partitions (/dev/hda1, /dev/hda2, ...) one-by-one instead of the >complete disc? Well, doing the partitioning manually, you could RSYNC >the server instead of DD+NETCATing, which probably is faster and fails >more gracefully. But would mean mucking around with the bootloader, which usually is the point for doing _complete_ disc-images. cheers, &rw -- / Ing. Robert Waldner | Security Engineer | CoreTec IT-Security \ \ <[EMAIL PROTECTED]> | T +43 1 503 72 73 | F +43 1 503 72 73 x99 /
Re: Cloning disks with dd and netcat
On Tue, 27 Jul 2004 14:05:14 +0200, "Volker Tanger" writes: >True - but DDing a 200GB system disc disc takes quite some time, while >manually handling partition+mkfs+lilo plus RSYNCing 1.2GB usually is >LOTS faster... > >Upgrading to servers with newer/bigger discs is also less painful than >with imaging. > >But for mostly uniform hardware or testlabs (with frequent system >bashing) it's the leisure-factor that is heavily in favour of DD images, >I confess... ;-) That's why I often do dump/restore followed by dd'ing the first couple bytes ;) cheers, &rw -- / Ing. Robert Waldner | Security Engineer | CoreTec IT-Security \ \ <[EMAIL PROTECTED]> | T +43 1 503 72 73 | F +43 1 503 72 73 x99 / pgp9BrqwQxo0N.pgp Description: PGP signature
Re: Cloning disks with dd and netcat
Greetings! > >Well, doing the partitioning manually, you could RSYNC > >the server instead of DD+NETCATing, which probably is faster and > >fails more gracefully. > > But would mean mucking around with the bootloader, which usually is > the point for doing _complete_ disc-images. True - but DDing a 200GB system disc disc takes quite some time, while manually handling partition+mkfs+lilo plus RSYNCing 1.2GB usually is LOTS faster... Upgrading to servers with newer/bigger discs is also less painful than with imaging. But for mostly uniform hardware or testlabs (with frequent system bashing) it's the leisure-factor that is heavily in favour of DD images, I confess... ;-) Bye Volker Tanger ITK Security -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Cloning disks with dd and netcat
On Tue, 27 Jul 2004 13:13:22 +0200, "Volker Tanger" writes: >What happens if you do the partitioning manually and image the >partitions (/dev/hda1, /dev/hda2, ...) one-by-one instead of the >complete disc? Well, doing the partitioning manually, you could RSYNC >the server instead of DD+NETCATing, which probably is faster and fails >more gracefully. But would mean mucking around with the bootloader, which usually is the point for doing _complete_ disc-images. cheers, &rw -- / Ing. Robert Waldner | Security Engineer | CoreTec IT-Security \ \ <[EMAIL PROTECTED]> | T +43 1 503 72 73 | F +43 1 503 72 73 x99 / pgpmWLH05wE9Q.pgp Description: PGP signature
Re: Cloning disks with dd and netcat
Greetings!
On Tue, 27 Jul 2004 12:12:33 +0200 "David Ross" <[EMAIL PROTECTED]>
wrote:
> tty1[/]# nc -v -w 60 XXX.XXX.XXX.XXX < /dev/null | gzip -dc | dd
> of=/dev/hda bs=512
> imgserver.whatever.co.za [XXX.XXX.XXX.XXX] (?) open
[...]
> hda: read_intr: status=0x59 { DriveReady SeekComplete DataRequest
> Error}
> hda: read_intr: error=0x40 { UncorrectableError }, LBAsect=19528,
> sector=19528
> end_request: I/O error, dev 03:00 (hda), sector 19528
> dd: writing `/dev/hda': Input/output error
> 19529+0 records in
> 19528+0 records out
> 9998336 bytes transferred in 9.226423 seconds (1083663 bytes/sec)
> too many output retries : Broken pipe
[...]
> Obviously the first thing I did was swap the
> harddrive just in case the one in the new pc is faulty, but I get the
> same error.
Obviously the problem is that DD cannot write (for whatever reason) to
/dev/hda - not a single byte.
Do you have any kind of BIOS-configurable write/virus protection
for that harddisc switched off?
What happens if you do the partitioning manually and image the
partitions (/dev/hda1, /dev/hda2, ...) one-by-one instead of the
complete disc? Well, doing the partitioning manually, you could RSYNC
the server instead of DD+NETCATing, which probably is faster and fails
more gracefully.
Bye
Volker Tanger
ITK Security
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
