copy full system from old disk to a new one
How can I pass my whole system from an old SATA disk (which is giving me some trouble) to a new disk. I have 4 partitions, all of which I have only passed /home using cp -.a I'm a bit concerned about the root file system, how to change grub, and if I'll have trouble with non-regular files from /var. TIA! -- Martín Marqués select 'martin.marques' || '@' || 'gmail.com' DBA, Programador, Administrador -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/16/2013 06:53 PM, Martín Marqués wrote: How can I pass my whole system from an old SATA disk (which is giving me some trouble) to a new disk. I have 4 partitions, all of which I have only passed /home using cp -.a I'm a bit concerned about the root file system, how to change grub, and if I'll have trouble with non-regular files from /var. TIA! http://www.grc.com/intro.htm Look at SpinRight. Also, while you are there, follow the links starting at Shields Up. Millions of internet routers have been produced with UpNP exposed on the internet side. Maybe yours??? -- _ °v° /(_)\ ^ ^ Mark LaPierre Registered Linux user No #267004 https://linuxcounter.net/ -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/16/2013 06:53 PM, Martín Marqués wrote: How can I pass my whole system from an old SATA disk (which is giving me some trouble) to a new disk. I have 4 partitions, all of which I have only passed /home using cp -.a I'm a bit concerned about the root file system, how to change grub, and if I'll have trouble with non-regular files from /var. TIA! There is a Linux program called Clonezilla which is supposed to be able to clone disks. I have not used it, but I have used a program called HDClone, from Miray. There are several versions of HD Clone: one is free, one costs some money, and others cost even more. The difference appears to be the speed of copying. I originally used the freebie, and it took about 7 hours to clone a 160G disk. Later, I bought the first paid one, and it took about 3 hours to do something similar. It will clone to another disk in the machine, or to an external usb drive, rotating or solid state. You download the software and burn to a CD and then run the bootable CD on one of the computers that you want to clone from or to. I did a "double clone": from laptop to external drive, to drive on second computer. Look here: http://www.miray.de/products/sat.hdclone.html (Unlike many Linuxers, I am not averse to buying something that will do a good job with a minimum of fuss for a reasonable price; I have also bought an office suite which I think is better than LO.) --doug -- Blessed are the peacemakers...for they shall be shot at from both sides. --A. M. Greeley -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 17.02.2013 01:36, schrieb doug: > On 02/16/2013 06:53 PM, Martín Marqués wrote: >> How can I pass my whole system from an old SATA disk (which is giving >> me some trouble) to a new disk. I have 4 partitions, all of which I >> have only passed /home using cp -.a >> >> I'm a bit concerned about the root file system, how to change grub, >> and if I'll have trouble with non-regular files from /var. >> > There is a Linux program called Clonezilla which is supposed to be able to > clone disks. > a program called HDClone, from Miray. There are several versions of HD Clone: > one is free, one costs some money, and > others cost even more. The difference appears to be the speed of copying. I > originally used the freebie, and it > took about 7 hours to clone a 160G disk. Later, I bought the first paid one, > and it took about 3 hours to do > something similar * boot from a live-cd and use dd * dd if=/dev/sdX of=/dev/sdY bs=16M /dev/sdX = your source disk /dev/sdY = your new disk be careful not make a mistake in source / target, afetr enter you are lost in this case with the script below if even mirrored 4 x 2 TB SATA disks from one machine to another with 3 linux-software-RAIDs in a few hours to a blank machine [root@localhost ~]# cat /clone.sh date dd if=/dev/sda | ssh root@192.168.196.129 "dd of=/dev/sda bs=16M" dd if=/dev/sdb | ssh root@192.168.196.129 "dd of=/dev/sdb bs=16M" dd if=/dev/sdc | ssh root@192.168.196.129 "dd of=/dev/sdc bs=16M" dd if=/dev/sdd | ssh root@192.168.196.129 "dd of=/dev/sdd bs=16M" date signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/17/2013 09:03 AM, Reindl Harald wrote: > > > Am 17.02.2013 01:36, schrieb doug: >> On 02/16/2013 06:53 PM, Martín Marqués wrote: >>> How can I pass my whole system from an old SATA disk (which is giving >>> me some trouble) to a new disk. I have 4 partitions, all of which I >>> have only passed /home using cp -.a >>> >>> I'm a bit concerned about the root file system, how to change grub, and >>> if I'll have trouble with non-regular files from /var. >>> >> There is a Linux program called Clonezilla which is supposed to be able >> to clone disks. a program called HDClone, from Miray. There are several >> versions of HD Clone: one is free, one costs some money, and others cost >> even more. The difference appears to be the speed of copying. I >> originally used the freebie, and it took about 7 hours to clone a 160G >> disk. Later, I bought the first paid one, and it took about 3 hours to >> do something similar > > * boot from a live-cd and use dd * dd if=/dev/sdX of=/dev/sdY bs=16M > > /dev/sdX = your source disk /dev/sdY = your new disk > > be careful not make a mistake in source / target, afetr enter you are lost > in this case > > with the script below if even mirrored 4 x 2 TB SATA disks from one machine > to another with 3 linux-software-RAIDs in a few hours to a blank machine > > [root@localhost ~]# cat /clone.sh date dd if=/dev/sda | ssh > root@192.168.196.129 "dd of=/dev/sda bs=16M" dd if=/dev/sdb | ssh > root@192.168.196.129 "dd of=/dev/sdb bs=16M" dd if=/dev/sdc | ssh > root@192.168.196.129 "dd of=/dev/sdc bs=16M" dd if=/dev/sdd | ssh > root@192.168.196.129 "dd of=/dev/sdd bs=16M" date You may benefit from having bs=16M on the input side as well, otherwise the input dd will use default bs size. i.e. dd if=/dev/sda bs=16M | ssh root@192.168.196.129 "dd of=/dev/sda bs=16M" All the best, - -Greg - -- +-+ Please also check the log file at "/dev/null" for additional information. (from /var/log/Xorg.setup.log) | Greg Hosler ghos...@redhat.com| +-+ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlEgLZ0ACgkQ404fl/0CV/SyygCePQaeE2EhbbQ4EJwTwxTV39R4 EYIAoLOCTx3YLyz/Tly2dLbWU0dYgUoD =xCwA -END PGP SIGNATURE- -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 2013/02/16 17:08, Gregory Hosler wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 02/17/2013 09:03 AM, Reindl Harald wrote:
Am 17.02.2013 01:36, schrieb doug:
On 02/16/2013 06:53 PM, Martín Marqués wrote:
How can I pass my whole system from an old SATA disk (which is giving
me some trouble) to a new disk. I have 4 partitions, all of which I
have only passed /home using cp -.a
I'm a bit concerned about the root file system, how to change grub, and
if I'll have trouble with non-regular files from /var.
There is a Linux program called Clonezilla which is supposed to be able
to clone disks. a program called HDClone, from Miray. There are several
versions of HD Clone: one is free, one costs some money, and others cost
even more. The difference appears to be the speed of copying. I
originally used the freebie, and it took about 7 hours to clone a 160G
disk. Later, I bought the first paid one, and it took about 3 hours to
do something similar
* boot from a live-cd and use dd * dd if=/dev/sdX of=/dev/sdY bs=16M
/dev/sdX = your source disk /dev/sdY = your new disk
be careful not make a mistake in source / target, afetr enter you are lost
in this case
with the script below if even mirrored 4 x 2 TB SATA disks from one machine
to another with 3 linux-software-RAIDs in a few hours to a blank machine
[root@localhost ~]# cat /clone.sh date dd if=/dev/sda | ssh
root@192.168.196.129 "dd of=/dev/sda bs=16M" dd if=/dev/sdb | ssh
root@192.168.196.129 "dd of=/dev/sdb bs=16M" dd if=/dev/sdc | ssh
root@192.168.196.129 "dd of=/dev/sdc bs=16M" dd if=/dev/sdd | ssh
root@192.168.196.129 "dd of=/dev/sdd bs=16M" date
You may benefit from having bs=16M on the input side as well, otherwise the
input dd will use default bs size.
i.e.
dd if=/dev/sda bs=16M | ssh root@192.168.196.129 "dd of=/dev/sda bs=16M"
All the best,
- -Greg
Also consider "conv=fsync".
Try adding "conv=noerror conv=sync" if you have read errors. That ensures
that the copy has all the blocks in the correct place even if some of them
are in error. If you have errors you may do best setting bs to the native
block size of the disk or the file system's block size. If you pick the
smaller of these two you get maximum data recovery possible.
{^_^}
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 16 Feb 2013 at 20:53, Martín Marqués wrote: Date sent: Sat, 16 Feb 2013 20:53:15 -0300 Subject:copy full system from old disk to a new one From: Martín Marqués To: Community support for Fedora users > How can I pass my whole system from an old SATA disk (which is giving > me some trouble) to a new disk. I have 4 partitions, all of which I > have only passed /home using cp -.a > > I'm a bit concerned about the root file system, how to change grub, > and if I'll have trouble with non-regular files from /var. > > TIA! Some additional comments. Spinrite is a program that I have used in the past that can correct errors to recovered data from disks that have issues. Once had a disk had massive problems from another user, and ran spinrite on it for days to complete, and was then able to copy all the data they needed luckly, but disk was beyond recovery. I'm the maintainer of the G4L disk imaging project, and it is basically using dd to make most copies, but also has options to use dd_rescue programs that can try to get arround errors if the regular copy doesn't work. It can do disk clones to copy all data from one disk to another, but doesn't resize to a larger disk, but keeps everything the exact same size. I generally use the project to make images of the disk or partitions to files, that can be used to restore to a disk in the event of a disk failure more than trying to recover after a failure. What exact errors are you getting from the disk? Have you run a memtest on the system? Good Luck. > > -- > Martín Marqués > select 'martin.marques' || '@' || 'gmail.com' > DBA, Programador, Administrador > -- > users mailing list > users@lists.fedoraproject.org > To unsubscribe or change subscription options: > https://admin.fedoraproject.org/mailman/listinfo/users > Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have > a question? Ask away: http://ask.fedoraproject.org +--+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mi...@kuentos.guam.net mailto:msetze...@gmail.com http://www.guam.net/home/mikes Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ +--+ http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489) BOINC@HOME CREDITS SETI13902506.299683 | EINSTEIN10019879.779852 ROSETTA 6117405.134244 | ABC 15881854.139214 -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 17.02.2013, Martín Marqués wrote: > How can I pass my whole system from an old SATA disk (which is giving > me some trouble) to a new disk. [] > I'm a bit concerned about the root file system, how to change grub, > and if I'll have trouble with non-regular files from /var. You can do that easily. Prepare your new disk (partition, format), boot from an external medium, e.g. http://www.sysresccd.org , mount the old and the new partitions and do a rsync -avxHSAX /old/ /new This is all you need. Afterwards, you'll have to reinstall GRUB, which you can do this way (assumed /dev/sda1 is yor root partition): mount /dev/sda1 /mnt grub2-install --root-directory=/mnt /dev/sda grub2-mkconfig -o /mnt/boot/grub2/grub.cfg That's it. Reboot, and you're done. (Did this procedure a lot, and it never failed for me). -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
RE: copy full system from old disk to a new one
The rsync approach Heinz suggests also has the advantage that you can change the size of your new partitions in the target system. The other block copying approaches would need either identically sized partitions or physical disks. - Greg -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/17/2013 03:30 AM, Heinz Diehl wrote: On 17.02.2013, Martín Marqués wrote: How can I pass my whole system from an old SATA disk (which is giving me some trouble) to a new disk. [] I'm a bit concerned about the root file system, how to change grub, and if I'll have trouble with non-regular files from /var. You can do that easily. Prepare your new disk (partition, format), boot from an external medium, e.g. http://www.sysresccd.org , mount the old and the new partitions and do a rsync -avxHSAX /old/ /new This is all you need. Afterwards, you'll have to reinstall GRUB, which you can do this way (assumed /dev/sda1 is yor root partition): mount /dev/sda1 /mnt grub2-install --root-directory=/mnt /dev/sda grub2-mkconfig -o /mnt/boot/grub2/grub.cfg That's it. Reboot, and you're done. (Did this procedure a lot, and it never failed for me). Just curious, will this apply to moving /_everything _/over from a "smaller" drive to a "bigger" one?...likesay from a 320GB SATA HDD to a 500GB SATA HDD?will the OS automatically be able to identify and recognize the free space on the new drive? EGO II -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/16/2013 04:36 PM, doug wrote: There is a Linux program called Clonezilla which is supposed to be able to clone disks. I've used it several times across the years with excellent results. Not only can it clone a disk, it can also expand partitions "on the fly" if the new disk is bigger than the old. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 17.02.2013 20:59, schrieb Eddie G. O'Connor Jr.: > On 02/17/2013 03:30 AM, Heinz Diehl wrote: >> On 17.02.2013, Martín Marqués wrote: >> >>> How can I pass my whole system from an old SATA disk (which is giving >>> me some trouble) to a new disk. >> [] >>> I'm a bit concerned about the root file system, how to change grub, >>> and if I'll have trouble with non-regular files from /var. >> You can do that easily. Prepare your new disk (partition, format), >> boot from an external medium, e.g. http://www.sysresccd.org , >> mount the old and the new partitions and do a >> >> rsync -avxHSAX /old/ /new >> >> This is all you need. Afterwards, you'll have to reinstall GRUB, which >> you can do this way (assumed /dev/sda1 is yor root partition): >> >> mount /dev/sda1 /mnt >> grub2-install --root-directory=/mnt /dev/sda >> grub2-mkconfig -o /mnt/boot/grub2/grub.cfg >> >> That's it. Reboot, and you're done. >> (Did this procedure a lot, and it never failed for me). >> > Just curious, will this apply to moving /_everything _/over from a "smaller" > drive to a "bigger" > one?...likesay from a 320GB SATA HDD to a 500GB SATA HDD?will the > OS automatically be able to identify > and recognize the free space on the new drive? if you are dealing with rsync you HAVE already partitions where you copy things around - so there is nothing to recognize if wouldn't do this and stick with dd / disk images and use gparted to resize partitions because if i clone machines i want to have all UUID's the same which matters in border cases like diks usgae widget of KDE in case you sync two of your machines regulary and because any LVM/RAID whatever IDs and there configs are 100% sane signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 17.02.2013, Eddie G. O'Connor Jr. wrote: > Just curious, will this apply to moving /_everything _/over from a "smaller" > drive to a "bigger" one?...likesay from a 320GB SATA HDD to a 500GB > SATA HDD?will the OS automatically be able to identify and recognize the > free space on the new drive? Yes, of course. You'll have to partition your new harddisk first. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 17.02.2013, Reindl Harald wrote: > if wouldn't do this and stick with dd / disk images and use gparted > to resize partitions because if i clone machines i want to have > all UUID's the same You can easily change/tailor your UUIDs with tools like tune2fs, xfs_admin etc.. The "dd" approach has a bunch of disadvantages, one of the biggest is misalignment if your new drive is a SSD or advanced format HDD. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 17.02.2013 22:03, schrieb Heinz Diehl: > On 17.02.2013, Reindl Harald wrote: > >> if wouldn't do this and stick with dd / disk images and use gparted >> to resize partitions because if i clone machines i want to have >> all UUID's the same > > You can easily change/tailor your UUIDs with tools like tune2fs have fun change also the RAID UUID's everywhere and setup all identical because you have also internal UUIDs on the RAID-partitions [root@srv-rhsoft:~]$ cat /etc/mdadm.conf MAILADDR root AUTO +imsm +1.x -all ARRAY /dev/md0 level=raid1 num-devices=4 UUID=1d691642:baed26df:1d197496:4fb00ff8 ARRAY /dev/md1 level=raid10 num-devices=4 UUID=b7475879:c95d9a47:c5043c02:0c5ae720 ARRAY /dev/md2 level=raid10 num-devices=4 UUID=ea253255:cb915401:f32794ad:ce0fe396 oh yeah and the tun2fs-output of /dev/md0 is [root@srv-rhsoft:~]$ tune2fs -l /dev/md0 | grep UUID Filesystem UUID: 1de836e4-e97c-43ee-b65c-400b0c29d3aa again: have fun to reacreate this setup 1:1 after rsync to new drives - i bet it would not work at the end - well it may not interest you on your setups, mine are always two identical machines which are synced daily > The "dd" approach has a bunch of disadvantages, one of the biggest is > misalignment if your new drive is a SSD or advanced format HDD which misalignment? if i switch to SSD some years later as soon as 4x2 TB are at a normal price i will simply remove disk 1, insert a SSD, rebuild RAID and after the rbuild the same for the other 3 drives signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/17/2013 03:04 PM, Reindl Harald wrote: Am 17.02.2013 20:59, schrieb Eddie G. O'Connor Jr.: On 02/17/2013 03:30 AM, Heinz Diehl wrote: On 17.02.2013, Martín Marqués wrote: How can I pass my whole system from an old SATA disk (which is giving me some trouble) to a new disk. [] I'm a bit concerned about the root file system, how to change grub, and if I'll have trouble with non-regular files from /var. You can do that easily. Prepare your new disk (partition, format), boot from an external medium, e.g. http://www.sysresccd.org , mount the old and the new partitions and do a rsync -avxHSAX /old/ /new This is all you need. Afterwards, you'll have to reinstall GRUB, which you can do this way (assumed /dev/sda1 is yor root partition): mount /dev/sda1 /mnt grub2-install --root-directory=/mnt /dev/sda grub2-mkconfig -o /mnt/boot/grub2/grub.cfg That's it. Reboot, and you're done. (Did this procedure a lot, and it never failed for me). Just curious, will this apply to moving /_everything _/over from a "smaller" drive to a "bigger" one?...likesay from a 320GB SATA HDD to a 500GB SATA HDD?will the OS automatically be able to identify and recognize the free space on the new drive? if you are dealing with rsync you HAVE already partitions where you copy things around - so there is nothing to recognize if wouldn't do this and stick with dd / disk images and use gparted to resize partitions because if i clone machines i want to have all UUID's the same which matters in border cases like diks usgae widget of KDE in case you sync two of your machines regulary and because any LVM/RAID whatever IDs and there configs are 100% sane Excellent!thanks for the info! Getting ready to swap out the drive for this F18 laptopfrom 320 GB to 500GB...(was thinking about a 750GB...but it's too muchdon't have that much stuff to save!) EGO II -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On Sunday 17 February 2013 05:23 AM, Martín Marqués wrote: How can I pass my whole system from an old SATA disk (which is giving me some trouble) to a new disk. I have 4 partitions, all of which I have only passed /home using cp -.a I'm a bit concerned about the root file system, how to change grub, and if I'll have trouble with non-regular files from /var. TIA! Just have a look at this[1], its what you are looking for [1] http://clonezilla.org/ -- °v° /(_)\ ^ ^ Jatin Khatri RHCSA,RHCE,CCNA Registerd Linux user No #501175 www.linuxcounter.net No M$ -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 18.02.2013, Reindl Harald wrote: > which misalignment? Aligment to a 4k blocksize. This is what most of the SSDs and advanced format HDDs use. Both your partitions have to be aligned to 4k block boundaries and the filesystem you create on top of them must support the 4k blocksize. If you do a straight "dd" from a 512/512 disk with a filesystem using 512 bytes blocksize on top, you'll encounter a performance degradation. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 16:31, schrieb Heinz Diehl: > On 18.02.2013, Reindl Harald wrote: > >> which misalignment? > > Aligment to a 4k blocksize. This is what most of the SSDs and advanced > format HDDs use. Both your partitions have to be aligned to 4k block > boundaries and the filesystem you create on top of them must support > the 4k blocksize. > > If you do a straight "dd" from a 512/512 disk with a filesystem using > 512 bytes blocksize on top, you'll encounter a performance > degradation i can not remember when the last ext3/ext4 had 512 bytes blocksize our main-server's sysroot created 2008: Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 393216 Block count: 1572354 Reserved block count: 2 Free blocks: 1049068 Free inodes: 329327 First block: 0 Block size: 4096 Fragment size:4096 Filesystem created: Mon Aug 18 06:48:05 2008 my RAID10 sysroot created 2011: Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1921360 Block count: 7679232 Reserved block count: 2 Free blocks: 5910849 Free inodes: 1743536 First block: 0 Block size: 4096 Fragment size:4096 Filesystem created: Wed Jun 8 13:10:52 2011 signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 17 February 2013 21:12, Reindl Harald wrote: > > > Am 17.02.2013 22:03, schrieb Heinz Diehl: >> On 17.02.2013, Reindl Harald wrote: >> >>> if wouldn't do this and stick with dd / disk images and use gparted >>> to resize partitions because if i clone machines i want to have >>> all UUID's the same >> >> You can easily change/tailor your UUIDs with tools like tune2fs > > have fun change also the RAID UUID's everywhere > and setup all identical because you have also > internal UUIDs on the RAID-partitions > > > [root@srv-rhsoft:~]$ cat /etc/mdadm.conf > MAILADDR root > AUTO +imsm +1.x -all > ARRAY /dev/md0 level=raid1 num-devices=4 > UUID=1d691642:baed26df:1d197496:4fb00ff8 > ARRAY /dev/md1 level=raid10 num-devices=4 > UUID=b7475879:c95d9a47:c5043c02:0c5ae720 > ARRAY /dev/md2 level=raid10 num-devices=4 > UUID=ea253255:cb915401:f32794ad:ce0fe396 > > oh yeah and the tun2fs-output of /dev/md0 is > [root@srv-rhsoft:~]$ tune2fs -l /dev/md0 | grep UUID > Filesystem UUID: 1de836e4-e97c-43ee-b65c-400b0c29d3aa > > again: have fun to reacreate this setup 1:1 after rsync to > new drives - i bet it would not work at the end - well it > may not interest you on your setups, mine are always two > identical machines which are synced daily > >> The "dd" approach has a bunch of disadvantages, one of the biggest is >> misalignment if your new drive is a SSD or advanced format HDD > > which misalignment? > > if i switch to SSD some years later as soon as 4x2 TB are at a > normal price i will simply remove disk 1, insert a SSD, rebuild > RAID and after the rbuild the same for the other 3 drives > > I don't have a solid reference for this, only hearsay, but dd as a way to transfer filesystems or partitions to SSD probably interferes with wear-levelling. http://serverfault.com/questions/282555/zeroing-ssd-drives http://www.wonkity.com/~wblock/docs/html/ssd.html I guess RAID tools must now have some way to address this. -- imalone http://ibmalone.blogspot.co.uk -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 18:19, schrieb Ian Malone: > On 17 February 2013 21:12, Reindl Harald wrote: >> >> >> Am 17.02.2013 22:03, schrieb Heinz Diehl: >>> On 17.02.2013, Reindl Harald wrote: >>> if wouldn't do this and stick with dd / disk images and use gparted to resize partitions because if i clone machines i want to have all UUID's the same >>> >>> You can easily change/tailor your UUIDs with tools like tune2fs >> >> have fun change also the RAID UUID's everywhere >> and setup all identical because you have also >> internal UUIDs on the RAID-partitions >> >> >> [root@srv-rhsoft:~]$ cat /etc/mdadm.conf >> MAILADDR root >> AUTO +imsm +1.x -all >> ARRAY /dev/md0 level=raid1 num-devices=4 >> UUID=1d691642:baed26df:1d197496:4fb00ff8 >> ARRAY /dev/md1 level=raid10 num-devices=4 >> UUID=b7475879:c95d9a47:c5043c02:0c5ae720 >> ARRAY /dev/md2 level=raid10 num-devices=4 >> UUID=ea253255:cb915401:f32794ad:ce0fe396 >> >> oh yeah and the tun2fs-output of /dev/md0 is >> [root@srv-rhsoft:~]$ tune2fs -l /dev/md0 | grep UUID >> Filesystem UUID: 1de836e4-e97c-43ee-b65c-400b0c29d3aa >> >> again: have fun to reacreate this setup 1:1 after rsync to >> new drives - i bet it would not work at the end - well it >> may not interest you on your setups, mine are always two >> identical machines which are synced daily >> >>> The "dd" approach has a bunch of disadvantages, one of the biggest is >>> misalignment if your new drive is a SSD or advanced format HDD >> >> which misalignment? >> >> if i switch to SSD some years later as soon as 4x2 TB are at a >> normal price i will simply remove disk 1, insert a SSD, rebuild >> RAID and after the rbuild the same for the other 3 drives >> >> > > I don't have a solid reference for this, only hearsay, but dd as a way > to transfer filesystems or partitions to SSD probably interferes with > wear-levelling. > http://serverfault.com/questions/282555/zeroing-ssd-drives > http://www.wonkity.com/~wblock/docs/html/ssd.html > > I guess RAID tools must now have some way to address this that is why i said "if 2 TB SSD's are at a normal price" which will not happen now AND not until i trust them which is not the case now and at least a disk have to survive running full without damage/impact for call it trustable the SD-card of my HTC hero destroyed the whole phone with overheat and the root cause was that it no longer did write any data but confirmed "all OK", i tried to put the SD-card on my PC, formatted it with different filesystems, did overwrite it with zero as also with random crap and each time after i took it out of the reader and inserted it again the last written data days ago where untouched NOTICE: and not a single warning in "dmesg" or /var/log/messages the whole time while any spinning disk would lead to warnings or errors in such cases - you understand while i do not trust them currently signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19.02.2013, Reindl Harald wrote: > i can not remember when the last ext3/ext4 > had 512 bytes blocksize [] Most of the "conventional" harddisks have a sectorsize/blocksize of 512/512. All the newer and bigger WD/Seagate drives and SSDs are using "advanced format", which means 512/4096. To support this, there are two factors which must be present: 1. Proper alignment to 4k block boundaries 2. A filesystem on top which supports 4k blocksize If you use "dd" to raw-copy a 512/512 drive to a newer WD/Seagate or a SSD, this will result in misalignment (even if your FS supports the 4k blocksize) causing huge performance loss. To avoid this, you'll have to create proper aligned partitions on the new drive first, formatted with a FS supporting 4k blocksize, and copy the data the "traditional" way, e.g. using cp, rsync and brothers. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 18:48, schrieb Heinz Diehl: > On 19.02.2013, Reindl Harald wrote: > >> i can not remember when the last ext3/ext4 >> had 512 bytes blocksize > [] > > Most of the "conventional" harddisks have a sectorsize/blocksize of > 512/512. All the newer and bigger WD/Seagate drives and SSDs are using > "advanced format", which means 512/4096. > > To support this, there are two factors which must be present: > > 1. Proper alignment to 4k block boundaries > 2. A filesystem on top which supports 4k blocksize > > If you use "dd" to raw-copy a 512/512 drive to a newer WD/Seagate or a > SSD, this will result in misalignment (even if your FS supports the 4k > blocksize) causing huge performance loss. and why did you strip the part where i showed you that any of my filesystems of the last 5 years have a 4 KB blocksize as also any of my hardware of the last 5 years is "newer"? signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19/02/2013 17:48, Heinz Diehl wrote: On 19.02.2013, Reindl Harald wrote: i can not remember when the last ext3/ext4 had 512 bytes blocksize [] Most of the "conventional" harddisks have a sectorsize/blocksize of 512/512. All the newer and bigger WD/Seagate drives and SSDs are using "advanced format", which means 512/4096. To support this, there are two factors which must be present: 1. Proper alignment to 4k block boundaries 2. A filesystem on top which supports 4k blocksize If you use "dd" to raw-copy a 512/512 drive to a newer WD/Seagate or a SSD, this will result in misalignment (even if your FS supports the 4k blocksize) causing huge performance loss. To avoid this, you'll have to create proper aligned partitions on the new drive first, formatted with a FS supporting 4k blocksize, and copy the data the "traditional" way, e.g. using cp, rsync and brothers. To ensure proper alignment it is sufficient to manually partition to 4KB boundaries and then dd individual partitions over (as opposed to the whole disk in one transfer). Of course, mkfs + [cp | rsync | tar] will be much more efficient as it only transfers the files, rather than also the empty space. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 19:19, schrieb Gordan Bobic: > On 19/02/2013 17:48, Heinz Diehl wrote: >> On 19.02.2013, Reindl Harald wrote: >> >>> i can not remember when the last ext3/ext4 >>> had 512 bytes blocksize >> [] >> >> Most of the "conventional" harddisks have a sectorsize/blocksize of >> 512/512. All the newer and bigger WD/Seagate drives and SSDs are using >> "advanced format", which means 512/4096. >> >> To support this, there are two factors which must be present: >> >> 1. Proper alignment to 4k block boundaries >> 2. A filesystem on top which supports 4k blocksize >> >> If you use "dd" to raw-copy a 512/512 drive to a newer WD/Seagate or a >> SSD, this will result in misalignment (even if your FS supports the 4k >> blocksize) causing huge performance loss. >> >> To avoid this, you'll have to create proper aligned partitions on the >> new drive first, formatted with a FS supporting 4k blocksize, >> and copy the data the "traditional" way, e.g. using cp, rsync and brothers. > > To ensure proper alignment it is sufficient to manually partition to 4KB > boundaries and then dd individual > partitions over (as opposed to the whole disk in one transfer). Of course, > mkfs + [cp | rsync | tar] will be much > more efficient as it only transfers the files, rather than also the empty > space but does not transfer whole RAID setups with all their UUIDs and detail config and so better wait for SSD's which are not crippled by get ONE TIME written completly which iw ould consider as broken hardware also with dd over ssh the empty space is compressed and not transferred, with modern CPUs and RC4 cipher this goes up to 98 MB/second over Gigabit ethernet before i reinstall the machine eblow whcih exists more than one time i stay with 2 TB non SSD-disks and in a few years you can expand the RAID10 to a spindle with 6x2TB which is fast enough for most workloads if not all, currently the 4X2 TB RAID10 is reading up to 300MB/sec and writes 150-200 MB/sec [root@srv-rhsoft:~]$ df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/md1 ext4 29G 6.4G 23G 23% / /dev/md0 ext4 485M 38M 447M 8% /boot /dev/md2 ext4 3.6T 1.7T 2.0T 46% /mnt/data [root@srv-rhsoft:~]$ df -iT Filesystem Type Inodes IUsed IFree IUse% Mounted on /dev/md1 ext4 1.9M 174K 1.7M 10% / /dev/md0 ext4 126K 350 125K1% /boot /dev/md2 ext4 231M 944K 231M1% /mnt/data signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19.02.2013, Reindl Harald wrote: > and why did you strip the part where i showed you that any of my > filesystems of the last 5 years have a 4 KB blocksize as also > any of my hardware of the last 5 years is "newer"? Because what you posted doesn't give any information on how your partitions are aligned, actually. Just having a FS using 4k blocksize on top of a misaligned partition doesn't work out. Cloning a disk using "dd", you most probably will create misaligned partitions on any advanced format disk, which most of the newer drives are. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 19:44, schrieb Heinz Diehl: > On 19.02.2013, Reindl Harald wrote: > >> and why did you strip the part where i showed you that any of my >> filesystems of the last 5 years have a 4 KB blocksize as also >> any of my hardware of the last 5 years is "newer"? > > Because what you posted doesn't give any information on how your > partitions are aligned, actually. Just having a FS using 4k blocksize > on top of a misaligned partition doesn't work out. > > Cloning a disk using "dd", you most probably will create misaligned > partitions on any advanced format disk, which most of the newer drives > are what exactly do you need to align on the partitions? for raid-setups you usually left 10-30 MB completly unpartitioned to make 100% sure that you get not troubles by replace a drive signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19/02/2013 18:25, Reindl Harald wrote: Am 19.02.2013 19:19, schrieb Gordan Bobic: On 19/02/2013 17:48, Heinz Diehl wrote: On 19.02.2013, Reindl Harald wrote: i can not remember when the last ext3/ext4 had 512 bytes blocksize [] Most of the "conventional" harddisks have a sectorsize/blocksize of 512/512. All the newer and bigger WD/Seagate drives and SSDs are using "advanced format", which means 512/4096. To support this, there are two factors which must be present: 1. Proper alignment to 4k block boundaries 2. A filesystem on top which supports 4k blocksize If you use "dd" to raw-copy a 512/512 drive to a newer WD/Seagate or a SSD, this will result in misalignment (even if your FS supports the 4k blocksize) causing huge performance loss. To avoid this, you'll have to create proper aligned partitions on the new drive first, formatted with a FS supporting 4k blocksize, and copy the data the "traditional" way, e.g. using cp, rsync and brothers. To ensure proper alignment it is sufficient to manually partition to 4KB boundaries and then dd individual partitions over (as opposed to the whole disk in one transfer). Of course, mkfs + [cp | rsync | tar] will be much more efficient as it only transfers the files, rather than also the empty space but does not transfer whole RAID setups with all their UUIDs and detail config and so better wait for SSD's which are not crippled by get ONE TIME written completly which iw ould consider as broken hardware Indeed. Depends on the SSD. USB sticks and SD/CF card based stuff tends to have wear leveling so poor that you can kill them pretty easily with this sort of activity. The good ones will do block level deduplication and compression for you and throw away 0-filled blocks instead of storing them. also with dd over ssh the empty space is compressed and not transferred, with modern CPUs and RC4 cipher this goes up to 98 MB/second over Gigabit ethernet The most efficient way to transfer it over ssh with plenty of cpu is probably using tar, piping through a parallel compressor (pigz, pbzip2, pxz, ...), and then the revers on the target side ssh command. before i reinstall the machine eblow whcih exists more than one time i stay with 2 TB non SSD-disks and in a few years you can expand the RAID10 to a spindle with 6x2TB which is fast enough for most workloads if not all, currently the 4X2 TB RAID10 is reading up to 300MB/sec and writes 150-200 MB/sec I wouldn't really trust traditional any RAID with disks over 500GB. ZFS for me, thanks. Otherwise you get bit-rot every time a sector fails (no way to tell which mirror has the right data in RAID1/10 or whether the stripe or the parity is correct in RAID5 - you need at least RAID6, and even then it's very questionable and dependant on the implementation whether you'll get the right data combination back). Traditional RAID is, IMO, not really fit for purpose with disks of the size in common use today. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19/02/2013 18:53, Reindl Harald wrote: Am 19.02.2013 19:44, schrieb Heinz Diehl: On 19.02.2013, Reindl Harald wrote: and why did you strip the part where i showed you that any of my filesystems of the last 5 years have a 4 KB blocksize as also any of my hardware of the last 5 years is "newer"? Because what you posted doesn't give any information on how your partitions are aligned, actually. Just having a FS using 4k blocksize on top of a misaligned partition doesn't work out. Cloning a disk using "dd", you most probably will create misaligned partitions on any advanced format disk, which most of the newer drives are what exactly do you need to align on the partitions? For a start, making sure your RAID implementation puts the metadata at the end of the disk, rather than the beginning. If it puts it at the beginning you need to infer how big it is, and then pre-compensate by positioning the partition so that it begins at a 4KB boundary on the raw disk. And then make sure all the other partitions also begin on a 4KB raw disk boundary. for raid-setups you usually left 10-30 MB completly unpartitioned to make 100% sure that you get not troubles by replace a drive That doesn't have anything to do with alignment. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 20:02, schrieb Gordan Bobic: >> what exactly do you need to align on the partitions? > > For a start, making sure your RAID implementation puts the metadata > at the end of the disk, rather than the beginning. "my RAID implementation"? LINUX SOFTWARE RAID and this is how the raid-partitions are looking no problem since years Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xae2c Device Boot Start End Blocks Id System /dev/sda1 *2048 1026047 512000 fd Linux raid autodetect /dev/sda2 1026048317460471536 fd Linux raid autodetect /dev/sda331746048 3906971647 1937612800 fd Linux raid autodetect [root@srv-rhsoft:/downloads]$ sfdisk -d /dev/sda # partition table of /dev/sda unit: sectors /dev/sda1 : start= 2048, size= 1024000, Id=fd, bootable /dev/sda2 : start= 1026048, size= 3072, Id=fd /dev/sda3 : start= 31746048, size=3875225600, Id=fd /dev/sda4 : start=0, size=0, Id= 0 [root@srv-rhsoft:/downloads]$ parted /dev/sda align-check opt 1 1 aligned [root@srv-rhsoft:/downloads]$ parted /dev/sda align-check opt 2 2 aligned [root@srv-rhsoft:/downloads]$ parted /dev/sda align-check opt 3 3 aligned signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19/02/2013 19:05, Reindl Harald wrote: Am 19.02.2013 20:02, schrieb Gordan Bobic: what exactly do you need to align on the partitions? For a start, making sure your RAID implementation puts the metadata at the end of the disk, rather than the beginning. "my RAID implementation"? LINUX SOFTWARE RAID and this is how the raid-partitions are looking no problem since years Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xae2c Device Boot Start End Blocks Id System /dev/sda1 *2048 1026047 512000 fd Linux raid autodetect /dev/sda2 1026048317460471536 fd Linux raid autodetect /dev/sda331746048 3906971647 1937612800 fd Linux raid autodetect [root@srv-rhsoft:/downloads]$ sfdisk -d /dev/sda # partition table of /dev/sda unit: sectors /dev/sda1 : start= 2048, size= 1024000, Id=fd, bootable /dev/sda2 : start= 1026048, size= 3072, Id=fd /dev/sda3 : start= 31746048, size=3875225600, Id=fd /dev/sda4 : start=0, size=0, Id= 0 That's the MD partition alignment, not the alignment of the FS space within the MD device. The two are not the same. MD superblock versions 0.9 and 1.0 are the the end of the block device. Versions 1.1 and 1.2 are at the beginning of the device. I don't recall off the top of my head whether the superblock size and offset ensure that the /dev/mdX device's space is actually aligned to a 4KB boundary on the raw underlying device. Throw in the SSD erase block sizes and RAID chunk size and you actually have to think through your alignment throughout the entire stack or you'll get it wrong. And if you are using ext*, make sure you adjust your block group size (-g) so that the block group beginnings rotate around the disks - default values more often than not lead to all the block groups starting on the same disk which then becomes a massive bottleneck. See here for details: http://www.altechnative.net/2010/12/31/disk-and-file-system-optimisation/ The article is old enough that it refers to fdisk in CHS rather than LBA mode (CHS is still default in EL6), but the general principle still applies in exactly the same way. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 20:24, schrieb Gordan Bobic: > On 19/02/2013 19:05, Reindl Harald wrote: >> >> >> Am 19.02.2013 20:02, schrieb Gordan Bobic: what exactly do you need to align on the partitions? >>> >>> For a start, making sure your RAID implementation puts the metadata >>> at the end of the disk, rather than the beginning. >> >> "my RAID implementation"? >> LINUX SOFTWARE RAID >> >> and this is how the raid-partitions are looking >> no problem since years >> >> Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors >> Units = sectors of 1 * 512 = 512 bytes >> Sector size (logical/physical): 512 bytes / 512 bytes >> I/O size (minimum/optimal): 512 bytes / 512 bytes >> Disk identifier: 0xae2c >> Device Boot Start End Blocks Id System >> /dev/sda1 *2048 1026047 512000 fd Linux raid autodetect >> /dev/sda2 1026048317460471536 fd Linux raid autodetect >> /dev/sda331746048 3906971647 1937612800 fd Linux raid autodetect >> >> [root@srv-rhsoft:/downloads]$ sfdisk -d /dev/sda >> # partition table of /dev/sda >> unit: sectors >> /dev/sda1 : start= 2048, size= 1024000, Id=fd, bootable >> /dev/sda2 : start= 1026048, size= 3072, Id=fd >> /dev/sda3 : start= 31746048, size=3875225600, Id=fd >> /dev/sda4 : start=0, size=0, Id= 0 > > That's the MD partition alignment, not the alignment of the FS space within > the MD device. The two are not the same. maybe you should read older posts in the thread [root@srv-rhsoft:/downloads]$ tune2fs -l /dev/md1 tune2fs 1.42.5 (29-Jul-2012) Filesystem volume name: system Last mounted on: / Filesystem UUID: b935b5db-0051-4f7f-83ac-6a6651fe0988 Filesystem magic number: 0xEF53 Filesystem revision #:1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options:journal_data_writeback user_xattr acl nobarrier Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1921360 Block count: 7679232 Reserved block count: 0 Free blocks: 5785905 Free inodes: 1728014 First block: 0 Block size: 4096 Fragment size:4096 Reserved GDT blocks: 1022 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8176 Inode blocks per group: 511 RAID stride: 128 RAID stripe width:256 Flex block group size:16 Filesystem created: Wed Jun 8 13:10:52 2011 Last mount time: Fri Feb 15 20:27:01 2013 Last write time: Fri Feb 15 20:26:59 2013 Mount count: 7 Maximum mount count: -1 Last checked: Tue Feb 5 10:23:45 2013 Check interval: 31104000 (12 months) Next check after: Fri Jan 31 10:23:45 2014 Lifetime writes: 1279 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode:8 First orphan inode: 1308180 Default directory hash: half_md4 Directory Hash Seed: d48fdbd1-5e0c-4dc8-8d0b-7e3c0d6c1e18 Journal backup: inode blocks [root@srv-rhsoft:/downloads]$ tune2fs -l /dev/md2 tune2fs 1.42.5 (29-Jul-2012) Filesystem volume name: data Last mounted on: /tmp Filesystem UUID: 1abf071b-0c78-4b82-bb21-b3dfb269afa8 Filesystem magic number: 0xEF53 Filesystem revision #:1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options:journal_data_writeback user_xattr acl nobarrier Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 242204672 Block count: 968805632 Reserved block count: 0 Free blocks: 524215841 Free inodes: 241237943 First block: 0 Block size: 4096 Fragment size:4096 Reserved GDT blocks: 793 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 RAID stride: 128 RAID stripe width:256 Flex block group size:16 Filesystem created: Wed Jun 8 13:10:56 2011 Last mount time: Fri Feb 15 20:27:03 2013 Last write time: Fri Feb 15 20:27:03 2013 Mount count: 7 Maximum mount count: -1 Last checked: Tue Feb 5 10:23:53 2013 Check interval: 31104000 (12 months) Next check afte
Re: copy full system from old disk to a new one
On 19/02/2013 19:42, Reindl Harald wrote: Am 19.02.2013 20:24, schrieb Gordan Bobic: On 19/02/2013 19:05, Reindl Harald wrote: Am 19.02.2013 20:02, schrieb Gordan Bobic: what exactly do you need to align on the partitions? For a start, making sure your RAID implementation puts the metadata at the end of the disk, rather than the beginning. "my RAID implementation"? LINUX SOFTWARE RAID and this is how the raid-partitions are looking no problem since years Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xae2c Device Boot Start End Blocks Id System /dev/sda1 *2048 1026047 512000 fd Linux raid autodetect /dev/sda2 1026048317460471536 fd Linux raid autodetect /dev/sda331746048 3906971647 1937612800 fd Linux raid autodetect [root@srv-rhsoft:/downloads]$ sfdisk -d /dev/sda # partition table of /dev/sda unit: sectors /dev/sda1 : start= 2048, size= 1024000, Id=fd, bootable /dev/sda2 : start= 1026048, size= 3072, Id=fd /dev/sda3 : start= 31746048, size=3875225600, Id=fd /dev/sda4 : start=0, size=0, Id= 0 That's the MD partition alignment, not the alignment of the FS space within the MD device. The two are not the same. maybe you should read older posts in the thread Looking for what, exactly? [root@srv-rhsoft:/downloads]$ tune2fs -l /dev/md1 [...] This won't tell you the FS alignment against the raw underlying disk sectors. Blocks per group: 32768 This is sub-optimal in almost all cases except RAID1 or single-disk, as I explained earlier. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On Sat, Feb 16, 2013 at 8:53 PM, Martín Marqués wrote: > How can I pass my whole system from an old SATA disk (which is giving > me some trouble) to a new disk. I have 4 partitions, all of which I > have only passed /home using cp -.a If you have access to any system running Windows: EaseUS Partition Manager (Free Home Edition). Point and click freeware, just like the good old Partition Magic http://www.partition-tool.com/download.htm Supports ext2, ext3, does NOT support ext4 or other Linux FS FC -- During times of Universal Deceit, telling the truth becomes a revolutionary act Durante épocas de Engaño Universal, decir la verdad se convierte en un Acto Revolucionario - George Orwell -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 20:59, schrieb Gordan Bobic: > On 19/02/2013 19:42, Reindl Harald wrote: >> >> >> Am 19.02.2013 20:24, schrieb Gordan Bobic: >>> On 19/02/2013 19:05, Reindl Harald wrote: Am 19.02.2013 20:02, schrieb Gordan Bobic: >> what exactly do you need to align on the partitions? > > For a start, making sure your RAID implementation puts the metadata > at the end of the disk, rather than the beginning. "my RAID implementation"? LINUX SOFTWARE RAID and this is how the raid-partitions are looking no problem since years Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xae2c Device Boot Start End Blocks Id System /dev/sda1 *2048 1026047 512000 fd Linux raid autodetect /dev/sda2 1026048317460471536 fd Linux raid autodetect /dev/sda331746048 3906971647 1937612800 fd Linux raid autodetect [root@srv-rhsoft:/downloads]$ sfdisk -d /dev/sda # partition table of /dev/sda unit: sectors /dev/sda1 : start= 2048, size= 1024000, Id=fd, bootable /dev/sda2 : start= 1026048, size= 3072, Id=fd /dev/sda3 : start= 31746048, size=3875225600, Id=fd /dev/sda4 : start=0, size=0, Id= 0 >>> >>> That's the MD partition alignment, not the alignment of the FS space within >>> the MD device. The two are not the >>> same. >> >> maybe you should read older posts in the thread > > Looking for what, exactly? > >> [root@srv-rhsoft:/downloads]$ tune2fs -l /dev/md1 > [...] > This won't tell you the FS alignment against the raw underlying disk sectors. > >> Blocks per group: 32768 > > This is sub-optimal in almost all cases except RAID1 or single-disk, as I > explained earlier how do you meassure "sub-optimal"? the time wasted to re-install a complex setup and start configuration from scratch, especially if you have more than one clones of the same machine will never be worth 1,2.3% of theoretical performance so i do not give a damn about a few percent and as long SSD's are way to expensive to store some TB in RAID setups they are no option and if they are at a valid price they are free from early-adopters problems as said: NOW i would not save any important data on a SSD and i do not own any unimportant data beause if it is unimportant i go ahead and delete it at all maybe you install from scratch regulary i do never and i am on board since Fedora Core 3 signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 21:12, schrieb Fernando Cassia: > On Sat, Feb 16, 2013 at 8:53 PM, Martín Marqués > wrote: >> How can I pass my whole system from an old SATA disk (which is giving >> me some trouble) to a new disk. I have 4 partitions, all of which I >> have only passed /home using cp -.a > > If you have access to any system running Windows: > EaseUS Partition Manager (Free Home Edition). > > Point and click freeware, just like the good old Partition Magic > > http://www.partition-tool.com/download.htm > > Supports ext2, ext3, does NOT support ext4 or other Linux FS oh no do not come up with windows they other side around using linux to clone windows makes more sense signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19/02/2013 20:15, Reindl Harald wrote: Am 19.02.2013 20:59, schrieb Gordan Bobic: On 19/02/2013 19:42, Reindl Harald wrote: Am 19.02.2013 20:24, schrieb Gordan Bobic: On 19/02/2013 19:05, Reindl Harald wrote: Am 19.02.2013 20:02, schrieb Gordan Bobic: what exactly do you need to align on the partitions? For a start, making sure your RAID implementation puts the metadata at the end of the disk, rather than the beginning. "my RAID implementation"? LINUX SOFTWARE RAID and this is how the raid-partitions are looking no problem since years Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xae2c Device Boot Start End Blocks Id System /dev/sda1 *2048 1026047 512000 fd Linux raid autodetect /dev/sda2 1026048317460471536 fd Linux raid autodetect /dev/sda331746048 3906971647 1937612800 fd Linux raid autodetect [root@srv-rhsoft:/downloads]$ sfdisk -d /dev/sda # partition table of /dev/sda unit: sectors /dev/sda1 : start= 2048, size= 1024000, Id=fd, bootable /dev/sda2 : start= 1026048, size= 3072, Id=fd /dev/sda3 : start= 31746048, size=3875225600, Id=fd /dev/sda4 : start=0, size=0, Id= 0 That's the MD partition alignment, not the alignment of the FS space within the MD device. The two are not the same. maybe you should read older posts in the thread Looking for what, exactly? [root@srv-rhsoft:/downloads]$ tune2fs -l /dev/md1 [...] This won't tell you the FS alignment against the raw underlying disk sectors. Blocks per group: 32768 This is sub-optimal in almost all cases except RAID1 or single-disk, as I explained earlier how do you meassure "sub-optimal"? If you have a disk that you have to hit for every file access that isn't cached, that's extremely sub-optimal when you could be distributing that load across all of your disks. It's essentialy 1/n of performance you could be getting where n is the number of disks. the time wasted to re-install a complex setup and start configuration from scratch, especially if you have more than one clones of the same machine will never be worth 1,2.3% of theoretical performance Theoretical performance gains are much greater than that. Real performance gains largely depend on the size of your FS and the amount of free RAM you have for caching the block group header metadata. And yes - not getting things right the first time is expensive. So get it right first time. :) so i do not give a damn about a few percent and as long SSD's are way to expensive to store some TB in RAID setups they are no option and if they are at a valid price they are free from early-adopters problems as said: NOW i would not save any important data on a SSD and i do not own any unimportant data beause if it is unimportant i go ahead and delete it at all You really have little or no idea what you are talking about. I have had many more mechanical disks fail than SSDs, and I have yet to see an SSD (proper SSD, not generic USB/CF/SD flash media) actually fail due to wearout. SSDs at the very least tend to fail more gracefully (become read-only) than mechanical disks (typically massive media failure). Flash write endurance on reasonable branded SSDs simply isn't an issue in any way, even with worst case scenario (e.g. runnaway logging application). Even if your media is only good for 3000 erase cycles and you have a 60GB SSD, that is 180TB of writes before you wear it out. Even if you overwrite the whole disk once per day that is still 8 years of continuous operation. It's not worth worrying about. Not by a long way. To give you an idea, I have a 24/7 server here, with a 4GB rootfs (ext4, no journal), including /var/log, and gets yum updated reasonably regularly. It was created in May 2011, and has since then seen a grand total of 31GB of writes, according to dumpe2fs. If it were on flash media, that would be about 8 full overwrites. 2992 remaining. Or to put it another way, it has used up about 0.26% of it's life expectancy so far over 20 months. I would worry far more about silent bit-rot on traditional RAID mechanical disks. maybe you install from scratch regulary i do never and i am on board since Fedora Core 3 My re-installation cycle broadly follows the EL release cycles. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 21:55, schrieb Gordan Bobic: > To give you an idea, I have a 24/7 server here, with a 4GB rootfs (ext4, no > journal), including /var/log, and gets > yum updated reasonably regularly. It was created in May 2011, and has since > then seen a grand total of 31GB of > writes, according to dumpe2fs. If it were on flash media, that would be about > 8 full overwrites. 2992 remaining. Or > to put it another way, it has used up about 0.26% of it's life expectancy so > far over 20 months. I would worry far > more about silent bit-rot on traditional RAID mechanical disks. you really do not need to give me an idea :-) Filesystem created: Wed Jun 8 13:10:56 2011 Lifetime writes: 44 TB signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 19/02/2013 21:01, Reindl Harald wrote: Am 19.02.2013 21:55, schrieb Gordan Bobic: To give you an idea, I have a 24/7 server here, with a 4GB rootfs (ext4, no journal), including /var/log, and gets yum updated reasonably regularly. It was created in May 2011, and has since then seen a grand total of 31GB of writes, according to dumpe2fs. If it were on flash media, that would be about 8 full overwrites. 2992 remaining. Or to put it another way, it has used up about 0.26% of it's life expectancy so far over 20 months. I would worry far more about silent bit-rot on traditional RAID mechanical disks. you really do not need to give me an idea :-) Filesystem created: Wed Jun 8 13:10:56 2011 Lifetime writes: 44 TB On how big a file system / disk? Even if it was a 4GB SSD, you'd still be at 25% of it's life expectancy after 18 months. That's 6 years of total projected life expectancy. I have grown to expect less out of my mechanical disks. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 19.02.2013 22:42, schrieb Gordan Bobic: > On 19/02/2013 21:01, Reindl Harald wrote: >> >> >> Am 19.02.2013 21:55, schrieb Gordan Bobic: >>> To give you an idea, I have a 24/7 server here, with a 4GB rootfs (ext4, no >>> journal), including /var/log, and gets >>> yum updated reasonably regularly. It was created in May 2011, and has since >>> then seen a grand total of 31GB of >>> writes, according to dumpe2fs. If it were on flash media, that would be >>> about 8 full overwrites. 2992 remaining. Or >>> to put it another way, it has used up about 0.26% of it's life expectancy >>> so far over 20 months. I would worry far >>> more about silent bit-rot on traditional RAID mechanical disks. >> >> you really do not need to give me an idea :-) >> >> Filesystem created: Wed Jun 8 13:10:56 2011 >> Lifetime writes: 44 TB > > On how big a file system / disk? 3,6T = 12 overwrites > Even if it was a 4GB SSD, you'd still be at 25% of it's life expectancy after > 18 months. That's 6 years of total > projected life expectancy. I have grown to expect less out of my mechanical > disks ah and you have long time expierience about SSD's over 6 years? i doubt no and i doubt they are not as relieable as you would wish data without a raid are useless and in case of RAID you have to have at least one full backup and so it does not care me if disks are dying the real important daza are on RAID6 with 12 SAS disks, have daily backups offsite and weekly de-duplication backups as also monthly off-site backups with switch backup media every second month the other machines are mirrored daily to the other side of the city and backuped one a month that said: i have never lost any data in my whole life signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
2013/2/19 Fernando Cassia : > On Sat, Feb 16, 2013 at 8:53 PM, Martín Marqués > wrote: >> How can I pass my whole system from an old SATA disk (which is giving >> me some trouble) to a new disk. I have 4 partitions, all of which I >> have only passed /home using cp -.a > > If you have access to any system running Windows: > EaseUS Partition Manager (Free Home Edition). > > Point and click freeware, just like the good old Partition Magic > > http://www.partition-tool.com/download.htm > > Supports ext2, ext3, does NOT support ext4 or other Linux FS > FC > Don't have Windows here. Also, old disk had XFS on all but root partition. Anyway, had lots of problems with the data so I re-installed Fedora (I was able to get almost all that was in the /home directory). Thanks anyway Fernando. -- Martín Marqués select 'martin.marques' || '@' || 'gmail.com' DBA, Programador, Administrador -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/19/2013 03:14 PM, Martín Marqués wrote: Anyway, had lots of problems with the data so I re-installed Fedora (I was able to get almost all that was in the /home directory). If you're planning to start doing backups to a flash drive, consider reformatting it to ext4. Most Linux backup software will use links to save space when the file hasn't changed, but you can't do that in vfat. Right now, I have a 16GB flash drive with 17.3GB used and 14.5GB available, which will give you an idea of how much more economical this is. And, if you're not planning on keeping (at least) /home backed up, you should. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/19/2013 10:00 PM, Reindl Harald wrote: Am 19.02.2013 22:42, schrieb Gordan Bobic: On 19/02/2013 21:01, Reindl Harald wrote: Am 19.02.2013 21:55, schrieb Gordan Bobic: To give you an idea, I have a 24/7 server here, with a 4GB rootfs (ext4, no journal), including /var/log, and gets yum updated reasonably regularly. It was created in May 2011, and has since then seen a grand total of 31GB of writes, according to dumpe2fs. If it were on flash media, that would be about 8 full overwrites. 2992 remaining. Or to put it another way, it has used up about 0.26% of it's life expectancy so far over 20 months. I would worry far more about silent bit-rot on traditional RAID mechanical disks. you really do not need to give me an idea :-) Filesystem created: Wed Jun 8 13:10:56 2011 Lifetime writes: 44 TB On how big a file system / disk? 3,6T = 12 overwrites Even if it was a 4GB SSD, you'd still be at 25% of it's life expectancy after 18 months. That's 6 years of total projected life expectancy. I have grown to expect less out of my mechanical disks ah and you have long time expierience about SSD's over 6 years? i doubt no and i doubt they are not as relieable as you would wish No, my experience does not go as far back 6 years for obvious reasons. My exprience with mechanical disks, however, goes as far back as 25 years, and I can promise you, they are every bit as unreliable as you fear the SSDs might be. data without a raid are useless My point was that even RAID is next to useless because it doesn't protect you against bit-rot. and in case of RAID you have to have at least one full backup and so it does not care me if disks are dying Depends how many versioned backups you have, I suppose. It is possible to not notice RAID silenced bit-rot for a long time, especially with a lot of data. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On Tue, Feb 19, 2013 at 3:52 PM, Gordan Bobic wrote: > On 02/19/2013 10:00 PM, Reindl Harald wrote: > >> >> > No, my experience does not go as far back 6 years for obvious reasons. My > exprience with mechanical disks, however, goes as far back as 25 years, and > I can promise you, they are every bit as unreliable as you fear the SSDs > might be. > > So, my experience with mechanical disks dates back 25 years as well (my first was a 5.25" HH 20M in a PC I bought in 1986), but I've had more frightening experiences with SSDs (and yet I still use them) than I have with conventional drives. I've had 3 complete and total failures of name-brand SSD (all from the same vendor, unfortunately) within the course of 1 year, all drives were less than 1 year old, and were deployed in fairly conventional desktop machines--one was a warranty replacement of an earlier failure. I've had unpleasant experiences with conventional disks as well, but I don't believe I've ever had more than one conventional drive fail so completely that _no_ data could be recovered--all of my SSD failures were like that. > > data without a raid are useless >> > > My point was that even RAID is next to useless because it doesn't protect > you against bit-rot. > > As we all know, both conventional drives (and I believe SSDs) use extensive error detection/correction so that the drive will know if a block is unreliable (most of the time the drive will manage to remap that block elsewhere before it becomes unrecoverable)--individual drives only _very_ rarely manage to return the wrong data (I'm actually not sure I've _ever_ seen that happen). The problem with RAID is when no one is looking to see if the RAID system had to correct blocks--once you see more than a couple of RAID corrections happen, it is time to replace a disk--if no one looks at the logs, then eventually, there will be double (or in the case of RAID6, triple) failure, and you will lose data. A further problem with RAID is when some of the blocks are never read. Any reasonable RAID controller will not only make the log of RAID corrections available (mine helpfully emails me when corrections happen), but will also have the option of scanning the entire RAID volume periodically to look for undetected individual block failures (my system does this scan 2x per week). I've never used software RAID, so I don't know if these options are available (but I assume they are). It would be suicidal to rely on any RAID system that didn't offer both logs of corrections as well as an easy way to scan every single block (including unused blocks) looking for unnoticed issues. > > and in case of >> RAID you have to have at least one full backup >> and so it does not care me if disks are dying >> > > Depends how many versioned backups you have, I suppose. It is possible to > not notice RAID silenced bit-rot for a long time, especially with a lot of > data. > > I have a 5x1TB RAID5 (plus 1 hot spare) system (I suppose this is no longer considered a lot of data, but it was to me when I built it) that has _never_ had an unrecoverable problem--and I've now replaced every drive at least once (and I just started a migration to a 3x3TB RAID 5 w/ spare before any more fail)--I built my system in late 2003 (with 250GB drives), and the only time the RAID system has been down for more than a few minutes is when I migrate either drives or controller (or when I upgrade fedora). -se -- They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. --Benjamin Franklin, 1759 -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/20/13 00:52, Gordan Bobic wrote: > On 02/19/2013 10:00 PM, Reindl Harald wrote: […] >> data without a raid are useless > > My point was that even RAID is next to useless because it doesn't > protect you against bit-rot. Stone carving is what we aim? Again!? :) Can I take this[1] for comfort? Cheers, poma [1] tune2fs -l /dev/md2 … Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Checksum: 0x62b7798f -- -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
Am 20.02.2013 00:52, schrieb Gordan Bobic: > On 02/19/2013 10:00 PM, Reindl Harald wrote: >> data without a raid are useless > > My point was that even RAID is next to useless because it doesn't protect you > against bit-rot. it does >> and in case of >> RAID you have to have at least one full backup >> and so it does not care me if disks are dying > > Depends how many versioned backups you have, I suppose. It is possible to not > notice RAID silenced bit-rot for a > long time, especially with a lot of data what do you think why a weakly raid-check is running per default and any high-grade RAID does srub? signature.asc Description: OpenPGP digital signature -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 20/02/2013 06:22, Steve Ellis wrote: On Tue, Feb 19, 2013 at 3:52 PM, Gordan Bobic mailto:gor...@bobich.net>> wrote: On 02/19/2013 10:00 PM, Reindl Harald wrote: No, my experience does not go as far back 6 years for obvious reasons. My exprience with mechanical disks, however, goes as far back as 25 years, and I can promise you, they are every bit as unreliable as you fear the SSDs might be. So, my experience with mechanical disks dates back 25 years as well (my first was a 5.25" HH 20M in a PC I bought in 1986), but I've had more frightening experiences with SSDs (and yet I still use them) than I have with conventional drives. I've had 3 complete and total failures of name-brand SSD (all from the same vendor, unfortunately) within the course of 1 year, all drives were less than 1 year old, and were deployed in fairly conventional desktop machines--one was a warranty replacement of an earlier failure. I've had unpleasant experiences with conventional disks as well, but I don't believe I've ever had more than one conventional drive fail so completely that _no_ data could be recovered--all of my SSD failures were like that. 3 out of how many? Bad models happen all the time, for example ST31000340AS. I originally bought 4, and out of those 4 drives I originally bought I've had 6 replacements under warranty so far (6 months of warranty left). Some with total media failure, some bricked completely after a secure-erase (often the only way to get some of the pending sectors to reallocate reliably with some drives with broken firmwares). Some just ran out of spare sectors to reallocate (softest failure I've seen on those. A few years ago I got a pair of HD501LJ drives - both suffered massive media failure, and while no doubt some data would have been recoverable it would have taken so long with the failing drives restoring onto a fresh pair of drives was more expedient. It took, IIRC, 8 replacement drives to actually get a pair that fully worked and passed all of it's built in SMART tests. I wrote up some of the experience here http://www.altechnative.net/2011/03/21/the-appalling-design-of-hard-disks/ along with other "shouldn't happen" failure modes. I'm not saying SSDs are any better, but I don't think they are any worse, either. data without a raid are useless My point was that even RAID is next to useless because it doesn't protect you against bit-rot. As we all know, both conventional drives (and I believe SSDs) use extensive error detection/correction so that the drive will know if a block is unreliable (most of the time the drive will manage to remap that block elsewhere before it becomes unrecoverable) Drives simply do not do that in normal operation. Once the sector rots out, it'll get reallocated on next write and you'll lose it's contents. The only case where the drive will automatically do any re-mapping before data loss occurs is when Write-Read-Verify feature is enabled: http://www.altechnative.net/2011/04/06/enabling-write-read-verify-feature-on-disks/ I upstreamed a patch to toggle this to hdparm (it's now been in the main release for a year or so). Unfortunately, very few disks have this feature. I've only found Seagates to have it, and not even all of them. --individual drives only _very_ rarely manage to return the wrong data (I'm actually not sure I've _ever_ seen that happen). I've seen it happen pretty regularly. Healing the pending sectors tends to have massive knock on effects on performance, though, especially if there is more than one (and they usually come in sets). I just stick with ZFS - I can run SMART tests to identify the bad sectors, then just clobber them to try to get them to reallocate. Scrub can then still use the checksums to establish which is the correct set of block to reassemble and restore the data to the one that has been clobbered. Far better and more reliable than depending on traditional RAID. The problem with RAID is when no one is looking to see if the RAID system had to correct blocks--once you see more than a couple of RAID corrections happen, it is time to replace a disk--if no one looks at the logs, then eventually, there will be double (or in the case of RAID6, triple) failure, and you will lose data. Replacing disks after only couple of reallocated sectors is going to get expensive. Most disks today have the specified unrecoverable error rate of 10^-14 bits, which means an unrecoverable sector every 11TB or so. So if you have a 5+1 RAID5 array, and you lose a disk, the chances of encountering an unrecoverable sector during rebuild is about 50% - not good. A further problem with RAID is when some of the blocks are never read. Any reasonable RAID controller will not only make the log of RAID corrections available (mine helpfully emails me when corrections happen), but will also have the option of scanning the entire RAID volume periodically to l
Re: copy full system from old disk to a new one
On 20/02/2013 08:20, poma wrote: On 02/20/13 00:52, Gordan Bobic wrote: On 02/19/2013 10:00 PM, Reindl Harald wrote: […] data without a raid are useless My point was that even RAID is next to useless because it doesn't protect you against bit-rot. Stone carving is what we aim? Again!? :) Well, the data retention rates have been tested on those to be good for thousands of years. The data density in bits per square inch per platter, however, is quite poor. Depends on how much data you need to store and for how long, I suppose. :) Can I take this[1] for comfort? > [1] tune2fs -l /dev/md2 … Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Checksum: 0x62b7798f -- Not really. That will keep some of your metadata safe, but not your data. Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 20/02/2013 09:01, Reindl Harald wrote: Am 20.02.2013 00:52, schrieb Gordan Bobic: On 02/19/2013 10:00 PM, Reindl Harald wrote: data without a raid are useless My point was that even RAID is next to useless because it doesn't protect you against bit-rot. it does OK, say you have RAID1. One of the disks has a rotted sector. Your RAID implementation can detect that the mirrors don't match but there's no way to tell which is correct without an additional checksum. Similar for RAID5 - you can detect that the data doesn't match the checksum, but you have no way of being able to tell which n out of n+1 blocks are correct. and in case of RAID you have to have at least one full backup and so it does not care me if disks are dying Depends how many versioned backups you have, I suppose. It is possible to not notice RAID silenced bit-rot for a long time, especially with a lot of data what do you think why a weakly raid-check is running per default and any high-grade RAID does srub? My point is that there isn't enough data in n+1 (RAID[1|5]) to work out which set of blocks is correct to rebuild the data from. With RAID6 you can do this, but not with lower levels of RAID. -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/20/13 11:55, Gordan Bobic wrote: […] >> Filesystem features: has_journal ext_attr resize_inode dir_index >> filetype needs_recovery extent flex_bg sparse_super large_file huge_file >> dir_nlink extra_isize metadata_csum >> Checksum: 0x62b7798f >> -- > > Not really. That will keep some of your metadata safe, but not your data. > Oh dear! modinfo zfs modinfo: ERROR: Module zfs not found. grep ZFS /boot/config-3.8.0-1.fc18.x86_64 zilch Is it Solaris? Cheers, poma -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 20/02/2013 18:01, poma wrote: On 02/20/13 11:55, Gordan Bobic wrote: […] Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Checksum: 0x62b7798f -- Not really. That will keep some of your metadata safe, but not your data. Oh dear! modinfo zfs modinfo: ERROR: Module zfs not found. grep ZFS /boot/config-3.8.0-1.fc18.x86_64 zilch Is it Solaris? It originated on Solaris, yes. There is now a Linux port: http://zfsonlinux.org/ Gordan -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On 02/20/13 19:17, Gordan Bobic wrote: > On 20/02/2013 18:01, poma wrote: >> On 02/20/13 11:55, Gordan Bobic wrote: >> […] Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Checksum: 0x62b7798f -- >>> >>> Not really. That will keep some of your metadata safe, but not your >>> data. >>> >> >> Oh dear! >> >> modinfo zfs >> modinfo: ERROR: Module zfs not found. >> grep ZFS /boot/config-3.8.0-1.fc18.x86_64 >> zilch >> Is it Solaris? > > It originated on Solaris, yes. There is now a Linux port: > http://zfsonlinux.org/ > modinfo zfs | grep parm | wc -l 75 WTF!? Are these people sane!? :) Kernel's module isn't NCC-1701! Besides zfs(rpm) collides with zfs-fuse(man, bin) installation - needed by libguestfs. "It's designed from the ground up with the notion that storage media is unreliable." is ingenious hypothesis. I wonder how it compares to Btrfs. Anyway, thanks for pointing out ;) Cheers, poma -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
Re: copy full system from old disk to a new one
On Wed, 2013-02-20 at 10:52 +, Gordan Bobic wrote: > 3x3TB RAID5 is _brave_, IMO. But hey, it's your data. :) So, are you saying there should be more drives in the array, or should he use a different RAID level altogether? Ranbir -- Kanwar Ranbir Sandhu Linux 3.7.9-101.fc17.x86_64 x86_64 GNU/Linux 09:07:45 up 1 day, 22:22, 2 users, load average: 0.29, 0.08, 0.08 -- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
