[gentoo-user] backup hardware setup
Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1
Re: [gentoo-user] backup hardware setup
On 06/24/2014 03:43 PM, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 I do this using hard links and rsync to only copy changed data. this creates a dated folder structure that i can then rsync / cp using a livecd to baremetal and basically allows best recoverability, imho. so long as the filesystem supports hard links you are golden. you might want btrfs for this for long term storage to help in case of bitrot, but rsync should refresh the file if it is suddenly unreadable (meaning any other hard lnked versoins are also up the swanny) ymmv depending on what it is you are backing up #!/bin/bash echo 'preparing..' date=`date +%Y-%m-%d_%H.%M.%S` workingfolder=/mnt/usb/backupsyncs/myhost1 fromfolder=root@myhost1:/* --exclude=/var/tmp --exclude=/dev --exclude=/mnt --exclude=/opt --exclude=/proc --exclude=/sys --exclude=/usr/portage --exclude=/usr/src echo Date $date echo From $fromfolder echo To$workingfolder echo move current to be dated mv $workingfolder/current $workingfolder/backup-$date echo now syncing into dated folder rsync -vz --partial --modify-window 5 -W --delete -a $fromfolder $workingfolder/backup-$date echo cleaning up..linkcopying dated folder to current cp -al $workingfolder/backup-$date $workingfolder/current
Re: [gentoo-user] backup hardware setup
thegeezer thegee...@thegeezer.net [14-06-24 17:16]: On 06/24/2014 03:43 PM, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 I do this using hard links and rsync to only copy changed data. this creates a dated folder structure that i can then rsync / cp using a livecd to baremetal and basically allows best recoverability, imho. so long as the filesystem supports hard links you are golden. you might want btrfs for this for long term storage to help in case of bitrot, but rsync should refresh the file if it is suddenly unreadable (meaning any other hard lnked versoins are also up the swanny) ymmv depending on what it is you are backing up #!/bin/bash echo 'preparing..' date=`date +%Y-%m-%d_%H.%M.%S` workingfolder=/mnt/usb/backupsyncs/myhost1 fromfolder=root@myhost1:/* --exclude=/var/tmp --exclude=/dev --exclude=/mnt --exclude=/opt --exclude=/proc --exclude=/sys --exclude=/usr/portage --exclude=/usr/src echo Date $date echo From $fromfolder echo To$workingfolder echo move current to be dated mv $workingfolder/current $workingfolder/backup-$date echo now syncing into dated folder rsync -vz --partial --modify-window 5 -W --delete -a $fromfolder $workingfolder/backup-$date echo cleaning up..linkcopying dated folder to current cp -al $workingfolder/backup-$date $workingfolder/current Hi, thank you for your reply! :) ...I am sure, whether I want btrfs. On the net I found for example this: http://www.phoronix.com/scan.php?page=news_itempx=MTY1MDU with sentences like: The Btrfs file-system changes for the Linux 3.15 kernel mostly deal with bug fixes and performance fixes while some corruption fixes are also expected to come. ...sounds a little different to stable I think... What do you think?
Re: [gentoo-user] backup hardware setup
On 06/24/2014 04:28 PM, meino.cra...@gmx.de wrote: http://www.phoronix.com/scan.php?page=news_itempx=MTY1MDU with sentences like: The Btrfs file-system changes for the Linux 3.15 kernel mostly deal with bug fixes and performance fixes while some corruption fixes are also expected to come. ...sounds a little different to stable I think... What do you think? i hear ya, but if all you are doing is something like code i submitted, you have no concerns. even ext4 had an odd corruption bug not too long ago, and that was after it was stable. more importantly there is a wide group of folks using btrfs and active development. I wouldn't bother with ext2 and ext3 -- fsck will take _forever_ and heavens help you if you unplug the usb without unmounting if you are that concerned you might want to make one drive ext4, one drive btrfs (or another pairing of your choice) and you can then guard against the filesystem choice by cycling the disks daily/weekly
Re: [gentoo-user] backup hardware setup
On 24/06/2014 16:43, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 You haven't given much in the way of detail, so I assume you have regular needs, nothing fancy, and it's all a bunch of files right? In that case, partitioning and filesystem type are largely irrelevant as long as you don't have corruption. With one caveat: You must always make sure the source drive is intact and ok. If not, and you back it up anyway, then you are already toast (you will overwrite your last backup with new faulty data). There's several approaches to how to do the transfer: If you have say a general fileserver with lots of files that don't change much or often, just rsync everything in one go. There is no optimization you can do that will perform much faster than rsync. If you have a big busy filesystem that changes often and lots, then use lvm (or anything that can make snapshots) and rsync that. If you have a huge database where everything is changing all the time, don't do filesystem copies, use the tools provided by the db vendor. I doubt this is your need as you would have said so, but it's worth mentioning. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] backup hardware setup
Alan McKinnon alan.mckin...@gmail.com [14-06-24 19:12]: On 24/06/2014 16:43, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 You haven't given much in the way of detail, so I assume you have regular needs, nothing fancy, and it's all a bunch of files right? In that case, partitioning and filesystem type are largely irrelevant as long as you don't have corruption. With one caveat: You must always make sure the source drive is intact and ok. If not, and you back it up anyway, then you are already toast (you will overwrite your last backup with new faulty data). There's several approaches to how to do the transfer: If you have say a general fileserver with lots of files that don't change much or often, just rsync everything in one go. There is no optimization you can do that will perform much faster than rsync. If you have a big busy filesystem that changes often and lots, then use lvm (or anything that can make snapshots) and rsync that. If you have a huge database where everything is changing all the time, don't do filesystem copies, use the tools provided by the db vendor. I doubt this is your need as you would have said so, but it's worth mentioning. -- Alan McKinnon alan.mckin...@gmail.com Hi Alan, thanks for your reply! :) Yes...your are right. I have a lot static (=not changing) data on my harddisk...mostly things like video tutorials (blender), videos of birds I filmed, dokuments and such... They are eating up the space on my systems harddisk. Do I decided to put them on a extern hd and an identical copy on another identical external harddisk. Its mainly a task of updateing the data on the external drives with that what is new (and static and big and falls under what I described above) on my systems harddisk. I will check rsync for that!
Re: [gentoo-user] backup hardware setup
On Tue, Jun 24, 2014 at 11:28 AM, meino.cra...@gmx.de wrote: ...I am sure, whether I want btrfs. On the net I found for example this: http://www.phoronix.com/scan.php?page=news_itempx=MTY1MDU with sentences like: The Btrfs file-system changes for the Linux 3.15 kernel mostly deal with bug fixes and performance fixes while some corruption fixes are also expected to come. ...sounds a little different to stable I think... I have mixed feelings regarding btrfs. For a disk that is going to be mostly offline, low-activity, long-term storage btrfs has some plusses and minuses, especially since you intend to mirror it (manually or otherwise). The only real minus is that btrfs is still fairly experimental. You could run into problems. However, what you're doing is a very simple use case and the write loads are certainly going to be quite low. Most of the scenarios that cause issues with btrfs are unlikely to come up. Btrfs has a few advantages. I'd say the biggest one is that it checksums everything and can detect silent corruption. For a disk that is just going to sit around for a long time offline that is a big plus - any other filesystem option isn't going to detect any corruption to your archive other than unreadable clusters (or disks). You could get around this by only storing data in a format that already can detect corruption, and then writing scripts to check everything, but you'll be manually copying data across your mirror if you find issues, and that will be really tricky if you're using something like mdadm since there is no easy way to pick which copy it gives you. Btrfs can also mirror your data which guarantees that all of it is replicated. Sure, rsync can do this also, but if for whatever reason something gets changed without updating mtime/ctime/size it won't spot it unless you have it set to hash everything (which is VERY slow so nobody does this). Mdadm would be a better choice, but as I pointed out it can't detect silent corruption, and is hard to recover if you discover it yourself. With btrfs everything is always mirrored (if you set that up) and a simple scrub command periodically will test to make sure everything is fine and restore anything that isn't. I have some old hard drives that I'm using for storage and I use btrfs on them. However, ultimately it isn't anything I can't afford to lose either. There is nothing wrong with ext4+rsync and maybe an occasional recursive diff. You just need to do a bit more with scripting/etc and make sure you stay on top of it, and watch out for gotchas like having the wrong disk mounted/etc. That will be less efficient than mdadm or btrfs since rsync has no idea what has changed without walking the whole tree, but it probably isn't a big deal and minus the automation is probably the most bulletproof option. Just keep in mind it is only bulletproof if you don't miss anything when you set it up - this is relying on you to manage the mirroring/etc and if you drop the ball then data will be at risk. So, I won't enthusiastically recommend btrfs, but it may be worth consideration... Rich
Re: [gentoo-user] backup hardware setup
On 24/06/2014 19:32, meino.cra...@gmx.de wrote: Alan McKinnon alan.mckin...@gmail.com [14-06-24 19:12]: On 24/06/2014 16:43, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 You haven't given much in the way of detail, so I assume you have regular needs, nothing fancy, and it's all a bunch of files right? In that case, partitioning and filesystem type are largely irrelevant as long as you don't have corruption. With one caveat: You must always make sure the source drive is intact and ok. If not, and you back it up anyway, then you are already toast (you will overwrite your last backup with new faulty data). There's several approaches to how to do the transfer: If you have say a general fileserver with lots of files that don't change much or often, just rsync everything in one go. There is no optimization you can do that will perform much faster than rsync. If you have a big busy filesystem that changes often and lots, then use lvm (or anything that can make snapshots) and rsync that. If you have a huge database where everything is changing all the time, don't do filesystem copies, use the tools provided by the db vendor. I doubt this is your need as you would have said so, but it's worth mentioning. -- Alan McKinnon alan.mckin...@gmail.com Hi Alan, thanks for your reply! :) Yes...your are right. I have a lot static (=not changing) data on my harddisk...mostly things like video tutorials (blender), videos of birds I filmed, dokuments and such... They are eating up the space on my systems harddisk. Do I decided to put them on a extern hd and an identical copy on another identical external harddisk. Its mainly a task of updateing the data on the external drives with that what is new (and static and big and falls under what I described above) on my systems harddisk. I will check rsync for that! That changes things just a little bit - I thought your two drives were going to be one for live and one for backup. Do you intend to move these files off your main drive onto the identical externals, or just copy the files? I would have those two external drives using different filesystems, just in case as they are your only copy and external drives are fragile in use and in storage. Exact fs type doesn't really matter - ext4 and xfs, or ext* and btrfs, it's all good. Just do make sure you don't use rsync with --delete for this :-) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] backup hardware setup
Alan McKinnon alan.mckin...@gmail.com [14-06-24 20:00]: On 24/06/2014 19:32, meino.cra...@gmx.de wrote: Alan McKinnon alan.mckin...@gmail.com [14-06-24 19:12]: On 24/06/2014 16:43, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 You haven't given much in the way of detail, so I assume you have regular needs, nothing fancy, and it's all a bunch of files right? In that case, partitioning and filesystem type are largely irrelevant as long as you don't have corruption. With one caveat: You must always make sure the source drive is intact and ok. If not, and you back it up anyway, then you are already toast (you will overwrite your last backup with new faulty data). There's several approaches to how to do the transfer: If you have say a general fileserver with lots of files that don't change much or often, just rsync everything in one go. There is no optimization you can do that will perform much faster than rsync. If you have a big busy filesystem that changes often and lots, then use lvm (or anything that can make snapshots) and rsync that. If you have a huge database where everything is changing all the time, don't do filesystem copies, use the tools provided by the db vendor. I doubt this is your need as you would have said so, but it's worth mentioning. -- Alan McKinnon alan.mckin...@gmail.com Hi Alan, thanks for your reply! :) Yes...your are right. I have a lot static (=not changing) data on my harddisk...mostly things like video tutorials (blender), videos of birds I filmed, dokuments and such... They are eating up the space on my systems harddisk. Do I decided to put them on a extern hd and an identical copy on another identical external harddisk. Its mainly a task of updateing the data on the external drives with that what is new (and static and big and falls under what I described above) on my systems harddisk. I will check rsync for that! That changes things just a little bit - I thought your two drives were going to be one for live and one for backup. Do you intend to move these files off your main drive onto the identical externals, or just copy the files? I would have those two external drives using different filesystems, just in case as they are your only copy and external drives are fragile in use and in storage. Exact fs type doesn't really matter - ext4 and xfs, or ext* and btrfs, it's all good. Just do make sure you don't use rsync with --delete for this :-) -- Alan McKinnon alan.mckin...@gmail.com Yes, I will delete the data from my systems drive... You wrote: I would have those two external drives using different filesystems Different to what? Different to the fs on the system drive? Both external drives use different filesystems? All three use different filesystems? And how can this help, if the drives are fragile? (I understand fragile as mechanical not robust (sorry I am no native english speaker)) I will use this mobile disks not really as the word mobile implies. They will only travel manually between a secure place and my PC. When in use, they will rest on the floor of the room (so they can not be dropped) and _under_ the case of my PC (ole school big tower metal case with a gap between the bottom of the case and the floor of the room.)
Re: [gentoo-user] backup hardware setup
On 24/06/2014 20:34, meino.cra...@gmx.de wrote: Alan McKinnon alan.mckin...@gmail.com [14-06-24 20:00]: On 24/06/2014 19:32, meino.cra...@gmx.de wrote: Alan McKinnon alan.mckin...@gmail.com [14-06-24 19:12]: On 24/06/2014 16:43, meino.cra...@gmx.de wrote: Hi, I bought two identical external harddrives, USB 3.0, with 1 TByte each (no SSD - the good ole mechanical ones...;). The intended use is for backup of longer files. The drives will contain the same contents. Currently there are still clean metal (no partitioning, no fs). Data integrity and recoverability (Uhhh...that words looks wrong...) in case of an desaster is more important than speed. What is the recommended way of partitioning ? What filesystem to choose? Thank you very much in advance for any help! Best regards, mcc PS: Running vanilla kernel 3.15.1 You haven't given much in the way of detail, so I assume you have regular needs, nothing fancy, and it's all a bunch of files right? In that case, partitioning and filesystem type are largely irrelevant as long as you don't have corruption. With one caveat: You must always make sure the source drive is intact and ok. If not, and you back it up anyway, then you are already toast (you will overwrite your last backup with new faulty data). There's several approaches to how to do the transfer: If you have say a general fileserver with lots of files that don't change much or often, just rsync everything in one go. There is no optimization you can do that will perform much faster than rsync. If you have a big busy filesystem that changes often and lots, then use lvm (or anything that can make snapshots) and rsync that. If you have a huge database where everything is changing all the time, don't do filesystem copies, use the tools provided by the db vendor. I doubt this is your need as you would have said so, but it's worth mentioning. -- Alan McKinnon alan.mckin...@gmail.com Hi Alan, thanks for your reply! :) Yes...your are right. I have a lot static (=not changing) data on my harddisk...mostly things like video tutorials (blender), videos of birds I filmed, dokuments and such... They are eating up the space on my systems harddisk. Do I decided to put them on a extern hd and an identical copy on another identical external harddisk. Its mainly a task of updateing the data on the external drives with that what is new (and static and big and falls under what I described above) on my systems harddisk. I will check rsync for that! That changes things just a little bit - I thought your two drives were going to be one for live and one for backup. Do you intend to move these files off your main drive onto the identical externals, or just copy the files? I would have those two external drives using different filesystems, just in case as they are your only copy and external drives are fragile in use and in storage. Exact fs type doesn't really matter - ext4 and xfs, or ext* and btrfs, it's all good. Just do make sure you don't use rsync with --delete for this :-) -- Alan McKinnon alan.mckin...@gmail.com Yes, I will delete the data from my systems drive... You wrote: I would have those two external drives using different filesystems Different to what? Different to the fs on the system drive? Both external drives use different filesystems? All three use different filesystems? Different to each other And how can this help, if the drives are fragile? (I understand fragile as mechanical not robust (sorry I am no native english speaker)) If one drive is say btrfs and the other say ext4 and you hit a corruption bug in btrfs, then you still have an uncorrupted ext4 copy I will use this mobile disks not really as the word mobile implies. They will only travel manually between a secure place and my PC. When in use, they will rest on the floor of the room (so they can not be dropped) and _under_ the case of my PC (ole school big tower metal case with a gap between the bottom of the case and the floor of the room.) External drives have a much higher failure rate than internal drives. people don't expect them to fail or be dropped or accidentally plugged in in the wrong order and the wrong one to be mkfs'ed (until it does happen). These are real risks that you can't ignore whereas with a good internal drive you can often get away with it. So it only make sense to take sensible precautions that cost you very little, especially considering these two drives will be your only copy. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] backup hardware setup
On Tue, Jun 24, 2014 at 4:02 PM, Alan McKinnon alan.mckin...@gmail.com wrote: External drives have a much higher failure rate than internal drives. people don't expect them to fail or be dropped or accidentally plugged in in the wrong order and the wrong one to be mkfs'ed (until it does happen). These are real risks that you can't ignore whereas with a good internal drive you can often get away with it. ++ Don't ignore the potential for logical errors. If you have some script that magically rsyncs stuff then don't make the mistake of moving data over and rsyncing the old copy over the new, or mounting the devices in a manner that isn't robust when udev changes all your device labels, and so on. That seems like the most likely way your data is going to get scrambled, unless you have them both in your car and end up in a crash. That was one of the reasons I went with btrfs for my offline copy. If it unmounts, then I know I have two copies of everything. If it mounts, I know it found both mirrors. If I scrub and there are no errors, then I know both copies are good. You can do that in other ways, but make sure you actually catch the failure modes. Rich
Re: [gentoo-user] backup hardware setup
On 06/24/2014 09:52 PM, Rich Freeman wrote: Don't ignore the potential for logical errors. If you have some script that magically rsyncs stuff then don't make the mistake of moving data over and rsyncing the old copy over the new, or mounting the devices in a manner that isn't robust when udev changes all your device labels, and so on. That seems like the most likely way your data is going to get scrambled, unless you have them both in your car and end up in a crash. That was one of the reasons I went with btrfs for my offline copy. cunning, i like it. i like it so much i think i will do this myself If it unmounts, then I know I have two copies of everything. If it mounts, I know it found both mirrors. If I scrub and there are no errors, then I know both copies are good. You can do that in other ways, but make sure you actually catch the failure modes. Rich
