[gentoo-amd64] Quick rsync backup before changing a RAID
Hi, I need to change the RAID layout on my home server. This change does _not_ involve the running RAID6 with Gentoo on it (at least at the moment) but only a rather old, large and ill-placed second RAID6 that has some _very_ important Virtualbox VMs. Now, I have backup scripts for protecting this stuff, and I use them every night, but I've never had to do a restore so at this point I don't know whether they really work. The files are there (checked by hand anyway) but who knows if permissions or something else might cause a problem so I want to be extra careful. Today, I created a new backup USB drive and want to do an rsync to that drive as a second backup. I intend to restore from this drive later today when the RAIDs are rebuilt and ready to go. I'd like some ideas on best practices for using rsync to do this job. My current scripts use this basic format: mark@c2RAID6 ~ $ cat ./DoBackups mount /backups rsync -avx /VirtualMachines/VMWare/. /backups/VMWare rsync -avx /VirtualMachines/Virtualbox/. /backups/Virtualbox rsync -avx /home/. /backups/home cd umount /backups mark@c2RAID6 ~ $ I'm wondering whether -avx is what others would use or whether there's a better set of options. When the RAID comes back up I want to reverse and copy back from USB drive to the /VirtualMachines directory. Thanks in advance, Mark
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 1:59 PM, Mark Knecht markkne...@gmail.com wrote: I'm wondering whether -avx is what others would use or whether there's a better set of options. I think that is the typical rsync backup command set as far as I have ever seen. It should work as you expect it to work. Note that if you use ACLs or extended attributes, you would need to additionally specify -A and/or -X since they are not implied by -a. On a side note, I recently rsync'ed a drive between computers, one running a Live CD and one not, over ssh, and somehow got all my user/groups mixed up (I guess group 20 on one box was X but on another box it was Y). Luckily I was able to identify it and fix it before things got too broken. I'm not entirely sure what I did wrong, to be honest.
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 9:59 PM, Mark Knecht markkne...@gmail.com wrote: ... I'm wondering whether -avx is what others would use or whether there's a better set of options. When the RAID comes back up I want to reverse and copy back from USB drive to the /VirtualMachines directory. Add --progress to track progress while copying. Also, be careful what filesystem the USB drive will have (better to use what you're already using), in order to preserve permissions correctly etc. Panagiotis
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 12:15 PM, Paul Hartman paul.hartman+gen...@gmail.com wrote: On Wed, Jun 19, 2013 at 1:59 PM, Mark Knecht markkne...@gmail.com wrote: I'm wondering whether -avx is what others would use or whether there's a better set of options. I think that is the typical rsync backup command set as far as I have ever seen. It should work as you expect it to work. Note that if you use ACLs or extended attributes, you would need to additionally specify -A and/or -X since they are not implied by -a. On a side note, I recently rsync'ed a drive between computers, one running a Live CD and one not, over ssh, and somehow got all my user/groups mixed up (I guess group 20 on one box was X but on another box it was Y). Luckily I was able to identify it and fix it before things got too broken. I'm not entirely sure what I did wrong, to be honest. Thanks for the reply Paul. Is 'using ACLs' one of those things that if you don't know then you are not? I just run basic Gentoo here. All installs done as per the install guide. I don't even use LVM or anything fancy. (Which is likely part of why I'm having to reconfigure as much as I am but so be it.) Anyway, thanks for the info. I'll just give it a try and then see about maybe diffing the two drives or something like to to check. Cheers, Mark
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 12:19 PM, Panagiotis Christopoulos pxr...@gmail.com wrote: On Wed, Jun 19, 2013 at 9:59 PM, Mark Knecht markkne...@gmail.com wrote: ... I'm wondering whether -avx is what others would use or whether there's a better set of options. When the RAID comes back up I want to reverse and copy back from USB drive to the /VirtualMachines directory. Add --progress to track progress while copying. Also, be careful what filesystem the USB drive will have (better to use what you're already using), in order to preserve permissions correctly etc. Panagiotis Hi Panagiotis, Good point about the file system type. Both are ext3 so no problems expected, although on restore I suspect it will end up ext4. Cheers, Mark
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
on 06/19/2013 10:15 PM Paul Hartman wrote the following: I recently rsync'ed a drive between computers, one running a Live CD and one not, over ssh, and somehow got all my user/groups mixed up You should have used the option --numeric-ids (see rsync man page)
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 12:37 PM, Thanasis thana...@asyr.hopto.org wrote: on 06/19/2013 10:15 PM Paul Hartman wrote the following: I recently rsync'ed a drive between computers, one running a Live CD and one not, over ssh, and somehow got all my user/groups mixed up You should have used the option --numeric-ids (see rsync man page) Ah, interesting option in Paul's case. In my case, as I'm going back onto the same system, I think I won't need this. Thanks, Mark
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 2:37 PM, Thanasis thana...@asyr.hopto.org wrote: on 06/19/2013 10:15 PM Paul Hartman wrote the following: I recently rsync'ed a drive between computers, one running a Live CD and one not, over ssh, and somehow got all my user/groups mixed up You should have used the option --numeric-ids (see rsync man page) Ah-ha, that looks like it would have prevented it. I will try to remember that for next time. Thank you very much for the tip.
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 3:21 PM, Mark Knecht markkne...@gmail.com wrote: Is 'using ACLs' one of those things that if you don't know then you are not? I just run basic Gentoo here. All installs done as per the install guide. I don't even use LVM or anything fancy. (Which is likely part of why I'm having to reconfigure as much as I am but so be it.) Gentoo can potentially make use of them if you're using POSIX capabilities, but I think that has to be enabled by USE flag or FEATURE. I think you might also have to enable them in fstab (and the kernel as well, though that is likely enabled by default). Chances are you'd know if you were using them, and it would only apply to stuff installed by portage (generally binaries). Oh, and using POSIX capabilities is a good thing (less suid root binaries). One thing to mention - I'm not sure how much data you're talking about, but if it is a lot and downtime is a concern you could probably get creative and avoid some copying. You could copy the files onto an LVM vg, and then after your RAID is set up you could move the logical volume onto your RAID, which is an online operation. (Ie set up your USB drive as LVM, copy the files over, adjust your mounts so that the USB drive is mounted where you want the files, and at this point they're in production. Then get your RAID set up, add it to the volume group, and pvmove the USB drive so that it gets moved to your RAID. Then remove the USB drive from your volume group and you're in the final config, with no further need to adjust mountpoints since they're already pointed to LVM.) I've done a lot of messing around with online migrations of RAID and LVM - quite handy when you have lots of stuff like mythtv recordings that you don't want to have offline backups of. Rich
Re: [gentoo-amd64] Quick rsync backup before changing a RAID
On Wed, Jun 19, 2013 at 1:20 PM, Rich Freeman ri...@gentoo.org wrote: On Wed, Jun 19, 2013 at 3:21 PM, Mark Knecht markkne...@gmail.com wrote: Is 'using ACLs' one of those things that if you don't know then you are not? I just run basic Gentoo here. All installs done as per the install guide. I don't even use LVM or anything fancy. (Which is likely part of why I'm having to reconfigure as much as I am but so be it.) Gentoo can potentially make use of them if you're using POSIX capabilities, but I think that has to be enabled by USE flag or FEATURE. I think you might also have to enable them in fstab (and the kernel as well, though that is likely enabled by default). Chances are you'd know if you were using them, and it would only apply to stuff installed by portage (generally binaries). Oh, and using POSIX capabilities is a good thing (less suid root binaries). One thing to mention - I'm not sure how much data you're talking about, but if it is a lot and downtime is a concern you could probably get creative and avoid some copying. You could copy the files onto an LVM vg, and then after your RAID is set up you could move the logical volume onto your RAID, which is an online operation. (Ie set up your USB drive as LVM, copy the files over, adjust your mounts so that the USB drive is mounted where you want the files, and at this point they're in production. Then get your RAID set up, add it to the volume group, and pvmove the USB drive so that it gets moved to your RAID. Then remove the USB drive from your volume group and you're in the final config, with no further need to adjust mountpoints since they're already pointed to LVM.) I've done a lot of messing around with online migrations of RAID and LVM - quite handy when you have lots of stuff like mythtv recordings that you don't want to have offline backups of. Rich Interesting info Rich. Thanks. Approximately 250GB involved. Not worried about downtime. The data is all Windows VMs which I use for stock futures trading. Only need the machine back up by tomorrow morning. Also, I've never used LVM so for an afternoon project it's unlikely I'd be successful going that direction I think. The second USB backup drive is about 50% done so I should be able to start working on reconfiguring the RAID in about an hour and hopefully be back up and running by 5PM if things go well. Cheers, Mark
