[slim] A Cautionary Tale..
Hi All, Thought I'd share with the board my big error over the weekend and also to highlight the importance of backup! I recently bought a 1tb external HDD to mirror my /storage folder on my server (running vortexbox), its got about 450gb worth of music, photos etc, I set this backup to run using rsync and it ran daily. Over the weekend I travlled to my to parents house and brought the server with me, on Sat I couldn't get Squeezecenter to run, I worked out this was a space issue and from that noticed that the previous nights rsync job had run without the External HDD plugged in and had filled the 20gb root partition on the server. I navigated to /media/externalHdd to delete the storage folder there which had incorrectly been created with the backup job, however I made a fatal mistake and instead of rm -rf storage/* I did rm -rf /storage/* This then deleted the entire 450gb worth of music etc. it was too late to change it once I realised. Thankfully when I return home on weds my External HD will be there with the server backed up from thurs (I hope nothing has happened to it!) however I have been kicking myself ever since for the error ! Does anyone with Linux experience know of a smarter way of doing the backup, I'm presuming something like a shell script which checks for the external HDD being mounted before running the backup ? Secondly is there a way of protecting the /storage part of the server to stop running commands like the above, I log in as root when ever I'm doing work on the server Thanks Ian -- socistep socistep's Profile: http://forums.slimdevices.com/member.php?userid=18860 View this thread: http://forums.slimdevices.com/showthread.php?t=66269 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] A Cautionary Tale..
I would suggest a look at rdiff-backup (http://rdiff-backup.nongnu.org) for a similar rdiff-like solution but which will also efficiently maintain a number of previous backup snapshots as well. It's easy to script and automate, and has worked well for me for a while. I run it nightly to an internal disk (automated, only mounting that disk when needed so I couldn't accidentally delete the files on it), and weekly to one I take off-site. That way if I make a mistake such as yours I'd have an backup close to hand. To prevent accidentally backing up without the drive mounted a poor-mans way is to make sure the backup is to a directory under the root of the mount-point (eg /storage/ext/backup), rather than to the mount point root itself (/storage/ext). That way the backup should immediately fail since the target directory isn't there (ie /storage/ext/backup won't exist unless the drive is already mounted). Alternatively, wrap the backup with a short script that checks that directory is there, is present in /proc/mounts, or something similar. The obvious answer to avoiding this kind of error is not to run as root - make your media owned by a group (eg "media") and put yourself in that group. Of course you'll still be able to quickly delete everything available to that group, but the consequences wouldn't be as bad if you accidentally ran it on the root directory, for example. Other alternatives are to make 'rm' prompt for your approval (alias "rm" to "rm -i", which you can add to a login script) - according to the man page it'll then prompt if there are more than three files or a recursive rm is being performed, which would have saved you in this situation if you'd read the prompt. Also, some other shells (eg fish or zsh) can be better at catching classic errors such as this - I can't remember for sure but I recall one or both of these having some special protection. There are as many ways of backing up as there are people who reply to threads such as this, so you'll be unlikely to hear of a 'best' way...! My only other advice is to also encrypt your backups as an external drive is easy to lose or have stolen - Linux makes that easy with dm-crypt/LUKS (and it's easily scriptable). Stuart On 03/08/2009 10:16 AM, socistep wrote: > > Hi All, > > Thought I'd share with the board my big error over the weekend and also > to highlight the importance of backup! > > I recently bought a 1tb external HDD to mirror my /storage folder on my > server (running vortexbox), its got about 450gb worth of music, photos > etc, I set this backup to run using rsync and it ran daily. Over the > weekend I travlled to my to parents house and brought the server with > me, on Sat I couldn't get Squeezecenter to run, I worked out this was a > space issue and from that noticed that the previous nights rsync job had > run without the External HDD plugged in and had filled the 20gb root > partition on the server. > > I navigated to /media/externalHdd to delete the storage folder there > which had incorrectly been created with the backup job, however I made a > fatal mistake and instead of > > rm -rf storage/* > > I did > > rm -rf /storage/* > > This then deleted the entire 450gb worth of music etc. it was too > late to change it once I realised. > > Thankfully when I return home on weds my External HD will be there with > the server backed up from thurs (I hope nothing has happened to it!) > however I have been kicking myself ever since for the error ! > > Does anyone with Linux experience know of a smarter way of doing the > backup, I'm presuming something like a shell script which checks for the > external HDD being mounted before running the backup ? > > Secondly is there a way of protecting the /storage part of the server > to stop running commands like the above, I log in as root when ever I'm > doing work on the server > > Thanks > Ian > > ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] A Cautionary Tale..
socistep;445326 Wrote: > Hi All, > > > I navigated to /media/externalHdd to delete the storage folder there > which had incorrectly been created with the backup job, however I made a > fatal mistake and instead of > > rm -rf storage/* > > I did > > rm -rf /storage/* > > This then deleted the entire 450gb worth of music etc. it was too > late to change it once I realised. > > This form of the rm command is VERY dangerous, as you've discovered, and should always be use with great care. If you use the wrong form of it, you can delete every single file on your hard drive with a single command! I would suggest in the future using some form of the find command to validate what you are passing to the rm command. If this shows what you expect to see, then use the string with the rm command. Another solution would be to call your back up directory something completely different (i.e. ServerBackup). If you can't do this directly, you could create a symbolic link of this name to what you want to back up. At least you learned your lesson without causing any damage that you cannot recover from. -- maggior Rich - Setup: 2 SB3s, 3 Booms, 1 duet, 1 receiver. SuSE 11.0 Server running SqueezeCenter 7.3.3, MusicIP, and SqueezeSlave. Current library stats: 27,462 songs, 2,276 albums, 434 artists. http://www.last.fm/user/maggior maggior's Profile: http://forums.slimdevices.com/member.php?userid=9080 View this thread: http://forums.slimdevices.com/showthread.php?t=66269 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] A Cautionary Tale..
A long time ago, well before sudo came to existence, I did, as root: "rm -rf . /foobar" where I wanted "./foobar". Oh well, homedir gone, ridiculed by colleagues etc. - never work as root. never. - always use "." to prepend your path when expressing relative paths - prefer developing a small wrapper to often-used commands. " alias rm='rm -i' " in your .bashrc (or equiv.) is a simple solution. Writing a shell script called mybackup.sh or something is usually the way I go. After my mishap, I had a colleague who shared with me his personal trick: alias rm to his version of rm, which was in fact a "move" command to a specific trash directory. The trash being emptied via cron every so often. In the specific case of backing up to USB, you could use the /etc/udev/ rules files to recognize the brand of bridge that gets connected, and if this is the backup drive, then run a script: - mount to a specific location, with specific rights, - stat the filesystem, - launch the backup as a specific user, - auto-dismount at the end. This is a bit stringent, as plugging the drive will automatically command a long, uninterruptible process. Personally I prefer running a custom command at will. -- epoch1970 epoch1970's Profile: http://forums.slimdevices.com/member.php?userid=16711 View this thread: http://forums.slimdevices.com/showthread.php?t=66269 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
