On 21/03/13 10:41, Phil Kennedy wrote: > Self-replying to add a little detail here; > > The system that failed is a Red Hat Enterprise system (5.9 at the > moment.) The system has the backuppc (3.2.0) pool living on a Promise > vTrak system in a Software RAID 10 (softlinked to a partition called > /backup). That part is fine. > > The OS drives were supposed to have been configured as a software RAID 1 > but (and for the life of me, I cannot figure out how this could happen > aside from malice or gross incompetence) the secondary drive (/dev/sdb) gross incompetence on whose part? The sysadmin that left some time ago, or the sysadmin that has been responsible for the system since the previous admin left? > apparently hadn't synced with the primary drive (/dev/sda) since August > of 2009. Literally everything (passwd, group, grub, fstab, the works) > there was almost four years old. Unfortunately for me, the primary drive > failed taking the current (though supposedly, mirrored) configs with it. > The system obviously has undergone a great deal of expansion and > tweaking in the interim. Thus, the config files in /etc/backuppc were > essentially the defaults. > > Now, the folders within the directories are there. There are directories > under /backup/pc/hostname/ by those directories do not show in the menu > when you try to to browse via the web interface. > > I'll set TreashCleanSleepSec to a ridiculous number as was suggested. > Obviously, once I realized that the system may have been eating or had > eaten data, I stopped the backuppc service. There is some data currently > in /backup/trash (though murphey's law says it won't be any of the more > important data that may be missing). If I can ID which system the data > came from, I can probably just move the data back under its host > directory, correct? under ../trash the directories are named something > like 1363794493_24518_0, if I move them under ../pc/hostname/ and give > them a name like 100, it should show as backup 100, correct? (assuming > all permissions are correct?) That sounds right to me, technically you should also rebuild the backups file, but this is not required to be able to browse the backup (in my experience). > Thanks for the pointers. This event has furthered my belief that > software RAID is crap. Interesting, in my opinion, it would further my belief that software RAID is fantastic. Hardware RAID usually uses different tools for each RAID controller brand (or model), which can be frustratingly difficult to get a proper current status from it. In addition, they often (in my experience) can have a failed drive without anyone becoming aware of it (no user close enough to hear the alarm, no alarm sounding, hard to get status, etc), and as always, if the controller fails you are SOL as far as getting a working system again.
A simple cat /proc/mdstat would have shown the current status of your software RAID array, installing mdadm and configuring would allow it to automatically send you alert emails when any drive was missing from the array, etc. Personally, I use a complete (free open source) monitoring system (www.xymon.com) with a plugin which will monitor my software raid array, this then alerts me via SMS of any failures. It may be overkill in your situation, but I would strongly suggest a minimum of mdadm to send emails on failure, though you should also consider other failures that might bite you in the future (such as backuppc dying and never running a backup, not discovered until you need to restore something, or many other possible issues). IMHO, a server which is not monitored is a disaster that you just don't know about yet. PS, that is not to say that you will never experience a disaster just because you monitor a system, there is always some new way things can break which the monitoring system did not test for, but these are much more rare, and once you experience them once, you can write the monitoring script for it (very easy with xymon). Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_mar _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/