Re: What logs etc do I need to checkfrequently?
As you with good memories know, I lost 3000 pictures of my first sons first year this month. I did have a RAID-5 system with fresh disks, however, shit happens and I have a feeling that this could have been avoided if I read my log files better. As has been mentioned, if you have critical or irreplaceable data, back it up.If your partitions are too large for any backup system you can use, then make a scheme for breaking the data in to more manageable sized partitions - use more partitions and even more slices and disk devices if needed. Those huge raid partitions are nice for some things, but have their drawbacks. You can get backup devices - LTO and high end DLT that hold up to 160 or more uncompressed, so that is a good single sized partition. They are expensive, but not compared to losing your data. Also, you are supposed to be able to use more than one tape in a backup. I don't know how well that works in FreeBSD dump/restore. I don't have a big enough disk under FreeBSD to make it go to more than one tape yet. But, I have successfully dumped partitions that needed up to 10 DAT tapes before on other UNIX systems. jerry So basically, ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need to checkfrequently?
Chuck Swiger wrote: Joachim Dagerot wrote: As you with good memories know, I lost 3000 pictures of my first sons first year this month. I did have a RAID-5 system with fresh disks, however, shit happens and I have a feeling that this could have been avoided if I read my log files better. I'm sorry that you lost data. While you may have been able to notice the problem with the RAID-5 array in time to do something, what you ought to do to avoid losing more data sometime in the future involves making good backups-- not poring over the system log files, not configuring RAID. So basically, a) I get a mail each time my a cron-event fires, this happens every 30 min so the mailbox are quite loaded, not very funny going through. If you can, change the cron task to not generate output unless there is a problem that you should know about. Failing that, append /dev/null 21 to the line in your crontab, which will discard the output, meaning you won't get mail from cron. a1) Is it possible to only get a mail with critical information, where and what do I need to do to achieve this? My comments above should help you reduce the amount of junk mail you get from cron. b1) Where will information about ongoing disk-problems appear? How can I see that there is a flaky disk in a non-rebooted system? /var/log/messages. The system will complain quite noticably in the face of hardware errors, and should log one or more lines for every bad sector it runs into. On the other hand, depending on the hard drive to fail gradually is risky: hard drives can fail catestrophically without giving significant warning. Some failure modes-- stiction in particular-- can sometimes be worked around on a temporary basis long enough to recover data without heroic measures (ie, paying a data recovery company a few grand). It's important to realize that while RAID modes which provide fault-tolerance do improve availability (ie, they can save your data if a drive goes), RAID is not a substitute for backups. In particular, RAID-5 or RAID-1 doesn't help a bit if someone deletes or overwrites a file In addition to the questions above, is there something else I need to tune/install/setup/configurare to get a very reliable system that report critical data to me but where non-critical data is filtered out? /etc/syslog.conf defines the configuration of system logging, and it is worth reviewing that to understand what is being logged and where. Just to add my .02c here- all of the above is excellent advise, and I think someone else already mentioned logcheck, which can be useful for other thins as well (port scanning, Nimda attacks (which are STILL out there), and others)I guess my only disagreement here is about RAID or disk vs tape backups. At this point, it's generally more inexpensive to buy a secondary disk, or even a RAID setup. than to go with tape. Tape itself isn't a guaranteed medium by any means...meaning there have been times I've gone to backup from tape to then find out it was a less than full backup or data was corrupted on the tape. I won't even get into tape drive issues where you write data on one unit and a different tape drive won't read the same tapes, other than to say it happens. If you've got data that doesn't change often (ie like your pictures), and only grows, you've got a few options: 1. Set up a RAID-5 array. IMPORTANT- designate at least one hot spare! If your main use is for backup, you can go with an older SCSI solution, but if so, I _highly_ recommend using a RAID enclosure (with backplane, not a 'homemade' cabled setup- older SCSI (scsi2, UW, etc) is pretty picky about cable length, and using an external cabled enclosure can cause read or write errors and other issues (dropping a drive offline). Fiber channel or older scsi hardware RAID solutions can be had on eBay for pennies on the dollar right now.. Note that I'm talking about hardware RAID here...software RAID is slower (generally), and IMHO just not as reliable...yet. You can use the setup as a 'live data logical drive,' and if you're overly paranoid, do a scheduled tar archive (or other means of backup, but with tar you can add the incrementals to your archive) to the drives, or to an alternate drive, as well. 2. Buy a secondary IDE disk, sized at least 2x the size of your data to allow for growth. Do NOT use this drive for anything other than backup, eg mount the drive as /backup and only use it for (cronned) data dumps. You'll only be writing and reading to the drive occasionally, and as such, you should have a reasonably decent length of time the drive will work for, meaning it's likely to get replaced during upgrades years from now before the drive itself fails. Large IDE disks are getting insanely inexpensive... DLT drives in the 15gb(uncompressed) range are $100, but as usual, become progressively more expensive as you go larger in size, woth a 40gb running
Re: What logs etc do I need to checkfrequently?
Joachim Dagerot wrote: [ ... ] I realise you are right. The thing is that this is a home system and I have (had!) around 230 GB of data that was non-replicable. I am not aware of a deasent backup system that can handle that amount of data. There are three main alternatives: 1) hard drives -- get another HD or two and back up all data from your main filesystem daily via cron (perhaps locally, perhaps to a second machine); very cheap but iffy reliability...more of a hot spare solution than a true backup solution, but reasonably effective. 2) DVD burner -- cheap, limited media capacity but you can burn 20-odd DVD's (modulo achieved compression ratio) in a sane amount of time 3) tape-- main candidates are probably sDLT (up to 320GB per tape capacity) or maybe Ultrium (LTO?)...expensive (~ $3000+), best reliability by far, and best support for incremental backups to conserve media costs -- -Chuck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need to checkfrequently?
On 12/27/2003 at 7:40 PM Joachim Dagerot wrote: |As you with good memories know, I lost 3000 pictures of my first sons |first year this month. I did have a RAID-5 system with fresh disks, |however, shit happens and I have a feeling that this could have been |avoided if I read my log files better. | |So basically, | |a) I get a mail each time my a cron-event fires, this happens every 30 |min so the mailbox are quite loaded, not very funny going through. | |b) The file messages contained information that my raid-system was |about to blow, I didn't see this because I never read that file. Also |it appears to only log information after a reboot. |The vinum history file seems to only hold information about the vinum |setup, not the health of the volumes. | |Questions: | |a1) Is it possible to only get a mail with critical information, where |and what do I need to do to achieve this? /usr/ports/security/logcheck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: What logs etc do I need to checkfrequently?
Joachim Dagerot wrote: As you with good memories know, I lost 3000 pictures of my first sons first year this month. I did have a RAID-5 system with fresh disks, however, shit happens and I have a feeling that this could have been avoided if I read my log files better. I'm sorry that you lost data. While you may have been able to notice the problem with the RAID-5 array in time to do something, what you ought to do to avoid losing more data sometime in the future involves making good backups-- not poring over the system log files, not configuring RAID. So basically, a) I get a mail each time my a cron-event fires, this happens every 30 min so the mailbox are quite loaded, not very funny going through. If you can, change the cron task to not generate output unless there is a problem that you should know about. Failing that, append /dev/null 21 to the line in your crontab, which will discard the output, meaning you won't get mail from cron. a1) Is it possible to only get a mail with critical information, where and what do I need to do to achieve this? My comments above should help you reduce the amount of junk mail you get from cron. b1) Where will information about ongoing disk-problems appear? How can I see that there is a flaky disk in a non-rebooted system? /var/log/messages. The system will complain quite noticably in the face of hardware errors, and should log one or more lines for every bad sector it runs into. On the other hand, depending on the hard drive to fail gradually is risky: hard drives can fail catestrophically without giving significant warning. Some failure modes-- stiction in particular-- can sometimes be worked around on a temporary basis long enough to recover data without heroic measures (ie, paying a data recovery company a few grand). It's important to realize that while RAID modes which provide fault-tolerance do improve availability (ie, they can save your data if a drive goes), RAID is not a substitute for backups. In particular, RAID-5 or RAID-1 doesn't help a bit if someone deletes or overwrites a file In addition to the questions above, is there something else I need to tune/install/setup/configurare to get a very reliable system that report critical data to me but where non-critical data is filtered out? /etc/syslog.conf defines the configuration of system logging, and it is worth reviewing that to understand what is being logged and where. -- -Chuck ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
