Re: Recovering Linux TSM server from partial filesystem failure

2014-03-12 Thread Zoltan Forray
Thanks for the suggestion. As SOP, we update all firmware, every few months, so this box is up-to-date. The Dell diagnostics finally finished, CLEAN. No issues found. I have some of the FODC logs when the first error occurred (was moving them to another filespace as fast as I could when the

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Zoltan Forray
With the lack of replies, I am guessing I can't recover this server from what is left behind. I do have an old DB backups but for what this server does, it isn't worth bothering. I can rebuild it faster. I do have additional questions that somebody might have an answer to. 1. Any reason NOT

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Chavdar Cholev
Zoltan, you can check db2diag log fir more info, I would start from there. I am nor sure about TSM6.1 to TSM7.1, my main concern here is different DB2 versions. If you thing to rebuild TSM 6.1 on new HW by mounting LUNs from old crashed TSM it may work, and after that to upgrade to TSM 7.1 I

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Zoltan Forray
The db2diag.log file was lost along with the root and /home partition. All I have is ghost messages from the activity log (TSMManager console saves a lot of the messages in its buffers) On Tue, Mar 11, 2014 at 12:21 PM, Chavdar Cholev chavdar.cho...@gmail.comwrote: Zoltan, if not you can

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Thomas Denier
-Zoltan Forray wrote: - 2. When doing postmortem on this failed server (still waiting for results from hardware diagnostics - my OS guy is head to the offsite location to check on the results and to start reinstalling the OS), I notice this message from my monitoring system: 3/6/2014

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Skylar Thompson
Encapsulating the term in quotes (-980) ought to do the trick. Looks like -980 is associated with a disk error, which unfortunately doesn't help Zoltan too much at this point... http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/core/rsql0900.htm On Tue, Mar

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Zoltan Forray
Thanks for finding that but, as you said, it doesn't help much. The disk filled to 100% due to DB2 taking dumps but that doesn't tell me what caused the dumping in the first place. We are still running Dell full diagnostics (started yesterday afternoon and was at 78% as of 2pm EDT). My OS guy

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-11 Thread Skylar Thompson
Since you mentioned Dell, one thing to check would be PERC and hard drive firmware levels. There have been a number of updates to both over the past few years concerning silent data corruption under a variety of conditions. On Tue, Mar 11, 2014 at 03:07:24PM -0400, Zoltan Forray wrote: Thanks

Recovering Linux TSM server from partial filesystem failure

2014-03-10 Thread Zoltan Forray
We recently had our offsite/recover TSM server (RH Linux 6.4, TSM 6.3.4.200) go south. Something happened that caused DB2 to start crashing/dumping and subsequently completely filled the filesystem containing /home/tsminst1 directory. Since this was the root folder, the system tanked and is now

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-10 Thread Arbogast, Warren K
Zoltan, We are all eager to know if the something that happened had anything to do with TSM 6.3.2 or DB2. Since they seem to be fine and the OS needs to be rebuilt, presumably not. Sometimes i's and t's beg to dotted and crosed. Best wishes, Keith Arbogast Indiana University On Mar 10, 2014,

Re: Recovering Linux TSM server from partial filesystem failure

2014-03-10 Thread Zoltan Forray
As soon as I know more, I will post here. My OS guy (offsite with the box) just reported/confirmed the root filesystem is a loss and will have to rebuild/reinstall. He is running Dell hardware diagnostics right now. Going back through what logs/reports I have available, I found that there was