Database/system Crashing

[Webber Valerie H]

Title: Database/system Crashing





I have a 8.1.7 repository database for Designer 9i (client) running on SUN Solaris 8. I can start the database up but when I access the data via Designer it crashes. I try to start it back up and I get the following message

ORA-09925: Unable to create audit trail file
SVR4 Error: 6: No such device or address
Additional information: 9925


This happens even after I have removed all of the .aud files from $ORACLE_HOME/rdbms/audit. 


Can anyone give me some clues as to what the problem might be? How do you determine whether it's a hardware problem or a database problem. I have checked everything I know to check as far as the database is concerned. Nothing major has changed on this system since or before this started. 

I think its a hardware problem (controller/buffer cache/data block ?) because after my database crashes the UNIX system doesn't work properly. For example, sometimes I can't even do a ps -ef or cat/more a file. The SA can't reboot the system without flipping the switch but still thinks its a database problem. Once the SA flips the switch (fsck runs fine) then I can start the database. The UNIX system has hung on one occasion with the database shutdown. It also happened once in the middle of a cold backup of the database.

Thanks in advance!
Val


Valerie H. Webber
Management Systems Designers, Inc
Database Administrator
[EMAIL PROTECTED]
704-566-5321 

Re: Database/system Crashing

[Suzy Vordos]

My mistake.  The parameter is audit_file_dest, which is the location
where audit files are written if audit_trail=os.  By default
audit_file_dest is $ORACLE_HOME/rdbms/audit.  I suspect audit_file_dest
is explicitly set to another location which either doesn't exist or
lacks proper permissions.

Suzy Vordos wrote:
> 
> Check your init.ora for the audit_trail parameter.  It's probably set to
> write a file to a location that either doesn't exist or lacks proper
> permissions.
> 
> > Webber Valerie H wrote:
> >
> > I have a 8.1.7 repository database for Designer 9i (client) running on
> > SUN Solaris 8. I can start the database up but when I access the data
> > via Designer it crashes. I try to start it back up and I get the
> > following message
> >
> > ORA-09925: Unable to create audit trail file
> > SVR4 Error: 6: No such device or address
> > Additional information: 9925
> >
> > This happens even after I have removed all of the .aud files from
> > $ORACLE_HOME/rdbms/audit.
> >
> > Can anyone give me some clues as to what the problem might be? How do
> > you determine whether it's a hardware problem or a database problem. I
> > have checked everything I know to check as far as the database is
> > concerned. Nothing major has changed on this system since or before
> > this started.
> >
> > I think its a hardware problem (controller/buffer cache/data block ?)
> > because after my database crashes the UNIX system doesn't work
> > properly. For example, sometimes I can't even do a ps -ef or cat/more
> > a file. The SA can't reboot the system without flipping the switch but
> > still thinks its a database problem. Once the SA flips the switch
> > (fsck runs fine) then I can start the database. The UNIX system has
> > hung on one occasion with the database shutdown. It also happened once
> > in the middle of a cold backup of the database.
> >
> > Thanks in advance!
> > Val
> >
> > Valerie H. Webber
> > Management Systems Designers, Inc
> > Database Administrator
> > [EMAIL PROTECTED]
> > 704-566-5321
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Suzy Vordos
  INET: [EMAIL PROTECTED]

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
San Diego, California-- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Re: Database/system Crashing

[Suzy Vordos]

Check your init.ora for the audit_trail parameter.  It's probably set to
write a file to a location that either doesn't exist or lacks proper
permissions.

> Webber Valerie H wrote:
> 
> I have a 8.1.7 repository database for Designer 9i (client) running on
> SUN Solaris 8. I can start the database up but when I access the data
> via Designer it crashes. I try to start it back up and I get the
> following message
> 
> ORA-09925: Unable to create audit trail file
> SVR4 Error: 6: No such device or address
> Additional information: 9925
> 
> This happens even after I have removed all of the .aud files from
> $ORACLE_HOME/rdbms/audit.
> 
> Can anyone give me some clues as to what the problem might be? How do
> you determine whether it's a hardware problem or a database problem. I
> have checked everything I know to check as far as the database is
> concerned. Nothing major has changed on this system since or before
> this started.
> 
> I think its a hardware problem (controller/buffer cache/data block ?)
> because after my database crashes the UNIX system doesn't work
> properly. For example, sometimes I can't even do a ps -ef or cat/more
> a file. The SA can't reboot the system without flipping the switch but
> still thinks its a database problem. Once the SA flips the switch
> (fsck runs fine) then I can start the database. The UNIX system has
> hung on one occasion with the database shutdown. It also happened once
> in the middle of a cold backup of the database.
> 
> Thanks in advance!
> Val
> 
> Valerie H. Webber
> Management Systems Designers, Inc
> Database Administrator
> [EMAIL PROTECTED]
> 704-566-5321
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Suzy Vordos
  INET: [EMAIL PROTECTED]

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
San Diego, California-- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Re: Database/system Crashing

[Reginald . W . Bailey]

Valerie:

What is the audit parameter in the init.ora file set to?  Is it set to
database (db) or os files? What are the other audit parameters in the
init.ora file set to?

RWB




Webber Valerie H <[EMAIL PROTECTED]>@fatcity.com on 01/02/2003
01:45:17 PM

Please respond to [EMAIL PROTECTED]



Sent by:  [EMAIL PROTECTED]


To:   Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]>
cc:




I have a 8.1.7 repository database for Designer 9i (client) running on SUN
Solaris 8. I can start the database up but when I access the data via
Designer it crashes. I try to start it back up and I get the following
message

ORA-09925: Unable to create audit trail file
SVR4 Error: 6: No such device or address
Additional information: 9925

This happens even after I have removed all of the .aud files from
$ORACLE_HOME/rdbms/audit.

Can anyone give me some clues as to what the problem might be? How do you
determine whether it's a hardware problem or a database problem. I have
checked everything I know to check as far as the database is concerned.
Nothing major has changed on this system since or before this started.

I think its a hardware problem (controller/buffer cache/data block ?)
because after my database crashes the UNIX system doesn't work properly.
For example, sometimes I can't even do a ps -ef or cat/more a file. The SA
can't reboot the system without flipping the switch but still thinks its a
database problem. Once the SA flips the switch (fsck runs fine) then I can
start the database. The UNIX system has hung on one occasion with the
database shutdown. It also happened once in the middle of a cold backup of
the database.

Thanks in advance!
Val

Valerie H. Webber
Management Systems Designers, Inc
Database Administrator
[EMAIL PROTECTED]
704-566-5321





-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: 
  INET: [EMAIL PROTECTED]

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
San Diego, California-- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

RE: Database/system Crashing

[Kurth, Michael J.]

Title: Database/system Crashing



Check 
the oracle executable and make sure the setuid bit is set.

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Thursday, January 02, 2003 
  1:45 PMTo: Multiple recipients of list ORACLE-LSubject: 
  Database/system Crashing
  I have a 8.1.7 repository database for 
  Designer 9i (client) running on SUN Solaris 8. I can start the database up but 
  when I access the data via Designer it crashes. I try to start it back up and 
  I get the following message
  ORA-09925: Unable to create audit trail 
  file SVR4 Error: 6: No such device 
  or address Additional information: 
  9925 
  This happens even after I have removed all 
  of the .aud files from $ORACLE_HOME/rdbms/audit. 
  Can anyone give me some clues as to what the 
  problem might be? How do you determine whether it's a hardware problem or a 
  database problem. I have checked everything I know to check as far as the 
  database is concerned. Nothing major has changed on this system since or 
  before this started. 
  I think its a hardware problem 
  (controller/buffer cache/data block ?) because after my database crashes the 
  UNIX system doesn't work properly. For example, sometimes I can't even do a ps 
  -ef or cat/more a file. The SA can't reboot the system without flipping the 
  switch but still thinks its a database problem. Once the SA flips the switch 
  (fsck runs fine) then I can start the database. The UNIX system has hung on 
  one occasion with the database shutdown. It also happened once in the middle 
  of a cold backup of the database.
  Thanks in advance! Val 
  Valerie H. WebberManagement Systems Designers, Inc Database 
  Administrator[EMAIL PROTECTED]704-566-5321 




Privileged/Confidential information may be contained in this message.  The information contained in this message is intended only for the use of the recipient(s) named above and their co-workers who are working on the same matter.


The recipient of this information is prohibited from disclosing the information to any other party unless this disclosure has been authorized in advance.


If you are not intended recipient of this message or any agent responsible for delivery of the message to the intended recipient, you are hereby notified that any disclosure, copying, distribution or action taken in reliance on the contents of this message is strictly prohibited.  You should immediately destroy this message and kindly notify the sender by reply E-Mail.


Please advise immediately if you or your employer does not consent to Internet E-Mail for messages of this kind.  Opinions, conclusions and other information in this message that do not relate to the official business of the firm shall be understood as neither given nor endorsed by it.

RE: Database/system Crashing

[Fink, Dan]

Title: Database/system Crashing



Valerie,
    What happens when the database first crashes? Is there an 
error in the alert log? 
    As for the second error, the issue is not that the device 
is full (you would get a different O/S error). It sounds more like either the 
audit_file_dest value is not set properly, $ORACLE_HOME is not defined correctly 
for that process, or that the init.ora used is not the one you think should be 
used.
 
    It is very important to find out what is causing the 
original crash. Focus on that and resolve it first.
 
Dan 
Fink

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Thursday, January 02, 2003 
  12:45 PMTo: Multiple recipients of list ORACLE-LSubject: 
  Database/system Crashing
  I have a 8.1.7 repository database for 
  Designer 9i (client) running on SUN Solaris 8. I can start the database up but 
  when I access the data via Designer it crashes. I try to start it back up and 
  I get the following message
  ORA-09925: Unable to create audit trail 
  file SVR4 Error: 6: No such device 
  or address Additional information: 
  9925 
  This happens even after I have removed all 
  of the .aud files from $ORACLE_HOME/rdbms/audit. 
  Can anyone give me some clues as to what the 
  problem might be? How do you determine whether it's a hardware problem or a 
  database problem. I have checked everything I know to check as far as the 
  database is concerned. Nothing major has changed on this system since or 
  before this started. 
  I think its a hardware problem 
  (controller/buffer cache/data block ?) because after my database crashes the 
  UNIX system doesn't work properly. For example, sometimes I can't even do a ps 
  -ef or cat/more a file. The SA can't reboot the system without flipping the 
  switch but still thinks its a database problem. Once the SA flips the switch 
  (fsck runs fine) then I can start the database. The UNIX system has hung on 
  one occasion with the database shutdown. It also happened once in the middle 
  of a cold backup of the database.
  Thanks in advance! Val 
  Valerie H. WebberManagement Systems Designers, Inc Database 
  Administrator[EMAIL PROTECTED]704-566-5321 

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing





Yes, you're correct and it can write the file to $ORACLE_HOME/rdbms/audit once the system is rebooted. Its just that when the database crashes, it can't write to that location until its rebooted.

Is it possible that I need to beef up my init.ora parameters? Can low system/database resources make the system crash like this and make the system lose track of UNIX commands and/or file directories? I might add that there are 2 disks on this box and the Oracle server software is located on the same disk as root but a different slice.

thanks for the reply
Val


-Original Message-
From: Suzy Vordos [mailto:[EMAIL PROTECTED]]
Sent: Thursday, January 02, 2003 4:59 PM
To: Multiple recipients of list ORACLE-L
Subject: Re: Database/system Crashing




My mistake.  The parameter is audit_file_dest, which is the location
where audit files are written if audit_trail=os.  By default
audit_file_dest is $ORACLE_HOME/rdbms/audit.  I suspect audit_file_dest
is explicitly set to another location which either doesn't exist or
lacks proper permissions.


Suzy Vordos wrote:
> 
> Check your init.ora for the audit_trail parameter.  It's probably set to
> write a file to a location that either doesn't exist or lacks proper
> permissions.
> 
> > Webber Valerie H wrote:
> >
> > I have a 8.1.7 repository database for Designer 9i (client) running on
> > SUN Solaris 8. I can start the database up but when I access the data
> > via Designer it crashes. I try to start it back up and I get the
> > following message
> >
> > ORA-09925: Unable to create audit trail file
> > SVR4 Error: 6: No such device or address
> > Additional information: 9925
> >
> > This happens even after I have removed all of the .aud files from
> > $ORACLE_HOME/rdbms/audit.
> >
> > Can anyone give me some clues as to what the problem might be? How do
> > you determine whether it's a hardware problem or a database problem. I
> > have checked everything I know to check as far as the database is
> > concerned. Nothing major has changed on this system since or before
> > this started.
> >
> > I think its a hardware problem (controller/buffer cache/data block ?)
> > because after my database crashes the UNIX system doesn't work
> > properly. For example, sometimes I can't even do a ps -ef or cat/more
> > a file. The SA can't reboot the system without flipping the switch but
> > still thinks its a database problem. Once the SA flips the switch
> > (fsck runs fine) then I can start the database. The UNIX system has
> > hung on one occasion with the database shutdown. It also happened once
> > in the middle of a cold backup of the database.
> >
> > Thanks in advance!
> > Val
> >
> > Valerie H. Webber
> > Management Systems Designers, Inc
> > Database Administrator
> > [EMAIL PROTECTED]
> > 704-566-5321
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Suzy Vordos
  INET: [EMAIL PROTECTED]


Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California    -- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Re: Database/system Crashing

[Suzy Vordos]

One thing to try is set audit_trail=none in init.ora, then bounce the
instance.  If the problem goes away then it's related to something gone
awry in system-wide auditing. 

Check for errors in the alert.log and for trace files in the *_dump_dest
directories.  In sqlplus do 'show paramter audit' to verify what
directory audit_file_dest uses, and do an 'ls -ld' on that directory to
verify ownership and permissions.

> Webber Valerie H wrote:
> 
> Yes, you're correct and it can write the file to
> $ORACLE_HOME/rdbms/audit once the system is rebooted. Its just that
> when the database crashes, it can't write to that location until its
> rebooted.
> 
> Is it possible that I need to beef up my init.ora parameters? Can low
> system/database resources make the system crash like this and make the
> system lose track of UNIX commands and/or file directories? I might
> add that there are 2 disks on this box and the Oracle server software
> is located on the same disk as root but a different slice.
> 
> thanks for the reply
> Val
> 
> -Original Message-
> From: Suzy Vordos [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, January 02, 2003 4:59 PM
> To: Multiple recipients of list ORACLE-L
> Subject: Re: Database/system Crashing
> 
> My mistake.  The parameter is audit_file_dest, which is the location
> where audit files are written if audit_trail=os.  By default
> audit_file_dest is $ORACLE_HOME/rdbms/audit.  I suspect
> audit_file_dest
> is explicitly set to another location which either doesn't exist or
> lacks proper permissions.
> 
> Suzy Vordos wrote:
> >
> > Check your init.ora for the audit_trail parameter.  It's probably
> set to
> > write a file to a location that either doesn't exist or lacks proper
> 
> > permissions.
> >
> > > Webber Valerie H wrote:
> > >
> > > I have a 8.1.7 repository database for Designer 9i (client)
> running on
> > > SUN Solaris 8. I can start the database up but when I access the
> data
> > > via Designer it crashes. I try to start it back up and I get the
> > > following message
> > >
> > > ORA-09925: Unable to create audit trail file
> > > SVR4 Error: 6: No such device or address
> > > Additional information: 9925
> > >
> > > This happens even after I have removed all of the .aud files from
> > > $ORACLE_HOME/rdbms/audit.
> > >
> > > Can anyone give me some clues as to what the problem might be? How
> do
> > > you determine whether it's a hardware problem or a database
> problem. I
> > > have checked everything I know to check as far as the database is
> > > concerned. Nothing major has changed on this system since or
> before
> > > this started.
> > >
> > > I think its a hardware problem (controller/buffer cache/data block
> ?)
> > > because after my database crashes the UNIX system doesn't work
> > > properly. For example, sometimes I can't even do a ps -ef or
> cat/more
> > > a file. The SA can't reboot the system without flipping the switch
> but
> > > still thinks its a database problem. Once the SA flips the switch
> > > (fsck runs fine) then I can start the database. The UNIX system
> has
> > > hung on one occasion with the database shutdown. It also happened
> once
> > > in the middle of a cold backup of the database.
> > >
> > > Thanks in advance!
> > > Val
> > >
> > > Valerie H. Webber
> > > Management Systems Designers, Inc
> > > Database Administrator
> > > [EMAIL PROTECTED]
> > > 704-566-5321
> --
> Please see the official ORACLE-L FAQ: http://www.orafaq.net
> --
> Author: Suzy Vordos
>   INET: [EMAIL PROTECTED]
> 
> Fat City Network Services-- 858-538-5051 http://www.fatcity.com
> San Diego, California-- Mailing list and web hosting services
> -
> To REMOVE yourself from this mailing list, send an E-Mail message
> to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
> the message BODY, include a line containing: UNSUB ORACLE-L
> (or the name of mailing list you want to be removed from).  You may
> also send the HELP command for other information (like subscribing).
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Suzy Vordos
  INET: [EMAIL PROTECTED]

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
San Diego, California-- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

RE: Database/system Crashing

[Stephen Lee]

I wonder if a file lock is being left in place when the instance crashes,
and the OS does not clear the lock until a reboot.  I would think the OS
should clear this without a reboot, but stranger things have been seen with
OS's ... even Unix.  This doesn't explain why the instance crashes.  I
wonder if fuser would show anything.  Are there any NFS mounts involved?

-Original Message-
Yes, you're correct and it can write the file to $ORACLE_HOME/rdbms/audit
once the system is rebooted. Its just that when the database crashes, it
can't write to that location until its rebooted.
Is it possible that I need to beef up my init.ora parameters?
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Stephen Lee
  INET: [EMAIL PROTECTED]

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
San Diego, California-- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing





Yes there are NFS mounts involved. What you said about the OS locks on the audit directory makes a lot of sense. My SA's are back to thinking it's a OS problem because it crashed again with the database shut down. 

The odd thing is that there is nothing written to the Oracle alert log file nor are there any entries in the trace files. But when the system is rebooted and I bring the db back up, Oracle knows it previously crashed and recovers itself. That's in the alert log file. Its like the system is losing its pointers or something. I suggested reinstalling the OS and Oracle then put my database back and see if that helps. Are there huge risks with this scenario?

Another odd thing that the SA's can't figure out is there are no entries in the message file nor can they get a dump file to determine why the system crashed. There is nothing. It crashed over the weekend with no activity and they got some sort of i-nodes error. 

Thanks for all your replies. Any ideas are helpful and I will relay them to our SA's...


Val


-Original Message-
From: Stephen Lee [mailto:[EMAIL PROTECTED]]
Sent: Friday, January 03, 2003 10:09 AM
To: Multiple recipients of list ORACLE-L
Subject: RE: Database/system Crashing




I wonder if a file lock is being left in place when the instance crashes,
and the OS does not clear the lock until a reboot.  I would think the OS
should clear this without a reboot, but stranger things have been seen with
OS's ... even Unix.  This doesn't explain why the instance crashes.  I
wonder if fuser would show anything.  Are there any NFS mounts involved?


-Original Message-
Yes, you're correct and it can write the file to $ORACLE_HOME/rdbms/audit
once the system is rebooted. Its just that when the database crashes, it
can't write to that location until its rebooted.
Is it possible that I need to beef up my init.ora parameters?
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Stephen Lee
  INET: [EMAIL PROTECTED]


Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California    -- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

RE: Database/system Crashing

[DENNIS WILLIAMS]

Val - I had a horrible problem using RMAN over NFS. Just hanging. Eventually
it turned out that NFS was configured for only a few connections (I can't
recall the exact term used). The assumption was that if we were only doing
one thing, then that should be adequate. It turned out this needed to be
increased. Hope that is some help.


Dennis Williams 
DBA, 40%OCP 
Lifetouch, Inc. 
[EMAIL PROTECTED]   

-Original Message-
Sent: Monday, January 06, 2003 10:40 AM
To: Multiple recipients of list ORACLE-L



Yes there are NFS mounts involved. What you said about the OS locks on the
audit directory makes a lot of sense. My SA's are back to thinking it's a OS
problem because it crashed again with the database shut down. 

The odd thing is that there is nothing written to the Oracle alert log file
nor are there any entries in the trace files. But when the system is
rebooted and I bring the db back up, Oracle knows it previously crashed and
recovers itself. That's in the alert log file. Its like the system is losing
its pointers or something. I suggested reinstalling the OS and Oracle then
put my database back and see if that helps. Are there huge risks with this
scenario?

Another odd thing that the SA's can't figure out is there are no entries in
the message file nor can they get a dump file to determine why the system
crashed. There is nothing. It crashed over the weekend with no activity and
they got some sort of i-nodes error. 

Thanks for all your replies. Any ideas are helpful and I will relay them to
our SA's... 

Val 

-Original Message- 
Sent: Friday, January 03, 2003 10:09 AM 
To: Multiple recipients of list ORACLE-L 



I wonder if a file lock is being left in place when the instance crashes, 
and the OS does not clear the lock until a reboot.  I would think the OS 
should clear this without a reboot, but stranger things have been seen with 
OS's ... even Unix.  This doesn't explain why the instance crashes.  I 
wonder if fuser would show anything.  Are there any NFS mounts involved? 

-Original Message- 
Yes, you're correct and it can write the file to $ORACLE_HOME/rdbms/audit 
once the system is rebooted. Its just that when the database crashes, it 
can't write to that location until its rebooted. 
Is it possible that I need to beef up my init.ora parameters? 
-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
  
-- 
Author: Stephen Lee 
  INET: [EMAIL PROTECTED] 

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
  
San Diego, California-- Mailing list and web hosting services 
- 
To REMOVE yourself from this mailing list, send an E-Mail message 
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in 
the message BODY, include a line containing: UNSUB ORACLE-L 
(or the name of mailing list you want to be removed from).  You may 
also send the HELP command for other information (like subscribing). 

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: DENNIS WILLIAMS
  INET: [EMAIL PROTECTED]

Fat City Network Services-- 858-538-5051 http://www.fatcity.com
San Diego, California-- Mailing list and web hosting services
-
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

RE: Database/system Crashing

[Stephen Lee]

Title: RE: Database/system Crashing



If the database 
has data files on NFS mounts, then any network problems or problems on the NFS 
server can crash things.  Disappearing NFS mounts can be very nasty.  
That's why it's a big no-no to put ANY files related to the database -- 
data files, log files, oracle binaries, etc. -- on NFS 
mounts.

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  10:40 AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Yes there are NFS mounts involved. What you said about the OS 
  locks on the audit directory makes a lot of sense. My SA's are back to 
  thinking it's a OS problem because it crashed again with the database shut 
  down. 
  The odd thing is that there is nothing written to the Oracle 
  alert log file nor are there any entries in the trace files. But when the 
  system is rebooted and I bring the db back up, Oracle knows it previously 
  crashed and recovers itself. That's in the alert log file. Its like the system 
  is losing its pointers or something. I suggested reinstalling the OS and 
  Oracle then put my database back and see if that helps. Are there huge risks 
  with this scenario?
  Another odd thing that the SA's can't figure out is there are 
  no entries in the message file nor can they get a dump file to determine why 
  the system crashed. There is nothing. It crashed over the weekend with no 
  activity and they got some sort of i-nodes error. 
  Thanks for all your replies. Any ideas are helpful and I will 
  relay them to our SA's... 
  Val 
  -Original Message- From: 
  Stephen Lee [mailto:[EMAIL PROTECTED]] Sent: Friday, January 03, 2003 10:09 AM To: 
  Multiple recipients of list ORACLE-L Subject: RE: 
  Database/system Crashing 
  I wonder if a file lock is being left in place when the 
  instance crashes, and the OS does not clear the lock 
  until a reboot.  I would think the OS should 
  clear this without a reboot, but stranger things have been seen with 
  OS's ... even Unix.  This doesn't explain why the 
  instance crashes.  I wonder if fuser would show 
  anything.  Are there any NFS mounts involved? 
  -Original Message- Yes, you're 
  correct and it can write the file to $ORACLE_HOME/rdbms/audit once the system is rebooted. Its just that when the database crashes, 
  it can't write to that location until its 
  rebooted. Is it possible that I need to beef up my 
  init.ora parameters? -- Please 
  see the official ORACLE-L FAQ: http://www.orafaq.net -- 
  Author: Stephen Lee   
  INET: [EMAIL PROTECTED] 
  Fat City Network Services    -- 858-538-5051 http://www.fatcity.com 
  San Diego, 
  California    -- Mailing list and web 
  hosting services - 
  To REMOVE yourself from this mailing list, send an E-Mail 
  message to: [EMAIL PROTECTED] (note EXACT spelling 
  of 'ListGuru') and in the message BODY, include a line 
  containing: UNSUB ORACLE-L (or the name of mailing 
  list you want to be removed from).  You may also 
  send the HELP command for other information (like subscribing). 

RE: Database/system Crashing

[Fink, Dan]

Title: RE: Database/system Crashing



Val,
    Not having an entry in the alert log 
or having trace files is not all that odd. This indicates a hard crash of the 
instance, where the background processes were unable to write to the files. This 
could be a result of the instance being forcefully terminated without using the 
Oracle shutdown process or it could be the result of the processes being unable 
to write to the device containing the log and trace files. Try moving the 
background_dump_dest to another device (preferably internally connected to the 
server).
    I would not reinstall the OS and 
Oracle unless it can be reasonably determined that the OS is causing the 
problem. What are the reasons the SAs say it is the OS? It is a lot of work to 
recreate the system and you have no guarantee that this will solve it. It sounds 
like a more detailed inspection of all the systems is in order instead of 
spinning the 'Wheel Of Blame' to stop on the 'most likely' suspect. More 
troubleshooting is called for, not the drastic step of "wipe it clean and start 
over"
 
Dan 
Fink

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  9:40 AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Yes there are NFS mounts involved. What you said about the OS 
  locks on the audit directory makes a lot of sense. My SA's are back to 
  thinking it's a OS problem because it crashed again with the database shut 
  down. 
  The odd thing is that there is nothing written to the Oracle 
  alert log file nor are there any entries in the trace files. But when the 
  system is rebooted and I bring the db back up, Oracle knows it previously 
  crashed and recovers itself. That's in the alert log file. Its like the system 
  is losing its pointers or something. I suggested reinstalling the OS and 
  Oracle then put my database back and see if that helps. Are there huge risks 
  with this scenario?
  Another odd thing that the SA's can't figure out is there are 
  no entries in the message file nor can they get a dump file to determine why 
  the system crashed. There is nothing. It crashed over the weekend with no 
  activity and they got some sort of i-nodes error. 
  Thanks for all your replies. Any ideas are helpful and I will 
  relay them to our SA's... 
  Val

RE: Database/system Crashing

[Burke, William F (Bill)]

Title: RE: Database/system Crashing



I'd 
agree with Dan.  You need to find the root cause of the crash.  If you 
rebuild to the current state from scratch, the odds are you'll see the same 
problem reoccur.  Secondly, while NFS mounted volumes will work, they 
should always be a last resort as any network, remote IO load on the server 
where the NFS mounted volume lives "could" cause IO corruption and panic the 
host server.  I didn't see the start of this thread so these are after the 
thought comments.  Maybe they're helpful.
 
Regards, 
Bill Burke "The 
Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 11:55 
  AMTo: Multiple recipients of list ORACLE-LSubject: RE: 
  Database/system Crashing
  Val,
      Not having an entry in the alert 
  log or having trace files is not all that odd. This indicates a hard crash of 
  the instance, where the background processes were unable to write to the 
  files. This could be a result of the instance being forcefully terminated 
  without using the Oracle shutdown process or it could be the result of the 
  processes being unable to write to the device containing the log and trace 
  files. Try moving the background_dump_dest to another device (preferably 
  internally connected to the server).
      I would not reinstall the OS and 
  Oracle unless it can be reasonably determined that the OS is causing the 
  problem. What are the reasons the SAs say it is the OS? It is a lot of work to 
  recreate the system and you have no guarantee that this will solve it. It 
  sounds like a more detailed inspection of all the systems is in order instead 
  of spinning the 'Wheel Of Blame' to stop on the 'most likely' suspect. More 
  troubleshooting is called for, not the drastic step of "wipe it clean and 
  start over"
   
  Dan 
  Fink
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
9:40 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Yes there are NFS mounts involved. What you said about the 
OS locks on the audit directory makes a lot of sense. My SA's are back to 
thinking it's a OS problem because it crashed again with the database shut 
down. 
The odd thing is that there is nothing written to the Oracle 
alert log file nor are there any entries in the trace files. But when the 
system is rebooted and I bring the db back up, Oracle knows it previously 
crashed and recovers itself. That's in the alert log file. Its like the 
system is losing its pointers or something. I suggested reinstalling the OS 
and Oracle then put my database back and see if that helps. Are there huge 
risks with this scenario?
Another odd thing that the SA's can't figure out is there 
are no entries in the message file nor can they get a dump file to determine 
why the system crashed. There is nothing. It crashed over the weekend with 
no activity and they got some sort of i-nodes error. 
Thanks for all your replies. Any ideas are helpful and I 
will relay them to our SA's... 
Val

RE: Database/system Crashing

[Richard Ji]

Title: RE: Database/system Crashing



Val,
 
Sorry 
I missed the previous messages.  Was this a Sun platform?  Did the 
system crash with a CPU panic
in 
/var/adm/messages?  We resovled a Sun NFS related crashes a couple of 
months ago.
 
Richard Ji

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  11:40 AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Yes there are NFS mounts involved. What you said about the OS 
  locks on the audit directory makes a lot of sense. My SA's are back to 
  thinking it's a OS problem because it crashed again with the database shut 
  down. 
  The odd thing is that there is nothing written to the Oracle 
  alert log file nor are there any entries in the trace files. But when the 
  system is rebooted and I bring the db back up, Oracle knows it previously 
  crashed and recovers itself. That's in the alert log file. Its like the system 
  is losing its pointers or something. I suggested reinstalling the OS and 
  Oracle then put my database back and see if that helps. Are there huge risks 
  with this scenario?
  Another odd thing that the SA's can't figure out is there are 
  no entries in the message file nor can they get a dump file to determine why 
  the system crashed. There is nothing. It crashed over the weekend with no 
  activity and they got some sort of i-nodes error. 
  Thanks for all your replies. Any ideas are helpful and I will 
  relay them to our SA's... 
  Val 
  -Original Message- From: 
  Stephen Lee [mailto:[EMAIL PROTECTED]] Sent: Friday, January 03, 2003 10:09 AM To: 
  Multiple recipients of list ORACLE-L Subject: RE: 
  Database/system Crashing 
  I wonder if a file lock is being left in place when the 
  instance crashes, and the OS does not clear the lock 
  until a reboot.  I would think the OS should 
  clear this without a reboot, but stranger things have been seen with 
  OS's ... even Unix.  This doesn't explain why the 
  instance crashes.  I wonder if fuser would show 
  anything.  Are there any NFS mounts involved? 
  -Original Message- Yes, you're 
  correct and it can write the file to $ORACLE_HOME/rdbms/audit once the system is rebooted. Its just that when the database crashes, 
  it can't write to that location until its 
  rebooted. Is it possible that I need to beef up my 
  init.ora parameters? -- Please 
  see the official ORACLE-L FAQ: http://www.orafaq.net -- 
  Author: Stephen Lee   
  INET: [EMAIL PROTECTED] 
  Fat City Network Services    -- 858-538-5051 http://www.fatcity.com 
  San Diego, 
  California    -- Mailing list and web 
  hosting services - 
  To REMOVE yourself from this mailing list, send an E-Mail 
  message to: [EMAIL PROTECTED] (note EXACT spelling 
  of 'ListGuru') and in the message BODY, include a line 
  containing: UNSUB ORACLE-L (or the name of mailing 
  list you want to be removed from).  You may also 
  send the HELP command for other information (like subscribing). 

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing



Richard, 
 
Yes this is a SUN platform. One SA now believes it is 
independent of the database since it happens when the database isn't running. I 
moved the background dest files to the other disk (other than the root disk.) 
Normally it would have crashed by now after starting the database and using 
Designer but so far so good.
 
There are no error messages in /var/adm/messages. There's 
no core dump file either. Its like the system gets corrupt before it can write 
to the /var/adm/messages file.
 
What were the symptoms of your Sun NFS related crashed and 
how did you diagnois the problem and what was the solution (if it were that 
simple)
 
Val

  -Original Message-From: Richard Ji 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  4:19 PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Val,
   
  Sorry I missed the previous messages.  Was this 
  a Sun platform?  Did the system crash with a CPU 
panic
  in 
  /var/adm/messages?  We resovled a Sun NFS related crashes a couple of 
  months ago.
   
  Richard Ji
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
11:40 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Yes there are NFS mounts involved. What you said about the 
OS locks on the audit directory makes a lot of sense. My SA's are back to 
thinking it's a OS problem because it crashed again with the database shut 
down. 
The odd thing is that there is nothing written to the Oracle 
alert log file nor are there any entries in the trace files. But when the 
system is rebooted and I bring the db back up, Oracle knows it previously 
crashed and recovers itself. That's in the alert log file. Its like the 
system is losing its pointers or something. I suggested reinstalling the OS 
and Oracle then put my database back and see if that helps. Are there huge 
risks with this scenario?
Another odd thing that the SA's can't figure out is there 
are no entries in the message file nor can they get a dump file to determine 
why the system crashed. There is nothing. It crashed over the weekend with 
no activity and they got some sort of i-nodes error. 
Thanks for all your replies. Any ideas are helpful and I 
will relay them to our SA's... 
Val 
-Original Message- From: 
Stephen Lee [mailto:[EMAIL PROTECTED]] Sent: Friday, January 03, 2003 10:09 AM To: 
Multiple recipients of list ORACLE-L Subject: RE: 
Database/system Crashing 
I wonder if a file lock is being left in place when the 
instance crashes, and the OS does not clear the lock 
until a reboot.  I would think the OS should 
clear this without a reboot, but stranger things have been seen with 
OS's ... even Unix.  This doesn't explain why the 
instance crashes.  I wonder if fuser would show 
anything.  Are there any NFS mounts involved? 
-Original Message- Yes, 
you're correct and it can write the file to $ORACLE_HOME/rdbms/audit 
once the system is rebooted. Its just that when the 
database crashes, it can't write to that location 
until its rebooted. Is it possible that I need to 
beef up my init.ora parameters? -- Please see the official ORACLE-L FAQ: http://www.orafaq.net -- Author: Stephen Lee   INET: [EMAIL PROTECTED] 
Fat City Network Services    -- 858-538-5051 
http://www.fatcity.com San Diego, California    -- 
Mailing list and web hosting services - 
To REMOVE yourself from this mailing list, send an E-Mail 
message to: [EMAIL PROTECTED] (note EXACT 
spelling of 'ListGuru') and in the message BODY, 
include a line containing: UNSUB ORACLE-L (or the 
name of mailing list you want to be removed from).  You may 
also send the HELP command for other information (like 
subscribing). 

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing



Well I relocated the background dest files and I got the 
following error... that was a great idea!
 
ORA-00206: error in writing (block 3, # blocks 1) of 
controlfileORA-00202: controlfile: 
'/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not found;  
product=RDBMS; facility=ORASVR4 Error: 5: I/O errorAdditional 
information: -1Additional information: 2048error 221 detected in 
background process
 
The SA's think its a data block corruption. If anyone has 
any additional information, it will be greatly appreciated.
At least now I know why the database crashed to begin with. 
Now the SA's just have to figure out how to fix it.
 
Thanks for all the help!!
Val

  -Original Message-From: Burke, William F (Bill) 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 2:49 
  PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
  Database/system Crashing
  I'd 
  agree with Dan.  You need to find the root cause of the crash.  If 
  you rebuild to the current state from scratch, the odds are you'll see the 
  same problem reoccur.  Secondly, while NFS mounted volumes will work, 
  they should always be a last resort as any network, remote IO load on the 
  server where the NFS mounted volume lives "could" cause IO corruption and 
  panic the host server.  I didn't see the start of this thread so these 
  are after the thought comments.  Maybe they're 
  helpful.
   
  Regards, 
  Bill Burke "The 
  Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 
  
-Original Message-From: Fink, Dan 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 11:55 
AMTo: Multiple recipients of list ORACLE-LSubject: RE: 
Database/system Crashing
Val,
    Not having an entry in the alert 
log or having trace files is not all that odd. This indicates a hard crash 
of the instance, where the background processes were unable to write to the 
files. This could be a result of the instance being forcefully terminated 
without using the Oracle shutdown process or it could be the result of the 
processes being unable to write to the device containing the log and trace 
files. Try moving the background_dump_dest to another device (preferably 
internally connected to the server).
    I would not reinstall the OS and 
Oracle unless it can be reasonably determined that the OS is causing the 
problem. What are the reasons the SAs say it is the OS? It is a lot of work 
to recreate the system and you have no guarantee that this will solve it. It 
sounds like a more detailed inspection of all the systems is in order 
instead of spinning the 'Wheel Of Blame' to stop on the 'most likely' 
suspect. More troubleshooting is called for, not the drastic step of "wipe 
it clean and start over"
 
Dan Fink

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  9:40 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Yes there are NFS mounts involved. What you said about the 
  OS locks on the audit directory makes a lot of sense. My SA's are back to 
  thinking it's a OS problem because it crashed again with the database shut 
  down. 
  The odd thing is that there is nothing written to the 
  Oracle alert log file nor are there any entries in the trace files. But 
  when the system is rebooted and I bring the db back up, Oracle knows it 
  previously crashed and recovers itself. That's in the alert log file. Its 
  like the system is losing its pointers or something. I suggested 
  reinstalling the OS and Oracle then put my database back and see if that 
  helps. Are there huge risks with this scenario?
  Another odd thing that the SA's can't figure out is there 
  are no entries in the message file nor can they get a dump file to 
  determine why the system crashed. There is nothing. It crashed over the 
  weekend with no activity and they got some sort of i-nodes error. 
  
  Thanks for all your replies. Any ideas are helpful and I 
  will relay them to our SA's... 
  Val

Re: Database/system Crashing

[Yechiel Adar]

Title: RE: Database/system Crashing



Got the same error on NT last week.
Check if there are system backups that backup the control 
files.
When Veritas backup the control file as a regular file the 
database can not write to it and you get this message.
 
Yechiel AdarMehish

  - Original Message - 
  From: 
  Webber Valerie H 
  To: Multiple recipients of list ORACLE-L 
  Sent: Tuesday, January 07, 2003 6:34 
  PM
  Subject: RE: Database/system 
  Crashing
  
  Well I relocated the background dest files 
  and I got the following error... that was a great idea!
   
  ORA-00206: error in writing (block 3, # 
  blocks 1) of controlfileORA-00202: controlfile: 
  '/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not found;  
  product=RDBMS; facility=ORASVR4 Error: 5: I/O errorAdditional 
  information: -1Additional information: 2048error 221 detected in 
  background process
   
  The SA's think its a data block corruption. 
  If anyone has any additional information, it will be greatly 
  appreciated.
  At least now I know why the database crashed 
  to begin with. Now the SA's just have to figure out how to fix 
  it.
   
  Thanks for all the help!!
  Val
  
-Original Message-From: Burke, William F (Bill) 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 2:49 
PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
Database/system Crashing
I'd agree with Dan.  You need to find the root cause of the 
crash.  If you rebuild to the current state from scratch, the odds are 
you'll see the same problem reoccur.  Secondly, while NFS mounted 
volumes will work, they should always be a last resort as any network, 
remote IO load on the server where the NFS mounted volume lives "could" 
cause IO corruption and panic the host server.  I didn't see the start 
of this thread so these are after the thought comments.  Maybe they're 
helpful.
 
Regards, 
Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 11:55 
  AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Val,
      Not having an entry in the 
  alert log or having trace files is not all that odd. This indicates a hard 
  crash of the instance, where the background processes were unable to write 
  to the files. This could be a result of the instance being forcefully 
  terminated without using the Oracle shutdown process or it could be the 
  result of the processes being unable to write to the device containing the 
  log and trace files. Try moving the background_dump_dest to another device 
  (preferably internally connected to the server).
      I would not reinstall the OS 
  and Oracle unless it can be reasonably determined that the OS is causing 
  the problem. What are the reasons the SAs say it is the OS? It is a lot of 
  work to recreate the system and you have no guarantee that this will solve 
  it. It sounds like a more detailed inspection of all the systems is in 
  order instead of spinning the 'Wheel Of Blame' to stop on the 'most 
  likely' suspect. More troubleshooting is called for, not the drastic step 
  of "wipe it clean and start over"
   
  Dan Fink
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
    2003 9:40 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Yes there are NFS mounts involved. What you said about 
the OS locks on the audit directory makes a lot of sense. My SA's are 
back to thinking it's a OS problem because it crashed again with the 
database shut down. 
The odd thing is that there is nothing written to the 
Oracle alert log file nor are there any entries in the trace files. But 
when the system is rebooted and I bring the db back up, Oracle knows it 
previously crashed and recovers itself. That's in the alert log file. 
Its like the system is losing its pointers or something. I suggested 
reinstalling the OS and Oracle then put my database back and see if that 
helps. Are there huge risks with this scenario?
Another odd thing that the SA's can't figure out is 
there are no entries in the message file nor can they get a dump file to 
determine why the system crashed. There is nothing. It crashed over the 
weekend with no activity and they got some sort of i-nodes error. 

Thanks for all your replies. Any ideas are helpful and I 
will relay them to our SA's... 
Val

RE: Database/system Crashing

[Fink, Dan]

Title: RE: Database/system Crashing



Val,
    Have you tried copying a known good controlfile in place of 
the bad one? If not, try it and report the result. If it corrupts as well, it 
seems to me that there is a much bigger problem. If it does not corrupt, then 
the question is, why didn't oracle report the corruption in the first place. I 
hate to say this, but I'm not certain you have found the problem, you may only 
be experiencing another symptom.
 
Dan

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
  9:34 AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Well I relocated the background dest files and I got the 
  following error... that was a great idea!
   
  ORA-00206: error in writing (block 3, # blocks 1) of 
  controlfileORA-00202: controlfile: 
  '/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not found;  
  product=RDBMS; facility=ORASVR4 Error: 5: I/O errorAdditional 
  information: -1Additional information: 2048error 221 detected in 
  background process
   
  The SA's think its a data block corruption. If anyone has 
  any additional information, it will be greatly 
appreciated.
  At least now I know why the database crashed to begin 
  with. Now the SA's just have to figure out how to fix it.
   
  Thanks for all the help!!
  Val
  
-Original Message-From: Burke, William F (Bill) 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 2:49 
PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
Database/system Crashing
I'd agree with Dan.  You need to find the root cause of the 
crash.  If you rebuild to the current state from scratch, the odds are 
you'll see the same problem reoccur.  Secondly, while NFS mounted 
volumes will work, they should always be a last resort as any network, 
remote IO load on the server where the NFS mounted volume lives "could" 
cause IO corruption and panic the host server.  I didn't see the start 
of this thread so these are after the thought comments.  Maybe they're 
helpful.
 
Regards, 
Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 11:55 
  AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Val,
      Not having an entry in the 
  alert log or having trace files is not all that odd. This indicates a hard 
  crash of the instance, where the background processes were unable to write 
  to the files. This could be a result of the instance being forcefully 
  terminated without using the Oracle shutdown process or it could be the 
  result of the processes being unable to write to the device containing the 
  log and trace files. Try moving the background_dump_dest to another device 
  (preferably internally connected to the server).
      I would not reinstall the OS 
  and Oracle unless it can be reasonably determined that the OS is causing 
  the problem. What are the reasons the SAs say it is the OS? It is a lot of 
  work to recreate the system and you have no guarantee that this will solve 
  it. It sounds like a more detailed inspection of all the systems is in 
  order instead of spinning the 'Wheel Of Blame' to stop on the 'most 
  likely' suspect. More troubleshooting is called for, not the drastic step 
  of "wipe it clean and start over"
   
  Dan Fink
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
    2003 9:40 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Yes there are NFS mounts involved. What you said about 
the OS locks on the audit directory makes a lot of sense. My SA's are 
back to thinking it's a OS problem because it crashed again with the 
database shut down. 
The odd thing is that there is nothing written to the 
Oracle alert log file nor are there any entries in the trace files. But 
when the system is rebooted and I bring the db back up, Oracle knows it 
previously crashed and recovers itself. That's in the alert log file. 
Its like the system is losing its pointers or something. I suggested 
reinstalling the OS and Oracle then put my database back and see if that 
helps. Are there huge risks with this scenario?
Another odd thing that the SA's can't figure out is 
there are no entries in the message file nor can they get a dump file to 
determine why the system crashed. There is nothing. It crashed over the 
w

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing



Dan,
 
I meant to say that I found out why Oracle crashed. There 
is a bigger problem with the OS since it crashes when the db is down and it 
seems to lose parts of itself if that makes sense. After the OS "sorta crashes" 
or partially crashes, some Unix commands are invalid like CAT or MORE or even 
VI. The SAs are looking into it. I thought about re-creating the control file or 
replacing it with a good one but they like you think that is just a symptom of a 
bigger OS problem.
 
Val

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 12:16 
  PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
  Database/system Crashing
  Val,
      Have you tried copying a known good controlfile in place 
  of the bad one? If not, try it and report the result. If it corrupts as well, 
  it seems to me that there is a much bigger problem. If it does not corrupt, 
  then the question is, why didn't oracle report the corruption in the first 
  place. I hate to say this, but I'm not certain you have found the problem, you 
  may only be experiencing another symptom.
   
  Dan
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
9:34 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Well I relocated the background dest files and I got 
the following error... that was a great idea!
 
ORA-00206: error in writing (block 3, # blocks 1) of 
controlfileORA-00202: controlfile: 
'/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not 
found;  product=RDBMS; facility=ORASVR4 Error: 5: I/O 
errorAdditional information: -1Additional information: 2048error 
221 detected in background process
 
The SA's think its a data block corruption. If anyone 
has any additional information, it will be greatly 
appreciated.
At least now I know why the database crashed to begin 
with. Now the SA's just have to figure out how to fix 
it.
 
Thanks for all the help!!
Val

  -Original Message-From: Burke, William F (Bill) 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 2:49 
  PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  I'd agree with Dan.  You need to find the root cause of the 
  crash.  If you rebuild to the current state from scratch, the odds 
  are you'll see the same problem reoccur.  Secondly, while NFS mounted 
  volumes will work, they should always be a last resort as any network, 
  remote IO load on the server where the NFS mounted volume lives "could" 
  cause IO corruption and panic the host server.  I didn't see the 
  start of this thread so these are after the thought comments.  Maybe 
  they're helpful.
   
  Regards, 
  Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 
  
-Original Message-From: Fink, Dan 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 11:55 
    AMTo: Multiple recipients of list ORACLE-LSubject: 
RE: Database/system Crashing
Val,
    Not having an entry in the 
alert log or having trace files is not all that odd. This indicates a 
hard crash of the instance, where the background processes were unable 
to write to the files. This could be a result of the instance being 
forcefully terminated without using the Oracle shutdown process or it 
could be the result of the processes being unable to write to the device 
containing the log and trace files. Try moving the background_dump_dest 
to another device (preferably internally connected to the 
server).
    I would not reinstall the OS 
and Oracle unless it can be reasonably determined that the OS is causing 
the problem. What are the reasons the SAs say it is the OS? It is a lot 
of work to recreate the system and you have no guarantee that this will 
solve it. It sounds like a more detailed inspection of all the systems 
is in order instead of spinning the 'Wheel Of Blame' to stop on the 
'most likely' suspect. More troubleshooting is called for, not the 
drastic step of "wipe it clean and start over"
 
Dan Fink

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
      2003 9:40 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Yes there are NFS mounts involved. What you said about 
  the OS locks on the audit directory makes a lot of sense. My SA's are 
 

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing





"PS..  do we all get a virtual 
"pass" on a future audit for helping?  :)"
 
ABSOLUTELY!! 
;)

  -Original Message-From: Mercadante, Thomas F 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
  3:52 PMTo: '[EMAIL PROTECTED]'Cc: 
  '[EMAIL PROTECTED]'Subject: RE: Database/system 
  Crashing
  Val,
   
  if 
  the unix commands are disappearing, then it sounds like you are either losing 
  disk directories, or the paths that point at them.
   
  when 
  I first read your post last week, I had a sneaky feeling that this was an OS 
  problem and not an Oracle one.  but not having anything solid to offer 
  you, I just lurked until someone with better unix experience could 
  help.
   
  glad 
  you are "on your way" to figuring it out.
   
   
  PS..  do we all get a virtual "pass" on a future audit for 
  helping?  :)
  Tom Mercadante Oracle Certified Professional 
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
3:00 PMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Dan,
 
I meant to say that I found out why Oracle crashed. 
There is a bigger problem with the OS since it crashes when the db is down 
and it seems to lose parts of itself if that makes sense. After the OS 
"sorta crashes" or partially crashes, some Unix commands are invalid like 
CAT or MORE or even VI. The SAs are looking into it. I thought about 
re-creating the control file or replacing it with a good one but they like 
you think that is just a symptom of a bigger OS problem.
 
Val

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 12:16 
  PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Val,
      Have you tried copying a known good controlfile in 
  place of the bad one? If not, try it and report the result. If it corrupts 
  as well, it seems to me that there is a much bigger problem. If it does 
  not corrupt, then the question is, why didn't oracle report the corruption 
  in the first place. I hate to say this, but I'm not certain you have found 
  the problem, you may only be experiencing another 
  symptom.
   
  Dan
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 
    2003 9:34 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Well I relocated the background dest files and I 
got the following error... that was a great idea!
 
ORA-00206: error in writing (block 3, # blocks 1) 
of controlfileORA-00202: controlfile: 
'/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not 
found;  product=RDBMS; facility=ORASVR4 Error: 5: I/O 
errorAdditional information: -1Additional information: 
2048error 221 detected in background process
 
The SA's think its a data block corruption. If 
anyone has any additional information, it will be greatly 
appreciated.
At least now I know why the database crashed to 
begin with. Now the SA's just have to figure out how to fix 
it.
 
Thanks for all the help!!
Val

  -Original Message-From: Burke, William F 
  (Bill) [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
  2003 2:49 PMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  I'd agree with Dan.  You need to find the root cause of 
  the crash.  If you rebuild to the current state from scratch, the 
  odds are you'll see the same problem reoccur.  Secondly, while 
  NFS mounted volumes will work, they should always be a last resort as 
  any network, remote IO load on the server where the NFS mounted volume 
  lives "could" cause IO corruption and panic the host server.  I 
  didn't see the start of this thread so these are after the thought 
  comments.  Maybe they're helpful.
   
  Regards, 
  Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 
  
-Original Message-From: Fink, Dan 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
11:55 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Val,
    Not having an entry in 
the alert log or having trace files is not all that odd. This 
indicates a hard cras

RE: Database/system Crashing

[Mercadante, Thomas F]

Title: RE: Database/system Crashing



Val,
 
if the 
unix commands are disappearing, then it sounds like you are either losing disk 
directories, or the paths that point at them.
 
when I 
first read your post last week, I had a sneaky feeling that this was an OS 
problem and not an Oracle one.  but not having anything solid to offer you, 
I just lurked until someone with better unix experience could 
help.
 
glad 
you are "on your way" to figuring it out.
 
 
PS..  do we all get a virtual "pass" on a future audit for 
helping?  :)
Tom Mercadante Oracle Certified Professional 

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
  3:00 PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Dan,
   
  I meant to say that I found out why Oracle crashed. There 
  is a bigger problem with the OS since it crashes when the db is down and it 
  seems to lose parts of itself if that makes sense. After the OS "sorta 
  crashes" or partially crashes, some Unix commands are invalid like CAT or MORE 
  or even VI. The SAs are looking into it. I thought about re-creating the 
  control file or replacing it with a good one but they like you think that is 
  just a symptom of a bigger OS problem.
   
  Val
  
-Original Message-From: Fink, Dan 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 12:16 
PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
Database/system Crashing
Val,
    Have you tried copying a known good controlfile in 
place of the bad one? If not, try it and report the result. If it corrupts 
as well, it seems to me that there is a much bigger problem. If it does not 
corrupt, then the question is, why didn't oracle report the corruption in 
the first place. I hate to say this, but I'm not certain you have found the 
problem, you may only be experiencing another symptom.
 
Dan

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 
  2003 9:34 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Well I relocated the background dest files and I got 
  the following error... that was a great idea!
   
  ORA-00206: error in writing (block 3, # blocks 1) of 
  controlfileORA-00202: controlfile: 
  '/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not 
  found;  product=RDBMS; facility=ORASVR4 Error: 5: I/O 
  errorAdditional information: -1Additional information: 
  2048error 221 detected in background process
   
  The SA's think its a data block corruption. If anyone 
  has any additional information, it will be greatly 
  appreciated.
  At least now I know why the database crashed to begin 
  with. Now the SA's just have to figure out how to fix 
  it.
   
  Thanks for all the help!!
  Val
  
-Original Message-From: Burke, William F 
(Bill) [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
2003 2:49 PMTo: Multiple recipients of list 
    ORACLE-LSubject: RE: Database/system 
Crashing
I'd agree with Dan.  You need to find the root cause of the 
crash.  If you rebuild to the current state from scratch, the odds 
are you'll see the same problem reoccur.  Secondly, while NFS 
mounted volumes will work, they should always be a last resort as any 
network, remote IO load on the server where the NFS mounted volume lives 
"could" cause IO corruption and panic the host server.  I didn't 
see the start of this thread so these are after the thought 
comments.  Maybe they're helpful.
 
Regards, 
Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  11:55 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Val,
      Not having an entry in the 
  alert log or having trace files is not all that odd. This indicates a 
  hard crash of the instance, where the background processes were unable 
  to write to the files. This could be a result of the instance being 
  forcefully terminated without using the Oracle shutdown process or it 
  could be the result of the processes being unable to write to the 
  device containing the log and trace files. Try moving the 
  background_dump_dest to another device (preferably internally 
  connected to the server).
      I would not reinstall the 
  OS and Oracle unless it can be reasona

RE: Database/system Crashing

[Burke, William F (Bill)]

Title: RE: Database/system Crashing



I'm 
late on this thread...  Is this a SAN which is mirrored by any 
chance?
 
Regards, 
Bill Burke "The 
Kinder and Gentler DBA" IOUG University 
Master Class Faculty 2001-2002 "iDBA 
Management, High Performance Infrastructure and HA" IOUG Board of Directors 2000-2002 ODTUG Board of Directors 1996-2000 www.OracleGuru.com www.KBMotorsports.biz 

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
  2:00 PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Dan,
   
  I meant to say that I found out why Oracle crashed. There 
  is a bigger problem with the OS since it crashes when the db is down and it 
  seems to lose parts of itself if that makes sense. After the OS "sorta 
  crashes" or partially crashes, some Unix commands are invalid like CAT or MORE 
  or even VI. The SAs are looking into it. I thought about re-creating the 
  control file or replacing it with a good one but they like you think that is 
  just a symptom of a bigger OS problem.
   
  Val
  
-Original Message-From: Fink, Dan 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 12:16 
PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
Database/system Crashing
Val,
    Have you tried copying a known good controlfile in 
place of the bad one? If not, try it and report the result. If it corrupts 
as well, it seems to me that there is a much bigger problem. If it does not 
corrupt, then the question is, why didn't oracle report the corruption in 
the first place. I hate to say this, but I'm not certain you have found the 
problem, you may only be experiencing another symptom.
 
Dan

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 
  2003 9:34 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Well I relocated the background dest files and I got 
  the following error... that was a great idea!
   
  ORA-00206: error in writing (block 3, # blocks 1) of 
  controlfileORA-00202: controlfile: 
  '/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not 
  found;  product=RDBMS; facility=ORASVR4 Error: 5: I/O 
  errorAdditional information: -1Additional information: 
  2048error 221 detected in background process
   
  The SA's think its a data block corruption. If anyone 
  has any additional information, it will be greatly 
  appreciated.
  At least now I know why the database crashed to begin 
  with. Now the SA's just have to figure out how to fix 
  it.
   
  Thanks for all the help!!
  Val
  
-Original Message-From: Burke, William F 
(Bill) [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
2003 2:49 PMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
I'd agree with Dan.  You need to find the root cause of the 
crash.  If you rebuild to the current state from scratch, the odds 
are you'll see the same problem reoccur.  Secondly, while NFS 
mounted volumes will work, they should always be a last resort as any 
network, remote IO load on the server where the NFS mounted volume lives 
"could" cause IO corruption and panic the host server.  I didn't 
see the start of this thread so these are after the thought 
comments.  Maybe they're helpful.
 
Regards, 
Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  11:55 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Val,
      Not having an entry in the 
  alert log or having trace files is not all that odd. This indicates a 
  hard crash of the instance, where the background processes were unable 
  to write to the files. This could be a result of the instance being 
  forcefully terminated without using the Oracle shutdown process or it 
  could be the result of the processes being unable to write to the 
  device containing the log and trace files. Try moving the 
  background_dump_dest to another device (preferably internally 
  connected to the server).
      I would not reinstall the 
  OS and Oracle unless it can be reasonably determined that the OS is 
  causing the problem. What are the reasons the SAs say it is the OS? It 
  is a lot of work to recreate the system and you have no guarantee th

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing



No sorry... just a simple 2 disk development Unix box... 
Sun Solaris 8

  -Original Message-From: Burke, William F (Bill) 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 5:09 
  PMTo: Multiple recipients of list ORACLE-LSubject: RE: 
  Database/system Crashing
  I'm 
  late on this thread...  Is this a SAN which is mirrored by any 
  chance?
   
  Regards, 
  Bill Burke "The 
  Kinder and Gentler DBA" IOUG University 
  Master Class Faculty 2001-2002 "iDBA 
  Management, High Performance Infrastructure and HA" IOUG Board of Directors 2000-2002 ODTUG Board of Directors 1996-2000 www.OracleGuru.com www.KBMotorsports.biz 
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 
2:00 PMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Dan,
 
I meant to say that I found out why Oracle crashed. 
There is a bigger problem with the OS since it crashes when the db is down 
and it seems to lose parts of itself if that makes sense. After the OS 
"sorta crashes" or partially crashes, some Unix commands are invalid like 
CAT or MORE or even VI. The SAs are looking into it. I thought about 
re-creating the control file or replacing it with a good one but they like 
you think that is just a symptom of a bigger OS problem.
 
Val

  -Original Message-From: Fink, Dan 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 2003 12:16 
  PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Val,
      Have you tried copying a known good controlfile in 
  place of the bad one? If not, try it and report the result. If it corrupts 
  as well, it seems to me that there is a much bigger problem. If it does 
  not corrupt, then the question is, why didn't oracle report the corruption 
  in the first place. I hate to say this, but I'm not certain you have found 
  the problem, you may only be experiencing another 
  symptom.
   
  Dan
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 07, 
2003 9:34 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Well I relocated the background dest files and I 
got the following error... that was a great idea!
 
ORA-00206: error in writing (block 3, # blocks 1) 
of controlfileORA-00202: controlfile: 
'/u04/oradata/ERCS/ora_control2'ORA-27063: Message 27063 not 
found;  product=RDBMS; facility=ORASVR4 Error: 5: I/O 
errorAdditional information: -1Additional information: 
2048error 221 detected in background process
 
The SA's think its a data block corruption. If 
anyone has any additional information, it will be greatly 
appreciated.
At least now I know why the database crashed to 
begin with. Now the SA's just have to figure out how to fix 
it.
 
Thanks for all the help!!
Val

  -Original Message-From: Burke, William F 
  (Bill) [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
  2003 2:49 PMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  I'd agree with Dan.  You need to find the root cause of 
  the crash.  If you rebuild to the current state from scratch, the 
  odds are you'll see the same problem reoccur.  Secondly, while 
  NFS mounted volumes will work, they should always be a last resort as 
  any network, remote IO load on the server where the NFS mounted volume 
  lives "could" cause IO corruption and panic the host server.  I 
  didn't see the start of this thread so these are after the thought 
  comments.  Maybe they're helpful.
   
  Regards, 
  Bill Burke "The Kinder and Gentler DBA" www.OracleGuru.com www.KBMotorsports.biz 
  
-Original Message-From: Fink, Dan 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
    11:55 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Val,
    Not having an entry in 
the alert log or having trace files is not all that odd. This 
indicates a hard crash of the instance, where the background 
processes were unable to write to the files. This could be a result 
of the instance being forcefully terminated without using the Oracle 
shutdown process or it could be the result of the processes being 

RE: Database/system Crashing

[Richard Ji]

Title: RE: Database/system Crashing



Sorry 
for getting back to you late on this.  In our case, the server crashes each 
time with CPU panic message
in 
/var/adm/messages.  We used adb to analyze the core dump and saw in each 
instance of the crash, NFS
lead 
to it.  It was trying to free memory twice which caused the panic.  
And talking to Sun confirmed that there
was a 
bug in NFS.  If you are interested in it I can send you the patch 
number.

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  5:26 PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Richard, 
   
  Yes this is a SUN platform. One SA now 
  believes it is independent of the database since it happens when the database 
  isn't running. I moved the background dest files to the other disk (other than 
  the root disk.) Normally it would have crashed by now after starting the 
  database and using Designer but so far so good.
   
  There are no error messages in 
  /var/adm/messages. There's no core dump file either. Its like the system gets 
  corrupt before it can write to the /var/adm/messages file.
   
  What were the symptoms of your Sun NFS 
  related crashed and how did you diagnois the problem and what was the solution 
  (if it were that simple)
   
  Val
  
-Original Message-From: Richard Ji 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
2003 4:19 PMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Val,
 
Sorry I missed the previous messages.  Was 
this a Sun platform?  Did the system crash with a CPU 
panic
in 
/var/adm/messages?  We resovled a Sun NFS related crashes a couple of 
months ago.
 
Richard Ji

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  11:40 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Yes there are NFS mounts involved. What you said about the 
  OS locks on the audit directory makes a lot of sense. My SA's are back to 
  thinking it's a OS problem because it crashed again with the database shut 
  down. 
  The odd thing is that there is nothing written to the 
  Oracle alert log file nor are there any entries in the trace files. But 
  when the system is rebooted and I bring the db back up, Oracle knows it 
  previously crashed and recovers itself. That's in the alert log file. Its 
  like the system is losing its pointers or something. I suggested 
  reinstalling the OS and Oracle then put my database back and see if that 
  helps. Are there huge risks with this scenario?
  Another odd thing that the SA's can't figure out is there 
  are no entries in the message file nor can they get a dump file to 
  determine why the system crashed. There is nothing. It crashed over the 
  weekend with no activity and they got some sort of i-nodes error. 
  
  Thanks for all your replies. Any ideas are helpful and I 
  will relay them to our SA's... 
  Val 
  -Original Message- From: 
  Stephen Lee [mailto:[EMAIL PROTECTED]] Sent: Friday, January 03, 2003 10:09 AM To: 
  Multiple recipients of list ORACLE-L Subject: RE: 
  Database/system Crashing 
  I wonder if a file lock is being left in place when the 
  instance crashes, and the OS does not clear the 
  lock until a reboot.  I would think the OS should clear this without a reboot, but stranger things have been 
  seen with OS's ... even Unix.  This doesn't 
  explain why the instance crashes.  I wonder 
  if fuser would show anything.  Are there any NFS mounts 
  involved? 
  -Original Message- Yes, 
  you're correct and it can write the file to 
  $ORACLE_HOME/rdbms/audit once the system is 
  rebooted. Its just that when the database crashes, it can't write to that location until its rebooted. Is it possible that I need to beef up my init.ora 
  parameters? -- Please see 
  the official ORACLE-L FAQ: http://www.orafaq.net -- 
  Author: Stephen Lee   
  INET: [EMAIL PROTECTED] 
  Fat City Network Services    -- 
  858-538-5051 http://www.fatcity.com San 
  Diego, California    -- Mailing 
  list and web hosting services - 
  To REMOVE yourself from this mailing list, send an E-Mail 
  message to: [EMAIL PROTECTED] (note EXACT 
  spelling of 'ListGuru') and in the message BODY, 
  include a line containing: UNSUB ORACLE-L (or the 
  name of mailing list you want to be removed from).  You may 
  also send the HELP command for other information (like 
  subscribing). 

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing



Please do. Any information at this point will help. The 
only difference is that when it fails it doesn't even get to write the 
/var/adm/messages or the core dump. Oracle wasn't writing to the alert log file 
until I moved it off the root disk an onto another disk. That has helped but its 
definitely an OS problem.
 
Is it possible to re-locate the messages and/or core dump 
file for UNIX to another disk? If we did, maybe it could write to it like Oracle 
was able to. I don't know if that's a possibility on the Unix system or does it 
have to reside on the root disk?
 
Thanks in advance!Val

  -Original Message-From: Richard Ji 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 14, 2003 
  12:39 AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Sorry for getting back to you late on this.  In 
  our case, the server crashes each time with CPU panic 
  message
  in 
  /var/adm/messages.  We used adb to analyze the core dump and saw in each 
  instance of the crash, NFS
  lead 
  to it.  It was trying to free memory twice which caused the panic.  
  And talking to Sun confirmed that there
  was 
  a bug in NFS.  If you are interested in it I can send you the patch 
  number.
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
5:26 PMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Richard, 
 
Yes this is a SUN platform. One SA now believes it is 
independent of the database since it happens when the database isn't 
running. I moved the background dest files to the other disk (other than the 
root disk.) Normally it would have crashed by now after starting the 
database and using Designer but so far so good.
 
There are no error messages in /var/adm/messages. 
There's no core dump file either. Its like the system gets corrupt before it 
can write to the /var/adm/messages file.
 
What were the symptoms of your Sun NFS related crashed 
and how did you diagnois the problem and what was the solution (if it were 
that simple)
 
Val

  -Original Message-From: Richard Ji 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
  2003 4:19 PMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Val,
   
  Sorry I missed the previous messages.  Was 
  this a Sun platform?  Did the system crash with a CPU 
  panic
  in /var/adm/messages?  We resovled a Sun NFS 
  related crashes a couple of months ago.
   
  Richard Ji
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
2003 11:40 AMTo: Multiple recipients of list 
    ORACLE-LSubject: RE: Database/system 
Crashing
Yes there are NFS mounts involved. What you said about 
the OS locks on the audit directory makes a lot of sense. My SA's are 
back to thinking it's a OS problem because it crashed again with the 
database shut down. 
The odd thing is that there is nothing written to the 
Oracle alert log file nor are there any entries in the trace files. But 
when the system is rebooted and I bring the db back up, Oracle knows it 
previously crashed and recovers itself. That's in the alert log file. 
Its like the system is losing its pointers or something. I suggested 
reinstalling the OS and Oracle then put my database back and see if that 
helps. Are there huge risks with this scenario?
Another odd thing that the SA's can't figure out is 
there are no entries in the message file nor can they get a dump file to 
determine why the system crashed. There is nothing. It crashed over the 
weekend with no activity and they got some sort of i-nodes error. 

Thanks for all your replies. Any ideas are helpful and I 
will relay them to our SA's... 
Val 
-Original Message- From: 
Stephen Lee [mailto:[EMAIL PROTECTED]] 
Sent: Friday, January 03, 2003 10:09 AM 
To: Multiple recipients of list ORACLE-L 
Subject: RE: Database/system Crashing 

I wonder if a file lock is being left in place when the 
instance crashes, and the OS does not clear the 
lock until a reboot.  I would think the OS should clear this without a reboot, but stranger things have been 
seen with OS's ... even Unix.  This doesn't 
explain why the instance crashes.  I wonder 
if fuser would show anything.  Are there any NFS mounts 
involved? 
-Original Message- Yes, 
you're correct and it can write the file to 
$

RE: Database/system Crashing

[Richard Ji]

Title: RE: Database/system Crashing



Val,
 
Here is the Sun patch 
108727-16 that was related to our problem.  
Does the system have savecore running?
You 
can try to symbolic link /var/adm to another disk.  In our case Oracle 
never wrote anything
to 
alert log because it was a system crash.  Sounds like your system is having 
IO related issues.  Is it internal disks?
or 
SAN?

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 14, 2003 
  11:59 AMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Please do. Any information at this point will 
  help. The only difference is that when it fails it doesn't even get to write 
  the /var/adm/messages or the core dump. Oracle wasn't writing to the alert log 
  file until I moved it off the root disk an onto another disk. That has helped 
  but its definitely an OS problem.
   
  Is it possible to re-locate the messages 
  and/or core dump file for UNIX to another disk? If we did, maybe it could 
  write to it like Oracle was able to. I don't know if that's a possibility on 
  the Unix system or does it have to reside on the root 
disk?
   
  Thanks in advance!Val
  
-Original Message-From: Richard Ji 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 14, 
2003 12:39 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Sorry for getting back to you late on this.  
In our case, the server crashes each time with CPU panic 
message
in 
/var/adm/messages.  We used adb to analyze the core dump and saw in 
each instance of the crash, NFS
lead to it.  It was trying to free memory 
twice which caused the panic.  And talking to Sun confirmed that 
there
was a bug in NFS.  If you are interested in it 
I can send you the patch number.

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 2003 
  5:26 PMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Richard, 
   
  Yes this is a SUN platform. One SA now 
  believes it is independent of the database since it happens when the 
  database isn't running. I moved the background dest files to the other 
  disk (other than the root disk.) Normally it would have crashed by now 
  after starting the database and using Designer but so far so 
  good.
   
  There are no error messages in 
  /var/adm/messages. There's no core dump file either. Its like the system 
  gets corrupt before it can write to the /var/adm/messages 
  file.
   
  What were the symptoms of your Sun NFS 
  related crashed and how did you diagnois the problem and what was the 
  solution (if it were that simple)
   
  Val
  
-Original Message-From: Richard Ji 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
2003 4:19 PMTo: Multiple recipients of list 
    ORACLE-LSubject: RE: Database/system 
Crashing
Val,
 
Sorry I missed the previous messages.  Was 
this a Sun platform?  Did the system crash with a CPU 
panic
in /var/adm/messages?  We resovled a Sun 
NFS related crashes a couple of months ago.
 
Richard Ji

  -Original Message-From: Webber Valerie H 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
  2003 11:40 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Yes there are NFS mounts involved. What you said about 
  the OS locks on the audit directory makes a lot of sense. My SA's are 
  back to thinking it's a OS problem because it crashed again with the 
  database shut down. 
  The odd thing is that there is nothing written to the 
  Oracle alert log file nor are there any entries in the trace files. 
  But when the system is rebooted and I bring the db back up, Oracle 
  knows it previously crashed and recovers itself. That's in the alert 
  log file. Its like the system is losing its pointers or something. I 
  suggested reinstalling the OS and Oracle then put my database back and 
  see if that helps. Are there huge risks with this scenario?
  Another odd thing that the SA's can't figure out is 
  there are no entries in the message file nor can they get a dump file 
  to determine why the system crashed. There is nothing. It crashed over 
  the weekend with no activity and they got some sort of i-nodes error. 
  
  Thanks for all your replies. Any ideas are helpful and 
  I will relay them to our SA's... 
  Val 
  -Original Message- From: St

RE: Database/system Crashing

[Webber Valerie H]

Title: RE: Database/system Crashing



I'm not sure about savecore but I do know that the disks 
are internal. No SAN... They're about to yank SUN's chain because they are 
trying to resolve this remotely and its taking too long. Hopefully we'll have 
something solid from SUN next week. I am going to check out this SUN patch. 
Thanks for sending!
 

  -Original Message-From: Richard Ji 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 14, 2003 
  1:10 PMTo: Multiple recipients of list ORACLE-LSubject: 
  RE: Database/system Crashing
  Val,
   
  Here is the Sun patch 
  108727-16 that was related to our 
  problem.  Does the system have savecore 
  running?
  You 
  can try to symbolic link /var/adm to another disk.  In our case Oracle 
  never wrote anything
  to 
  alert log because it was a system crash.  Sounds like your system is 
  having IO related issues.  Is it internal disks?
  or 
  SAN?
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 14, 2003 
11:59 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Please do. Any information at this point will help. The 
only difference is that when it fails it doesn't even get to write the 
/var/adm/messages or the core dump. Oracle wasn't writing to the alert log 
file until I moved it off the root disk an onto another disk. That has 
helped but its definitely an OS problem.
 
Is it possible to re-locate the messages and/or core 
dump file for UNIX to another disk? If we did, maybe it could write to it 
like Oracle was able to. I don't know if that's a possibility on the Unix 
system or does it have to reside on the root disk?
 
Thanks in advance!Val

  -Original Message-From: Richard Ji 
  [mailto:[EMAIL PROTECTED]]Sent: Tuesday, January 14, 
  2003 12:39 AMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Sorry for getting back to you late on this.  
  In our case, the server crashes each time with CPU panic 
  message
  in /var/adm/messages.  We used adb to 
  analyze the core dump and saw in each instance of the crash, 
  NFS
  lead to it.  It was trying to free memory 
  twice which caused the panic.  And talking to Sun confirmed that 
  there
  was a bug in NFS.  If you are interested in 
  it I can send you the patch number.
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 06, 
2003 5:26 PMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Richard, 
 
Yes this is a SUN platform. One SA now believes it 
is independent of the database since it happens when the database isn't 
running. I moved the background dest files to the other disk (other than 
the root disk.) Normally it would have crashed by now after starting the 
database and using Designer but so far so good.
 
There are no error messages in /var/adm/messages. 
There's no core dump file either. Its like the system gets corrupt 
before it can write to the /var/adm/messages file.
 
What were the symptoms of your Sun NFS related 
crashed and how did you diagnois the problem and what was the solution 
(if it were that simple)
 
Val

  -Original Message-From: Richard Ji 
  [mailto:[EMAIL PROTECTED]]Sent: Monday, January 
  06, 2003 4:19 PMTo: Multiple recipients of list 
  ORACLE-LSubject: RE: Database/system 
  Crashing
  Val,
   
  Sorry I missed the previous messages.  
  Was this a Sun platform?  Did the system crash with a CPU 
  panic
  in /var/adm/messages?  We resovled a Sun 
  NFS related crashes a couple of months ago.
   
  Richard Ji
  
-Original Message-From: Webber Valerie H 
[mailto:[EMAIL PROTECTED]]Sent: Monday, January 
06, 2003 11:40 AMTo: Multiple recipients of list 
ORACLE-LSubject: RE: Database/system 
Crashing
Yes there are NFS mounts involved. What you said 
about the OS locks on the audit directory makes a lot of sense. My 
SA's are back to thinking it's a OS problem because it crashed again 
with the database shut down. 
The odd thing is that there is nothing written to 
the Oracle alert log file nor are there any entries in the trace 
files. But when the system is rebooted and I bring the db back up, 
Oracle knows it previously crashed and recovers itself. That's in