Our DBA mentioned that: DB can accommodate up to 750 oracle processes per DB instances.
So, how does this number relates to the Server Ports and Queues we have configured. Any recommendations? Even the last time, issue was encountered in this specific form only. This form has many table fields ~15 and overall fields count ~400. (Basically, this is a heavily customized form but not as much as our HPD form) On Thursday, June 28, 2012 11:35:09 AM UTC-7, Raj wrote: > > Sorry! Screenshot didn't work. > Here is the information again: > Type/RPC/Min/Max > Admin / 390600 / 1/ 1 > Fast / 390620 / 20 / 50 > List / 390635 / 20 / 50 > Private / 390623 / 1 / 1 > Private / 390624 / 1 / 1 > Private / 390680 / 1 / 1 > Private / 390694 / 1 / 1 > > On Thursday, June 28, 2012 11:29:00 AM UTC-7, Raj wrote: >> >> ** Thanks Fred for your prompt reply. >> I checked the Server Ports and Queues settings: >> >> reaching out to DBA to check "how many connections and processes the >> database (and database O/S for the user the database runs under) is >> configured for." as you suggested. >> Will let you know my findings. >> >> On Thursday, June 28, 2012 11:02:13 AM UTC-7, Grooms, Frederick W wrote: >>> >>> ** >>> >>> Let me make sure I have the facts from your earlier email in the chain … >>> >>> >>> Server : 5.01.02 Patch 1313 >>> Hardware : sun4u >>> OS: SunOS 5.10 >>> DB: SQL -- Oracle >>> DB Version : 10.2.0.5.0 - 64bit >>> >>> And since it is ARS version 5 you are using the Oracle 8 client. >>> >>> Since the DBA said processes reached the threshold you should ask the >>> DBA how many connections and processes the database (and database O/S for >>> the user the database runs under) is configured for. >>> >>> If the number of connections and processes is less than the max number >>> of threads you have configured (Fast, List, Escalation, Admin, Private, …) >>> you will need to increase the number on the database or reduce the max >>> number on the ARS server. >>> >>> >>> >>> If I remember correctly ARS always dropped the view and created a new >>> one when database field names were changed or added (the table uses the >>> field IDs and there is a database view that has the names). Remedy >>> doesn’t do anything with the view itself, it is created just to make >>> outside querying of the data easier. >>> >>> >>> >>> Fred >>> >>> >>> >>> >>> >>> -----Original Message----- >>> *From:* Action Request System discussion list(ARSList) [mailto: >>> arslist@ARSLIST.ORG] *On Behalf Of *Axton >>> *Sent:* Thursday, June 28, 2012 12:35 PM >>> *To:* arslist@ARSLIST.ORG >>> *Subject:* Re: Can remedy code push cause DB crash, etc? >>> >>> >>> >>> ** From https://forums.oracle.com/forums/thread.jspa?threadID=364644 >>> >>> >>> >>> Oracle finally admit there was a bug: BUG 5607984 - ORACLE DOES NOT >>> CLOSE TCP CONNECTIONS. REMAINS IN CLOSE_WAIT STATE. [On Windows 32-bit]. >>> >>> >>> >>> The patch 10 (patch number 5639232) is supposed to solve the problem for >>> 10.2.0.2.0. We applied it monday morning and everything is fine up to now. >>> >>> >>> >>> This bug is also supposed to be solved in the 10.2.0.3.0 patchset that >>> is availlable on the Metalink site. >>> >>> >>> Do you happen to be running Oracle on a Windows 32-bit server? >>> >>> -----Original Message----- >>> On Thu, Jun 28, 2012 at 11:55 AM, Raj <ravi6...@gmail.com> wrote: >>> >>> ** Thank everyone for their help. >>> >>> Last night we experienced the same issue again. >>> Added a new field to the form and modified DB name of the 4 fields and >>> hit save. >>> System froze and users started getting ARERR 94. >>> When checked arerror.log, noticed the following entries: >>> >>> Wed Jun 27 21:54:13 2012 390635 : Failure while trying to connect to >>> the SQL database. >>> >>> >>> Please ensure the SQL database is running or contact the Database >>> Administrator for help (ARERR 550) >>> >>> Wed Jun 27 21:54:13 2012 ORA-12535: TNS:operation timed out >>> Wed Jun 27 21:54:13 2012 390635 : Cannot initialize contact with SQL >>> database (ARERR 551) >>> Wed Jun 27 21:54:13 2012 Thread 63 not handling connection >>> >>> DBA mentioned that the processes on DB instances reached the threshold. >>> The server/DB became unresponsive. When we looked in the DB, the view >>> (for the form I was trying to modify) was missing in Production. >>> So, looks like the Save operation on the form never completed. We had to >>> bounce the DB instances and AR Server. >>> After that We had to manually copy the view DDL from our QA environment >>> and modify it for Prod and run it to enable the view to compile and other >>> related views to compile. >>> >>> Could anyone throw some light what might have happened and how we can >>> avoid this in future to avoid any impact to customers. >>> >>> >>> Here's the environment info: >>> >>> >>> Server : 5.01.02 Patch 1313 >>> Hardware : sun4u >>> OS: SunOS 5.10 >>> DB: SQL -- Oracle >>> DB Version : 0.2.0.5.0 - 64bi >>> >>> Thank you, Raj >>> >>> >>> -----Original Message----- >>> On Friday, March 30, 2012 1:14:40 PM UTC-7, Joe Martin D'Souza wrote: >>> >>> Yes in my previous email I was talking about the /tmp directory on the >>> OS >>> and files created there too.. The files created there are done by AR >>> process >>> that are a result of an operation from a client.. in your case the >>> client >>> was a import operation from the DS (the new toy for us adults from >>> Remedyville). >>> >>> I do no think in this case its your TEMP database which we thought it >>> may be >>> earlier. Your DBA's information directs us to the OS. Running out of >>> volume >>> space on the where the /tmp is equally critical. >>> >>> Check this directory to see if there are a lot of files owned by the AR >>> System OS user after you shut down the AR System process.. It might be >>> an >>> indication of an application level problem.. Without shutting down the >>> processes, you will be unable to delete the files that are owned by the >>> AR >>> System OS user.. These files will be locked if created after the process >>> was >>> last started.. >>> >>> Joe >>> >>> -----Original Message----- >>> From: Raj >>> Sent: Friday, March 30, 2012 4:04 PM Newsgroups: >>> public.remedy.arsystem.general >>> To: arslist@ARSLIST.ORG >>> Subject: Re: Can remedy code push cause DB crash, etc? >>> >>> Thanks Joe: >>> >>> Here some additional details I gathered from our DBA: >>> It was DataWarehouse that had issues and it was not complaining about >>> TEMP >>> tablespace but the errors in the alert log were pertaining the OS >>> resource >>> problem >>> >>> Wed Mar 28 08:22:30 UTC 2012 >>> Process startup failed, error stack: >>> Wed Mar 28 08:22:31 UTC 2012 >>> ORA-27300: OS system dependent operation:fork failed with status: 11 >>> ORA-27301: OS failure message: Resource temporarily unavailable >>> ORA-27302: failure occurred at: skgpspawn3 >>> Wed Mar 28 08:22:32 UTC 2012 >>> Process q000 died, see its trace file >>> Wed Mar 28 08:22:32 UTC 2012 >>> .. >>> >>> Wed Mar 28 08:49:31 UTC 2012 >>> Process startup failed, error stack: >>> Wed Mar 28 08:49:31 UTC 2012 >>> ORA-27300: OS system dependent operation:fork failed with status: 12 >>> ORA-27301: OS failure message: Not enough space >>> ORA-27302: failure occurred at: ...... >>> Wed Mar 28 08:49:31 UTC 2012 >>> Process J001 died, see its trace file >>> >>> So, what are the recommended size, settings? whom should I approach >>> at my end to get these settings in place? >>> >>> -----Original Message----- >>> On Mar 30, 12:50 pm, Joe Martin D'Souza <jdso...@shyle.net> wrote: >>> > The /tmp directory is needed by the AR Server process to process some >>> of >>> > its >>> > internal stuff.. running out of space here is not good too.. >>> > >>> > Every client connection that is opened up from a client, creates a >>> unique >>> > temp file in the /tmp directory and this directory can grow pretty >>> large >>> > as >>> > well.. The size of that file is directly directly dependent on the >>> > operation >>> > performed.. The operation you were performing is an expensive AR System >>> > operation so I can see that this file could grow pretty large. After >>> the >>> > client connection is closed the file is typically purged.. >>> > >>> > If a file for whatever reason does not get purged after the operation, >>> its >>> > usually an indication of some sort of an application problem.. The >>> file is >>> > locked so cannot be deleted until you terminate the AR System server >>> > process >>> > in case that happens.. Also when that happens, you can get this >>> directory >>> > fill up with files that are large, and not deleted. Monitoring this >>> > directory once in a while may be a good idea.. >>> > >>> > In short you can't be having a low volume dedicated to the tmp >>> directory >>> > on >>> > your AR Server.. >>> > >>> > Joe >>> > >>> > -----Original Message----- >>> > From: Raj >>> > Sent: Friday, March 30, 2012 3:30 PM Newsgroups: >>> > >>> > public.remedy.arsystem.general >>> > To: arsl...@arslist.org >>> > Subject: Re: Can remedy code push cause DB crash, etc? >>> > >>> > Thanks for all your replies, I discussed this with our DBA. Here' the >>> > response when I asked to check the following(1. What is the size of >>> the DB >>> > Temp Space? >>> > 2. Is Temp Database set to autoextend and If there is sufficient free >>> disk >>> > space ?) >>> > Response: >>> > The problem we have was not the TEMP tablespace it was the /tmp on the >>> > unix >>> > server got 100% full. Also it was not production server that had >>> problem >>> > it >>> > was the Datawarehouse server and in the alert log of datawarehouse >>> server, >>> > it did point to TEMP tablespace problem but of resource unable and not >>> > enough space on the unix system >>> > >>> > Any comments? >>> > >>> > On Mar 30, 12:02 pm, Joe Martin D'Souza <jdso...@shyle.net> wrote: >>> > > Raj, >>> > >>> > > Turn autoextend on for your temp database, and make sure the disk has >>> > > sufficient free disk space to allow similar operations in the >>> future. >>> > > I'm >>> > > guessing you must have been missing one or both of these? >>> > >>> > > Joe >>> > >>> > > -----Original Message----- >>> > > From: Raj >>> > > Sent: Friday, March 30, 2012 2:52 PM Newsgroups: >>> > >>> > > public.remedy.arsystem.general >>> > > To: arsl...@arslist.org >>> > > Subject: Re: Can remedy code push cause DB crash, etc? >>> > >>> > > Yup exactly, we noticed the same thing. DB ran out of temp space and >>> > > ultimately crashed. >>> > > What are your recommendations, how can we avoid this in future? >>> > >>> > > On Mar 30, 11:47 am, "Shellman, David" <dave.shell...@te.com> wrote: >>> > > > Been there. Done that. In my case, we ran up against not enough >>> temp >>> > > > space and froze the system. >>> > >>> > > > Dave >>> > >>> > > > -----Original Message----- >>> > > > From: Action Request System discussion list(ARSList) >>> > > > [mailto:arsl...@arslist.org] On Behalf Of Raj >>> > > > Sent: Friday, March 30, 2012 2:36 PM >>> > > > To: arsl...@arslist.org >>> > > > Subject: Re: Can remedy code push cause DB crash, etc? >>> > >>> > > > Typo ; >>> > > > DB Version : 10.2.0.5.0 - 64bi >>> > >>> > > > -----Original Message----- >>> > > > On Mar 30, 11:33 am, Raj <ravi6...@gmail.com> wrote: >>> > > > > Sorry, missed the environment info: >>> > >>> > > > > Server : 5.01.02 Patch 1313 >>> > > > > Hardware : sun4u >>> > > > > OS: SunOS 5.10 >>> > > > > DB: SQL -- Oracle >>> > > > > DB Version : 0.2.0.5.0 - 64bi >>> > >>> > > > > On Mar 30, 11:30 am, Raj <ravi6...@gmail.com> wrote: >>> > >>> > > > > > Hi All, >>> > >>> > > > > > Very recently I performed Remedy code push on our production >>> > > > > > server. >>> > >>> > > > > > This involved importing some new forms and related workflow >>> and >>> > > > > > also >>> > > > > > modifying existing forms like HelpDesk and Change forms.(Using >>> > > > > > ARAdmin >>> > > > > > tool) >>> > >>> > > > > > I tested this same on our Dev and QA Servers and the code push >>> was >>> > > > > > a >>> > > > > > success without any issues. >>> > >>> > > > > > But when I performed this on our Production server, it caused >>> DB >>> > > > > > crash and brought down Remedy. >>> > >>> > > > > > The outage was caused by the fact that the database became >>> > > > > > unresponsive. >>> > >>> > > > > > After we brought back Remedy, we noticed some of the DB view >>> > > > > > needed >>> > > > > > to be recompiled. >>> > >>> > > > > > As we are still investigating if it has to do with issues in >>> non >>> > > > > > remedy components but we couldn't find any solid clues and >>> > > > > > ultimately brings it back to Remedy code push. >>> > >>> > > > > > Here are the logs : >>> > >>> > > > > > ------------------------------ >>> > > > > > Wed Mar 28 01:09:59 2012 Distrib : AR System Distributed Server >>> > > > > > terminated when a signal was received by the server (ARDSNOTE >>> > > > > > 3000) >>> > > > > > Wed Mar 28 01:09:59 2012 15 Wed Mar 28 01:09:59 2012 Distrib : >>> AR >>> > > > > > System Distributed Server terminated when a signal was >>> received by >>> > > > > > the server (ARDSNOTE 3000) Wed Mar 28 01:09:59 2012 15 Wed Mar >>> 28 >>> > > > > > 01:11:23 2012 390600 : Requested database table not found. >>> > > > > > Please check the spelling (table name is case-sensitive) >>> (ARERR >>> > > > > > 481) >>> > > > > > Wed Mar 28 01:29:29 2012 390620 : Failure while trying to >>> connect >>> > > > > > to >>> > > > > > the SQL database. >>> > > > > > Please ensure the SQL database is running or contact the >>> Database >>> > > > > > Administrator for help (ARERR 550) Wed Mar 28 01:29:29 2012 >>> > > > > > ORA-12535: TNS:operation timed out Wed Mar 28 01:29:29 2012 >>> 390620 >>> > > > > > : >>> > > > > > Cannot initialize contact with SQL database (ARERR 551) Wed >>> Mar 28 >>> > > > > > 01:29:29 2012 Thread 67 not handling connection >>> > >>> > > > > > ------------------------------- >>> > >>> > > > > > At this point, I would like to understand from experts in here >>> to >>> > > > > > see if anything like the above mentioned is possible? If yes, >>> what >>> > > > > > could be the reasons and what can I look into to find the root >>> > > > > > cause. >>> > >>> > > > > > Thanks, >>> > >>> > > > > > Raj >>> >>> >>> _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ >> >> _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
