Ron, I wanted to suggest looking at any actions that trigger at 6 am. Escalations come to mind because I noticed a line in your log:
Fri Feb 10 06:30:05 2012 BRIE : Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation (OUR_AR_SERVER) ARERR - 94 I see this repeated. I am thinking there maybe be something in the background running a bad query. Just a thought. ________________________________ From: "Peters, Ron" <rpet...@columbia.com> To: arslist@ARSLIST.ORG Sent: Tuesday, February 14, 2012 3:12:25 PM Subject: Re: AR Server or DB issue ** Hmm, that KB article is interesting. It sure does sound plausible. If the server is stuck in a loop, resetting its brain seems like the right solution. Since the server is unresponsive during our issues, what should I look for on the DB to see what is causing the issue? I’m working with the DBA on this and since it happened last week mon/wed/fri and again yesterday, I can reasonably assume it will happen again tomorrow morning. I’d like to set up some type of process to catch this in the act and I’d like to gather all the appropriate data. Thougths? Thanks. From:Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jonas Stumph Stevnsvig Sent: Monday, February 13, 2012 2:47 PM To: arslist@ARSLIST.ORG Subject: Re: AR Server or DB issue ** Hi Ron I noticed the arAPPnote - indicating an approval notice. I've recently hotfxed our servers which were presenting us with ARERR 93 (server busy) or the same error you are getting (ARERR 92). Our servers would hang upon handling some approval actions. The symtom turned out to be unbelievable numbers of notifications being triggered - by a single Approval on a change. We got 130000 entries in NTE:SYS-NT Proces Control for a single change. hotfix was SW00369891 Knowledgebase article is KA 354923 - just a hunch - but check your NTE:SYS-NT Proces Control form (or select from the Database if your server is unresponsive as ours was). br Jonas Den 10-02-2012 16:34, Peters, Ron skrev: ** Hello all, Having some AR server issues. Something has been happening in the mornings about 6am in that our service desk can’t access Remedy. This started in Decemeber but only once. Then twice in January and now a couple times this week. The first few times it resolved itself after about 30 minutes but this week we had to restart the AR service on the server to get it back. The following is the arerror log from this morning. I’ve contacted our DBA’s to see if they notice anything but since an AR restart resolves it, I’m not convinced it’s a DB issue. Any thoughts? Thanks. Fri Feb 10 06:25:30 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:25:30 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:25:31 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:25:31 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:27:32 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:27:32 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:27:32 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:27:32 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:29:32 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:29:32 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:29:32 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:29:32 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:30:05 2012 BRIE : Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation (OUR_AR_SERVER) ARERR - 94 Fri Feb 10 06:31:32 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:31:32 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:31:32 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:31:32 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:33:32 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:33:32 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:33:32 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:33:32 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:35:02 2012 CMDB Dispatcher : Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation (OUR_AR_SERVER) ARERR - 94 Fri Feb 10 06:35:05 2012 BRIE : Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation (OUR_AR_SERVER) ARERR - 94 Fri Feb 10 06:35:32 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:35:32 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:35:32 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:35:32 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:37:32 2012 AssignEng : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:37:32 2012 AssignEng : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) Fri Feb 10 06:37:32 2012 SLMCS : Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (OUR_AR_SERVER) ARERR - 92 Fri Feb 10 06:37:32 2012 SLMCS : AR System Application server terminated -- fatal error encountered (ARAPPNOTE 4501) _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"