If I remember correctly, the threads are for concurrent database processes. If you have a max of 15 and 15 (for List and Fast), then at most, 15 Fast processes and 15 List processes can run simultaneously.
If you're reaching the max, and your system starts to crawl, I'd say you don't have enough threads open... Calls to the db would be queued for the next thread-type to come available (as determined by the dispatcher). (Kinda like how floating licenses work...if a license isn't available for a floating user, then they'll wait until one comes available). But going back to your thread log...it sure looks like a user is invoking some process that is causing a fault...(although the frequency between restart and stop doesn't seem like a user is pushing a button and waiting...it's happening too fast.) Your error logs do show *who* caused the problem. I'd watch that closely and when the dump occurs, contact them (if you can) to find out what they're doing. If you can replicate it...you can nail down the problem. J.T. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Wednesday, July 11, 2007 10:36 AM To: arslist@ARSLIST.ORG Subject: Re: ARS v7.0.1P2 LIST threads initiate inexplainably Maybe it's not available for use. There is nothing intrinsically wrong with hitting the max, it's only a problem if none of them are free to work. Have your looked to see if the threads appear busy to arserver (server stats and api logs)? Axton Grams On 7/11/07, Susan Palmer <[EMAIL PROTECTED]> wrote: > ** > It's almost like ar server creates the threads initially. Then as it > feels there's a need it creates more threads. But at some point it > cannot see or doesn't think an existing thread is available for use so > it keeps creating them until whaalaa ... you hit the max. > > Why can't the ar server see that the thread is there and available for > use ??? > > Susan > > > > > > > On 7/11/07, Susan Palmer <[EMAIL PROTECTED]> wrote: > > > > The stack errors come across as ARNOTE20 errors and they restart > themselves. They can be FAST or LIST, although mostly FAST. > > > > We are not getting arerror 93's. > > > > We don't have any plugins. We do have HD v5.0. So I am getting > > some > periodic rappsrv errors. > > > > It almost seems like when someone, obviously not everyone, logs in > > it can > start the thread-storm. I'm just grasping at straws because the time > correlation to the initial thread error doesn't seem quite close enough. > > > > We've had 3 storms already today. I feel like I'm in Toronado alley. > > > > I think we'll try and go to Patch 3 tonight. > > > > If that doesn't work better I need to figure out a plan to move back > > to > some other version because this is a killer. How does bmc expect to > keep customers if we can't keep the server up! It will take years to > win back confidence in the product with my customers. > > > > Susan > > > > > > > > > > On 7/11/07, Tony Worthington < [EMAIL PROTECTED]> wrote: > > > > > Susan, Axton, Ben, etc... > > > > > > We have the same issues. Our server comes and goes as it pleases, > > > and we've yet to track down exactly what the cause is. We do know > > > some of > our > > > problems stem from plugins, most of the time it's ardbcquery. The > easiest > > > way to crash a thread is to open either the Overview or Problem console. > > > Other times, the server just gives up, we get ARERR93's and end up > having > > > to restart the service. > > > > > > I think things have gotten worse after patch002 and 003, but > > > things have never been stable. > > > > > > Here's an excerpt from a High priority ticket we have open -- with > > > no solutions so far: > > > > > > Faulting application arcmdbd.exe, version 2.0.1.3, faulting module > > > arcmdbd.exe, version 2.0.1.3, fault address 0x0000a981. > > > > > > Mon Jul 02 12:45:05 2007 390695 : AR System server terminated > > > when a signal/exception was received by the server (ARNOTE 20) > > > Mon Jul 02 12:45:05 2007 0xc0000005 > > > Mon Jul 02 12:45:05 2007 390695 : AR System server terminated -- > > > fatal error encountered (ARNOTE 21) Mon Jul 02 12:46:12 2007 > > > 390695 : AR System server terminated when a signal/exception was > > > received by the server (ARNOTE 20) > > > Mon Jul 02 12:46:12 2007 0xc0000005 > > > > > > We're running with lots of threads, but are on a 8 processor (dual > > > core > > > 3ghz) box with 18gb of ram. :-) Remedy is still slow. Our DB > > > flies, > and > > > network is fine. > > > > > > Private-RPC-Socket: 390601 2 8 > > > Private-RPC-Socket: 390620 4 8 > > > Private-RPC-Socket: 390621 2 8 > > > Private-RPC-Socket: 390626 4 12 > > > Private-RPC-Socket: 390627 2 8 > > > Private-RPC-Socket: 390628 12 12 > > > Private-RPC-Socket: 390629 2 8 > > > Private-RPC-Socket: 390635 4 8 > > > Private-RPC-Socket: 390690 1 6 > > > > > > > > > Windows 2003R2 EE > > > 7.0.0.1 patch 003 > > > Remote Oracle 10gR2 RAC/Linux > > > ITSM7p004 > > > > > > -tony > > > > > > -- > > > Tony Worthington > > > [EMAIL PROTECTED] > > > 262-703-5911 > > > > > > > > > > > > Ben Cantatore < [EMAIL PROTECTED]> Sent by: "Action Request > > > System discussion list(ARSList)" > > > <arslist@ARSLIST.ORG> > > > 07/11/2007 12:02 AM > > > Please respond to > > > arslist@ARSLIST.ORG > > > > > > > > > To > > > arslist@ARSLIST.ORG > > > cc > > > > > > Subject > > > Re: ARS v7.0.1P2 LIST threads initiate inexplainably > > > > > > > > > > > > > > > > > > > > > ** > > > Axton and Susan, you're not alone with the stack errors. I've > > > been plagued with them since my launch back in Feb. In my > > > situation the > server > > > runs about 2 weeks and crashes and usually recovers on its own. > > > One problem I had which I could reproduce crashing the server was > > > fixed by patch 3 so I think what Davies is saying applied to that. > > > The stack problems have gotten a little better, but still happens. > > > I have API, Escalation, Filter, SQL and Thread logging turned on > > > and am currently waiting for the next crash. So I'll post then > > > with any interesting information that surfaces from that. > > > > > > Susan, sounds like you're having a worse problem than I am. How > > > often > do > > > the crashes happen, does the arserver recover on its own? > > > > > > Currently on Server 7.0.1 patch 2 and ITSM 7.0.2 patch 3 for all > > > modules connecting to Oracle 10g db. > > > > > > Ben Cantatore > > > Remedy Administrator > > > Avon > > > (914) 935-2946 > > > > > > > > > "Davies, J.T." < [EMAIL PROTECTED]> Sent by: "Action > > > Request System discussion list(ARSList)" > > > < arslist@ARSLIST.ORG> > > > 07/10/2007 05:38 PM > > > > > > Please respond to > > > arslist@ARSLIST.ORG > > > > > > > > > To > > > arslist@ARSLIST.ORG > > > cc > > > > > > Subject > > > Re: ARS v7.0.1P2 LIST threads initiate inexplainably > > > > > > > > > > > > > > > > > > > > > > > > > > > ** > > > Hi Susan, > > > > > > I've experienced these errors before, too, a long time ago: Signal > > > 5 termination on a specific thread. > > > > > > I was able to trace it back to a workflow problem, and with your > > > jump > from > > > 5.1.2 to 7.0, I'd imagine you might be in the same boat. > > > > > > What I found was with the Run Process commands. > > > > > > To give an example: > > > > > > You're calling the Application-Delete-Entry command. It's > > > expecting two > > > inputs: Form and ID. > > > > > > The problem is akin to receiving a bad input value. Perhaps the > > > ID was missing, or the Form name was invalid...some wacky scenario > > > where the command raised an error, and thus, the users execution died. > > > > > > I also found the errors were occurring if a command was expecting > > > an integer value (say, the number of seconds to offset on a > > > Business Time command), but the input was some character string ("ABC"). > > > > > > You might see if your logs (Filter and SQL) can narrow this down > > > for > you. > > > You might have to mentally process some of these commands, because > > > I do remember that it wasn't very apparent in the logs that it was > > > failing on these. > > > > > > Hope this helps or provides a little more direction! > > > > > > J.T. > > > New Edge Networks > > > An Earthlink Company > > > > > > From: Action Request System discussion list(ARSList) > > > [mailto: [EMAIL PROTECTED] On Behalf Of Susan Palmer > > > Sent: Tuesday, July 10, 2007 2:16 PM > > > To: arslist@ARSLIST.ORG > > > Subject: ARS v7.0.1P2 LIST threads initiate inexplainably > > > > > > ** > > > Since our upgrade from ARS 5.1.2 to 7.0.1P2 on 6/27/07 I have no life. > Ok, > > > that's the end of my rant. > > > > > > Everyday, with the exception of Sat/Sun even though there are some > > > users on the system, we have a thread run wild issue. It is with > > > the LIST threads. This manifested itself in a arerror94 - > > > database timeout > > > > > > Of course everyone points to Remedy but it didn't feel like a > > > Remedy issue. I felt like network or database. Well after 4 days > > > of denial > the > > > other guys finally started looking at their sides and various > > > changes > were > > > made. Optimizations in oracle and some minor server changes. > > > Yesterday we actually ran the full high-production hours schedule > > > without timing out. Thought I was home free. > > > > > > Well on the way home the threads started to initiate again. And > > > finally by 10:30p I had to restart the services because we were at > > > 14 threads > and > > > our max is 15. Some sleep is required even after a v7 upgrade. > > > > > > Restarting the services clears the problem. Yesterday we ran most > > > of > the > > > day at 10 threads. I check old statistics from May and that's > > > what we were running at then. I know, it seems high ... that's > > > not the issue right now. > > > > > > We were having 343 errors appear after a restart which didn't seem > > > all that bad but apparently are VERY bad. Those got cleared away > > > today and > we > > > are not seeing them anymore. > > > > > > So today, it's been a horrific day. Four times we pushed the > > > threads to the limit and had to restart services. > > > > > > Over the last week or so I've become intimately familiar with my > > > sql > logs > > > again. Looking for trends, patterns, hints. Sometimes it seems like > > > when a person logs in it causes a thread to stop and restart. Or > > > maybe it's the start of the thread buildup. > > > > > > But what I do see periodically are the following errors. AND I > > > think > I'm > > > see a pattern that after one of these there is a slow buildup of > > > threads from 7-11 over 20-30 minutes then 12-15 is faster. We > > > start seeing hourglasses when I think we're at 14 on the way to 15. > > > > > > I'd like to know what the entry below is telling me. I've asked > > > bmc support but have received no answer. I've looked in the sql > > > logs at > this > > > time and don't see a real pattern to what the person was doing > > > that > would > > > drive me in a certain direction. > > > > > > Any knowledge on this you can share would be appreciated. > > > > > > Thanks, > > > Susan Palmer > > > ShopperTrak > > > > > > ARS v7.0.1P1 > > > Oracle 10g > > > Windows 2003 > > > Tue Jul 10 14:55:33 2007 390635 : AR System server terminated > > > when a signal/exception was received by the server (ARNOTE 20) Tue > > > Jul 10 14:55:33 2007 > > > Timestamp: Tue Jul 10 2007 14:55:34.1250 Thread Id: 2664 > > > Version: 7.0.01 patch 002 200704021644 Apr 2 2007 20:12:39 > > > ServerName: remsrv > > > Database: SQL -- Oracle > > > Hardware: Intel Pentium > > > OS: Windows NT 5.2 > > > RPC Id: 13200 > > > RPC Call: 34 (EXP) > > > RPC Queue: 390635 > > > Client: User AdamsT from Remedy User (protocol 9) at IP address > > > 10.0.4.68 > > > Form: > > > Logging On: API SQL Thread > > > Code: c0000005 > > > Operation: read > > > Access Addr: 736C6C69 > > > Stack Begin: > > > Stack End > > > Tue Jul 10 14:55:33 2007 390635 : AR System server terminated > > > when a signal/exception was received by the server (ARNOTE 20) > > > Tue Jul 10 14:55:33 2007 0xc0000005 > > > Tue Jul 10 14:55:33 2007 390635 : AR System server terminated -- > > > fatal error encountered (ARNOTE 21) Thread log that relates to the > > > above error from a timing perspective: > > > <THRD> /* Tue Jul 10 2007 14:15:16.3430 */ Thread Id 3108 (thread > > > number > > > 21) application statistics thread started. > > > <THRD> /* Tue Jul 10 2007 14:55:34.3430 */ Thread Id 2664 (thread > > > number > > > 16) on LIST queue died. > > > <THRD> /* Tue Jul 10 2007 14:55:34.3430 */ Thread Id 4864 (thread > > > number > > > 16) on LIST queue restarted. > > > <THRD> /* Tue Jul 10 2007 14:57:39.8750 */ Thread Id 5784 (thread > > > number > > > 22) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 14:59:05.8750 */ Thread Id 5640 (thread > > > number > > > 23) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 14:59:08.0620 */ Thread Id 5568 (thread > > > number > > > 24) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 14:59:27.4530 */ Thread Id 5652 (thread > > > number > > > 25) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 15:02:14.5620 */ Thread Id 2332 (thread > > > number > > > 26) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 15:03: 11.6560 */ Thread Id 5328 (thread > number > > > 27) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 15:03:17.9370 */ Thread Id 4908 (thread > > > number > > > 28) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 15:23:11.9370 */ Thread Id 5704 (thread > > > number > > > 29) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 15:26:09.5460 */ Thread Id 964 (thread > > > number > > > 30) on LIST queue started. > > > > > > > > > This thread terror occurred today too. First time FAST threads > > > have > been > > > involved: > > > <THRD> /* Tue Jul 10 2007 13:11:11.7650 */ Thread Trace Log -- ON > > > <THRD> /* Tue Jul 10 2007 13:11:11.7810 */ Thread Id 5856 (thread > > > number > > > 0) Thread Manager started. > > > <THRD> /* Tue Jul 10 2007 13:11: 11.7810 */ Thread Id 3236 (thread > number > > > 1) timed call thread started. > > > <THRD> /* Tue Jul 10 2007 13:11: 11.7810 */ Thread Id 5236 (thread > number > > > 2) on ADMIN queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5740 (thread > number > > > 3) on ALERT queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5384 (thread > number > > > 4) on ESCALATION queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5880 (thread > number > > > 5) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 3256 (thread > number > > > 6) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3432 (thread > > > number > > > 7) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 4812 (thread > number > > > 8) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3692 (thread > > > number > > > 9) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 904 (thread > > > number > > > 10) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4508 (thread > > > number > > > 11) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 4816 (thread > number > > > 12) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 5792 (thread > > > number > > > 13) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 4120 (thread > number > > > 14) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4216 (thread > > > number > > > 15) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 2960 (thread > number > > > 16) on LIST queue started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 2832 (thread > > > number > > > 17) license monitor thread started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5104 (thread > number > > > 18) archive thread started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7180 */ Thread Id 4960 (thread > > > number > > > 19) computed group call thread started. > > > <THRD> /* Tue Jul 10 2007 13:11: 37.7180 */ Thread Id 5340 (thread > number > > > 20) server statistics thread started. > > > <THRD> /* Tue Jul 10 2007 13:11:37.7180 */ Thread Id 4940 (thread > > > number > > > 21) application statistics thread started. > > > <THRD> /* Tue Jul 10 2007 13:40: 36.1400 */ Thread Id 5880 (thread > number > > > 5) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 5372 (thread > > > number > > > 5) on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:40: 36.1400 */ Thread Id 5372 (thread > number > > > 5) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 4196 (thread > > > number > > > 5) on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:40: 36.1710 */ Thread Id 4812 (thread > number > > > 8) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:40:36.1710 */ Thread Id 884 (thread > > > number > 8) > > > on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:40:38.6710 */ Thread Id 3692 (thread > > > number > > > 9) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:40:38.6710 */ Thread Id 1996 (thread > > > number > > > 9) on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:40:59.2500 */ Thread Id 4120 (thread > > > number > > > 14) on LIST queue died. > > > <THRD> /* Tue Jul 10 2007 13:40: 59.2500 */ Thread Id 5860 (thread > number > > > 14) on LIST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:41:39.4060 */ Thread Id 4216 (thread > > > number > > > 15) on LIST queue died. > > > <THRD> /* Tue Jul 10 2007 13:41:39.4060 */ Thread Id 5108 (thread > > > number > > > 15) on LIST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:41:42.9840 */ Thread Id 2960 (thread > > > number > > > 16) on LIST queue died. > > > <THRD> /* Tue Jul 10 2007 13:41:42.9840 */ Thread Id 2700 (thread > > > number > > > 16) on LIST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:42:19.5150 */ Thread Id 3256 (thread > > > number > > > 6) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:42:19.5150 */ Thread Id 2108 (thread > > > number > > > 6) on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:42:36.1710 */ Thread Id 3432 (thread > > > number > > > 7) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:42:36.1710 */ Thread Id 4064 (thread > > > number > > > 7) on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:42: 46.6090 */ Thread Id 480 (thread > > > number > > > 22) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:42:58.3590 */ Thread Id 4700 (thread > > > number > > > 23) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:43:05.3430 */ Thread Id 4816 (thread > > > number > > > 12) on LIST queue died. > > > <THRD> /* Tue Jul 10 2007 13:43:05.3430 */ Thread Id 5756 (thread > > > number > > > 12) on LIST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:43:13.6710 */ Thread Id 356 (thread > > > number > > > 24) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:43:13.7180 */ Thread Id 904 (thread > > > number > > > 10) on FAST queue died. > > > <THRD> /* Tue Jul 10 2007 13:43:13.7180 */ Thread Id 2812 (thread > > > number > > > 10) on FAST queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:43:25.6870 */ Thread Id 6140 (thread > > > number > > > 25) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:43:31.7180 */ Thread Id 3960 (thread > > > number > > > 26) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:43: 41.7810 */ Thread Id 2484 (thread > number > > > 27) on FAST queue started. > > > <THRD> /* Tue Jul 10 2007 13:46:39.1870 */ Thread Id 5384 (thread > > > number > > > 4) on ESCALATION queue died. > > > <THRD> /* Tue Jul 10 2007 13:46: 39.1870 */ Thread Id 3704 (thread > number > > > 4) on ESCALATION queue restarted. > > > <THRD> /* Tue Jul 10 2007 13:48:45.8750 */ Thread Id 5792 (thread > > > number > > > 13) on LIST queue died. > > > <THRD> /* Tue Jul 10 2007 13:48: 45.8750 */ Thread Id 824 (thread > > > number > > > 13) on LIST queue restarted. > > > __20060125_______________________This posting was > submitted with HTML in > > > it___ > > > __20060125_______________________This posting was > submitted with HTML in > > > it___ > > > __20060125_______________________This posting was > submitted with HTML in > > > it___ > > > > > > > > > CONFIDENTIALITY NOTICE: > > > This is a transmission from Kohl's Department Stores, Inc. > > > and may contain information which is confidential and proprietary. > > > If you are not the addressee, any disclosure, copying or > > > distribution or > use of the contents of this message is expressly prohibited. > > > If you have received this transmission in error, please destroy it > > > and > notify us immediately at 262-703-7000. > > > > > > CAUTION: > > > Internet and e-mail communications are Kohl's property and Kohl's > reserves the right to retrieve and read any message created, sent and > received. Kohl's reserves the right to monitor messages to or from > authorized Kohl's Associates at any time > > > without any further consent. > > > > > > > ______________________________________________________________________ > _________ > > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > > ARSlist:"Where > the Answers Are" > > > > > > > > > __20060125_______________________This posting was submitted with HTML > in it___ ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
