The stack errors come across as ARNOTE20 errors and they restart
themselves. They can be FAST or LIST, although mostly FAST.
We are not getting arerror 93's.
We don't have any plugins. We do have HD v5.0. So I am getting some
periodic rappsrv errors.
It almost seems like when someone, obviously not everyone, logs in it can
start the thread-storm. I'm just grasping at straws because the time
correlation to the initial thread error doesn't seem quite close enough.
We've had 3 storms already today. I feel like I'm in Toronado alley.
I think we'll try and go to Patch 3 tonight.
If that doesn't work better I need to figure out a plan to move back to some
other version because this is a killer. How does bmc expect to keep
customers if we can't keep the server up! It will take years to win back
confidence in the product with my customers.
Susan
On 7/11/07, Tony Worthington <[EMAIL PROTECTED]> wrote:
Susan, Axton, Ben, etc...
We have the same issues. Our server comes and goes as it pleases, and
we've yet to track down exactly what the cause is. We do know some of our
problems stem from plugins, most of the time it's ardbcquery. The easiest
way to crash a thread is to open either the Overview or Problem console.
Other times, the server just gives up, we get ARERR93's and end up having
to restart the service.
I think things have gotten worse after patch002 and 003, but things have
never been stable.
Here's an excerpt from a High priority ticket we have open -- with no
solutions so far:
Faulting application arcmdbd.exe, version 2.0.1.3, faulting module
arcmdbd.exe, version 2.0.1.3, fault address 0x0000a981.
Mon Jul 02 12:45:05 2007 390695 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Mon Jul 02 12:45:05 2007 0xc0000005
Mon Jul 02 12:45:05 2007 390695 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Mon Jul 02 12:46:12 2007 390695 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Mon Jul 02 12:46:12 2007 0xc0000005
We're running with lots of threads, but are on a 8 processor (dual core
3ghz) box with 18gb of ram. :-) Remedy is still slow. Our DB flies, and
network is fine.
Private-RPC-Socket: 390601 2 8
Private-RPC-Socket: 390620 4 8
Private-RPC-Socket: 390621 2 8
Private-RPC-Socket: 390626 4 12
Private-RPC-Socket: 390627 2 8
Private-RPC-Socket: 390628 12 12
Private-RPC-Socket: 390629 2 8
Private-RPC-Socket: 390635 4 8
Private-RPC-Socket: 390690 1 6
Windows 2003R2 EE
7.0.0.1 patch 003
Remote Oracle 10gR2 RAC/Linux
ITSM7p004
-tony
--
Tony Worthington
[EMAIL PROTECTED]
262-703-5911
Ben Cantatore <[EMAIL PROTECTED]>
Sent by: "Action Request System discussion list(ARSList)"
<arslist@ARSLIST.ORG>
07/11/2007 12:02 AM
Please respond to
arslist@ARSLIST.ORG
To
arslist@ARSLIST.ORG
cc
Subject
Re: ARS v7.0.1P2 LIST threads initiate inexplainably
**
Axton and Susan, you're not alone with the stack errors. I've been
plagued with them since my launch back in Feb. In my situation the server
runs about 2 weeks and crashes and usually recovers on its own. One
problem I had which I could reproduce crashing the server was fixed by
patch 3 so I think what Davies is saying applied to that. The stack
problems have gotten a little better, but still happens. I have API,
Escalation, Filter, SQL and Thread logging turned on and am currently
waiting for the next crash. So I'll post then with any interesting
information that surfaces from that.
Susan, sounds like you're having a worse problem than I am. How often do
the crashes happen, does the arserver recover on its own?
Currently on Server 7.0.1 patch 2 and ITSM 7.0.2 patch 3 for all modules
connecting to Oracle 10g db.
Ben Cantatore
Remedy Administrator
Avon
(914) 935-2946
"Davies, J.T." <[EMAIL PROTECTED]>
Sent by: "Action Request System discussion list(ARSList)"
<arslist@ARSLIST.ORG>
07/10/2007 05:38 PM
Please respond to
arslist@ARSLIST.ORG
To
arslist@ARSLIST.ORG
cc
Subject
Re: ARS v7.0.1P2 LIST threads initiate inexplainably
**
Hi Susan,
I've experienced these errors before, too, a long time ago: Signal 5
termination on a specific thread.
I was able to trace it back to a workflow problem, and with your jump from
5.1.2 to 7.0, I'd imagine you might be in the same boat.
What I found was with the Run Process commands.
To give an example:
You're calling the Application-Delete-Entry command. It's expecting two
inputs: Form and ID.
The problem is akin to receiving a bad input value. Perhaps the ID was
missing, or the Form name was invalid...some wacky scenario where the
command raised an error, and thus, the users execution died.
I also found the errors were occurring if a command was expecting an
integer value (say, the number of seconds to offset on a Business Time
command), but the input was some character string ("ABC").
You might see if your logs (Filter and SQL) can narrow this down for you.
You might have to mentally process some of these commands, because I do
remember that it wasn't very apparent in the logs that it was failing on
these.
Hope this helps or provides a little more direction!
J.T.
New Edge Networks
An Earthlink Company
From: Action Request System discussion list(ARSList)
[mailto:[EMAIL PROTECTED] On Behalf Of Susan Palmer
Sent: Tuesday, July 10, 2007 2:16 PM
To: arslist@ARSLIST.ORG
Subject: ARS v7.0.1P2 LIST threads initiate inexplainably
**
Since our upgrade from ARS 5.1.2 to 7.0.1P2 on 6/27/07 I have no life. Ok,
that's the end of my rant.
Everyday, with the exception of Sat/Sun even though there are some users
on the system, we have a thread run wild issue. It is with the LIST
threads. This manifested itself in a arerror94 - database timeout
Of course everyone points to Remedy but it didn't feel like a Remedy
issue. I felt like network or database. Well after 4 days of denial the
other guys finally started looking at their sides and various changes were
made. Optimizations in oracle and some minor server changes. Yesterday
we actually ran the full high-production hours schedule without timing
out. Thought I was home free.
Well on the way home the threads started to initiate again. And finally
by 10:30p I had to restart the services because we were at 14 threads and
our max is 15. Some sleep is required even after a v7 upgrade.
Restarting the services clears the problem. Yesterday we ran most of the
day at 10 threads. I check old statistics from May and that's what we
were running at then. I know, it seems high ... that's not the issue
right now.
We were having 343 errors appear after a restart which didn't seem all
that bad but apparently are VERY bad. Those got cleared away today and we
are not seeing them anymore.
So today, it's been a horrific day. Four times we pushed the threads to
the limit and had to restart services.
Over the last week or so I've become intimately familiar with my sql logs
again. Looking for trends, patterns, hints. Sometimes it seems like
when a person logs in it causes a thread to stop and restart. Or maybe
it's the start of the thread buildup.
But what I do see periodically are the following errors. AND I think I'm
see a pattern that after one of these there is a slow buildup of threads
from 7-11 over 20-30 minutes then 12-15 is faster. We start seeing
hourglasses when I think we're at 14 on the way to 15.
I'd like to know what the entry below is telling me. I've asked bmc
support but have received no answer. I've looked in the sql logs at this
time and don't see a real pattern to what the person was doing that would
drive me in a certain direction.
Any knowledge on this you can share would be appreciated.
Thanks,
Susan Palmer
ShopperTrak
ARS v7.0.1P1
Oracle 10g
Windows 2003
Tue Jul 10 14:55:33 2007 390635 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Tue Jul 10 14:55:33 2007
Timestamp: Tue Jul 10 2007 14:55:34.1250
Thread Id: 2664
Version: 7.0.01 patch 002 200704021644 Apr 2 2007 20:12:39
ServerName: remsrv
Database: SQL -- Oracle
Hardware: Intel Pentium
OS: Windows NT 5.2
RPC Id: 13200
RPC Call: 34 (EXP)
RPC Queue: 390635
Client: User AdamsT from Remedy User (protocol 9) at IP address
10.0.4.68
Form:
Logging On: API SQL Thread
Code: c0000005
Operation: read
Access Addr: 736C6C69
Stack Begin:
Stack End
Tue Jul 10 14:55:33 2007 390635 : AR System server terminated when a
signal/exception was received by the server (ARNOTE 20)
Tue Jul 10 14:55:33 2007 0xc0000005
Tue Jul 10 14:55:33 2007 390635 : AR System server terminated -- fatal
error encountered (ARNOTE 21)
Thread log that relates to the above error from a timing perspective:
<THRD> /* Tue Jul 10 2007 14:15:16.3430 */ Thread Id 3108 (thread number
21) application statistics thread started.
<THRD> /* Tue Jul 10 2007 14:55:34.3430 */ Thread Id 2664 (thread number
16) on LIST queue died.
<THRD> /* Tue Jul 10 2007 14:55:34.3430 */ Thread Id 4864 (thread number
16) on LIST queue restarted.
<THRD> /* Tue Jul 10 2007 14:57:39.8750 */ Thread Id 5784 (thread number
22) on LIST queue started.
<THRD> /* Tue Jul 10 2007 14:59:05.8750 */ Thread Id 5640 (thread number
23) on LIST queue started.
<THRD> /* Tue Jul 10 2007 14:59:08.0620 */ Thread Id 5568 (thread number
24) on LIST queue started.
<THRD> /* Tue Jul 10 2007 14:59:27.4530 */ Thread Id 5652 (thread number
25) on LIST queue started.
<THRD> /* Tue Jul 10 2007 15:02:14.5620 */ Thread Id 2332 (thread number
26) on LIST queue started.
<THRD> /* Tue Jul 10 2007 15:03: 11.6560 */ Thread Id 5328 (thread number
27) on LIST queue started.
<THRD> /* Tue Jul 10 2007 15:03:17.9370 */ Thread Id 4908 (thread number
28) on LIST queue started.
<THRD> /* Tue Jul 10 2007 15:23:11.9370 */ Thread Id 5704 (thread number
29) on LIST queue started.
<THRD> /* Tue Jul 10 2007 15:26:09.5460 */ Thread Id 964 (thread number
30) on LIST queue started.
This thread terror occurred today too. First time FAST threads have been
involved:
<THRD> /* Tue Jul 10 2007 13:11:11.7650 */ Thread Trace Log -- ON
<THRD> /* Tue Jul 10 2007 13:11:11.7810 */ Thread Id 5856 (thread number
0) Thread Manager started.
<THRD> /* Tue Jul 10 2007 13:11: 11.7810 */ Thread Id 3236 (thread number
1) timed call thread started.
<THRD> /* Tue Jul 10 2007 13:11:11.7810 */ Thread Id 5236 (thread number
2) on ADMIN queue started.
<THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5740 (thread number
3) on ALERT queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 5384 (thread number
4) on ESCALATION queue started.
<THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5880 (thread number
5) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3256 (thread number
6) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3432 (thread number
7) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4812 (thread number
8) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 3692 (thread number
9) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 904 (thread number
10) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4508 (thread number
11) on LIST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4816 (thread number
12) on LIST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 5792 (thread number
13) on LIST queue started.
<THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 4120 (thread number
14) on LIST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 4216 (thread number
15) on LIST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 2960 (thread number
16) on LIST queue started.
<THRD> /* Tue Jul 10 2007 13:11:37.7030 */ Thread Id 2832 (thread number
17) license monitor thread started.
<THRD> /* Tue Jul 10 2007 13:11: 37.7030 */ Thread Id 5104 (thread number
18) archive thread started.
<THRD> /* Tue Jul 10 2007 13:11:37.7180 */ Thread Id 4960 (thread number
19) computed group call thread started.
<THRD> /* Tue Jul 10 2007 13:11: 37.7180 */ Thread Id 5340 (thread number
20) server statistics thread started.
<THRD> /* Tue Jul 10 2007 13:11:37.7180 */ Thread Id 4940 (thread number
21) application statistics thread started.
<THRD> /* Tue Jul 10 2007 13:40: 36.1400 */ Thread Id 5880 (thread number
5) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 5372 (thread number
5) on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 5372 (thread number
5) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:40:36.1400 */ Thread Id 4196 (thread number
5) on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:40:36.1710 */ Thread Id 4812 (thread number
8) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:40:36.1710 */ Thread Id 884 (thread number 8)
on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:40:38.6710 */ Thread Id 3692 (thread number
9) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:40:38.6710 */ Thread Id 1996 (thread number
9) on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:40:59.2500 */ Thread Id 4120 (thread number
14) on LIST queue died.
<THRD> /* Tue Jul 10 2007 13:40: 59.2500 */ Thread Id 5860 (thread number
14) on LIST queue restarted.
<THRD> /* Tue Jul 10 2007 13:41:39.4060 */ Thread Id 4216 (thread number
15) on LIST queue died.
<THRD> /* Tue Jul 10 2007 13:41:39.4060 */ Thread Id 5108 (thread number
15) on LIST queue restarted.
<THRD> /* Tue Jul 10 2007 13:41:42.9840 */ Thread Id 2960 (thread number
16) on LIST queue died.
<THRD> /* Tue Jul 10 2007 13:41:42.9840 */ Thread Id 2700 (thread number
16) on LIST queue restarted.
<THRD> /* Tue Jul 10 2007 13:42:19.5150 */ Thread Id 3256 (thread number
6) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:42:19.5150 */ Thread Id 2108 (thread number
6) on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:42:36.1710 */ Thread Id 3432 (thread number
7) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:42:36.1710 */ Thread Id 4064 (thread number
7) on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:42: 46.6090 */ Thread Id 480 (thread number
22) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:42:58.3590 */ Thread Id 4700 (thread number
23) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:43:05.3430 */ Thread Id 4816 (thread number
12) on LIST queue died.
<THRD> /* Tue Jul 10 2007 13:43:05.3430 */ Thread Id 5756 (thread number
12) on LIST queue restarted.
<THRD> /* Tue Jul 10 2007 13:43:13.6710 */ Thread Id 356 (thread number
24) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:43:13.7180 */ Thread Id 904 (thread number
10) on FAST queue died.
<THRD> /* Tue Jul 10 2007 13:43:13.7180 */ Thread Id 2812 (thread number
10) on FAST queue restarted.
<THRD> /* Tue Jul 10 2007 13:43:25.6870 */ Thread Id 6140 (thread number
25) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:43:31.7180 */ Thread Id 3960 (thread number
26) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:43: 41.7810 */ Thread Id 2484 (thread number
27) on FAST queue started.
<THRD> /* Tue Jul 10 2007 13:46:39.1870 */ Thread Id 5384 (thread number
4) on ESCALATION queue died.
<THRD> /* Tue Jul 10 2007 13:46: 39.1870 */ Thread Id 3704 (thread number
4) on ESCALATION queue restarted.
<THRD> /* Tue Jul 10 2007 13:48:45.8750 */ Thread Id 5792 (thread number
13) on LIST queue died.
<THRD> /* Tue Jul 10 2007 13:48: 45.8750 */ Thread Id 824 (thread number
13) on LIST queue restarted.
__20060125_______________________This posting was submitted with HTML in
it___
__20060125_______________________This posting was submitted with HTML in
it___
__20060125_______________________This posting was submitted with HTML in
it___
CONFIDENTIALITY NOTICE:
This is a transmission from Kohl's Department Stores, Inc.
and may contain information which is confidential and proprietary.
If you are not the addressee, any disclosure, copying or distribution or
use of the contents of this message is expressly prohibited.
If you have received this transmission in error, please destroy it and
notify us immediately at 262-703-7000.
CAUTION:
Internet and e-mail communications are Kohl's property and Kohl's reserves
the right to retrieve and read any message created, sent and
received. Kohl's reserves the right to monitor messages to or from
authorized Kohl's Associates at any time
without any further consent.
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where
the Answers Are"
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers
Are"
