I'm going to try to explain this problem today to BMC Remedy support too.
I believe I've done my homework and investigated well enough to characterize it.
Certainly we can't be the only Remedy installation to experience what I'm going
to
talk about????
I have searched the BMC Remedy KB for issues related to threads, Solaris, memory usage,
notification issues, phase 3, etc.
CURRENT ENVIRONMENT:
ARS 5.1.2 patch 1303/1484 production/development
Mid Tier 5.1.2/6.3 patch 14 production/development
RMS Help Desk 5.5.1 w/ Change Mgt & Asset modules
Solaris 8, patched as per ARS 6.3 compatibility matrix
Oracle 9.2.0.7, with 8.1.7 client
Oracle, Mid Tier, ARS all on same box per server
PROBLEM: We have a silent, intermittent problem identified by incomplete phase 3 notify
workflow with new Help Desk requests. I say "silent" because no error messages are being
generated anywhere. Remedy keeps working, no freezing, etc.
I need to get this fixed for my users (and my sanity).
I investigated because users were reporting occasionally missing their notifications. I
investigated the auto group assignment notifications specifically. It didn't seem to be
ALL groups but some don't rely on them as much as others so I can't tell. I think this
has been going on since upgrading to 5 but perhaps before that as well.
Email messages/alerts are not being created for some members of a group when this occurs.
The respective forms show no entries. Usually, it seems that at least one member's
notification is created for the auto-assign group when a request is created.
Comparing bad/good logs in one case, the SELECT on user_cache for the group is the same on
both; SELECTs on user_cache for individual group members are all missing in the broken
environment. It seems at least one member's notification is created, then, bad logs are
missing the repeat workflow for additional notifications. I could send the logs.
I upped Maximum Stack of Filters to 35 and 50 but it doesn't make this go away.
It seems that it is stress and memory-related, not a slow memory leak.
Restarting the Remedy processes fixes the problem but it is apparently
triggered again, as far as I can duplicate it, by a combination of
1) registered user change(s)
2) server configuration change(s)
3) high volume queries/reports
Daily restarts for cold backups were the norm for years but we were able to change to a
24/7 Remedy; this made the problem obvious. Daily restarts again reduced problem but it
still occurs, if the stressors occur.
The order of stressing activities does not seem to matter.
One change in and of itself does not trigger the problem.
Here are things I tested that do not seem to be involved.
-) Memory--our production server has plenty of memory; test server does not.
Same problem on both servers. Swap is available.
-) Leaving the server run 24/7 does not seem to trigger the problem until the activity
mentioned occurs. The problem can be triggered within a business day after a restart
or take several days. Server events form is helpful here.
-) Fast/List min threads equivalent vs not equivalent to max threads.
-) Increasing max threads.
-) Number of people in assigned group. A simple group of two people shows same
problem
as one:many CTI:group mappings with 8 people to the group of the problem
request.
I've seen 1 of 2, 4 of 8, and 1 of 8 get missed for notifications.
-) Email and alert notifications misbehave the same.
My testing was not done with Mid Tier web client but I remember it happening
there.
Makes sense since we are dealing with filters. I remember this problem with Chg Mgt
notifications as well.
Anyway, if anyone has a CLUE about the root cause, we'd very much appreciate it!
Ideally, BMC Remedy support will send an answer, it will work, and I'll post the
fix or workaround here!
I hope everyone is having a great Spring!
Regards,
Ann Kosch
~*~ ~*~ ~*~
A. R. Kosch
Special Projects/Analyst
Remedy ARS Administrator/Coordinator
[EMAIL PROTECTED]
785-532-4933
Kansas State University
Computing and Network Services
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at http://www.wwrug.org
