Hi Rod, Filter logging against the relevant thread just shows processing stopping. No errors, it just hangs.
I am considering a change in server configuration - moving the escalations processing from the "user" primary to the integration server. Users (aside from me and the other developers) never log onto there, it would in theory free up some server load on the user facing servers. Regards Dave On 16 March 2016 at 22:14, Rod Harris <r...@smapps.com.au> wrote: > ** > Hi Dave, > > From what you describe it does sound like your escalation is stuck in some > kind of infinite loop. You should put filter logging on the server that is > hosting your escalation. You may find that your filters being triggered by > the escalation are never completing and perhaps timing out prior to the end > of the transaction. This may cause the transaction to roll back and for the > same condition to trigger again and again, not giving anything else a look > in. When this happens you may see very little in the escalation log for the > impacted pool. > > The root cause of this may be a difference in data or configuration > between servers or may also be that the workflow is not designed to scale > to production loaded servers. > > Hope this helps, > > Rod Harris > > > > On 17 March 2016 at 08:49, Dave Barber <daddy.bar...@gmail.com> wrote: > >> ** >> Just realised a flaw in my earlier checks - all 3 servers are in a server >> group, escalations can only run on one server in the group. >> >> So regardless of which server triggers the escalation process into action >> (setting a value on a form), the escalation will always run on the same >> server - and subsequently the related filters will also only run on the >> server nominated for escalations ... >> >> And doing these checks after having been up for about 18 hours isn't >> advisable .... >> >> Regards >> >> Dave >> >> On 16 March 2016 at 20:26, Dave Barber <daddy.bar...@gmail.com> wrote: >> >>> All 3 servers are running from the same database (ie. same codebase). >>> >>> The two solaris servers are (supposedly) identical hardware and builds >>> (Solaris, Java, etc.). The Red Hat box is totally different in almost >>> every possible way (processors, memory, probably java version). And it is >>> only on one of the two Solaris boxes where the escalation fails. >>> >>> The escalation in question triggers a table loop through a list of >>> impacted areas related to a change, in order to build a contact list for >>> notifications. The data set in question has around 6,500 impacted areas to >>> loop through, remove duplicate customers/contacts, etc. The actual set of >>> filters triggered are best described as "complicated". But as it runs fine >>> on two of the three servers, that is why I am considering AR Server >>> configuration issues - and it is easier for me to investigate and change >>> such things. >>> >>> I'll check out your confdiff app Misi, much appreciated (currently using >>> Notepad++ with a comparison plugin). >>> >>> Regards >>> >>> Dave >>> >>> On 16 March 2016 at 19:26, <remedy...@gmail.com> wrote: >>> >>>> Same version of Java ? >>>> Just wondering >>>> >>>> >>>> Sent from my iPhone >>>> >>>> > On Mar 16, 2016, at 2:18 PM, Misi Mladoniczky <m...@rrr.se> wrote: >>>> > >>>> > Hi, >>>> > >>>> > It would be interesting to know more about what the escalation is >>>> actually >>>> > doing. Number of records, what is the action, etc. >>>> > >>>> > I have a handy tool to compare config files that you can download >>>> here: >>>> > https://rrr.se/cgi/tools/main?tool=rrrConfDiff >>>> > >>>> > Best Regards - Misi, RRR AB, http://www.rrr.se (ARSList MVP >>>> 2011) >>>> > >>>> > Ask the Remedy Licensing Experts (Best R.O.I. Award at >>>> WWRUG10/11/12/13): >>>> > * RRR|License - Not enough Remedy licenses? Save money by optimizing. >>>> > * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy >>>> logs. >>>> > Find these products, and many free tools and utilities, at >>>> http://rrr.se. >>>> > >>>> >> All, >>>> >> >>>> >> Have an interesting one; we have an ARS 7.5/ITSM 7.6 system running >>>> on : >>>> >> 1 server RHL, used for integration/CMDB processing >>>> >> 2 servers running Solaris, in an active/passive load balanced >>>> environment, >>>> >> users will typically use either of these. >>>> >> >>>> >> Users normally connect to the server designated as primary (ie. the >>>> active). >>>> >> >>>> >> We implemented some new functionality driven from escalations - runs >>>> >> absolutely fine on our test environments, but it has started to fail >>>> on the >>>> >> active server. And when I say "fail", I mean that all escalations >>>> just >>>> >> grind to a halt the moment when this new functionality is invoked. >>>> Nothing >>>> >> explanatory in arerror.log, I have full filter/escalation logging >>>> enabled >>>> >> (on a per thread basis), and it gives no error either - escalations >>>> just >>>> >> plain stop. >>>> >> >>>> >> What makes this interesting is that if I run this same process (same >>>> change >>>> >> request, same data, etc) via the integration server or the >>>> secondary, it >>>> >> runs without any problems. >>>> >> >>>> >> As far as I can tell we have the escalations threads setup the same >>>> on both >>>> >> of the "user" solaris servers >>>> >> Server #1 (which fails) >>>> >> Private-RPC-Socket: 390601 1 1 >>>> >> Private-RPC-Socket: 390603 3 8 (escalations threads) >>>> >> Private-RPC-Socket: 390620 12 30 >>>> >> Private-RPC-Socket: 390630 3 4 >>>> >> Private-RPC-Socket: 390635 12 30 >>>> >> Private-RPC-Socket: 390680 15 30 >>>> >> >>>> >> Server #2 (which runs fine) >>>> >> Private-RPC-Socket: 390601 1 1 >>>> >> Private-RPC-Socket: 390603 3 8 >>>> >> Private-RPC-Socket: 390620 12 30 >>>> >> Private-RPC-Socket: 390635 12 30 >>>> >> Private-RPC-Socket: 390680 15 30 >>>> >> >>>> >> I am thinking of going through the various configurations in ar.conf >>>> and >>>> >> aligning the failing primary wherever possible/practical - but any >>>> hints as >>>> >> to why it could be failing welcome! >>>> >> >>>> >> Regards >>>> >> >>>> >> Dave Barber >>>> >> >>>> >> >>>> _______________________________________________________________________________ >>>> >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >>>> >> "Where the Answers Are, and have been for 20 years" >>>> > >>>> > >>>> _______________________________________________________________________________ >>>> > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >>>> > "Where the Answers Are, and have been for 20 years" >>>> >>>> >>>> _______________________________________________________________________________ >>>> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >>>> "Where the Answers Are, and have been for 20 years" >>>> >>> >>> >> _ARSlist: "Where the Answers Are" and have been for 20 years_ >> > > _ARSlist: "Where the Answers Are" and have been for 20 years_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
