We reported this problem with 7.6.04.01 last summer and have been living with it ever since. The mid-tier caches incorrect login names or incorrect case login names that are passed to AREA LDAP, especially when people try to use the mid-tier with a mobile device that capitalizes their login, and then kills threads on the AR Server when AREA allows them in and they do not exist in the User table in that form (Jsmith versus jsmith). These incorrect or invalid account names are cached by the mid-tier, especially the ones where non-case-sensitive LDAP will authenticate them whereas case-sensitive ARS would not have. On the end of the next prefetch, the mid-tier throws them at the AR Server without passwords and they are in turn thrown at AREA and you get three or four thread crashes per account. I have seen the production mid-tier unusable for 30 minutes after a restart (a different mid-tier already connected is unaffected) as thread after thread crashed on the AR Server - exactly the same problem you experienced. Real high quality code, here; I don't think BMC ever tested the ehcache prefetching on systems with AREA authentication enabled.
We also saw the same thread deaths when someone used an ipad to hit our Kinetic portal - invariably a capitalized version of their login name - so AREA would let them in, then Kinetic would halt them when it could not match their credentials (Jsmith != jsmith) and could not pull their customer data. We finally killed that off with javascript in the Kinetic login page that forced the login name to lower case; we may have to do exactly the same thing on the mid-tier login.jsp pages (anyone have a working snippet of code for that??). I am testing now to see if this is fixed in SP3; we identified so many fatal installer problems in SP2 (ARS, mid-tier, Atrium, ITSM - against ITSM Suite servers upgraded from 7.1/7.0.02) last November that I have refused to use it for ANYTHING ever since. This defect is definitely NOT fixed if you only update the mid-tier to SP3 and the AR Server is still on SP1; I tried that first. I only just today got a full ARS SP3 over SP1 install completed on a clone of production, with a local and remote SP3 mid-tier, so I should be able to tell if they really fixed it tomorrow. If not, then I'll give up and start forcing the login to lower case on each mid-tier. That will keep Demo and similar mixed-case "system" accounts out too, but that's what the User Tool is still essential for. SP3 does fix the date-range crystal report problem we reported last summer (must be SP3 on BOTH the AR Server and mid-tier), as well as the Firefox-Work Info-text box defect we reported at the same time, so _some_ issues have been fixed that have caused us problems on production for the last 7 months. I think they fixed the Migrator login timeout problem introduced in 7.6.04.00 and retained in .01 and .02 (they finally gave me a post-SP2 hotfix that worked) but I have to test that as well. I'll post an update tomorrow. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ On 3/1/12 9:16 PM, "h@rry" <hari...@yahoo.com> wrote: >Hello Experts, > >We are on ARS and ITSM 7.6.04 and are facing an issue where we are >finding the below content in the error log after which application goes >into a timeout and users are unable to access Remedy, only solution is to >restart the application. This is not happening at any defined time >intervals, occurs sometimes when we have most of the users logged in and >sometimes when only 1 or 2 of them are logged in. Only similarity we >could find for all of these messages is that the user name appearing is >not existing in Remedy but existing in AD (we have the LDAP integration >done). However, there was were instances when the timeout occurred and >the user name which appeared in the log was an active user in Remedy and >at one instance, it also showed the user name as Demo. >We have raised a ticket with support and looking from the >LDAP/Authentication side of it, they are mentioning that this is a defect >which has been fixed in ARS 7.6.04 SP2. But we are not very sure if this >content in error log is occurring only because of authentication related >issue as mentioned by support or could there be any other reason. > >Though we have raised a case with support, i wanted to know if anyone >else has faced a similar issue and is upgrading to SP2 the solution for >this and moreover, is this issue related to authentication itself. Would >be great if anyone can help on this. > >Below content in error log is for a user who does not exist in Remedy and >is existing in AD. This did not lead to a timeout, but there was slowness >(almost inaccessible) in accessing the application for around 30 mins. > >Thu Mar 01 18:31:47 2012: AR System server terminated when a >signal/exception was received by the server (ARNOTE 20) > > Thread Id: 5728 > Version: 7.6.04 Build 002 201101141059 Jan 14 2011 11:55:43 > ServerName: RMDAPD01 > Database: SQL -- SQL Server > Hardware: x86_64 > OS: Windows 6.1 > RPC Id: 18539 > RPC Call: 33 (GLG) > RPC Queue: 390620 > Client: User z28801 from Mid-tier (protocol 18) at IP address >10.226.138.68 > Logging On: > Code: c0000005 > Operation: read > Access Addr: 0000000000000000 > Stack Begin: > Stack End >Thu Mar 01 18:31:47 2012 390620 : AR System server terminated when a >signal/exception was received by the server (ARNOTE 20) >Thu Mar 01 18:31:47 2012 0xc0000005 >Thu Mar 01 18:31:47 2012 390620 : AR System server terminated ‹ fatal >error occurred in ARSERVER (ARNOTE 21) > > >Below content in error log is for a user existing in Remedy and when it >led to a timeout and subsequent restart of application. > >Fri Mar 02 09:38:59 2012: AR System server terminated when a >signal/exception was received by the server (ARNOTE 20) > > Thread Id: 3576 > Version: 7.6.04 Build 002 201101141059 Jan 14 2011 11:55:43 > ServerName: RMDAPD01 > Database: SQL -- SQL Server > Hardware: x86_64 > OS: Windows 6.1 > RPC Id: 49046 > RPC Call: 36 (GSI) > RPC Queue: 390620 > Client: User Demo from SIM Publishing Server (protocol 12) at IP >address 10.226.138.72 > Logging On: > Code: c0000005 > Operation: read > Access Addr: 000000000000000C > Stack Begin: > Stack End >Fri Mar 02 09:38:59 2012 390620 : AR System server terminated when a >signal/exception was received by the server (ARNOTE 20) >Fri Mar 02 09:38:59 2012 0xc0000005 >Fri Mar 02 09:38:59 2012 390620 : AR System server terminated ‹ fatal >error occurred in ARSERVER (ARNOTE 21) >Fri Mar 02 09:42:04 2012 AssignEng : Timeout during database update -- >the operation has been accepted by the server and will usually complete >successfully (RMDAPD01) ARERR - 92 >Fri Mar 02 09:42:05 2012 AssignEng : AR System Application server >terminated -- fatal error encountered (ARAPPNOTE 4501) >Fri Mar 02 09:43:37 2012 : Action Request System(R) Server x64 Version >7.6.04 Build 002 201101141059 >(c) Copyright 1991-2010 BMC Software, Inc. >Fri Mar 02 09:51:26 2012 AssignEng : Timeout during database update -- >the operation has been accepted by the server and will usually complete >successfully (RMDAPD01) ARERR - 92 >Fri Mar 02 09:51:26 2012 AssignEng : AR System Application server >terminated -- fatal error encountered (ARAPPNOTE 4501) >Fri Mar 02 09:53:26 2012 AssignEng : Timeout during database update -- >the operation has been accepted by the server and will usually complete >successfully (RMDAPD01) ARERR - 92 >Fri Mar 02 09:53:26 2012 AssignEng : AR System Application server >terminated -- fatal error encountered (ARAPPNOTE 4501) >Fri Mar 02 09:54:26 2012 : Action Request System(R) Server x64 Version >7.6.04 Build 002 201101141059 >(c) Copyright 1991-2010 BMC Software, Inc. > >__________________________________________________________________________ >_____ >UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
