The plugin consists of a jar containing the plugin itself, some configs that need to happen in several files, and a set of forms and workflow to utilize the plugin. Essentially turning a 'Push-Modify' operation into a separate transaction.
-----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Friday, November 09, 2012 12:34 PM To: arslist@ARSLIST.ORG Subject: Re: XMLCE hangs, Oracle locks, arschema won't update (long, boring, but important...) Linux, Windows, or non-denominational? :) Is this just a set of xml files + plugin calls in the armonitor and ar.conf? Or are there binaries too? William Rentfrow wrentf...@stratacominc.com http://www.stratacominc.com 715-204-3061 Office 715-498-5056 Cell -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Longwing, LJ CTR MDA/IC Sent: Friday, November 09, 2012 10:04 AM To: arslist@ARSLIST.ORG Subject: Re: XMLCE hangs, Oracle locks, arschema won't update (long, boring, but important...) William, I did a presentation during WWRUG12 that discussed this exact topic. When you have high concurrency and have specific portions of the transaction, that if isolated, would allow for the main items to work without error, can allow for higher concurrency and prevent DB blocking. This topic was exceedingly difficult for me to troubleshoot and identify when I came across it, but eventually discovered that a specific set of transactions within my process were causing DB blocking that caused my transactions to become serial instead of parallel. I developed a plugin and a process that allowed me to isolate that portion of the transaction to a separate DB transaction, and thus allowed for the locking to clear on those tables, then allowing my overall transactions to process faster. I can provide you a copy of the plugin if you would like. It isn't 'production ready', but is 'demo ready' :) -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Friday, November 09, 2012 8:42 AM To: arslist@ARSLIST.ORG Subject: Re: XMLCE hangs, Oracle locks, arschema won't update (long, boring, but important...) Those two are actually in our short list of proposed fixes. Unfortunately we don't have definitive proof of what exactly the problem is. Our proposed list of possible fixes is this: -Set Next-ID-Commit: T (<---default prior to 7.6.03 is F, default in newest versions is T, ours is old hence we need to specify it). -Set Next-ID-Block-Size: 10 -Purge the table the rows are being put into. The performance actually seems pretty good when things are working well but the data is transactional throw-away data once a person is authenticated. We are doing this one for sure this weekend just to be safe -Offload the IVR transactions to their own web server (not likely due to redundancy issues). William Rentfrow wrentf...@stratacominc.com http://www.stratacominc.com 715-204-3061 Office 715-498-5056 Cell -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Longwing, LJ CTR MDA/IC Sent: Friday, November 09, 2012 9:29 AM To: arslist@ARSLIST.ORG Subject: Re: XMLCE hangs, Oracle locks, arschema won't update (long, boring, but important...) William, If you feel you are dealing with a concurrency issue on arschema, please look into two ar.cfg parameters Next-ID-Commit Next-ID-Block-Size Either one, or a combination of both may help you with your situation. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Friday, November 09, 2012 8:25 AM To: arslist@ARSLIST.ORG Subject: XMLCE hangs, Oracle locks, arschema won't update (long, boring, but important...) ** Hi listers - We're seeing something peculiar. This is on a fairly high volume (5000 incidents/day) system. There's an integration with an IVR that does a lookup via web services to authenticate someone is actually an employee. That uses a web service that's been dependable for a couple of years (at least). All this particular WS does is create a new record in a table. The system was recently migrated from Solaris to Linux. It's using ARS 7.1 and ITSM 7.1x - the database is Oracle RAC, remote. I am wondering if there is a known bug because when we look at the log we see this when things are behaving fine (IP removed): Tue Nov 06 2012 10:23:51.7183 */+XMLCE ARXMLCreateEntry from Webservice (protocol 13) at IP address xx.xx.xx.xx Tue Nov 06 2012 10:23:51.7686 */-XMLCE OK But occasionally we see the first line and no corresponding -XMLCE. About 11 minutes (note milliseconds in the error below) later in our SystemOut.log from Websphere: [11/6/12 13:34:04:866 CST] 00000037 SystemOut O - Connects to ARServer (server name removew) through Java Rpc failed with: ERROR (91): RPC call failed; ONC/RPC call timed out [11/6/12 13:34:04:882 CST] 0000006c SystemOut O - getApiRecording: 0 [11/6/12 13:34:04:883 CST] 00000049 SystemOut O - getApiRecording: 0 [11/6/12 13:34:04:920 CST] 00000069 SystemOut O - getApiRecording: 0 [11/6/12 13:36:04:847 CST] 0000006c SystemOut O - getApiRecording: 0 [11/6/12 13:36:04:878 CST] 00000069 SystemOut O - getApiRecording: 0 [11/6/12 13:36:53:342 CST] 000000a2 SystemOut O - getApiRecording: 0 [11/6/12 13:36:53:343 CST] 000000a1 SystemOut O - getApiRecording: 0 [11/6/12 13:37:15:266 CST] 000000a4 SystemOut O - getApiRecording: 0 [11/6/12 13:37:15:266 CST] 000000a5 SystemOut O - getApiRecording: 0 [11/6/12 13:37:35:817 CST] 000000a7 SystemOut O - getApiRecording: 0 [11/6/12 13:38:04:820 CST] 00000037 SystemOut O - getApiRecording: 0 [11/6/12 13:38:04:822 CST] 00000053 SystemOut O - getApiRecording: 0 [11/6/12 13:38:04:826 CST] 0000006c SystemOut O - getApiRecording: 0 [11/6/12 13:38:04:843 CST] 0000002e SystemOut O - getApiRecording: 0 [11/6/12 13:38:04:852 CST] 00000069 SystemOut O - getApiRecording: 0 [11/6/12 13:38:53:307 CST] 000000a2 SystemOut O - getApiRecording: 0 [11/6/12 13:40:04:784 CST] 00000053 SystemOut O - getApiRecording: 0 [11/6/12 13:40:04:802 CST] 0000003c SystemOut O - getApiRecording: 0 [11/6/12 13:40:53:267 CST] 000000a2 SystemOut O - getApiRecording: 0 [11/6/12 13:41:09:066 CST] 00000032 ThreadMonitor W WSVR0605W: Thread "WebContainer : 51" (00000063) has been active for 675278 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung. at com.remedy.arsys.goat.preferences.ARUserPreferences.getUserPreferences(Unknown Source) at com.remedy.arsys.stubs.SessionData.<init>(Unknown Source) at com.remedy.arsys.session.Login.initWebService(Unknown Source) at com.remedy.arsys.ws.services.ARService.login(Unknown Source) at com.remedy.arsys.ws.services.ARService.processRequest(Unknown Source) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.apache.axis.providers.java.MsgProvider.processMessage(MsgProvider.java:141) at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319) at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32) at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118) at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83) at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453) at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281) at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:763) at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1213) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1154) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:118) at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:848) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:691) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:654) at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:526) at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:764) at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1478) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:133) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:450) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:508) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:296) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:270) at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1069) at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:728) at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:415) at com.ibm.ws.ssl.channel.impl.SSLUtils.handleHandshake(SSLUtils.java:942) at com.ibm.ws.ssl.channel.impl.SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:70) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:196) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:792) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1551) At this point we experience a database lock on the arschema table where Oracle has been trying to update the nextid for the table in the XMLCE and it has not released the lock for the row in question. This is the row for the table that the XMLCE is firing against. Our DBA has to manually clear the locked thread(s - usually more than one) and then the final -XMLCE entry is put into the arapi log file and the system frees up. I've never see this before - and there's not enough information in the logs to determine what exactly is happening. We have a case open with BMC but this has all of the hallmarks of an obscure bug that only happens on Tuesdays during a thunderstorm. Unfortunately it's becoming more frequent. William Rentfrow wrentf...@stratacominc.com http://www.stratacominc.com 715-204-3061 Office 715-498-5056 Cell _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
