Hi Axton, Thanks for reading and for your suggestions.
I have done a little bit of JVM monitoring. I used jconsole, since it was already there and was easy enough to startup. I watched the various heaps over a weeks time and I didn't see anything alarming. GC seemed to be happening regularly and doesn't seem to to take very long. These are my JAVA_OPTS: -XX:MaxPermSize=256m -Xms2048m -Xmx2048m -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewRatio=3 Apache Tomcat/6.0.32 JVM: Sun 1.6.0_31-b04 Linux: 2.6.32-131.17.1.el6.x86_64 Your question about how the Mid-tier servers connect to the AR server was a good one. I believe I am using proxy connections with the default of 80 as a max. I was using the default queues but I am wondering if there would be any advantage to setting up a dedicated queue for the Mid-tier servers to hit. I think I'll give that a try to see if anything changes. Probably won't, since RRR|Log indicates no thread contention. But maybe those threads are managed differently. Do you agree that using proxy is more efficient or would you expect better performance if that was disabled? If it is disabled, does the Mid-tier make a dedicated connection to the AR server for each Mid-tier user? I'm a little vague on how that works and I was not able to find much in the documentation about it. Any pointers on where to learn more about that? catalina.out shows the following settings: - jrpcMode: true - jniLoadMode: 1 - apiRecordingMode: 0 - useConnectionPooling: true - maxProxiesPerServer: 80 - idleConnectionsPerServer: 5 - connectionTimeout: 0 - connectionLifespan: 0 - timeUnit: MINUTES - minimumSupportedServerRpcVersion: 12 - connectionMaxRetries: 1 - timeLagBetweenRetriesMillSec: 100 - stringizeFieldAssignments: false - stringizeSetIfQualification: false - useLegacyQualParser: false - useLegacyAssignParser: false - useLegacyQualFormatter: false - useLegacyAssignFormatter: false - UsePooling: true - , Pool setting info: , # maxProxyPerServer- 80, # idleConnectionsPerServer- 5, connectionTimeout setting- 00:00:00, connectionLifespan setting- 00:00:00 - Currently, connection pooling true - Client adjusts connection variables maximumPoolSize = 80 minIdleProxySize = 5 connectionTimeout = 0 timeUnit = null - Client sets useConnectionPooling to : true - Root WebApplicationContext: initialization completed in 2594 ms - Currently, connection pooling true - Client sets connection limits per server to :80 - UsePooling: true - , Pool setting info: , # maxProxyPerServer- 80, # idleConnectionsPerServer- 5, connectionTimeout setting- 00:00:00, connectionLifespan setting- 00:00:00 - Client sets useConnectionPooling to : true - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@7952506e - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@60e1e567 - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@477588d5 - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@756a7c99 Thanks again for your help. Larry On Thu, Mar 29, 2012 at 11:03 AM, Axton <axton.gr...@gmail.com> wrote: > ** Well constructed post. It gives me a lot to work with when thinking > about your problem. I applaud you for that. > > The interesting thing is that a restart of either the Tomcat servers or > the ARServer fixes your issue. This throws an unknown variable into the > mix (a monkey wrench, of sorts). > > If you are inclined to dig into the Tomcat performance, I suggest enabling > JMX and hooking up a JMX monitoring tool to the Tomcat instances (I prefer > jvisualvm, which is part of the jdk) to see what the server is busy doing > and where the memory allocation is going. You might find that the CPU time > required for garbage collection gets higher and higher. If this is the > case, depending on the configuration of the Linux VMs (cpu count, memory > size, etc.) that house Tomcat, you may be able to play with the GC settings > to alleviate this. It may point you to some issue within the midtier that > you can't address, but if you know where the problem is you can work with > support. > > If using JMX, you will want to use a local jvisualvm so that you can get > cpu profiling statistics. > > You might also look at how the midtier connects to the arserver. Do you > use a proxy connection? Do you use a private queue? If you use a private > queue, how many threads does it have and are those threads congested? > > Axton Grams > > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
