Re: [Resin-interest] Resin Pro 3.1.9 - JDK Deadlock on Startup

Thu, 18 Jun 2009 19:47:19 -0700 


On Jun 18, 2009, at 19:31, Scott Ferguson wrote:



On Jun 18, 2009, at 5:22 PM, Rob Lockstone wrote:
I've never seen this happen with Resin. But ever since upgrading to Resin Pro 3.1.9 (from Pro 3.0.21), it's happening several times per week with different servers during restarts.
Environment: Windows 2003 64-bit Server (SP2), Resin Pro 3.1.9 (100 Server License), JDK 1.6.0_13 (64-bit)
I looked in the 1.6.0_14 release notes and didn't see anything specific to deadlock situations that were fixed in the .14 release. I don't know if this is a problem with our code, or Resin, or the JDK, or some combination thereof.
It might be a classloading-related deadlock. The log is horrible (it's straight JMX output), but I think it has some classloader classes that might be a problem.
Other than the obvious, "Why is this happening and how can I fix it?" question, I'm also curious about the 15 minutes that it always takes resin to notice the deadlock and restart. Is this 15 minutes configurable? If so, where can I adjust it and what are the implications of making it < 15 minutes?
The deadlock check is part of the <ping> functionality. There's an <initial-sleep-time> that defaults to 15 minutes, because the ping needs to wait for the server to start before pinging.
Interesting, so even though I haven't defined a jsp page for the ping to check, it still operates? What is it checking?
In any event, good to know. I'm going to crank this value down since our servers generally get started within about a minute or so. I'll pad it out a little to account for recompiling jsp's and such. 

Rob



I've added a bug report to at least clean up that log report.

-- Scott



Rob
Here's an example of the arguments we pass to Java. Most of our servers are configured similarly with variations in the heap sizes. 

      <jvm-arg>-server</jvm-arg>
      <jvm-arg>-verbose:gc</jvm-arg>
      <jvm-arg>-Xmn500m</jvm-arg>
      <jvm-arg>-Xms5000m</jvm-arg>
      <jvm-arg>-Xmx5000m</jvm-arg>
      <jvm-arg>-Xss128k</jvm-arg>
      <jvm-arg>-Xrs</jvm-arg>
      <jvm-arg>-Xloggc:log/gc.log</jvm-arg>
      <jvm-arg>-Xdebug</jvm-arg>
      <jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg>
      <jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg>
      <jvm-arg>-XX:+PrintGCDetails</jvm-arg>
      <jvm-arg>-Dhttp.maxConnections=400</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.port=9337</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</ jvm-arg> 
      <jvm-arg>-agentlib:resin</jvm-arg>

And here's the portion of the log file from when a deadlock occurs.
RESIN [02:58:38.369] {resin-destroy} Server[id=,cluster=app-tier] stopping [This is when the restart request is sent. --Rob] 
RESIN [02:58:38.416] {http--80-2843$1916052035} WebApp[] active
RESIN [02:59:06.136] {main} Proxy Cache disk-size=1024M memory- size=64M [Resin starts to restart here. --Rob] 
RESIN [02:59:06.168] {main} PingThread[] starting, checking []
RESIN [02:59:06.340] {main}
RESIN [02:59:06.340] {main} Windows 2003 5.2 amd64
RESIN [02:59:06.340] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en RESIN [02:59:06.340] {main} Java HotSpot(TM) 64-Bit Server VM 11.3- b02, 64, mixed mode, Sun Microsystems Inc. 
RESIN [02:59:06.340] {main} user.name: SYSTEM
RESIN [02:59:06.340] {main} resin.home = d:\resin
RESIN [02:59:06.340] {main} resin.root = d:\resin
RESIN [02:59:06.340] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [02:59:06.340] {main}
RESIN [03:14:06.169] {resin-7} JDK detected deadlock. Restarting Resin. [15 minutes later is when Resin has detected the deadlock and restarts. --Rob] RESIN [03:14:06.169] {resin-7} javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .ThreadInfo ,items = ((itemName = blockedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = blockedTime ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = inNative ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = lockInfo ,itemType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer))))), (itemName = lockName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockOwnerId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = lockOwnerName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockedMonitors ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .MonitorInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackDepth ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackFrame ,itemType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)), (itemName = lockedSynchronizers ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Integer)))),primitiveArray=false)), (itemName = stackTrace ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean)))),primitiveArray=false)), (itemName = suspended ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = threadId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = threadName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = threadState ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = waitedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = waitedTime ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Long)))),contents={blockedCount=12, blockedTime=-1, inNative=false, lockInfo = javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType (name = java .lang .Integer )))),contents={className=com.caucho.loader.EnvironmentClassLoader, identityHashCode=324813035}), lockname=com.caucho.loader.environmentclassloa...@135c40eb, lockOwnerId=25, lockOwnerName=resin-8, lockedMonitors =[Ljavax.management.openmbean.CompositeData;@62135133, lockedSynchronizers =[Ljavax.management.openmbean.CompositeData;@62381d92, stackTrace=[Ljavax.management.openmbean.CompositeData;@521d8e9e, suspended=false, threadId=27, threadName=resin-10, threadState=BLOCKED, waitedCount=7, waitedTime=-1}) RESIN [03:14:06.184] {resin-7} javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .ThreadInfo ,items = ((itemName = blockedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = blockedTime ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = inNative ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = lockInfo ,itemType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer))))), (itemName = lockName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockOwnerId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = lockOwnerName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockedMonitors ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .MonitorInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackDepth ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackFrame ,itemType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)), (itemName = lockedSynchronizers ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Integer)))),primitiveArray=false)), (itemName = stackTrace ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean)))),primitiveArray=false)), (itemName = suspended ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = threadId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = threadName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = threadState ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = waitedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = waitedTime ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Long)))),contents={blockedCount=7, blockedTime=-1, inNative=false, lockInfo = javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType (name = java .lang.Integer)))),contents={className=com.caucho.loader.ClassEntry, identityHashCode=587476940}), lockname=com.caucho.loader.classen...@23042fcc, lockOwnerId=27, lockOwnerName=resin-10, lockedMonitors =[Ljavax.management.openmbean.CompositeData;@67de0c09, lockedSynchronizers =[Ljavax.management.openmbean.CompositeData;@22e85825, stackTrace=[Ljavax.management.openmbean.CompositeData;@43f2f70a, suspended=false, threadId=25, threadName=resin-8, threadState=BLOCKED, waitedCount=1, waitedTime=-1}) RESIN [03:14:11.091] {main} Proxy Cache disk-size=1024M memory- size=64M 
RESIN [03:14:11.122] {main} PingThread[] starting, checking []
RESIN [03:14:11.263] {main}
RESIN [03:14:11.263] {main} Windows 2003 5.2 amd64
RESIN [03:14:11.278] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en RESIN [03:14:11.278] {main} Java HotSpot(TM) 64-Bit Server VM 11.3- b02, 64, mixed mode, Sun Microsystems Inc. 
RESIN [03:14:11.278] {main} user.name: SYSTEM
RESIN [03:14:11.278] {main} resin.home = d:\resin
RESIN [03:14:11.278] {main} resin.root = d:\resin
RESIN [03:14:11.278] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [03:14:11.278] {main}

_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest



_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Reply via email to