Re: [Resin-interest] Resin Pro 3.1.9 - JDK Deadlock on Startup

Thu, 18 Jun 2009 19:32:20 -0700 


On Jun 18, 2009, at 5:22 PM, Rob Lockstone wrote:
I've never seen this happen with Resin. But ever since upgrading to Resin Pro 3.1.9 (from Pro 3.0.21), it's happening several times per week with different servers during restarts.
Environment: Windows 2003 64-bit Server (SP2), Resin Pro 3.1.9 (100 Server License), JDK 1.6.0_13 (64-bit)
I looked in the 1.6.0_14 release notes and didn't see anything specific to deadlock situations that were fixed in the .14 release. I don't know if this is a problem with our code, or Resin, or the JDK, or some combination thereof.
It might be a classloading-related deadlock. The log is horrible (it's straight JMX output), but I think it has some classloader classes that might be a problem.
Other than the obvious, "Why is this happening and how can I fix it?" question, I'm also curious about the 15 minutes that it always takes resin to notice the deadlock and restart. Is this 15 minutes configurable? If so, where can I adjust it and what are the implications of making it < 15 minutes?
The deadlock check is part of the <ping> functionality. There's an <initial-sleep-time> that defaults to 15 minutes, because the ping needs to wait for the server to start before pinging. 

I've added a bug report to at least clean up that log report.

-- Scott



Rob
Here's an example of the arguments we pass to Java. Most of our servers are configured similarly with variations in the heap sizes. 

      <jvm-arg>-server</jvm-arg>
      <jvm-arg>-verbose:gc</jvm-arg>
      <jvm-arg>-Xmn500m</jvm-arg>
      <jvm-arg>-Xms5000m</jvm-arg>
      <jvm-arg>-Xmx5000m</jvm-arg>
      <jvm-arg>-Xss128k</jvm-arg>
      <jvm-arg>-Xrs</jvm-arg>
      <jvm-arg>-Xloggc:log/gc.log</jvm-arg>
      <jvm-arg>-Xdebug</jvm-arg>
      <jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg>
      <jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg>
      <jvm-arg>-XX:+PrintGCDetails</jvm-arg>
      <jvm-arg>-Dhttp.maxConnections=400</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.port=9337</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</ jvm-arg> 
      <jvm-arg>-agentlib:resin</jvm-arg>

And here's the portion of the log file from when a deadlock occurs.
RESIN [02:58:38.369] {resin-destroy} Server[id=,cluster=app-tier] stopping [This is when the restart request is sent. --Rob] 
RESIN [02:58:38.416] {http--80-2843$1916052035} WebApp[] active
RESIN [02:59:06.136] {main} Proxy Cache disk-size=1024M memory- size=64M [Resin starts to restart here. --Rob] 
RESIN [02:59:06.168] {main} PingThread[] starting, checking []
RESIN [02:59:06.340] {main}
RESIN [02:59:06.340] {main} Windows 2003 5.2 amd64
RESIN [02:59:06.340] {main} Java(TM) SE Runtime Environment 1.6.0_13- b03, Cp1252, en RESIN [02:59:06.340] {main} Java HotSpot(TM) 64-Bit Server VM 11.3- b02, 64, mixed mode, Sun Microsystems Inc. 
RESIN [02:59:06.340] {main} user.name: SYSTEM
RESIN [02:59:06.340] {main} resin.home = d:\resin
RESIN [02:59:06.340] {main} resin.root = d:\resin
RESIN [02:59:06.340] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [02:59:06.340] {main}
RESIN [03:14:06.169] {resin-7} JDK detected deadlock. Restarting Resin. [15 minutes later is when Resin has detected the deadlock and restarts. --Rob] RESIN [03:14:06.169] {resin-7} javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .ThreadInfo ,items = ((itemName = blockedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = blockedTime ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = inNative ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = lockInfo ,itemType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer))))), (itemName = lockName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockOwnerId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = lockOwnerName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockedMonitors ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .MonitorInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackDepth ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackFrame ,itemType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)), (itemName = lockedSynchronizers ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Integer)))),primitiveArray=false)), (itemName = stackTrace ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean)))),primitiveArray=false)), (itemName = suspended ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = threadId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = threadName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = threadState ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = waitedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = waitedTime ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Long)))),contents={blockedCount=12, blockedTime=-1, inNative=false, lockInfo = javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType (name = java .lang .Integer )))),contents={className=com.caucho.loader.EnvironmentClassLoader, identityHashCode=324813035}), lockname=com.caucho.loader.environmentclassloa...@135c40eb, lockOwnerId=25, lockOwnerName=resin-8, lockedMonitors=[Ljavax.management.openmbean.CompositeData;@62135133, lockedSynchronizers =[Ljavax.management.openmbean.CompositeData;@62381d92, stackTrace=[Ljavax.management.openmbean.CompositeData;@521d8e9e, suspended=false, threadId=27, threadName=resin-10, threadState=BLOCKED, waitedCount=7, waitedTime=-1}) RESIN [03:14:06.184] {resin-7} javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .ThreadInfo ,items = ((itemName = blockedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = blockedTime ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = inNative ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = lockInfo ,itemType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer))))), (itemName = lockName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockOwnerId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = lockOwnerName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockedMonitors ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .MonitorInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackDepth ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackFrame ,itemType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)), (itemName = lockedSynchronizers ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Integer)))),primitiveArray=false)), (itemName = stackTrace ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean)))),primitiveArray=false)), (itemName = suspended ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = threadId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = threadName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = threadState ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = waitedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = waitedTime ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Long)))),contents={blockedCount=7, blockedTime=-1, inNative=false, lockInfo = javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType (name = java .lang.Integer)))),contents={className=com.caucho.loader.ClassEntry, identityHashCode=587476940}), lockname=com.caucho.loader.classen...@23042fcc, lockOwnerId=27, lockOwnerName=resin-10, lockedMonitors=[Ljavax.management.openmbean.CompositeData;@67de0c09, lockedSynchronizers =[Ljavax.management.openmbean.CompositeData;@22e85825, stackTrace=[Ljavax.management.openmbean.CompositeData;@43f2f70a, suspended=false, threadId=25, threadName=resin-8, threadState=BLOCKED, waitedCount=1, waitedTime=-1}) RESIN [03:14:11.091] {main} Proxy Cache disk-size=1024M memory- size=64M 
RESIN [03:14:11.122] {main} PingThread[] starting, checking []
RESIN [03:14:11.263] {main}
RESIN [03:14:11.263] {main} Windows 2003 5.2 amd64
RESIN [03:14:11.278] {main} Java(TM) SE Runtime Environment 1.6.0_13- b03, Cp1252, en RESIN [03:14:11.278] {main} Java HotSpot(TM) 64-Bit Server VM 11.3- b02, 64, mixed mode, Sun Microsystems Inc. 
RESIN [03:14:11.278] {main} user.name: SYSTEM
RESIN [03:14:11.278] {main} resin.home = d:\resin
RESIN [03:14:11.278] {main} resin.root = d:\resin
RESIN [03:14:11.278] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [03:14:11.278] {main}

_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest



_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Reply via email to