Re: [Resin-interest] Resin Pro 3.1.9 - JDK Deadlock on Startup

Mon, 22 Jun 2009 20:11:43 -0700 


On Jun 18, 2009, at 19:31, Scott Ferguson wrote:


On Jun 18, 2009, at 5:22 PM, Rob Lockstone wrote:
I've never seen this happen with Resin. But ever since upgrading to Resin Pro 3.1.9 (from Pro 3.0.21), it's happening several times per week with different servers during restarts.
Environment: Windows 2003 64-bit Server (SP2), Resin Pro 3.1.9 (100 Server License), JDK 1.6.0_13 (64-bit)
I looked in the 1.6.0_14 release notes and didn't see anything specific to deadlock situations that were fixed in the .14 release. I don't know if this is a problem with our code, or Resin, or the JDK, or some combination thereof.
It might be a classloading-related deadlock. The log is horrible (it's straight JMX output), but I think it has some classloader classes that might be a problem.
Other than the obvious, "Why is this happening and how can I fix it?" question, I'm also curious about the 15 minutes that it always takes resin to notice the deadlock and restart. Is this 15 minutes configurable? If so, where can I adjust it and what are the implications of making it < 15 minutes?
The deadlock check is part of the <ping> functionality. There's an <initial-sleep-time> that defaults to 15 minutes, because the ping needs to wait for the server to start before pinging.
Fyi, the online documentation for both 3.1 and 4.0 doesn't make mention of initial-sleep-time. In fact they explicitly state, "The first ping is always 15m after the server starts, ..." 

http://caucho.com/resin-3.1/doc/cluster-tags.xtp#ping

http://caucho.com/resin/doc/cluster-tags.xtp#ping
However, setting initial-sleep-time does work. But you do have to specify a URL for it to call. So even though there is a built-in deadlock detection mechanism that kicks in after 15 minutes (what I was seeing below), you can't control that one via the config file. 

Rob



I've added a bug report to at least clean up that log report.

-- Scott



Rob
Here's an example of the arguments we pass to Java. Most of our servers are configured similarly with variations in the heap sizes. 

      <jvm-arg>-server</jvm-arg>
      <jvm-arg>-verbose:gc</jvm-arg>
      <jvm-arg>-Xmn500m</jvm-arg>
      <jvm-arg>-Xms5000m</jvm-arg>
      <jvm-arg>-Xmx5000m</jvm-arg>
      <jvm-arg>-Xss128k</jvm-arg>
      <jvm-arg>-Xrs</jvm-arg>
      <jvm-arg>-Xloggc:log/gc.log</jvm-arg>
      <jvm-arg>-Xdebug</jvm-arg>
      <jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg>
      <jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg>
      <jvm-arg>-XX:+PrintGCDetails</jvm-arg>
      <jvm-arg>-Dhttp.maxConnections=400</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.port=9337</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</ jvm-arg> 
      <jvm-arg>-agentlib:resin</jvm-arg>

And here's the portion of the log file from when a deadlock occurs.
RESIN [02:58:38.369] {resin-destroy} Server[id=,cluster=app-tier] stopping [This is when the restart request is sent. --Rob] 
RESIN [02:58:38.416] {http--80-2843$1916052035} WebApp[] active
RESIN [02:59:06.136] {main} Proxy Cache disk-size=1024M memory- size=64M [Resin starts to restart here. --Rob] 
RESIN [02:59:06.168] {main} PingThread[] starting, checking []
RESIN [02:59:06.340] {main}
RESIN [02:59:06.340] {main} Windows 2003 5.2 amd64
RESIN [02:59:06.340] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en RESIN [02:59:06.340] {main} Java HotSpot(TM) 64-Bit Server VM 11.3- b02, 64, mixed mode, Sun Microsystems Inc. 
RESIN [02:59:06.340] {main} user.name: SYSTEM
RESIN [02:59:06.340] {main} resin.home = d:\resin
RESIN [02:59:06.340] {main} resin.root = d:\resin
RESIN [02:59:06.340] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [02:59:06.340] {main}
RESIN [03:14:06.169] {resin-7} JDK detected deadlock. Restarting Resin. [15 minutes later is when Resin has detected the deadlock and restarts. --Rob] RESIN [03:14:06.169] {resin-7} javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .ThreadInfo ,items = ((itemName = blockedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = blockedTime ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = inNative ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = lockInfo ,itemType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer))))), (itemName = lockName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockOwnerId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = lockOwnerName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockedMonitors ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .MonitorInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackDepth ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackFrame ,itemType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)), (itemName = lockedSynchronizers ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Integer)))),primitiveArray=false)), (itemName = stackTrace ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean)))),primitiveArray=false)), (itemName = suspended ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = threadId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = threadName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = threadState ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = waitedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = waitedTime ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Long)))),contents={blockedCount=12, blockedTime=-1, inNative=false, lockInfo = javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType (name = java .lang .Integer )))),contents={className=com.caucho.loader.EnvironmentClassLoader, identityHashCode=324813035}), lockname=com.caucho.loader.environmentclassloa...@135c40eb, lockOwnerId=25, lockOwnerName=resin-8, lockedMonitors =[Ljavax.management.openmbean.CompositeData;@62135133, lockedSynchronizers =[Ljavax.management.openmbean.CompositeData;@62381d92, stackTrace=[Ljavax.management.openmbean.CompositeData;@521d8e9e, suspended=false, threadId=27, threadName=resin-10, threadState=BLOCKED, waitedCount=7, waitedTime=-1}) RESIN [03:14:06.184] {resin-7} javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .ThreadInfo ,items = ((itemName = blockedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = blockedTime ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = inNative ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = lockInfo ,itemType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer))))), (itemName = lockName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockOwnerId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = lockOwnerName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lockedMonitors ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .MonitorInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackDepth ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = lockedStackFrame ,itemType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)), (itemName = lockedSynchronizers ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Integer)))),primitiveArray=false)), (itemName = stackTrace ,itemType = javax .management .openmbean .ArrayType (name = [Ljavax .management .openmbean .CompositeData ;,dimension = 1 ,elementType = javax .management .openmbean .CompositeType (name = java .lang .StackTraceElement ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = fileName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = lineNumber ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Integer)), (itemName = methodName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = nativeMethod ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Boolean)))),primitiveArray=false)), (itemName = suspended ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Boolean)), (itemName = threadId ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = threadName ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = threadState ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = waitedCount ,itemType =javax.management.openmbean.SimpleType(name=java.lang.Long)), (itemName = waitedTime ,itemType = javax .management .openmbean .SimpleType(name=java.lang.Long)))),contents={blockedCount=7, blockedTime=-1, inNative=false, lockInfo = javax .management .openmbean .CompositeDataSupport (compositeType = javax .management .openmbean .CompositeType (name = java .lang .management .LockInfo ,items = ((itemName = className ,itemType =javax.management.openmbean.SimpleType(name=java.lang.String)), (itemName = identityHashCode ,itemType = javax .management .openmbean .SimpleType (name = java .lang.Integer)))),contents={className=com.caucho.loader.ClassEntry, identityHashCode=587476940}), lockname=com.caucho.loader.classen...@23042fcc, lockOwnerId=27, lockOwnerName=resin-10, lockedMonitors =[Ljavax.management.openmbean.CompositeData;@67de0c09, lockedSynchronizers =[Ljavax.management.openmbean.CompositeData;@22e85825, stackTrace=[Ljavax.management.openmbean.CompositeData;@43f2f70a, suspended=false, threadId=25, threadName=resin-8, threadState=BLOCKED, waitedCount=1, waitedTime=-1}) RESIN [03:14:11.091] {main} Proxy Cache disk-size=1024M memory- size=64M 
RESIN [03:14:11.122] {main} PingThread[] starting, checking []
RESIN [03:14:11.263] {main}
RESIN [03:14:11.263] {main} Windows 2003 5.2 amd64
RESIN [03:14:11.278] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en RESIN [03:14:11.278] {main} Java HotSpot(TM) 64-Bit Server VM 11.3- b02, 64, mixed mode, Sun Microsystems Inc. 
RESIN [03:14:11.278] {main} user.name: SYSTEM
RESIN [03:14:11.278] {main} resin.home = d:\resin
RESIN [03:14:11.278] {main} resin.root = d:\resin
RESIN [03:14:11.278] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [03:14:11.278] {main}



_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Reply via email to