I've never seen this happen with Resin. But ever since upgrading to
Resin Pro 3.1.9 (from Pro 3.0.21), it's happening several times per
week with different servers during restarts.
Environment: Windows 2003 64-bit Server (SP2), Resin Pro 3.1.9 (100
Server License), JDK 1.6.0_13 (64-bit)
I looked in the 1.6.0_14 release notes and didn't see anything
specific to deadlock situations that were fixed in the .14 release. I
don't know if this is a problem with our code, or Resin, or the JDK,
or some combination thereof.
Other than the obvious, "Why is this happening and how can I fix it?"
question, I'm also curious about the 15 minutes that it always takes
resin to notice the deadlock and restart. Is this 15 minutes
configurable? If so, where can I adjust it and what are the
implications of making it < 15 minutes?
Rob
Here's an example of the arguments we pass to Java. Most of our
servers are configured similarly with variations in the heap sizes.
<jvm-arg>-server</jvm-arg>
<jvm-arg>-verbose:gc</jvm-arg>
<jvm-arg>-Xmn500m</jvm-arg>
<jvm-arg>-Xms5000m</jvm-arg>
<jvm-arg>-Xmx5000m</jvm-arg>
<jvm-arg>-Xss128k</jvm-arg>
<jvm-arg>-Xrs</jvm-arg>
<jvm-arg>-Xloggc:log/gc.log</jvm-arg>
<jvm-arg>-Xdebug</jvm-arg>
<jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg>
<jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg>
<jvm-arg>-XX:+PrintGCDetails</jvm-arg>
<jvm-arg>-Dhttp.maxConnections=400</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote.port=9337</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
<jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</jvm-
arg>
<jvm-arg>-agentlib:resin</jvm-arg>
And here's the portion of the log file from when a deadlock occurs.
RESIN [02:58:38.369] {resin-destroy} Server[id=,cluster=app-tier]
stopping [This is when the restart request is sent. --Rob]
RESIN [02:58:38.416] {http--80-2843$1916052035} WebApp[] active
RESIN [02:59:06.136] {main} Proxy Cache disk-size=1024M memory-
size=64M [Resin starts to restart here. --Rob]
RESIN [02:59:06.168] {main} PingThread[] starting, checking []
RESIN [02:59:06.340] {main}
RESIN [02:59:06.340] {main} Windows 2003 5.2 amd64
RESIN [02:59:06.340] {main} Java(TM) SE Runtime Environment 1.6.0_13-
b03, Cp1252, en
RESIN [02:59:06.340] {main} Java HotSpot(TM) 64-Bit Server VM 11.3-
b02, 64, mixed mode, Sun Microsystems Inc.
RESIN [02:59:06.340] {main} user.name: SYSTEM
RESIN [02:59:06.340] {main} resin.home = d:\resin
RESIN [02:59:06.340] {main} resin.root = d:\resin
RESIN [02:59:06.340] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [02:59:06.340] {main}
RESIN [03:14:06.169] {resin-7} JDK detected deadlock. Restarting
Resin. [15 minutes later is when Resin has detected the deadlock and
restarts. --Rob]
RESIN [03:14:06.169] {resin-7}
javax
.management
.openmbean
.CompositeDataSupport
(compositeType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.ThreadInfo
,items
=
((itemName
=
blockedCount
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
blockedTime
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
inNative
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),
(itemName
=
lockInfo
,itemType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.LockInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer))))),
(itemName
=
lockName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lockOwnerId
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
lockOwnerName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lockedMonitors
,itemType
=
javax
.management
.openmbean
.ArrayType
(name
=
[Ljavax
.management
.openmbean
.CompositeData
;,dimension
=
1
,elementType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.MonitorInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
lockedStackDepth
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
lockedStackFrame
,itemType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.StackTraceElement
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
fileName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lineNumber
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
methodName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
nativeMethod
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)),
(itemName
=
lockedSynchronizers
,itemType
=
javax
.management
.openmbean
.ArrayType
(name
=
[Ljavax
.management
.openmbean
.CompositeData
;,dimension
=
1
,elementType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.LockInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Integer)))),primitiveArray=false)),
(itemName
=
stackTrace
,itemType
=
javax
.management
.openmbean
.ArrayType
(name
=
[Ljavax
.management
.openmbean
.CompositeData
;,dimension
=
1
,elementType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.StackTraceElement
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
fileName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lineNumber
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
methodName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
nativeMethod
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Boolean)))),primitiveArray=false)),
(itemName
=
suspended
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),
(itemName
=
threadId
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
threadName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
threadState
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
waitedCount
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
waitedTime
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Long)))),contents={blockedCount=12,
blockedTime=-1, inNative=false,
lockInfo
=
javax
.management
.openmbean
.CompositeDataSupport
(compositeType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.LockInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=
javax
.management
.openmbean
.SimpleType
(name
=
java
.lang
.Integer
)))),contents={className=com.caucho.loader.EnvironmentClassLoader,
identityHashCode=324813035}),
lockname=com.caucho.loader.environmentclassloa...@135c40eb,
lockOwnerId=25, lockOwnerName=resin-8,
lockedMonitors=[Ljavax.management.openmbean.CompositeData;@62135133,
lockedSynchronizers
=[Ljavax.management.openmbean.CompositeData;@62381d92,
stackTrace=[Ljavax.management.openmbean.CompositeData;@521d8e9e,
suspended=false, threadId=27, threadName=resin-10,
threadState=BLOCKED, waitedCount=7, waitedTime=-1})
RESIN [03:14:06.184] {resin-7}
javax
.management
.openmbean
.CompositeDataSupport
(compositeType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.ThreadInfo
,items
=
((itemName
=
blockedCount
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
blockedTime
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
inNative
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),
(itemName
=
lockInfo
,itemType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.LockInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer))))),
(itemName
=
lockName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lockOwnerId
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
lockOwnerName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lockedMonitors
,itemType
=
javax
.management
.openmbean
.ArrayType
(name
=
[Ljavax
.management
.openmbean
.CompositeData
;,dimension
=
1
,elementType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.MonitorInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
lockedStackDepth
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
lockedStackFrame
,itemType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.StackTraceElement
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
fileName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lineNumber
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
methodName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
nativeMethod
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)),
(itemName
=
lockedSynchronizers
,itemType
=
javax
.management
.openmbean
.ArrayType
(name
=
[Ljavax
.management
.openmbean
.CompositeData
;,dimension
=
1
,elementType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.LockInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Integer)))),primitiveArray=false)),
(itemName
=
stackTrace
,itemType
=
javax
.management
.openmbean
.ArrayType
(name
=
[Ljavax
.management
.openmbean
.CompositeData
;,dimension
=
1
,elementType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.StackTraceElement
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
fileName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
lineNumber
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Integer)),
(itemName
=
methodName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
nativeMethod
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Boolean)))),primitiveArray=false)),
(itemName
=
suspended
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),
(itemName
=
threadId
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
threadName
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
threadState
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
waitedCount
,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName
=
waitedTime
,itemType
=
javax
.management
.openmbean
.SimpleType(name=java.lang.Long)))),contents={blockedCount=7,
blockedTime=-1, inNative=false,
lockInfo
=
javax
.management
.openmbean
.CompositeDataSupport
(compositeType
=
javax
.management
.openmbean
.CompositeType
(name
=
java
.lang
.management
.LockInfo
,items
=
((itemName
=
className
,itemType
=javax.management.openmbean.SimpleType(name=java.lang.String)),
(itemName
=
identityHashCode
,itemType
=
javax
.management
.openmbean
.SimpleType
(name
=
java
.lang.Integer)))),contents={className=com.caucho.loader.ClassEntry,
identityHashCode=587476940}),
lockname=com.caucho.loader.classen...@23042fcc, lockOwnerId=27,
lockOwnerName=resin-10,
lockedMonitors=[Ljavax.management.openmbean.CompositeData;@67de0c09,
lockedSynchronizers
=[Ljavax.management.openmbean.CompositeData;@22e85825,
stackTrace=[Ljavax.management.openmbean.CompositeData;@43f2f70a,
suspended=false, threadId=25, threadName=resin-8, threadState=BLOCKED,
waitedCount=1, waitedTime=-1})
RESIN [03:14:11.091] {main} Proxy Cache disk-size=1024M memory-size=64M
RESIN [03:14:11.122] {main} PingThread[] starting, checking []
RESIN [03:14:11.263] {main}
RESIN [03:14:11.263] {main} Windows 2003 5.2 amd64
RESIN [03:14:11.278] {main} Java(TM) SE Runtime Environment 1.6.0_13-
b03, Cp1252, en
RESIN [03:14:11.278] {main} Java HotSpot(TM) 64-Bit Server VM 11.3-
b02, 64, mixed mode, Sun Microsystems Inc.
RESIN [03:14:11.278] {main} user.name: SYSTEM
RESIN [03:14:11.278] {main} resin.home = d:\resin
RESIN [03:14:11.278] {main} resin.root = d:\resin
RESIN [03:14:11.278] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [03:14:11.278] {main}
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest