Hi,
Found one Java-level deadlock:
=============================
"SocketListener0-7":
waiting to lock monitor 0x0845e1fc (object 0x54f95838, a
org.apache.hadoop.mapred.JobTracker),
which is held by "IPC Server handler 0 on 54311"
"IPC Server handler 0 on 54311":
waiting to lock monitor 0x4d671064 (object 0x57250a60, a
org.apache.hadoop.mapred.JobInProgress),
which is held by "initJobs"
"initJobs":
waiting to lock monitor 0x0845e1fc (object 0x54f95838, a
org.apache.hadoop.mapred.JobTracker),
which is held by "IPC Server handler 0 on 54311"
Java stack information for the threads listed above:
===================================================
"SocketListener0-7":
at
org.apache.hadoop.mapred.JobTracker.getClusterStatus(JobTracker.java:1826)
- waiting to lock <0x54f95838> (a
org.apache.hadoop.mapred.JobTracker)
at
org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:135)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
"IPC Server handler 0 on 54311":
at
org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:1451)
- waiting to lock <0x57250a60> (a
org.apache.hadoop.mapred.JobInProgress)
at org.apache.hadoop.mapred.JobTracker.killJob(JobTracker.java:1843)
- locked <0x54f95838> (a org.apache.hadoop.mapred.JobTracker)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
"initJobs":
at
org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:1015)
- waiting to lock <0x54f95838> (a
org.apache.hadoop.mapred.JobTracker)
at
org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:1656)
- locked <0x57250a60> (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:1469)
- locked <0x57250a60> (a org.apache.hadoop.mapred.JobInProgress)
at
org.apache.hadoop.mapred.JobTracker$JobInitThread.run(JobTracker.java:416)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
I found this condition. I will try to work on this
-Sagar
