Re: 100% CPU utilization on idle HDFS Data Nodes

2014-06-03 Thread Shayan Pooya
1. The jstack result is attached. 2. These are the two processes: 25050 shayan 20 0 1643M 87872 2512 S 100. 10.11434h /usr/lib/jvm/java-1.6.0//bin/java -Dproc_datanode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/shayan/hadoop/logs -Dhadoo 25094 shayan 20 0 1643

Re: 100% CPU utilization on idle HDFS Data Nodes

2014-06-03 Thread Ian Brooks
Hi Shayan, If you restart one of the datanodes, does that node go back to normal cpu usage? if so that looks like the same issue im seeing on my nodes, though mine will go to 200% over time on a 4 cpu host. I havent been able to track the cause down yet. Heavy use of HDFS will cause the node t

Re: 100% CPU utilization on idle HDFS Data Nodes

2014-06-03 Thread Ted Yu
Can you pastebin data node log snippet and jstack of datanode process ? Thanks On Tue, Jun 3, 2014 at 9:34 AM, Shayan Pooya wrote: > I have a three node HDFS cluster with a name-node. There is absolutely no > IO going on this cluster or any jobs running on it and I just use it for > testing t

100% CPU utilization on idle HDFS Data Nodes

2014-06-03 Thread Shayan Pooya
I have a three node HDFS cluster with a name-node. There is absolutely no IO going on this cluster or any jobs running on it and I just use it for testing the Disco HDFS integration. I noticed that two of the three data-nodes are using 100% CPU. They have been running for a long time (2 months) b