Re: hbase compaction stuck

2015-03-16 Thread Nicolas Liochon
If the node is dead, the NoRouteToHostException can happen. This could be a hdfs or hbase bug (or something else). For how long to you see the NoRouteToHostException exception? Basically hbase will try to use that node under hdfs discovers that the node is stale or dead. With the default hdfs set

Re: hbase compaction stuck

2015-03-16 Thread Stack
Your networking is broken. Fix the 'java.net.NoRouteToHostException: No route to host' exceptions then come back to this list if still issues. Yours, St.Ack On Mon, Mar 16, 2015 at 7:54 AM, Chen Song wrote: > We ran a hbase cluser with version 0.98.1+cdh5.1.0 and with auto > compaction. I have n

Re: hbase compaction stuck

2015-03-16 Thread Ted Yu
bq. Some server in the cluster is hard dead How many servers were down in the cluster ? What hadoop release are you using ? What's the replication factor for hdfs ? Thanks On Mon, Mar 16, 2015 at 7:54 AM, Chen Song wrote: > We ran a hbase cluser with version 0.98.1+cdh5.1.0 and with auto > co

hbase compaction stuck

2015-03-16 Thread Chen Song
We ran a hbase cluser with version 0.98.1+cdh5.1.0 and with auto compaction. I have noticed a few times that compaction stuck under the following circumstances. 1. Some server in the cluster is hard dead and physical down. 2. At the same time, if any region servers are running major compaction and