Hi, hbase experts:
You probably have seen this in the past. Can someone share some quick
ideas?
I have a MR job that is loading data into hbase using hbase client API.
The job failed after a while. Below is the error info and stack trace.
2014-02-04 14:15:32,587 WARN org.apache.hadoop.hbase.client.AsyncProcess:
Attempt #6/35 failed for 182 ops on hdtest203.svl.ibm.com,60020,1391489734651
NOT resubmitting.
region=tpch_hb_1000.lineitem,\x01\x8Ao\x83\xF0\x01\x80'`\x04\x01\x80\x00\x00\x00\x02ufc\x01\x80\x00\x00\x01,1391544629530.b4a41acd34723629c417571b524b80ab.,
hostname=hdtest203.svl.ibm.com,60020,1391489734651, seqNum=682710
2014-02-04 14:19:12,987 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=302, tasksDone=301, currentTasksDone=301,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:19:32,452 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=306, tasksDone=305, currentTasksDone=305,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:19:42,544 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=307, tasksDone=306, currentTasksDone=306,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:19:52,624 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=308, tasksDone=307, currentTasksDone=307,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:20:03,044 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=309, tasksDone=308, currentTasksDone=308,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:20:13,133 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=310, tasksDone=309, currentTasksDone=309,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:20:23,240 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=311, tasksDone=310, currentTasksDone=310,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:20:33,328 INFO org.apache.hadoop.hbase.client.AsyncProcess: :
Waiting for the global number of running tasks to be equals or less than 0,
tasksSent=312, tasksDone=311, currentTasksDone=311,
tableName=tpch_hb_1000.lineitem
2014-02-04 14:20:38,444 WARN org.apache.hadoop.hbase.client.AsyncProcess:
Attempt #14/35 failed for 837 ops on hdtest203.svl.ibm.com,60020,1391489734651
NOT resubmitting.
region=tpch_hb_1000.lineitem,\x01\x8Ao\x83\xF0\x01\x80'`\x04\x01\x80\x00\x00\x00\x02ufc\x01\x80\x00\x00\x01,1391544629530.b4a41acd34723629c417571b524b80ab.,
hostname=hdtest203.svl.ibm.com,60020,1391489734651, seqNum=682710
2014-02-04 14:20:38,452 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2014-02-04 14:20:38,472 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hive (auth:SIMPLE)
cause:org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 837 actions: RegionTooBusyException: 837 times,
2014-02-04 14:20:38,473 WARN org.apache.hadoop.mapred.Child: Error running
child
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
837 actions: RegionTooBusyException: 837 times,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:185)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:169)
at
org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:782)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:934)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1193)
at
com.ibm.jaql.module.hbase.HBaseRecordWriter.close(HBaseRecordWriter.java:42)
at
com.ibm.jaql.io.hadoop.CompositeOutputAdapter$1.close(CompositeOutputAdapter.java:383)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:806)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:439)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at
java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2014-02-04 14:20:38,477 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
