Thanks Ted, I'll try to do a major compact. Hi Steven,
Yes, most of my rows are hashed to make it randomly distributed, but one column family has monotonically increasing rowkeys, and it's used for recording sequence of events. Do you have a solution how to bulk import this kind of data? Jianshi On Sat, Sep 6, 2014 at 12:00 AM, Magana-zook, Steven Alan < maganazo...@llnl.gov> wrote: > Hi Jianshi, > > What are the field(s) in your row key? If your row key is monotonically > increasing then you will be sending all of your requests to one region > server. Even after the region splits, all new entries will keep punishing > one server (the region responsible for the split containing the new keys). > > See these articles that may help if this is indeed your issue: > 1. http://hbase.apache.org/book/rowkey.design.html > 2. > http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-incre > asing-values-are-bad/ > > Regards, > Steven Magana-Zook > > > > > > > On 9/5/14 8:54 AM, "Jianshi Huang" <jianshi.hu...@gmail.com> wrote: > > >Hi JM, > > > >What do you mean by the 'destination cluster'? The files are in the same > >Hadoop/HDFS cluster where HBase is running. > > > >Do you mean do the bulk importing on HBase Master node? > > > > > >Jianshi > > > > > >On Fri, Sep 5, 2014 at 11:18 PM, Jean-Marc Spaggiari < > >jean-m...@spaggiari.org> wrote: > > > >> Hi Jianshi, > >> > >> You might want to upload the file on the destination cluster first and > >>then > >> re-run your bulk load from there. That way the transfer time will not be > >> taken into consideration for the timeout size the files will be local. > >> > >> JM > >> > >> > >> 2014-09-05 11:15 GMT-04:00 Jianshi Huang <jianshi.hu...@gmail.com>: > >> > >> > I'm importing 2TB of generated HFiles to HBase and I constantly get > >>the > >> > following errors: > >> > > >> > Caused by: > >> > > >> > > >> > >>org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.h > >>base.RegionTooBusyException): > >> > org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock > >>in > >> > 60000 ms. > >> > > >> > > >> > >>regionName=grapple_edges_v2,ff000000,1409817320781.6d2955c780b39523de733f > >>3565642d96., > >> > server=xxxxx.xxx.xxx,60020,1404854700728 > >> > at > >> > org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5851) > >> > at > >> > org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5837) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.regionserver.HRegion.startBulkRegionOperation(HRe > >>gion.java:5795) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java: > >>3543) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java: > >>3525) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionS > >>erver.java:3277) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.c > >>allBlockingMethod(ClientProtos.java:28863) > >> > at > >> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008) > >> > at > >>org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcSche > >>duler.java:160) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcSchedu > >>ler.java:38) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.j > >>ava:110) > >> > at java.lang.Thread.run(Thread.java:724) > >> > > >> > at > >> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1498) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1 > >>684) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.ca > >>llBlockingMethod(RpcClient.java:1737) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blo > >>ckingStub.bulkLoadHFile(ClientProtos.java:29276) > >> > at > >> > > >> > > >> > >>org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil. > >>java:1548) > >> > ... 11 more > >> > > >> > > >> > What makes the region too busy? Is there a way to improve it? > >> > > >> > Does that also mean some part of my data are not correctly imported? > >> > > >> > > >> > Thanks, > >> > > >> > -- > >> > Jianshi Huang > >> > > >> > LinkedIn: jianshi > >> > Twitter: @jshuang > >> > Github & Blog: http://huangjs.github.com/ > >> > > >> > > > > > > > >-- > >Jianshi Huang > > > >LinkedIn: jianshi > >Twitter: @jshuang > >Github & Blog: http://huangjs.github.com/ > > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
