this is because the table region changes, not match with the regions when you get the HFiles if the bulkload process is over, the files should be moved to hbase
i think it is better to delete all hfiles and dirs when the bulkload over. At 2015-12-23 16:35:10, "qihuang.zheng" <qihuang.zh...@fraudmetrix.cn> wrote: >I Have a HFile generate by importtsv, the file is really large, from 100mb to >10G. >I have changed hbase.hregion.max.filesize to 50GB(53687091200). also specify >src CanonicalServiceName same with hbase. >hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1 data.md5_id2 HADOOP_CLASSPATH=`hbase >classpath` hadoop jar hbase-1.0.2/lib/hbase-server-1.0.2.jar completebulkload >/user/tongdun/id_hbase/1 data.md5_id2 >But both completebulkload and LoadIncrementalHFiles did't just mv/rename hfile >expected. but instead copy and split hfile happening, which take long time. >the logSplit occured while grouping HFiles, retry attempt XXXwill create child >_tmp dir one by one level. >2015-12-23 15:52:04,909 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig: >CacheConfig:disabled 2015-12-23 15:52:05,006 INFO [LoadIncrementalHFiles-0] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/01114a58782b4c369819673e4b3678ae > first=f6eb30074a52ebb8c5f52ed1c85c2f0d last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:52:05,007 INFO [LoadIncrementalHFiles-0] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/01114a58782b4c369819673e4b3678ae no >longer fits inside a single region. Splitting... 2015-12-23 15:53:38,639 INFO >[LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles: Successfully split >into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/9f6fe2d28ddc4f209be62757ace8611b.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/9f6fe2d28ddc4f209be62757ace8611b.top > 2015-12-23 15:53:39,173 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 1 with 2 files remaining to group >or split 2015-12-23 15:53:39,186 INFO [LoadIncrementalHFiles-1] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/9f6fe2d28ddc4f209be62757ace8611b.bottom > first=f6eb30074a52ebb8c5f52ed1c85c2f0d last=f733d2c504f22f71b191014d72e4d124 >2015-12-23 15:53:39,188 INFO [LoadIncrementalHFiles-2] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/9f6fe2d28ddc4f209be62757ace8611b.top > first=f733d2c6407f5758e860195b6d2c10c1 last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:53:39,189 INFO [LoadIncrementalHFiles-2] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/9f6fe2d28ddc4f209be62757ace8611b.top > no longer fits inside a single region. Splitting... 2015-12-23 15:54:27,722 >INFO [LoadIncrementalHFiles-2] mapreduce.LoadIncrementalHFiles: Successfully >split into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/17ba0f42c4934f4c96218c784d3c3bb0.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/17ba0f42c4934f4c96218c784d3c3bb0.top > 2015-12-23 15:54:28,557 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 2 with 2 files remaining to group >or split 2015-12-23 15:54:28,568 INFO [LoadIncrementalHFiles-4] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/17ba0f42c4934f4c96218c784d3c3bb0.bottom > first=f733d2c6407f5758e860195b6d2c10c1 last=f77c7d357a76ff92bb16ec1ef79f31fb >2015-12-23 15:54:28,568 INFO [LoadIncrementalHFiles-5] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/17ba0f42c4934f4c96218c784d3c3bb0.top > first=f77c7d3915c9a8b71c83c414aabd587d last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:54:28,568 INFO [LoadIncrementalHFiles-5] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/17ba0f42c4934f4c96218c784d3c3bb0.top > no longer fits inside a single region. Splitting... 2015-12-23 15:55:08,992 >INFO [LoadIncrementalHFiles-5] mapreduce.LoadIncrementalHFiles: Successfully >split into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/f7162cec4e404eabbea479b2a5446294.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/f7162cec4e404eabbea479b2a5446294.top > 2015-12-23 15:55:09,424 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 3 with 2 files remaining to group >or split 2015-12-23 15:55:09,431 INFO [LoadIncrementalHFiles-7] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/f7162cec4e404eabbea479b2a5446294.bottom > first=f77c7d3915c9a8b71c83c414aabd587d last=f7c525a83ee19ea166414e972c5d5541 >2015-12-23 15:55:09,433 INFO [LoadIncrementalHFiles-8] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/f7162cec4e404eabbea479b2a5446294.top > first=f7c525aa2ec661c1c0707b02d1c4b4b3 last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:55:09,433 INFO [LoadIncrementalHFiles-8] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/f7162cec4e404eabbea479b2a5446294.top > no longer fits inside a single region. Splitting... 2015-12-23 15:55:42,165 >INFO [LoadIncrementalHFiles-8] mapreduce.LoadIncrementalHFiles: Successfully >split into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/6610bd5d178e423fbe02db1865f834f0.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/6610bd5d178e423fbe02db1865f834f0.top > 2015-12-23 15:55:42,490 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 4 with 2 files remaining to group >or split 2015-12-23 15:55:42,498 INFO [LoadIncrementalHFiles-10] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/6610bd5d178e423fbe02db1865f834f0.bottom > first=f7c525aa2ec661c1c0707b02d1c4b4b3 last=f80dcce8a4a14be406ddd1bdebc2eda2 >2015-12-23 15:55:42,502 INFO [LoadIncrementalHFiles-11] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/6610bd5d178e423fbe02db1865f834f0.top > first=f80dccecf159d4999cb8e17446103d72 last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:55:42,502 INFO [LoadIncrementalHFiles-11] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/6610bd5d178e423fbe02db1865f834f0.top > no longer fits inside a single region. Splitting... 2015-12-23 15:56:09,560 >INFO [LoadIncrementalHFiles-11] mapreduce.LoadIncrementalHFiles: Successfully >split into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/8f07441d8b7c4d3ba37b6b0917860f68.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/8f07441d8b7c4d3ba37b6b0917860f68.top > 2015-12-23 15:56:09,933 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 5 with 2 files remaining to group >or split 2015-12-23 15:56:09,942 INFO [LoadIncrementalHFiles-13] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/8f07441d8b7c4d3ba37b6b0917860f68.bottom > first=f80dccecf159d4999cb8e17446103d72 last=f85673f473ead63c89e96c83b2058ca7 >2015-12-23 15:56:09,943 INFO [LoadIncrementalHFiles-14] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/8f07441d8b7c4d3ba37b6b0917860f68.top > first=f85673fde3138dac07ce08881c9d0ccc last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:56:09,944 INFO [LoadIncrementalHFiles-14] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/8f07441d8b7c4d3ba37b6b0917860f68.top > no longer fits inside a single region. Splitting... 2015-12-23 15:56:30,890 >INFO [LoadIncrementalHFiles-14] mapreduce.LoadIncrementalHFiles: Successfully >split into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/feaa0a6428f24a5294c87dd87c6bc5a6.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/feaa0a6428f24a5294c87dd87c6bc5a6.top > 2015-12-23 15:56:31,145 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 6 with 2 files remaining to group >or split 2015-12-23 15:56:31,151 INFO [LoadIncrementalHFiles-16] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/feaa0a6428f24a5294c87dd87c6bc5a6.bottom > first=f85673fde3138dac07ce08881c9d0ccc last=f89f12a56b5af206188639f736877563 >2015-12-23 15:56:31,151 INFO [LoadIncrementalHFiles-17] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/feaa0a6428f24a5294c87dd87c6bc5a6.top > first=f89f12a59e4a9c9bcbb42d0504318e25 last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:56:31,151 INFO [LoadIncrementalHFiles-17] >mapreduce.LoadIncrementalHFiles: HFile at >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/feaa0a6428f24a5294c87dd87c6bc5a6.top > no longer fits inside a single region. Splitting... 2015-12-23 15:56:44,959 >INFO [LoadIncrementalHFiles-17] mapreduce.LoadIncrementalHFiles: Successfully >split into new HFiles >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/3886569dba4041deb4487f49d0417ca6.bottom > and >hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/3886569dba4041deb4487f49d0417ca6.top > 2015-12-23 15:56:46,826 INFO [main] mapreduce.LoadIncrementalHFiles: Split >occured while grouping HFiles, retry attempt 7 with 2 files remaining to group >or split 2015-12-23 15:56:46,832 INFO [LoadIncrementalHFiles-19] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/3886569dba4041deb4487f49d0417ca6.bottom > first=f89f12a59e4a9c9bcbb42d0504318e25 last=f8e7bc423ca4799459898439bf0f68b2 >2015-12-23 15:56:46,833 INFO [LoadIncrementalHFiles-20] >mapreduce.LoadIncrementalHFiles: Trying to load >hfile=hdfs://tdhdfs/user/tongdun/id_hbase/1/id/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/3886569dba4041deb4487f49d0417ca6.top > first=f8e7bc4bc8c2e7eac7f7e31bc116f8e0 last=f93061a29e9458fada2521ffe45ca385 >2015-12-23 15:56:46,930 INFO [main] >client.ConnectionManager$HConnectionImplementation: Closing master protocol: >MasterService 2015-12-23 15:56:46,931 INFO [main] >client.ConnectionManager$HConnectionImplementation: Closing zookeeper >sessionid=0x3515d529acedbaa 2015-12-23 15:56:46,960 INFO [main] >zookeeper.ZooKeeper: Session: 0x3515d529acedbaa closed 2015-12-23 15:56:46,960 >INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down >even though the process finished, original hfile did't delete. I was wondering >why mv/rename command not happend. >[qihuang.zheng@spark047213 ~]$ hadoop fs -du -h /user/tongdun/id_hbase/1/id/ >3.3 G /user/tongdun/id_hbase/1/id/01114a58782b4c369819673e4b3678ae 6.0 G >/user/tongdun/id_hbase/1/id/_tmp > > > > >tks, qihuang.zheng
