bq. on each Region Server there are about 800 hfiles Is it possible to control the number of hfiles during each bulk load ?
For this big table, are the regions evenly spread across the servers ? If so, consider increasing the capacity of your cluster. >From the doc for hbase.hstore.compactionThreshold : Larger values delay compaction, but when compaction does occur, it takes longer to complete. On Sat, Aug 26, 2017 at 6:48 AM, Liu, Ming (Ming) <ming....@esgyn.cn> wrote: > hi, all, > > We have a system with 17 nodes, with a big table about 28T in size. We use > native hbase bulkloader (importtsv) to load data, and it generated a lot of > hfiles, on each Region Server there are about 800 hfiles. We turned off > Major Compact, but the Minor compaction is running due to so many hfile. > The problem is, after the initial loading, there are about 80% disk space > used, when minor compaction is going on, we notice the disk space is > reducing rapidly until all disk spaces are used and hbase went down. > > We try to change the hbase.hstore.compactionThreshold to 2000, but the > minor compaction is still triggered. > > The system is CDH 5.7, HBase is 1.2. > > Could anyone help to give us some suggestions? We are really stuck. Thanks > in advance. > > Thanks, > Ming >
