Hello Lu: Any other environmental or hardware differences between dev and prod? E.g. is the compaction the only thing running on prod? Are there adjacent processes that might be using up i/o or network? Are the machines same in prod as in dev?
Also, you compare one compaction on dev to another on prod. If you compare a larger sample set, do you see the phenomenon still or is it this single compaction only? There is a compaction tool that you might run on both clusters to more directly compare compaction throughputs: http://hbase.apache.org/book.html#compaction.tool S On Mon, Aug 20, 2018 at 3:11 AM Lu Dillon <[email protected]> wrote: > Hi All, > > > I'm a newbie against Hbase. When I use YCSB to load data into Hbase, the > compaction in two cluster looks different. > > DEV: 6 nodes. A and B are master and standby for HDFS and Hbase. C/D/E/F > are the HDFS datanode and regionservers. > > PROD: 8 nodes. A and B are master and standby for HDFS and Hbase. C/D/E/F/G/H > are the HDFS datanode and regionservers. > > Network is 1000mbs. > > Hadoop: 2.7.3 > > Hbase: 1.2.6 > > JDK:1.8 > > Zookeep: install on C/D/E > > > > In DEV, I notice the compaction 4G files are about 30 seconds; however, it > tooks about 30 seconds to process 400MB files in PROD. > > For example, this is logs from one node of DEV: > > 2018-08-10 16:30:47,535 INFO > [regionserver/node103/172.28.200.103:16020-shortCompactions-1533803656579] > regionserver.HStore: Completed compaction of 6 file(s) in family of > usertable_nosplit,,1533878791857.be420543a609c32475386855dee93a82. into > abe3a5c0dbbd4708ae2523f9b4a53ad8(size=1.4 G), total size for store is 4.0 > G. This selection was in queue for 0sec, and took 18sec to execute. > > > This is from PROD: > > 2018-08-17 23:29:28,751 INFO > > [regionserver/YJB-HADOOP-74-23/192.168.74.23:16020-shortCompactions-1534519629607] > regionserver.HStore: Completed compaction of 3 (all) file(s) in family of > usertable_nosplit,,1534518587365.95595277af1c56260b85de113c9687f1. into > 934669bf6a8a407aa6ee385439c4a7b5(size=430.8 M), total size for store is > 430.8 M. This selection was in queue for 0sec, and took 30sec to execute. > > > I thought this might be a problem of GC. So I change to use CG1 with this > article "https://product.hubspot.com/blog/g1gc-tuning-your-hbase-cluster"; > . The performance improve a little, not to good. > > > Is there any suggestion to find out what causes the problem and improve > the performance? > > > BTW: I'm not sure what kind information can help so I attach my > hbase-site.xml and hbase-env.sh first. > > BTW2: to use GC1 against regionserver, I create a speraeted hbase-env.sh > for regionserver. > > > Thanks, > > Dillon >
