Thanks for your reviewing and flaky test checking, Duo. Will file a separate issue to address your comment if necessary.
Thanks. On Wed, Jun 19, 2019 at 9:55 PM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > +1 from me. > > Left a few comments on github PR, not big problems. And the flaky dashboard > is pretty good. > > > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/HBASE-21879/lastSuccessfulBuild/artifact/dashboard.html > > > The TestConnectionImplementation was also failing on master, and was fixed > after merging back HBASE-21512. > > 张铎(Duo Zhang) <palomino...@gmail.com> 于2019年6月18日周二 下午9:48写道： > > > Good. Will take a look soon. > > > > OpenInx <open...@gmail.com> 于2019年6月18日周二 下午9:41写道： > > > >> > Could please open a PR, just like what I have done for HBASE-21512, so > >> that others could have a overall view on the modified code? > >> > >> OK, created a PR for this: https://github.com/apache/hbase/pull/320 > >> Thanks for your suggestion, Duo. > >> > >> On Tue, Jun 18, 2019 at 9:24 PM 张铎(Duo Zhang) <palomino...@gmail.com> > >> wrote: > >> > >> > The performance number is great. > >> > > >> > Could please open a PR, just like what I have done for HBASE-21512, so > >> that > >> > others could have a overall view on the modified code? > >> > > >> > Thanks. > >> > > >> > OpenInx <open...@gmail.com> 于2019年6月18日周二 下午6:58写道： > >> > > >> > > BTW, when testing this branch, we found some performance issues > >> about > >> > > HDFS Client: > >> > > 1. we reduced the DFS client's heap allocation from 45% to 27% > >> > > in HDFS-14535 [1]; > >> > > 2. we also increased get throughput by 17.8% in disabled block > cache > >> > case > >> > > in HDFS-14541[2]. > >> > > In theory, it should also helps a lot (especially p99/p999) > even > >> if > >> > RS > >> > > has a high cacheHitRatio. > >> > > > >> > > I think the next HDFS 2.8 release will include those patches, > they're > >> > very > >> > > good points for our > >> > > HBase performance. > >> > > > >> > > [1]. https://issues.apache.org/jira/browse/HDFS-14535 > >> > > [2]. > >> > > > >> > > > >> > > >> > https://issues.apache.org/jira/browse/HDFS-14541?focusedCommentId=16866472&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16866472 > >> > > > >> > > Thanks. > >> > > > >> > > On Tue, Jun 18, 2019 at 12:05 PM OpenInx <open...@gmail.com> wrote: > >> > > > >> > > > The HBASE-21879 has lots of changes: 123 files changed, 5833 > >> > > > insertions(+), 3015 deletions(-). > >> > > > Currently we developed this issue based on master branch, and > >> expect to > >> > > > release it in future HBase3.x. > >> > > > Of course, if branch-2 want this feature we can do the backport, > >> should > >> > > > have some conflicts now but I > >> > > > don't think it would be hard to fix because I believe the branch-2 > >> > > > shouldn't have so much diff with master now > >> > > > (at least in read path). > >> > > > The first priority thing for now, I think it would be merging > the > >> > > > HBASE-21879 branch to master branch > >> > > > before diverging. After that, I can do the backport. > >> > > > > >> > > > Thanks for your suggestion, Guanghao ! > >> > > > > >> > > > > >> > > > > >> > > > On Tue, Jun 18, 2019 at 11:39 AM Guanghao Zhang < > zghao...@gmail.com > >> > > >> > > > wrote: > >> > > > > >> > > >> This is a improvement not a new feature? So backport to branch-2, > >> too? > >> > > >> > >> > > >> OpenInx <open...@gmail.com> 于2019年6月17日周一 下午2:45写道： > >> > > >> > >> > > >> > Dear HBase dev: > >> > > >> > > >> > > >> > In HBASE-21879[1], we redesigned the offheap read path: read > the > >> > > >> HFileBlock > >> > > >> > from HDFS to pooled offheap > >> > > >> > ByteBuffers directly, while before HBASE-21879 we just read the > >> > > >> HFileBlock > >> > > >> > to heap which would still lead > >> > > >> > to high GC pressure. > >> > > >> > > >> > > >> > After few months of development and testing, all subtasks have > >> been > >> > > >> > resovled now except the HBASE-21946[2] > >> > > >> > (It depends on HDFS-14483[3] and our HDFS teams are working on > >> this, > >> > > we > >> > > >> > expect the HDFS-14483 to be included > >> > > >> > in hadoop 2.8.6 and after that the HBASE-21946 will get > >> resolved). > >> > we > >> > > >> think > >> > > >> > the feature is stable enough now and it's > >> > > >> > time to merge branch HBASE-21879 back to master now. > >> > > >> > > >> > > >> > We have designed 3 test cases to prove the performance > improvment > >> > with > >> > > >> > HBASE-21879: > >> > > >> > 1. Disabled BlockCache, which means the cacheHitRatio is 0%; > >> > > >> > 2. CacheHitRatio~65%; > >> > > >> > 3. CachehHitRatio~100%; > >> > > >> > > >> > > >> > In our performance results[4], we can see that: the case#1 have > >> an > >> > > great > >> > > >> > performance improvement > >> > > >> > ( > >> > > >> > *throughput increased about 17%, heap allocation decreased > about > >> > 95%, > >> > > >> Young > >> > > >> > generaion size decreased about 81.7%*), that's because after > >> > > HBASE-21879 > >> > > >> > all reads will allocate from pooled offheap bytebuffers > >> > > >> > and almost no heap allocation, while before HBASE-21879 the > read > >> > path > >> > > >> will > >> > > >> > create so many heap allocations. > >> > > >> > On the other hand, from the testing results of case#2 and > case#3 > >> we > >> > > can > >> > > >> > also see that: > >> > > >> > > >> > > >> > *As the cacheHitRatioincreasing, the difference between > >> > > >> before-HBASE-21879 > >> > > >> > and after-HBASE-21879 will decrease, when cacheHitRatio is > 100%, > >> > they > >> > > >> > almost have no much difference in both throughput and latency.* > >> > > >> > > >> > > >> > For more details please see the document[4]. Thanks > >> > > >> > Anoop/Ram/DuoZhang/Stack/GuanghaoZhang very much > >> > > >> > for your meticulous work (Suggession, discussion, patch > >> reviewing, > >> > doc > >> > > >> > reviewing etc). > >> > > >> > > >> > > >> > Please vote > >> > > >> > > >> > > >> > [] +1 > >> > > >> > [] +0/-0 > >> > > >> > [] -1 Do not merge the branch back because... > >> > > >> > > >> > > >> > Thanks. Any suggestions are welcomed. > >> > > >> > > >> > > >> > [1] https://issues.apache.org/jira/browse/HBASE-21879 > >> > > >> > [2] https://issues.apache.org/jira/browse/HBASE-21946 > >> > > >> > [3] https://issues.apache.org/jira/browse/HDFS-14483 > >> > > >> > [4] > >> > > >> > > >> > > >> > > >> > > >> > >> > > > >> > > >> > https://docs.google.com/document/d/1xSy9axGxafoH-Qc17zbD2Bd--rWjjI00xTWQZ8ZwI_E > >> > > >> > > >> > > >> > >> > > > > >> > > > >> > > >> > > >
