Thanks for your reply Sree Vaddi. As you said, in our internal hadoop branch we have already backported these changes, and also it is fine for us to upgrade to hadoop 3, in fact we have already started to test hadoop 3.1.x in our company.
But here we are talking about the open source version of hbase being affected by these changes. For the open source version, I do not think we can drop all the hadoop 2.x support for hbase 2.x releases. We have to follow some rules, for example, only drop hadoop support when there are CVEs and the hadoop team will not fix it for a minor release line. And when changing the major versions, we can drop some legacy hadoop support, for example, when releasing hbase 3.x, probably we could drop all the support for hadoop 2.x. Thanks. Regards OpenInx <open...@gmail.com> 于2019年6月28日周五 上午10:58写道： > > What are your challenges upgrading to Hadoop 3 ? > > It's a major release upgradation from hadoop 2 to hadoop 3. we upstreams > will need some time to test the stability > and performance, so I think the HBase will not upgrade to hadoop 3 so quick > in short time. Backporting the ByteBuffer > pread to hadoop 2.9 is more suitable for now (it's a minor feature, should > be OK to port in branch-2.9). I believe in future > HBase release we will include the hadoop 3 > > > Did you verify if your problem statement valid, if you use Ozone API ? > We have some work about the object-store api [1]. but I'm not quite sure > how will it affect the GC. will take a look about that. > > Thanks for your reply. > > [1]. https://issues.apache.org/jira/browse/HBASE-22149 > On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sree_at_ch...@yahoo.com> > wrote: > > > What are your challenges upgrading to Hadoop 3 ? > > > > Did you verify if your problem statement valid, if you use Ozone API ? > > > > Sent from Yahoo Mail on Android > > < > https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature > > > > > > On Thu, Jun 27, 2019 at 9:20 AM, OpenInx > > <open...@gmail.com> wrote: > > Dear HDFS Team: > > > > I'm apache HBase member and working on HBASE-21879 [1]: it's a feature > > which help to optimize the > > HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading > > HDFS block into offheap > > ByteBuffers and it had great performance improvement in our benchmark > test > > [2]. While the offheap > > HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such > > as: ByteBuffer stream > > read and ByteBuffer positional read. > > > > The problem is ByteBuffer pread interface: we've finished the > HDFS-3246[3] > > for Hadoop 3.3.0 version. but > > the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not > > upgrade the dependency to > > Hadoop 3.3.0 soon. We have to backport HDFS-3246 to Hadoop branch-2.9 > so > > that in the next hadoop > > 2.9.3 can include it , and our Apache HBase can upgrade the dependency to > > 2.9.3. > > > > Now all the patches of HBASE-21879 have been merged into HBase master > > branch. If we cannot upgrade to > > the Hadoop which include the ByteBuffer pread implementation, then I > > believe almost all of work from HBASE-21879 > > will worth nothing because we still read heap block and still high GC > > pressure in some special case ( such as > > cacheHitRatio is not very high). That will be quite frustrating for us. > > > > Actually, we have been pushing the backport for the past few weeks. our > > XiaoMi HDFS team have prepared the > > backport patches, but seems the guys we ping didn't have the bandwidth to > > help review the patches. we > > expect the backport can be included in Hadoop 2.9.3 so that our upstream > > can benefit from this as soon as > > possible. > > > > If any HDFS committer or PMC have some bandwidth to review the backport > > patches, please give us a hand. > > The Apache HBase team really need your help. > > > > The ByteBuffer pread backport issues are: > > https://issues.apache.org/jira/browse/HDFS-14585 > > https://issues.apache.org/jira/browse/HDFS-14483 > > > > Thank you very much. > > > > [1]. https://issues.apache.org/jira/browse/HBASE-21879 > > [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html > > [3]. https://issues.apache.org/jira/browse/HDFS-3246 > > > > >
