The reason why we only publish artifacts compiled with hadoop 2 for HBase is that, you can use these artifacts to run HBase against a 3.x HDFS cluster, no problem, you do not need to replace the hadoop jars in HBase to hadoop 3, as we will not make use any new features at client side...
I'm not sure the usage of phoenix, if it is necessary, I think we could try to publish extra artifacts which are compiled against hadoop 3.x. Thanks. Istvan Toth <[email protected]> 于2021年3月5日周五 上午1:40写道： > Hi! > > Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track this. > > Istvan > > On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <[email protected]> > wrote: > > > If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it > would > > be a huge help if we could do that. A couple different reasons: > > > > 1. Ease of development. The workaround described in BUILDING.md adds > extra > > friction to getting an environment setup, and it has to be redone for > each > > patch release of HBase an engineer needs to work with. It's confusing for > > both new contributors and experienced ones who don't usually work with > 5.x > > or haven't lately. > > > > 2. Ease of upgrade. At my dayjob, we try to avoid upgrading combinations > of > > Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing > and > > debugging simpler. HBase 2.x and Phoenix 5.x have to be done in lockstep, > > and there's no avoiding that. But since HBase 1.x doesn't support Hadoop > 3, > > and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an > > upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop, > > HBase and Phoenix to new major versions simultaneously. It would be > really > > nice to do HBase / Phoenix first, and then Hadoop later. > > > > Geoffrey > > > > > > > > On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <[email protected]> > > wrote: > > > > > Hi! > > > > > > For the first part of your question: Yes, you are correct. > > > > > > I think that there is no inherent technical issue that would make it > > > impossible to support Hbase 2 on Hadoop 2 for Phoenix. > > > > > > While this happened before my time, I guess that supporting Hadoop 3 > only > > > was a decision that partly stemmed from the then current Phoenix maven > > > setup, > > > which wasn't equipped to support multiple profiles (see the multiple > > > branches for the HBase 1.x versions), and partly because this was the > > > configuration that > > > the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in. > > > > > > I don't know how much work and additional complexity it would take to > add > > > Hadoop2 support for master. > > > My guess is that it would need some major changes on the maven side for > > > building, packaging, testing, and docs, > > > but probably little in the way of Java code changes. > > > > > > It would also make providing binaries for Phoenix, connectors and PQS > > even > > > more hairy than now. > > > > > > If anyone has more historical context, or a better idea of what it > would > > > take to do this, please let us know! > > > > > > Istvan > > > > > > > > > > > > On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <[email protected]> > wrote: > > > > > > > Hi, > > > > > > > > > However, up to now, we could use these artifacts with Hadoop 3.0 > and > > > they > > > > > worked. (Or we just didn't hit the incompatibilities in our test > > > suite.) > > > > > > > > I have a question reg Phoenix 5.x supporting Hadoop 3 only. > > > > I believe this means Phoenix 5.x cannot work with HBase 2.x that > > > > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 > > > > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). > > > > Is this correct? > > > > > > > > If this is correct, shall we not support Hadoop 2 because HBase 2 > > > > still has default Hadoop profile: 2.0? Moreover, what is the root > > > > cause behind Phoenix 5.x only supporting Hadoop 3.x? > > > > As per the compatibility guidelines [1], HBase 2.3 is fully > compatible > > > > with Hadoop 2.10.x. > > > > > > > > Apologies for these questions might be repeated ones but > > > > somehow I still could not trace back the root cause. > > > > > > > > > > > > 1. https://hbase.apache.org/book.html#basic.prerequisites > > > > > > > > > > > > On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > > > > > Hi! > > > > > > > > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for > > this, > > > > but I > > > > > am repeating it here for greater visibility) > > > > > > > > > > The HBase artifacts downloadable from maven central are built with > > > Hadoop > > > > > 2.x > > > > > > > > > > However, up to now, we could use these artifacts with Hadoop 3.0 > and > > > they > > > > > worked. (Or we just didn't hit the incompatibilities in our test > > > suite.) > > > > > > > > > > This seems to have changed with 2.2.5, as the public maven artifact > > > > doesn't > > > > > work with Hadoop 3.0.3 or 3.1.2 . > > > > > > > > > > This is a known issue in HBase, and documented, but this means that > > > > > > > > > > Any client JAR we'd build with Hbase 2.2.5 would have the same > > problem. > > > > > We cannot run our tests with Hbase 2.2.5 > > > > > HBase's suggested solution is to rebuild HBase with the Hadoop > > version > > > > used > > > > > in the cluster, and use that. This, however, doesn't fit into our > > test > > > or > > > > > distribution process. > > > > > > > > > > I'm looking for your input on how to fix this. > > > > > > > > > > The options I could think of are: > > > > > > > > > > - Ask HBase to fix it for us (it doesn't seem to be a priority) > > > > > - HBase could publish artifacts for Hadoop3 > > > > > - Or could try to stick the common subset of Hadoop APIs > > > > > - Re-publish the official HBase releases built with Hadoop 3 under > > the > > > > > org.apache.phoenix groupId > > > > > - Hack up our build system to rebuild HBase from source, and use > that > > > for > > > > > Phoenix > > > > > > > > > > None of these are particularly compelling, but I expect that this > > will > > > > have > > > > > to be solved somehow if we want to keep up with future Hadoop > > releases. > > > > > > > > > > Looking forward to hearing your ideas and opinion on this > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > -- > > > *István Tóth* | Staff Software Engineer > > > [email protected] <https://www.cloudera.com> > > > [image: Cloudera] <https://www.cloudera.com/> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > Cloudera > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > <https://www.cloudera.com/> > > > ------------------------------ > > > > > >
