Thanks for chiming in Geoffrey, Istvan and Duo. > I'm not sure the usage of phoenix, if it is necessary, I think we could try > to publish extra artifacts which are compiled against hadoop 3.x.
Duo, recently published Phoenix 5.1.0 can run only with HBase 2.x compiled with Hadoop 3.0 profile. While releasing Phoenix 5, all HBase supported version tarballs are released based on each HBase profile supported by Phoenix (e.g https://dist.apache.org/repos/dist/release/phoenix/phoenix-5.1.0/) What I am trying to convince is the fact that Phoenix 5 should support Hadoop 2 compiled HBase (default) as well because not all in-place rolling upgrades would like to deal with upgrading Hadoop while upgrading HBase from HBase 1 to HBase 2 (and hence, Phoenix 4 to Phoenix 5). Geoffrey has correctly mentioned all the points in details, Thanks Geoffrey. Also, as for publishing HBase 2 extra artifacts compiled against Hadoop 3, I think it's a good idea but we might not have to do it (unless explicitly mentioned by our users, has anyone requested the same in the past?) as we at least have Hadoop profiles available for both Hadoop 2 and 3, hence users do have a way to generate artifacts themselves if they are not fine with default Hadoop 2. For Phoenix too, having Hadoop profiles for using Hadoop 2 or 3 with any of supported HBase versions would be a great option, and IMHO Phoenix also might not need to release artifacts for both Hadoop profiles because users have a way to get any combination they would like to have as long as we have profiles available. If we decide to release all combinations, it might look something like phoenix-hbase-hadoop-2.1-5.1.0-3.1.4, phoenix-hbase-hadoop-2.1-5.1.0-2.10.0 and so on. Anyways, this decision could be taken up on a separate thread :) On Fri, Mar 5, 2021 at 7:07 PM 张铎(Duo Zhang) <[email protected]> wrote: > The reason why we only publish artifacts compiled with hadoop 2 for HBase > is that, you can use these artifacts to run HBase against a 3.x HDFS > cluster, no problem, you do not need to replace the hadoop jars in HBase to > hadoop 3, as we will not make use any new features at client side... > > I'm not sure the usage of phoenix, if it is necessary, I think we could try > to publish extra artifacts which are compiled against hadoop 3.x. > > Thanks. > > Istvan Toth <[email protected]> 于2021年3月5日周五 上午1:40写道： > > > Hi! > > > > Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track this. > > > > Istvan > > > > On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <[email protected]> > > wrote: > > > > > If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it > > would > > > be a huge help if we could do that. A couple different reasons: > > > > > > 1. Ease of development. The workaround described in BUILDING.md adds > > extra > > > friction to getting an environment setup, and it has to be redone for > > each > > > patch release of HBase an engineer needs to work with. It's confusing > for > > > both new contributors and experienced ones who don't usually work with > > 5.x > > > or haven't lately. > > > > > > 2. Ease of upgrade. At my dayjob, we try to avoid upgrading > combinations > > of > > > Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing > > and > > > debugging simpler. HBase 2.x and Phoenix 5.x have to be done in > lockstep, > > > and there's no avoiding that. But since HBase 1.x doesn't support > Hadoop > > 3, > > > and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an > > > upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop, > > > HBase and Phoenix to new major versions simultaneously. It would be > > really > > > nice to do HBase / Phoenix first, and then Hadoop later. > > > > > > Geoffrey > > > > > > > > > > > > On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <[email protected] > > > > > wrote: > > > > > > > Hi! > > > > > > > > For the first part of your question: Yes, you are correct. > > > > > > > > I think that there is no inherent technical issue that would make it > > > > impossible to support Hbase 2 on Hadoop 2 for Phoenix. > > > > > > > > While this happened before my time, I guess that supporting Hadoop 3 > > only > > > > was a decision that partly stemmed from the then current Phoenix > maven > > > > setup, > > > > which wasn't equipped to support multiple profiles (see the multiple > > > > branches for the HBase 1.x versions), and partly because this was the > > > > configuration that > > > > the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested > in. > > > > > > > > I don't know how much work and additional complexity it would take to > > add > > > > Hadoop2 support for master. > > > > My guess is that it would need some major changes on the maven side > for > > > > building, packaging, testing, and docs, > > > > but probably little in the way of Java code changes. > > > > > > > > It would also make providing binaries for Phoenix, connectors and PQS > > > even > > > > more hairy than now. > > > > > > > > If anyone has more historical context, or a better idea of what it > > would > > > > take to do this, please let us know! > > > > > > > > Istvan > > > > > > > > > > > > > > > > On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <[email protected]> > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > However, up to now, we could use these artifacts with Hadoop 3.0 > > and > > > > they > > > > > > worked. (Or we just didn't hit the incompatibilities in our test > > > > suite.) > > > > > > > > > > I have a question reg Phoenix 5.x supporting Hadoop 3 only. > > > > > I believe this means Phoenix 5.x cannot work with HBase 2.x that > > > > > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 > > > > > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). > > > > > Is this correct? > > > > > > > > > > If this is correct, shall we not support Hadoop 2 because HBase 2 > > > > > still has default Hadoop profile: 2.0? Moreover, what is the root > > > > > cause behind Phoenix 5.x only supporting Hadoop 3.x? > > > > > As per the compatibility guidelines [1], HBase 2.3 is fully > > compatible > > > > > with Hadoop 2.10.x. > > > > > > > > > > Apologies for these questions might be repeated ones but > > > > > somehow I still could not trace back the root cause. > > > > > > > > > > > > > > > 1. https://hbase.apache.org/book.html#basic.prerequisites > > > > > > > > > > > > > > > On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > > > > > > Hi! > > > > > > > > > > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for > > > this, > > > > > but I > > > > > > am repeating it here for greater visibility) > > > > > > > > > > > > The HBase artifacts downloadable from maven central are built > with > > > > Hadoop > > > > > > 2.x > > > > > > > > > > > > However, up to now, we could use these artifacts with Hadoop 3.0 > > and > > > > they > > > > > > worked. (Or we just didn't hit the incompatibilities in our test > > > > suite.) > > > > > > > > > > > > This seems to have changed with 2.2.5, as the public maven > artifact > > > > > doesn't > > > > > > work with Hadoop 3.0.3 or 3.1.2 . > > > > > > > > > > > > This is a known issue in HBase, and documented, but this means > that > > > > > > > > > > > > Any client JAR we'd build with Hbase 2.2.5 would have the same > > > problem. > > > > > > We cannot run our tests with Hbase 2.2.5 > > > > > > HBase's suggested solution is to rebuild HBase with the Hadoop > > > version > > > > > used > > > > > > in the cluster, and use that. This, however, doesn't fit into our > > > test > > > > or > > > > > > distribution process. > > > > > > > > > > > > I'm looking for your input on how to fix this. > > > > > > > > > > > > The options I could think of are: > > > > > > > > > > > > - Ask HBase to fix it for us (it doesn't seem to be a priority) > > > > > > - HBase could publish artifacts for Hadoop3 > > > > > > - Or could try to stick the common subset of Hadoop APIs > > > > > > - Re-publish the official HBase releases built with Hadoop 3 > under > > > the > > > > > > org.apache.phoenix groupId > > > > > > - Hack up our build system to rebuild HBase from source, and use > > that > > > > for > > > > > > Phoenix > > > > > > > > > > > > None of these are particularly compelling, but I expect that this > > > will > > > > > have > > > > > > to be solved somehow if we want to keep up with future Hadoop > > > releases. > > > > > > > > > > > > Looking forward to hearing your ideas and opinion on this > > > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *István Tóth* | Staff Software Engineer > > > > [email protected] <https://www.cloudera.com> > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > <https://www.cloudera.com/> > > > > ------------------------------ > > > > > > > > > >
