Hi! Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track this.
Istvan On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <[email protected]> wrote: > If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it would > be a huge help if we could do that. A couple different reasons: > > 1. Ease of development. The workaround described in BUILDING.md adds extra > friction to getting an environment setup, and it has to be redone for each > patch release of HBase an engineer needs to work with. It's confusing for > both new contributors and experienced ones who don't usually work with 5.x > or haven't lately. > > 2. Ease of upgrade. At my dayjob, we try to avoid upgrading combinations of > Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing and > debugging simpler. HBase 2.x and Phoenix 5.x have to be done in lockstep, > and there's no avoiding that. But since HBase 1.x doesn't support Hadoop 3, > and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an > upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop, > HBase and Phoenix to new major versions simultaneously. It would be really > nice to do HBase / Phoenix first, and then Hadoop later. > > Geoffrey > > > > On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <[email protected]> > wrote: > > > Hi! > > > > For the first part of your question: Yes, you are correct. > > > > I think that there is no inherent technical issue that would make it > > impossible to support Hbase 2 on Hadoop 2 for Phoenix. > > > > While this happened before my time, I guess that supporting Hadoop 3 only > > was a decision that partly stemmed from the then current Phoenix maven > > setup, > > which wasn't equipped to support multiple profiles (see the multiple > > branches for the HBase 1.x versions), and partly because this was the > > configuration that > > the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in. > > > > I don't know how much work and additional complexity it would take to add > > Hadoop2 support for master. > > My guess is that it would need some major changes on the maven side for > > building, packaging, testing, and docs, > > but probably little in the way of Java code changes. > > > > It would also make providing binaries for Phoenix, connectors and PQS > even > > more hairy than now. > > > > If anyone has more historical context, or a better idea of what it would > > take to do this, please let us know! > > > > Istvan > > > > > > > > On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <[email protected]> wrote: > > > > > Hi, > > > > > > > However, up to now, we could use these artifacts with Hadoop 3.0 and > > they > > > > worked. (Or we just didn't hit the incompatibilities in our test > > suite.) > > > > > > I have a question reg Phoenix 5.x supporting Hadoop 3 only. > > > I believe this means Phoenix 5.x cannot work with HBase 2.x that > > > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 > > > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). > > > Is this correct? > > > > > > If this is correct, shall we not support Hadoop 2 because HBase 2 > > > still has default Hadoop profile: 2.0? Moreover, what is the root > > > cause behind Phoenix 5.x only supporting Hadoop 3.x? > > > As per the compatibility guidelines [1], HBase 2.3 is fully compatible > > > with Hadoop 2.10.x. > > > > > > Apologies for these questions might be repeated ones but > > > somehow I still could not trace back the root cause. > > > > > > > > > 1. https://hbase.apache.org/book.html#basic.prerequisites > > > > > > > > > On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > > > > Hi! > > > > > > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for > this, > > > but I > > > > am repeating it here for greater visibility) > > > > > > > > The HBase artifacts downloadable from maven central are built with > > Hadoop > > > > 2.x > > > > > > > > However, up to now, we could use these artifacts with Hadoop 3.0 and > > they > > > > worked. (Or we just didn't hit the incompatibilities in our test > > suite.) > > > > > > > > This seems to have changed with 2.2.5, as the public maven artifact > > > doesn't > > > > work with Hadoop 3.0.3 or 3.1.2 . > > > > > > > > This is a known issue in HBase, and documented, but this means that > > > > > > > > Any client JAR we'd build with Hbase 2.2.5 would have the same > problem. > > > > We cannot run our tests with Hbase 2.2.5 > > > > HBase's suggested solution is to rebuild HBase with the Hadoop > version > > > used > > > > in the cluster, and use that. This, however, doesn't fit into our > test > > or > > > > distribution process. > > > > > > > > I'm looking for your input on how to fix this. > > > > > > > > The options I could think of are: > > > > > > > > - Ask HBase to fix it for us (it doesn't seem to be a priority) > > > > - HBase could publish artifacts for Hadoop3 > > > > - Or could try to stick the common subset of Hadoop APIs > > > > - Re-publish the official HBase releases built with Hadoop 3 under > the > > > > org.apache.phoenix groupId > > > > - Hack up our build system to rebuild HBase from source, and use that > > for > > > > Phoenix > > > > > > > > None of these are particularly compelling, but I expect that this > will > > > have > > > > to be solved somehow if we want to keep up with future Hadoop > releases. > > > > > > > > Looking forward to hearing your ideas and opinion on this > > > > > > > > Istvan > > > > > > > > > > > > > -- > > *István Tóth* | Staff Software Engineer > > [email protected] <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > <https://www.cloudera.com/> > > ------------------------------ > > >
