Hi! For the first part of your question: Yes, you are correct.
I think that there is no inherent technical issue that would make it impossible to support Hbase 2 on Hadoop 2 for Phoenix. While this happened before my time, I guess that supporting Hadoop 3 only was a decision that partly stemmed from the then current Phoenix maven setup, which wasn't equipped to support multiple profiles (see the multiple branches for the HBase 1.x versions), and partly because this was the configuration that the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in. I don't know how much work and additional complexity it would take to add Hadoop2 support for master. My guess is that it would need some major changes on the maven side for building, packaging, testing, and docs, but probably little in the way of Java code changes. It would also make providing binaries for Phoenix, connectors and PQS even more hairy than now. If anyone has more historical context, or a better idea of what it would take to do this, please let us know! Istvan On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <[email protected]> wrote: > Hi, > > > However, up to now, we could use these artifacts with Hadoop 3.0 and they > > worked. (Or we just didn't hit the incompatibilities in our test suite.) > > I have a question reg Phoenix 5.x supporting Hadoop 3 only. > I believe this means Phoenix 5.x cannot work with HBase 2.x that > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). > Is this correct? > > If this is correct, shall we not support Hadoop 2 because HBase 2 > still has default Hadoop profile: 2.0? Moreover, what is the root > cause behind Phoenix 5.x only supporting Hadoop 3.x? > As per the compatibility guidelines [1], HBase 2.3 is fully compatible > with Hadoop 2.10.x. > > Apologies for these questions might be repeated ones but > somehow I still could not trace back the root cause. > > > 1. https://hbase.apache.org/book.html#basic.prerequisites > > > On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > > Hi! > > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for this, > but I > > am repeating it here for greater visibility) > > > > The HBase artifacts downloadable from maven central are built with Hadoop > > 2.x > > > > However, up to now, we could use these artifacts with Hadoop 3.0 and they > > worked. (Or we just didn't hit the incompatibilities in our test suite.) > > > > This seems to have changed with 2.2.5, as the public maven artifact > doesn't > > work with Hadoop 3.0.3 or 3.1.2 . > > > > This is a known issue in HBase, and documented, but this means that > > > > Any client JAR we'd build with Hbase 2.2.5 would have the same problem. > > We cannot run our tests with Hbase 2.2.5 > > HBase's suggested solution is to rebuild HBase with the Hadoop version > used > > in the cluster, and use that. This, however, doesn't fit into our test or > > distribution process. > > > > I'm looking for your input on how to fix this. > > > > The options I could think of are: > > > > - Ask HBase to fix it for us (it doesn't seem to be a priority) > > - HBase could publish artifacts for Hadoop3 > > - Or could try to stick the common subset of Hadoop APIs > > - Re-publish the official HBase releases built with Hadoop 3 under the > > org.apache.phoenix groupId > > - Hack up our build system to rebuild HBase from source, and use that for > > Phoenix > > > > None of these are particularly compelling, but I expect that this will > have > > to be solved somehow if we want to keep up with future Hadoop releases. > > > > Looking forward to hearing your ideas and opinion on this > > > > Istvan > > > -- *István Tóth* | Staff Software Engineer [email protected] <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> <https://www.cloudera.com/> ------------------------------
