Maybe we could try to see if we could make Async WAL work with both hadoop 2 and hadoop 3.
But please do not rely on me, I'm not sure if I can make this done soon. Thanks. Andrew Purtell <[email protected]> 于2021年3月7日周日 上午1:36写道： > Because the Async WAL module will not link with Hadoop 3 jars if it has > been compiled for Hadoop 2, Phoenix builds that download the Hadoop 3 and > HBase 2 jars from public repositories will not work. Tests cannot run > during the Maven verify phase because the minicluster will terminate with > linkage errors. To work around this Phoenix build instructions tell the > user to recompile HBase for Hadoop 3 and install it into the local Maven > cache first. > > To the extent Hadoop and HBase classes expecting to link with Hadoop 3 are > incorporated unshaded into the Phoenix server jar there is also risk of > real runtime problems in a deployment into a server classpath expecting and > including Hadoop 2. > > Publishing artifacts built against Hadoop 3 would be a solution but there > are a couple of risks. The first is the time and complexity of the release > candidate build and publish process will double. The second is Maven > classifiers somehow are not up to the task of distinguishing between > default and Hadoop 3 profile artifacts. The latter would be something we > can determine right away. The former would be potentially be a problem > every time an RM attempts a release. A nightly build for Hadoop 3 with > people actually paying attention and fixing failures there would mitigate. > > > > On Mar 5, 2021, at 5:37 AM, 张铎 <[email protected]> wrote: > > > > The reason why we only publish artifacts compiled with hadoop 2 for > HBase > > is that, you can use these artifacts to run HBase against a 3.x HDFS > > cluster, no problem, you do not need to replace the hadoop jars in HBase > to > > hadoop 3, as we will not make use any new features at client side... > > > > I'm not sure the usage of phoenix, if it is necessary, I think we could > try > > to publish extra artifacts which are compiled against hadoop 3.x. > > > > Thanks. > > > > Istvan Toth <[email protected]> 于2021年3月5日周五 上午1:40写道： > > > >> Hi! > >> Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track > this. > >> Istvan > >>> On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <[email protected]> > >>> wrote: > >>> If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it > >> would > >>> be a huge help if we could do that. A couple different reasons: > >>> 1. Ease of development. The workaround described in BUILDING.md adds > >> extra > >>> friction to getting an environment setup, and it has to be redone for > >> each > >>> patch release of HBase an engineer needs to work with. It's confusing > for > >>> both new contributors and experienced ones who don't usually work with > >> 5.x > >>> or haven't lately. > >>> 2. Ease of upgrade. At my dayjob, we try to avoid upgrading > combinations > >> of > >>> Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing > >> and > >>> debugging simpler. HBase 2.x and Phoenix 5.x have to be done in > lockstep, > >>> and there's no avoiding that. But since HBase 1.x doesn't support > Hadoop > >> 3, > >>> and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an > >>> upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop, > >>> HBase and Phoenix to new major versions simultaneously. It would be > >> really > >>> nice to do HBase / Phoenix first, and then Hadoop later. > >>> Geoffrey > >>> On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <[email protected] > > > >>> wrote: > >>>> Hi! > >>>> For the first part of your question: Yes, you are correct. > >>>> I think that there is no inherent technical issue that would make it > >>>> impossible to support Hbase 2 on Hadoop 2 for Phoenix. > >>>> While this happened before my time, I guess that supporting Hadoop 3 > >> only > >>>> was a decision that partly stemmed from the then current Phoenix maven > >>>> setup, > >>>> which wasn't equipped to support multiple profiles (see the multiple > >>>> branches for the HBase 1.x versions), and partly because this was the > >>>> configuration that > >>>> the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in. > >>>> I don't know how much work and additional complexity it would take to > >> add > >>>> Hadoop2 support for master. > >>>> My guess is that it would need some major changes on the maven side > for > >>>> building, packaging, testing, and docs, > >>>> but probably little in the way of Java code changes. > >>>> It would also make providing binaries for Phoenix, connectors and PQS > >>> even > >>>> more hairy than now. > >>>> If anyone has more historical context, or a better idea of what it > >> would > >>>> take to do this, please let us know! > >>>> Istvan > >>>> On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <[email protected]> > >> wrote: > >>>>> Hi, > >>>>>> However, up to now, we could use these artifacts with Hadoop 3.0 > >> and > >>>> they > >>>>>> worked. (Or we just didn't hit the incompatibilities in our test > >>>> suite.) > >>>>> I have a question reg Phoenix 5.x supporting Hadoop 3 only. > >>>>> I believe this means Phoenix 5.x cannot work with HBase 2.x that > >>>>> is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 > >>>>> jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). > >>>>> Is this correct? > >>>>> If this is correct, shall we not support Hadoop 2 because HBase 2 > >>>>> still has default Hadoop profile: 2.0? Moreover, what is the root > >>>>> cause behind Phoenix 5.x only supporting Hadoop 3.x? > >>>>> As per the compatibility guidelines [1], HBase 2.3 is fully > >> compatible > >>>>> with Hadoop 2.10.x. > >>>>> Apologies for these questions might be repeated ones but > >>>>> somehow I still could not trace back the root cause. > >>>>> 1. https://hbase.apache.org/book.html#basic.prerequisites > >>>>> On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > >>>>>> Hi! > >>>>>> (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for > >>> this, > >>>>> but I > >>>>>> am repeating it here for greater visibility) > >>>>>> The HBase artifacts downloadable from maven central are built with > >>>> Hadoop > >>>>>> 2.x > >>>>>> However, up to now, we could use these artifacts with Hadoop 3.0 > >> and > >>>> they > >>>>>> worked. (Or we just didn't hit the incompatibilities in our test > >>>> suite.) > >>>>>> This seems to have changed with 2.2.5, as the public maven artifact > >>>>> doesn't > >>>>>> work with Hadoop 3.0.3 or 3.1.2 . > >>>>>> This is a known issue in HBase, and documented, but this means that > >>>>>> Any client JAR we'd build with Hbase 2.2.5 would have the same > >>> problem. > >>>>>> We cannot run our tests with Hbase 2.2.5 > >>>>>> HBase's suggested solution is to rebuild HBase with the Hadoop > >>> version > >>>>> used > >>>>>> in the cluster, and use that. This, however, doesn't fit into our > >>> test > >>>> or > >>>>>> distribution process. > >>>>>> I'm looking for your input on how to fix this. > >>>>>> The options I could think of are: > >>>>>> - Ask HBase to fix it for us (it doesn't seem to be a priority) > >>>>>> - HBase could publish artifacts for Hadoop3 > >>>>>> - Or could try to stick the common subset of Hadoop APIs > >>>>>> - Re-publish the official HBase releases built with Hadoop 3 under > >>> the > >>>>>> org.apache.phoenix groupId > >>>>>> - Hack up our build system to rebuild HBase from source, and use > >> that > >>>> for > >>>>>> Phoenix > >>>>>> None of these are particularly compelling, but I expect that this > >>> will > >>>>> have > >>>>>> to be solved somehow if we want to keep up with future Hadoop > >>> releases. > >>>>>> Looking forward to hearing your ideas and opinion on this > >>>>>> Istvan > >>>> -- > >>>> *István Tóth* | Staff Software Engineer > >>>> [email protected] <https://www.cloudera.com> > >>>> [image: Cloudera] <https://www.cloudera.com/> > >>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > >>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > >>> Cloudera > >>>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > >>>> <https://www.cloudera.com/> > >>>> ------------------------------ >
