> To further complicate matters, we may have similar incompatibilities > between Hadoop 3.0 and 3.1, as I have seen similar errors when building > Phoenix with Hadoop 3.1
This is unfortunate. Anyone know of Hadoop releasing plans for 3.0.x? Perhaps you can ignore it and go straight to 3.1.x. And propose on dev@hbase that future releases attempt publishing Hadoop 3 artifacts, with appropriate Maven classifier or some other way to discriminate, built against 3.1. On Sun, Mar 7, 2021 at 11:37 PM Istvan Toth <[email protected]> wrote: > The Phoenix server JARs should not have any HBase or Hadoop classes shaded > into them, and should rely on the ones on the HBase classpath. > We've fixed that before the 5.1 release, and if any have crept back, then > it's a bug that should be fixed. > > So it's possible that a phoenix-server JAR compiled for Hadoop3 will work > on a Hadoop2 deployment and vice versa. > (Anyone who deploys HBase2 on a Hadoop 3 cluster will still need to > recompile HBase, but that's outside Phoenix's scope) > > Phoenix-client is self-contained, and will clash with any standalone > HBase/Hadoop libraries anyway, so it's not really an issue there. > > However, without Hadoop3 specific official binaries, both deploying HBase > on Hadoop3, and running the Phoenix test suite will require recompiling > HBase from source. > > To further complicate matters, we may have similar incompatibilities > between Hadoop 3.0 and 3.1, as I have seen similar errors when building > Phoenix with Hadoop 3.1 > against an HBase built for Hadoop 3.0 (I did not save the error, or opened > an HBase ticket for it at the time, unfortunately, only noted it in > PHOENIX-6333) > > Istvan > > > On Sat, Mar 6, 2021 at 6:36 PM Andrew Purtell <[email protected]> > wrote: > > > Because the Async WAL module will not link with Hadoop 3 jars if it has > > been compiled for Hadoop 2, Phoenix builds that download the Hadoop 3 and > > HBase 2 jars from public repositories will not work. Tests cannot run > > during the Maven verify phase because the minicluster will terminate with > > linkage errors. To work around this Phoenix build instructions tell the > > user to recompile HBase for Hadoop 3 and install it into the local Maven > > cache first. > > > > To the extent Hadoop and HBase classes expecting to link with Hadoop 3 > are > > incorporated unshaded into the Phoenix server jar there is also risk of > > real runtime problems in a deployment into a server classpath expecting > and > > including Hadoop 2. > > > > Publishing artifacts built against Hadoop 3 would be a solution but there > > are a couple of risks. The first is the time and complexity of the > release > > candidate build and publish process will double. The second is Maven > > classifiers somehow are not up to the task of distinguishing between > > default and Hadoop 3 profile artifacts. The latter would be something we > > can determine right away. The former would be potentially be a problem > > every time an RM attempts a release. A nightly build for Hadoop 3 with > > people actually paying attention and fixing failures there would > mitigate. > > > > > > > On Mar 5, 2021, at 5:37 AM, 张铎 <[email protected]> wrote: > > > > > > The reason why we only publish artifacts compiled with hadoop 2 for > > HBase > > > is that, you can use these artifacts to run HBase against a 3.x HDFS > > > cluster, no problem, you do not need to replace the hadoop jars in > HBase > > to > > > hadoop 3, as we will not make use any new features at client side... > > > > > > I'm not sure the usage of phoenix, if it is necessary, I think we could > > try > > > to publish extra artifacts which are compiled against hadoop 3.x. > > > > > > Thanks. > > > > > > Istvan Toth <[email protected]> 于2021年3月5日周五 上午1:40写道： > > > > > >> Hi! > > >> Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track > > this. > > >> Istvan > > >>> On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <[email protected]> > > >>> wrote: > > >>> If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it > > >> would > > >>> be a huge help if we could do that. A couple different reasons: > > >>> 1. Ease of development. The workaround described in BUILDING.md adds > > >> extra > > >>> friction to getting an environment setup, and it has to be redone for > > >> each > > >>> patch release of HBase an engineer needs to work with. It's confusing > > for > > >>> both new contributors and experienced ones who don't usually work > with > > >> 5.x > > >>> or haven't lately. > > >>> 2. Ease of upgrade. At my dayjob, we try to avoid upgrading > > combinations > > >> of > > >>> Hadoop, HBase, and/or Phoenix at the same time to make upgrade > testing > > >> and > > >>> debugging simpler. HBase 2.x and Phoenix 5.x have to be done in > > lockstep, > > >>> and there's no avoiding that. But since HBase 1.x doesn't support > > Hadoop > > >> 3, > > >>> and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that > an > > >>> upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading > Hadoop, > > >>> HBase and Phoenix to new major versions simultaneously. It would be > > >> really > > >>> nice to do HBase / Phoenix first, and then Hadoop later. > > >>> Geoffrey > > >>> On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth > <[email protected] > > > > > >>> wrote: > > >>>> Hi! > > >>>> For the first part of your question: Yes, you are correct. > > >>>> I think that there is no inherent technical issue that would make it > > >>>> impossible to support Hbase 2 on Hadoop 2 for Phoenix. > > >>>> While this happened before my time, I guess that supporting Hadoop 3 > > >> only > > >>>> was a decision that partly stemmed from the then current Phoenix > maven > > >>>> setup, > > >>>> which wasn't equipped to support multiple profiles (see the multiple > > >>>> branches for the HBase 1.x versions), and partly because this was > the > > >>>> configuration that > > >>>> the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested > in. > > >>>> I don't know how much work and additional complexity it would take > to > > >> add > > >>>> Hadoop2 support for master. > > >>>> My guess is that it would need some major changes on the maven side > > for > > >>>> building, packaging, testing, and docs, > > >>>> but probably little in the way of Java code changes. > > >>>> It would also make providing binaries for Phoenix, connectors and > PQS > > >>> even > > >>>> more hairy than now. > > >>>> If anyone has more historical context, or a better idea of what it > > >> would > > >>>> take to do this, please let us know! > > >>>> Istvan > > >>>> On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <[email protected]> > > >> wrote: > > >>>>> Hi, > > >>>>>> However, up to now, we could use these artifacts with Hadoop 3.0 > > >> and > > >>>> they > > >>>>>> worked. (Or we just didn't hit the incompatibilities in our test > > >>>> suite.) > > >>>>> I have a question reg Phoenix 5.x supporting Hadoop 3 only. > > >>>>> I believe this means Phoenix 5.x cannot work with HBase 2.x that > > >>>>> is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 > > >>>>> jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). > > >>>>> Is this correct? > > >>>>> If this is correct, shall we not support Hadoop 2 because HBase 2 > > >>>>> still has default Hadoop profile: 2.0? Moreover, what is the root > > >>>>> cause behind Phoenix 5.x only supporting Hadoop 3.x? > > >>>>> As per the compatibility guidelines [1], HBase 2.3 is fully > > >> compatible > > >>>>> with Hadoop 2.10.x. > > >>>>> Apologies for these questions might be repeated ones but > > >>>>> somehow I still could not trace back the root cause. > > >>>>> 1. https://hbase.apache.org/book.html#basic.prerequisites > > >>>>> On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > > >>>>>> Hi! > > >>>>>> (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for > > >>> this, > > >>>>> but I > > >>>>>> am repeating it here for greater visibility) > > >>>>>> The HBase artifacts downloadable from maven central are built with > > >>>> Hadoop > > >>>>>> 2.x > > >>>>>> However, up to now, we could use these artifacts with Hadoop 3.0 > > >> and > > >>>> they > > >>>>>> worked. (Or we just didn't hit the incompatibilities in our test > > >>>> suite.) > > >>>>>> This seems to have changed with 2.2.5, as the public maven > artifact > > >>>>> doesn't > > >>>>>> work with Hadoop 3.0.3 or 3.1.2 . > > >>>>>> This is a known issue in HBase, and documented, but this means > that > > >>>>>> Any client JAR we'd build with Hbase 2.2.5 would have the same > > >>> problem. > > >>>>>> We cannot run our tests with Hbase 2.2.5 > > >>>>>> HBase's suggested solution is to rebuild HBase with the Hadoop > > >>> version > > >>>>> used > > >>>>>> in the cluster, and use that. This, however, doesn't fit into our > > >>> test > > >>>> or > > >>>>>> distribution process. > > >>>>>> I'm looking for your input on how to fix this. > > >>>>>> The options I could think of are: > > >>>>>> - Ask HBase to fix it for us (it doesn't seem to be a priority) > > >>>>>> - HBase could publish artifacts for Hadoop3 > > >>>>>> - Or could try to stick the common subset of Hadoop APIs > > >>>>>> - Re-publish the official HBase releases built with Hadoop 3 under > > >>> the > > >>>>>> org.apache.phoenix groupId > > >>>>>> - Hack up our build system to rebuild HBase from source, and use > > >> that > > >>>> for > > >>>>>> Phoenix > > >>>>>> None of these are particularly compelling, but I expect that this > > >>> will > > >>>>> have > > >>>>>> to be solved somehow if we want to keep up with future Hadoop > > >>> releases. > > >>>>>> Looking forward to hearing your ideas and opinion on this > > >>>>>> Istvan > > >>>> -- > > >>>> *István Tóth* | Staff Software Engineer > > >>>> [email protected] <https://www.cloudera.com> > > >>>> [image: Cloudera] <https://www.cloudera.com/> > > >>>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > >>>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > >>> Cloudera > > >>>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >>>> <https://www.cloudera.com/> > > >>>> ------------------------------ > > > > > -- > *István Tóth* | Staff Software Engineer > [email protected] <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > <https://www.cloudera.com/> > ------------------------------ > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
