Have you tried to just export Hadoop 3's classpath to `HADOOP_CLASSPATH` and see if that works out of the box?
If the main use case is HDFS access, then there is a fair chance it might just work, because Flink uses only a small subset of the Hadoop FS API which is stable between 2.x and 3.x, as far as I tried it out a while back. On Tue, Mar 3, 2020 at 6:03 PM LINZ, Arnaud <al...@bouyguestelecom.fr> wrote: > Hello, > > Have you shared it somewhere on the web already? > > Best, > > Arnaud > > > > *De :* vino yang <yanghua1...@gmail.com> > *Envoyé :* mercredi 4 décembre 2019 11:55 > *À :* Márton Balassi <balassi.mar...@gmail.com> > *Cc :* Chesnay Schepler <ches...@apache.org>; Foster, Craig < > foscr...@amazon.com>; user@flink.apache.org; d...@flink.apache.org > *Objet :* Re: Building with Hadoop 3 > > > > Hi Marton, > > > > Thanks for your explanation. Personally, I look forward to your > contribution! > > > > Best, > > Vino > > > > Márton Balassi <balassi.mar...@gmail.com> 于2019年12月4日周三 下午5:15写道： > > Wearing my Cloudera hat I can tell you that we have done this exercise for > our distros of the 3.0 and 3.1 Hadoop versions. We have not contributed > these back just yet, but we are open to do so. If the community is > interested we can contribute those changes back to flink-shaded and suggest > the necessay changes to flink too. The task was not overly complex, but it > certainly involved a bit of dependency hell. :-) > > > > Right now we are focused on internal timelines, but we could invest into > contributing this back in the end of January timeframe if the community > deems this a worthwhile effort. > > > > Best, > > Marton > > > > On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <ches...@apache.org> > wrote: > > There's no JIRA and no one actively working on it. I'm not aware of any > investigations on the matter; hence the first step would be to just try it > out. > > > > A flink-shaded artifact isn't a hard requirement; Flink will work with any > 2.X hadoop distribution (provided that there aren't any dependency clashes). > > > > On 03/12/2019 18:22, Foster, Craig wrote: > > Hi: > > I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here > from a year ago that no one is looking into Hadoop 3 support [1]. Is there > a document or JIRA that now exists which would point to what needs to be > done to support Hadoop 3? Right now builds with Hadoop 3 don’t work > obviously because there’s no flink-shaded-hadoop-3 artifacts. > > > > Thanks! > > Craig > > > > [1] https://issues.apache.org/jira/browse/FLINK-11086 > > > > > > > ------------------------------ > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. >
