Hi All, Looks like we are out of the woods for now (so we think) - we went with Hadoop free version and relied on client libraries on edge node.
However, I am still not very confident as I started digging into that stack as well and realized what Till pointed out (traces leads to a class that is part of 2.9). I did dig around env variables and nothing was set. This is a brand new clustered installed a week back and our team is literally the first hands on deck. I will fish around and see if Hortonworks back-ported something for HDP (dots are still not completely connected but nonetheless, we have a test session and app running in our brand new Prod) Thanks, Ashish > On Mar 22, 2018, at 4:47 AM, Till Rohrmann <trohrm...@apache.org> wrote: > > Hi Ashish, > > the class `RequestHedgingRMFailoverProxyProvider` was only introduced with > Hadoop 2.9.0. My suspicion is thus that you start the client with some Hadoop > 2.9.0 dependencies on the class path. Could you please check the logs of the > client what's on its class path? Maybe you could also share the logs with us. > Please also check whether HADOOP_CLASSPATH is set to something suspicious. > > Thanks a lot! > > Cheers, > Till > > On Wed, Mar 21, 2018 at 6:25 PM, ashish pok <ashish...@yahoo.com > <mailto:ashish...@yahoo.com>> wrote: > Hi Piotrek, > > At this point we are simply trying to start a YARN session. > > BTW, we are on Hortonworks HDP 2.6 which is on 2.7 Hadoop if anyone has > experienced similar issues. > > We actually pulled 2.6 binaries for the heck of it and ran into same issues. > > I guess we are left with getting non-hadoop binaries and set HADOOP_CLASSPATH > then? > > -- Ashish > > On Wed, Mar 21, 2018 at 12:03 PM, Piotr Nowojski > <pi...@data-artisans.com <mailto:pi...@data-artisans.com>> wrote: > Hi, > > > Does some simple word count example works on the cluster after the upgrade? > > If not, maybe your job is pulling some dependency that’s causing this version > conflict? > > Piotrek > >> On 21 Mar 2018, at 16:52, ashish pok <ashish...@yahoo.com >> <mailto:ashish...@yahoo.com>> wrote: >> >> Hi Piotrek, >> >> Yes, this is a brand new Prod environment. 2.6 was in our lab. >> >> Thanks, >> >> -- Ashish >> >> On Wed, Mar 21, 2018 at 11:39 AM, Piotr Nowojski >> <pi...@data-artisans.com <mailto:pi...@data-artisans.com>> wrote: >> Hi, >> >> Have you replaced all of your old Flink binaries with freshly downloaded >> <https://flink.apache.org/downloads.html> Hadoop 2.7 versions? Are you sure >> that something hasn't mix in the process? >> >> Does some simple word count example works on the cluster after the upgrade? >> >> Piotrek >> >>> On 21 Mar 2018, at 16:11, ashish pok <ashish...@yahoo.com >>> <mailto:ashish...@yahoo.com>> wrote: >>> >>> Hi All, >>> >>> We ran into a roadblock in our new Hadoop environment, migrating from 2.6 >>> to 2.7. It was supposed to be an easy lift to get a YARN session but doesnt >>> seem like :) We definitely are using 2.7 binaries but it looks like there >>> is a call here to a private methos which screams runtime incompatibility. >>> >>> Anyone has seen this and have pointers? >>> >>> Thanks, Ashish >>> Exception in thread "main" java.lang.IllegalAccessError: tried to access >>> method >>> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; >>> from class >>> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider <> >>> at >>> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75) >>> at >>> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163) >>> at >>> org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) >>> at >>> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) >>> at >>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) >>> at >>> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) >>> at >>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:314) >>> at >>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:417) >>> at >>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:367) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:679) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:514) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:511) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) >>> at >>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:511) >>> >> > >
