Hey Vitaliy, Check this documentation on how to use Flink with Hadoop: https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/hadoop.html For your setup, I would recommend referencing the Hadoop jars from your Hadoop vendor by setting
export HADOOP_CLASSPATH=`hadoop classpath` Is it possible that the files on your cluster are Flink 1.7.0 files, while your Flink job maven project has Flink 1.10 dependencies? On your server, what version do the flink jar files in lib/ have? If your are launching Flink like this... ./bin/flink run -m yarn-cluster -p 4 -yjm 1024m -ytm 4096m ./my/project.jar ... it will use the files in lib/ for starting Flink. Best, Robert On Mon, Mar 30, 2020 at 5:39 PM Vitaliy Semochkin <vitaliy...@gmail.com> wrote: > Hello Robert, > Thank you for quick response! > Indeed logs says the hadoop version is 2.4.1 this is probably because of > > https://github.com/apache/flink/blob/b17a597dec80e590db2beedda446aa3cae9920dd/pom.xml#L96 > How can I make 1.10 to work with my current hadoop version? > > Regarding flink reporting in logs its 1.7.0 > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting > YarnJobClusterEntrypoint (Version: 1.7.0 > while I'm using 1.10 and this is application cluster (everything is > bundled and we don't have session cluster running). > Here are the whole dependencies list: > mvn dependency:tree | grep flink | cut -d'-' -f2- > org.apache.flink:flink-yarn_2.11:jar:1.10.0:runtime > org.apache.flink:flink-clients_2.11:jar:1.10.0:compile > org.apache.flink:flink-optimizer_2.11:jar:1.10.0:compile > org.apache.flink:flink-shaded-hadoop-2:jar:2.4.1-9.0:runtime > org.apache.flink:force-shading:jar:1.10.0:compile > org.apache.flink:flink-runtime_2.11:jar:1.10.0:runtime > org.apache.flink:flink-core:jar:1.10.0:compile > org.apache.flink:flink-annotations:jar:1.10.0:compile > org.apache.flink:flink-metrics-core:jar:1.10.0:compile > org.apache.flink:flink-java:jar:1.10.0:compile > org.apache.flink:flink-queryable-state-client-java:jar:1.10.0:runtime > org.apache.flink:flink-hadoop-fs:jar:1.10.0:runtime > org.apache.flink:flink-shaded-netty:jar:4.1.39.Final-9.0:compile > org.apache.flink:flink-shaded-guava:jar:18.0-9.0:compile > org.apache.flink:flink-shaded-asm-7:jar:7.1-9.0:compile > org.apache.flink:flink-shaded-jackson:jar:2.10.1-9.0:compile > org.apache.flink:flink-jdbc_2.11:jar:1.10.0:compile > org.apache.flink:flink-hbase_2.11:jar:1.10.0:compile > org.apache.flink:flink-runtime-web_2.11:jar:1.10.0:compile > As you can see all flink related libs are 1.10. > > Can you please tell which class in flinks identifies the version(I'll try > to debug it locally)? > > Regards, > Vitaliy > > > On Mon, Mar 30, 2020 at 5:10 PM Robert Metzger <rmetz...@apache.org> > wrote: > >> Hey Vitaliy, >> is it okay for you if we keep the discussion on the list, so that others >> can chime in to help, and that Google can index the conversation, in case >> somebody else has a similar problem? >> >> I just checked, and Flink on YARN in Flink 1.10 does set the >> property correctly. Maybe in Flink 1.7, accessing the logs in the web ui >> was not yet supported. >> >> You said in your email, that you are using Flink 1.10, however, your logs >> state that you are running Flink 1.7.0. >> It also seems that you have the Hadoop 2.4.1 dependencies of Flink, but >> your Hadoop environment is Hadoop 2.7.3. I believe this error is caused by >> that version mismatch: >> >> Caused by: java.lang.IllegalAccessError: tried to access method >>> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; >>> from class >>> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider >> >> >> >> >> On Mon, Mar 30, 2020 at 1:58 PM Vitaliy Semochkin <vitaliy...@gmail.com> >> wrote: >> >>> Hello Robert, >>> >Where exactly are you seeing the "Log file environment variable >>> 'log.file' is not set." message? >>> I get this message when I check yarn logs. >>> >>> >Can you post some context around it? (is this shown from the command >>> line? what are the arguments? is it shown in a file? >>> I'm creating an application cluster from a java application using flink >>> 1.10. I didn't have this issue with 1.8.1 version, >>> however when I upgraded configuration slightly changed, e.g. >>> ClusterSpecification taskManagerMemoryMB is now ignored, and should be set >>> via fink Configuration. >>> (though it's value still validated, but no longer used after that). >>> My main issue, is that Application seems to fail to start properly, it >>> seems that JobMaster fails to connect to ResourceManager, but I can't >>> figure out why. >>> The yarn log is attached. >>> >>> I'll appreciate if you tell me to which direction I should dig. >>> >>> Regards, >>> Vitaliy >>> >>> >>> On Mon, Mar 30, 2020 at 12:00 PM Robert Metzger <rmetz...@apache.org> >>> wrote: >>> >>>> Hey, >>>> which Flink version are you using? >>>> >>>> Where exactly are you seeing the "Log file environment variable >>>> 'log.file' is not set." message? Can you post some context around it? (is >>>> this shown from the command line? what are the arguments? is it shown in a >>>> file? >>>> >>>> Usually, the "log.file" property is used to pass the name of the log >>>> file into the log4j configuration. If this property is not set, I have to >>>> assume that you are using modified or custom scripts, or you are executing >>>> Flink in an environment that fails to set the property. >>>> >>>> When running Flink on YARN, the JobManager logs are stored on the >>>> machine running the JobManager. The logs accessible through "yarn logs" are >>>> the same as you would see in the JM interface. >>>> >>>> Best, >>>> Robert >>>> >>>> >>>> >>>> On Sun, Mar 29, 2020 at 11:22 PM Vitaliy Semochkin < >>>> vitaliy...@gmail.com> wrote: >>>> >>>>> Hello Yun, >>>>> >>>>> I see this error reported by: >>>>> *org.apache.flink.runtime.webmonitor.WebMonitorUtils* - *JobManager >>>>> log files are unavailable in the web dashboard. Log file location not >>>>> found >>>>> in environment variable 'log.file' or configuration key 'Key: >>>>> 'web.log.path' , default: null (fallback keys: >>>>> [{key=jobmanager.web.log.path, isDeprecated=true}])'.* >>>>> >>>>> I wonder where the JobManager files are stored in case running on a >>>>> YARN cluster? >>>>> Are these logs same to those I get via yarn logs -applicationId? >>>>> >>>>> Regards, >>>>> Vitaliy >>>>> >>>>> >>>>> >>>>> On Sun, Mar 29, 2020 at 8:24 PM Yun Tang <myas...@live.com> wrote: >>>>> >>>>>> Hi Vitaliy >>>>>> >>>>>> Property of 'log.file' would be configured if you have uploaded >>>>>> 'logback.xml' or 'log4j.properties' [1]. >>>>>> The file would contain logs of job manager or task manager which is >>>>>> decided by the component itself. And as you can see, this is only a local >>>>>> file path, I am afraid this cannot understand hdfs paths. >>>>>> >>>>>> >>>>>> [1] >>>>>> https://github.com/apache/flink/blob/ae3b0ff80b93a83a358ab474060473863d2c30d6/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L420 >>>>>> >>>>>> Best >>>>>> Yun Tang >>>>>> ------------------------------ >>>>>> *From:* Vitaliy Semochkin <vitaliy...@gmail.com> >>>>>> *Sent:* Sunday, March 29, 2020 4:32 >>>>>> *To:* user <user@flink.apache.org> >>>>>> *Subject:* Log file environment variable 'log.file' is not set. >>>>>> >>>>>> Hi, >>>>>> >>>>>> When I launch Flink Application Cluster I keep getting a message >>>>>> " Log file environment variable 'log.file' is not set." >>>>>> >>>>>> I use console logging via log4j >>>>>> and I read logs via yarn logs -applicationId .... >>>>>> >>>>>> What's the purpose of log.file property? >>>>>> What this file will contain and on which host should I search for the >>>>>> log? >>>>>> Does this property understands hdfs paths? >>>>>> >>>>>> Regards, >>>>>> Vitaliy >>>>>> >>>>>
