[ https://issues.apache.org/jira/browse/FLINK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392680#comment-17392680 ]
Kai Chen edited comment on FLINK-23449 at 8/4/21, 3:10 AM: ----------------------------------------------------------- [~gaborgsomogyi] [~trohrmann] I'm not stick to code change. It was just a little uncompatible and confusing when upgrading from flink-1.10 to upper versions when I followed official guide and used flink hive connector. Flink-1.10 still uses flink-shaded-hadoop-2-uber.jar which includes hadoop-mapreduce-client-core.jar. And starting from Flink 1.11, using flink-shaded-hadoop-2-uber releases is not officially supported by the Flink project and thus hadoop-mapreduce-client-core.jar is not included in JM/TM classpath. In the other hand, users follow [the documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/yarn] , set up HADOOP_CLASSPATH and usually take it for granted that HADOOP_CLASSPATH is added to JM/TM classpath. That's why I changed the code and added MAPREDUCE_APPLICATION_CLASSPATH. As I said, I'm not stick to code change. An updating/extending documentation looks good to me, too. was (Author: yuchuanchen): [~gaborgsomogyi] [~trohrmann] I'm not stick to code change. It was just a little uncompatible and confusing when upgrading from flink-1.10 to upper versions when I followed official guide and used flink hive connector. Flink-1.10 still uses flink-shaded-hadoop-2-uber.jar which includes hadoop-mapreduce-client-core.jar. And starting from Flink 1.11, using flink-shaded-hadoop-2-uber releases is not officially supported by the Flink project and thus hadoop-mapreduce-client-core.jar is not included in JM/TM classpath. In the other hand, users follow [the documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/yarn] , set up HADOOP_CLASSPATH and usually take it for granted that HADOOP_CLASSPATH is added to JM/TM classpath. That's why I changed the code and added MAPREDUCE_APPLICATION_CLASSPATH. As I said, I'm not stick to code change. A better documentation looks good to me, too. > YarnTaskExecutorRunner does not contains MapReduce classes > ----------------------------------------------------------- > > Key: FLINK-23449 > URL: https://issues.apache.org/jira/browse/FLINK-23449 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Deployment / YARN > Affects Versions: 1.11.3 > Environment: flink-1.11 > flink on yarn cluster > jdk1.8 > hive1.2.1 > hadoop2.7 > hadoop classes is provided with {{export HADOOP_CLASSPATH=`hadoop classpath` > when submitting test APP. (described in > [https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html] > )}} > {{}} > Reporter: Kai Chen > Priority: Major > Labels: pull-request-available > > I followed instructions described in > [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive] > and tested hive streaming sink, met this exception > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.mapred.JobConf > [http://apache-flink.147419.n8.nabble.com/Flink-td7866.html] met the same > problem. > > I checked TM jvm envs and the code and found that flink only set up > YARN_APPLICATION_CLASSPATH, but without MAPREDUCE_APPLICATION_CLASSPATH. > See: > [https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119] > > I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as > spark does. -- This message was sent by Atlassian Jira (v8.3.4#803005)