[
https://issues.apache.org/jira/browse/FLINK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391546#comment-17391546
]
Gabor Somogyi commented on FLINK-23449:
---------------------------------------
[~yuchuanchen] I've read through the conversation on all places and honestly I
don't understand why Flink code change is needed at all?!
There are multiple possible solutions to solve the problem. One is a simple
configuration change, the other is different application packaging.
I've just tried out them and both worked like charm. What blocks you to use
the mentioned suggestions?
Related
{quote}not very convenient to use
{quote}
can you share exact code lines what is not convenient? That would help the
evaluation.
Having my current knowledge there is the "yarn.application.classpath" where all
dependencies can be added.
Introducing further variables which would do the same but with higher code
complexity is something I wouldn't add.
What I can imagine is fixing/extending the documentation.
> YarnTaskExecutorRunner does not contains MapReduce classes
> -----------------------------------------------------------
>
> Key: FLINK-23449
> URL: https://issues.apache.org/jira/browse/FLINK-23449
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Deployment / YARN
> Affects Versions: 1.11.3
> Environment: flink-1.11
> flink on yarn cluster
> jdk1.8
> hive1.2.1
> hadoop2.7
> hadoop classes is provided with {{export HADOOP_CLASSPATH=`hadoop classpath`
> when submitting test APP. （described in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html]
> ）}}
> {{}}
> Reporter: Kai Chen
> Priority: Major
> Labels: pull-request-available
>
> I followed instructions described in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive]
> and tested hive streaming sink, met this exception
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConf
> [http://apache-flink.147419.n8.nabble.com/Flink-td7866.html] met the same
> problem.
>
> I checked TM jvm envs and the code and found that flink only set up
> YARN_APPLICATION_CLASSPATH, but without MAPREDUCE_APPLICATION_CLASSPATH.
> See:
> [https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119]
>
> I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as
> spark does.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
