[jira] [Commented] (FLINK-14626) User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn
[
https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969267#comment-16969267
]
Chesnay Schepler commented on FLINK-14626:
--
This would change existing behavior and could result in subtle bugs. I'm also a
bit concerned about making the existing YARN code even more complicated (it is
a minefield for bugs as is), in particular without a use-case that can't be
summed up as "user error".
> User jar packaged with hadoop dependencies may cause class conflit with
> hadoop jars on yarn
> ---
>
> Key: FLINK-14626
> URL: https://issues.apache.org/jira/browse/FLINK-14626
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.9.1
>Reporter: Victor Wong
>Priority: Major
>
> Currently, the yarn application classpath is placed behind Flink classpath
> (including user jars), which will cause conflict if the user jar accidentally
> included hadoop dependencies.
> {code:java}
> // org.apache.flink.yarn.Utils#setupYarnClassPath
> public static void setupYarnClassPath(Configuration conf, Map
> appMasterEnv) {
>addToEnvironment(
> appMasterEnv,
> Environment.CLASSPATH.name(),
> appMasterEnv.get(ENV_FLINK_CLASSPATH));
>String[] applicationClassPathEntries = conf.getStrings(
> YarnConfiguration.YARN_APPLICATION_CLASSPATH,
> YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH);
>for (String c : applicationClassPathEntries) {
> addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim());
>}
> }
> {code}
> Maybe we should place the user jars behind yarn application classpath when
> `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is
> set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar".
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14626) User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn
[
https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969388#comment-16969388
]
Victor Wong commented on FLINK-14626:
-
[~chesnay], thanks for your quick response. As an engineer providing
company-wide stream processing capabilities for other teams, I have seen quite
a few errors related to this issue. It's really prone to including Hadoop
dependencies in the uber jar, especially when your job deals with Hadoop
systems, e.g. HBase.
But I agree that it _would change existing behavior and could result in subtle
bugs,_ so I will close this PR and find another way to solve the related
problems.
> User jar packaged with hadoop dependencies may cause class conflit with
> hadoop jars on yarn
> ---
>
> Key: FLINK-14626
> URL: https://issues.apache.org/jira/browse/FLINK-14626
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.9.1
>Reporter: Victor Wong
>Priority: Major
>
> Currently, the yarn application classpath is placed behind Flink classpath
> (including user jars), which will cause conflict if the user jar accidentally
> included hadoop dependencies.
> {code:java}
> // org.apache.flink.yarn.Utils#setupYarnClassPath
> public static void setupYarnClassPath(Configuration conf, Map
> appMasterEnv) {
>addToEnvironment(
> appMasterEnv,
> Environment.CLASSPATH.name(),
> appMasterEnv.get(ENV_FLINK_CLASSPATH));
>String[] applicationClassPathEntries = conf.getStrings(
> YarnConfiguration.YARN_APPLICATION_CLASSPATH,
> YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH);
>for (String c : applicationClassPathEntries) {
> addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim());
>}
> }
> {code}
> Maybe we should place the user jars behind yarn application classpath when
> `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is
> set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar".
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
