[jira] [Commented] (FLINK-14626) User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn
[ https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969267#comment-16969267 ] Chesnay Schepler commented on FLINK-14626: -- This would change existing behavior and could result in subtle bugs. I'm also a bit concerned about making the existing YARN code even more complicated (it is a minefield for bugs as is), in particular without a use-case that can't be summed up as "user error". > User jar packaged with hadoop dependencies may cause class conflit with > hadoop jars on yarn > --- > > Key: FLINK-14626 > URL: https://issues.apache.org/jira/browse/FLINK-14626 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.1 >Reporter: Victor Wong >Priority: Major > > Currently, the yarn application classpath is placed behind Flink classpath > (including user jars), which will cause conflict if the user jar accidentally > included hadoop dependencies. > {code:java} > // org.apache.flink.yarn.Utils#setupYarnClassPath > public static void setupYarnClassPath(Configuration conf, Map > appMasterEnv) { >addToEnvironment( > appMasterEnv, > Environment.CLASSPATH.name(), > appMasterEnv.get(ENV_FLINK_CLASSPATH)); >String[] applicationClassPathEntries = conf.getStrings( > YarnConfiguration.YARN_APPLICATION_CLASSPATH, > YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH); >for (String c : applicationClassPathEntries) { > addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim()); >} > } > {code} > Maybe we should place the user jars behind yarn application classpath when > `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is > set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar". > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-14626) User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn
[ https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969388#comment-16969388 ] Victor Wong commented on FLINK-14626: - [~chesnay], thanks for your quick response. As an engineer providing company-wide stream processing capabilities for other teams, I have seen quite a few errors related to this issue. It's really prone to including Hadoop dependencies in the uber jar, especially when your job deals with Hadoop systems, e.g. HBase. But I agree that it _would change existing behavior and could result in subtle bugs,_ so I will close this PR and find another way to solve the related problems. > User jar packaged with hadoop dependencies may cause class conflit with > hadoop jars on yarn > --- > > Key: FLINK-14626 > URL: https://issues.apache.org/jira/browse/FLINK-14626 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.9.1 >Reporter: Victor Wong >Priority: Major > > Currently, the yarn application classpath is placed behind Flink classpath > (including user jars), which will cause conflict if the user jar accidentally > included hadoop dependencies. > {code:java} > // org.apache.flink.yarn.Utils#setupYarnClassPath > public static void setupYarnClassPath(Configuration conf, Map > appMasterEnv) { >addToEnvironment( > appMasterEnv, > Environment.CLASSPATH.name(), > appMasterEnv.get(ENV_FLINK_CLASSPATH)); >String[] applicationClassPathEntries = conf.getStrings( > YarnConfiguration.YARN_APPLICATION_CLASSPATH, > YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH); >for (String c : applicationClassPathEntries) { > addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim()); >} > } > {code} > Maybe we should place the user jars behind yarn application classpath when > `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is > set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar". > -- This message was sent by Atlassian Jira (v8.3.4#803005)