[ https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969388#comment-16969388 ]
Victor Wong commented on FLINK-14626: ------------------------------------- [~chesnay], thanks for your quick response. As an engineer providing company-wide stream processing capabilities for other teams, I have seen quite a few errors related to this issue. It's really prone to including Hadoop dependencies in the uber jar, especially when your job deals with Hadoop systems, e.g. HBase. But I agree that it _would change existing behavior and could result in subtle bugs,_ so I will close this PR and find another way to solve the related problems. > User jar packaged with hadoop dependencies may cause class conflit with > hadoop jars on yarn > ------------------------------------------------------------------------------------------- > > Key: FLINK-14626 > URL: https://issues.apache.org/jira/browse/FLINK-14626 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN > Affects Versions: 1.9.1 > Reporter: Victor Wong > Priority: Major > > Currently, the yarn application classpath is placed behind Flink classpath > (including user jars), which will cause conflict if the user jar accidentally > included hadoop dependencies. > {code:java} > // org.apache.flink.yarn.Utils#setupYarnClassPath > public static void setupYarnClassPath(Configuration conf, Map<String, String> > appMasterEnv) { > addToEnvironment( > appMasterEnv, > Environment.CLASSPATH.name(), > appMasterEnv.get(ENV_FLINK_CLASSPATH)); > String[] applicationClassPathEntries = conf.getStrings( > YarnConfiguration.YARN_APPLICATION_CLASSPATH, > YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH); > for (String c : applicationClassPathEntries) { > addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim()); > } > } > {code} > Maybe we should place the user jars behind yarn application classpath when > `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is > set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar". > -- This message was sent by Atlassian Jira (v8.3.4#803005)