[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015646#comment-16015646 ] Robert Metzger commented on FLINK-6031: --- Merged for master (1.4) in http://git-wip-us.apache.org/repos/asf/flink/commit/4a314a80 Merged for 1.3 in: http://git-wip-us.apache.org/repos/asf/flink/commit/ca3e403e > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.3.0, 1.4.0 > > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015644#comment-16015644 ] ASF GitHub Bot commented on FLINK-6031: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3931 > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015640#comment-16015640 ] ASF GitHub Bot commented on FLINK-6031: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3931 I'll merge the change now ... > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015540#comment-16015540 ] ASF GitHub Bot commented on FLINK-6031: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3931 (for both 1.3 and 1.4) > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015539#comment-16015539 ] ASF GitHub Bot commented on FLINK-6031: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3931 Thank you for addressing my comments so quickly. The change is good to merge. > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015525#comment-16015525 ] ASF GitHub Bot commented on FLINK-6031: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3931 @rmetzger I've addressed your comments. > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[
https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015509#comment-16015509
]
ASF GitHub Bot commented on FLINK-6031:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3931#discussion_r117204729
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -665,58 +685,52 @@ public ApplicationReport startAppMaster(JobGraph
jobGraph, YarnClient yarnClient
1));
}
+ String configuredUserJarInclusion =
flinkConfiguration.getString(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR);
+ YarnConfigOptions.UserJarInclusion userJarInclusion;
+ try {
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(configuredUserJarInclusion.toUpperCase());
+ } catch (IllegalArgumentException e) {
+ LOG.warn("Configuration parameter {} was configured
with an invalid value {}. Falling back to default ({}).",
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.key(),
+ configuredUserJarInclusion,
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
--- End diff --
this shouldn't be here actually.
> Add parameter for per job yarn clusters to control whether the user code jar
> is included into the system classloader.
> -
>
> Key: FLINK-6031
> URL: https://issues.apache.org/jira/browse/FLINK-6031
> Project: Flink
> Issue Type: Bug
> Components: YARN
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Critical
>
> FLINK-4913 added the user jar into the system classloader, when starting a
> Flink per job YARN cluster.
> Some users were experiencing issues with the changed behavior.
> I suggest to introduce a new yarn specific configuration parameter (for the
> flink-conf.yaml file) to control if the user jar is added into system
> classloader.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015505#comment-16015505 ] ASF GitHub Bot commented on FLINK-6031: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3931#discussion_r117204403 --- Diff: docs/setup/yarn_setup.md --- @@ -245,6 +245,18 @@ Note: You can use a different configuration directory per job by setting the env Note: It is possible to combine `-m yarn-cluster` with a detached YARN submission (`-yd`) to "fire and forget" a Flink job to the YARN cluster. In this case, your application will not get any accumulator results or exceptions from the ExecutionEnvironment.execute() call! +### User jars & Classpath + +By default Flink will include the user jars into the system classpath when running a single job. This behavior can be controlled with the `yarn.per-job-cluster.include-job-jar` parameter. --- End diff -- The configuration parameter is actually called `yarn.per-job-cluster.include-user-jar`. > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[
https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015490#comment-16015490
]
ASF GitHub Bot commented on FLINK-6031:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3931#discussion_r117199176
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -665,58 +685,52 @@ public ApplicationReport startAppMaster(JobGraph
jobGraph, YarnClient yarnClient
1));
}
+ String configuredUserJarInclusion =
flinkConfiguration.getString(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR);
+ YarnConfigOptions.UserJarInclusion userJarInclusion;
+ try {
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(configuredUserJarInclusion.toUpperCase());
+ } catch (IllegalArgumentException e) {
+ LOG.warn("Configuration parameter {} was configured
with an invalid value {}. Falling back to default ({}).",
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.key(),
+ configuredUserJarInclusion,
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
--- End diff --
This also seems very similar to the other duplicates.
> Add parameter for per job yarn clusters to control whether the user code jar
> is included into the system classloader.
> -
>
> Key: FLINK-6031
> URL: https://issues.apache.org/jira/browse/FLINK-6031
> Project: Flink
> Issue Type: Bug
> Components: YARN
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Critical
>
> FLINK-4913 added the user jar into the system classloader, when starting a
> Flink per job YARN cluster.
> Some users were experiencing issues with the changed behavior.
> I suggest to introduce a new yarn specific configuration parameter (for the
> flink-conf.yaml file) to control if the user jar is added into system
> classloader.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015491#comment-16015491 ] ASF GitHub Bot commented on FLINK-6031: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3931#discussion_r117199375 --- Diff: docs/setup/yarn_setup.md --- @@ -245,6 +245,18 @@ Note: You can use a different configuration directory per job by setting the env Note: It is possible to combine `-m yarn-cluster` with a detached YARN submission (`-yd`) to "fire and forget" a Flink job to the YARN cluster. In this case, your application will not get any accumulator results or exceptions from the ExecutionEnvironment.execute() call! +### User jars & Classpath + +By default Flink will include the user jars into the system classpath when running a single job. This behavior can be controlled with the `yarn.per-job-cluster.include-job-jar` parameter. + +When setting this to `DISABLED` Flink will include the jar in the user classpath instead. + +The user-jars position in the class path can be controlled by setting the parameter to one of the following: + +- `ORDER`: (default) Adds the jar to the system class path based on the lexicographic order. +- `FIRST`: Adds the jar to the beginning of the system class path. +- `LAST`: Adds the jar to the end of the system class path. + --- End diff -- Ideally, you put the configuration property also into the `setup/config.md` page, where all config options are listed. > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[
https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015492#comment-16015492
]
ASF GitHub Bot commented on FLINK-6031:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3931#discussion_r117198446
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
---
@@ -200,6 +214,17 @@ public void setTaskManagerMemory(int memoryMb) {
public void
setFlinkConfiguration(org.apache.flink.configuration.Configuration conf) {
this.flinkConfiguration = conf;
+
+ String configuredUserJarInclusion =
flinkConfiguration.getString(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR);
+ try {
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(configuredUserJarInclusion.toUpperCase());
+ } catch (IllegalArgumentException e) {
+ LOG.warn("Configuration parameter {} was configured
with an invalid value {}. Falling back to default ({}).",
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.key(),
+ configuredUserJarInclusion,
+
YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ userJarInclusion =
YarnConfigOptions.UserJarInclusion.valueOf(YarnConfigOptions.CLASSPATH_INCLUDE_USER_JAR.defaultValue());
+ }
--- End diff --
This code and the code above looks like exactly the same.
If this is duplicate code, it should be extracted into one method?
> Add parameter for per job yarn clusters to control whether the user code jar
> is included into the system classloader.
> -
>
> Key: FLINK-6031
> URL: https://issues.apache.org/jira/browse/FLINK-6031
> Project: Flink
> Issue Type: Bug
> Components: YARN
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Critical
>
> FLINK-4913 added the user jar into the system classloader, when starting a
> Flink per job YARN cluster.
> Some users were experiencing issues with the changed behavior.
> I suggest to introduce a new yarn specific configuration parameter (for the
> flink-conf.yaml file) to control if the user jar is added into system
> classloader.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[
https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014338#comment-16014338
]
ASF GitHub Bot commented on FLINK-6031:
---
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/3931
[FLINK-6031][yarn] Add config parameter for user-jar inclusion in cla…
This PR adds a config parameter to control how user-jars are being handled
in regards ot the system class path for per-job yarn clusters.
The parameter allows:
* to disable the inclusion in the system classpath and use the user
classpath instead ("DISABLE")
* prepend the user jars to the system class path ("FIRST")
* append the user jars to the system class path ("LAST")
* (default) add the user jars to the system class path based on the
lexicographic order ("ORDER")
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 6031_yarn_userjars
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3931.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3931
> Add parameter for per job yarn clusters to control whether the user code jar
> is included into the system classloader.
> -
>
> Key: FLINK-6031
> URL: https://issues.apache.org/jira/browse/FLINK-6031
> Project: Flink
> Issue Type: Bug
> Components: YARN
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Critical
>
> FLINK-4913 added the user jar into the system classloader, when starting a
> Flink per job YARN cluster.
> Some users were experiencing issues with the changed behavior.
> I suggest to introduce a new yarn specific configuration parameter (for the
> flink-conf.yaml file) to control if the user jar is added into system
> classloader.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013762#comment-16013762 ] Robert Metzger commented on FLINK-6031: --- I responded on the BEAM issue. Back to this one: >From the ML discussion, it seems that we need to have the following three >options for controlling the user jar inclusion: - "user code first" or "user code last" in the classpath - no user code in the classpath > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6031) Add parameter for per job yarn clusters to control whether the user code jar is included into the system classloader.
[ https://issues.apache.org/jira/browse/FLINK-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907567#comment-15907567 ] Aljoscha Krettek commented on FLINK-6031: - [~rmetzger] do you think this issue in the Beam Flink Runner could be cause by this: https://issues.apache.org/jira/browse/BEAM-1640? > Add parameter for per job yarn clusters to control whether the user code jar > is included into the system classloader. > - > > Key: FLINK-6031 > URL: https://issues.apache.org/jira/browse/FLINK-6031 > Project: Flink > Issue Type: Bug > Components: YARN >Reporter: Robert Metzger >Priority: Critical > > FLINK-4913 added the user jar into the system classloader, when starting a > Flink per job YARN cluster. > Some users were experiencing issues with the changed behavior. > I suggest to introduce a new yarn specific configuration parameter (for the > flink-conf.yaml file) to control if the user jar is added into system > classloader. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
