[jira] [Created] (FLINK-18362) Shipping jdk by shipping archive
Noah created FLINK-18362: Summary: Shipping jdk by shipping archive Key: FLINK-18362 URL: https://issues.apache.org/jira/browse/FLINK-18362 Project: Flink Issue Type: Wish Affects Versions: 1.10.1 Reporter: Noah Hello, Our team are running flink cluster on YARN, and it works so well h4. Functional requirements Is there any option to ship archive to YARN applications? h4. Backgrounds Recently, one of job has been shut down with jdk8 version related issues. https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699 It's easy problem if we could set latest jdk on `containerized.taskmanager.env.JAVA_HOME`. However, cluster administrator said it's difficult to install the latest jdk on all cluster machines. So, we planned to run a job on latest jdk that is shipped via shared resources. There's an option `yarn.ship-directories` but it's quite slow because jdk has large number of files. If Flink supports to ship archive such as `yarn.ship-archive`, we can ship jdk archive to remote machines and use shipped jdk location as JAVA_HOME (using `yarn.container-start-command-template` ) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16527) hadoop.version of flink-shaded-hadoop-2-uber should be checked
Noah created FLINK-16527:
Summary: hadoop.version of flink-shaded-hadoop-2-uber should be
checked
Key: FLINK-16527
URL: https://issues.apache.org/jira/browse/FLINK-16527
Project: Flink
Issue Type: Bug
Reporter: Noah
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/
several pom of above path seem to have wrong property value,
`hadoop.version` with `2.4.1`
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.6.5-7.0/flink-shaded-hadoop-2-uber-2.6.5-7.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.6.5-8.0/flink-shaded-hadoop-2-uber-2.6.5-8.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.6.5-9.0/flink-shaded-hadoop-2-uber-2.6.5-9.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-8.0/flink-shaded-hadoop-2-uber-2.7.5-8.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-9.0/flink-shaded-hadoop-2-uber-2.7.5-9.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-8.0/flink-shaded-hadoop-2-uber-2.8.3-8.0.pom
https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-9.0/flink-shaded-hadoop-2-uber-2.8.3-9.0.pom
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/hive/#depedencies
Documentation recommended `2.6.5-8.0` that cannot be built.
It makes below errors.
{code:groovy}
Warning:project ':streaming-flink': Unable to build Scala project
configuration
Details:
org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException:
Could not resolve all files for configuration
':streaming-flink:compileClasspath'.
Caused by: org.gradle.internal.resolve.ModuleVersionResolveException: Could not
resolve org.apache.flink:flink-shaded-hadoop-2-uber:2.7.5-7.0.
Required by:
project :streaming-flink
Caused by: org.gradle.internal.resolve.ModuleVersionResolveException: Could not
resolve org.apache.flink:flink-shaded-hadoop-2-uber:2.7.5-7.0.
Caused by:
org.gradle.api.internal.artifacts.ivyservice.ivyresolve.parser.MetaDataParseException:
inconsistent module metadata found. Descriptor:
org.apache.flink:flink-shaded-hadoop-2-uber:2.4.1-7.0 Errors: bad version:
expected='2.7.5-7.0' found='2.4.1-7.0'
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
