[
https://issues.apache.org/jira/browse/FLINK-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414427#comment-16414427
]
ASF GitHub Bot commented on FLINK-9086:
---------------------------------------
GitHub user ruankd opened a pull request:
https://github.com/apache/flink/pull/5768
[FLINK-9086] Use INTERNAL_HADOOP_CLASSPATHS as classpath for Yarn Session
## What is the purpose of the change
* This change improves user experience for Flink on some environments such
as AWS, which don't have hadoop jars in `HADOOP_CLASSPATH` but has `hadoop
classpath` result set in `INTERNAL_HADOOP_CLASSPATHS` by `config.sh`. With this
commit, users don't have to set `HADOOP_CLASSPATH` manually.
## Brief change log
- Change `HADOOP_CLASSPATH` with `INTERNAL_HADOOP_CLASSPATHS` in
`start-scala-shell.sh`.
## Verifying this change
- Test on AWS environment to start a Flink Yarn cluster for Scala Shell.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ruankd/flink FLINK-9086
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5768.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5768
----
commit 8f0aa7a9a3cb9547463a1b0cbe480a055a55aeb0
Author: Keda Ruan <kedarua@...>
Date: 2018-03-26T19:14:05Z
[FLINK-9086] Use INTERNAL_HADOOP_CLASSPATHS as classpath for launching Yarn
cluster for Scala Shell
----
> Use INTERNAL_HADOOP_CLASSPATHS as classpath for launching Yarn cluster for
> Scala Shell
> --------------------------------------------------------------------------------------
>
> Key: FLINK-9086
> URL: https://issues.apache.org/jira/browse/FLINK-9086
> Project: Flink
> Issue Type: Improvement
> Components: Startup Shell Scripts
> Affects Versions: 1.4.0
> Reporter: Keda Ruan
> Priority: Major
>
> Some environments don't have hadoop jars in {{HADOOP_CLASSPATH}}, causing
> Flink Yarn cluster unable to start due to some jars missing. For example
> Flink 1.4.0 drops jersey dependency in its shaded hadoop jar, causing Yarn
> cluster unable to start:
> {quote}Exception in thread "main" java.lang.NoClassDefFoundError:
> com/sun/jersey/core/util/FeaturesAndProperties
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> {quote}
> Since there is a {{INTERNAL_HADOOP_CLASSPATHS}} in {{config.sh}} that has
> {{hadoop classpath}} result, we can utilize this and improve the user
> experience on starting Flink Yarn cluster for Scala Shell.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
