[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/6140 ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199817183 --- Diff: flink-scala-shell/start-script/start-scala-shell.sh --- @@ -19,6 +19,9 @@ # from scala-lang 2.10.4 +# Uncomment the following line to enable remote debug +# export FLINK_SCALA_SHELL_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" --- End diff -- Created PR #6245 for it ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199813552 --- Diff: flink-scala-shell/start-script/start-scala-shell.sh --- @@ -19,6 +19,9 @@ # from scala-lang 2.10.4 +# Uncomment the following line to enable remote debug +# export FLINK_SCALA_SHELL_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" --- End diff -- Created FLINK-9718 for it ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199760246 --- Diff: flink-scala-shell/start-script/start-scala-shell.sh --- @@ -19,6 +19,9 @@ # from scala-lang 2.10.4 +# Uncomment the following line to enable remote debug +# export FLINK_SCALA_SHELL_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" --- End diff -- please move this into a separate commit ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199760506 --- Diff: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java --- @@ -166,6 +166,10 @@ public Configuration getConfiguration() { return copiedConfiguration; } + public static Options getCustomCommandLineOptions() { --- End diff -- it should be possible for this getter to be non-static. Then the changes to the `customCommandLineOptions` field are unnecessary, and the in `FlinkShell` replace `CliFrontend.getCustomCommandLineOptions()` with `frontend.getCustomCommandLineOptions` ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199752165 --- Diff: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java --- @@ -166,6 +166,10 @@ public Configuration getConfiguration() { return copiedConfiguration; } + public static Options getCustomCommandLineOptions() { --- End diff -- so.why does this getter have to be static? ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199750956 --- Diff: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala --- @@ -255,14 +257,25 @@ object FlinkShell { yarnConfig.queue.foreach((queue) => args ++= Seq("-yqu", queue.toString)) yarnConfig.slots.foreach((slots) => args ++= Seq("-ys", slots.toString)) +val customCommandLines = CliFrontend.loadCustomCommandLines( + configuration,configurationDirectory) +val commandOptions = CliFrontendParser.getRunCommandOptions +val customCommandLineOptions = new Options() +customCommandLines.asScala.foreach(cmd => { --- End diff -- @zentol I have fixed that, but I have to make `customCommandLineOptions` of `CliFrontend` to be static (a little ugly) ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199724031 --- Diff: flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala --- @@ -255,14 +257,25 @@ object FlinkShell { yarnConfig.queue.foreach((queue) => args ++= Seq("-yqu", queue.toString)) yarnConfig.slots.foreach((slots) => args ++= Seq("-ys", slots.toString)) +val customCommandLines = CliFrontend.loadCustomCommandLines( + configuration,configurationDirectory) +val commandOptions = CliFrontendParser.getRunCommandOptions +val customCommandLineOptions = new Options() +customCommandLines.asScala.foreach(cmd => { --- End diff -- this is already done int he `CliFrontend` constructor. It may be sufficient to switch the initialization of `frontend` and `commandLine`. ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
GitHub user zjffdu opened a pull request: https://github.com/apache/flink/pull/6140 [FLINK-9554] flink scala shell doesn't work in yarn mode ## What is the purpose of the change This PR is trying to fix the issue of scala-shell unable to run in yarn mode. ## Brief change log The root cause is the options of CustomCommandLine is missed which cause the "-m yarn-cluster" can not be parsed correctly. ## Verifying this change I verify it manually on a single node hadoop cluster. I think it is better to add integration for that so that we can avoid the regression issue in future. If necessary, I can create a ticket for it and do it in another PR. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no ) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? ( not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/flink FLINK-9554 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6140.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6140 commit f3d37d4799e51323fbbcd52cf972ac1d874d2514 Author: Jeff Zhang Date: 2018-06-07T09:47:32Z save commit d65058133ce220e2d4212b906d521b38b9ef53dd Author: Jeff Zhang Date: 2018-06-08T02:39:27Z [FLINK-9554] flink scala shell doesn't work in yarn mode ---