[
https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024769#comment-17024769
]
Kyle Weaver edited comment on BEAM-8970 at 1/27/20 11:54 PM:
-------------------------------------------------------------
Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark
REST API along with YARN, because normally the Spark REST API is started along
with the Spark master.
You should be able to spark-submit portable jars. To create portable jars:
[--runner=SparkRunner,
--output_executable_path=~/path/to/output.jar]
(Without using the spark_submit_uber_jar option.)
Also, note that this will require YARN nodes to have installed or otherwise be
able to access Beam worker code. [~angoenka] might know more.
was (Author: ibzib):
Hi Enis, thanks for the feedback. I'm not sure it's possible to use the Spark
REST API along with YARN, because normally the Spark REST API is started along
with the Spark master.
You should be able to spark-submit portable jars. To create portable jars:
{{
['--runner=SparkRunner',
--output_executable_path "$OUTPUT_JAR"]
}}
(Without using the spark_submit_uber_jar option.)
Also, note that this will require YARN nodes to have installed or otherwise be
able to access Beam worker code. [~angoenka] might know more.
> Spark portable runner supports Yarn
> -----------------------------------
>
> Key: BEAM-8970
> URL: https://issues.apache.org/jira/browse/BEAM-8970
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-spark
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
