[ https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023657#comment-17023657 ]
Enis Nazif edited comment on BEAM-8970 at 1/25/20 10:11 PM: ------------------------------------------------------------ looking at this issue, to run a pipeline on YARN backed sparked, a user should be able to specify runner options of {code:java} ['--runner=SparkRunner', '--spark_submit_uber_jar', '--spark_rest_url=http://spark-rest-api:6066', '--spark_master_url='yarn']{code} As it stands, the 'spark_master_url' isn't being passed into the request created in in [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L145] It seems that this is necessary to support YARN Failing this, an alternative way may be to bypass the Spark REST API (which seems like fairly hidden functionality) and instead directly {noformat} spark-submit{noformat} spark-submit` the portable jars that are created. was (Author: enazif): looking at this issue, to run a pipeline on YARN backed sparked, a user should be able to specify runner options of {code:java} ['--runner=SparkRunner', '--spark_submit_uber_jar', '--spark_rest_url=http://spark-rest-api:6066', '--spark_master_url='yarn']{code} As it stands, the `spark_master_url` isn't being passed into the request created in in [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L145] It seems that this is necessary to support YARN Failing this, an alternative way may be to bypass the Spark REST API (which seems like fairly hidden functionality) and instead directly `spark-submit` the portable jars that are created. > Spark portable runner supports Yarn > ----------------------------------- > > Key: BEAM-8970 > URL: https://issues.apache.org/jira/browse/BEAM-8970 > Project: Beam > Issue Type: Wish > Components: runner-spark > Reporter: Kyle Weaver > Assignee: Kyle Weaver > Priority: Major > Labels: portability-spark > -- This message was sent by Atlassian Jira (v8.3.4#803005)