[ https://issues.apache.org/jira/browse/BEAM-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023657#comment-17023657 ]
Enis Nazif edited comment on BEAM-8970 at 1/25/20 10:12 PM: ------------------------------------------------------------ looking at this issue, to run a pipeline on YARN backed sparked, a user should be able to specify runner options of {code:java} ['--runner=SparkRunner', '--spark_submit_uber_jar', '--spark_rest_url=http://spark-rest-api:6066', '--spark_master_url='yarn']{code} As it stands, the 'spark_master_url' isn't being passed into the request created in in [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L145] It seems that this is necessary to support YARN Failing this, an alternative way may be to bypass the Spark REST API (which seems like fairly hidden functionality) and instead directly spark-submit the portable jars that are created. was (Author: enazif): looking at this issue, to run a pipeline on YARN backed sparked, a user should be able to specify runner options of {code:java} ['--runner=SparkRunner', '--spark_submit_uber_jar', '--spark_rest_url=http://spark-rest-api:6066', '--spark_master_url='yarn']{code} As it stands, the 'spark_master_url' isn't being passed into the request created in in [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L145] It seems that this is necessary to support YARN Failing this, an alternative way may be to bypass the Spark REST API (which seems like fairly hidden functionality) and instead directly {noformat} spark-submit{noformat} spark-submit` the portable jars that are created. > Spark portable runner supports Yarn > ----------------------------------- > > Key: BEAM-8970 > URL: https://issues.apache.org/jira/browse/BEAM-8970 > Project: Beam > Issue Type: Wish > Components: runner-spark > Reporter: Kyle Weaver > Assignee: Kyle Weaver > Priority: Major > Labels: portability-spark > -- This message was sent by Atlassian Jira (v8.3.4#803005)