Ismaël Mejía created BEAM-470:
---------------------------------
Summary: Spark Runner does not send the job execution information
into the Spark History Server
Key: BEAM-470
URL: https://issues.apache.org/jira/browse/BEAM-470
Project: Beam
Issue Type: Improvement
Components: runner-spark
Affects Versions: 0.2.0-incubating
Reporter: Ismaël Mejía
Assignee: Amit Sela
Priority: Minor
If you run a Beam pipeline using the spark runner from spark (via
spark-submit), the execution is registered in the spark-history-server if it is
active and configured.
if you do this directly from a main method with --runner=SparkRunner (the beam
way) the Beam runner does not report the execution to the history server, it
seems the issue is the runner does not take into account an existing spark
configuration file SPARK_HOME/conf/spark-defaults.conf (or there is not a way
to tell the runner to take such conf into account).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
