[
https://issues.apache.org/jira/browse/SPARK-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das resolved SPARK-6331.
----------------------------------
Resolution: Fixed
> New Spark Master URL is not picked up when streaming context is started from
> checkpoint
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-6331
> URL: https://issues.apache.org/jira/browse/SPARK-6331
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.1.1, 1.3.0, 1.2.1
> Reporter: Tathagata Das
> Assignee: Tathagata Das
>
> When the SparkConf is reconstructed based on the checkpointed configuration,
> it recovers the old master URL. This okay if the cluster on which the
> streaming application is relaunched is the same cluster as it was running
> before. But if that cluster changes, there is no way to inject the new master
> URL of the new cluster. As a result, the restarted app tries to connect to
> the non-existent old cluster and fails.
> The solution is to check whether a master URL is set in the System properties
> (by Spark submit) before recreating the SparkConf. If a new master url is set
> in the properties, then use it as that is obviously the most relevant one.
> Otherwise load the old one (to maintain existing behavior).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
