[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364713#comment-14364713 ] Marius Soutier commented on SPARK-6304: --- Got it, thanks. > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364574#comment-14364574 ] Apache Spark commented on SPARK-6304: - User 'jerryshao' has created a pull request for this issue: https://github.com/apache/spark/pull/5060 > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364254#comment-14364254 ] Saisai Shao commented on SPARK-6304: Hi [~msoutier], the reason to remove these two configurations, especially "spark.driver.port" is that: SparkContext itself will randomly choose a port and set it to configuration even user didn't set it, next time after application is recovered, previous configuration "spark.driver.port" need to remove and let SparkContext itself to randomly choose again and set into the SparkConf. So that's why checkpoint need to remove these two configurations. > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363047#comment-14363047 ] Marius Soutier commented on SPARK-6304: --- Yeah but if the user doesn't set the port, why remove it? When Spark deserializes the checkpoint, the port shouldn't be set by default, right? > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362932#comment-14362932
]
Saisai Shao commented on SPARK-6304:
As I said, normally user will not set these two configurations
{{spark.driver.host}} and {{spark.driver.port}} to let SparkContext to set.
SparkContext will internally choose driver's host name and random port for
these two configurations, the reason to do so is to avoid port contention whey
multiple driver running on the same machine.
Spark Streaming takes this assumption to remove these two configurations when
recovering from checkpoint file, to avoid port contention. Yes this is a bug
for usage scenarios like yours.
> Checkpointing doesn't retain driver port
>
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362928#comment-14362928 ] Marius Soutier commented on SPARK-6304: --- I'm just reporting the bug. As you said, the code explicitly removes "spark.driver.host" and "spark.driver.port" when recovering from a checkpoint, so I first would like to understand why that is. > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362925#comment-14362925 ] Saisai Shao commented on SPARK-6304: OK, got it. So are you going to fix this issue or just report a bug here? > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[ https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362918#comment-14362918 ] Marius Soutier commented on SPARK-6304: --- Simple, I'm using `actorStream` and want to send data to it via remoting. For that I need to have a fixed port to send data to. As a workaround I'm now starting a second ActorSystem, but it seems to have issues communicating with Spark's ActorSystem. > Checkpointing doesn't retain driver port > > > Key: SPARK-6304 > URL: https://issues.apache.org/jira/browse/SPARK-6304 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: Marius Soutier > > In a check-pointed Streaming application running on a fixed driver port, the > setting "spark.driver.port" is not loaded when recovering from a checkpoint. > (The driver is then started on a random port.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port
[
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362907#comment-14362907
]
Saisai Shao commented on SPARK-6304:
Hi [~msoutier], seldom user will set this {{spark.driver.port}}, Spark itself
will randomly choose a port and set it to SparkConf, so according to this,
Spark Streaming will remove previously set port when recovering from checkpoint.
Basically why {{spark.driver.port}} need to be fixed? And potentially it needs
to be fixed when {{spark.driver.*}} is set by user.
> Checkpointing doesn't retain driver port
>
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
