[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-17 Thread Marius Soutier (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364713#comment-14364713
 ] 

Marius Soutier commented on SPARK-6304:
---

Got it, thanks.

> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Apache Spark (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364574#comment-14364574
 ] 

Apache Spark commented on SPARK-6304:
-

User 'jerryshao' has created a pull request for this issue:
https://github.com/apache/spark/pull/5060

> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Saisai Shao (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364254#comment-14364254
 ] 

Saisai Shao commented on SPARK-6304:


Hi [~msoutier], the reason to remove these two configurations, especially 
"spark.driver.port" is that: SparkContext itself will randomly choose a port 
and set it to configuration even user didn't set it, next time after 
application is recovered, previous configuration "spark.driver.port" need to 
remove and let SparkContext itself to randomly choose again and set into the 
SparkConf. So that's why checkpoint need to remove these two configurations.

> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Marius Soutier (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363047#comment-14363047
 ] 

Marius Soutier commented on SPARK-6304:
---

Yeah but if the user doesn't set the port, why remove it? When Spark 
deserializes the checkpoint, the port shouldn't be set by default, right?


> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Saisai Shao (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362932#comment-14362932
 ] 

Saisai Shao commented on SPARK-6304:


As I said, normally user will not set these two configurations 
{{spark.driver.host}} and {{spark.driver.port}} to let SparkContext to set. 
SparkContext will internally choose driver's host name and random port for 
these two configurations, the reason to do so is to avoid port contention whey 
multiple driver running on the same machine. 

Spark Streaming takes this assumption to remove these two configurations when 
recovering from checkpoint file, to avoid port contention. Yes this is a bug 
for usage scenarios like yours.

> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Marius Soutier (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362928#comment-14362928
 ] 

Marius Soutier commented on SPARK-6304:
---

I'm just reporting the bug. As you said, the code explicitly removes 
"spark.driver.host" and "spark.driver.port" when recovering from a checkpoint, 
so I first would like to understand why that is.




> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Saisai Shao (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362925#comment-14362925
 ] 

Saisai Shao commented on SPARK-6304:


OK, got it. So are you going to fix this issue or just report a bug here?

> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Marius Soutier (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362918#comment-14362918
 ] 

Marius Soutier commented on SPARK-6304:
---

Simple, I'm using `actorStream` and want to send data to it via remoting. For 
that I need to have a fixed port to send data to.

As a workaround I'm now starting a second ActorSystem, but it seems to have 
issues communicating with Spark's ActorSystem.


> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org



[jira] [Commented] (SPARK-6304) Checkpointing doesn't retain driver port

2015-03-16 Thread Saisai Shao (JIRA)

[ 
https://issues.apache.org/jira/browse/SPARK-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362907#comment-14362907
 ] 

Saisai Shao commented on SPARK-6304:


Hi [~msoutier], seldom user will set this {{spark.driver.port}}, Spark itself 
will randomly choose a port and set it to SparkConf, so according to this, 
Spark Streaming will remove previously set port when recovering from checkpoint.

Basically why {{spark.driver.port}} need to be fixed? And potentially it needs 
to be fixed when {{spark.driver.*}} is set by user.

> Checkpointing doesn't retain driver port
> 
>
> Key: SPARK-6304
> URL: https://issues.apache.org/jira/browse/SPARK-6304
> Project: Spark
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.1
>Reporter: Marius Soutier
>
> In a check-pointed Streaming application running on a fixed driver port, the 
> setting "spark.driver.port" is not loaded when recovering from a checkpoint.
> (The driver is then started on a random port.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org