[
https://issues.apache.org/jira/browse/SPARK-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137797#comment-15137797
]
Hari Shreedharan commented on SPARK-13216:
------------------------------------------
I disagree that checkpointing is only for failed applications. For any of the
receiver-based streaming applications, checkpoints are important to recover as
yet unprocessed data.
If the application cannot be reloaded from a checkpoint - then the old data is
pretty much gone. I know that checkpointing basically makes application and
spark upgrades difficult or impossible, but there are configuration parameters
that the users might want to change based on load requirements etc. I don't see
a reason why we should not allow this, since it has nothing to do with starting
the app from checkpoint or not - if we want the number of executors to change
we should be able to. This is especially true when migrating from a non-dynamic
allocation situation to a dynamic allocation situation.
> Spark streaming application not honoring --num-executors in restarting of an
> application from a checkpoint
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-13216
> URL: https://issues.apache.org/jira/browse/SPARK-13216
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit, Streaming
> Affects Versions: 1.5.0
> Reporter: Neelesh Srinivas Salian
> Priority: Minor
> Labels: Streaming
>
> Scenario to help understand:
> 1) The Spark streaming job with 12 executors was initiated with checkpointing
> enabled.
> 2) In version 1.3, the user was able to append the number of executors to 20
> using --num-executors but was unable to do so in version 1.5.
> In 1.5, the spark application still runs with 13 executors (1 for driver and
> 12 executors).
> There is a need to start from the checkpoint itself and not restart the
> application to avoid the loss of information.
> 3) Checked the code in 1.3 and 1.5, which shows the command
> ''--num-executors" has been deprecated.
> Any thoughts on this? Not sure if anyone hit this one specifically before.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
