[
https://issues.apache.org/jira/browse/SPARK-12137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-12137.
-------------------------------
Resolution: Not A Problem
> Spark Streaming State Recovery limitations
> ------------------------------------------
>
> Key: SPARK-12137
> URL: https://issues.apache.org/jira/browse/SPARK-12137
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.4.1
> Reporter: Ravindar
> Priority: Critical
>
> There was multiple threads in forums asking similar question without a clear
> answer and hence entering it here.
> We have a streaming application that goes through multi-step processing. In
> some of these steps stateful operations like *updateStateByKey* are used to
> maintain an accumulated running state (and other state info) with incoming
> RDD streams. As streaming application is incremental, it is imperative that
> we recover/restore from previous known state in the following two scenarios
> 1. On spark driver/streaming application failure.
> In this scenario the driver/streaming application shutdown and
> restarted. The recommended approach is enable the *checkpoint(checkpointDir)*
> and use *StreamingContext.getOrCreate* to restore the context from checkpoint
> state.
> 2. Upgrade driver/streaming application with additional steps in the
> processing
> In this scenario, we introduced new steps with downstream processing for
> new functionality without changes to existing steps. Upgrading the streaming
> application with the new fails on *StreamingContext.getOrCreate* as there is
> mismatch in checkpoint saved.
> Both of the above scenarios needs a unified approach where accumulated state
> has to be saved and restored. The first approach of restoring from checkpoint
> works for driver failure but not code upgrade. When the application code
> changed, there is a recommendation to delete checkpoint data when new code is
> deployed. If so, how do you reconstitute all of the stateful (e.g:
> updateStateByKey) information from the last run. Every streaming application
> has to save up-to-date state for each session represented by key and then
> initialize it from this when a new session starts for the same key. Does
> every application have to create their own mechanism given this is very
> similar to current state checkpointing to HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
