[ https://issues.apache.org/jira/browse/SPARK-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733383#comment-15733383 ]
Apache Spark commented on SPARK-18790: -------------------------------------- User 'tcondie' has created a pull request for this issue: https://github.com/apache/spark/pull/16219 > Keep a general offset history of stream batches > ----------------------------------------------- > > Key: SPARK-18790 > URL: https://issues.apache.org/jira/browse/SPARK-18790 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Reporter: Tyson Condie > > Instead of only keeping the minimum number of offsets around, we should keep > enough information to allow us to roll back n batches and reexecute the > stream starting from a given point. In particular, we should create a config > in SQLConf, spark.sql.streaming.retainedBatches that defaults to 100 and > ensure that we keep enough log files in the following places to roll back the > specified number of batches: > the offsets that are present in each batch > versions of the state store > the files lists stored for the FileStreamSource > the metadata log stored by the FileStreamSink -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org