[ https://issues.apache.org/jira/browse/SPARK-48589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim reassigned SPARK-48589: ------------------------------------ Assignee: Yuchen Liu > Add option snapshotStartBatchId and snapshotPartitionId to state data source > ---------------------------------------------------------------------------- > > Key: SPARK-48589 > URL: https://issues.apache.org/jira/browse/SPARK-48589 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming > Affects Versions: 4.0.0 > Reporter: Yuchen Liu > Assignee: Yuchen Liu > Priority: Major > Labels: pull-request-available > > Define two new options, _snapshotStartBatchId_ and _snapshotPartitionId_, for > the existing state reader. Both of them should be provided at the same time. > # When there is no snapshot file at that batch (note there is an off-by-one > issue between version and batch Id), throw an exception. > # Otherwise, the reader should continue to rebuild the state by reading > delta files only, and ignore all snapshot files afterwards. > # Note that if a batchId option is already specified. That batchId is the > ending batchId, we should then end at that batchId. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org