Yuchen Liu created SPARK-48772:
----------------------------------
Summary: State Data Source Read Change Feed
Key: SPARK-48772
URL: https://issues.apache.org/jira/browse/SPARK-48772
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Yuchen Liu
The current state reader can only return the entire state at a specific
version. If an error occurs related to state, knowing the change of state
across versions to find out at which version state starts to go wrong is
important for debugging purposes. This adds ability of showing the evolution of
state as Change Data Capture (CDC) format to state data source.
An example usage:
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required
.option("changeEndBatchId", 10) #not required, default: latest batch Id
available
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
