[
https://issues.apache.org/jira/browse/SPARK-48772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuchen Liu updated SPARK-48772:
-------------------------------
Description:
The current state reader can only return the entire state at a specific
version. If an error occurs related to state, knowing the change of state
across versions to find out at which version state starts to go wrong is
important for debugging purposes. This adds ability of showing the evolution of
state as Change Data Capture (CDC) format to state data source.
An example usage:
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required
.option("changeEndBatchId", 10) #not required, default: latest batch Id
available
{code}
was:
The current state reader can only return the entire state at a specific
version. If an error occurs related to state, knowing the change of state
across versions to find out at which version state starts to go wrong is
important for debugging purposes. This adds ability of showing the evolution of
state as Change Data Capture (CDC) format to state data source.
An example usage:
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required
.option("changeEndBatchId", 10) #not required, default: latest batch Id
available
{code}
> State Data Source Read Change Feed
> ----------------------------------
>
> Key: SPARK-48772
> URL: https://issues.apache.org/jira/browse/SPARK-48772
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Yuchen Liu
> Priority: Major
>
> The current state reader can only return the entire state at a specific
> version. If an error occurs related to state, knowing the change of state
> across versions to find out at which version state starts to go wrong is
> important for debugging purposes. This adds ability of showing the evolution
> of state as Change Data Capture (CDC) format to state data source.
> An example usage:
> {code:java}
> .format("statestore")
> .option("readChangeFeed", true)
> .option("changeStartBatchId", 5) #required
> .option("changeEndBatchId", 10) #not required, default: latest batch Id
> available
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
