[
https://issues.apache.org/jira/browse/SPARK-60348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60348:
-----------------------------------
Labels: pull-request-available (was: )
> Fix state data source change feed dropping list state values after the first
> ----------------------------------------------------------------------------
>
> Key: SPARK-60348
> URL: https://issues.apache.org/jira/browse/SPARK-60348
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.1.0, 4.2.0, 5.0.0
> Reporter: Kavpreet Grewal
> Priority: Major
> Labels: pull-request-available
>
> Since SPARK-54118, {{ListState.put}} and {{appendList}} write all of their
> values in a single changelog record, but
> {{RocksDBStateStoreChangeDataReader}} decodes only the first value of each
> record. So reading list state with {{readChangeFeed}} silently drops every
> value after the first. For example, {{put(Array("a", "b", "c"))}} shows up
> only as {{update a}}. The stored state is not affected.
> We should return one row per value: the first value keeps the record's change
> type and the rest are returned as {{append}}, which matches the change feed
> from before SPARK-54118.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
