[
https://issues.apache.org/jira/browse/SPARK-59841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-59841:
-----------------------------------
Description:
{{TableCatalog.tableStateOptionKeys()}} was introduced in Spark 4.3.0 by [PR
#57799|https://github.com/apache/spark/pull/57799] ([commit
4b4a2f7b69d5|https://github.com/apache/spark/commit/4b4a2f7b69d555a2b28a130092f7d75411cb4c49]).
That change made ordinary table loading and metadata reuse honor declared
table-state options, while {{loadChangelog}} retained its all-options contract.
TableCatalog currently has different option-handling contracts for ordinary
table reads and changelog reads:
* {{tableStateOptionKeys()}} declares the connector options that select table
state. Spark passes only those options to {{loadTable(Identifier, TableContext,
stateOptions)}} and uses them for table metadata reuse. Complete per-read
options remain available for scan planning.
* {{loadChangelog(Identifier, ChangelogContext, options)}} receives all query
options, and changelog resolution bypasses the corresponding state-aware
table-loading/cache path. There is no separate changelog state-option
declaration.
State options do reach CDC connectors, but Spark does not apply the same option
projection and metadata-reuse rules to the two paths. This complicates
deduplicating loads and refreshes, including the ongoing work on concurrent
table refresh.
For example, a current-table read and a changes-from-version-1 read with
matching table-state options should be able to share the underlying captured
table state. Their scan options and changelog range still describe distinct
reads. Different state options must remain isolated, and different changelog
contexts must not be treated as equivalent reads.
The goal is to define consistent state-option, caching, and refresh semantics
for ordinary and changelog reads, while retaining complete scan options and
preserving the requested CDC context through refresh.
The API approach is undecided. This ticket does not prescribe changing the
Changelog hierarchy, adding a baseTable accessor, or any particular wrapper or
refresh-request abstraction. The implementation should be evaluated for
compatibility with existing connector APIs for the Spark 4.3.0 release.
References:
* [TableCatalog option
contracts|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java#L104]
* [Current changelog
resolution|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala#L382]
* [Exploratory prototype for discussion; API not
decided|https://github.com/gengliangwang/spark/pull/20]
Target release: Spark 4.3.0.
was:
TableCatalog currently has different option-handling contracts for ordinary
table reads and changelog reads:
* {{tableStateOptionKeys()}} declares the connector options that select table
state. Spark passes only those options to {{loadTable(Identifier, TableContext,
stateOptions)}} and uses them for table metadata reuse. Complete per-read
options remain available for scan planning.
* {{loadChangelog(Identifier, ChangelogContext, options)}} receives all query
options, and changelog resolution bypasses the corresponding state-aware
table-loading/cache path. There is no separate changelog state-option
declaration.
State options do reach CDC connectors, but Spark does not apply the same option
projection and metadata-reuse rules to the two paths. This complicates
deduplicating loads and refreshes, including the ongoing work on concurrent
table refresh.
For example, a current-table read and a changes-from-version-1 read with
matching table-state options should be able to share the underlying captured
table state. Their scan options and changelog range still describe distinct
reads. Different state options must remain isolated, and different changelog
contexts must not be treated as equivalent reads.
The goal is to define consistent state-option, caching, and refresh semantics
for ordinary and changelog reads, while retaining complete scan options and
preserving the requested CDC context through refresh.
The API approach is undecided. This ticket does not prescribe changing the
Changelog hierarchy, adding a baseTable accessor, or any particular wrapper or
refresh-request abstraction. The implementation should be evaluated for
compatibility with existing connector APIs for the Spark 4.3.0 release.
References:
* [TableCatalog option
contracts|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java#L104]
* [Current changelog
resolution|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala#L382]
* [Exploratory prototype for discussion; API not
decided|https://github.com/gengliangwang/spark/pull/20]
Target release: Spark 4.3.0.
> Align table and changelog state-option handling and metadata reuse
> ------------------------------------------------------------------
>
> Key: SPARK-59841
> URL: https://issues.apache.org/jira/browse/SPARK-59841
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Gengliang Wang
> Priority: Major
> Fix For: 4.3.0
>
>
> {{TableCatalog.tableStateOptionKeys()}} was introduced in Spark 4.3.0 by [PR
> #57799|https://github.com/apache/spark/pull/57799] ([commit
> 4b4a2f7b69d5|https://github.com/apache/spark/commit/4b4a2f7b69d555a2b28a130092f7d75411cb4c49]).
> That change made ordinary table loading and metadata reuse honor declared
> table-state options, while {{loadChangelog}} retained its all-options
> contract.
> TableCatalog currently has different option-handling contracts for ordinary
> table reads and changelog reads:
> * {{tableStateOptionKeys()}} declares the connector options that select table
> state. Spark passes only those options to {{loadTable(Identifier,
> TableContext, stateOptions)}} and uses them for table metadata reuse.
> Complete per-read options remain available for scan planning.
> * {{loadChangelog(Identifier, ChangelogContext, options)}} receives all query
> options, and changelog resolution bypasses the corresponding state-aware
> table-loading/cache path. There is no separate changelog state-option
> declaration.
> State options do reach CDC connectors, but Spark does not apply the same
> option projection and metadata-reuse rules to the two paths. This complicates
> deduplicating loads and refreshes, including the ongoing work on concurrent
> table refresh.
> For example, a current-table read and a changes-from-version-1 read with
> matching table-state options should be able to share the underlying captured
> table state. Their scan options and changelog range still describe distinct
> reads. Different state options must remain isolated, and different changelog
> contexts must not be treated as equivalent reads.
> The goal is to define consistent state-option, caching, and refresh semantics
> for ordinary and changelog reads, while retaining complete scan options and
> preserving the requested CDC context through refresh.
> The API approach is undecided. This ticket does not prescribe changing the
> Changelog hierarchy, adding a baseTable accessor, or any particular wrapper
> or refresh-request abstraction. The implementation should be evaluated for
> compatibility with existing connector APIs for the Spark 4.3.0 release.
> References:
> * [TableCatalog option
> contracts|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java#L104]
> * [Current changelog
> resolution|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala#L382]
> * [Exploratory prototype for discussion; API not
> decided|https://github.com/gengliangwang/spark/pull/20]
> Target release: Spark 4.3.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
