Yan Yan created SPARK-58389:
-------------------------------
Summary: Pass all options while loading tables
Key: SPARK-58389
URL: https://issues.apache.org/jira/browse/SPARK-58389
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Yan Yan
SPARK-56961 changed TableCatalog.loadChangelog to receive all user-specified
options (via a ChangelogContext + CaseInsensitiveStringMap), so connectors can
see the options a query passes. The regular table read path was not updated:
TableCatalog.loadTable and its overloads do not forward the user's options to
the catalog, so a connector cannot access them when a table is loaded.
This is a follow-up that does the same for table reads. It adds
```
TableCatalog.loadTable(Identifier ident, TableContext context,
CaseInsensitiveStringMap options)
```
where TableContext carries the parsed, Spark-recognized load parameters (time
travel and write privileges) and the CaseInsensitiveStringMap carries all raw
user options. The new method has a default implementation that delegates to the
existing loadTable overloads, so existing connectors keep working unchanged.
TableContext models time travel with a new public connector type, TimeTravel
(a sealed interface with Version/Timestamp), mirroring how SPARK-56961
introduced ChangelogContext/ChangelogRange rather than exposing the internal
catalyst TimeTravelSpec.
This is needed to make the read API usable in connectors like Iceberg and
Delta. The functionality is unreleased.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
