Yan Yan created SPARK-58392:
-------------------------------
Summary: Pass all options while loading relations
Key: SPARK-58392
URL: https://issues.apache.org/jira/browse/SPARK-58392
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Yan Yan
Follow-up to SPARK-58389 ("Pass all options while loading tables"), which
forwarded
all user-specified options to TableCatalog.loadTable(Identifier, TableContext,
CaseInsensitiveStringMap) on the table read path.
That change does not cover the RelationCatalog single-RPC read path. A
RelationCatalog
(a connector that exposes both tables and views in one identifier namespace) is
resolved
through RelationCatalog.loadRelation(Identifier) -- a single call that answers
"table or
view?" in one round trip. For such a catalog this is the primary plain-read
path, and it
currently does not receive the user options, so a connector cannot see them when
producing the Table/View metadata for a plain read.
This is the direct analogue of SPARK-58389 for the single-RPC path. Add
RelationCatalog.loadRelation(Identifier, CaseInsensitiveStringMap) with a
default
implementation that ignores the options and delegates to
loadRelation(Identifier) --
so existing connectors are unaffected -- and forward the user options to it
from the
resolver's plain-read branch (which fires only when there is no time travel and
no write
privileges; both apply to tables only and route through the loadTable path
instead).
This is API completeness for the combined table+view catalog: it lets such a
catalog
customize the returned relation from the read options at metadata-load time,
symmetric
to what SPARK-58389 enables for table-only catalogs. The feature has not been
released.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
