[
https://issues.apache.org/jira/browse/SPARK-60688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60688:
---------------------------------
Issue Type: Task (was: New Feature)
> Add the DSv2 connector API for external lookup join
> ---------------------------------------------------
>
> Key: SPARK-60688
> URL: https://issues.apache.org/jira/browse/SPARK-60688
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> Part of SPARK-60031. This ticket adds only the connector-facing DSv2 API for
> external lookup join. No engine code calls it yet, so it has no user-visible
> behavior.
> In scope:
> - `TableCapability.EXTERNAL_LOOKUP`.
> - `SupportsExternalLookup`, a `ScanBuilder` mix-in:
> `selectLookupPredicates(Predicate[])`.
> - `LookupParameter`: the typed `?` in an offered predicate.
> - `Scan.toExternalLookup()`, `ExternalLookup`,
> `ExternalLookupReaderFactory`, `ExternalLookupReader` and
> `ExternalLookupResultCursor`.
> - The error condition `DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED`, and `?`
> rendering for `LookupParameter` in `ToStringSQLBuilder`.
> Out of scope:
> - Planning rules.
> - Execution.
> - The join hint.
> - Lookup-side operators and more predicate shapes.
> - Cache.
> - Predicate pushdown.
> - Python Data Source support.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
