[
https://issues.apache.org/jira/browse/SPARK-60689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60689:
-----------------------------------
Labels: pull-request-available (was: )
> Add the planning rules for external lookup join
> -----------------------------------------------
>
> Key: SPARK-60689
> URL: https://issues.apache.org/jira/browse/SPARK-60689
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> Part of SPARK-60031. This ticket adds the logical nodes and the analyzer and
> optimizer rules. They are **not registered** yet, so no query reaches them
> and nothing changes for users; plan-level tests exercise them.
> In scope:
> - Logical nodes `ExternalLookupJoin` and `PreparedExternalLookupJoin`, and
> the placeholder `ExternalLookupRelation`.
> - `ResolveExternalLookupJoin`:
> - rewrites an inner or left outer join with a condition, whose left side is
> a streaming query and whose right side **is** a lookup-only table
> (`EXTERNAL_LOOKUP` without `BATCH_READ`);
> - rejects ineligible joins with named errors: `JOIN_TYPE`,
> `STATIC_LEFT_SIDE`, `LOOKUP_TABLE_ON_LEFT` and `CONFLICTING_JOIN_STRATEGY`;
> - reports `MISSING_JOIN_CONDITION` on the final plan.
> - `PrepareExternalLookupJoin`:
> - offers `=` comparisons as candidates and maps the connector's selection
> back by identity;
> - keeps everything else as a residual condition;
> - validates the connector contract
> (`DATA_SOURCE_INVALID_EXTERNAL_LOOKUP.*`);
> - matches the read schema the way V2 scans do (CHAR/VARCHAR, nested
> nullability).
> Out of scope:
> - Registering the rules and executing joins.
> - Lookup sides beneath hints, aliases, views, projections or filters.
> - The `EXTERNAL_LOOKUP` hint and dual-capability opt-in.
> - Left semi/anti joins, `<=>` and range predicates.
> - Cache.
> - Predicate pushdown through the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
