Jungtaek Lim created SPARK-60689:
------------------------------------
Summary: Add the planning rules for external lookup join
Key: SPARK-60689
URL: https://issues.apache.org/jira/browse/SPARK-60689
Project: Spark
Issue Type: Task
Components: SQL, Structured Streaming
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031. This ticket adds the logical nodes and the analyzer and
optimizer rules. They are **not registered** yet, so no query reaches them and
nothing changes for users; plan-level tests exercise them.
In scope:
- Logical nodes `ExternalLookupJoin` and `PreparedExternalLookupJoin`, and the
placeholder `ExternalLookupRelation`.
- `ResolveExternalLookupJoin`:
- rewrites an inner or left outer join with a condition, whose left side is a
streaming query and whose right side **is** a lookup-only table
(`EXTERNAL_LOOKUP` without `BATCH_READ`);
- rejects ineligible joins with named errors: `JOIN_TYPE`,
`STATIC_LEFT_SIDE`, `LOOKUP_TABLE_ON_LEFT` and `CONFLICTING_JOIN_STRATEGY`;
- reports `MISSING_JOIN_CONDITION` on the final plan.
- `PrepareExternalLookupJoin`:
- offers `=` comparisons as candidates and maps the connector's selection
back by identity;
- keeps everything else as a residual condition;
- validates the connector contract (`DATA_SOURCE_INVALID_EXTERNAL_LOOKUP.*`);
- matches the read schema the way V2 scans do (CHAR/VARCHAR, nested
nullability).
Out of scope:
- Registering the rules and executing joins.
- Lookup sides beneath hints, aliases, views, projections or filters.
- The `EXTERNAL_LOOKUP` hint and dual-capability opt-in.
- Left semi/anti joins, `<=>` and range predicates.
- Cache.
- Predicate pushdown through the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
