Jungtaek Lim created SPARK-60690:
------------------------------------
Summary: Execute basic stream-static external lookup joins
Key: SPARK-60690
URL: https://issues.apache.org/jira/browse/SPARK-60690
Project: Spark
Issue Type: Task
Components: SQL, Structured Streaming
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031. This ticket registers the planning rules and adds the
physical operator, the first user-visible behavior.
In scope:
- **Wiring:** the analyzer and optimizer register the rules, and
`ExternalLookupJoinStrategy` plans `ExternalLookupJoinExec`.
- **Execution:**
- batched, synchronous requests;
- parameter rows deduplicated under Spark equality;
- one-to-many results, with an early stop under `LIMIT`;
- readers created lazily per input partition and closed on success, failure
or cancellation;
- connector metrics reported under `source.<name>`;
- a connector failure fails the task and is never treated as a miss.
- **Analysis:**
- lookup-only `DataFrameReader` sources load as `ExternalLookupRelation` and
pass analysis;
- a lookup-only table read outside a lookup join fails with `TABLE_OPERATION`.
- **Other:**
- CHAR read padding for the lookup relation;
- a provably empty lookup side, e.g. `WHERE false`, returns its result
without calling the connector.
User-visible result:
- Micro-batch stream-static inner and left outer joins work when the right side
is **directly** a lookup-only DSv2 table loaded with `DataFrameReader`, with
`=` keys (composite included).
- Real-time mode rejects the operator.
- Batch (static-static) joins over lookup-only tables fail with
`STATIC_LEFT_SIDE`.
Out of scope (these fail with a named error, not bugs):
- SQL and catalog table references, and aliases, views, projections or filters
above the lookup table (`TABLE_OPERATION`).
- The `EXTERNAL_LOOKUP` hint, and strategy hints on the lookup side.
- Left semi/anti joins, `<=>` and range predicates.
- CHAR lookup columns (`LOOKUP_PLAN`).
- Cache.
- Pushing stream-side filters below the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
