Jungtaek Lim created SPARK-61403:
------------------------------------
Summary: Support external lookup joins in real-time mode
Key: SPARK-61403
URL: https://issues.apache.org/jira/browse/SPARK-61403
Project: Spark
Issue Type: Task
Components: Structured Streaming, SQL
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031. Real-time mode rejects the external lookup join today,
because `ExternalLookupJoinExec` is not in `RealTimeModeAllowlist`.
In scope:
- Allow `ExternalLookupJoinExec` in real-time mode. The operator is stateless,
runs in its input's task and keeps its input's determinism level, so a stateful
operator after it can read it through a pipelined shuffle.
- Tests in real-time mode: inner, left outer, left semi and left anti joins;
the cache within a batch and its reset at the next batch; a streaming
aggregation after the join; a shuffle on the stream side before it; a stop
during a lookup request.
- Known limitations, pinned by a test and documented:
- a stream row waits until its lookup request is full or the batch ends;
- the output of a full request waits for the next stream row;
- a cache hit waits the same way;
- the cache starts empty at every batch.
Out of scope:
- A time-based flush of a lookup request that is not full.
- Asynchronous lookup requests.
- A cache that lives across batches.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]