Jungtaek Lim created SPARK-61403:
------------------------------------

             Summary: Support external lookup joins in real-time mode
                 Key: SPARK-61403
                 URL: https://issues.apache.org/jira/browse/SPARK-61403
             Project: Spark
          Issue Type: Task
          Components: Structured Streaming, SQL
    Affects Versions: 4.4.0
            Reporter: Jungtaek Lim


Part of SPARK-60031. Real-time mode rejects the external lookup join today, 
because `ExternalLookupJoinExec` is not in `RealTimeModeAllowlist`.

In scope:
- Allow `ExternalLookupJoinExec` in real-time mode. The operator is stateless, 
runs in its input's task and keeps its input's determinism level, so a stateful 
operator after it can read it through a pipelined shuffle.
- Tests in real-time mode: inner, left outer, left semi and left anti joins; 
the cache within a batch and its reset at the next batch; a streaming 
aggregation after the join; a shuffle on the stream side before it; a stop 
during a lookup request.
- Known limitations, pinned by a test and documented:
  - a stream row waits until its lookup request is full or the batch ends;
  - the output of a full request waits for the next stream row;
  - a cache hit waits the same way;
  - the cache starts empty at every batch.

Out of scope:
- A time-based flush of a lookup request that is not full.
- Asynchronous lookup requests.
- A cache that lives across batches.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to