Jungtaek Lim created SPARK-61509:
------------------------------------

             Summary: Fail at start when a held-back stream-stream join output 
watermark reaches a later join, event-time timeout or aggregation
                 Key: SPARK-61509
                 URL: https://issues.apache.org/jira/browse/SPARK-61509
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


h3. Problem

SPARK-61345 fails a query at start when an append-mode aggregation uses the 
event time of a
stream-stream join input whose state the join cannot bound. The join holds back 
its output
watermark for that input (the input's contribution is 0), so the aggregation 
never emits. The
check judges only the nearest join (the tracer stops there) and only 
append-mode aggregations. Three
shapes with the same cause still start without an error or a warning:

Let J1 = A inner join B on {{ak = bk AND at = bt}} with a watermark only on 
{{at}}. J1 holds the
watermark of {{at}} at 0.

# J2 = J1 left outer join C on {{ak = ck AND at = ct}}, with C watermarked: J2 
never evicts the
  J1 rows and never emits them as unmatched.
# An append-mode window aggregation over {{ct}} after J2 never emits, but the 
check passes because
  it judges only J2.
# A {{flatMapGroupsWithState}} or {{transformWithState}} operator with 
event-time timeouts after
  J1 never fires its timeouts.

The analyzer already rejects append-mode aggregations and event-time timeouts 
without a watermark,
and outer joins that can never finalize. A watermark held at 0 has the same 
effect.

h3. Proposal

Make {{checkJoinOutputWatermarkConsumers}} follow the tracer through later 
joins to the join that
holds the watermark back, and also reject stream-stream outer joins and 
operators with event-time
timeouts, naming the operator. The same conf, 
{{spark.sql.streaming.join.checkFinalization.enabled}},
turns the error into a warning, as in SPARK-61345.

h3. Why a separate change

Covering later joins needs the propagated watermark through each join, not one 
join's state bound,
and it touches every stateful operator kind. It deserves its own review. The 
migration guide
documents the current scope: such operators keep running and do not emit 
results that wait for the
watermark.

Related: SPARK-61345.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to