Jungtaek Lim created SPARK-61509:
------------------------------------
Summary: Fail at start when a held-back stream-stream join output
watermark reaches a later join, event-time timeout or aggregation
Key: SPARK-61509
URL: https://issues.apache.org/jira/browse/SPARK-61509
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
h3. Problem
SPARK-61345 fails a query at start when an append-mode aggregation uses the
event time of a
stream-stream join input whose state the join cannot bound. The join holds back
its output
watermark for that input (the input's contribution is 0), so the aggregation
never emits. The
check judges only the nearest join (the tracer stops there) and only
append-mode aggregations. Three
shapes with the same cause still start without an error or a warning:
Let J1 = A inner join B on {{ak = bk AND at = bt}} with a watermark only on
{{at}}. J1 holds the
watermark of {{at}} at 0.
# J2 = J1 left outer join C on {{ak = ck AND at = ct}}, with C watermarked: J2
never evicts the
J1 rows and never emits them as unmatched.
# An append-mode window aggregation over {{ct}} after J2 never emits, but the
check passes because
it judges only J2.
# A {{flatMapGroupsWithState}} or {{transformWithState}} operator with
event-time timeouts after
J1 never fires its timeouts.
The analyzer already rejects append-mode aggregations and event-time timeouts
without a watermark,
and outer joins that can never finalize. A watermark held at 0 has the same
effect.
h3. Proposal
Make {{checkJoinOutputWatermarkConsumers}} follow the tracer through later
joins to the join that
holds the watermark back, and also reject stream-stream outer joins and
operators with event-time
timeouts, naming the operator. The same conf,
{{spark.sql.streaming.join.checkFinalization.enabled}},
turns the error into a warning, as in SPARK-61345.
h3. Why a separate change
Covering later joins needs the propagated watermark through each join, not one
join's state bound,
and it touches every stateful operator kind. It deserves its own review. The
migration guide
documents the current scope: such operators keep running and do not emit
results that wait for the
watermark.
Related: SPARK-61345.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]