[
https://issues.apache.org/jira/browse/SPARK-61448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-61448.
----------------------------------
Resolution: Duplicate
> Bound stream-stream join state by a window key over the first watermarked
> column in legacy mode
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-61448
> URL: https://issues.apache.org/jira/browse/SPARK-61448
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> With {{spark.sql.streaming.statefulOperator.allowMultiple=false}} (legacy
> mode), an input of a stream-stream join can keep both its event time {{ts}}
> and a time window over it. Both columns carry the watermark, and the join
> filters late rows by the first of them.
> SPARK-61345 lets a window equality key bound the other input's state only
> when the window is that first column. Before it, Spark used any watermarked
> equality key. That was sound for a window over the first watermarked column:
> a row that passes late filtering on {{ts}} has a window that ends after the
> watermark. But it was unsound for a window over another watermarked column,
> where it could remove rows that could still match. SPARK-61345 cannot tell
> the two apart without plan lineage, so it drops the bound for both.
> As a result, an input built with {{withColumn("window", window($"ts",
> ...))}}, which keeps {{ts}} before the window, gets no bound from the window
> key in legacy mode:
> - Outer and left semi joins on the window fail analysis. The error says which
> column filters late rows and how to reorder the columns.
> - Inner joins keep the other input's rows for the life of the query.
> No result is wrong; the join only keeps more state than it needs, or is
> rejected.
> This ticket asks Spark to recognize a window key whose window was computed
> from the input's first watermarked column, and to bound state by its end
> again, as before SPARK-61345. A window over another column must stay
> unrecognized.
> h3. Reproducer
> - Setup: Scala with {{MemoryStream}}, legacy mode, and no-data micro-batches
> disabled, so that each step below is one batch.
> - Build: a build with SPARK-61345.
> {code:scala}
> spark.conf.set("spark.sql.streaming.statefulOperator.allowMultiple", "false")
> spark.conf.set("spark.sql.streaming.noDataMicroBatches.enabled", "false")
> val leftInput = MemoryStream[(Int, Int)]
> val rightInput = MemoryStream[(Int, Int)]
> def windowed(input: MemoryStream[(Int, Int)], side: String) =
> input.toDF().toDF(s"${side}k", "time")
> .select(col(s"${side}k"), timestamp_seconds($"time").as(s"${side}t"))
> .withWatermark(s"${side}t", "0 seconds")
> .withColumn(s"${side}w", window(col(s"${side}t"), "10 seconds"))
> val left = windowed(leftInput, "l")
> val right = windowed(rightInput, "r")
> val joined = left.join(right, expr("lk = rk AND lw = rw"), "left_outer")
> // batch 1: left (1, 5 s), (2, 6 s); right (1, 7 s)
> // batch 2: left (3, 30 s); right (9, 30 s) -> watermark becomes 30 s
> // batch 3: right (9, 40 s)
> {code}
> - *Expected:* the query starts. Batch 1 outputs the match for key 1. Batch 3
> outputs {{(2, 6 s, null)}}, because the window [0 s, 10 s) of the unmatched
> left row ends at or below the 30-second watermark.
> - *Actual:* the query fails analysis with {{Stream-stream LeftOuter join
> between two streaming DataFrame/Datasets is not supported: it requires a
> watermark on the right input and a time constraint ...}}. The error also
> says: {{With spark.sql.streaming.statefulOperator.allowMultiple=false, the
> right input filters late rows by its first watermarked column, rt, so its
> window key rw is not a time constraint. Put rw before rt in that input's
> columns, or drop rt, to bound state by the window.}}
> An inner join of the same shape runs but never evicts either input's rows.
> h3. Workaround
> Put the window column before the event time column in each input, for example
> {{.select($"lk", $"lw", $"lt")}}, or drop the event time column. The window
> is then the input's first watermarked column, the join filters late rows by
> its end, and it bounds the other input's state by the window end.
> h3. How the affected versions were checked
> *The released lines.* I read {{StreamingJoinHelper.isWatermarkInJoinKeys}}
> (the admission check) and {{StreamingSymmetricHashJoinHelper}} (the eviction
> key ordinal) with {{git show <ref>:<path>}} at these tags:
> ||Tag||Commit||
> |v3.5.9|7c14a3c28b1|
> |v4.0.5|b472a3aa3c4|
> |v4.1.4|45b9b176e6e|
> |v4.2.1|61761839608|
> |v4.3.0|98bc0212c94|
> At every tag, both accept any equality key that carries watermark metadata,
> and a window copies that metadata from its time column. So these lines bound
> the reproducer by window end, and the gap is not there.
> *The unreleased lines.* The gap starts with SPARK-61345, which its migration
> note places in Spark 4.4.0, without a backport to earlier releases. So the
> affected versions are 4.4.0 (branch-4.x) and 5.0.0 (master).
> *The reproducer.* The analysis error and the workaround run are covered by
> the test {{legacy-mode join on a window after its event time names the column
> order}} in {{StreamingJoinWatermarkAdmissionSuite}}. It ran on the
> SPARK-61345 branch with the error hint. The inner join's unbounded state
> comes from reading the code: it has no key ordinal and no range bound. It was
> not run.
> h3. Possible direction
> - *Shared lineage rule:* for each join input, find the watermarked window
> attributes that the time-window rewrite built over the column that reaches
> the join as the input's first watermarked column. Walk down by expression ID:
> - through the rewrite's marked alias for tumbling windows, and its
> {{Expand}} for sliding windows;
> - through nodes that pass values on unchanged: filters, projections of
> attributes and renames, and the watermark node.
> Any other node, including a union or an earlier join, leaves the window
> unrecognized. The output-watermark tracer in {{IncrementalExecution}}, which
> SPARK-61344 extends to follow {{Expand}} and unions, can be a starting point.
> But it runs on the physical plan only.
> - *Same keys everywhere:* the analysis check, the state eviction predicates,
> and the V4 lookup key ordinal must accept exactly the same keys. A practical
> way is to walk the analyzed logical plan once in each place, and give each
> join exec the result, because execution chooses the V4 ordinal without the
> plan.
> - *Accept a recognized key as the event time:* treat a recognized window key
> as the input's event time, bounded by its end, as a window that is the only
> watermarked column is today.
> - *Tests:* legacy inner and left outer joins in both column orders; a window
> over the second watermarked column; a renamed {{ts}}; {{Union}}; a sliding
> window; and a V4 restart.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]