[
https://issues.apache.org/jira/browse/SPARK-62046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-62046:
---------------------------------
Affects Version/s: 4.4.0
> Stream-stream outer join never outputs a row whose watermarked equality key
> is NULL, on the input that has that watermark
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62046
> URL: https://issues.apache.org/jira/browse/SPARK-62046
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> h3. Problem
> A stream-stream outer join can evict state by an equality key that carries a
> watermark, for
> example {{lk = rk AND lw = rw}} with the watermark on the left input's
> {{lw}}. If a left row has
> a NULL {{lw}}, the row can never match, so a left outer or full outer join
> should output it
> padded with NULLs once the watermark moves, and then drop it from state.
> Instead the row stays
> in state for the life of the query and is never output.
> The late-row filter keeps such a row, because {{lw <= watermark}} is NULL for
> it. Then:
> - State formats 2 and 3: the key eviction predicate is NULL for a NULL key,
> so the row is never
> evicted.
> - State format 4: a NULL key is read as event time 0. On the input whose own
> watermark is the
> key, eviction after the first micro-batch scans only above the previous
> watermark, so a row
> at time 0 that arrives after the first micro-batch is never evicted. In the
> first micro-batch
> the scan starts at 0, so the row is output there.
> A NULL watermarked key can reach that input through a streaming union, or
> from the NULL-padded
> side of an upstream outer join; both keep the watermark metadata.
> h3. Reproducer
> Scala, {{MemoryStream}},
> {{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB state
> store.
> {code:scala}
> val withTime = leftInput.toDF().toDF("lk", "t")
> .select($"lk", timestamp_seconds($"t").as("lt"))
> .withWatermark("lt", "10 seconds")
> val windowed = withTime.select($"lk", window($"lt", "10 seconds").as("lw"))
> val nulls = nullInput.toDF().toDF("lk")
> .select($"lk", lit(null).cast(windowed.schema("lw").dataType).as("lw"))
> val rightTime = when($"s" >= 0, timestamp_seconds($"s"))
> val right = rightInput.toDF().toDF("rk", "s")
> .select($"rk", when($"s" >= 0, struct(rightTime.as("start"),
> (rightTime + expr("INTERVAL 10 SECONDS")).as("end"))).as("rw"))
> val joined = windowed.union(nulls)
> .join(right, expr("lk = rk AND lw = rw"), "left_outer") // or "full_outer"
> .select($"lk", $"lw.end".cast("long"), $"rk", $"rw.end".cast("long"))
> // batch 0: leftInput (1, 5)
> // batch 1: nullInput 7
> // batch 2: leftInput (2, 100) -> the watermark moves to 90 seconds
> // batch 3: leftInput (no rows)
> // batch 4: leftInput (3, 300)
> // batch 5: leftInput (no rows)
> {code}
> The same happens with a timestamp key: use {{$"lt".as("lw")}} instead of the
> window, and
> {{rightTime.as("rw")}} on the right.
> - *Expected:* {{[1,10,null,null]}}, {{[7,null,null,null]}},
> {{[2,110,null,null]}}; after batch 5,
> state holds only the row of {{(3, 300)}}.
> - *Actual:* {{[1,10,null,null]}}, {{[2,110,null,null]}}. The row {{(7,
> null)}} is never output
> and stays in state (2 state rows after batch 5).
> Results (left outer and full outer give the same rows here; full outer has no
> right rows):
> ||Build||State format 2||State format 3||State format 4||
> |master {{4ccce2781cb}}, window key|actual (row never output)|actual|query
> fails: NullPointerException (SPARK-61986)|
> |master {{4ccce2781cb}}, timestamp key|actual|actual|actual|
> |with SPARK-61986 (#60665), window or timestamp key|actual|actual|actual|
> If the NULL-key row arrives in the first micro-batch, state format 4 outputs
> it in that batch,
> while state formats 2 and 3 still never output it.
> h3. Notes
> - Related: SPARK-61986 (#60665) removes the NullPointerException for a NULL
> window key in state
> format 4; with it, the window case behaves like the timestamp case above.
> SPARK-61382 covers a
> different NULL-key problem: NULL keys of both inputs matching each other in
> a full outer join.
> - Fix idea: for state format 4, also evict the time-0 slot on each eviction
> of the input whose
> own watermark is the key. For state formats 2 and 3, treat a NULL
> watermarked key as evictable
> at the first eviction. Either way, an outer join then outputs the row once.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]