Jungtaek Lim created SPARK-62046:
------------------------------------
Summary: Stream-stream outer join never outputs a row whose
watermarked equality key is NULL, on the input that has that watermark
Key: SPARK-62046
URL: https://issues.apache.org/jira/browse/SPARK-62046
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
h3. Problem
A stream-stream outer join can evict state by an equality key that carries a
watermark, for
example {{lk = rk AND lw = rw}} with the watermark on the left input's {{lw}}.
If a left row has
a NULL {{lw}}, the row can never match, so a left outer or full outer join
should output it
padded with NULLs once the watermark moves, and then drop it from state.
Instead the row stays
in state for the life of the query and is never output.
The late-row filter keeps such a row, because {{lw <= watermark}} is NULL for
it. Then:
- State formats 2 and 3: the key eviction predicate is NULL for a NULL key, so
the row is never
evicted.
- State format 4: a NULL key is read as event time 0. On the input whose own
watermark is the
key, eviction after the first micro-batch scans only above the previous
watermark, so a row
at time 0 that arrives after the first micro-batch is never evicted. In the
first micro-batch
the scan starts at 0, so the row is output there.
A NULL watermarked key can reach that input through a streaming union, or from
the NULL-padded
side of an upstream outer join; both keep the watermark metadata.
h3. Reproducer
Scala, {{MemoryStream}},
{{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB state
store.
{code:scala}
val withTime = leftInput.toDF().toDF("lk", "t")
.select($"lk", timestamp_seconds($"t").as("lt"))
.withWatermark("lt", "10 seconds")
val windowed = withTime.select($"lk", window($"lt", "10 seconds").as("lw"))
val nulls = nullInput.toDF().toDF("lk")
.select($"lk", lit(null).cast(windowed.schema("lw").dataType).as("lw"))
val rightTime = when($"s" >= 0, timestamp_seconds($"s"))
val right = rightInput.toDF().toDF("rk", "s")
.select($"rk", when($"s" >= 0, struct(rightTime.as("start"),
(rightTime + expr("INTERVAL 10 SECONDS")).as("end"))).as("rw"))
val joined = windowed.union(nulls)
.join(right, expr("lk = rk AND lw = rw"), "left_outer") // or "full_outer"
.select($"lk", $"lw.end".cast("long"), $"rk", $"rw.end".cast("long"))
// batch 0: leftInput (1, 5)
// batch 1: nullInput 7
// batch 2: leftInput (2, 100) -> the watermark moves to 90 seconds
// batch 3: leftInput (no rows)
// batch 4: leftInput (3, 300)
// batch 5: leftInput (no rows)
{code}
The same happens with a timestamp key: use {{$"lt".as("lw")}} instead of the
window, and
{{rightTime.as("rw")}} on the right.
- *Expected:* {{[1,10,null,null]}}, {{[7,null,null,null]}},
{{[2,110,null,null]}}; after batch 5,
state holds only the row of {{(3, 300)}}.
- *Actual:* {{[1,10,null,null]}}, {{[2,110,null,null]}}. The row {{(7, null)}}
is never output
and stays in state (2 state rows after batch 5).
Results (left outer and full outer give the same rows here; full outer has no
right rows):
||Build||State format 2||State format 3||State format 4||
|master {{4ccce2781cb}}, window key|actual (row never output)|actual|query
fails: NullPointerException (SPARK-61986)|
|master {{4ccce2781cb}}, timestamp key|actual|actual|actual|
|with SPARK-61986 (#60665), window or timestamp key|actual|actual|actual|
If the NULL-key row arrives in the first micro-batch, state format 4 outputs it
in that batch,
while state formats 2 and 3 still never output it.
h3. Notes
- Related: SPARK-61986 (#60665) removes the NullPointerException for a NULL
window key in state
format 4; with it, the window case behaves like the timestamp case above.
SPARK-61382 covers a
different NULL-key problem: NULL keys of both inputs matching each other in a
full outer join.
- Fix idea: for state format 4, also evict the time-0 slot on each eviction of
the input whose
own watermark is the key. For state formats 2 and 3, treat a NULL watermarked
key as evictable
at the first eviction. Either way, an outer join then outputs the row once.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]