Jungtaek Lim created SPARK-62046:
------------------------------------

             Summary: Stream-stream outer join never outputs a row whose 
watermarked equality key is NULL, on the input that has that watermark
                 Key: SPARK-62046
                 URL: https://issues.apache.org/jira/browse/SPARK-62046
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


h3. Problem

A stream-stream outer join can evict state by an equality key that carries a 
watermark, for
example {{lk = rk AND lw = rw}} with the watermark on the left input's {{lw}}. 
If a left row has
a NULL {{lw}}, the row can never match, so a left outer or full outer join 
should output it
padded with NULLs once the watermark moves, and then drop it from state. 
Instead the row stays
in state for the life of the query and is never output.

The late-row filter keeps such a row, because {{lw <= watermark}} is NULL for 
it. Then:

- State formats 2 and 3: the key eviction predicate is NULL for a NULL key, so 
the row is never
  evicted.
- State format 4: a NULL key is read as event time 0. On the input whose own 
watermark is the
  key, eviction after the first micro-batch scans only above the previous 
watermark, so a row
  at time 0 that arrives after the first micro-batch is never evicted. In the 
first micro-batch
  the scan starts at 0, so the row is output there.

A NULL watermarked key can reach that input through a streaming union, or from 
the NULL-padded
side of an upstream outer join; both keep the watermark metadata.

h3. Reproducer

Scala, {{MemoryStream}}, 
{{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB state
store.

{code:scala}
val withTime = leftInput.toDF().toDF("lk", "t")
  .select($"lk", timestamp_seconds($"t").as("lt"))
  .withWatermark("lt", "10 seconds")
val windowed = withTime.select($"lk", window($"lt", "10 seconds").as("lw"))
val nulls = nullInput.toDF().toDF("lk")
  .select($"lk", lit(null).cast(windowed.schema("lw").dataType).as("lw"))
val rightTime = when($"s" >= 0, timestamp_seconds($"s"))
val right = rightInput.toDF().toDF("rk", "s")
  .select($"rk", when($"s" >= 0, struct(rightTime.as("start"),
    (rightTime + expr("INTERVAL 10 SECONDS")).as("end"))).as("rw"))
val joined = windowed.union(nulls)
  .join(right, expr("lk = rk AND lw = rw"), "left_outer")   // or "full_outer"
  .select($"lk", $"lw.end".cast("long"), $"rk", $"rw.end".cast("long"))

// batch 0: leftInput (1, 5)
// batch 1: nullInput 7
// batch 2: leftInput (2, 100)    -> the watermark moves to 90 seconds
// batch 3: leftInput (no rows)
// batch 4: leftInput (3, 300)
// batch 5: leftInput (no rows)
{code}

The same happens with a timestamp key: use {{$"lt".as("lw")}} instead of the 
window, and
{{rightTime.as("rw")}} on the right.

- *Expected:* {{[1,10,null,null]}}, {{[7,null,null,null]}}, 
{{[2,110,null,null]}}; after batch 5,
  state holds only the row of {{(3, 300)}}.
- *Actual:* {{[1,10,null,null]}}, {{[2,110,null,null]}}. The row {{(7, null)}} 
is never output
  and stays in state (2 state rows after batch 5).

Results (left outer and full outer give the same rows here; full outer has no 
right rows):

||Build||State format 2||State format 3||State format 4||
|master {{4ccce2781cb}}, window key|actual (row never output)|actual|query 
fails: NullPointerException (SPARK-61986)|
|master {{4ccce2781cb}}, timestamp key|actual|actual|actual|
|with SPARK-61986 (#60665), window or timestamp key|actual|actual|actual|

If the NULL-key row arrives in the first micro-batch, state format 4 outputs it 
in that batch,
while state formats 2 and 3 still never output it.

h3. Notes

- Related: SPARK-61986 (#60665) removes the NullPointerException for a NULL 
window key in state
  format 4; with it, the window case behaves like the timestamp case above. 
SPARK-61382 covers a
  different NULL-key problem: NULL keys of both inputs matching each other in a 
full outer join.
- Fix idea: for state format 4, also evict the time-0 slot on each eviction of 
the input whose
  own watermark is the key. For state formats 2 and 3, treat a NULL watermarked 
key as evictable
  at the first eviction. Either way, an outer join then outputs the row once.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to