Jungtaek Lim created SPARK-62057:
------------------------------------

             Summary: Stream-stream left semi join over a streaming union fails 
with a missing state file when a union input gets its first data after the 
first micro-batch
                 Key: SPARK-62057
                 URL: https://issues.apache.org/jira/browse/SPARK-62057
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


h3. Problem

A stream-stream left semi join whose left input is a streaming union can fail 
in the second
micro-batch with {{CANNOT_LOAD_STATE_STORE}} ("... state/1/2/1.changelog does 
not exist"). It
happens when one union input has no data in the first micro-batch and gets data 
later.

The physical plan changes between the batches. In batch 0 the plan has one
{{StreamingSymmetricHashJoinExec}} (operator id 0). In batch 1 a stateful 
operator with id 1
tries to load state version 1, which batch 0 never wrote. This fits the 
optimizer pushing the
left semi join below the Union ({{PushDownLeftSemiAntiJoin}}) once both union 
inputs have data,
which gives one join per union input; the batch 1 plan itself was not captured, 
because the
batch fails.

h3. Reproducer

Scala, {{MemoryStream}}, 
{{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB.
Fails on state formats 2, 3 and 4.

{code:scala}
val first = input1.toDF().toDF("lk", "t")
  .select($"lk", timestamp_seconds($"t").as("lt"))
  .withWatermark("lt", "10 seconds")
val second = input2.toDF().toDF("lk", "t")
  .select($"lk", timestamp_seconds($"t").as("lt"))      // no watermark on this 
input
val right = rightInput.toDF().toDF("rk", "s")
  .select($"rk", timestamp_seconds($"s").as("rt"))
val joined = first.union(second)
  .join(right, expr("lk = rk AND lt = rt"), "left_semi")
  .select($"lk", $"lt".cast("long"))

// batch 0: input1 (1, 5) and rightInput (1, 5)  -> (1, 5); one stream-stream 
join, id 0
// batch 1: input2 (2, 6) and rightInput (2, 6)  -> expected (2, 6)
{code}

- *Expected:* {{(1, 5)}}, then {{(2, 6)}}.
- *Actual:* batch 1 fails with 
{{[CANNOT_LOAD_STATE_STORE.CANNOT_READ_STREAMING_STATE_FILE]}},
  for {{state/1/<partition>/1.changelog}}.

||State format||Result on unmodified master {{4ccce2781cb}}||
|2|fails in batch 1|
|3|fails in batch 1|
|4|fails in batch 1|

Note: a variant where both union inputs and the right input have watermarks, 
with
{{noDataMicroBatches}} on, did not fail; there the final plan had two joins. 
Which shapes avoid
the plan change was not narrowed down.

h3. Notes

- Reproduced on unmodified master {{4ccce2781cb}}, with the query above: batch 
0 plans one
  stream-stream join (operator id 0), and batch 1 fails loading 
{{state/1/2/1.changelog}}.
- Fix idea: the number of stateful operators must not depend on whether a 
source has data in a
  batch. Either do not push a streaming left semi or anti join below a Union, 
or plan an empty
  source so that the plan shape stays the same across batches.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to