Jungtaek Lim created SPARK-62057:
------------------------------------
Summary: Stream-stream left semi join over a streaming union fails
with a missing state file when a union input gets its first data after the
first micro-batch
Key: SPARK-62057
URL: https://issues.apache.org/jira/browse/SPARK-62057
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
h3. Problem
A stream-stream left semi join whose left input is a streaming union can fail
in the second
micro-batch with {{CANNOT_LOAD_STATE_STORE}} ("... state/1/2/1.changelog does
not exist"). It
happens when one union input has no data in the first micro-batch and gets data
later.
The physical plan changes between the batches. In batch 0 the plan has one
{{StreamingSymmetricHashJoinExec}} (operator id 0). In batch 1 a stateful
operator with id 1
tries to load state version 1, which batch 0 never wrote. This fits the
optimizer pushing the
left semi join below the Union ({{PushDownLeftSemiAntiJoin}}) once both union
inputs have data,
which gives one join per union input; the batch 1 plan itself was not captured,
because the
batch fails.
h3. Reproducer
Scala, {{MemoryStream}},
{{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB.
Fails on state formats 2, 3 and 4.
{code:scala}
val first = input1.toDF().toDF("lk", "t")
.select($"lk", timestamp_seconds($"t").as("lt"))
.withWatermark("lt", "10 seconds")
val second = input2.toDF().toDF("lk", "t")
.select($"lk", timestamp_seconds($"t").as("lt")) // no watermark on this
input
val right = rightInput.toDF().toDF("rk", "s")
.select($"rk", timestamp_seconds($"s").as("rt"))
val joined = first.union(second)
.join(right, expr("lk = rk AND lt = rt"), "left_semi")
.select($"lk", $"lt".cast("long"))
// batch 0: input1 (1, 5) and rightInput (1, 5) -> (1, 5); one stream-stream
join, id 0
// batch 1: input2 (2, 6) and rightInput (2, 6) -> expected (2, 6)
{code}
- *Expected:* {{(1, 5)}}, then {{(2, 6)}}.
- *Actual:* batch 1 fails with
{{[CANNOT_LOAD_STATE_STORE.CANNOT_READ_STREAMING_STATE_FILE]}},
for {{state/1/<partition>/1.changelog}}.
||State format||Result on unmodified master {{4ccce2781cb}}||
|2|fails in batch 1|
|3|fails in batch 1|
|4|fails in batch 1|
Note: a variant where both union inputs and the right input have watermarks,
with
{{noDataMicroBatches}} on, did not fail; there the final plan had two joins.
Which shapes avoid
the plan change was not narrowed down.
h3. Notes
- Reproduced on unmodified master {{4ccce2781cb}}, with the query above: batch
0 plans one
stream-stream join (operator id 0), and batch 1 fails loading
{{state/1/2/1.changelog}}.
- Fix idea: the number of stateful operators must not depend on whether a
source has data in a
batch. Either do not push a streaming left semi or anti join below a Union,
or plan an empty
source so that the plan shape stays the same across batches.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]