[ 
https://issues.apache.org/jira/browse/SPARK-62057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-62057:
---------------------------------
    Affects Version/s: 4.3.0

> Stream-stream left semi join over a streaming union fails with a missing 
> state file when a union input gets its first data after the first micro-batch
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62057
>                 URL: https://issues.apache.org/jira/browse/SPARK-62057
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> h3. Problem
> A stream-stream left semi join whose left input is a streaming union can fail 
> in the second
> micro-batch with {{CANNOT_LOAD_STATE_STORE}} ("... state/1/2/1.changelog does 
> not exist"). It
> happens when one union input has no data in the first micro-batch and gets 
> data later.
> The physical plan changes between the batches. In batch 0 the plan has one
> {{StreamingSymmetricHashJoinExec}} (operator id 0). In batch 1 a stateful 
> operator with id 1
> tries to load state version 1, which batch 0 never wrote. This fits the 
> optimizer pushing the
> left semi join below the Union ({{PushDownLeftSemiAntiJoin}}) once both union 
> inputs have data,
> which gives one join per union input; the batch 1 plan itself was not 
> captured, because the
> batch fails.
> h3. Reproducer
> Scala, {{MemoryStream}}, 
> {{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB.
> Fails on state formats 2, 3 and 4.
> {code:scala}
> val first = input1.toDF().toDF("lk", "t")
>   .select($"lk", timestamp_seconds($"t").as("lt"))
>   .withWatermark("lt", "10 seconds")
> val second = input2.toDF().toDF("lk", "t")
>   .select($"lk", timestamp_seconds($"t").as("lt"))      // no watermark on 
> this input
> val right = rightInput.toDF().toDF("rk", "s")
>   .select($"rk", timestamp_seconds($"s").as("rt"))
> val joined = first.union(second)
>   .join(right, expr("lk = rk AND lt = rt"), "left_semi")
>   .select($"lk", $"lt".cast("long"))
> // batch 0: input1 (1, 5) and rightInput (1, 5)  -> (1, 5); one stream-stream 
> join, id 0
> // batch 1: input2 (2, 6) and rightInput (2, 6)  -> expected (2, 6)
> {code}
> - *Expected:* {{(1, 5)}}, then {{(2, 6)}}.
> - *Actual:* batch 1 fails with 
> {{[CANNOT_LOAD_STATE_STORE.CANNOT_READ_STREAMING_STATE_FILE]}},
>   for {{state/1/<partition>/1.changelog}}.
> ||State format||Result on unmodified master {{4ccce2781cb}}||
> |2|fails in batch 1|
> |3|fails in batch 1|
> |4|fails in batch 1|
> Note: a variant where both union inputs and the right input have watermarks, 
> with
> {{noDataMicroBatches}} on, did not fail; there the final plan had two joins. 
> Which shapes avoid
> the plan change was not narrowed down.
> h3. Notes
> - Reproduced on unmodified master {{4ccce2781cb}}, with the query above: 
> batch 0 plans one
>   stream-stream join (operator id 0), and batch 1 fails loading 
> {{state/1/2/1.changelog}}.
> - Fix idea: the number of stateful operators must not depend on whether a 
> source has data in a
>   batch. Either do not push a streaming left semi or anti join below a Union, 
> or plan an empty
>   source so that the plan shape stays the same across batches.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to