Livia Zhu created SPARK-59877:
---------------------------------
Summary: Fix race condition causing
`STATE_STORE_INVALID_STATE_MACHINE_TRANSITION` for stateful op + CartesianRDD
Key: SPARK-59877
URL: https://issues.apache.org/jira/browse/SPARK-59877
Project: Spark
Issue Type: Documentation
Components: Structured Streaming
Affects Versions: 4.3.0
Reporter: Livia Zhu
{{CartesianProductExec}} can run several output-partition tasks for the same
stateful left partition. Those sibling tasks may run on different executors,
causing the StateStore coordinator to move the active provider and unload the
previous executor's copy.
On that previous executor, {{StateStore.get}} can read the provider from
{{{}loadedProviders{}}}, release the map lock, and then call
{{{}provider.getStore{}}}. If coordinator-driven unload closes that provider in
the gap, {{getStore}} tries to acquire a state-machine stamp from the
now-closed provider and fails with
{{STATE_STORE_INVALID_STATE_MACHINE_TRANSITION}} ({{{}CLOSED{}}} →
{{{}ACQUIRED{}}}, operation {{{}load{}}}). The observed tasks were distinct
first attempts, not retries or speculation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
