[
https://issues.apache.org/jira/browse/SPARK-59877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59877:
-----------------------------------
Labels: pull-request-available (was: )
> Fix race condition causing `STATE_STORE_INVALID_STATE_MACHINE_TRANSITION` for
> stateful op + CartesianRDD
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59877
> URL: https://issues.apache.org/jira/browse/SPARK-59877
> Project: Spark
> Issue Type: Documentation
> Components: Structured Streaming
> Affects Versions: 4.3.0
> Reporter: Livia Zhu
> Priority: Major
> Labels: pull-request-available
>
> {{CartesianProductExec}} can run several output-partition tasks for the same
> stateful left partition. Those sibling tasks may run on different executors,
> causing the StateStore coordinator to move the active provider and unload the
> previous executor's copy.
> On that previous executor, {{StateStore.get}} can read the provider from
> {{{}loadedProviders{}}}, release the map lock, and then call
> {{{}provider.getStore{}}}. If coordinator-driven unload closes that provider
> in the gap, {{getStore}} tries to acquire a state-machine stamp from the
> now-closed provider and fails with
> {{STATE_STORE_INVALID_STATE_MACHINE_TRANSITION}} ({{{}CLOSED{}}} →
> {{{}ACQUIRED{}}}, operation {{{}load{}}}). The observed tasks were distinct
> first attempts, not retries or speculation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
