Herman van Hövell created SPARK-39366:
-----------------------------------------
Summary: BlockInfoManager should not release write locks on task
end
Key: SPARK-39366
URL: https://issues.apache.org/jira/browse/SPARK-39366
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.1
Reporter: Herman van Hövell
Assignee: Herman van Hövell
The BlockInfoManager releases all locks held by a task when the task is done.
It also release write locks, the problem with that is that a thread (other than
the main task thread) might still be modifying the block. By releasing it the
block now seems readable, and a reader might observe a block in a partial or
non-existent state.
This is fortunately not the massive problem as it appears to be because the
BlockManager (only place where we write blocks) is well behaved and always puts
the block in a consistent state. This means the errors caused by this are only
transient.
Given the fact that the write code is well behaved we don't need to release the
write locks on task end. We should remove that.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
