[
https://issues.apache.org/jira/browse/SPARK-52637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007912#comment-18007912
]
Anish Shrigondekar commented on SPARK-52637:
--------------------------------------------
PR merged here -> https://github.com/apache/spark/pull/51520
> Fix VERSIONID_MISMATCH error when RocksDB compaction leads to incorrect file
> mapping during load
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-52637
> URL: https://issues.apache.org/jira/browse/SPARK-52637
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Livia Zhu
> Priority: Major
> Labels: pull-request-available
>
> We found a bug leading to checkpoint corruption with RocksDB VersionID
> Mismatch error due to local file mappings not being cleared correctly when
> native RocksDB does a compaction resulting in the original SST file not being
> deleted. This resulted in the DFS file UUID being reused for 2 SST files that
> were not the same, leading the version ID mismatch.
> We need a change that purges from the local file mappings any mapping that
> was created by a version equal or greater than the one loaded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
