[
https://issues.apache.org/jira/browse/SPARK-47036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-47036.
---------------------------------
> RocksDB versionID Mismatch in SST files with Compaction
> -------------------------------------------------------
>
> Key: SPARK-47036
> URL: https://issues.apache.org/jira/browse/SPARK-47036
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.5.0, 3.5.1, 3.5.2, 4.0.0
> Reporter: Bhuwan Sahni
> Assignee: Bhuwan Sahni
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.2, 4.0.0
>
>
> RocksDB compaction can result in version Id mismatch errors if the same
> version is committed twice from the same executor. (Multiple commits can
> happen due to Spark Stage/task retry).
> A particular scenario where this can happen is provided below:
> # Version V1 is loaded on executor A, RocksDB State Store has 195.sst,
> 196.sst, 197.sst and 198.sst files.
> 2. State changes are made, which result in creation of a new table file
> 200.sst.
> 3. State store is committed as version V2. The SST file 200.sst (as
> 000200-8c80161a-bc23-4e3b-b175-cffe38e427c7.sst) is uploaded to DFS, and
> previous 4 files are reused. A new metadata file is created to track the
> exact SST files with unique IDs, and uploaded with RocksDB Manifest as part
> of V1.zip.
> 4. Rocks DB compaction is triggered at the same time. The compaction creates
> a new L1 file (201.sst), and deletes existing 5 SST files.
> 5. Spark Stage is retried.
> 6. Version V1 is reloaded on the same executor. The local files are
> inspected, and 201.sst is deleted. The 4 SST files in version V1 are
> downloaded again to local file system.
> 7. Any local files which are deleted (as part of version load) are also
> removed from local → DFS file upload tracking. **However, the files already
> deleted as a result of compaction are not removed from tracking. This is the
> bug which resulted in the failure.**
> 8. State store is committed as version V1. However, the local mapping of SST
> files to DFS file path still has 200.sst in its tracking, hence the SST file
> is not re-uploaded. A new metadata file is created to track the exact SST
> files with unique IDs, and uploaded with the new RocksDB Manifest as part of
> V2.zip. (The V2.zip file is overwritten here atomically)
> 9. A new executor tried to load version V2. However, the SST files in (1) are
> now incompatible with Manifest file in (6) resulting in the version Id
> mismatch failure.
>
> We need to ensure that any files deleted from local filesystem post
> compaction are not tracked in uploadedDFSFiles mapping if the same version is
> loaded again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
