[ https://issues.apache.org/jira/browse/SPARK-47036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-47036. ---------------------------------- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45092 [https://github.com/apache/spark/pull/45092] > RocksDB versionID Mismatch in SST files with Compaction > ------------------------------------------------------- > > Key: SPARK-47036 > URL: https://issues.apache.org/jira/browse/SPARK-47036 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 3.5.0, 4.0.0, 3.5.1, 3.5.2 > Reporter: Bhuwan Sahni > Assignee: Bhuwan Sahni > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > RocksDB compaction can result in version Id mismatch errors if the same > version is committed twice from the same executor. (Multiple commits can > happen due to Spark Stage/task retry). > A particular scenario where this can happen is provided below: > # Version V1 is loaded on executor A, RocksDB State Store has 195.sst, > 196.sst, 197.sst and 198.sst files. > 2. State changes are made, which result in creation of a new table file > 200.sst. > 3. State store is committed as version V2. The SST file 200.sst (as > 000200-8c80161a-bc23-4e3b-b175-cffe38e427c7.sst) is uploaded to DFS, and > previous 4 files are reused. A new metadata file is created to track the > exact SST files with unique IDs, and uploaded with RocksDB Manifest as part > of V1.zip. > 4. Rocks DB compaction is triggered at the same time. The compaction creates > a new L1 file (201.sst), and deletes existing 5 SST files. > 5. Spark Stage is retried. > 6. Version V1 is reloaded on the same executor. The local files are > inspected, and 201.sst is deleted. The 4 SST files in version V1 are > downloaded again to local file system. > 7. Any local files which are deleted (as part of version load) are also > removed from local → DFS file upload tracking. **However, the files already > deleted as a result of compaction are not removed from tracking. This is the > bug which resulted in the failure.** > 8. State store is committed as version V1. However, the local mapping of SST > files to DFS file path still has 200.sst in its tracking, hence the SST file > is not re-uploaded. A new metadata file is created to track the exact SST > files with unique IDs, and uploaded with the new RocksDB Manifest as part of > V2.zip. (The V2.zip file is overwritten here atomically) > 9. A new executor tried to load version V2. However, the SST files in (1) are > now incompatible with Manifest file in (6) resulting in the version Id > mismatch failure. > > We need to ensure that any files deleted from local filesystem post > compaction are not tracked in uploadedDFSFiles mapping if the same version is > loaded again. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org