[ https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Kudinkin updated HUDI-3322: ---------------------------------- Summary: Rollback Plan for Delta Commits constructed incorrectly for MOR tables (was: Rollback of Delta Commits performed incorrectly for MOR tables) > Rollback Plan for Delta Commits constructed incorrectly for MOR tables > ---------------------------------------------------------------------- > > Key: HUDI-3322 > URL: https://issues.apache.org/jira/browse/HUDI-3322 > Project: Apache Hudi > Issue Type: Bug > Reporter: Alexey Kudinkin > Priority: Blocker > Fix For: 0.11.0 > > > Diving deeper into the issue of HUDI-3279, i've realized that the root-cause > of the problem is actually a Rollback of Delta Commits that is performed > incorrectly for MOR tables. Consider the case below (we will continue to rely > on test of > {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}): > Hoodie Timeline: > {code:java} > alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie > total 400 > drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 . > drwx------ 5 alexey.kudinkin staff 160 Jan 25 12:56 .. > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc > -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56 > .001.deltacommit.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 > .001.deltacommit.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 > .002.deltacommit.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 > .002.deltacommit.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 > .003.deltacommit.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56 > .003.deltacommit.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 > .004.deltacommit.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 > .004.deltacommit.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 > .005.compaction.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57 > .005.compaction.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 > .006.deltacommit.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 > .006.deltacommit.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc > -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 > .007.deltacommit.inflight.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57 > .007.deltacommit.requested.crc > -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08 > .20220125130818473.restore.inflight.crc > drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux > -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 > .hoodie.properties.crc > drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp > -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit > -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56 > 001.deltacommit.inflight > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 > 001.deltacommit.requested > -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit > -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56 > 002.deltacommit.inflight > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 > 002.deltacommit.requested > -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit > -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 > 003.deltacommit.inflight > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56 > 003.deltacommit.requested > -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit > -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 > 004.deltacommit.inflight > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 > 004.deltacommit.requested > -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 > 005.compaction.inflight > -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57 > 005.compaction.requested > -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit > -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 > 006.deltacommit.inflight > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 > 006.deltacommit.requested > -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit > -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57 > 007.deltacommit.inflight > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 > 007.deltacommit.requested > -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08 > 20220125130818473.restore.inflight > drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived > -rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties > drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code} > Commit "007" metadata: > {code:java} > alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat > .hoodie/007.deltacommit > { > "partitionToWriteStats" : { > "2016/03/15" : [ { > "fileId" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0", > "path" : > "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471", > "prevCommit" : "005", > "numWrites" : 57, > "numDeletes" : 0, > "numUpdateWrites" : 57, > "numInserts" : 0, > "totalWriteBytes" : 18054, > "totalWriteErrors" : 0, > "tempPath" : null, > "partitionPath" : "2016/03/15", > "totalLogRecords" : 0, > "totalLogFilesCompacted" : 0, > "totalLogSizeCompacted" : 0, > "totalUpdatedRecordsCompacted" : 0, > "totalLogBlocks" : 0, > "totalCorruptLogBlock" : 0, > "totalRollbackBlocks" : 0, > "fileSizeInBytes" : 18054, > "minEventTime" : null, > "maxEventTime" : null, > "logVersion" : 2, > "logOffset" : 0, > "baseFile" : > "a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet", > "logFiles" : [ > ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398", > ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" ] > } ], > "2015/03/16" : [ { > "fileId" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0", > "path" : > "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472", > "prevCommit" : "005", > "numWrites" : 69, > "numDeletes" : 0, > "numUpdateWrites" : 69, > "numInserts" : 0, > "totalWriteBytes" : 21457, > "totalWriteErrors" : 0, > "tempPath" : null, > "partitionPath" : "2015/03/16", > "totalLogRecords" : 0, > "totalLogFilesCompacted" : 0, > "totalLogSizeCompacted" : 0, > "totalUpdatedRecordsCompacted" : 0, > "totalLogBlocks" : 0, > "totalCorruptLogBlock" : 0, > "totalRollbackBlocks" : 0, > "fileSizeInBytes" : 21457, > "minEventTime" : null, > "maxEventTime" : null, > "logVersion" : 2, > "logOffset" : 0, > "baseFile" : > "0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet", > "logFiles" : [ > ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399", > ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" ] > } ], > "2015/03/17" : [ { > "fileId" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0", > "path" : > "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473", > "prevCommit" : "005", > "numWrites" : 74, > "numDeletes" : 0, > "numUpdateWrites" : 74, > "numInserts" : 0, > "totalWriteBytes" : 22889, > "totalWriteErrors" : 0, > "tempPath" : null, > "partitionPath" : "2015/03/17", > "totalLogRecords" : 0, > "totalLogFilesCompacted" : 0, > "totalLogSizeCompacted" : 0, > "totalUpdatedRecordsCompacted" : 0, > "totalLogBlocks" : 0, > "totalCorruptLogBlock" : 0, > "totalRollbackBlocks" : 0, > "fileSizeInBytes" : 22889, > "minEventTime" : null, > "maxEventTime" : null, > "logVersion" : 2, > "logOffset" : 0, > "baseFile" : > "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet", > "logFiles" : [ > ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400", > ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" ] > } ] > }, > "compacted" : false, > "extraMetadata" : { > "schema" : > "{\"type\":\"record\",\"name\":\"triprec\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"_row_key\",\"type\":\"string\"},{\"name\":\"partition_path\",\"type\":\"string\"},{\"name\":\"rider\",\"type\":\"string\"},{\"name\":\"driver\",\"type\":\"string\"},{\"name\":\"begin_lat\",\"type\":\"double\"},{\"name\":\"begin_lon\",\"type\":\"double\"},{\"name\":\"end_lat\",\"type\":\"double\"},{\"name\":\"end_lon\",\"type\":\"double\"},{\"name\":\"distance_in_meters\",\"type\":\"int\"},{\"name\":\"seconds_since_epoch\",\"type\":\"long\"},{\"name\":\"weight\",\"type\":\"float\"},{\"name\":\"nation\",\"type\":\"bytes\"},{\"name\":\"current_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"current_ts\",\"type\":\"long\"},{\"name\":\"height\",\"type\":{\"type\":\"fixed\",\"name\":\"abc\",\"size\":5,\"logicalType\":\"decimal\",\"precision\":10,\"scale\":6}},{\"name\":\"city_to_state\",\"type\":{\"type\":\"map\",\"values\":\"string\"}},{\"name\":\"fare\",\"type\":{\"type\":\"record\",\"name\":\"fare\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}},{\"name\":\"tip_history\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"tip_history\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}],\"default\":null},\"default\":[]},\"default\":[]},{\"name\":\"_hoodie_is_deleted\",\"type\":\"boolean\",\"default\":false}]}" > }, > "operationType" : "UPSERT", > "totalLogFilesCompacted" : 0, > "fileIdAndRelativePaths" : { > "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0" : > "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473", > "0ac88bb7-ebd4-4384-94f1-593e6827a868-0" : > "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472", > "a685909e-14e8-4adf-b6a2-e0b669e89e61-0" : > "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" > }, > "totalLogRecordsCompacted" : 0, > "totalCompactedRecordsUpdated" : 0, > "totalRecordsDeleted" : 0, > "totalLogFilesSize" : 0, > "totalScanTime" : 0, > "totalCreateTime" : 0, > "totalUpsertTime" : 139, > "minAndMaxEventTime" : { > "Optional.empty" : { > "val" : null, > "present" : false > } > }, > "writePartitionPaths" : [ "2016/03/15", "2015/03/16", "2015/03/17" ] > }% {code} > > From the metadata it's not hard to see that only 3 files have changed as part > of "007" commit: > {code:java} > "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" > "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" > "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" > {code} > However, rollback of the "007" instant produces following plan: > {code:java} > { > "instantToRollback": { > "commitTime": "007", > "action": "deltacommit" > }, > "RollbackRequests": [ > { > "partitionPath": "2015/03/16", > "fileId": "", > "latestBaseInstant": "", > "filesToBeDeleted": [], > "logBlocksToBeDeleted": {} > }, > { > "partitionPath": "2015/03/16", > "fileId": "0ac88bb7-ebd4-4384-94f1-593e6827a868-0", > "latestBaseInstant": "005", > "filesToBeDeleted": [], > "logBlocksToBeDeleted": { > > "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399": > 21464, > > "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472": > 21457 > } > }, > { > "partitionPath": "2015/03/17", > "fileId": "", > "latestBaseInstant": "", > "filesToBeDeleted": [], > "logBlocksToBeDeleted": {} > }, > { > "partitionPath": "2015/03/17", > "fileId": "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0", > "latestBaseInstant": "005", > "filesToBeDeleted": [], > "logBlocksToBeDeleted": { > > "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473": > 22889, > > "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400": > 22881 > } > }, > { > "partitionPath": "2016/03/15", > "fileId": "", > "latestBaseInstant": "", > "filesToBeDeleted": [], > "logBlocksToBeDeleted": {} > }, > { > "partitionPath": "2016/03/15", > "fileId": "a685909e-14e8-4adf-b6a2-e0b669e89e61-0", > "latestBaseInstant": "005", > "filesToBeDeleted": [], > "logBlocksToBeDeleted": { > > "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398": > 18022, > > "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471": > 18054 > } > } > ], > "version": 1 > } {code} > Which contains files that have not been changed in the "007" instant at all. -- This message was sent by Atlassian Jira (v8.20.1#820001)