[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Mahindra updated HUDI-3322:
--
Sprint: Hudi-Sprint-Jan-24, Hudi-Sprint-Jan-31 (was: Hudi-Sprint-Jan-24)
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
> Components: metadata
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
>
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Story Points: 0 (was: 4)
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
> Components: metadata
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-3322:
--
Component/s: metadata
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
> Components: metadata
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Reviewers: sivabalan narayanan, Y Ethan Guo
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Status: Patch Available (was: In Progress)
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-3322:
-
Labels: pull-request-available (was: )
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Story Points: 4 (was: 2)
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Mahindra updated HUDI-3322:
--
Status: In Progress (was: Open)
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Epic Link: HUDI-3081
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Story Points: 2
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Sprint: Hudi-Sprint-Jan-24
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Assignee: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback Plan for Delta Commits is composed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Description:
Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of
the problem is actually a Rollback Plan for Delta Commits is composed
incorrectly for MOR tables. Consider the case below (we will continue to rely
on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
Hoodie Timeline:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
total 400
drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
.001.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.001.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
.002.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.002.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.003.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.003.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.004.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.004.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.005.compaction.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
.005.compaction.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.006.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.006.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.007.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.007.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
.20220125130818473.restore.inflight.crc
drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
-rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
-rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
001.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
001.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
002.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
002.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
003.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
003.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
004.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
004.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
-rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
005.compaction.requested
-rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
006.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
006.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
007.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
007.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08
20220125130818473.restore.inflight
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
-rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
Data Folder layout:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -al **/**
-rw-r--r-- 1 alexey.kudinkin staff 447405 Jan 25 12:56
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Description:
Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of
the problem is actually a Rollback Plan for Delta Commits is composed
incorrectly for MOR tables. Consider the case below (we will continue to rely
on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
Hoodie Timeline:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
total 400
drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
.001.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.001.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
.002.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.002.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.003.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.003.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.004.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.004.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.005.compaction.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
.005.compaction.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.006.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.006.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.007.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.007.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
.20220125130818473.restore.inflight.crc
drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
-rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
-rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
001.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
001.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
002.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
002.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
003.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
003.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
004.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
004.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
-rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
005.compaction.requested
-rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
006.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
006.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
007.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
007.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08
20220125130818473.restore.inflight
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
-rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
Data Folder layout:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -al **/**
-rw-r--r-- 1 alexey.kudinkin staff 448853 Jan 25 12:57
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Description:
Diving deeper into the issue of HUDI-3279, i've realized that the root-cause of
the problem is actually a Rollback Plan for Delta Commits is composed
incorrectly for MOR tables. Consider the case below (we will continue to rely
on test of {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
Hoodie Timeline:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
total 400
drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
.001.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.001.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
.002.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.002.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.003.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
.003.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.004.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.004.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.005.compaction.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
.005.compaction.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.006.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.006.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
-rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
.007.deltacommit.inflight.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
.007.deltacommit.requested.crc
-rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
.20220125130818473.restore.inflight.crc
drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
-rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56 .hoodie.properties.crc
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
-rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
001.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
001.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
002.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
002.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
003.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
003.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
004.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
004.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57 005.compaction.inflight
-rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
005.compaction.requested
-rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
006.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
006.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 5551 Jan 25 12:57 007.deltacommit
-rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
007.deltacommit.inflight
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
007.deltacommit.requested
-rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 13:08
20220125130818473.restore.inflight
drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:56 archived
-rw-r--r-- 1 alexey.kudinkin staff 347 Jan 25 12:56 hoodie.properties
drwxr-xr-x 4 alexey.kudinkin staff 128 Jan 25 12:56 metadata {code}
Commit "007" metadata:
{code:java}
alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat
.hoodie/007.deltacommit
{
"partitionToWriteStats" : {
"2016/03/15" : [ {
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Summary: Rollback Plan for Delta Commits constructed incorrectly (was:
Rollback Plan for Delta Commits constructed incorrectly for MOR tables)
> Rollback Plan for Delta Commits constructed incorrectly
> ---
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback of Delta Commits that is performed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff
[jira] [Updated] (HUDI-3322) Rollback Plan for Delta Commits constructed incorrectly for MOR tables
[
https://issues.apache.org/jira/browse/HUDI-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-3322:
--
Summary: Rollback Plan for Delta Commits constructed incorrectly for MOR
tables (was: Rollback of Delta Commits performed incorrectly for MOR tables)
> Rollback Plan for Delta Commits constructed incorrectly for MOR tables
> --
>
> Key: HUDI-3322
> URL: https://issues.apache.org/jira/browse/HUDI-3322
> Project: Apache Hudi
> Issue Type: Bug
>Reporter: Alexey Kudinkin
>Priority: Blocker
> Fix For: 0.11.0
>
>
> Diving deeper into the issue of HUDI-3279, i've realized that the root-cause
> of the problem is actually a Rollback of Delta Commits that is performed
> incorrectly for MOR tables. Consider the case below (we will continue to rely
> on test of
> {{{}TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore{}}}):
> Hoodie Timeline:
> {code:java}
> alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
> total 400
> drwxr-xr-x 52 alexey.kudinkin staff 1664 Jan 25 13:08 .
> drwx-- 5 alexey.kudinkin staff 160 Jan 25 12:56 ..
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56 .001.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:56
> .001.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .001.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:56 .002.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:56
> .002.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .002.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .003.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .003.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:56
> .003.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 56 Jan 25 12:57 .004.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .004.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .004.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57 .005.commit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .005.compaction.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 28 Jan 25 12:57
> .005.compaction.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .006.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .006.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .006.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 52 Jan 25 12:57 .007.deltacommit.crc
> -rw-r--r-- 1 alexey.kudinkin staff 48 Jan 25 12:57
> .007.deltacommit.inflight.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 12:57
> .007.deltacommit.requested.crc
> -rw-r--r-- 1 alexey.kudinkin staff 8 Jan 25 13:08
> .20220125130818473.restore.inflight.crc
> drwxr-xr-x 5 alexey.kudinkin staff 160 Jan 25 12:57 .aux
> -rw-r--r-- 1 alexey.kudinkin staff 12 Jan 25 12:56
> .hoodie.properties.crc
> drwxr-xr-x 2 alexey.kudinkin staff 64 Jan 25 12:57 .temp
> -rw-r--r-- 1 alexey.kudinkin staff 4822 Jan 25 12:56 001.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 2499 Jan 25 12:56
> 001.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 001.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5451 Jan 25 12:56 002.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:56
> 002.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 002.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5646 Jan 25 12:57 003.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 003.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:56
> 003.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5835 Jan 25 12:57 004.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 004.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 004.deltacommit.requested
> -rw-r--r-- 1 alexey.kudinkin staff 4756 Jan 25 12:57 005.commit
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 005.compaction.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 2507 Jan 25 12:57
> 005.compaction.requested
> -rw-r--r-- 1 alexey.kudinkin staff 5362 Jan 25 12:57 006.deltacommit
> -rw-r--r-- 1 alexey.kudinkin staff 4620 Jan 25 12:57
> 006.deltacommit.inflight
> -rw-r--r-- 1 alexey.kudinkin staff 0 Jan 25 12:57
> 006.deltacommit.requested
>
