[ https://issues.apache.org/jira/browse/HUDI-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-2476: -------------------------------------- Parent: HUDI-1292 Issue Type: Sub-task (was: Improvement) > Fix retried compaction commit in datatable fails when applied to metadata w/ > sync updates > ----------------------------------------------------------------------------------------- > > Key: HUDI-2476 > URL: https://issues.apache.org/jira/browse/HUDI-2476 > Project: Apache Hudi > Issue Type: Sub-task > Components: Writer Core > Reporter: sivabalan narayanan > Assignee: sivabalan narayanan > Priority: Major > Fix For: 0.10.0 > > > Compaction and clustering has a static instant time. So, when retried it may > not have a new instant time, but the same. > So, lets walk through the scenario of what happens when compaction fails > after synced to metadata table. > c1, c2, cc3( compaction commit), c4. > c1, c2, c4 on completion is applied to metadata table. > cc3 also gets synced to metadata table, but before committing to data table, > it failed(process crashed). Its is a small window, but still a possibility. > So, from a timeline standpoint this is what looks like > data timeline: > c1 complete, c2 complete, cc3 inflight. c4 complete. > metadata timeline: > dc_c1, dc_c2, dc_cc3, dc_c4 > Lets say there are few more commits went in. > data timeline: > c1 complete, c2 complete, cc3 inflight. c4 complete, c5 complete. c6 complete. > metadata timeline: > dc_c1, dc_c2, dc_cc3, dc_c4, dc_c5, dc_c6 > > Now, compaction in datatable is being re-attempted. So, first we rollback > pending compaction in data table. So, this will trigger an upsert to metadata > table. even thought this is a rollback, all updates to metadata table is an > upsert which would result in a delta table. > And then, the compaction will be retried in datatable. when this is nearing > completion, we try to upsert to metadata table. which will fail. because > already we have a completed dc_cc3 in metadata table. > > Fix: > when a commit is being retried, we delete the completed instant and then > proceed with upsert. So, when log blocks/files are merged together, final > state will be intact and will ensure only those files added in 2nd attempt is > returned and those added during 1st attempt is not returned (since there will > be complimenting log block corresponding to a rollback). > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)