[
https://issues.apache.org/jira/browse/HIVE-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844339#comment-16844339
]
Gopal V commented on HIVE-21757:
--------------------------------
Found the other data-loss comment in RB - https://reviews.apache.org/r/66485/
"That's a REPL event & the trouble with IOW is that it also destroys commits in
progress with the new base_n files, where n > all previous open txns."
So even if you're compacting transaction 10, if transaction 11,12 are open, you
cannot create a base_13 until 11 & 12 are done. The compactor now blocks behind
writes it is not compacting either.
> ACID: use a new write id for compaction's output instead of the visibility id
> -----------------------------------------------------------------------------
>
> Key: HIVE-21757
> URL: https://issues.apache.org/jira/browse/HIVE-21757
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 4.0.0
> Reporter: Vaibhav Gumashta
> Priority: Major
>
> HIVE-20823 added support for running compaction within a transaction. To
> control the visibility of the output directory, it uses
> base_writeId_visibilityId, where visibilityId is the transaction id of the
> transaction that the compactor ran in. Perhaps we can keep using the
> base_writeId format, by allocating a new writeId for the compactor and
> creating the new base/delta with that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
