[
https://issues.apache.org/jira/browse/HIVE-29251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-29251:
----------------------------------
Labels: ACID Compaction hive-4.2.0-must (was: ACID Compaction)
> Hive ACID: HiveConf object shouldn't be shared between multiple cleanup tasks
> -----------------------------------------------------------------------------
>
> Key: HIVE-29251
> URL: https://issues.apache.org/jira/browse/HIVE-29251
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 4.1.0
> Reporter: Denys Kuzmenko
> Assignee: Denys Kuzmenko
> Priority: Major
> Labels: ACID, Compaction, hive-4.2.0-must
>
> The {{HiveConf}} object is shared across multiple cleanup tasks. When the
> Cleaner runs with multiple threads, this can lead to a race condition.
> At the end of a cleanup task, the open transaction cap is lifted to check for
> any deltas that may become eligible for cleanup once the open transaction
> completes. If none are found, the cleanup request is marked complete;
> otherwise, it’s returned to the queue with a “ready-for-cleaning” status.
>
> {code:java}
> conf.set(ValidTxnList.VALID_TXNS_KEY, new ValidReadTxnList().toString());
> dir = AcidUtils.getAcidState(fs, path, conf,
> new ValidReaderWriteIdList(info.getFullTableName(), new long[0], new
> BitSet(), info.highestWriteId, Long.MAX_VALUE),
> Ref.from(false), false,
> dirSnapshots);
> List<Path> remained = subtract(CompactorUtil.getObsoleteDirs(dir,
> isDynPartAbort), deleted);
> if (!remained.isEmpty()) {
> LOG.warn("Remained {} obsolete directories from {}. {}",
> remained.size(), location, CompactorUtil.getDebugInfo(remained));
> } else {
> LOG.debug("All cleared below the watermark: {} from {}",
> info.highestWriteId, location);
> success = true;
> }{code}
> that might override the `{{{}ValidTxnList.VALID_TXNS_KEY`{}}} set in a
> concurrent task, leading to premature cleanup of data still in use by an
> active transaction.
> {code:java}
> private void cleanUsingAcidDir(CompactionInfo ci, String location, long
> minOpenTxn) throws Exception {
> ValidTxnList validTxnList =
> TxnUtils.createValidTxnListForCleaner(txnHandler.getOpenTxns(),
> minOpenTxn, false);
> //save it so that getAcidState() sees it
> conf.set(ValidTxnList.VALID_TXNS_KEY, validTxnList.writeToString());{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
