[ https://issues.apache.org/jira/browse/HIVE-18772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604371#comment-16604371 ]
Hive QA commented on HIVE-18772: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12938398/HIVE-18772.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 14921 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolutionAndCompaction (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testCleanerForTxnToWriteId (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite1 (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite2 (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2 (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testACIDwithSchemaEvolutionAndCompaction (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCleanerForTxnToWriteId (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInitiatorWithMultipleFailedCompactions (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite1 (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite2 (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1 (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2 (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3 (batchId=310) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable (batchId=275) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithUnionAll (batchId=275) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testOperationsOnCompletedTxnComponentsForMmTable (batchId=275) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testSnapshotIsolationWithAbortedTxnOnMmTable (batchId=275) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteForPartitionedMmTable (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithUnionAll (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testOperationsOnCompletedTxnComponentsForMmTable (batchId=297) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testSnapshotIsolationWithAbortedTxnOnMmTable (batchId=297) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=297) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNoBuckets (batchId=299) org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMetastoreTablesCleanup (batchId=314) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction (batchId=284) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase (batchId=284) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction (batchId=284) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction (batchId=284) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction (batchId=284) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMajorPartitionCompaction (batchId=285) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMajorPartitionCompactionNoBase (batchId=285) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMajorTableCompaction (batchId=285) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMinorPartitionCompaction (batchId=285) org.apache.hadoop.hive.ql.txn.compactor.TestCleaner2.cleanupAfterMinorTableCompaction (batchId=285) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMajorPartitionCompaction (batchId=241) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMajorTableCompaction (batchId=241) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMinorPartitionCompaction (batchId=241) org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMinorTableCompaction (batchId=241) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.mmTableOpenWriteId[0] (batchId=240) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.mmTableOpenWriteId[1] (batchId=240) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testTableProperties[0] (batchId=240) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testTableProperties[1] (batchId=240) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13598/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13598/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13598/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 45 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12938398 - PreCommit-HIVE-Build > Make Acid Cleaner use MIN_HISTORY_LEVEL > --------------------------------------- > > Key: HIVE-18772 > URL: https://issues.apache.org/jira/browse/HIVE-18772 > Project: Hive > Issue Type: Improvement > Components: Transactions > Affects Versions: 3.0.0 > Reporter: Eugene Koifman > Assignee: Eugene Koifman > Priority: Major > Attachments: HIVE-18772.01.patch, HIVE-18772.02.patch, > HIVE-18772.02.patch > > > Instead of using Lock Manager state as it currently does. > This will eliminate possible race conditions > See this > [comment|https://issues.apache.org/jira/browse/HIVE-18192?focusedCommentId=16338208&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16338208] > Suppose A is the set of all ValidTxnList across all active readers. Each > ValidTxnList has minOpenTxnId. > MIN_HISTORY_LEVEL allows us to determine X = min(minOpenTxnId) across all > currently active readers > This means that no active transaction in the system sees any txn with txnid < > X as open. > This means if construct ValidTxnIdList with HWM=X-1 and use that in > getAcidState(), any files determined by this call as 'obsolete', will be seen > as obsolete by any existing/future reader, i.e. can be physically deleted. > This is also necessary for multi-statement transactions where relying on the > state of Lock Manager is not sufficient. For example > Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. > 13 commits (via it's parent txn) at t2 > t1. (17 is still running). > Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on > Table1/Part1 (17 is still running) > Now delta_13 may be cleaned since it can be seen as obsolete and there may be > no locks on it, i.e. no one is reading it. > Now at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It cannot > use base_14 is that may have absorbed delete events from delete_delta_14. > Using MIN_HISTORY_LEVEL solves this. > See description of HIVE-18747 for more details on MIN_HISTORY_LEVEL -- This message was sent by Atlassian JIRA (v7.6.3#76005)