[
https://issues.apache.org/jira/browse/HIVE-27019?focusedWorklogId=845607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-845607
]
ASF GitHub Bot logged work on HIVE-27019:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Feb/23 10:51
Start Date: 15/Feb/23 10:51
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #4032:
URL: https://github.com/apache/hive/pull/4032#discussion_r1106898869
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/handler/Handler.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor.handler;
+
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.ql.io.AcidDirectory;
+import org.apache.hadoop.hive.ql.txn.compactor.CleaningRequest;
+import org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil;
+import org.apache.thrift.TBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hive.metastore.HMSHandler.getMSForConf;
+import static
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.getDefaultCatalog;
+
+/**
+ * An abstract class which defines the list of utility methods for performing
cleanup activities.
+ */
+public abstract class Handler<T extends CleaningRequest> {
Review Comment:
Class name could be sth more meaningful like CleaningRequestHandler
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CleaningRequest.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.util.List;
+
+/**
+ * A class which specifies the required information for cleanup.
+ * Objects from this class are passed to FSRemover for cleanup.
+ */
+public class CleaningRequest {
+ public enum RequestType {
+ COMPACTION,
+ }
+ private final RequestType type;
+ private final String location;
+ private final List<Path> obsoleteDirs;
+ private final boolean purge;
+ private final FileSystem fs;
+ private String runAs;
+ protected String cleanerMetric;
+ protected String dbName;
+ protected String tableName;
+ protected String partitionName;
+ protected boolean dropPartition;
+ protected String fullPartitionName;
Review Comment:
The only place where these are written is in CompactionCleaningRequest. I
don't understand why these fields are protected and set directly in
CompactionCleaningRequest, while other fields are private final and set through
the constructor?
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/handler/Handler.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor.handler;
+
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.ql.io.AcidDirectory;
+import org.apache.hadoop.hive.ql.txn.compactor.CleaningRequest;
+import org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil;
+import org.apache.thrift.TBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hadoop.hive.metastore.HMSHandler.getMSForConf;
+import static
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.getDefaultCatalog;
+
+/**
+ * An abstract class which defines the list of utility methods for performing
cleanup activities.
+ */
+public abstract class Handler<T extends CleaningRequest> {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(Handler.class.getName());
+ protected final TxnStore txnHandler;
+ protected final HiveConf conf;
+ protected final boolean metricsEnabled;
+ private Optional<Cache<String, TBase>> metaCache;
+
+ Handler(HiveConf conf, TxnStore txnHandler, boolean metricsEnabled) {
+ this.conf = conf;
+ this.txnHandler = txnHandler;
+ boolean tableCacheOn = MetastoreConf.getBoolVar(this.conf,
MetastoreConf.ConfVars.COMPACTOR_CLEANER_TABLECACHE_ON);
+ this.metaCache = initializeCache(tableCacheOn);
+ this.metricsEnabled = metricsEnabled;
+ }
+
+ public HiveConf getConf() {
+ return conf;
+ }
+
+ public TxnStore getTxnHandler() {
+ return txnHandler;
+ }
+
+ public boolean isMetricsEnabled() {
+ return metricsEnabled;
+ }
+
+ /**
+ * Find the list of objects which are ready for cleaning.
+ * @return Cleaning requests
+ */
+ public abstract List<T> findReadyToClean() throws MetaException;
+
+ /**
+ * Execute just before cleanup
+ * @param cleaningRequest - Cleaning request
+ */
+ public abstract void beforeExecutingCleaningRequest(T cleaningRequest)
throws MetaException;
+
+ /**
+ * Execute just after cleanup
+ * @param cleaningRequest Cleaning request
+ * @param deletedFiles List of deleted files
+ * @return True if cleanup was successful, false otherwise
+ * @throws MetaException
+ */
+ public abstract boolean afterExecutingCleaningRequest(T cleaningRequest,
List<Path> deletedFiles) throws MetaException;
+
+ /**
+ * Execute in the event of failure
+ * @param cleaningRequest Cleaning request
+ * @param ex Failure exception
+ * @throws MetaException
+ */
+ public abstract void failureExecutingCleaningRequest(T cleaningRequest,
Exception ex) throws MetaException;
+
+ public Table resolveTable(String dbName, String tableName) throws
MetaException {
+ try {
+ return getMSForConf(conf).getTable(getDefaultCatalog(conf), dbName,
tableName);
+ } catch (MetaException e) {
+ LOG.error("Unable to find table {}.{}, {}", dbName, tableName,
e.getMessage());
+ throw e;
+ }
+ }
+
+ protected Partition resolvePartition(String dbName, String tableName, String
partName) throws MetaException {
+ if (partName != null) {
+ List<Partition> parts;
+ try {
+ parts = CompactorUtil.getPartitionsByNames(conf, dbName, tableName,
partName);
+ if (parts == null || parts.isEmpty()) {
+ // The partition got dropped before we went looking for it.
+ return null;
+ }
+ } catch (Exception e) {
+ LOG.error("Unable to find partition: {}.{}.{}", dbName, tableName,
partName, e);
+ throw e;
+ }
+ if (parts.size() != 1) {
+ LOG.error("{}.{}.{} does not refer to a single partition. {}", dbName,
tableName, partName,
+ Arrays.toString(parts.toArray()));
+ throw new MetaException(String.join("Too many partitions for : ",
dbName, tableName, partName));
+ }
+ return parts.get(0);
+ } else {
+ return null;
+ }
+ }
+
+ public <B extends TBase<B,?>> B computeIfAbsent(String key, Callable<B>
callable) throws Exception {
+ if (metaCache.isPresent()) {
+ try {
+ return (B) metaCache.get().get(key, callable);
+ } catch (ExecutionException e) {
+ throw (Exception) e.getCause();
+ }
+ }
+ return callable.call();
+ }
+
+ Optional<Cache<String, TBase>> initializeCache(boolean tableCacheOn) {
+ if (tableCacheOn) {
+ metaCache = Optional.of(CacheBuilder.newBuilder().softValues().build());
+ }
+ return metaCache;
+ }
+
+ public void invalidateMetaCache() {
+ metaCache.ifPresent(Cache::invalidateAll);
+ }
+
+ protected List<Path> getObsoleteDirs(AcidDirectory dir, boolean
isDynPartAbort) {
+ List<Path> obsoleteDirs = dir.getObsolete();
+ /*
+ * add anything in 'dir' that only has data from aborted transactions -
no one should be
+ * trying to read anything in that dir (except getAcidState() that only
reads the name of
+ * this dir itself)
+ * So this may run ahead of {@link CompactionInfo#highestWriteId} but it's
ok (suppose there
+ * are no active txns when cleaner runs). The key is to not delete
metadata about aborted
+ * txns with write IDs > {@link CompactionInfo#highestWriteId}.
+ * See {@link TxnStore#markCleaned(CompactionInfo)}
+ */
+ obsoleteDirs.addAll(dir.getAbortedDirectories());
+ if (isDynPartAbort) {
+ // In the event of an aborted DP operation, we should only consider the
aborted directories for cleanup.
+ // Including obsolete directories for partitioned tables can result in
data loss.
+ obsoleteDirs = dir.getAbortedDirectories();
+ }
+ return obsoleteDirs;
+ }
Review Comment:
Im not sure if these methods should be common. Will the AbortedTxnHandler
use all of them?
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java:
##########
@@ -141,49 +93,37 @@ public void run() {
new
CleanerCycleUpdater(MetricsConstants.COMPACTION_CLEANER_CYCLE_DURATION,
startedAt));
}
- long minOpenTxnId = txnHandler.findMinOpenTxnIdForCleaner();
-
- checkInterrupt();
-
- List<CompactionInfo> readyToClean =
txnHandler.findReadyToClean(minOpenTxnId, retentionTime);
-
- checkInterrupt();
-
- if (!readyToClean.isEmpty()) {
- long minTxnIdSeenOpen = txnHandler.findMinTxnIdSeenOpen();
- final long cleanerWaterMark =
- minTxnIdSeenOpen < 0 ? minOpenTxnId : Math.min(minOpenTxnId,
minTxnIdSeenOpen);
-
- LOG.info("Cleaning based on min open txn id: " + cleanerWaterMark);
- List<CompletableFuture<Void>> cleanerList = new ArrayList<>();
- // For checking which compaction can be cleaned we can use the
minOpenTxnId
- // However findReadyToClean will return all records that were
compacted with old version of HMS
- // where the CQ_NEXT_TXN_ID is not set. For these compactions we
need to provide minTxnIdSeenOpen
- // to the clean method, to avoid cleaning up deltas needed for
running queries
- // when min_history_level is finally dropped, than every HMS will
commit compaction the new way
- // and minTxnIdSeenOpen can be removed and minOpenTxnId can be
used instead.
- for (CompactionInfo compactionInfo : readyToClean) {
-
- //Check for interruption before scheduling each compactionInfo
and return if necessary
+ for (Handler handler : handlers) {
+ try {
+ List<CleaningRequest> readyToClean = handler.findReadyToClean();
checkInterrupt();
- CompletableFuture<Void> asyncJob =
- CompletableFuture.runAsync(
- ThrowingRunnable.unchecked(() ->
clean(compactionInfo, cleanerWaterMark, metricsEnabled)),
- cleanerExecutor)
- .exceptionally(t -> {
- LOG.error("Error clearing {}",
compactionInfo.getFullPartitionName(), t);
- return null;
- });
- cleanerList.add(asyncJob);
+ if (!readyToClean.isEmpty()) {
+ List<CompletableFuture<Void>> cleanerList = new ArrayList<>();
+ for (CleaningRequest cr : readyToClean) {
+
+ //Check for interruption before scheduling each cleaning
request and return if necessary
+ checkInterrupt();
+
+ CompletableFuture<Void> asyncJob =
CompletableFuture.runAsync(
+ ThrowingRunnable.unchecked(new
FSRemover(handler, cr)), cleanerExecutor)
+ .exceptionally(t -> {
+ LOG.error("Error clearing: {}",
cr.getFullPartitionName(), t);
+ return null;
+ });
Review Comment:
This construct means that each request will have it's own FSRemover, but the
handler will be common. It would be better if the "scope" of the two services
would be the same, in other words: There should be a single FSRemover instance
in Cleaner, it's run method could be renamed to clean and accept a
CleaningRequest. The wrapping for the exception handling could be done outside:
`ThrowingRunnable.unchecked(() -> remover.clean(cr)), cleanerExecutor)`
##########
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/handler/TestHandler.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor.handler;
+
+import org.apache.hadoop.hive.metastore.api.ShowCompactResponseElement;
+import org.apache.hadoop.hive.metastore.api.CompactionRequest;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.ShowCompactRequest;
+import org.apache.hadoop.hive.metastore.api.ShowCompactResponse;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.txn.CompactionInfo;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.ql.txn.compactor.Cleaner;
+import org.apache.hadoop.hive.ql.txn.compactor.CleaningRequest;
+import org.apache.hadoop.hive.ql.txn.compactor.TestCleaner;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_COMPACTOR_DELAYED_CLEANUP_ENABLED;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+
+public class TestHandler extends TestCleaner {
+
+ @Test
+ public void testCompactionHandlerForSuccessfulCompaction() throws Exception {
+ Table t = newTable("default", "handler_success_table", true);
+ Partition p = newPartition(t, "today");
+ addBaseFile(t, p, 20L, 20);
+ addDeltaFile(t, p, 21L, 22L, 2);
+ addDeltaFile(t, p, 23L, 24L, 2);
+ addBaseFile(t, p, 25L, 25);
+
+ burnThroughTransactions(t.getDbName(), t.getTableName(), 25);
+
+ CompactionRequest rqst = new CompactionRequest(t.getDbName(),
t.getTableName(), CompactionType.MAJOR);
+ rqst.setPartitionname("ds=today");
+ compactInTxn(rqst);
+
+ Handler handler = new CompactionHandler(conf, txnHandler, false);
+
+ // Fetch the compaction request using the handler
+ List<CleaningRequest> cleaningRequests = handler.findReadyToClean();
+ Assert.assertEquals(1, cleaningRequests.size());
+ CleaningRequest cr = cleaningRequests.get(0);
+ Assert.assertEquals(t.getDbName(), cr.getDbName());
+ Assert.assertEquals(t.getTableName(), cr.getTableName());
+ Assert.assertEquals("ds=today", cr.getPartitionName());
+ Assert.assertEquals(CleaningRequest.RequestType.COMPACTION, cr.getType());
+
+ // Check whether appropriate handler utility methods are called exactly
once in a successful compaction scenario.
+ Handler mockedHandler = Mockito.spy(handler);
+ AtomicBoolean stop = new AtomicBoolean(true);
+ Cleaner cleaner = new Cleaner(Arrays.asList(mockedHandler));
+ cleaner.setConf(conf);
+ cleaner.init(stop);
+ cleaner.run();
+
+ Mockito.verify(mockedHandler, Mockito.times(1)).findReadyToClean();
+ Mockito.verify(mockedHandler,
Mockito.times(1)).beforeExecutingCleaningRequest(any(CleaningRequest.class));
+ Mockito.verify(mockedHandler,
Mockito.times(1)).afterExecutingCleaningRequest(any(CleaningRequest.class),
any(List.class));
Review Comment:
You could also assert that failureExecutingCleaningRequest was not invoked.
##########
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/handler/TestHandler.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.txn.compactor.handler;
+
+import org.apache.hadoop.hive.metastore.api.ShowCompactResponseElement;
+import org.apache.hadoop.hive.metastore.api.CompactionRequest;
+import org.apache.hadoop.hive.metastore.api.CompactionType;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.ShowCompactRequest;
+import org.apache.hadoop.hive.metastore.api.ShowCompactResponse;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.txn.CompactionInfo;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.ql.txn.compactor.Cleaner;
+import org.apache.hadoop.hive.ql.txn.compactor.CleaningRequest;
+import org.apache.hadoop.hive.ql.txn.compactor.TestCleaner;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_COMPACTOR_DELAYED_CLEANUP_ENABLED;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+
+public class TestHandler extends TestCleaner {
+
+ @Test
+ public void testCompactionHandlerForSuccessfulCompaction() throws Exception {
+ Table t = newTable("default", "handler_success_table", true);
+ Partition p = newPartition(t, "today");
+ addBaseFile(t, p, 20L, 20);
+ addDeltaFile(t, p, 21L, 22L, 2);
+ addDeltaFile(t, p, 23L, 24L, 2);
+ addBaseFile(t, p, 25L, 25);
+
+ burnThroughTransactions(t.getDbName(), t.getTableName(), 25);
+
+ CompactionRequest rqst = new CompactionRequest(t.getDbName(),
t.getTableName(), CompactionType.MAJOR);
+ rqst.setPartitionname("ds=today");
+ compactInTxn(rqst);
+
+ Handler handler = new CompactionHandler(conf, txnHandler, false);
+
+ // Fetch the compaction request using the handler
+ List<CleaningRequest> cleaningRequests = handler.findReadyToClean();
+ Assert.assertEquals(1, cleaningRequests.size());
+ CleaningRequest cr = cleaningRequests.get(0);
+ Assert.assertEquals(t.getDbName(), cr.getDbName());
+ Assert.assertEquals(t.getTableName(), cr.getTableName());
+ Assert.assertEquals("ds=today", cr.getPartitionName());
+ Assert.assertEquals(CleaningRequest.RequestType.COMPACTION, cr.getType());
+
+ // Check whether appropriate handler utility methods are called exactly
once in a successful compaction scenario.
+ Handler mockedHandler = Mockito.spy(handler);
+ AtomicBoolean stop = new AtomicBoolean(true);
+ Cleaner cleaner = new Cleaner(Arrays.asList(mockedHandler));
+ cleaner.setConf(conf);
+ cleaner.init(stop);
+ cleaner.run();
+
+ Mockito.verify(mockedHandler, Mockito.times(1)).findReadyToClean();
+ Mockito.verify(mockedHandler,
Mockito.times(1)).beforeExecutingCleaningRequest(any(CleaningRequest.class));
+ Mockito.verify(mockedHandler,
Mockito.times(1)).afterExecutingCleaningRequest(any(CleaningRequest.class),
any(List.class));
+ }
+
+ @Test
+ public void testCompactionHandlerForFailureCompaction() throws Exception {
+ Table t = newTable("default", "handler_failure_table", true);
+ Partition p = newPartition(t, "today");
+ addBaseFile(t, p, 20L, 20);
+ addDeltaFile(t, p, 21L, 22L, 2);
+ addDeltaFile(t, p, 23L, 24L, 2);
+ addBaseFile(t, p, 25L, 25);
+
+ burnThroughTransactions(t.getDbName(), t.getTableName(), 25);
+
+ CompactionRequest rqst = new CompactionRequest(t.getDbName(),
t.getTableName(), CompactionType.MAJOR);
+ rqst.setPartitionname("ds=today");
+ compactInTxn(rqst);
+
+ // Check whether appropriate handler utility methods are called exactly
once in a failure compaction scenario.
+ TxnStore mockedTxnHandler = Mockito.spy(txnHandler);
+ doThrow(new RuntimeException()).when(mockedTxnHandler).markCleaned(any());
+ Handler mockedHandler = Mockito.spy(new CompactionHandler(conf,
mockedTxnHandler, false));
+ AtomicBoolean stop = new AtomicBoolean(true);
+ Cleaner cleaner = new Cleaner(Arrays.asList(mockedHandler));
+ cleaner.setConf(conf);
+ cleaner.init(stop);
+ cleaner.run();
+
+ Mockito.verify(mockedHandler, Mockito.times(1)).findReadyToClean();
+ Mockito.verify(mockedHandler,
Mockito.times(1)).beforeExecutingCleaningRequest(any(CleaningRequest.class));
+ Mockito.verify(mockedHandler,
Mockito.times(1)).failureExecutingCleaningRequest(any(CleaningRequest.class),
any(Exception.class));
Review Comment:
You could also assert that afterExecutingCleaningRequest was not invoked.
Issue Time Tracking
-------------------
Worklog Id: (was: 845607)
Time Spent: 3h 10m (was: 3h)
> Split Cleaner into separate manageable modular entities
> -------------------------------------------------------
>
> Key: HIVE-27019
> URL: https://issues.apache.org/jira/browse/HIVE-27019
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sourabh Badhya
> Assignee: Sourabh Badhya
> Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> As described by the parent task -
> Cleaner can be divided into separate entities like -
> *1) Handler* - This entity fetches the data from the metastore DB from
> relevant tables and converts it into a request entity called CleaningRequest.
> It would also do SQL operations post cleanup (postprocess). Every type of
> cleaning request is provided by a separate handler.
> *2) Filesystem remover* - This entity fetches the cleaning requests from
> various handlers and deletes them according to the cleaning request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
