Rajesh Balamohan created HIVE-24805: ---------------------------------------
Summary: Compactor: Initiator shouldn't fetch table details again and again for partitioned tables Key: HIVE-24805 URL: https://issues.apache.org/jira/browse/HIVE-24805 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan Initiator shouldn't be fetch table details for all its partitions. When there are large number of databases/tables, it takes lot of time for Initiator to complete its initial iteration and load on DB also goes higher. https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129 https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456 For all the following partitions, table details would be the same. However, it ends up fetching table details from HMS again and again. {noformat} 2021-02-22 08:13:16,106 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899 2021-02-22 08:13:16,124 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830 2021-02-22 08:13:16,140 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586 2021-02-22 08:13:16,149 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698 2021-02-22 08:13:16,158 INFO org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see if we should compact tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063 {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)