Rajesh Balamohan created HIVE-23521:
---------------------------------------
Summary: REPL: Optimise partition loading during bootstrap
Key: HIVE-23521
URL: https://issues.apache.org/jira/browse/HIVE-23521
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
When bootstrapping with large "REPL dump" with ~10K partitions, it starts
executing "addPartition" in sequential manner and takes very long time as it
communicates with HMS/registers partition etc for every call.
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L399]
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L165]
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java#L210]
When bootstrap loading has to deal with DDL, it would be good to collate all
partitions in single call to HMS. This would help in reducing overall runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
