vikramahuja1001 commented on code in PR #4501:
URL: https://github.com/apache/hive/pull/4501#discussion_r1273057101
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -4139,27 +4141,42 @@ public List<Partition> getPartitions(Table tbl) throws
HiveException {
}
/**
- * Get all the partitions; unlike {@link #getPartitions(Table)}, does not
include auth.
+ * Get all the partitions in batches; unlike {@link #getPartitions(Table)},
does not include auth.
* @param tbl table for which partitions are needed
* @return list of partition objects
*/
public Set<Partition> getAllPartitionsOf(Table tbl) throws HiveException {
+ int batchSize= MetastoreConf.getIntVar(
+ Hive.get().getConf(), MetastoreConf.ConfVars.BATCH_RETRIEVE_MAX);
+ int maxRetries = HiveConf.getIntVar(conf,
HiveConf.ConfVars.HIVE_GETPARTITIONS_MAX_RETRIES);
+ return getAllPartitionsOf(tbl, batchSize, 2, maxRetries);
+ }
+
+ public Set<Partition> getAllPartitionsOf(Table tbl, int batchSize, int
decayingFactor,
+ int maxRetries) throws HiveException {
if (!tbl.isPartitioned()) {
return Sets.newHashSet(new Partition(tbl));
}
-
- List<org.apache.hadoop.hive.metastore.api.Partition> tParts;
+ Set<Partition> result = new LinkedHashSet<>();
+ RetryUtilities.ExponentiallyDecayingBatchWork batchTask = new
RetryUtilities
+ .ExponentiallyDecayingBatchWork<Void>(batchSize, decayingFactor,
maxRetries) {
+ @Override
+ public Void execute(int size) throws HiveException {
+ try {
+ result.clear();
+ new PartitionIterable(Hive.get(), tbl, null,
size).forEach(result::add);
Review Comment:
We won't get partial or duplicate partitions due to some exception. In case
of any exception, retry will happen by reducing the batch size. It will keep on
retrying unless batchSize becomes 0 or maxRetry limit is exhausted. I have
added test cases for all these 3 scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
