vikramahuja1001 commented on code in PR #4807:
URL: https://github.com/apache/hive/pull/4807#discussion_r1435010219
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -4104,49 +4104,20 @@ public List<String> getPartitionNames(Table tbl,
ExprNodeGenericFuncDesc expr, S
}
/**
- * get all the partitions that the table has
+ * get all the partitions that the table has along with auth info
*
* @param tbl
* object for which partition is needed
- * @return list of partition objects
+ * @return list of partition objects along with auth info
*/
public List<Partition> getPartitions(Table tbl) throws HiveException {
PerfLogger perfLogger = SessionState.getPerfLogger();
perfLogger.perfLogBegin(CLASS_NAME, PerfLogger.HIVE_GET_PARTITIONS);
-
try {
- if (tbl.isPartitioned()) {
- List<org.apache.hadoop.hive.metastore.api.Partition> tParts;
- try {
- GetPartitionsPsWithAuthRequest req = new
GetPartitionsPsWithAuthRequest();
- req.setTblName(tbl.getTableName());
- req.setDbName(tbl.getDbName());
- req.setUserName(getUserName());
- req.setMaxParts((short) -1);
- req.setGroupNames(getGroupNames());
- if (AcidUtils.isTransactionalTable(tbl)) {
- ValidWriteIdList validWriteIdList =
getValidWriteIdList(tbl.getDbName(), tbl.getTableName());
- req.setValidWriteIdList(validWriteIdList != null ?
validWriteIdList.toString() : null);
- req.setId(tbl.getTTable().getId());
- }
- GetPartitionsPsWithAuthResponse res =
getMSC().listPartitionsWithAuthInfoRequest(req);
- tParts = res.getPartitions();
-
- } catch (NoSuchObjectException nsoe) {
- return Lists.newArrayList();
- } catch (Exception e) {
- LOG.error("Failed getPartitions", e);
- throw new HiveException(e);
- }
- List<Partition> parts = new ArrayList<>(tParts.size());
- for (org.apache.hadoop.hive.metastore.api.Partition tpart : tParts) {
- parts.add(new Partition(tbl, tpart));
- }
-
- return parts;
- } else {
- return Collections.singletonList(new Partition(tbl));
- }
+ int batchSize= MetastoreConf.getIntVar(Hive.get().getConf(),
MetastoreConf.ConfVars.BATCH_RETRIEVE_MAX);
Review Comment:
Yes, All calls are by batching an retry(only in case if a call fails the
retry will kick in). The user can control the number of hms calls by changing
the batch size property(hive.metastore.batch.retrieve.max). This way for users
for which this call never fails will not experience any delay.
Only in case of massive tables with high number of columns and high number
of partitions this value will be set relatively low. GetPartitions on such
tables will anyways not work without this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
