Sushanth Sowmyan created HIVE-10381:
---------------------------------------
Summary: Allow pushing of property-key-value based predicate
filter to Metastore dropPartitions
Key: HIVE-10381
URL: https://issues.apache.org/jira/browse/HIVE-10381
Project: Hive
Issue Type: Sub-task
Reporter: Sushanth Sowmyan
After HIVE-10228, we now have a case wherein we assume, based on our knowledge
of how replication will work, that DROP_PARTITION replication will act on a
per-partition level.
In order to be robust, however, we do handle the case where this might not be
the case, wherein the DROP_PARTITION replication command generated would work
across multiple partitions, it's just that that handling is very inefficient if
ever used, since it has to fetch each of those partitions from the metastore,
decide whether or not it matches the ReplicationSpec.allowEventReplacementInto
filter, and then drop it.
Ideally, we should allow pushing this filter to the metastore, and let the
metastore do a smart drop based on that.
{code}
ReplicationSpec replicationSpec = dropTbl.getReplicationSpec();
if (replicationSpec.isInReplicationScope()){
// TODO: Current implementation of replication will result in
DROP_PARTITION under replication
// scope being called per-partition instead of multiple partitions.
However, to be robust, we
// must still handle the case of multiple partitions in case this
assumption changes in the
// future. However, if this assumption changes, we will not be very
performant if we fetch
// each partition one-by-one, and then decide on inspection whether or
not this is a candidate
// for dropping. Thus, we need a way to push this filter
(replicationSpec.allowEventReplacementInto)
// to the metastore to allow it to do drop a partition or not, depending
on a Predicate on the
// parameter key values.
for (DropTableDesc.PartSpec partSpec : dropTbl.getPartSpecs()){
try {
for (Partition p : Iterables.filter(
db.getPartitionsByFilter(tbl,
partSpec.getPartSpec().getExprString()),
replicationSpec.allowEventReplacementInto())){
db.dropPartition(tbl.getDbName(),tbl.getTableName(),p.getValues(),true);
}
} catch (NoSuchObjectException e){
// ignore NSOE because that means there's nothing to drop.
} catch (Exception e) {
throw new HiveException(e.getMessage(), e);
}
}
return;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
