[
https://issues.apache.org/jira/browse/HIVE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159396#comment-16159396
]
Alan Gates commented on HIVE-17466:
-----------------------------------
I'm certainly ok with the intention.
At some point I'd like to rationalize the thrift API, as it has many many
calls. But we should do that holistically rather than start blocking
individual changes.
> Metastore API to list unique partition-key-value combinations
> -------------------------------------------------------------
>
> Key: HIVE-17466
> URL: https://issues.apache.org/jira/browse/HIVE-17466
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Affects Versions: 2.2.0, 3.0.0
> Reporter: Mithun Radhakrishnan
> Assignee: Thiruvel Thirumoolan
> Attachments: HIVE-17466.1.patch, HIVE-17466.2-branch-2.patch,
> HIVE-17466.2.patch
>
>
> Raising this on behalf of [~thiruvel], who wrote this initially as part of a
> tangential "data-discovery" system.
> Programs like Apache Oozie, Apache Falcon (or Yahoo GDM), etc. launch
> workflows based on the availability of table/partitions. Partitions are
> currently discovered by listing partitions using (what boils down to)
> {{HiveMetaStoreClient.listPartitions()}}. This can be slow and cumbersome,
> given that {{Partition}} objects are heavyweight and carry redundant
> information. The alternative is to use partition-names, which will need
> client-side parsing to extract part-key values.
> When checking which hourly partitions for a particular day have been
> published already, it would be preferable to have an API that pushed down
> part-key extraction into the {{RawStore}} layer, and returned key-values as
> the result. This would be similar to how {{SELECT DISTINCT part_key FROM
> my_table;}} would run, but at the {{HiveMetaStoreClient}} level.
> Here's what we've been using at Yahoo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
