[ 
https://issues.apache.org/jira/browse/HIVE-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655335#action_12655335
 ] 

Joydeep Sen Sarma commented on HIVE-142:
----------------------------------------

i was thinking that inferring of the URI from the partition spec is best done 
in the server (rather than in the Java client side code as today). Currently 
the thrift interface does not return the URI. i am assuming that you would have 
to do something like that for hive-91 anyway.

If the URI is returned by the metastore - we could do the check outside the 
server. it's probably better that the check be done outside (since it could 
potentially be a heavyweight operation and might slow down the metastore). also 
- typically - checks like these are admin commands and not really that useful 
exposed through a thrift like api. so a simple java program to do the check 
using metastore + hdfs calls would be nice. the checks suggested sound like a 
good start.

are u planning to do this as a separate utility (might be easier) or do you 
want to integrate this into alter table family of commands?

> Create a metastore check command
> --------------------------------
>
>                 Key: HIVE-142
>                 URL: https://issues.apache.org/jira/browse/HIVE-142
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Metastore
>    Affects Versions: 0.19.0
>            Reporter: Johan Oskarsson
>             Fix For: 0.19.0
>
>
> We need a command to verify that the information in the metastore reflects 
> the data that is on hdfs. For example partitions can be deleted on hdfs but 
> still be in the metastore.
> From Joydeep Sen Sarma, see ticket HIVE-126 for the full comment:
> for a command line interface - one might want to check the entire database or 
> just a table or even just one partition. other metadata checks will also be 
> added over time (for example - do the file types on disk agree with metadata 
> records, bucketing information etc). So, here's a strawman proposal for a new 
> command:
> alter table <DB>[.TABLE [PARTITION-SPEC]] check [TYPE-LIST]
> where TYPE by default is 'all' (check for all kinds of errors), but can be 
> specified to a specific type. For example - in this case - we can have a type 
> called 'partitions' (and then over time we can add other types like 
> 'fileformat' etc.). for v1 - we can just drop the type-list altogether.
> the check command can produce a list of things that need to be done to fix 
> the format (like adding any directories not in the metastore - but in hdfs - 
> to the metastore). actually performing of such steps would require a user 
> confirmation (y/n).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to