[jira] [Commented] (HIVE-13703) "msck repair" on table with non-partition subdirectories reporting partitions not in metastore

2016-09-21 Thread Alina Abramova (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509475#comment-15509475
 ] 

Alina Abramova commented on HIVE-13703:
---

Oh yeah, now I see that HIVE-14511 really fixes this issue. Thank you
This bug could be regarded as resolved

> "msck repair" on table with non-partition subdirectories reporting partitions 
> not in metastore
> --
>
> Key: HIVE-13703
> URL: https://issues.apache.org/jira/browse/HIVE-13703
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.2.1
>Reporter: Ana Gillan
>Assignee: Alina Abramova
> Attachments: HIVE-13703.patch
>
>
> PROBLEM: Subdirectories created with UNION ALL are listed in {{show 
> partitions}} output, but show up as {{Partitions not in metastore}} in {{msck 
> repair}} output. 
> STEPS TO REPRODUCE: Table created from {{CTAS ... UNION ALL}} DDL
> {code}
> hive> msck repair table meter_001; 
> OK 
> Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1 
> meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1 
> meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1 
> meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1 
> meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1 
> meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1 
> meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1 
> meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1 
> meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1 
> meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1 
> meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1 
> meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1 
> meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1 
> meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1 
> meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1 
> meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1 
> meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1 
> meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1 
> meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1 
> meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1 
> meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1 
> meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1 
> meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1 
> meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1 
> meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1 
> meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1 
> meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1 
> meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1 
> meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1 
> meter_001:tech_datestamp=2016-04-06/2 
> Time taken: 15.996 seconds, Fetched: 1 row(s) 
> {code}
> {code}
> hive> show partitions meter_001; 
> OK 
> tech_datestamp=2016-03-09 
> tech_datestamp=2016-03-10 
> tech_datestamp=2016-03-11 
> tech_datestamp=2016-03-12 
> tech_datestamp=2016-03-13 
> tech_datestamp=2016-03-14 
> tech_datestamp=2016-03-15 
> tech_datestamp=2016-03-16 
> tech_datestamp=2016-03-17 
> tech_datestamp=2016-03-18 
> tech_datestamp=2016-03-19 
> tech_datestamp=2016-03-20 
> tech_datestamp=2016-03-21 
> tech_datestamp=2016-03-22 
> tech_datestamp=2016-03-23 
> tech_datestamp=2016-03-24 
> tech_datestamp=2016-03-25 
> tech_datestamp=2016-03-26 
> tech_datestamp=2016-03-27 
> tech_datestamp=2016-03-28 
> tech_datestamp=2016-03-29 
> tech_datestamp=2016-03-30 
> tech_datestamp=2016-03-31 
> tech_datestamp=2016-04-01 
> tech_datestamp=2016-04-02 
> tech_datestamp=2016-04-03 
> tech_datestamp=2016-04-04 
> tech_datestamp=2016-04-05 
> tech_datestamp=2016-04-06 
> Time taken: 0.417 seconds, Fetched: 29 row(s) 
> {code}
> Ideally msck repair should ignore subdirectory if that additional partition 
> column doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13703) "msck repair" on table with non-partition subdirectories reporting partitions not in metastore

2016-09-19 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504433#comment-15504433
 ] 

Sergey Shelukhin commented on HIVE-13703:
-

Would this be fixed by HIVE-14511, or otherwise should it use a similar 
approach (looking for the expected directory structure in the first place, 
rather than catching errors)?

> "msck repair" on table with non-partition subdirectories reporting partitions 
> not in metastore
> --
>
> Key: HIVE-13703
> URL: https://issues.apache.org/jira/browse/HIVE-13703
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 0.14.0, 1.0.0, 1.2.1
>Reporter: Ana Gillan
>Assignee: Alina Abramova
> Attachments: HIVE-13703.patch
>
>
> PROBLEM: Subdirectories created with UNION ALL are listed in {{show 
> partitions}} output, but show up as {{Partitions not in metastore}} in {{msck 
> repair}} output. 
> STEPS TO REPRODUCE: Table created from {{CTAS ... UNION ALL}} DDL
> {code}
> hive> msck repair table meter_001; 
> OK 
> Partitions not in metastore: meter_001:tech_datestamp=2016-03-09/1 
> meter_001:tech_datestamp=2016-03-09/2 meter_001:tech_datestamp=2016-03-10/1 
> meter_001:tech_datestamp=2016-03-10/2 meter_001:tech_datestamp=2016-03-11/1 
> meter_001:tech_datestamp=2016-03-11/2 meter_001:tech_datestamp=2016-03-12/1 
> meter_001:tech_datestamp=2016-03-12/2 meter_001:tech_datestamp=2016-03-13/1 
> meter_001:tech_datestamp=2016-03-13/2 meter_001:tech_datestamp=2016-03-14/1 
> meter_001:tech_datestamp=2016-03-14/2 meter_001:tech_datestamp=2016-03-15/1 
> meter_001:tech_datestamp=2016-03-15/2 meter_001:tech_datestamp=2016-03-16/1 
> meter_001:tech_datestamp=2016-03-16/2 meter_001:tech_datestamp=2016-03-17/1 
> meter_001:tech_datestamp=2016-03-17/2 meter_001:tech_datestamp=2016-03-18/1 
> meter_001:tech_datestamp=2016-03-18/2 meter_001:tech_datestamp=2016-03-19/1 
> meter_001:tech_datestamp=2016-03-19/2 meter_001:tech_datestamp=2016-03-20/1 
> meter_001:tech_datestamp=2016-03-20/2 meter_001:tech_datestamp=2016-03-21/1 
> meter_001:tech_datestamp=2016-03-21/2 meter_001:tech_datestamp=2016-03-22/1 
> meter_001:tech_datestamp=2016-03-22/2 meter_001:tech_datestamp=2016-03-23/1 
> meter_001:tech_datestamp=2016-03-23/2 meter_001:tech_datestamp=2016-03-24/1 
> meter_001:tech_datestamp=2016-03-24/2 meter_001:tech_datestamp=2016-03-25/1 
> meter_001:tech_datestamp=2016-03-25/2 meter_001:tech_datestamp=2016-03-26/1 
> meter_001:tech_datestamp=2016-03-26/2 meter_001:tech_datestamp=2016-03-27/1 
> meter_001:tech_datestamp=2016-03-27/2 meter_001:tech_datestamp=2016-03-28/1 
> meter_001:tech_datestamp=2016-03-28/2 meter_001:tech_datestamp=2016-03-29/1 
> meter_001:tech_datestamp=2016-03-29/2 meter_001:tech_datestamp=2016-03-30/1 
> meter_001:tech_datestamp=2016-03-30/2 meter_001:tech_datestamp=2016-03-31/1 
> meter_001:tech_datestamp=2016-03-31/2 meter_001:tech_datestamp=2016-04-01/1 
> meter_001:tech_datestamp=2016-04-01/2 meter_001:tech_datestamp=2016-04-02/1 
> meter_001:tech_datestamp=2016-04-02/2 meter_001:tech_datestamp=2016-04-03/1 
> meter_001:tech_datestamp=2016-04-03/2 meter_001:tech_datestamp=2016-04-04/1 
> meter_001:tech_datestamp=2016-04-04/2 meter_001:tech_datestamp=2016-04-05/1 
> meter_001:tech_datestamp=2016-04-05/2 meter_001:tech_datestamp=2016-04-06/1 
> meter_001:tech_datestamp=2016-04-06/2 
> Time taken: 15.996 seconds, Fetched: 1 row(s) 
> {code}
> {code}
> hive> show partitions meter_001; 
> OK 
> tech_datestamp=2016-03-09 
> tech_datestamp=2016-03-10 
> tech_datestamp=2016-03-11 
> tech_datestamp=2016-03-12 
> tech_datestamp=2016-03-13 
> tech_datestamp=2016-03-14 
> tech_datestamp=2016-03-15 
> tech_datestamp=2016-03-16 
> tech_datestamp=2016-03-17 
> tech_datestamp=2016-03-18 
> tech_datestamp=2016-03-19 
> tech_datestamp=2016-03-20 
> tech_datestamp=2016-03-21 
> tech_datestamp=2016-03-22 
> tech_datestamp=2016-03-23 
> tech_datestamp=2016-03-24 
> tech_datestamp=2016-03-25 
> tech_datestamp=2016-03-26 
> tech_datestamp=2016-03-27 
> tech_datestamp=2016-03-28 
> tech_datestamp=2016-03-29 
> tech_datestamp=2016-03-30 
> tech_datestamp=2016-03-31 
> tech_datestamp=2016-04-01 
> tech_datestamp=2016-04-02 
> tech_datestamp=2016-04-03 
> tech_datestamp=2016-04-04 
> tech_datestamp=2016-04-05 
> tech_datestamp=2016-04-06 
> Time taken: 0.417 seconds, Fetched: 29 row(s) 
> {code}
> Ideally msck repair should ignore subdirectory if that additional partition 
> column doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)