[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993857#comment-16993857 ] Anant Mittal commented on HIVE-18922: - Thanks David!! It is possible that these are the cause, I will confirm once I can get this tried on even newer Hive versions. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. > Update: It seems HIVE-11940 did not cover this as the issue is seen in later > versions too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16956357#comment-16956357 ] Anant Mittal commented on HIVE-18922: - Issue still seen on newer Hive versions. EMR5.26 and mapr6.1 both show the same issue. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. > Update: It seems HIVE-11940 did not cover this as the issue is seen in later > versions too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anant Mittal updated HIVE-18922: Description: Hive is creating hdfs folders with format /.hive-staging_hive__-xx/-ext-x These are not being cleaned up even after long duration. The folder is used to load to the table. Example: Loading data to table default.tablename from hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 This might be covered to some extent by HIVE-11940 but, want to make sure all cases are addressed. Update: It seems HIVE-11940 did not cover this as the issue is seen in later versions too. was: Hive is creating hdfs folders with format /.hive-staging_hive__-xx/-ext-x These are not being cleaned up even after long duration. The folder is used to load to the table. Example: Loading data to table default.tablename from hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 This might be covered to some extent by HIVE-11940 but, want to make sure all cases are addressed. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. > Update: It seems HIVE-11940 did not cover this as the issue is seen in later > versions too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[
https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397425#comment-16397425
]
Anant Mittal commented on HIVE-18922:
-
It definitely happens with {{MERGE INTO}} queries. I tried some simple non
merge queries and did not see the behavior. You are right, it might be specific
to ACID but I cannot say for sure.
> Hive is not cleaning up staging directories
>
>
> Key: HIVE-18922
> URL: https://issues.apache.org/jira/browse/HIVE-18922
> Project: Hive
> Issue Type: Bug
>Reporter: Anant Mittal
>Priority: Major
>
> Hive is creating hdfs folders with format
> /.hive-staging_hive__-xx/-ext-x
> These are not being cleaned up even after long duration. The folder is used
> to load to the table. Example:
> Loading data to table default.tablename from
> hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1
>
> This might be covered to some extent by HIVE-11940 but, want to make sure all
> cases are addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395755#comment-16395755 ] Anant Mittal commented on HIVE-18922: - [~stakiar] This was noticed in my setup every time a MERGE INTO query is used. I am not aware whether other cases would cause the same to happen in some specific scenario. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anant Mittal reassigned HIVE-18922: --- Assignee: Sergio Peña > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Assignee: Sergio Peña >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
