[
https://issues.apache.org/jira/browse/TEZ-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931108#comment-17931108
]
Shohei Okumiya commented on TEZ-4604:
-------------------------------------
I'm still checking why MR on Tez doesn't clean up the directory.
[~gaya]
Alternatively,
[hive.compactor.crud.query.based=true|https://github.com/apache/hive/blob/rel/release-4.0.1/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L3412-L3414]
can be your solution. It will perform a compaction as a Hive on Tez query. It
successfully completed minor and major compactions in my local environment.
https://issues.apache.org/jira/browse/HIVE-20699
Disclaimer: I've not seriously used Hive ACID. Please get in touch with Hive's
user ML when you have a question about the feature.
[https://hive.apache.org/community/mailinglists/]
> Hive compaction in Tez does not delete files under staging directory
> --------------------------------------------------------------------
>
> Key: TEZ-4604
> URL: https://issues.apache.org/jira/browse/TEZ-4604
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Hiroyuki Nagaya
> Priority: Critical
> Attachments: createTable.sql.txt, hive-changed.xml,
> hive-default.xml.template, mapred-site.xml, tez-changed.xml,
> tez-default-template.xml
>
>
> I am using a combination of Hadoop, Hive and Tez.
> When I run major compaction with Hive, files under the staging directory are
> not deleted.
> With Mapreduce, files are deleted from the staging directory and files are
> created in the history directory.
> Hadoop 3.3.6
> Hive 4.0.1
> Tez 0.10.4
> *1. When using Mapreduce*
> The following data will be deleted.
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.jar
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.split
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.splitmetainfo
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.xml
> Historical data will be created in the following directories
> /tmp/hadoop-yarn/staging/history/done
> *2. When using Tez*
> The following data will not be deleted
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/.tez
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.jar
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.split
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.splitmetainfo
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.xml
> No historical data will be created.
> *Is it a bug that the following directories are not deleted?*
> *Or is it a Tez configuration problem?*
> *I would like it to be deleted because the process has been completed
> successfully and it is about 80MB in size.*
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
