[
https://issues.apache.org/jira/browse/TEZ-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931369#comment-17931369
]
Hiroyuki Nagaya edited comment on TEZ-4604 at 2/28/25 5:00 AM:
---------------------------------------------------------------
[~okumin] Thank you for your detailed research.
I will try the <{*}hive.compactor.crud.query.based=true>{*} you taught me here
as well!
Also, thank you for showing us how MRAppMaster#cleanupStagingDir works and how
Tez's MR works.
was (Author: JIRAUSER308760):
[~okumin] Thank you for your detailed research.
I will try the [{*}hive.compactor.crud.query.based=true{*}] you taught me here
as well!
Also, thank you for showing us how MRAppMaster#cleanupStagingDir works and how
Tez's MR works.
> Hive compaction in Tez does not delete files under staging directory
> --------------------------------------------------------------------
>
> Key: TEZ-4604
> URL: https://issues.apache.org/jira/browse/TEZ-4604
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Hiroyuki Nagaya
> Priority: Critical
> Attachments: createTable.sql.txt, hive-changed.xml,
> hive-default.xml.template, mapred-site.xml, tez-changed.xml,
> tez-default-template.xml
>
>
> I am using a combination of Hadoop, Hive and Tez.
> When I run major compaction with Hive, files under the staging directory are
> not deleted.
> With Mapreduce, files are deleted from the staging directory and files are
> created in the history directory.
> Hadoop 3.3.6
> Hive 4.0.1
> Tez 0.10.4
> *1. When using Mapreduce*
> The following data will be deleted.
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.jar
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.split
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.splitmetainfo
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1705466455536_3620/job.xml
> Historical data will be created in the following directories
> /tmp/hadoop-yarn/staging/history/done
> *2. When using Tez*
> The following data will not be deleted
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/.tez
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.jar
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.split
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.splitmetainfo
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002/job.xml
> No historical data will be created.
> *Is it a bug that the following directories are not deleted?*
> *Or is it a Tez configuration problem?*
> *I would like it to be deleted because the process has been completed
> successfully and it is about 80MB in size.*
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1740026697751_0002
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
