[
https://issues.apache.org/jira/browse/TEZ-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041870#comment-17041870
]
László Bodor commented on TEZ-3350:
-----------------------------------
it still applies, in recent logs:
{code}
2020-02-19 23:56:53,908 [INFO] [TezChild] |impl.PipelinedSorter|: Reducer 2:
Spilling to
/grid/05/hadoop/yarn/local/usercache/hive/appcache/application_1581931657676_3139/output/attempt_1581931657676_3139_1_00_000008_0_10002_0/file.out
{code}
https://github.com/apache/tez/blob/cf111b1c896db94d40ea92fc6da2a6fcce356114/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/task/local/output/TezTaskOutputFiles.java#L78-L95
> Shuffle spills are not spilled to a container-specific directory
> ----------------------------------------------------------------
>
> Key: TEZ-3350
> URL: https://issues.apache.org/jira/browse/TEZ-3350
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Jason Darrell Lowe
> Assignee: László Bodor
> Priority: Major
>
> If a Tez task receives too much input data and needs to spill the inputs to
> disk it ends up using a path that is not container-specific. Therefore YARN
> will not automatically cleanup these files when the container exits as it
> should, and instead the files linger until the entire application completes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
