[ https://issues.apache.org/jira/browse/YARN-8385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16536677#comment-16536677 ]
Marco Gaido commented on YARN-8385: ----------------------------------- Thanks for your answer [~jlowe]. As it is stated in the question on SO (https://stackoverflow.com/questions/46893123/how-can-i-make-spark-thrift-server-clean-up-its-cache) I think the application directory is used. I see why the data is not removed by YARN from you comment above, though. So I think we have to investigate why Spark is using the application directory in this case. Thanks. > Clean local directories when a container is killed > -------------------------------------------------- > > Key: YARN-8385 > URL: https://issues.apache.org/jira/browse/YARN-8385 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.7.0 > Reporter: Marco Gaido > Priority: Major > > In long running applications, it may happen that many containers are created > and killed. A use case is Spark Thrift Server when dynamic allocation is > enabled. A lot of containers are killed and the application keeps running > indefinitely. > Currently, YARN seems to remove the local directories only when the whole > application terminates. In the scenario described above, this can cause > serious resource leakages. Please, check > https://issues.apache.org/jira/browse/SPARK-22575. > I think YARN should clean up all the local directories of a container when it > is killed and not when the whole application terminates. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org