Rohini Palaniswamy created TEZ-3877:
---------------------------------------
Summary: Delete spill files once merge is done
Key: TEZ-3877
URL: https://issues.apache.org/jira/browse/TEZ-3877
Project: Apache Tez
Issue Type: Bug
Reporter: Rohini Palaniswamy
I see that spill files are not deleted right after merge completes. We should
do that as it takes up a lot of space and we can't afford that wastage when Tez
takes up a lot of shuffle space with complex DAGs. [~jlowe] told me they are
only cleaned up after application completes as they are written in app
directory and not container directory. That also has to be done so that they
are cleaned up by node manager during task failures or container crashes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
