[
https://issues.apache.org/jira/browse/TEZ-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774809#comment-17774809
]
Rajesh Balamohan commented on TEZ-4518:
---------------------------------------
There can be spills in reducer side in merger. IAC, I was trying to point out
that having a restriction on number of spills may not be a generic way (e.g
some apps gets launched with higher memory and their spill ratios & size will
be different than the regular ones). So having a limit of say 500 on this, can
be different for apps with different mem requirements. May be it left to
cluster admins to choose the right value for this.
> Limit number of spill files getting created
> -------------------------------------------
>
> Key: TEZ-4518
> URL: https://issues.apache.org/jira/browse/TEZ-4518
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Mudit Sharma
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hi,
>
> We have been facing some issues where many of our cluster node disks go full
> because of some rogue applications creating a lot of spill data
> We wanted to fail the app if more than a threshold amount of spill files are
> written
> Please let us know if any such capability is supported
>
> If the capability is not there, we are proposing it to support it via a
> config, we have added a PR for the same:
> https://github.com/apache/tez/pull/312, please let us know your thoughts on it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
