[ https://issues.apache.org/jira/browse/TEZ-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171964#comment-14171964 ]
Gopal V commented on TEZ-1277: ------------------------------ Need to add a NativeIO impl to do {{fallocate}}. > Tez Spill handler should truncate files to reserve space on disk > ---------------------------------------------------------------- > > Key: TEZ-1277 > URL: https://issues.apache.org/jira/browse/TEZ-1277 > Project: Apache Tez > Issue Type: Improvement > Affects Versions: 0.5.0 > Reporter: Gopal V > Assignee: Gopal V > > Occasionally tasks fail due to full disks because the disks had space when > the task was allocating via LocalDirAllocator, but the disk space was > actually promised to many tasks instead of just one. > This race condition shows up when a 1Gb spill can be done in ~10s or so. > There is no way to do this via the hadoop-fs abstraction - but an SSD based > spill wastes most of the IOPS on journal updates about the file length > changing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)