[
https://issues.apache.org/jira/browse/HDFS-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192856#comment-13192856
]
Harsh J commented on HDFS-329:
------------------------------
Agree, given reserved space config, which should be set to the minimum free
space you want your shared MR scratch dirs to have, this should not be an issue
-- coupled with periodic balancing.
> separate space reservation for hdfs blocks and intermediate storage
> -------------------------------------------------------------------
>
> Key: HDFS-329
> URL: https://issues.apache.org/jira/browse/HDFS-329
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Priority: Critical
>
> both dfs client buffering (and i imagine map-reduce intermediate data) and
> datanode try to honor the same space reservation (dfs.du.reserved). But this
> is problematic because once hdfs/data-node fill up a node - there's no space
> left for map-reduce computations.
> ideally - hdfs should be allowed to consume upto some watermark (say 60%) and
> then dfs buffering/intermediate storage should be allowed to consume space
> upto some higher watermark (say 90%). this way the node will always remain
> usable.
> we are hitting this problem in a cluster where a few nodes have lower amount
> of space. while the cluster overall has space left, these nodes are hitting
> their space limits. but now tasks scheduled on these nodes fail because dfs
> client does not find space to buffer to. there's no workaround really i can
> think of.
> another option would be to globally allocate hdfs blocks based on space
> availability (keep all nodes at the same space utilization % approx.).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
