[
https://issues.apache.org/jira/browse/HDFS-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147176#comment-14147176
]
Arpit Agarwal commented on HDFS-6988:
-------------------------------------
bq. Why would I want the number of bytes that I keep free at all times to
double when my ramdisk size goes from 4g to 8g, for example?
The memory is certainly not kept free at all times. We also cap the number of
evictions at a time. From my system testing even with a low to moderate write
load, RAM disk usage is usually at 100%.
During low or no load the eviction will gradually push out replicas to get to
10% free space (now capped at a maximum of 10 replicas). Write workloads are by
nature spiky and we need to keep some margin of free space to absorb sudden
load.
bq. I don't understand why we can't simply have this be a single number,
I explained this earlier, a single number fails to work well for a range of
disks and makes configuration mandatory. What would you choose as the default
value of this single setting. Let's say we choose 1GB or higher. Then we are
wasting at least 25% of space on a 4GB RAM disk. Or we choose 512MB. Then we
are not evicting fast enough to keep up with multiple writers on a 50GB disk.
I dislike adding more mandatory configuration given the complexity of
configuring Hadoop. You are underestimating the administrator/developer burden
when we ship with poor defaults and a single value makes for a poor default.
Having multiple control parameters makes it unlikely the administrator will
have to change any of them. Honestly my preference is not to expose any of
these settings. As a compromise I am fine with exposing the settings and you
expressed agreement with the configurable upper limit above.
> Add configurable limit for percentage-based eviction threshold
> --------------------------------------------------------------
>
> Key: HDFS-6988
> URL: https://issues.apache.org/jira/browse/HDFS-6988
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
> Affects Versions: HDFS-6581
> Reporter: Arpit Agarwal
> Assignee: Arpit Agarwal
> Fix For: HDFS-6581
>
> Attachments: HDFS-6988.01.patch, HDFS-6988.02.patch
>
>
> Per feedback from [~cmccabe] on HDFS-6930, we can make the eviction
> thresholds configurable. The hard-coded thresholds may not be appropriate for
> very large RAM disks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
