[
https://issues.apache.org/jira/browse/HDDS-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964348#comment-16964348
]
Anu Engineer commented on HDDS-2384:
------------------------------------
Thank you for flagging this issue. I think this is a hard to solve problem in
the current architecture. I would like to explore some possibilitues on how we
can solve this issue.
1. We add the support for a buffer pool inside data node. A buffer pool would a
large chunk of memory that data node will pin and internally treat it as a set
of buffers that can be used for I/O. When we read or write data, we will also
use this buffer pool. That way, we can limit the maximum committed memory that
we will end up using for data path.
2. In order to do that, we will now need the ability to read data not in 16 MB
chunks, but perhaps in smaller 8KB kind of size(assuming the page size is going
to be 8KB in the buffer pool).
3. The advantage of such an approach is that we will read data only as much
memory we have, but the network layer still might have to buffer this data.
4. This also allows us to push back against a client that is sending or trying
to read too much data from the data node at any given time.
Question: Do you think such a change would address this issue ? If you have
other suggestions, I would love to hear them. Once more, thank you for flagging
this issue.
> Large chunks during write can have memory pressure on DN with multiple clients
> ------------------------------------------------------------------------------
>
> Key: HDDS-2384
> URL: https://issues.apache.org/jira/browse/HDDS-2384
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Anu Engineer
> Priority: Major
> Labels: performance
>
> During large file writes, it ends up writing {{16 MB}} chunks.
> https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java#L691
> In large clusters, 100s of clients may connect to DN. In such cases,
> depending on the incoming write workload mem load on DN can increase
> significantly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
