[
https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=749102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749102
]
ASF GitHub Bot logged work on HDFS-16519:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Mar/22 03:43
Start Date: 29/Mar/22 03:43
Worklog Time Spent: 10m
Work Description: cndaimin commented on pull request #4101:
URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1081368937
@tomscut Thanks for your review. Yes, background reconstruction is
necessary. With missing block(s) of block group, client has to read `dataUnits`
blocks in block group to decode the target data. This causes `dataUnits` times
of network traffic and corresponding cpu costs.
In our product cluster, we encountered backgroud reconstruction exhausts the
network traffic of DN node. This leads to the failure of client read/write. The
speed of read/write and reconstruction is in conflict, we can not have both of
them actually. And in practice, we set throttler of EC reconstruction to half
of available network traffic, for example, 500 MB/s to 1GB/s NIC. In this PR,
we set the default value to 0, which indicates there is no limit to
reconstruction at all, to be sure there is no side effect to upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 749102)
Time Spent: 0.5h (was: 20m)
> Add throttler to EC reconstruction
> ----------------------------------
>
> Key: HDFS-16519
> URL: https://issues.apache.org/jira/browse/HDFS-16519
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, ec
> Affects Versions: 3.3.1, 3.3.2
> Reporter: daimin
> Assignee: daimin
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HDFS already have throttlers for data transfer(replication) and balancer, the
> throttlers reduce the impact of these background procedures to user
> read/write.
> We should add a throttler to EC background reconstruction too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
