[
https://issues.apache.org/jira/browse/HDFS-16016?focusedWorklogId=608351&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-608351
]
ASF GitHub Bot logged work on HDFS-16016:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Jun/21 08:43
Start Date: 08/Jun/21 08:43
Worklog Time Spent: 10m
Work Description: smengcl commented on a change in pull request #2998:
URL: https://github.com/apache/hadoop/pull/2998#discussion_r647197192
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
##########
@@ -1104,6 +1122,34 @@ private void sendLifeline() throws IOException {
}
}
+ class IBRTaskHandler implements Runnable {
+
+ @Override
+ public void run() {
+ LOG.info("Starting IBR Task Handler.");
+ while (shouldRun()) {
+ try {
+ final long startTime = scheduler.monotonicNow();
+ final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
+ if (!dn.areIBRDisabledForTests() &&
+ (ibrManager.sendImmediately() || sendHeartbeat)) {
+ synchronized (sendIBRLock) {
+ ibrManager.sendIBRs(bpNamenode, bpRegistration,
+ bpos.getBlockPoolId(), getRpcMetricSuffix());
+ }
+ }
+ // There is no work to do; sleep until hearbeat timer elapses,
Review comment:
```suggestion
// There is no work to do; sleep until heartbeat timer elapses,
```
Same typo at line 751 where this line is copied from. Let's fix that as well.
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeReport.java
##########
@@ -172,8 +172,20 @@ public void testDatanodeReportMissingBlock() throws
Exception {
// all bad datanodes
}
cluster.triggerHeartbeats(); // IBR delete ack
- lb = fs.getClient().getLocatedBlocks(p.toString(), 0).get(0);
- assertEquals(0, lb.getLocations().length);
+ int c = 0;
+ while (true) {
+ lb = fs.getClient().getLocatedBlocks(p.toString(), 0).get(0);
+ if (0 != lb.getLocations().length) {
+ c++;
+ if (c > 7) {
+ assertEquals(0, lb.getLocations().length);
Review comment:
The `assertEquals` condition is contradictory to the condition `(0 !=
lb.getLocations().length)`.
If the intent is to fail the test here, just use
`Assert.fail("getLocatedBlocks failed after 7 retries")` instead.
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
##########
@@ -1104,6 +1122,34 @@ private void sendLifeline() throws IOException {
}
}
+ class IBRTaskHandler implements Runnable {
+
+ @Override
+ public void run() {
+ LOG.info("Starting IBR Task Handler.");
+ while (shouldRun()) {
+ try {
+ final long startTime = scheduler.monotonicNow();
+ final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
+ if (!dn.areIBRDisabledForTests() &&
+ (ibrManager.sendImmediately() || sendHeartbeat)) {
+ synchronized (sendIBRLock) {
+ ibrManager.sendIBRs(bpNamenode, bpRegistration,
+ bpos.getBlockPoolId(), getRpcMetricSuffix());
+ }
+ }
+ // There is no work to do; sleep until hearbeat timer elapses,
+ // or work arrives, and then iterate again.
+ ibrManager.waitTillNextIBR(scheduler.getHeartbeatWaitTime());
Review comment:
This implies this IBR timer will expire around the same time as the FBR
one (`offerService()`). This should be fine.
With IBR separated in a new thread, maybe later we could have a new config
key that controls IBR interval separately, or add a configurable constant
offset (from the FBR timer) to the IBR timer. This isn't something we need to
add to this jira. Just a thought.
Just in case I miss anything, @sodonnel would you like to take a quick look
at this?
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java
##########
@@ -167,13 +168,24 @@ public void testReportBlockDeleted() throws
InterruptedException, IOException {
// Trigger a heartbeat, this also triggers an IBR.
DataNodeTestUtils.triggerHeartbeat(singletonDn);
- Thread.sleep(2000);
// Ensure that the deleted block is reported.
- Mockito.verify(nnSpy, times(1)).blockReceivedAndDeleted(
- any(DatanodeRegistration.class),
- anyString(),
- any(StorageReceivedDeletedBlocks[].class));
+ int c = 0;
Review comment:
Same here
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeReport.java
##########
@@ -172,8 +172,20 @@ public void testDatanodeReportMissingBlock() throws
Exception {
// all bad datanodes
}
cluster.triggerHeartbeats(); // IBR delete ack
- lb = fs.getClient().getLocatedBlocks(p.toString(), 0).get(0);
- assertEquals(0, lb.getLocations().length);
+ int c = 0;
Review comment:
nit: Try to use meaningful variable names, e.g.
`countRetries`/`count`/`counter` instead of just `c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 608351)
Time Spent: 3h 50m (was: 3h 40m)
> BPServiceActor add a new thread to handle IBR
> ---------------------------------------------
>
> Key: HDFS-16016
> URL: https://issues.apache.org/jira/browse/HDFS-16016
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: Viraj Jasani
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Now BPServiceActor#offerService() is doing many things, FBR, IBR, heartbeat.
> We can handle IBR independently to improve the performance of heartbeat and
> FBR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
