[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687154&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687154
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 07:07
Start Date: 29/Nov/21 07:07
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#discussion_r758084008
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
##
@@ -84,7 +84,7 @@ public void execute(CommandLine cmd) throws Exception {
System.out.printf("%s", workStatus.currentStateString());
}
} catch (DiskBalancerException ex) {
- LOG.error("Query plan failed. ex: {}", ex);
+ LOG.error("Query plan failed. ex: {}", ex.getMessage());
Review comment:
In general, it would depend on how useful the message is. I believe if
we are re-throwing Exceptions (like this case), and we are also logging the
error message at ERROR level, it's good to print the entire stracktrace for
better debugging. Sounds good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687154)
Time Spent: 50m (was: 40m)
> Fix log format for QueryCommand
> ---
>
> Key: HDFS-16361
> URL: https://issues.apache.org/jira/browse/HDFS-16361
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Fix log format for QueryCommand of disk balancer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16332) Expired block token causes slow read due to missing handling in sasl handshake
[
https://issues.apache.org/jira/browse/HDFS-16332?focusedWorklogId=687150&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687150
]
ASF GitHub Bot logged work on HDFS-16332:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 06:53
Start Date: 29/Nov/21 06:53
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3677:
URL: https://github.com/apache/hadoop/pull/3677#discussion_r758077671
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferClient.java
##
@@ -603,7 +603,17 @@ private IOStreamPair doSaslHandshake(InetAddress addr,
conf, cipherOption, underlyingOut, underlyingIn, false) :
sasl.createStreamPair(out, in);
} catch (IOException ioe) {
- sendGenericSaslErrorMessage(out, ioe.getMessage());
+ try {
Review comment:
Given this fix is commented in detail and creating the unit test seems
costly (mock can do this, but the maintenance cost seems too much), I think we
don't need a unit test to cover this case. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687150)
Time Spent: 3.5h (was: 3h 20m)
> Expired block token causes slow read due to missing handling in sasl handshake
> --
>
> Key: HDFS-16332
> URL: https://issues.apache.org/jira/browse/HDFS-16332
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, dfs, dfsclient
>Affects Versions: 2.8.5, 3.3.1
>Reporter: Shinya Yoshida
>Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot from 2021-11-18 12-11-34.png, Screenshot from
> 2021-11-18 12-14-29.png, Screenshot from 2021-11-18 13-31-35.png
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> We're operating the HBase 1.4.x cluster on Hadoop 2.8.5.
> We're recently evaluating Kerberos secured HBase and Hadoop cluster with
> production load and we observed HBase's response slows >= several seconds,
> and about several minutes for worst-case (about once~three times a month).
> The following image is a scatter plot of HBase's response slow, each circle
> is each base's slow response log.
> The X-axis is the date time of the log occurred, the Y-axis is the response
> slow time.
> !Screenshot from 2021-11-18 12-14-29.png!
> We could reproduce this issue by reducing "dfs.block.access.token.lifetime"
> and we could figure out the cause.
> (We used dfs.block.access.token.lifetime=60, i.e. 1 hour)
> When hedged read enabled:
> !Screenshot from 2021-11-18 12-11-34.png!
> When hedged read disabled:
> !Screenshot from 2021-11-18 13-31-35.png!
> As you can see, it's worst if the hedged read is enabled. However, it happens
> whether the hedged read is enabled or not.
> This impacts our 99%tile response time.
> This happens when the block token is expired and the root cause is the wrong
> handling of the InvalidToken exception in sasl handshake in
> SaslDataTransferServer.
> I propose to add a new response code for DataTransferEncryptorStatus to
> request the client to update the block token like DataTransferProtos does.
> The test code and patch is available in
> https://github.com/apache/hadoop/pull/3677
> We could reproduce this issue by the following test code in 2.8.5 branch and
> trunk as I tested
> {code:java}
> // HDFS is configured as secure cluster
> try (FileSystem fs = newFileSystem();
> FSDataInputStream in = fs.open(PATH)) {
> waitBlockTokenExpired(in);
> in.read(0, bytes, 0, bytes.length)
> }
> private void waitBlockTokenExpired(FSDataInputStream in1) throws Exception {
> DFSInputStream innerStream = (DFSInputStream) in1.getWrappedStream();
> for (LocatedBlock block : innerStream.getAllBlocks()) {
> while (!SecurityTestUtil.isBlockTokenExpired(block.getBlockToken())) {
> Thread.sleep(100);
> }
> }
> }
> {code}
> Here is the log we got, we added a custom log before and after the block
> token refresh:
> https://github.com/bitterfox/hadoop/commit/173a9f876f2264b76af01d658f624197936fd79c
> {code}
> 2021-11-16 09:40:20,330 WARN [hedgedRead-247] impl.BlockReaderFactory: I/O
> error constructing remote block reader.
> java.io.IOException: DIGEST-MD5: IO error acquiring password
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransfer
[jira] [Work logged] (HDFS-16332) Expired block token causes slow read due to missing handling in sasl handshake
[
https://issues.apache.org/jira/browse/HDFS-16332?focusedWorklogId=687149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687149
]
ASF GitHub Bot logged work on HDFS-16332:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 06:48
Start Date: 29/Nov/21 06:48
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3677:
URL: https://github.com/apache/hadoop/pull/3677#discussion_r758075488
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferClient.java
##
@@ -603,7 +603,17 @@ private IOStreamPair doSaslHandshake(InetAddress addr,
conf, cipherOption, underlyingOut, underlyingIn, false) :
sasl.createStreamPair(out, in);
} catch (IOException ioe) {
- sendGenericSaslErrorMessage(out, ioe.getMessage());
+ try {
+sendGenericSaslErrorMessage(out, ioe.getMessage());
+ } catch (Exception ioe2) {
Review comment:
Given we catch `Exception`, I think `ioe2` should be `e`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687149)
Time Spent: 3h 20m (was: 3h 10m)
> Expired block token causes slow read due to missing handling in sasl handshake
> --
>
> Key: HDFS-16332
> URL: https://issues.apache.org/jira/browse/HDFS-16332
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, dfs, dfsclient
>Affects Versions: 2.8.5, 3.3.1
>Reporter: Shinya Yoshida
>Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot from 2021-11-18 12-11-34.png, Screenshot from
> 2021-11-18 12-14-29.png, Screenshot from 2021-11-18 13-31-35.png
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> We're operating the HBase 1.4.x cluster on Hadoop 2.8.5.
> We're recently evaluating Kerberos secured HBase and Hadoop cluster with
> production load and we observed HBase's response slows >= several seconds,
> and about several minutes for worst-case (about once~three times a month).
> The following image is a scatter plot of HBase's response slow, each circle
> is each base's slow response log.
> The X-axis is the date time of the log occurred, the Y-axis is the response
> slow time.
> !Screenshot from 2021-11-18 12-14-29.png!
> We could reproduce this issue by reducing "dfs.block.access.token.lifetime"
> and we could figure out the cause.
> (We used dfs.block.access.token.lifetime=60, i.e. 1 hour)
> When hedged read enabled:
> !Screenshot from 2021-11-18 12-11-34.png!
> When hedged read disabled:
> !Screenshot from 2021-11-18 13-31-35.png!
> As you can see, it's worst if the hedged read is enabled. However, it happens
> whether the hedged read is enabled or not.
> This impacts our 99%tile response time.
> This happens when the block token is expired and the root cause is the wrong
> handling of the InvalidToken exception in sasl handshake in
> SaslDataTransferServer.
> I propose to add a new response code for DataTransferEncryptorStatus to
> request the client to update the block token like DataTransferProtos does.
> The test code and patch is available in
> https://github.com/apache/hadoop/pull/3677
> We could reproduce this issue by the following test code in 2.8.5 branch and
> trunk as I tested
> {code:java}
> // HDFS is configured as secure cluster
> try (FileSystem fs = newFileSystem();
> FSDataInputStream in = fs.open(PATH)) {
> waitBlockTokenExpired(in);
> in.read(0, bytes, 0, bytes.length)
> }
> private void waitBlockTokenExpired(FSDataInputStream in1) throws Exception {
> DFSInputStream innerStream = (DFSInputStream) in1.getWrappedStream();
> for (LocatedBlock block : innerStream.getAllBlocks()) {
> while (!SecurityTestUtil.isBlockTokenExpired(block.getBlockToken())) {
> Thread.sleep(100);
> }
> }
> }
> {code}
> Here is the log we got, we added a custom log before and after the block
> token refresh:
> https://github.com/bitterfox/hadoop/commit/173a9f876f2264b76af01d658f624197936fd79c
> {code}
> 2021-11-16 09:40:20,330 WARN [hedgedRead-247] impl.BlockReaderFactory: I/O
> error constructing remote block reader.
> java.io.IOException: DIGEST-MD5: IO error acquiring password
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOpt
[jira] [Work logged] (HDFS-16332) Expired block token causes slow read due to missing handling in sasl handshake
[
https://issues.apache.org/jira/browse/HDFS-16332?focusedWorklogId=687148&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687148
]
ASF GitHub Bot logged work on HDFS-16332:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 06:45
Start Date: 29/Nov/21 06:45
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3677:
URL: https://github.com/apache/hadoop/pull/3677#discussion_r758074550
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferClient.java
##
@@ -603,7 +603,17 @@ private IOStreamPair doSaslHandshake(InetAddress addr,
conf, cipherOption, underlyingOut, underlyingIn, false) :
sasl.createStreamPair(out, in);
} catch (IOException ioe) {
- sendGenericSaslErrorMessage(out, ioe.getMessage());
+ try {
Review comment:
I could reproduce the test failure by closing `out` before sending error.
```
} catch (IOException ioe) {
out.close();
sendGenericSaslErrorMessage(out, ioe.getMessage());
throw ioe;
}
```
Throwing the original exception as follows fixed the test case. Therefore
your fix is really helpful.
```
} catch (IOException ioe) {
out.close();
try {
sendGenericSaslErrorMessage(out, ioe.getMessage());
} catch (Exception e) {
ioe.addSuppressed(e);
}
throw ioe;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687148)
Time Spent: 3h 10m (was: 3h)
> Expired block token causes slow read due to missing handling in sasl handshake
> --
>
> Key: HDFS-16332
> URL: https://issues.apache.org/jira/browse/HDFS-16332
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, dfs, dfsclient
>Affects Versions: 2.8.5, 3.3.1
>Reporter: Shinya Yoshida
>Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot from 2021-11-18 12-11-34.png, Screenshot from
> 2021-11-18 12-14-29.png, Screenshot from 2021-11-18 13-31-35.png
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We're operating the HBase 1.4.x cluster on Hadoop 2.8.5.
> We're recently evaluating Kerberos secured HBase and Hadoop cluster with
> production load and we observed HBase's response slows >= several seconds,
> and about several minutes for worst-case (about once~three times a month).
> The following image is a scatter plot of HBase's response slow, each circle
> is each base's slow response log.
> The X-axis is the date time of the log occurred, the Y-axis is the response
> slow time.
> !Screenshot from 2021-11-18 12-14-29.png!
> We could reproduce this issue by reducing "dfs.block.access.token.lifetime"
> and we could figure out the cause.
> (We used dfs.block.access.token.lifetime=60, i.e. 1 hour)
> When hedged read enabled:
> !Screenshot from 2021-11-18 12-11-34.png!
> When hedged read disabled:
> !Screenshot from 2021-11-18 13-31-35.png!
> As you can see, it's worst if the hedged read is enabled. However, it happens
> whether the hedged read is enabled or not.
> This impacts our 99%tile response time.
> This happens when the block token is expired and the root cause is the wrong
> handling of the InvalidToken exception in sasl handshake in
> SaslDataTransferServer.
> I propose to add a new response code for DataTransferEncryptorStatus to
> request the client to update the block token like DataTransferProtos does.
> The test code and patch is available in
> https://github.com/apache/hadoop/pull/3677
> We could reproduce this issue by the following test code in 2.8.5 branch and
> trunk as I tested
> {code:java}
> // HDFS is configured as secure cluster
> try (FileSystem fs = newFileSystem();
> FSDataInputStream in = fs.open(PATH)) {
> waitBlockTokenExpired(in);
> in.read(0, bytes, 0, bytes.length)
> }
> private void waitBlockTokenExpired(FSDataInputStream in1) throws Exception {
> DFSInputStream innerStream = (DFSInputStream) in1.getWrappedStream();
> for (LocatedBlock block : innerStream.getAllBlocks()) {
> while (!SecurityTestUtil.isBlockTokenExpired(block.getBlockToken())) {
> Thread.sleep(100);
> }
> }
> }
> {code}
> Here is the log we got, we added a custom log before an
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687146&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687146 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 29/Nov/21 06:41 Start Date: 29/Nov/21 06:41 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-981332929 > @tomscut Indeed, we don't indent case statements from a switch statement. Please fix them. > > https://github.com/apache/hadoop/blob/329b95b620909da203afc41a942958e8d229496b/dev-support/code-formatter/hadoop_idea_formatter.xml#L48 > > And please fix the check style warnings too. Thanks @tasanuma for your comments. Do I need to open a separate JIRA and fix the `hadoop_IDEa_formatter`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687146) Time Spent: 2h 40m (was: 2.5h) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 2h 40m > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[
https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=687141&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687141
]
ASF GitHub Bot logged work on HDFS-16324:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 06:22
Start Date: 29/Nov/21 06:22
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3661:
URL: https://github.com/apache/hadoop/pull/3661#discussion_r758065524
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java
##
@@ -242,6 +242,20 @@ public void testCheckSafeMode9() throws Exception {
assertTrue(content.contains("Using 3000 as SafeModeMonitor Interval"));
}
+ @Test(timeout = 2)
+ public void testCheckSafeMode10(){
+Configuration conf = new HdfsConfiguration();
+conf.setLong(DFSConfigKeys.DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY, -1);
+GenericTestUtils.LogCapturer logs =
+GenericTestUtils.LogCapturer.captureLogs(BlockManagerSafeMode.LOG);
+BlockManagerSafeMode blockManagerSafeMode = new BlockManagerSafeMode(bm,
+fsn, true, conf);
+String content = logs.getOutput();
+assertTrue(content.contains("Invalid value for " +
+DFSConfigKeys.DFS_NAMENODE_SAFEMODE_RECHECK_INTERVAL_KEY +
+".Should be greater than 0, but is -1"));
Review comment:
Would you use assertJ API? It will output much helpful error message
when the assertion fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687141)
Time Spent: 2h 20m (was: 2h 10m)
> fix error log in BlockManagerSafeMode
> -
>
> Key: HDFS-16324
> URL: https://issues.apache.org/jira/browse/HDFS-16324
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.1
>Reporter: guophilipse
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> if `recheckInterval` was set as invalid value, there will be warning log
> output, but the message seems not that proper ,we can improve it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687132&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687132
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 05:54
Start Date: 29/Nov/21 05:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#issuecomment-981312964
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 226m 8s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 324m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3732 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3913d0ee0fdd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7599d1da0b9a433c6204867748d38b67ca2de2ca |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/1/testReport/ |
| Max. process+thread count | 3450 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/1/console |
| versions | git=2.2
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687129 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 29/Nov/21 05:31 Start Date: 29/Nov/21 05:31 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-981305486 @tomscut Indeed, we don't indent case statements from a switch statement. Please fix them. https://github.com/apache/hadoop/blob/329b95b620909da203afc41a942958e8d229496b/dev-support/code-formatter/hadoop_idea_formatter.xml#L48 And please fix the check style warnings too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687129) Time Spent: 2.5h (was: 2h 20m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687128
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 05:19
Start Date: 29/Nov/21 05:19
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#discussion_r758044218
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
##
@@ -84,7 +84,7 @@ public void execute(CommandLine cmd) throws Exception {
System.out.printf("%s", workStatus.currentStateString());
}
} catch (DiskBalancerException ex) {
- LOG.error("Query plan failed. ex: {}", ex);
+ LOG.error("Query plan failed. ex: {}", ex.getMessage());
Review comment:
Thanks @virajjasani for your review. I agree with you. But I saw that in
several other places, I see that in several other places, only `Message` is
output. Do we need to change it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687128)
Time Spent: 0.5h (was: 20m)
> Fix log format for QueryCommand
> ---
>
> Key: HDFS-16361
> URL: https://issues.apache.org/jira/browse/HDFS-16361
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Fix log format for QueryCommand of disk balancer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687127
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 05:15
Start Date: 29/Nov/21 05:15
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#discussion_r758044218
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
##
@@ -84,7 +84,7 @@ public void execute(CommandLine cmd) throws Exception {
System.out.printf("%s", workStatus.currentStateString());
}
} catch (DiskBalancerException ex) {
- LOG.error("Query plan failed. ex: {}", ex);
+ LOG.error("Query plan failed. ex: {}", ex.getMessage());
Review comment:
Thanks @virajjasani for your review. I agree with you and I will update
it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687127)
Time Spent: 20m (was: 10m)
> Fix log format for QueryCommand
> ---
>
> Key: HDFS-16361
> URL: https://issues.apache.org/jira/browse/HDFS-16361
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Fix log format for QueryCommand of disk balancer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687126
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 05:10
Start Date: 29/Nov/21 05:10
Worklog Time Spent: 10m
Work Description: virajjasani commented on a change in pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#discussion_r758042827
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
##
@@ -84,7 +84,7 @@ public void execute(CommandLine cmd) throws Exception {
System.out.printf("%s", workStatus.currentStateString());
}
} catch (DiskBalancerException ex) {
- LOG.error("Query plan failed. ex: {}", ex);
+ LOG.error("Query plan failed. ex: {}", ex.getMessage());
Review comment:
How about we print the entire stacktrace and remove placeholder from log
line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687126)
Remaining Estimate: 0h
Time Spent: 10m
> Fix log format for QueryCommand
> ---
>
> Key: HDFS-16361
> URL: https://issues.apache.org/jira/browse/HDFS-16361
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Fix log format for QueryCommand of disk balancer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16361) Fix log format for QueryCommand
[ https://issues.apache.org/jira/browse/HDFS-16361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16361: -- Labels: pull-request-available (was: ) > Fix log format for QueryCommand > --- > > Key: HDFS-16361 > URL: https://issues.apache.org/jira/browse/HDFS-16361 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Fix log format for QueryCommand of disk balancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687118
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 04:26
Start Date: 29/Nov/21 04:26
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#issuecomment-981282094
> This failed UT `hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure` is
unrelated to the change.
Hi @ayushtkn , I found this unit test runs out of time several times. Can we
just increase the timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687118)
Time Spent: 1h (was: 50m)
> RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
> ---
>
> Key: HDFS-16359
> URL: https://issues.apache.org/jira/browse/HDFS-16359
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> RouterRpcServer#invokeAtAvailableNs does not take effect when retrying. See
> HDFS-15543.
> The original code of RouterRpcServer#getNameSpaceInfo looks like this:
> {code:java}
> private Set getNameSpaceInfo(String nsId) {
> Set namespaceInfos = new HashSet<>();
> for (FederationNamespaceInfo ns : namespaceInfos) {
> if (!nsId.equals(ns.getNameserviceId())) {
> namespaceInfos.add(ns);
> }
> }
> return namespaceInfos;
> } {code}
> And _namespaceInfos_ is always empty here.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687116
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 04:18
Start Date: 29/Nov/21 04:18
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#issuecomment-981278760
This failed UT `hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure` is
unrelated to the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687116)
Time Spent: 50m (was: 40m)
> RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
> ---
>
> Key: HDFS-16359
> URL: https://issues.apache.org/jira/browse/HDFS-16359
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> RouterRpcServer#invokeAtAvailableNs does not take effect when retrying. See
> HDFS-15543.
> The original code of RouterRpcServer#getNameSpaceInfo looks like this:
> {code:java}
> private Set getNameSpaceInfo(String nsId) {
> Set namespaceInfos = new HashSet<>();
> for (FederationNamespaceInfo ns : namespaceInfos) {
> if (!nsId.equals(ns.getNameserviceId())) {
> namespaceInfos.add(ns);
> }
> }
> return namespaceInfos;
> } {code}
> And _namespaceInfos_ is always empty here.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687114
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 04:02
Start Date: 29/Nov/21 04:02
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#issuecomment-981271879
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 34m 33s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not
generate ASF License warnings. |
| | | 140m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3731 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f736984cf7b7 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b9c2d3ee604d528ebfefc8400c200f1736e2921f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/2/testReport/ |
| Max. process+thread count | 2235 (vs. ulimit of 5500) |
| modules | C: hadoop-h
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687091 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 29/Nov/21 02:58 Start Date: 29/Nov/21 02:58 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-981190467 > In the following code, `datanode.getDnConf().getBlockReportInterval()` can not get the reconfigured property. Do we need to fix it? > > https://github.com/apache/hadoop/blob/c9d64bad37fbfbaa77481f67338b39a029d7dc51/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2626 Hi @tasanuma , I fixed this. PTAL. Thank you very much. In addition, I find that `case DFS_DATANODE_DATA_DIR_KEY` and `case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY` are indented too much (two more indented), do I need to fix them? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687091) Time Spent: 2h 20m (was: 2h 10m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16361) Fix log format for QueryCommand
[ https://issues.apache.org/jira/browse/HDFS-16361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16361: --- Description: Fix log format for QueryCommand of disk balancer. (was: Fix log format for QueryCommand.) > Fix log format for QueryCommand > --- > > Key: HDFS-16361 > URL: https://issues.apache.org/jira/browse/HDFS-16361 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Minor > > Fix log format for QueryCommand of disk balancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16361) Fix log format for QueryCommand
tomscut created HDFS-16361: -- Summary: Fix log format for QueryCommand Key: HDFS-16361 URL: https://issues.apache.org/jira/browse/HDFS-16361 Project: Hadoop HDFS Issue Type: Bug Reporter: tomscut Assignee: tomscut Fix log format for QueryCommand. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16360) Adding confirm prompt when removing root directory on local filesystem
Yayu Wang created HDFS-16360: Summary: Adding confirm prompt when removing root directory on local filesystem Key: HDFS-16360 URL: https://issues.apache.org/jira/browse/HDFS-16360 Project: Hadoop HDFS Issue Type: Improvement Reporter: Yayu Wang Would you like to add a confirm prompt when removing the root directory, especially when using local filesystem? I forgot to set fs.defaultFS and executed `rm -r /*` and it deleted everything on my local system. I lost everything under my $HOME directory, and I think a sudo user may destroy the host by doing so. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16327) Make dfs.namenode.max.slowpeer.collect.nodes reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16327: --- Summary: Make dfs.namenode.max.slowpeer.collect.nodes reconfigurable (was: Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable) > Make dfs.namenode.max.slowpeer.collect.nodes reconfigurable > --- > > Key: HDFS-16327 > URL: https://issues.apache.org/jira/browse/HDFS-16327 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > As the HDFS cluster expands or shrinks, the number of slow nodes to be > filtered must be dynamically adjusted. So we should make > DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable. > See HDFS-15879. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
