[jira] [Work logged] (HDFS-16358) HttpFS implementation for getSnapshotDiffReportListing
[ https://issues.apache.org/jira/browse/HDFS-16358?focusedWorklogId=687810&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687810 ] ASF GitHub Bot logged work on HDFS-16358: - Author: ASF GitHub Bot Created on: 30/Nov/21 07:50 Start Date: 30/Nov/21 07:50 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3730: URL: https://github.com/apache/hadoop/pull/3730#issuecomment-982369919 @ferhui @tasanuma could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687810) Time Spent: 1h 10m (was: 1h) > HttpFS implementation for getSnapshotDiffReportListing > -- > > Key: HDFS-16358 > URL: https://issues.apache.org/jira/browse/HDFS-16358 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HttpFS should support getSnapshotDiffReportListing API for improved snapshot > diff. WebHdfs implementation available on HDFS-16091. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687801
]
ASF GitHub Bot logged work on HDFS-16331:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 07:36
Start Date: 30/Nov/21 07:36
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3676:
URL: https://github.com/apache/hadoop/pull/3676#discussion_r759001555
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -533,78 +536,119 @@ protected Configuration getNewConf() {
public String reconfigurePropertyImpl(String property, String newVal)
throws ReconfigurationException {
switch (property) {
- case DFS_DATANODE_DATA_DIR_KEY: {
-IOException rootException = null;
+case DFS_DATANODE_DATA_DIR_KEY: {
+ IOException rootException = null;
+ try {
+LOG.info("Reconfiguring {} to {}", property, newVal);
+this.refreshVolumes(newVal);
+return getConf().get(DFS_DATANODE_DATA_DIR_KEY);
+ } catch (IOException e) {
+rootException = e;
+ } finally {
+// Send a full block report to let NN acknowledge the volume changes.
try {
- LOG.info("Reconfiguring {} to {}", property, newVal);
- this.refreshVolumes(newVal);
- return getConf().get(DFS_DATANODE_DATA_DIR_KEY);
+ triggerBlockReport(
+ new BlockReportOptions.Factory().setIncremental(false).build());
} catch (IOException e) {
- rootException = e;
+ LOG.warn("Exception while sending the block report after refreshing"
+ + " volumes {} to {}", property, newVal, e);
+ if (rootException == null) {
+rootException = e;
+ }
} finally {
- // Send a full block report to let NN acknowledge the volume changes.
- try {
-triggerBlockReport(
-new
BlockReportOptions.Factory().setIncremental(false).build());
- } catch (IOException e) {
-LOG.warn("Exception while sending the block report after
refreshing"
-+ " volumes {} to {}", property, newVal, e);
-if (rootException == null) {
- rootException = e;
-}
- } finally {
-if (rootException != null) {
- throw new ReconfigurationException(property, newVal,
- getConf().get(property), rootException);
-}
+ if (rootException != null) {
+throw new ReconfigurationException(property, newVal,
+getConf().get(property), rootException);
}
}
-break;
}
- case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY: {
-ReconfigurationException rootException = null;
-try {
- LOG.info("Reconfiguring {} to {}", property, newVal);
- int movers;
- if (newVal == null) {
-// set to default
-movers = DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT;
- } else {
-movers = Integer.parseInt(newVal);
-if (movers <= 0) {
- rootException = new ReconfigurationException(
- property,
- newVal,
- getConf().get(property),
- new IllegalArgumentException(
- "balancer max concurrent movers must be larger than 0"));
-}
- }
- boolean success = xserver.updateBalancerMaxConcurrentMovers(movers);
- if (!success) {
+ break;
+}
+case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY: {
+ ReconfigurationException rootException = null;
+ try {
+LOG.info("Reconfiguring {} to {}", property, newVal);
+int movers;
+if (newVal == null) {
+ // set to default
+ movers = DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT;
+} else {
+ movers = Integer.parseInt(newVal);
+ if (movers <= 0) {
rootException = new ReconfigurationException(
property,
newVal,
getConf().get(property),
new IllegalArgumentException(
-"Could not modify concurrent moves thread count"));
+"balancer max concurrent movers must be larger than 0"));
}
- return Integer.toString(movers);
-} catch (NumberFormatException nfe) {
+}
+boolean success = xserver.updateBalancerMaxConcurrentMovers(movers);
+if (!success) {
rootException = new ReconfigurationException(
- property, newVal, getConf().get(property), nfe);
-} finally {
- if (rootException != null)
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687800
]
ASF GitHub Bot logged work on HDFS-16331:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 07:36
Start Date: 30/Nov/21 07:36
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3676:
URL: https://github.com/apache/hadoop/pull/3676#discussion_r759001415
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
##
@@ -293,4 +295,76 @@ private void
testAcquireOnMaxConcurrentMoversReconfiguration(
assertEquals("should not be able to get thread quota", false,
dataNode.xserver.balanceThrottler.acquire());
}
+
+ @Test
+ public void testBlockReportIntervalReconfiguration()
+ throws ReconfigurationException, IOException {
+int blockReportInterval = 300 * 1000;
+for (int i = 0; i < NUM_DATA_NODE; i++) {
+ DataNode dn = cluster.getDataNodes().get(i);
+
+ // Try invalid values.
+ try {
+dn.reconfigureProperty(
+DFS_BLOCKREPORT_INTERVAL_MSEC_KEY, "text");
+fail("ReconfigurationException expected");
+ } catch (ReconfigurationException expected) {
+assertTrue("expecting NumberFormatException",
+expected.getCause() instanceof NumberFormatException);
+ }
+ try {
+dn.reconfigureProperty(
+DFS_BLOCKREPORT_INTERVAL_MSEC_KEY,
+String.valueOf(-1));
+fail("ReconfigurationException expected");
+ } catch (ReconfigurationException expected) {
+assertTrue("expecting IllegalArgumentException",
+expected.getCause() instanceof IllegalArgumentException);
+ }
+
+ // Change properties.
+ dn.reconfigureProperty(DFS_BLOCKREPORT_INTERVAL_MSEC_KEY,
+ String.valueOf(blockReportInterval));
+
+ // Verify change.
+ BlockPoolManager blockPoolManager = new BlockPoolManager(dn);
+ blockPoolManager.refreshNamenodes(dn.getConf());
Review comment:
Thanks for your review. I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687800)
Time Spent: 4h 10m (was: 4h)
> Make dfs.blockreport.intervalMsec reconfigurable
>
>
> Key: HDFS-16331
> URL: https://issues.apache.org/jira/browse/HDFS-16331
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-11-18-09-33-24-236.png,
> image-2021-11-18-09-35-35-400.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> We have a cold data cluster, which stores as EC policy. There are 24 fast
> disks on each node and each disk is 7 TB.
> Recently, many nodes have more than 10 million blocks, and the interval of
> FBR is 6h as default. Frequent FBR caused great pressure on NN.
> !image-2021-11-18-09-35-35-400.png|width=334,height=229!
> !image-2021-11-18-09-33-24-236.png|width=566,height=159!
> We want to increase the interval of FBR, but have to rolling restart the DNs,
> this operation is very heavy. In this scenario, it is necessary to make
> _dfs.blockreport.intervalMsec_ reconfigurable.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687769&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687769
]
ASF GitHub Bot logged work on HDFS-16331:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 05:25
Start Date: 30/Nov/21 05:25
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #3676:
URL: https://github.com/apache/hadoop/pull/3676#discussion_r758940164
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
##
@@ -293,4 +295,76 @@ private void
testAcquireOnMaxConcurrentMoversReconfiguration(
assertEquals("should not be able to get thread quota", false,
dataNode.xserver.balanceThrottler.acquire());
}
+
+ @Test
+ public void testBlockReportIntervalReconfiguration()
+ throws ReconfigurationException, IOException {
+int blockReportInterval = 300 * 1000;
+for (int i = 0; i < NUM_DATA_NODE; i++) {
+ DataNode dn = cluster.getDataNodes().get(i);
+
+ // Try invalid values.
+ try {
+dn.reconfigureProperty(
+DFS_BLOCKREPORT_INTERVAL_MSEC_KEY, "text");
+fail("ReconfigurationException expected");
+ } catch (ReconfigurationException expected) {
+assertTrue("expecting NumberFormatException",
+expected.getCause() instanceof NumberFormatException);
+ }
+ try {
+dn.reconfigureProperty(
+DFS_BLOCKREPORT_INTERVAL_MSEC_KEY,
+String.valueOf(-1));
+fail("ReconfigurationException expected");
+ } catch (ReconfigurationException expected) {
+assertTrue("expecting IllegalArgumentException",
+expected.getCause() instanceof IllegalArgumentException);
+ }
+
+ // Change properties.
+ dn.reconfigureProperty(DFS_BLOCKREPORT_INTERVAL_MSEC_KEY,
+ String.valueOf(blockReportInterval));
+
+ // Verify change.
+ BlockPoolManager blockPoolManager = new BlockPoolManager(dn);
+ blockPoolManager.refreshNamenodes(dn.getConf());
Review comment:
You may not need to call `#refreshNamenodes()` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687769)
Time Spent: 4h (was: 3h 50m)
> Make dfs.blockreport.intervalMsec reconfigurable
>
>
> Key: HDFS-16331
> URL: https://issues.apache.org/jira/browse/HDFS-16331
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-11-18-09-33-24-236.png,
> image-2021-11-18-09-35-35-400.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> We have a cold data cluster, which stores as EC policy. There are 24 fast
> disks on each node and each disk is 7 TB.
> Recently, many nodes have more than 10 million blocks, and the interval of
> FBR is 6h as default. Frequent FBR caused great pressure on NN.
> !image-2021-11-18-09-35-35-400.png|width=334,height=229!
> !image-2021-11-18-09-33-24-236.png|width=566,height=159!
> We want to increase the interval of FBR, but have to rolling restart the DNs,
> this operation is very heavy. In this scenario, it is necessary to make
> _dfs.blockreport.intervalMsec_ reconfigurable.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687768
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 05:22
Start Date: 30/Nov/21 05:22
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#issuecomment-982296717
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 37s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 118m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3731 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9ae154449ea0 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8324d9bac06c126503e5831efd8cb4d6a2c93a49 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/5/testReport/ |
| Max. process+thread count | 2545 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687767
]
ASF GitHub Bot logged work on HDFS-16331:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 05:22
Start Date: 30/Nov/21 05:22
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #3676:
URL: https://github.com/apache/hadoop/pull/3676#discussion_r758939114
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -533,78 +536,119 @@ protected Configuration getNewConf() {
public String reconfigurePropertyImpl(String property, String newVal)
throws ReconfigurationException {
switch (property) {
- case DFS_DATANODE_DATA_DIR_KEY: {
-IOException rootException = null;
+case DFS_DATANODE_DATA_DIR_KEY: {
+ IOException rootException = null;
+ try {
+LOG.info("Reconfiguring {} to {}", property, newVal);
+this.refreshVolumes(newVal);
+return getConf().get(DFS_DATANODE_DATA_DIR_KEY);
+ } catch (IOException e) {
+rootException = e;
+ } finally {
+// Send a full block report to let NN acknowledge the volume changes.
try {
- LOG.info("Reconfiguring {} to {}", property, newVal);
- this.refreshVolumes(newVal);
- return getConf().get(DFS_DATANODE_DATA_DIR_KEY);
+ triggerBlockReport(
+ new BlockReportOptions.Factory().setIncremental(false).build());
} catch (IOException e) {
- rootException = e;
+ LOG.warn("Exception while sending the block report after refreshing"
+ + " volumes {} to {}", property, newVal, e);
+ if (rootException == null) {
+rootException = e;
+ }
} finally {
- // Send a full block report to let NN acknowledge the volume changes.
- try {
-triggerBlockReport(
-new
BlockReportOptions.Factory().setIncremental(false).build());
- } catch (IOException e) {
-LOG.warn("Exception while sending the block report after
refreshing"
-+ " volumes {} to {}", property, newVal, e);
-if (rootException == null) {
- rootException = e;
-}
- } finally {
-if (rootException != null) {
- throw new ReconfigurationException(property, newVal,
- getConf().get(property), rootException);
-}
+ if (rootException != null) {
+throw new ReconfigurationException(property, newVal,
+getConf().get(property), rootException);
}
}
-break;
}
- case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY: {
-ReconfigurationException rootException = null;
-try {
- LOG.info("Reconfiguring {} to {}", property, newVal);
- int movers;
- if (newVal == null) {
-// set to default
-movers = DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT;
- } else {
-movers = Integer.parseInt(newVal);
-if (movers <= 0) {
- rootException = new ReconfigurationException(
- property,
- newVal,
- getConf().get(property),
- new IllegalArgumentException(
- "balancer max concurrent movers must be larger than 0"));
-}
- }
- boolean success = xserver.updateBalancerMaxConcurrentMovers(movers);
- if (!success) {
+ break;
+}
+case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY: {
+ ReconfigurationException rootException = null;
+ try {
+LOG.info("Reconfiguring {} to {}", property, newVal);
+int movers;
+if (newVal == null) {
+ // set to default
+ movers = DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT;
+} else {
+ movers = Integer.parseInt(newVal);
+ if (movers <= 0) {
rootException = new ReconfigurationException(
property,
newVal,
getConf().get(property),
new IllegalArgumentException(
-"Could not modify concurrent moves thread count"));
+"balancer max concurrent movers must be larger than 0"));
}
- return Integer.toString(movers);
-} catch (NumberFormatException nfe) {
+}
+boolean success = xserver.updateBalancerMaxConcurrentMovers(movers);
+if (!success) {
rootException = new ReconfigurationException(
- property, newVal, getConf().get(property), nfe);
-} finally {
- if (rootException != null)
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687745&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687745
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 03:49
Start Date: 30/Nov/21 03:49
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#issuecomment-982257714
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 23m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 126m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3731 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 971ace51d45a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6a15a84d62f0036ec0bf82bf743a6597ac4a45f2 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/3/testReport/ |
| Max. process+thread count | 2383 (vs. ulimit of 5500) |
| modules
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687740&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687740
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 03:37
Start Date: 30/Nov/21 03:37
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#discussion_r758906333
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
##
@@ -726,9 +726,10 @@ static String getMethodName() {
* @return List of name spaces in the federation on
* removing the already invoked namespaceinfo.
*/
- private Set getNameSpaceInfo(String nsId) {
+ private Set getNameSpaceInfo(
Review comment:
Thanks @goiri for your comment, I noticed that this method is currently
only called in one place. So do we still need to change it to static? Could you
please give me some advice. Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687740)
Time Spent: 1h 50m (was: 1h 40m)
> RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
> ---
>
> Key: HDFS-16359
> URL: https://issues.apache.org/jira/browse/HDFS-16359
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> RouterRpcServer#invokeAtAvailableNs does not take effect when retrying. See
> HDFS-15543.
> The original code of RouterRpcServer#getNameSpaceInfo looks like this:
> {code:java}
> private Set getNameSpaceInfo(String nsId) {
> Set namespaceInfos = new HashSet<>();
> for (FederationNamespaceInfo ns : namespaceInfos) {
> if (!nsId.equals(ns.getNameserviceId())) {
> namespaceInfos.add(ns);
> }
> }
> return namespaceInfos;
> } {code}
> And _namespaceInfos_ is always empty here.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687735
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 03:27
Start Date: 30/Nov/21 03:27
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#discussion_r758902855
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
##
@@ -726,9 +726,10 @@ static String getMethodName() {
* @return List of name spaces in the federation on
* removing the already invoked namespaceinfo.
*/
- private Set getNameSpaceInfo(String nsId) {
+ private Set getNameSpaceInfo(
Review comment:
This could be a static method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687735)
Time Spent: 1h 40m (was: 1.5h)
> RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
> ---
>
> Key: HDFS-16359
> URL: https://issues.apache.org/jira/browse/HDFS-16359
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> RouterRpcServer#invokeAtAvailableNs does not take effect when retrying. See
> HDFS-15543.
> The original code of RouterRpcServer#getNameSpaceInfo looks like this:
> {code:java}
> private Set getNameSpaceInfo(String nsId) {
> Set namespaceInfos = new HashSet<>();
> for (FederationNamespaceInfo ns : namespaceInfos) {
> if (!nsId.equals(ns.getNameserviceId())) {
> namespaceInfos.add(ns);
> }
> }
> return namespaceInfos;
> } {code}
> And _namespaceInfos_ is always empty here.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687731&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687731
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 03:13
Start Date: 30/Nov/21 03:13
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#issuecomment-982242740
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 2m 12s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| -1 :x: | mvnsite | 0m 43s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/4/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in trunk failed. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/4/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs-rbf in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 24s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/4/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs-rbf in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 23s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/4/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in trunk failed. |
| +1 :green_heart: | shadedclient | 27m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| -1 :x: | javadoc | 0m 37s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3731/4/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 99 new + 0 unchanged
- 0 fixed = 99 total (was 0) |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 20m 40s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687713&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687713 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 30/Nov/21 01:57 Start Date: 30/Nov/21 01:57 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-982206173 We don't need to fix this [checkstyle issue](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/10/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) because variable `DNConf#blockReportInterval` is not `private` before the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687713) Time Spent: 3h 40m (was: 3.5h) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 3h 40m > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687712 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 30/Nov/21 01:55 Start Date: 30/Nov/21 01:55 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-982206173 We don't need to fix this [checkstyle issue](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/10/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) because `DNConf#blockReportInterval` is not `private` before the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687712) Time Spent: 3.5h (was: 3h 20m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 3.5h > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687709&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687709
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 30/Nov/21 01:40
Start Date: 30/Nov/21 01:40
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#discussion_r758865364
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
##
@@ -726,9 +727,10 @@ static String getMethodName() {
* @return List of name spaces in the federation on
* removing the already invoked namespaceinfo.
*/
- private Set getNameSpaceInfo(String nsId) {
+ private Set getNameSpaceInfo(
+ Set nss, String nsId) {
Set namespaceInfos = new HashSet<>();
-for (FederationNamespaceInfo ns : namespaceInfos) {
+for (FederationNamespaceInfo ns : nss) {
Review comment:
Thanks @goiri for your comments and suggestion. I rolled back the way
catching with log and replaced with calling #getServerDefaults (create a
separate unit test #testInvokeAtAvailableNs). PTAL. Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687709)
Time Spent: 1h 20m (was: 1h 10m)
> RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
> ---
>
> Key: HDFS-16359
> URL: https://issues.apache.org/jira/browse/HDFS-16359
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> RouterRpcServer#invokeAtAvailableNs does not take effect when retrying. See
> HDFS-15543.
> The original code of RouterRpcServer#getNameSpaceInfo looks like this:
> {code:java}
> private Set getNameSpaceInfo(String nsId) {
> Set namespaceInfos = new HashSet<>();
> for (FederationNamespaceInfo ns : namespaceInfos) {
> if (!nsId.equals(ns.getNameserviceId())) {
> namespaceInfos.add(ns);
> }
> }
> return namespaceInfos;
> } {code}
> And _namespaceInfos_ is always empty here.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[ https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687688 ] ASF GitHub Bot logged work on HDFS-16361: - Author: ASF GitHub Bot Created on: 29/Nov/21 23:58 Start Date: 29/Nov/21 23:58 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3732: URL: https://github.com/apache/hadoop/pull/3732#issuecomment-982141097 Hi @virajjasani @tasanuma @ferhui, PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687688) Time Spent: 1.5h (was: 1h 20m) > Fix log format for QueryCommand > --- > > Key: HDFS-16361 > URL: https://issues.apache.org/jira/browse/HDFS-16361 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Fix log format for QueryCommand of disk balancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687687&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687687 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 29/Nov/21 23:56 Start Date: 29/Nov/21 23:56 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-982140122 The failed unit tests are unrelated to the change and work fine locally. They failed because of OOM. We might need to take a look at the compile environment? Hi @tasanuma , I have fixed the two problems you mentioned before, please have a look, thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687687) Time Spent: 3h 20m (was: 3h 10m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 3h 20m > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16303) Losing over 100 datanodes in state decommissioning results in full blockage of all datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-16303?focusedWorklogId=687614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687614
]
ASF GitHub Bot logged work on HDFS-16303:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 21:00
Start Date: 29/Nov/21 21:00
Worklog Time Spent: 10m
Work Description: sodonnel commented on pull request #3675:
URL: https://github.com/apache/hadoop/pull/3675#issuecomment-982018340
For me, DECOMMISSION_IN_PROGRESS + DEAD is an error state that means
decommission has effectively failed. There is a case where it can complete, but
what does that really mean - if the node is dead, it has not been gracefully
stopped. If it wasn't for the way decommission is triggered using the hosts
files, I would suggest switching it back to IN_SERVICE + DEAD, and let it be
treated like any other dead host.
If you have some monitoring tool tracking the decommission, and it sees
"DECOMMISSIONED", then it assumes the decommission went fine.
If if sees DECOMMISSION_IN_PROGRESS + DEAD, then its a flag that the admin
needs to go look into it, as it should not have happened - perhaps they need to
bring the node back, or conclude that the cluster is still OK without it (no
missing blocks) and add it to the exclude list and forget about it.
My feeling is that the priority queue idea adds some more complexity to an
already hard to follow process / code area and I wonder if it is better to just
remove the node from the monitor and let it be dealt with manually, which may
be required a lot of the time anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687614)
Time Spent: 4h 20m (was: 4h 10m)
> Losing over 100 datanodes in state decommissioning results in full blockage
> of all datanode decommissioning
> ---
>
> Key: HDFS-16303
> URL: https://issues.apache.org/jira/browse/HDFS-16303
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.10.1, 3.3.1
>Reporter: Kevin Wikant
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> h2. Impact
> HDFS datanode decommissioning does not make any forward progress. For
> example, the user adds X datanodes to the "dfs.hosts.exclude" file and all X
> of those datanodes remain in state decommissioning forever without making any
> forward progress towards being decommissioned.
> h2. Root Cause
> The HDFS Namenode class "DatanodeAdminManager" is responsible for
> decommissioning datanodes.
> As per this "hdfs-site" configuration:
> {quote}Config = dfs.namenode.decommission.max.concurrent.tracked.nodes
> Default Value = 100
> The maximum number of decommission-in-progress datanodes nodes that will be
> tracked at one time by the namenode. Tracking a decommission-in-progress
> datanode consumes additional NN memory proportional to the number of blocks
> on the datnode. Having a conservative limit reduces the potential impact of
> decomissioning a large number of nodes at once. A value of 0 means no limit
> will be enforced.
> {quote}
> The Namenode will only actively track up to 100 datanodes for decommissioning
> at any given time, as to avoid Namenode memory pressure.
> Looking into the "DatanodeAdminManager" code:
> * a new datanode is only removed from the "tracked.nodes" set when it
> finishes decommissioning
> * a new datanode is only added to the "tracked.nodes" set if there is fewer
> than 100 datanodes being tracked
> So in the event that there are more than 100 datanodes being decommissioned
> at a given time, some of those datanodes will not be in the "tracked.nodes"
> set until 1 or more datanodes in the "tracked.nodes" finishes
> decommissioning. This is generally not a problem because the datanodes in
> "tracked.nodes" will eventually finish decommissioning, but there is an edge
> case where this logic prevents the namenode from making any forward progress
> towards decommissioning.
> If all 100 datanodes in the "tracked.nodes" are unable to finish
> decommissioning, then other datanodes (which may be able to be
> decommissioned) will never get added to "tracked.nodes" and therefore will
> never get the opportunity to be decommissioned.
> This can occur due the following issue:
> {quote}2021-10-21 12:39:24,048 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.Block
[jira] [Work logged] (HDFS-16303) Losing over 100 datanodes in state decommissioning results in full blockage of all datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-16303?focusedWorklogId=687594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687594
]
ASF GitHub Bot logged work on HDFS-16303:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 20:17
Start Date: 29/Nov/21 20:17
Worklog Time Spent: 10m
Work Description: KevinWikant commented on pull request #3675:
URL: https://github.com/apache/hadoop/pull/3675#issuecomment-981982641
@sodonnel let me know your thoughts, but I think the problem with removing a
dead node from the DatanodeAdminManager until it comes back alive again is that
it will never be decommissioned if it never comes alive again
Do you see any major downside in keeping the dead nodes in the pendingNodes
priority queue behind all the alive nodes? Because of the priority queue
ordering the dead nodes will not block decommissioning of alive nodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687594)
Time Spent: 4h 10m (was: 4h)
> Losing over 100 datanodes in state decommissioning results in full blockage
> of all datanode decommissioning
> ---
>
> Key: HDFS-16303
> URL: https://issues.apache.org/jira/browse/HDFS-16303
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.10.1, 3.3.1
>Reporter: Kevin Wikant
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> h2. Impact
> HDFS datanode decommissioning does not make any forward progress. For
> example, the user adds X datanodes to the "dfs.hosts.exclude" file and all X
> of those datanodes remain in state decommissioning forever without making any
> forward progress towards being decommissioned.
> h2. Root Cause
> The HDFS Namenode class "DatanodeAdminManager" is responsible for
> decommissioning datanodes.
> As per this "hdfs-site" configuration:
> {quote}Config = dfs.namenode.decommission.max.concurrent.tracked.nodes
> Default Value = 100
> The maximum number of decommission-in-progress datanodes nodes that will be
> tracked at one time by the namenode. Tracking a decommission-in-progress
> datanode consumes additional NN memory proportional to the number of blocks
> on the datnode. Having a conservative limit reduces the potential impact of
> decomissioning a large number of nodes at once. A value of 0 means no limit
> will be enforced.
> {quote}
> The Namenode will only actively track up to 100 datanodes for decommissioning
> at any given time, as to avoid Namenode memory pressure.
> Looking into the "DatanodeAdminManager" code:
> * a new datanode is only removed from the "tracked.nodes" set when it
> finishes decommissioning
> * a new datanode is only added to the "tracked.nodes" set if there is fewer
> than 100 datanodes being tracked
> So in the event that there are more than 100 datanodes being decommissioned
> at a given time, some of those datanodes will not be in the "tracked.nodes"
> set until 1 or more datanodes in the "tracked.nodes" finishes
> decommissioning. This is generally not a problem because the datanodes in
> "tracked.nodes" will eventually finish decommissioning, but there is an edge
> case where this logic prevents the namenode from making any forward progress
> towards decommissioning.
> If all 100 datanodes in the "tracked.nodes" are unable to finish
> decommissioning, then other datanodes (which may be able to be
> decommissioned) will never get added to "tracked.nodes" and therefore will
> never get the opportunity to be decommissioned.
> This can occur due the following issue:
> {quote}2021-10-21 12:39:24,048 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager
> (DatanodeAdminMonitor-0): Node W.X.Y.Z:50010 is dead while in Decommission In
> Progress. Cannot be safely decommissioned or be in maintenance since there is
> risk of reduced data durability or data loss. Either restart the failed node
> or force decommissioning or maintenance by removing, calling refreshNodes,
> then re-adding to the excludes or host config files.
> {quote}
> If a Datanode is lost while decommissioning (for example if the underlying
> hardware fails or is lost), then it will remain in state decommissioning
> forever.
> If 100 or more Datanodes are lost while decommissioning over the Hadoop
> cluster lifetime, then this is enough to completely fill up
[jira] [Work logged] (HDFS-16359) RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
[
https://issues.apache.org/jira/browse/HDFS-16359?focusedWorklogId=687558&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687558
]
ASF GitHub Bot logged work on HDFS-16359:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 19:24
Start Date: 29/Nov/21 19:24
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #3731:
URL: https://github.com/apache/hadoop/pull/3731#discussion_r758668281
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
##
@@ -726,9 +727,10 @@ static String getMethodName() {
* @return List of name spaces in the federation on
* removing the already invoked namespaceinfo.
*/
- private Set getNameSpaceInfo(String nsId) {
+ private Set getNameSpaceInfo(
+ Set nss, String nsId) {
Set namespaceInfos = new HashSet<>();
-for (FederationNamespaceInfo ns : namespaceInfos) {
+for (FederationNamespaceInfo ns : nss) {
Review comment:
Very good catch.
Catching the issue with the log is fine but I think there must be an easier
way to test that the feature works.
Is there any other unit test we can add?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687558)
Time Spent: 1h 10m (was: 1h)
> RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying
> ---
>
> Key: HDFS-16359
> URL: https://issues.apache.org/jira/browse/HDFS-16359
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> RouterRpcServer#invokeAtAvailableNs does not take effect when retrying. See
> HDFS-15543.
> The original code of RouterRpcServer#getNameSpaceInfo looks like this:
> {code:java}
> private Set getNameSpaceInfo(String nsId) {
> Set namespaceInfos = new HashSet<>();
> for (FederationNamespaceInfo ns : namespaceInfos) {
> if (!nsId.equals(ns.getNameserviceId())) {
> namespaceInfos.add(ns);
> }
> }
> return namespaceInfos;
> } {code}
> And _namespaceInfos_ is always empty here.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687528&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687528
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 18:57
Start Date: 29/Nov/21 18:57
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#issuecomment-981921507
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 1m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 254m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 369m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3732 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 8505a5fe4385 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 174c98d3ce4b337f2f21135577864e1dfd0e12b9 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:P
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687510
]
ASF GitHub Bot logged work on HDFS-16331:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 18:17
Start Date: 29/Nov/21 18:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3676:
URL: https://github.com/apache/hadoop/pull/3676#issuecomment-981892355
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 54s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/10/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 134 unchanged
- 55 fixed = 135 total (was 189) |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 259m 36s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 370m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3676 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0e45a9c7ca40 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d7329c
[jira] [Commented] (HDFS-16293) Client sleeps and holds 'dataQueue' when DataNodes are congested
[
https://issues.apache.org/jira/browse/HDFS-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450621#comment-17450621
]
Hadoop QA commented on HDFS-16293:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m
47s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 36s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 39m
22s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 7m
16s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 19s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/740/artifact/out/diff-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red} hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 647 unchanged
- 0 fixed = 649 total (was 647) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 47s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/740/artifact/out/diff-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color}
| {color:red}
hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with
JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new + 624
unchanged - 0 fixed = 626 total (was 624) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
[jira] [Commented] (HDFS-16293) Client sleeps and holds 'dataQueue' when DataNodes are congested
[
https://issues.apache.org/jira/browse/HDFS-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450565#comment-17450565
]
Hadoop QA commented on HDFS-16293:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
45s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m
35s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 27m
51s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-mvninstall-root.txt{color}
| {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
30s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red} hadoop-hdfs-project in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
29s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color}
| {color:red} hadoop-hdfs-project in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project.txt{color}
| {color:orange} The patch fails to run checkstyle in hadoop-hdfs-project
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt{color}
| {color:red} hadoop-hdfs-client in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
28s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
2m 28s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red} hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red} hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color}
| {color:red} hadoop-hdfs-client in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
30s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/741/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color}
| {color:red} hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 5m
28s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[
https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=687345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687345
]
ASF GitHub Bot logged work on HDFS-16324:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 14:40
Start Date: 29/Nov/21 14:40
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3661:
URL: https://github.com/apache/hadoop/pull/3661#issuecomment-981696558
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 36s | | trunk passed |
| +1 :green_heart: | compile | 5m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 4m 59s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | | the patch passed |
| +1 :green_heart: | compile | 5m 9s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 5m 9s |
[/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/6/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs-project-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 644 unchanged - 1
fixed = 645 total (was 645) |
| +1 :green_heart: | compile | 4m 50s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 4m 50s |
[/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/6/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs-project-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 1 new +
622 unchanged - 1 fixed = 623 total (was 623) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 7s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/6/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 133 new + 101 unchanged - 0 fixed
= 234 total (was 101) |
| +1 :green_heart: | mvnsite | 2m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 25s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 230m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s
[jira] [Commented] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450487#comment-17450487
]
Hadoop QA commented on HDFS-16349:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
10s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} branch-3.2.3 Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
8s{color} | {color:green}{color} | {color:green} branch-3.2.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green}{color} | {color:green} branch-3.2.3 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} branch-3.2.3 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} branch-3.2.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 7s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} branch-3.2.3 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 19m
26s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
18s{color} | {color:green}{color} | {color:green} branch-3.2.3 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 15s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}202m 26s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/739/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}288m 13s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFsck |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/739/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-16349 |
| JIRA Patch URL |
https://issues.apache.o
[jira] [Work logged] (HDFS-16317) Backport HDFS-14729 for branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16317?focusedWorklogId=687300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687300 ] ASF GitHub Bot logged work on HDFS-16317: - Author: ASF GitHub Bot Created on: 29/Nov/21 12:53 Start Date: 29/Nov/21 12:53 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on pull request #3692: URL: https://github.com/apache/hadoop/pull/3692#issuecomment-981606852 lgtm..thanks @AnanyaSingh2121 for contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687300) Time Spent: 1.5h (was: 1h 20m) > Backport HDFS-14729 for branch-3.2 > -- > > Key: HDFS-16317 > URL: https://issues.apache.org/jira/browse/HDFS-16317 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 3.2.2 >Reporter: Ananya Singh >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Our security tool raised the following security flaw on Hadoop 3.2.2: > +[CVE-2015-9251 : > |http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-9251] > [https://nvd.nist.gov/vuln/detail/|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] > > [CVE-2015-9251|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-9251]+ > +[CVE-2019-11358|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2019-11358] > : > [https://nvd.nist.gov/vuln/detail/|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] > > [CVE-2019-11358|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2019-11358]+ > +[CVE-2020-11022 > |http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-11022] : > [https://nvd.nist.gov/vuln/detail/|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] > > [CVE-2020-11022|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-11022]+ > > +[CVE-2020-11023 > |http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-11023] [ > |http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-11022] : > [https://nvd.nist.gov/vuln/detail/|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] > > [CVE-2020-11023|http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2020-11023]+ > > > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16358) HttpFS implementation for getSnapshotDiffReportListing
[
https://issues.apache.org/jira/browse/HDFS-16358?focusedWorklogId=687296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687296
]
ASF GitHub Bot logged work on HDFS-16358:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 12:39
Start Date: 29/Nov/21 12:39
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3730:
URL: https://github.com/apache/hadoop/pull/3730#issuecomment-981596729
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 33s | | trunk passed |
| +1 :green_heart: | compile | 6m 43s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 5m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 6m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 6m 41s | | the patch passed |
| +1 :green_heart: | compile | 5m 59s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 5m 59s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 31s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 8m 52s | | hadoop-hdfs-httpfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 148m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3730/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3730 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux aca8e46c383a 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9356fb168366414ba41e1262c1d6946631f661dc |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3730/3/testReport/ |
| Max. process+thread count | 657 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-proj
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687293
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 12:33
Start Date: 29/Nov/21 12:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#issuecomment-981592938
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 24s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 23s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 25s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 25s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 25s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 25s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 26s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3732/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 26s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](htt
[jira] [Work logged] (HDFS-16361) Fix log format for QueryCommand
[
https://issues.apache.org/jira/browse/HDFS-16361?focusedWorklogId=687285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687285
]
ASF GitHub Bot logged work on HDFS-16361:
-
Author: ASF GitHub Bot
Created on: 29/Nov/21 12:16
Start Date: 29/Nov/21 12:16
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3732:
URL: https://github.com/apache/hadoop/pull/3732#discussion_r758306918
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/diskbalancer/command/QueryCommand.java
##
@@ -84,7 +84,7 @@ public void execute(CommandLine cmd) throws Exception {
System.out.printf("%s", workStatus.currentStateString());
}
} catch (DiskBalancerException ex) {
- LOG.error("Query plan failed. ex: {}", ex);
+ LOG.error("Query plan failed. ex: {}", ex.getMessage());
Review comment:
Thanks @virajjasani for your comments. I checked several other places,
all specifying `message:{}`. So I'll just change this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 687285)
Time Spent: 1h (was: 50m)
> Fix log format for QueryCommand
> ---
>
> Key: HDFS-16361
> URL: https://issues.apache.org/jira/browse/HDFS-16361
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Fix log format for QueryCommand of disk balancer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15180) DataNode FsDatasetImpl Fine-Grained Locking via BlockPool.
[ https://issues.apache.org/jira/browse/HDFS-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450379#comment-17450379 ] Stephen O'Donnell commented on HDFS-15180: -- In Cloudera, we have not been looking into this issue actively, but it is an interesting one. We have went ahead with HDFS-15160 in our latest release and so far have not seen any problems from it. Our hope is the relatively minor change in HDFS-15160 can have a large benefit and is easy to disable with a config switch if any problems are detected. This change probably has a bigger impact that HDFS-15160, but is more complicated and so carries more risk. It is good to know you have been running it with no issue for some time - that does help give us more confidence there are no issues with the approach. > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool. > --- > > Key: HDFS-15180 > URL: https://issues.apache.org/jira/browse/HDFS-15180 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.2.0 >Reporter: Qi Zhu >Assignee: Aiphago >Priority: Major > Attachments: HDFS-15180.001.patch, HDFS-15180.002.patch, > HDFS-15180.003.patch, HDFS-15180.004.patch, > image-2020-03-10-17-22-57-391.png, image-2020-03-10-17-31-58-830.png, > image-2020-03-10-17-34-26-368.png, image-2020-04-09-11-20-36-459.png > > > Now the FsDatasetImpl datasetLock is heavy, when their are many namespaces in > big cluster. If we can split the FsDatasetImpl datasetLock via blockpool. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16293) Client sleeps and holds 'dataQueue' when DataNodes are congested
[ https://issues.apache.org/jira/browse/HDFS-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanxin Zhu updated HDFS-16293: --- Attachment: HDFS-16293.02.patch > Client sleeps and holds 'dataQueue' when DataNodes are congested > > > Key: HDFS-16293 > URL: https://issues.apache.org/jira/browse/HDFS-16293 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.2.2, 3.3.1, 3.2.3 >Reporter: Yuanxin Zhu >Priority: Major > Attachments: HDFS-16293.01-branch-3.2.2.patch, HDFS-16293.01.patch, > HDFS-16293.02.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When I open the ECN and use Terasort(500G data,8 DataNodes,76 vcores/DN) for > testing, DataNodes are congested(HDFS-8008). The client enters the sleep > state after receiving the ACK for many times, but does not release the > 'dataQueue'. The ResponseProcessor thread needs the 'dataQueue' to execute > 'ackQueue.getFirst()', so the ResponseProcessor will wait for the client to > release the 'dataQueue', which is equivalent to that the ResponseProcessor > thread also enters sleep, resulting in ACK delay.MapReduce tasks can be > delayed by tens of minutes or even hours. > The DataStreamer thread can first execute 'one = dataQueue. getFirst()', > release 'dataQueue', and then judge whether to execute 'backOffIfNecessary()' > according to 'one.isHeartbeatPacket()' > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16293) Client sleeps and holds 'dataQueue' when DataNodes are congested
[ https://issues.apache.org/jira/browse/HDFS-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanxin Zhu updated HDFS-16293: --- Attachment: (was: HDFS-16293.02.patch) > Client sleeps and holds 'dataQueue' when DataNodes are congested > > > Key: HDFS-16293 > URL: https://issues.apache.org/jira/browse/HDFS-16293 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.2.2, 3.3.1, 3.2.3 >Reporter: Yuanxin Zhu >Priority: Major > Attachments: HDFS-16293.01-branch-3.2.2.patch, HDFS-16293.01.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When I open the ECN and use Terasort(500G data,8 DataNodes,76 vcores/DN) for > testing, DataNodes are congested(HDFS-8008). The client enters the sleep > state after receiving the ACK for many times, but does not release the > 'dataQueue'. The ResponseProcessor thread needs the 'dataQueue' to execute > 'ackQueue.getFirst()', so the ResponseProcessor will wait for the client to > release the 'dataQueue', which is equivalent to that the ResponseProcessor > thread also enters sleep, resulting in ACK delay.MapReduce tasks can be > delayed by tens of minutes or even hours. > The DataStreamer thread can first execute 'one = dataQueue. getFirst()', > release 'dataQueue', and then judge whether to execute 'backOffIfNecessary()' > according to 'one.isHeartbeatPacket()' > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16293) Client sleeps and holds 'dataQueue' when DataNodes are congested
[ https://issues.apache.org/jira/browse/HDFS-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanxin Zhu updated HDFS-16293: --- Attachment: HDFS-16293.02.patch > Client sleeps and holds 'dataQueue' when DataNodes are congested > > > Key: HDFS-16293 > URL: https://issues.apache.org/jira/browse/HDFS-16293 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.2.2, 3.3.1, 3.2.3 >Reporter: Yuanxin Zhu >Priority: Major > Attachments: HDFS-16293.01-branch-3.2.2.patch, HDFS-16293.01.patch, > HDFS-16293.02.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When I open the ECN and use Terasort(500G data,8 DataNodes,76 vcores/DN) for > testing, DataNodes are congested(HDFS-8008). The client enters the sleep > state after receiving the ACK for many times, but does not release the > 'dataQueue'. The ResponseProcessor thread needs the 'dataQueue' to execute > 'ackQueue.getFirst()', so the ResponseProcessor will wait for the client to > release the 'dataQueue', which is equivalent to that the ResponseProcessor > thread also enters sleep, resulting in ACK delay.MapReduce tasks can be > delayed by tens of minutes or even hours. > The DataStreamer thread can first execute 'one = dataQueue. getFirst()', > release 'dataQueue', and then judge whether to execute 'backOffIfNecessary()' > according to 'one.isHeartbeatPacket()' > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16358) HttpFS implementation for getSnapshotDiffReportListing
[ https://issues.apache.org/jira/browse/HDFS-16358?focusedWorklogId=687212&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687212 ] ASF GitHub Bot logged work on HDFS-16358: - Author: ASF GitHub Bot Created on: 29/Nov/21 10:10 Start Date: 29/Nov/21 10:10 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3730: URL: https://github.com/apache/hadoop/pull/3730#issuecomment-981480689 Minor refactor in the latest commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687212) Time Spent: 50m (was: 40m) > HttpFS implementation for getSnapshotDiffReportListing > -- > > Key: HDFS-16358 > URL: https://issues.apache.org/jira/browse/HDFS-16358 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > HttpFS should support getSnapshotDiffReportListing API for improved snapshot > diff. WebHdfs implementation available on HDFS-16091. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687203&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687203 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 29/Nov/21 09:37 Start Date: 29/Nov/21 09:37 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-981454424 > @tomscut Oh, I meant please fix the indents of `case DFS_DATANODE_DATA_DIR_KEY` and `case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY` since they aren't in accordance with the hadoop_idea_formatter. I think we can do it in this PR. Sorry, I misunderstood. And I will fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687203) Time Spent: 3h (was: 2h 50m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 3h > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chuanjie.duan updated HDFS-16349:
-
Attachment: HDFS-16349-branch-3.2.3.patch
Fix Version/s: 3.2.3
3.2.2
Status: Patch Available (was: Open)
> FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
> -
>
> Key: HDFS-16349
> URL: https://issues.apache.org/jira/browse/HDFS-16349
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.2, 3.2.3
>Reporter: chuanjie.duan
>Priority: Blocker
> Fix For: 3.2.3, 3.2.2
>
> Attachments: HDFS-16349-branch-3.2.3.patch
>
>
> 2021-11-22 20:36:44,440 INFO
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Using longest
> log: 10.65.57.133:8485=segmentState {
> startTxId: 3906965
> endTxId: 3906965
> isInProgress: false
> }
> lastWriterEpoch: 5
> lastCommittedTxId: 3906964
> 2021-11-22 20:36:44,457 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in /data12/data/flashHadoopU/namenode/current
> 2021-11-22 20:36:44,495 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /data12/data/flashHadoopU/namenode/current/edits_inprogress_3898378
> ->
> /data12/data/flashHadoopU/namenode/current/edits_3898378-3898412
> 2021-11-22 20:36:44,657 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Gap in transactions. Expected to be able to read up
> until at least txid 2510934 but unable to find any edit logs containing txid
> 2510933
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1578)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1536)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:976)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> 2021-11-22 20:36:44,660 INFO org.mortbay.log: Stopped
> HttpServer2$selectchannelconnectorwithsafestar...@pro-hadoop-dc01-057133.vm.dc01.hellocloud.tech:50070
> 2021-11-22 20:36:44,760 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2021-11-22 20:36:44,761 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> Old version: 2.7.3
> New version: 3.2.2
> Steps to Reproduce
> Step 1: Start NN1 as active , NN2 as standby .
> Step 2: Perform "hdfs dfsadmin -rollingUpgrade prepare"
> Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
> option.
> Step 4: DN also restarted in upgrade mode.
> Step 5: Restart journalnode with new hadoop version
> Step 6: a few days later
> Step 7: bring down both NN, journalnode, DN
> Step 8: Start JN with old version
> Step 9: Start NN1 with rolling upgrade rollback option. nn started failed
> with above ERROR(Above mentioned txid version 2510933 has been deleted
> because of checkpoint mechanism)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chuanjie.duan updated HDFS-16349:
-
Attachment: (was: HDFS-16349-branch-3.2.3.patch)
> FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
> -
>
> Key: HDFS-16349
> URL: https://issues.apache.org/jira/browse/HDFS-16349
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.2, 3.2.3
>Reporter: chuanjie.duan
>Priority: Blocker
>
> 2021-11-22 20:36:44,440 INFO
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Using longest
> log: 10.65.57.133:8485=segmentState {
> startTxId: 3906965
> endTxId: 3906965
> isInProgress: false
> }
> lastWriterEpoch: 5
> lastCommittedTxId: 3906964
> 2021-11-22 20:36:44,457 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in /data12/data/flashHadoopU/namenode/current
> 2021-11-22 20:36:44,495 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /data12/data/flashHadoopU/namenode/current/edits_inprogress_3898378
> ->
> /data12/data/flashHadoopU/namenode/current/edits_3898378-3898412
> 2021-11-22 20:36:44,657 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Gap in transactions. Expected to be able to read up
> until at least txid 2510934 but unable to find any edit logs containing txid
> 2510933
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1578)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1536)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:976)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> 2021-11-22 20:36:44,660 INFO org.mortbay.log: Stopped
> HttpServer2$selectchannelconnectorwithsafestar...@pro-hadoop-dc01-057133.vm.dc01.hellocloud.tech:50070
> 2021-11-22 20:36:44,760 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2021-11-22 20:36:44,761 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> Old version: 2.7.3
> New version: 3.2.2
> Steps to Reproduce
> Step 1: Start NN1 as active , NN2 as standby .
> Step 2: Perform "hdfs dfsadmin -rollingUpgrade prepare"
> Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
> option.
> Step 4: DN also restarted in upgrade mode.
> Step 5: Restart journalnode with new hadoop version
> Step 6: a few days later
> Step 7: bring down both NN, journalnode, DN
> Step 8: Start JN with old version
> Step 9: Start NN1 with rolling upgrade rollback option. nn started failed
> with above ERROR(Above mentioned txid version 2510933 has been deleted
> because of checkpoint mechanism)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450297#comment-17450297
]
chuanjie.duan commented on HDFS-16349:
--
Upload Patch，I just remove "+ 2" for esclipe checkForGaps, because we won't
load any editlog
> FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
> -
>
> Key: HDFS-16349
> URL: https://issues.apache.org/jira/browse/HDFS-16349
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.2, 3.2.3
>Reporter: chuanjie.duan
>Priority: Blocker
> Attachments: HDFS-16349-branch-3.2.3.patch
>
>
> 2021-11-22 20:36:44,440 INFO
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Using longest
> log: 10.65.57.133:8485=segmentState {
> startTxId: 3906965
> endTxId: 3906965
> isInProgress: false
> }
> lastWriterEpoch: 5
> lastCommittedTxId: 3906964
> 2021-11-22 20:36:44,457 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in /data12/data/flashHadoopU/namenode/current
> 2021-11-22 20:36:44,495 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /data12/data/flashHadoopU/namenode/current/edits_inprogress_3898378
> ->
> /data12/data/flashHadoopU/namenode/current/edits_3898378-3898412
> 2021-11-22 20:36:44,657 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Gap in transactions. Expected to be able to read up
> until at least txid 2510934 but unable to find any edit logs containing txid
> 2510933
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1578)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1536)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:976)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> 2021-11-22 20:36:44,660 INFO org.mortbay.log: Stopped
> HttpServer2$selectchannelconnectorwithsafestar...@pro-hadoop-dc01-057133.vm.dc01.hellocloud.tech:50070
> 2021-11-22 20:36:44,760 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2021-11-22 20:36:44,761 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> Old version: 2.7.3
> New version: 3.2.2
> Steps to Reproduce
> Step 1: Start NN1 as active , NN2 as standby .
> Step 2: Perform "hdfs dfsadmin -rollingUpgrade prepare"
> Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
> option.
> Step 4: DN also restarted in upgrade mode.
> Step 5: Restart journalnode with new hadoop version
> Step 6: a few days later
> Step 7: bring down both NN, journalnode, DN
> Step 8: Start JN with old version
> Step 9: Start NN1 with rolling upgrade rollback option. nn started failed
> with above ERROR(Above mentioned txid version 2510933 has been deleted
> because of checkpoint mechanism)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chuanjie.duan updated HDFS-16349:
-
Affects Version/s: 3.2.3
> FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
> -
>
> Key: HDFS-16349
> URL: https://issues.apache.org/jira/browse/HDFS-16349
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.2, 3.2.3
>Reporter: chuanjie.duan
>Priority: Blocker
> Attachments: HDFS-16349-branch-3.2.3.patch
>
>
> 2021-11-22 20:36:44,440 INFO
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Using longest
> log: 10.65.57.133:8485=segmentState {
> startTxId: 3906965
> endTxId: 3906965
> isInProgress: false
> }
> lastWriterEpoch: 5
> lastCommittedTxId: 3906964
> 2021-11-22 20:36:44,457 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in /data12/data/flashHadoopU/namenode/current
> 2021-11-22 20:36:44,495 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /data12/data/flashHadoopU/namenode/current/edits_inprogress_3898378
> ->
> /data12/data/flashHadoopU/namenode/current/edits_3898378-3898412
> 2021-11-22 20:36:44,657 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Gap in transactions. Expected to be able to read up
> until at least txid 2510934 but unable to find any edit logs containing txid
> 2510933
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1578)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1536)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:976)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> 2021-11-22 20:36:44,660 INFO org.mortbay.log: Stopped
> HttpServer2$selectchannelconnectorwithsafestar...@pro-hadoop-dc01-057133.vm.dc01.hellocloud.tech:50070
> 2021-11-22 20:36:44,760 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2021-11-22 20:36:44,761 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> Old version: 2.7.3
> New version: 3.2.2
> Steps to Reproduce
> Step 1: Start NN1 as active , NN2 as standby .
> Step 2: Perform "hdfs dfsadmin -rollingUpgrade prepare"
> Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
> option.
> Step 4: DN also restarted in upgrade mode.
> Step 5: Restart journalnode with new hadoop version
> Step 6: a few days later
> Step 7: bring down both NN, journalnode, DN
> Step 8: Start JN with old version
> Step 9: Start NN1 with rolling upgrade rollback option. nn started failed
> with above ERROR(Above mentioned txid version 2510933 has been deleted
> because of checkpoint mechanism)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chuanjie.duan updated HDFS-16349:
-
Attachment: HDFS-16349-branch-3.2.3.patch
> FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
> -
>
> Key: HDFS-16349
> URL: https://issues.apache.org/jira/browse/HDFS-16349
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.2, 3.2.3
>Reporter: chuanjie.duan
>Priority: Blocker
> Attachments: HDFS-16349-branch-3.2.3.patch
>
>
> 2021-11-22 20:36:44,440 INFO
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Using longest
> log: 10.65.57.133:8485=segmentState {
> startTxId: 3906965
> endTxId: 3906965
> isInProgress: false
> }
> lastWriterEpoch: 5
> lastCommittedTxId: 3906964
> 2021-11-22 20:36:44,457 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in /data12/data/flashHadoopU/namenode/current
> 2021-11-22 20:36:44,495 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /data12/data/flashHadoopU/namenode/current/edits_inprogress_3898378
> ->
> /data12/data/flashHadoopU/namenode/current/edits_3898378-3898412
> 2021-11-22 20:36:44,657 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Gap in transactions. Expected to be able to read up
> until at least txid 2510934 but unable to find any edit logs containing txid
> 2510933
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1578)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1536)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:976)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> 2021-11-22 20:36:44,660 INFO org.mortbay.log: Stopped
> HttpServer2$selectchannelconnectorwithsafestar...@pro-hadoop-dc01-057133.vm.dc01.hellocloud.tech:50070
> 2021-11-22 20:36:44,760 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2021-11-22 20:36:44,761 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> Old version: 2.7.3
> New version: 3.2.2
> Steps to Reproduce
> Step 1: Start NN1 as active , NN2 as standby .
> Step 2: Perform "hdfs dfsadmin -rollingUpgrade prepare"
> Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
> option.
> Step 4: DN also restarted in upgrade mode.
> Step 5: Restart journalnode with new hadoop version
> Step 6: a few days later
> Step 7: bring down both NN, journalnode, DN
> Step 8: Start JN with old version
> Step 9: Start NN1 with rolling upgrade rollback option. nn started failed
> with above ERROR(Above mentioned txid version 2510933 has been deleted
> because of checkpoint mechanism)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=687195&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687195 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 29/Nov/21 09:08 Start Date: 29/Nov/21 09:08 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-981423868 @tomscut Oh, I meant please fix the indents of `case DFS_DATANODE_DATA_DIR_KEY` and `case DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_KEY` since they aren't in accordance with the hadoop_idea_formatter. I think we can do it in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687195) Time Spent: 2h 50m (was: 2h 40m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15788) Correct the statement for pmem cache to reflect cache persistence support
[ https://issues.apache.org/jira/browse/HDFS-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450279#comment-17450279 ] Rakesh Radhakrishnan commented on HDFS-15788: - +1 LGTM, thanks [~PhiloHe] for the contribution. > Correct the statement for pmem cache to reflect cache persistence support > - > > Key: HDFS-15788 > URL: https://issues.apache.org/jira/browse/HDFS-15788 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Minor > Attachments: HDFS-15788-01.patch, HDFS-15788-02.patch > > > Correct the statement for pmem cache to reflect cache persistence support. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16344) Improve DirectoryScanner.Stats#toString
[ https://issues.apache.org/jira/browse/HDFS-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16344. - Fix Version/s: 3.4.0 3.3.3 Resolution: Fixed > Improve DirectoryScanner.Stats#toString > --- > > Key: HDFS-16344 > URL: https://issues.apache.org/jira/browse/HDFS-16344 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Attachments: image-2021-11-21-19-35-16-838.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Improve DirectoryScanner.Stats#toString. > !image-2021-11-21-19-35-16-838.png|width=1019,height=71! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16344) Improve DirectoryScanner.Stats#toString
[ https://issues.apache.org/jira/browse/HDFS-16344?focusedWorklogId=687182&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687182 ] ASF GitHub Bot logged work on HDFS-16344: - Author: ASF GitHub Bot Created on: 29/Nov/21 08:48 Start Date: 29/Nov/21 08:48 Worklog Time Spent: 10m Work Description: tasanuma merged pull request #3695: URL: https://github.com/apache/hadoop/pull/3695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687182) Time Spent: 1.5h (was: 1h 20m) > Improve DirectoryScanner.Stats#toString > --- > > Key: HDFS-16344 > URL: https://issues.apache.org/jira/browse/HDFS-16344 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-21-19-35-16-838.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > Improve DirectoryScanner.Stats#toString. > !image-2021-11-21-19-35-16-838.png|width=1019,height=71! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16344) Improve DirectoryScanner.Stats#toString
[ https://issues.apache.org/jira/browse/HDFS-16344?focusedWorklogId=687184&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-687184 ] ASF GitHub Bot logged work on HDFS-16344: - Author: ASF GitHub Bot Created on: 29/Nov/21 08:48 Start Date: 29/Nov/21 08:48 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3695: URL: https://github.com/apache/hadoop/pull/3695#issuecomment-981408618 Thanks for your contribution, @tomscut. Thanks for your reviews, @virajjasani, @ayushtkn, @ferhui. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 687184) Time Spent: 1h 40m (was: 1.5h) > Improve DirectoryScanner.Stats#toString > --- > > Key: HDFS-16344 > URL: https://issues.apache.org/jira/browse/HDFS-16344 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-21-19-35-16-838.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Improve DirectoryScanner.Stats#toString. > !image-2021-11-21-19-35-16-838.png|width=1019,height=71! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15068) DataNode could meet deadlock if invoke refreshVolumes when register
[
https://issues.apache.org/jira/browse/HDFS-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lei w reassigned HDFS-15068:
Assignee: Aiphago (was: lei w)
> DataNode could meet deadlock if invoke refreshVolumes when register
> ---
>
> Key: HDFS-15068
> URL: https://issues.apache.org/jira/browse/HDFS-15068
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: Aiphago
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-15068.001.patch, HDFS-15068.002.patch,
> HDFS-15068.003.patch, HDFS-15068.004.patch, HDFS-15068.005.patch
>
>
> DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host
> start` to trigger #refreshVolumes.
> 1. DataNod#refreshVolumes hold datanode instance ownable {{synchronizer}}
> when enter this method first, then try to hold BPOfferService {{readlock}}
> when `bpos.getNamespaceInfo()` in following code segment.
> {code:java}
> for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
> nsInfos.add(bpos.getNamespaceInfo());
> }
> {code}
> 2. BPOfferService#registrationSucceeded (which is invoked by #register when
> DataNode start or #reregister when processCommandFromActor) hold
> BPOfferService {{writelock}} first, then try to hold datanode instance
> ownable {{synchronizer}} in following method.
> {code:java}
> synchronized void bpRegistrationSucceeded(DatanodeRegistration
> bpRegistration,
> String blockPoolId) throws IOException {
> id = bpRegistration;
> if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
> throw new IOException("Inconsistent Datanode IDs. Name-node returned "
> + bpRegistration.getDatanodeUuid()
> + ". Expecting " + storage.getDatanodeUuid());
> }
>
> registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15068) DataNode could meet deadlock if invoke refreshVolumes when register
[
https://issues.apache.org/jira/browse/HDFS-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lei w reassigned HDFS-15068:
Assignee: lei w (was: Aiphago)
> DataNode could meet deadlock if invoke refreshVolumes when register
> ---
>
> Key: HDFS-15068
> URL: https://issues.apache.org/jira/browse/HDFS-15068
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Xiaoqiao He
>Assignee: lei w
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-15068.001.patch, HDFS-15068.002.patch,
> HDFS-15068.003.patch, HDFS-15068.004.patch, HDFS-15068.005.patch
>
>
> DataNode could meet deadlock when invoke `dfsadmin -reconfig datanode ip:host
> start` to trigger #refreshVolumes.
> 1. DataNod#refreshVolumes hold datanode instance ownable {{synchronizer}}
> when enter this method first, then try to hold BPOfferService {{readlock}}
> when `bpos.getNamespaceInfo()` in following code segment.
> {code:java}
> for (BPOfferService bpos : blockPoolManager.getAllNamenodeThreads()) {
> nsInfos.add(bpos.getNamespaceInfo());
> }
> {code}
> 2. BPOfferService#registrationSucceeded (which is invoked by #register when
> DataNode start or #reregister when processCommandFromActor) hold
> BPOfferService {{writelock}} first, then try to hold datanode instance
> ownable {{synchronizer}} in following method.
> {code:java}
> synchronized void bpRegistrationSucceeded(DatanodeRegistration
> bpRegistration,
> String blockPoolId) throws IOException {
> id = bpRegistration;
> if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid())) {
> throw new IOException("Inconsistent Datanode IDs. Name-node returned "
> + bpRegistration.getDatanodeUuid()
> + ". Expecting " + storage.getDatanodeUuid());
> }
>
> registerBlockPoolWithSecretManager(bpRegistration, blockPoolId);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
