[jira] [Work logged] (HDFS-16203) Discover datanodes with unbalanced block pool usage by the standard deviation
[
https://issues.apache.org/jira/browse/HDFS-16203?focusedWorklogId=649031&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649031
]
ASF GitHub Bot logged work on HDFS-16203:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 05:20
Start Date: 10/Sep/21 05:20
Worklog Time Spent: 10m
Work Description: tomscut edited a comment on pull request #3366:
URL: https://github.com/apache/hadoop/pull/3366#issuecomment-916639192
@aajisaka @ayushtkn @ferhui @Hexiaoqiao Could you please review the code?
Thanks a lot.
The standard deviation is calculated by referring to
FSNamesystem#getNodeUsage. The key change is
FSNamesystem#getBlockPoolUsedPercentStdDev, :
```
public float getBlockPoolUsedPercentStdDev(StorageReport[] storageReports)
{
ArrayList usagePercentList = new ArrayList<>();
float totalUsagePercent = 0.0f;
float dev = 0.0f;
if (storageReports.length == 0) {
return dev;
}
for (StorageReport s : storageReports) {
usagePercentList.add(s.getBlockPoolUsagePercent());
totalUsagePercent += s.getBlockPoolUsagePercent();
}
totalUsagePercent /= storageReports.length;
Collections.sort(usagePercentList);
for (Float usagePercent : usagePercentList) {
dev += (usagePercent - totalUsagePercent)
* (usagePercent - totalUsagePercent);
}
dev = (float) Math.sqrt(dev / usagePercentList.size());
return dev;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649031)
Time Spent: 1h 40m (was: 1.5h)
> Discover datanodes with unbalanced block pool usage by the standard deviation
> -
>
> Key: HDFS-16203
> URL: https://issues.apache.org/jira/browse/HDFS-16203
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-09-01-19-16-27-172.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> *Discover datanodes with unbalanced volume usage by the standard deviation.*
> *In some scenarios, we may cause unbalanced datanode disk usage:*
> 1. Repair the damaged disk and make it online again.
> 2. Add disks to some Datanodes.
> 3. Some disks are damaged, resulting in slow data writing.
> 4. Use some custom volume choosing policies.
> In the case of unbalanced disk usage, a sudden increase in datanode write
> traffic may result in busy disk I/O with low volume usage, resulting in
> decreased throughput across datanodes.
> We need to find these nodes in time to do diskBalance, or other processing.
> Based on the volume usage of each datanode, we can calculate the standard
> deviation of the volume usage. The more unbalanced the volume, the higher the
> standard deviation.
> *We can display the result on the Web of namenode, and then sorting directly
> to find the nodes where the volumes usages are unbalanced.*
> *{color:#172b4d}This interface is only used to obtain metrics and does not
> adversely affect namenode performance.{color}*
>
> {color:#172b4d}!image-2021-09-01-19-16-27-172.png|width=581,height=216!{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16203) Discover datanodes with unbalanced block pool usage by the standard deviation
[
https://issues.apache.org/jira/browse/HDFS-16203?focusedWorklogId=649030&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649030
]
ASF GitHub Bot logged work on HDFS-16203:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 05:19
Start Date: 10/Sep/21 05:19
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3366:
URL: https://github.com/apache/hadoop/pull/3366#issuecomment-916639192
@ayushtkn @ferhui @Hexiaoqiao Could you please review the code? Thanks a lot.
The standard deviation is calculated by referring to
FSNamesystem#getNodeUsage. The key change is
FSNamesystem#getBlockPoolUsedPercentStdDev, :
```
public float getBlockPoolUsedPercentStdDev(StorageReport[] storageReports)
{
ArrayList usagePercentList = new ArrayList<>();
float totalUsagePercent = 0.0f;
float dev = 0.0f;
if (storageReports.length == 0) {
return dev;
}
for (StorageReport s : storageReports) {
usagePercentList.add(s.getBlockPoolUsagePercent());
totalUsagePercent += s.getBlockPoolUsagePercent();
}
totalUsagePercent /= storageReports.length;
Collections.sort(usagePercentList);
for (Float usagePercent : usagePercentList) {
dev += (usagePercent - totalUsagePercent)
* (usagePercent - totalUsagePercent);
}
dev = (float) Math.sqrt(dev / usagePercentList.size());
return dev;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649030)
Time Spent: 1.5h (was: 1h 20m)
> Discover datanodes with unbalanced block pool usage by the standard deviation
> -
>
> Key: HDFS-16203
> URL: https://issues.apache.org/jira/browse/HDFS-16203
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-09-01-19-16-27-172.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> *Discover datanodes with unbalanced volume usage by the standard deviation.*
> *In some scenarios, we may cause unbalanced datanode disk usage:*
> 1. Repair the damaged disk and make it online again.
> 2. Add disks to some Datanodes.
> 3. Some disks are damaged, resulting in slow data writing.
> 4. Use some custom volume choosing policies.
> In the case of unbalanced disk usage, a sudden increase in datanode write
> traffic may result in busy disk I/O with low volume usage, resulting in
> decreased throughput across datanodes.
> We need to find these nodes in time to do diskBalance, or other processing.
> Based on the volume usage of each datanode, we can calculate the standard
> deviation of the volume usage. The more unbalanced the volume, the higher the
> standard deviation.
> *We can display the result on the Web of namenode, and then sorting directly
> to find the nodes where the volumes usages are unbalanced.*
> *{color:#172b4d}This interface is only used to obtain metrics and does not
> adversely affect namenode performance.{color}*
>
> {color:#172b4d}!image-2021-09-01-19-16-27-172.png|width=581,height=216!{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16219) RBF: Set default map tasks and bandwidth in RouterFederationRename
Akira Ajisaka created HDFS-16219: Summary: RBF: Set default map tasks and bandwidth in RouterFederationRename Key: HDFS-16219 URL: https://issues.apache.org/jira/browse/HDFS-16219 Project: Hadoop HDFS Issue Type: Sub-task Components: rbf Environment: Hadoop 3.3.0 with patches Reporter: Akira Ajisaka If dfs.federation.router.federation.rename.map or dfs.federation.router.federation.rename.bandwidth is not set, DFSRouter fails to launch. This issue is similar to HDFS-16217. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16218) RBF: RouterFedbalance should load HDFS config
[
https://issues.apache.org/jira/browse/HDFS-16218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412957#comment-17412957
]
Akira Ajisaka commented on HDFS-16218:
--
In
https://github.com/apache/hadoop/blob/77299ae992b16066dd61e4fec9ff63b863ae2e21/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/rbfbalance/RouterFedBalance.java#L359,
we should use HdfsConfiguration instead of Configuration.
> RBF: RouterFedbalance should load HDFS config
> -
>
> Key: HDFS-16218
> URL: https://issues.apache.org/jira/browse/HDFS-16218
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Environment: Hadoop 3.3.0 + patches, Kerberos authentication is
> enabled
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: newbie
>
> RouterFedBalance fails to connect to DFSRouter when Kerberos is enabled
> because "dfs.federation.router.kerberos.principal" in hdfs-site.xml is not
> loaded.
> {quote}
> 21/09/08 17:21:38 ERROR rbfbalance.RouterFedBalance: Submit balance job
> failed.
> java.io.IOException: DestHost:destPort 0.0.0.0:8111 , LocalHost:localPort
> /:0. Failed on local exception: java.io.IOException: Couldn't set
> up IO streams: java.lang.IllegalArgumentException: Failed to specify server's
> Kerberos principal name
> at
> org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocolTranslatorPB.getMountTableEntries(RouterAdminProtocolTranslatorPB.java:198)
> at
> org.apache.hadoop.hdfs.rbfbalance.MountTableProcedure.getMountEntry(MountTableProcedure.java:140)
> at
> org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.getSrcPath(RouterFedBalance.java:326)
> at
> org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.access$000(RouterFedBalance.java:68)
> at
> org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance$Builder.build(RouterFedBalance.java:168)
> at
> org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.submit(RouterFedBalance.java:302)
> at
> org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.run(RouterFedBalance.java:216)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.main(RouterFedBalance.java:376)
> {quote}
> When adding the property specifically by "-D" option, the command worked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16218) RBF: RouterFedbalance should load HDFS config
Akira Ajisaka created HDFS-16218:
Summary: RBF: RouterFedbalance should load HDFS config
Key: HDFS-16218
URL: https://issues.apache.org/jira/browse/HDFS-16218
Project: Hadoop HDFS
Issue Type: Sub-task
Components: rbf
Environment: Hadoop 3.3.0 + patches, Kerberos authentication is enabled
Reporter: Akira Ajisaka
RouterFedBalance fails to connect to DFSRouter when Kerberos is enabled because
"dfs.federation.router.kerberos.principal" in hdfs-site.xml is not loaded.
{quote}
21/09/08 17:21:38 ERROR rbfbalance.RouterFedBalance: Submit balance job failed.
java.io.IOException: DestHost:destPort 0.0.0.0:8111 , LocalHost:localPort
/:0. Failed on local exception: java.io.IOException: Couldn't set up
IO streams: java.lang.IllegalArgumentException: Failed to specify server's
Kerberos principal name
at
org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocolTranslatorPB.getMountTableEntries(RouterAdminProtocolTranslatorPB.java:198)
at
org.apache.hadoop.hdfs.rbfbalance.MountTableProcedure.getMountEntry(MountTableProcedure.java:140)
at
org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.getSrcPath(RouterFedBalance.java:326)
at
org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.access$000(RouterFedBalance.java:68)
at
org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance$Builder.build(RouterFedBalance.java:168)
at
org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.submit(RouterFedBalance.java:302)
at
org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.run(RouterFedBalance.java:216)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.hdfs.rbfbalance.RouterFedBalance.main(RouterFedBalance.java:376)
{quote}
When adding the property specifically by "-D" option, the command worked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16217) RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri
Akira Ajisaka created HDFS-16217:
Summary: RBF: Set default value of
hdfs.fedbalance.procedure.scheduler.journal.uri
Key: HDFS-16217
URL: https://issues.apache.org/jira/browse/HDFS-16217
Project: Hadoop HDFS
Issue Type: Sub-task
Components: rbf
Environment: Hadoop 3.3.0 with patches
Reporter: Akira Ajisaka
When dfs.federation.router.federation.rename.option is set to DISTCP and
hdfs.fedbalance.procedure.scheduler.journal.uri is not set, DFSRouter fails to
launch.
{quote}
2021-09-08 15:39:11,818 ERROR
org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
router
java.lang.NullPointerException
at java.base/java.net.URI$Parser.parse(URI.java:3104)
at java.base/java.net.URI.(URI.java:600)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.initRouterFedRename(RouterRpcServer.java:444)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.(RouterRpcServer.java:419)
at
org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
at
org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
{quote}
hdfs.fedbalance.procedure.scheduler.journal.uri is
hdfs://localhost:8020/tmp/procedure by default, however, the default value is
not used in DFSRouter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=649027&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649027
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 04:42
Start Date: 10/Sep/21 04:42
Worklog Time Spent: 10m
Work Description: bshashikant commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916626429
> > @bshashikant , thanks! In that case, let's please just add the comments
I requested to `INodeDirectory` and `INodeDirectoryAttributes`. Then I'll
approve and merge.
>
> Thanks @cnauroth . As the problem is very specific to snapshot root,
"metedataEquals()" function is now overriden in Snapshot.Root() and added the
comments as per your suggestion
@szetszwo , can you have another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649027)
Time Spent: 2h 50m (was: 2h 40m)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=649026&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649026
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 04:41
Start Date: 10/Sep/21 04:41
Worklog Time Spent: 10m
Work Description: bshashikant commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916626334
> @bshashikant , thanks! In that case, let's please just add the comments I
requested to `INodeDirectory` and `INodeDirectoryAttributes`. Then I'll approve
and merge.
Thanks @cnauroth . As the problem is very specific to snapshot root,
"metedataEquals()" function is now overriden in Snapshot.Root() and added the
comments as per your suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649026)
Time Spent: 2h 40m (was: 2.5h)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16216) RBF: Wrong path when get mount point status
[
https://issues.apache.org/jira/browse/HDFS-16216?focusedWorklogId=649019&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649019
]
ASF GitHub Bot logged work on HDFS-16216:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 04:07
Start Date: 10/Sep/21 04:07
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3415:
URL: https://github.com/apache/hadoop/pull/3415#issuecomment-916613934
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 2
unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 29s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 104m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3415 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 06a972f6a1f4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 697004f46d57eb90cfb0d32f1ca35ef9e64e26fa |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3415/1/testReport/ |
| Max. process+thread count | 2734 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=649016&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649016
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 03:53
Start Date: 10/Sep/21 03:53
Worklog Time Spent: 10m
Work Description: cnauroth commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916609296
@bshashikant , thanks! In that case, let's please just add the comments I
requested to `INodeDirectory` and `INodeDirectoryAttributes`. Then I'll
approve and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649016)
Time Spent: 2.5h (was: 2h 20m)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=649010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649010
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 03:23
Start Date: 10/Sep/21 03:23
Worklog Time Spent: 10m
Work Description: bshashikant edited a comment on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916599655
Thanks @cnauroth . The problem here seems to be very specific to
snapshotRoot.
`
@Test
public void testXattrOnFileWithSnapshotAndNNRestart() throws Exception {
FileSystem.mkdirs(hdfs, path, FsPermission.createImmutable((short)
0700));
Path file = new Path(path, "file1");
FileSystem.create(hdfs, file, FsPermission.createImmutable((short)
0));
hdfs.setXAttr(file, name1, value1);
hdfs.allowSnapshot(path);
hdfs.createSnapshot(path, snapshotName);
SnapshotDiffReport report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
hdfs.saveNamespace();
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
}`
The same test with xattrs set on file inside the snapshotRoot/SubDirecory
will just work with/without the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649010)
Time Spent: 2h 20m (was: 2h 10m)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=649008&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649008
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 03:19
Start Date: 10/Sep/21 03:19
Worklog Time Spent: 10m
Work Description: bshashikant edited a comment on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916599655
Thanks @cnauroth . The problem here is very specific to snapshotRoot.
`@Test
public void testXattrOnFileWithSnapshotAndNNRestart() throws Exception {
// Init
FileSystem.mkdirs(hdfs, path, FsPermission.createImmutable((short)
0700));
Path file = new Path(path, "file1");
FileSystem.create(hdfs, file, FsPermission.createImmutable((short)
0));
hdfs.setXAttr(file, name1, value1);
hdfs.allowSnapshot(path);
hdfs.createSnapshot(path, snapshotName);
SnapshotDiffReport report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
hdfs.saveNamespace();
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
}`
The same test with xattrs set on file inside the snapshotRoot/SubDirecory
will just work with/without the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649008)
Time Spent: 2h 10m (was: 2h)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=649006&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649006
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 03:19
Start Date: 10/Sep/21 03:19
Worklog Time Spent: 10m
Work Description: bshashikant commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-916599655
Thanks @cnauroth . The problem here is very specific to snapshotRoot.
`@Test
public void testXattrOnFileWithSnapshotAndNNRestart() throws Exception {
// Init
FileSystem.mkdirs(hdfs, path, FsPermission.createImmutable((short)
0700));
Path file = new Path(path, "file1");
FileSystem.create(hdfs, file, FsPermission.createImmutable((short)
0));
hdfs.setXAttr(file, name1, value1);
hdfs.allowSnapshot(path);
hdfs.createSnapshot(path, snapshotName);
SnapshotDiffReport report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
hdfs.saveNamespace();
hdfs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
hdfs.getSnapshotDiffReport(path, snapshotName, "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
}`
The same test with xattrs set on file inside the snapshotRoot/SubDirecory
will just work with/without the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 649006)
Time Spent: 2h (was: 1h 50m)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15160) ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock
[ https://issues.apache.org/jira/browse/HDFS-15160?focusedWorklogId=649004&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649004 ] ASF GitHub Bot logged work on HDFS-15160: - Author: ASF GitHub Bot Created on: 10/Sep/21 03:11 Start Date: 10/Sep/21 03:11 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on pull request #3200: URL: https://github.com/apache/hadoop/pull/3200#issuecomment-916596838 Will commit to branch-3.2 for a short while if no more comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 649004) Time Spent: 5h 40m (was: 5.5h) > ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl > methods should use datanode readlock > --- > > Key: HDFS-15160 > URL: https://issues.apache.org/jira/browse/HDFS-15160 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15160-branch-3.3-001.patch, HDFS-15160.001.patch, > HDFS-15160.002.patch, HDFS-15160.003.patch, HDFS-15160.004.patch, > HDFS-15160.005.patch, HDFS-15160.006.patch, HDFS-15160.007.patch, > HDFS-15160.008.patch, HDFS-15160.branch-3-3.001.patch, > image-2020-04-10-17-18-08-128.png, image-2020-04-10-17-18-55-938.png > > Time Spent: 5h 40m > Remaining Estimate: 0h > > Now we have HDFS-15150, we can start to move some DN operations to use the > read lock rather than the write lock to improve concurrence. The first step > is to make the changes to ReplicaMap, as many other methods make calls to it. > This Jira switches read operations against the volume map to use the readLock > rather than the write lock. > Additionally, some methods make a call to replicaMap.replicas() (eg > getBlockReports, getFinalizedBlocks, deepCopyReplica) and only use the result > in a read only fashion, so they can also be switched to using a readLock. > Next is the directory scanner and disk balancer, which only require a read > lock. > Finally (for this Jira) are various "low hanging fruit" items in BlockSender > and fsdatasetImpl where is it fairly obvious they only need a read lock. > For now, I have avoided changing anything which looks too risky, as I think > its better to do any larger refactoring or risky changes each in their own > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16197) Simplify getting NNStorage in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16197?focusedWorklogId=649001&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-649001 ] ASF GitHub Bot logged work on HDFS-16197: - Author: ASF GitHub Bot Created on: 10/Sep/21 03:05 Start Date: 10/Sep/21 03:05 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3357: URL: https://github.com/apache/hadoop/pull/3357#issuecomment-916594491 @ferhui @virajjasani , can you help me check the code? If there is something to be corrected, I will continue to work. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 649001) Time Spent: 2.5h (was: 2h 20m) > Simplify getting NNStorage in FSNamesystem > -- > > Key: HDFS-16197 > URL: https://issues.apache.org/jira/browse/HDFS-16197 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > In FSNamesystem, there are many places where NNStorage needs to be used > (according to preliminary statistics, there are 15 times), and these places > are obtained using "getFSImage().getStorage()". We should try to use a > simpler way. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16210) RBF: Add the option of refreshCallQueue to RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412919#comment-17412919 ] Janus Chow commented on HDFS-16210: --- [~ferhui] Thanks for the help. Do I need to create a new ticket to backport to branch-3.2? > RBF: Add the option of refreshCallQueue to RouterAdmin > -- > > Key: HDFS-16210 > URL: https://issues.apache.org/jira/browse/HDFS-16210 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We enabled FairCallQueue to RouterRpcServer, but Router can not > refreshCallQueue as NameNode does. > This ticket is to enable the refreshCallQueue for Router so that we don't > have to restart the Routers when updating FairCallQueue configurations. > > The option is not to refreshCallQueue to NameNodes, just trying to refresh > the callQueue of Router itself. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16210) RBF: Add the option of refreshCallQueue to RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HDFS-16210. Resolution: Fixed > RBF: Add the option of refreshCallQueue to RouterAdmin > -- > > Key: HDFS-16210 > URL: https://issues.apache.org/jira/browse/HDFS-16210 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We enabled FairCallQueue to RouterRpcServer, but Router can not > refreshCallQueue as NameNode does. > This ticket is to enable the refreshCallQueue for Router so that we don't > have to restart the Routers when updating FairCallQueue configurations. > > The option is not to refreshCallQueue to NameNodes, just trying to refresh > the callQueue of Router itself. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16210) RBF: Add the option of refreshCallQueue to RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412917#comment-17412917 ] Hui Fei commented on HDFS-16210: Just cherry-pick to branch-3.3, branch-3.2 i s more beihnd. > RBF: Add the option of refreshCallQueue to RouterAdmin > -- > > Key: HDFS-16210 > URL: https://issues.apache.org/jira/browse/HDFS-16210 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We enabled FairCallQueue to RouterRpcServer, but Router can not > refreshCallQueue as NameNode does. > This ticket is to enable the refreshCallQueue for Router so that we don't > have to restart the Routers when updating FairCallQueue configurations. > > The option is not to refreshCallQueue to NameNodes, just trying to refresh > the callQueue of Router itself. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16210) RBF: Add the option of refreshCallQueue to RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei updated HDFS-16210: --- Fix Version/s: 3.3.2 > RBF: Add the option of refreshCallQueue to RouterAdmin > -- > > Key: HDFS-16210 > URL: https://issues.apache.org/jira/browse/HDFS-16210 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We enabled FairCallQueue to RouterRpcServer, but Router can not > refreshCallQueue as NameNode does. > This ticket is to enable the refreshCallQueue for Router so that we don't > have to restart the Routers when updating FairCallQueue configurations. > > The option is not to refreshCallQueue to NameNodes, just trying to refresh > the callQueue of Router itself. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16216) RBF: Wrong path when get mount point status
[
https://issues.apache.org/jira/browse/HDFS-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16216:
--
Labels: pull-request-available (was: )
> RBF: Wrong path when get mount point status
> ---
>
> Key: HDFS-16216
> URL: https://issues.apache.org/jira/browse/HDFS-16216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we get mount point status, we will get wrong path in file status.
> {code:java}
> // prepare mount env
> hdfs dfsrouteradmin -add /test ns1 /test
> hdfs dfsrouteradmin -add /test/test1/test2/test3 ns1 /test/test1/test2/test3
> {code}
> {code:java}
> // java code
> public class TmpOp {
> public static void main(String[] args) throws Exception {
> Configuration conf = new HdfsConfiguration();
> FileSystem fs = FileSystem.get(conf);
> FileStatus test = fs.getFileStatus(new Path("/test"));
> FileStatus test1 = fs.getFileStatus(new Path("/test/test1"));
> FileStatus test2 = fs.getFileStatus(new Path("/test/test1/test2"));
> System.out.println(test.getPath());
> System.out.println(test1.getPath());
> System.out.println(test2.getPath());
> }
> }
> {code}
>
> {code:java}
> // result
> hdfs://ns1/test/test
> hdfs://ns1/test/test1/test1
> hdfs://ns1/test/test1/test2/test2
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16216) RBF: Wrong path when get mount point status
[
https://issues.apache.org/jira/browse/HDFS-16216?focusedWorklogId=648985&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648985
]
ASF GitHub Bot logged work on HDFS-16216:
-
Author: ASF GitHub Bot
Created on: 10/Sep/21 02:20
Start Date: 10/Sep/21 02:20
Worklog Time Spent: 10m
Work Description: zhengzhuobinzzb opened a new pull request #3415:
URL: https://github.com/apache/hadoop/pull/3415
### Description of PR
https://issues.apache.org/jira/browse/HDFS-16216
### How was this patch tested?
UT. TestRouterMountTable.testGetMountPointFileStatus
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648985)
Remaining Estimate: 0h
Time Spent: 10m
> RBF: Wrong path when get mount point status
> ---
>
> Key: HDFS-16216
> URL: https://issues.apache.org/jira/browse/HDFS-16216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we get mount point status, we will get wrong path in file status.
> {code:java}
> // prepare mount env
> hdfs dfsrouteradmin -add /test ns1 /test
> hdfs dfsrouteradmin -add /test/test1/test2/test3 ns1 /test/test1/test2/test3
> {code}
> {code:java}
> // java code
> public class TmpOp {
> public static void main(String[] args) throws Exception {
> Configuration conf = new HdfsConfiguration();
> FileSystem fs = FileSystem.get(conf);
> FileStatus test = fs.getFileStatus(new Path("/test"));
> FileStatus test1 = fs.getFileStatus(new Path("/test/test1"));
> FileStatus test2 = fs.getFileStatus(new Path("/test/test1/test2"));
> System.out.println(test.getPath());
> System.out.println(test1.getPath());
> System.out.println(test2.getPath());
> }
> }
> {code}
>
> {code:java}
> // result
> hdfs://ns1/test/test
> hdfs://ns1/test/test1/test1
> hdfs://ns1/test/test1/test2/test2
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16216) RBF: Wrong path when get mount point status
[
https://issues.apache.org/jira/browse/HDFS-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuobin zheng reassigned HDFS-16216:
Assignee: zhuobin zheng
> RBF: Wrong path when get mount point status
> ---
>
> Key: HDFS-16216
> URL: https://issues.apache.org/jira/browse/HDFS-16216
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Major
>
> When we get mount point status, we will get wrong path in file status.
> {code:java}
> // prepare mount env
> hdfs dfsrouteradmin -add /test ns1 /test
> hdfs dfsrouteradmin -add /test/test1/test2/test3 ns1 /test/test1/test2/test3
> {code}
> {code:java}
> // java code
> public class TmpOp {
> public static void main(String[] args) throws Exception {
> Configuration conf = new HdfsConfiguration();
> FileSystem fs = FileSystem.get(conf);
> FileStatus test = fs.getFileStatus(new Path("/test"));
> FileStatus test1 = fs.getFileStatus(new Path("/test/test1"));
> FileStatus test2 = fs.getFileStatus(new Path("/test/test1/test2"));
> System.out.println(test.getPath());
> System.out.println(test1.getPath());
> System.out.println(test2.getPath());
> }
> }
> {code}
>
> {code:java}
> // result
> hdfs://ns1/test/test
> hdfs://ns1/test/test1/test1
> hdfs://ns1/test/test1/test2/test2
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16216) RBF: Wrong path when get mount point status
zhuobin zheng created HDFS-16216:
Summary: RBF: Wrong path when get mount point status
Key: HDFS-16216
URL: https://issues.apache.org/jira/browse/HDFS-16216
Project: Hadoop HDFS
Issue Type: Bug
Reporter: zhuobin zheng
When we get mount point status, we will get wrong path in file status.
{code:java}
// prepare mount env
hdfs dfsrouteradmin -add /test ns1 /test
hdfs dfsrouteradmin -add /test/test1/test2/test3 ns1 /test/test1/test2/test3
{code}
{code:java}
// java code
public class TmpOp {
public static void main(String[] args) throws Exception {
Configuration conf = new HdfsConfiguration();
FileSystem fs = FileSystem.get(conf);
FileStatus test = fs.getFileStatus(new Path("/test"));
FileStatus test1 = fs.getFileStatus(new Path("/test/test1"));
FileStatus test2 = fs.getFileStatus(new Path("/test/test1/test2"));
System.out.println(test.getPath());
System.out.println(test1.getPath());
System.out.println(test2.getPath());
}
}
{code}
{code:java}
// result
hdfs://ns1/test/test
hdfs://ns1/test/test1/test1
hdfs://ns1/test/test1/test2/test2
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412908#comment-17412908 ] JiangHua Zhu commented on HDFS-14703: - Thanks [~prasad-acit] for sharing. Yes, I have browsed through the design documents, which is very good. I think INodeMap#NAMESPACE_KEY_DEPTH should be configurable, which is conducive to the management of the cluster. (If necessary, I can create a jira) By the way, in our cluster, there are more than 100 million INodes. So I put forward this idea. > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16207) Remove NN logs stack trace for non-existent xattr query
[
https://issues.apache.org/jira/browse/HDFS-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-16207:
-
Fix Version/s: 3.2.4
3.3.2
2.10.2
3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~ahussein], thank you for the additional patch to backport to branch-2.10. I
just committed it.
> Remove NN logs stack trace for non-existent xattr query
> ---
>
> Key: HDFS-16207
> URL: https://issues.apache.org/jira/browse/HDFS-16207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>
> Attachments: HDFS-16207-branch-2.10.001.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The NN logs a full stack trace every time a getXAttrs is called for a
> non-existent xattr. The logging has zero value add. The increased logging
> load may harm performance. Something is now probing for xattrs resulting in
> many lines of:
> {code:bash}
> 2021-09-02 13:48:03,340 [IPC Server handler 5 on default port 59951] INFO
> ipc.Server (Server.java:logException(3149)) - IPC Server handler 5 on default
> port 59951, call Call#17 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getXAttrs from 127.0.0.1:59961
> java.io.IOException: At least one of the attributes provided was not found.
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirXAttrOp.getXAttrs(FSDirXAttrOp.java:134)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getXAttrs(FSNamesystem.java:8472)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getXAttrs(NameNodeRpcServer.java:2317)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getXAttrs(ClientNamenodeProtocolServerSideTranslatorPB.java:1745)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1155)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3088)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412801#comment-17412801 ] Renukaprasad C commented on HDFS-14703: --- Thanks [~jianghuazhu] for sharing your thoughts. Hope this will clarify your doubts. INodeMap#NAMESPACE_KEY_DEPTH is desighed with flexibility. Yes, by default it is 2 which is cobmination of (ParentINodeId, INodeId). When you set it to 3, then GrandParentId as well. We have tried upto level 3 with basic functionality. But performance not measured. We continued to use with the default value - 2. I am not sure of any use case to increase the values to higher number (Atleast i havent done any testing on this part). By default each partition capacity is 117965 (65536 * 1.8), we continue to use the default values in our test. We also checked the scenarios when dynamic partitions were added. No perf degrade on dynamic partitions, infact this is expected to give higher throuput. We havent noticed very high CPU usage upto 1M file write Ops (Resouce usage statistics we need to capture yet with base & FGL Patch), so this shouldnt have any impact of the other operations (RPC or any other server side processing tasks). In case if you have missed the design please go through the latest desigh doc - NameNode Fine-Grained Locking.pdf [~shv] [~xinglin] Would you like to share your inputs? > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16207) Remove NN logs stack trace for non-existent xattr query
[
https://issues.apache.org/jira/browse/HDFS-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412705#comment-17412705
]
Hadoop QA commented on HDFS-16207:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
41s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
25s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green}{color} | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} branch-2.10 passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 10m
1s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:red}-1{color} | {color:red} spotbugs {color} | {color:red} 2m
24s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/713/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html{color}
| {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 1 extant
spotbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-
[jira] [Updated] (HDFS-16207) Remove NN logs stack trace for non-existent xattr query
[
https://issues.apache.org/jira/browse/HDFS-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated HDFS-16207:
-
Attachment: HDFS-16207-branch-2.10.001.patch
Status: Patch Available (was: In Progress)
Submitted HDFS-16207-branch-2.10.001.patch
*test-patch*
{code:bash}
dev-support/bin/test-patch \
--plugins="maven,checkstyle,javac,javadoc" \
--test-parallel=true \
--branch=branch-2.10 HDFS-16207-branch-2.10.001.patch
{code}
*unit-tests*
{code:bash}
mvn test
-Dtest=TestDFSShell,TestEncryptionZones,TestFileContextXAttr,TestNameNodeXAttr,TestWebHDFSXAttr
{code}
> Remove NN logs stack trace for non-existent xattr query
> ---
>
> Key: HDFS-16207
> URL: https://issues.apache.org/jira/browse/HDFS-16207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16207-branch-2.10.001.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The NN logs a full stack trace every time a getXAttrs is called for a
> non-existent xattr. The logging has zero value add. The increased logging
> load may harm performance. Something is now probing for xattrs resulting in
> many lines of:
> {code:bash}
> 2021-09-02 13:48:03,340 [IPC Server handler 5 on default port 59951] INFO
> ipc.Server (Server.java:logException(3149)) - IPC Server handler 5 on default
> port 59951, call Call#17 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getXAttrs from 127.0.0.1:59961
> java.io.IOException: At least one of the attributes provided was not found.
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirXAttrOp.getXAttrs(FSDirXAttrOp.java:134)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getXAttrs(FSNamesystem.java:8472)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getXAttrs(NameNodeRpcServer.java:2317)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getXAttrs(ClientNamenodeProtocolServerSideTranslatorPB.java:1745)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1155)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3088)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15160) ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock
[ https://issues.apache.org/jira/browse/HDFS-15160?focusedWorklogId=648613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648613 ] ASF GitHub Bot logged work on HDFS-15160: - Author: ASF GitHub Bot Created on: 09/Sep/21 13:47 Start Date: 09/Sep/21 13:47 Worklog Time Spent: 10m Work Description: amahussein commented on pull request #3200: URL: https://github.com/apache/hadoop/pull/3200#issuecomment-916113850 Hi @Hexiaoqiao, Is this PR ready for merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648613) Time Spent: 5.5h (was: 5h 20m) > ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl > methods should use datanode readlock > --- > > Key: HDFS-15160 > URL: https://issues.apache.org/jira/browse/HDFS-15160 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.3.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15160-branch-3.3-001.patch, HDFS-15160.001.patch, > HDFS-15160.002.patch, HDFS-15160.003.patch, HDFS-15160.004.patch, > HDFS-15160.005.patch, HDFS-15160.006.patch, HDFS-15160.007.patch, > HDFS-15160.008.patch, HDFS-15160.branch-3-3.001.patch, > image-2020-04-10-17-18-08-128.png, image-2020-04-10-17-18-55-938.png > > Time Spent: 5.5h > Remaining Estimate: 0h > > Now we have HDFS-15150, we can start to move some DN operations to use the > read lock rather than the write lock to improve concurrence. The first step > is to make the changes to ReplicaMap, as many other methods make calls to it. > This Jira switches read operations against the volume map to use the readLock > rather than the write lock. > Additionally, some methods make a call to replicaMap.replicas() (eg > getBlockReports, getFinalizedBlocks, deepCopyReplica) and only use the result > in a read only fashion, so they can also be switched to using a readLock. > Next is the directory scanner and disk balancer, which only require a read > lock. > Finally (for this Jira) are various "low hanging fruit" items in BlockSender > and fsdatasetImpl where is it fairly obvious they only need a read lock. > For now, I have avoided changing anything which looks too risky, as I think > its better to do any larger refactoring or risky changes each in their own > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16207) Remove NN logs stack trace for non-existent xattr query
[
https://issues.apache.org/jira/browse/HDFS-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412574#comment-17412574
]
Ahmed Hussein commented on HDFS-16207:
--
Thanks [~cnauroth]!
I will upload a separate patch for branch-2.10 shortly. Let me know if you
prefer that I submit a separate Pull-request for that rather than attaching a
patch to the jira.
> Remove NN logs stack trace for non-existent xattr query
> ---
>
> Key: HDFS-16207
> URL: https://issues.apache.org/jira/browse/HDFS-16207
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.4.0, 2.10.2, 3.3.2, 3.2.4
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The NN logs a full stack trace every time a getXAttrs is called for a
> non-existent xattr. The logging has zero value add. The increased logging
> load may harm performance. Something is now probing for xattrs resulting in
> many lines of:
> {code:bash}
> 2021-09-02 13:48:03,340 [IPC Server handler 5 on default port 59951] INFO
> ipc.Server (Server.java:logException(3149)) - IPC Server handler 5 on default
> port 59951, call Call#17 Retry#0
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getXAttrs from 127.0.0.1:59961
> java.io.IOException: At least one of the attributes provided was not found.
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirXAttrOp.getXAttrs(FSDirXAttrOp.java:134)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getXAttrs(FSNamesystem.java:8472)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getXAttrs(NameNodeRpcServer.java:2317)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getXAttrs(ClientNamenodeProtocolServerSideTranslatorPB.java:1745)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1155)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1083)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3088)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412536#comment-17412536 ] JiangHua Zhu commented on HDFS-14703: - [~prasad-acit], I have some very curious questions. The first one is: I see that INodeMap#NAMESPACE_KEY_DEBTH is a fixed value, and the default is 2. What happens if the value is 4 or 5? What I can think of is that this will affect the range of INodes allocated. The second is: If the value of INodeMap#NUM_RANGES_STATIC is greater than 256, the parallelism of processing and writing data will increase, which will affect the performance of RPC? > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15616) Disable Observer NameNode to trigger edit log roll
[ https://issues.apache.org/jira/browse/HDFS-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei reassigned HDFS-15616: -- Assignee: Janus Chow > Disable Observer NameNode to trigger edit log roll > -- > > Key: HDFS-15616 > URL: https://issues.apache.org/jira/browse/HDFS-15616 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15616.001.patch, HDFS-15616.002.patch, > HDFS-15616.003.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Currently when Observer is transitioned from StandbyState, the editLogTailer > will still send the request to roll editLog to ActiveNN, which should be > disabled to keep the definition of "logRollPeriodMs" clear. > One thing I'm not sure is for a cluster with multi standby Namenode, all the > standby NN will trigger the roll. Should this feature be extended to all > standby NNs or implementing on observers first? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=648519&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648519
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 09/Sep/21 11:05
Start Date: 09/Sep/21 11:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-915986774
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 175 unchanged - 1
fixed = 175 total (was 176) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 236m 16s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 321m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3340 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d5e8ef15c320 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1f43713e1f30afaa32d95f3fbe1bbe644c601b5f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/5/testReport/ |
| Max. process+thread c
[jira] [Assigned] (HDFS-16065) RBF: Add metrics to record Router's operations
[ https://issues.apache.org/jira/browse/HDFS-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei reassigned HDFS-16065: -- Assignee: Janus Chow > RBF: Add metrics to record Router's operations > -- > > Key: HDFS-16065 > URL: https://issues.apache.org/jira/browse/HDFS-16065 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, Router's operations are not well recorded. It would be good to > have a similar metrics as "Hadoop:service=NameNode,name=NameNodeActivity" for > NameNode, which shows the count for each operations. > Besides, some operations are invoked concurrently in Routers, know the counts > for concurrent operations would help us better knowing about the cluster's > state. > This ticket is to add normal operation metrics and concurrent operation > metrics for Router. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16065) RBF: Add metrics to record Router's operations
[ https://issues.apache.org/jira/browse/HDFS-16065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HDFS-16065. Fix Version/s: 3.4.0 Resolution: Fixed > RBF: Add metrics to record Router's operations > -- > > Key: HDFS-16065 > URL: https://issues.apache.org/jira/browse/HDFS-16065 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, Router's operations are not well recorded. It would be good to > have a similar metrics as "Hadoop:service=NameNode,name=NameNodeActivity" for > NameNode, which shows the count for each operations. > Besides, some operations are invoked concurrently in Routers, know the counts > for concurrent operations would help us better knowing about the cluster's > state. > This ticket is to add normal operation metrics and concurrent operation > metrics for Router. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16065) RBF: Add metrics to record Router's operations
[ https://issues.apache.org/jira/browse/HDFS-16065?focusedWorklogId=648513&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648513 ] ASF GitHub Bot logged work on HDFS-16065: - Author: ASF GitHub Bot Created on: 09/Sep/21 10:45 Start Date: 09/Sep/21 10:45 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3100: URL: https://github.com/apache/hadoop/pull/3100#issuecomment-915973951 @symious Thanks for contribution. @goiri Thanks for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648513) Time Spent: 2.5h (was: 2h 20m) > RBF: Add metrics to record Router's operations > -- > > Key: HDFS-16065 > URL: https://issues.apache.org/jira/browse/HDFS-16065 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently, Router's operations are not well recorded. It would be good to > have a similar metrics as "Hadoop:service=NameNode,name=NameNodeActivity" for > NameNode, which shows the count for each operations. > Besides, some operations are invoked concurrently in Routers, know the counts > for concurrent operations would help us better knowing about the cluster's > state. > This ticket is to add normal operation metrics and concurrent operation > metrics for Router. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16065) RBF: Add metrics to record Router's operations
[ https://issues.apache.org/jira/browse/HDFS-16065?focusedWorklogId=648512&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648512 ] ASF GitHub Bot logged work on HDFS-16065: - Author: ASF GitHub Bot Created on: 09/Sep/21 10:45 Start Date: 09/Sep/21 10:45 Worklog Time Spent: 10m Work Description: ferhui merged pull request #3100: URL: https://github.com/apache/hadoop/pull/3100 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648512) Time Spent: 2h 20m (was: 2h 10m) > RBF: Add metrics to record Router's operations > -- > > Key: HDFS-16065 > URL: https://issues.apache.org/jira/browse/HDFS-16065 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently, Router's operations are not well recorded. It would be good to > have a similar metrics as "Hadoop:service=NameNode,name=NameNodeActivity" for > NameNode, which shows the count for each operations. > Besides, some operations are invoked concurrently in Routers, know the counts > for concurrent operations would help us better knowing about the cluster's > state. > This ticket is to add normal operation metrics and concurrent operation > metrics for Router. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
