[jira] [Updated] (HDFS-16059) dfsadmin -listOpenFiles -blockingDecommission can miss some files
[
https://issues.apache.org/jira/browse/HDFS-16059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-16059:
Attachment: HDFS-16059-WIP-01.patch
> dfsadmin -listOpenFiles -blockingDecommission can miss some files
> -
>
> Key: HDFS-16059
> URL: https://issues.apache.org/jira/browse/HDFS-16059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsadmin
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-16059-WIP-01.patch, HDFS-16059-regression-test.patch
>
>
> While reviewing HDFS-13671, I found "dfsadmin -listOpenFiles
> -blockingDecommission" can drop some files.
> [https://github.com/apache/hadoop/pull/3065#discussion_r647396463]
> {quote}If the DataNodes have the following open files and we want to list all
> the open files:
> DN1: [1001, 1002, 1003, ... , 2000]
> DN2: [1, 2, 3, ... , 1000]
> At first getFilesBlockingDecom(0, "/") is called and it returns [1001, 1002,
> ... , 2000] because it reached max size (=1000), and next
> getFilesBlockingDecom(2000, "/") is called because the last inode Id of the
> previous result is 2000. That way the open files of DN2 is missed
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16059) dfsadmin -listOpenFiles -blockingDecommission can miss some files
[
https://issues.apache.org/jira/browse/HDFS-16059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436427#comment-17436427
]
Ayush Saxena commented on HDFS-16059:
-
Thanx [~aajisaka] for the report. Was just casually trying this out. Can we not
leverage the code for listing all open files and just have a filter for files
blocking decommission, rather than trying out each datanode? I couldn't think
of any reason where it would cause a problem or why it wasn't done that way
initially.
I have uploaded a WIP patch with the said approach. Tried out running the
related tests, they passed for me. But couldn't repro the exact scenario,
reproducing that seems tough, post we change the approach, so relied on the
existing tests :)
> dfsadmin -listOpenFiles -blockingDecommission can miss some files
> -
>
> Key: HDFS-16059
> URL: https://issues.apache.org/jira/browse/HDFS-16059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsadmin
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-16059-WIP-01.patch, HDFS-16059-regression-test.patch
>
>
> While reviewing HDFS-13671, I found "dfsadmin -listOpenFiles
> -blockingDecommission" can drop some files.
> [https://github.com/apache/hadoop/pull/3065#discussion_r647396463]
> {quote}If the DataNodes have the following open files and we want to list all
> the open files:
> DN1: [1001, 1002, 1003, ... , 2000]
> DN2: [1, 2, 3, ... , 1000]
> At first getFilesBlockingDecom(0, "/") is called and it returns [1001, 1002,
> ... , 2000] because it reached max size (=1000), and next
> getFilesBlockingDecom(2000, "/") is called because the last inode Id of the
> previous result is 2000. That way the open files of DN2 is missed
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14735) File could only be replicated to 0 nodes instead of minReplication (=1)
[ https://issues.apache.org/jira/browse/HDFS-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HDFS-14735. - Resolution: Not A Problem This is a cluster level issue, Please reach out to the user mailing list for help > File could only be replicated to 0 nodes instead of minReplication (=1) > --- > > Key: HDFS-14735 > URL: https://issues.apache.org/jira/browse/HDFS-14735 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tatyana Alexeyev >Priority: Major > > Hello I have intermitent error when running my EMR Hadoop Cluster: > "Error: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /user/sphdadm/_sqoop/00501bd7b05e4182b5006b9d51 > bafb7f_f405b2f3/_temporary/1/_temporary/attempt_1565136887564_20057_m_00_0/part-m-0.snappy > could only be replicated to 0 nodes instead of minReplication (=1). There > are 5 datanode(s) running and no node(s) are excluded in this operation." > I am running Hadoop version > sphdadm@ip-10-6-15-108 hadoop]$ hadoop version > Hadoop 2.8.5-amzn-4 > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15638) Make Hive tables directory permission check flat
[
https://issues.apache.org/jira/browse/HDFS-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436467#comment-17436467
]
Ayush Saxena commented on HDFS-15638:
-
{quote}but for Hive tables directories/files, all the files under a partition
or even a table usually have the same permissions for the same set of ACL groups
{quote}
This is just an assumption!, There is so such bindings AFAIK atleast not for
external tables.
Mostly using default ACL should handle most of the cases. Having a specific bit
for all directories just for HIVE, doesn't seems very practical in the real
world...
> Make Hive tables directory permission check flat
> -
>
> Key: HDFS-15638
> URL: https://issues.apache.org/jira/browse/HDFS-15638
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Xinli Shang
>Priority: Major
>
> Problem: Currently, when a user tries to accesses a file he/she needs the
> permissions of it's parent and ancestors and the permission of that file.
> This is correct generally, but for Hive tables directories/files, all the
> files under a partition or even a table usually have the same permissions for
> the same set of ACL groups. Although the permissions and ACL groups are the
> same, the writer still need to call setfacl() for every file. This results in
> a huge amount of RPC calls to NN. HDFS has default ACL to solve that but that
> only applies to create and copy, but not apply for rename. However, in Hive
> ETL, rename is very common.
> Proposal: Add a 1-bit flag to directory inodes to indicate whether or not it
> is a Hive table directory. If that flag is set, then all the sub-directory
> and files under it will just use it's permission and ACL groups settings. By
> doing this way, Hive ETL doesn't need to set permissions at the file level.
> If that flag is not set(by default), work as before. To set/unset that flag,
> it would require admin privilege.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[ https://issues.apache.org/jira/browse/HDFS-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu reassigned HDFS-16291: --- Assignee: JiangHua Zhu > Make the comment of INode#ReclaimContext more standardized > -- > > Key: HDFS-16291 > URL: https://issues.apache.org/jira/browse/HDFS-16291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Attachments: image-2021-10-31-20-25-08-379.png > > > In the INode#ReclaimContext class, there are some comments that are not > standardized enough. > E.g: > !image-2021-10-31-20-25-08-379.png! > We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[ https://issues.apache.org/jira/browse/HDFS-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-16291: Affects Version/s: 3.4.0 > Make the comment of INode#ReclaimContext more standardized > -- > > Key: HDFS-16291 > URL: https://issues.apache.org/jira/browse/HDFS-16291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Attachments: image-2021-10-31-20-25-08-379.png > > > In the INode#ReclaimContext class, there are some comments that are not > standardized enough. > E.g: > !image-2021-10-31-20-25-08-379.png! > We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
JiangHua Zhu created HDFS-16291: --- Summary: Make the comment of INode#ReclaimContext more standardized Key: HDFS-16291 URL: https://issues.apache.org/jira/browse/HDFS-16291 Project: Hadoop HDFS Issue Type: Improvement Components: documentation, namenode Reporter: JiangHua Zhu Attachments: image-2021-10-31-20-25-08-379.png In the INode#ReclaimContext class, there are some comments that are not standardized enough. E.g: !image-2021-10-31-20-25-08-379.png! We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16286) Debug tool to verify the correctness of erasure coding on file
[ https://issues.apache.org/jira/browse/HDFS-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436476#comment-17436476 ] daimin commented on HDFS-16286: --- [~sodonnell] Thanks for your reply. I have visited the README document and the code implementation, the EC validator looks good enough to me. And in fact, if I had known this work has been done, I would not paying time to re-work it out. Our motive to build such a tool initially is because we had data corruption of EC block group in our production environment and one big problem is that we could not tell which files are good or bad. After we checked all the suspicious files, we think this tool may be useful to those who are using EC too. Could you please pay some time on reviewing the patch? Thanks a lot. > Debug tool to verify the correctness of erasure coding on file > -- > > Key: HDFS-16286 > URL: https://issues.apache.org/jira/browse/HDFS-16286 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.3.0, 3.3.1 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Block data in erasure coded block group may corrupt and the block meta > (checksum) is unable to discover the corruption in some cases such as EC > reconstruction, related issues are: HDFS-14768, HDFS-15186, HDFS-15240. > In addition to HDFS-15759, there needs a tool to check erasure coded file > whether any block group has data corruption in case of other conditions > rather than EC reconstruction, or the feature HDFS-15759(validation during EC > reconstruction) is not open(which is close by default now). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[ https://issues.apache.org/jira/browse/HDFS-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16291: -- Labels: pull-request-available (was: ) > Make the comment of INode#ReclaimContext more standardized > -- > > Key: HDFS-16291 > URL: https://issues.apache.org/jira/browse/HDFS-16291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Attachments: image-2021-10-31-20-25-08-379.png > > Time Spent: 10m > Remaining Estimate: 0h > > In the INode#ReclaimContext class, there are some comments that are not > standardized enough. > E.g: > !image-2021-10-31-20-25-08-379.png! > We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[ https://issues.apache.org/jira/browse/HDFS-16291?focusedWorklogId=672456&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672456 ] ASF GitHub Bot logged work on HDFS-16291: - Author: ASF GitHub Bot Created on: 31/Oct/21 13:19 Start Date: 31/Oct/21 13:19 Worklog Time Spent: 10m Work Description: jianghuazhu opened a new pull request #3602: URL: https://github.com/apache/hadoop/pull/3602 ### Description of PR In the INode#ReclaimContext class, there are some irregular comments. The purpose of this pr is to make them more standardized. Details: HDFS-16291 ### How was this patch tested? Here is just some work on the documentation, the test pressure is not great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 672456) Remaining Estimate: 0h Time Spent: 10m > Make the comment of INode#ReclaimContext more standardized > -- > > Key: HDFS-16291 > URL: https://issues.apache.org/jira/browse/HDFS-16291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Attachments: image-2021-10-31-20-25-08-379.png > > Time Spent: 10m > Remaining Estimate: 0h > > In the INode#ReclaimContext class, there are some comments that are not > standardized enough. > E.g: > !image-2021-10-31-20-25-08-379.png! > We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[ https://issues.apache.org/jira/browse/HDFS-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16291 started by JiangHua Zhu. --- > Make the comment of INode#ReclaimContext more standardized > -- > > Key: HDFS-16291 > URL: https://issues.apache.org/jira/browse/HDFS-16291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Attachments: image-2021-10-31-20-25-08-379.png > > Time Spent: 10m > Remaining Estimate: 0h > > In the INode#ReclaimContext class, there are some comments that are not > standardized enough. > E.g: > !image-2021-10-31-20-25-08-379.png! > We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16269) [Fix] Improve NNThroughputBenchmark#blockReport operation
[
https://issues.apache.org/jira/browse/HDFS-16269?focusedWorklogId=672473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672473
]
ASF GitHub Bot logged work on HDFS-16269:
-
Author: ASF GitHub Bot
Created on: 31/Oct/21 18:12
Start Date: 31/Oct/21 18:12
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3544:
URL: https://github.com/apache/hadoop/pull/3544#discussion_r739857661
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
##
@@ -166,4 +166,25 @@ public void testNNThroughputForAppendOp() throws Exception
{
}
}
}
+
+ /**
+ * This test runs {@link NNThroughputBenchmark} against a mini DFS cluster
+ * for block report operation.
+ */
+ @Test(timeout = 12)
+ public void testNNThroughputForBlockReportOp() throws Exception {
+final Configuration conf = new HdfsConfiguration();
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(3).build()) {
+ cluster.waitActive();
+ final Configuration benchConf = new HdfsConfiguration();
+ benchConf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+ benchConf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+ NNThroughputBenchmark.runBenchmark(benchConf,
+ new String[]{"-fs", cluster.getURI().toString(), "-op",
+ "blockReport", "-datanodes", "3", "-reports", "2"});
Review comment:
Would you fix the checkstyle warning in L187? Sorry my example was wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672473)
Time Spent: 3h 40m (was: 3.5h)
> [Fix] Improve NNThroughputBenchmark#blockReport operation
> -
>
> Key: HDFS-16269
> URL: https://issues.apache.org/jira/browse/HDFS-16269
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, namenode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> When using NNThroughputBenchmark to verify the blockReport, you will get some
> exception information.
> Commands used:
> ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
> -op blockReport -datanodes 3 -reports 1
> The exception information:
> 21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark:
> blockReport
> 21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with
> 10 blocks each.
> 21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark:
> java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoo
[jira] [Work logged] (HDFS-16269) [Fix] Improve NNThroughputBenchmark#blockReport operation
[
https://issues.apache.org/jira/browse/HDFS-16269?focusedWorklogId=672474&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672474
]
ASF GitHub Bot logged work on HDFS-16269:
-
Author: ASF GitHub Bot
Created on: 31/Oct/21 18:14
Start Date: 31/Oct/21 18:14
Worklog Time Spent: 10m
Work Description: aajisaka commented on a change in pull request #3544:
URL: https://github.com/apache/hadoop/pull/3544#discussion_r739857661
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
##
@@ -166,4 +166,25 @@ public void testNNThroughputForAppendOp() throws Exception
{
}
}
}
+
+ /**
+ * This test runs {@link NNThroughputBenchmark} against a mini DFS cluster
+ * for block report operation.
+ */
+ @Test(timeout = 12)
+ public void testNNThroughputForBlockReportOp() throws Exception {
+final Configuration conf = new HdfsConfiguration();
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(3).build()) {
+ cluster.waitActive();
+ final Configuration benchConf = new HdfsConfiguration();
+ benchConf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+ benchConf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+ NNThroughputBenchmark.runBenchmark(benchConf,
+ new String[]{"-fs", cluster.getURI().toString(), "-op",
+ "blockReport", "-datanodes", "3", "-reports", "2"});
Review comment:
Would you fix the checkstyle warning in L187? Sorry my example
(https://github.com/apache/hadoop/pull/3544#discussion_r738940853) was wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672474)
Time Spent: 3h 50m (was: 3h 40m)
> [Fix] Improve NNThroughputBenchmark#blockReport operation
> -
>
> Key: HDFS-16269
> URL: https://issues.apache.org/jira/browse/HDFS-16269
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, namenode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> When using NNThroughputBenchmark to verify the blockReport, you will get some
> exception information.
> Commands used:
> ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
> -op blockReport -datanodes 3 -reports 1
> The exception information:
> 21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark:
> blockReport
> 21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with
> 10 blocks each.
> 21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark:
> java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.h
[jira] [Work logged] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[
https://issues.apache.org/jira/browse/HDFS-16291?focusedWorklogId=672493&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672493
]
ASF GitHub Bot logged work on HDFS-16291:
-
Author: ASF GitHub Bot
Created on: 31/Oct/21 20:59
Start Date: 31/Oct/21 20:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3602:
URL: https://github.com/apache/hadoop/pull/3602#issuecomment-955790559
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 352m 49s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 459m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3602/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3602 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 475888c7d128 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 89a6347517792d0f8e9c302b4f8cdbfad211cfee |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3602/1/testReport/ |
| Max. process+thread count | 1981 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3602/1/console |
| versions | git=
[jira] [Work logged] (HDFS-16285) Make HDFS ownership tools cross platform
[
https://issues.apache.org/jira/browse/HDFS-16285?focusedWorklogId=672499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672499
]
ASF GitHub Bot logged work on HDFS-16285:
-
Author: ASF GitHub Bot
Created on: 31/Oct/21 21:36
Start Date: 31/Oct/21 21:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3588:
URL: https://github.com/apache/hadoop/pull/3588#issuecomment-955795792
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 0s | | trunk passed |
| +1 :green_heart: | compile | 3m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 61m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed |
| +1 :green_heart: | cc | 3m 18s | | the patch passed |
| +1 :green_heart: | golang | 3m 18s | | the patch passed |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 96m 53s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 189m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3588 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux ca030a88dd73 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15677758e0447b99c25fc7e0158bc3969a6c1544 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/testReport/ |
| Max. process+thread count | 565 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672499)
Time Spent: 50m (was: 40m)
> Make HDFS ownership tools cross platform
>
>
> Key: HDFS-16285
> URL: https://issues.apache.org/jira/browse/HDFS-16285
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The source files for *hdfs_chown*, *hdfs_chmod* and *hdfs_chgrp* uses getopt
> for parsing the command line arguments. getopt is available only on Linux and
> thus, isn't cross platform. We need to replace getopt with
> boo
[jira] [Work logged] (HDFS-16285) Make HDFS ownership tools cross platform
[
https://issues.apache.org/jira/browse/HDFS-16285?focusedWorklogId=672508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672508
]
ASF GitHub Bot logged work on HDFS-16285:
-
Author: ASF GitHub Bot
Created on: 01/Nov/21 00:37
Start Date: 01/Nov/21 00:37
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3588:
URL: https://github.com/apache/hadoop/pull/3588#issuecomment-955826153
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 35s | | trunk passed |
| +1 :green_heart: | compile | 3m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 50m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 3s | | the patch passed |
| +1 :green_heart: | cc | 3m 3s | | the patch passed |
| +1 :green_heart: | golang | 3m 3s | | the patch passed |
| +1 :green_heart: | javac | 3m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 29s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 180m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3588 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 833404994e10 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15677758e0447b99c25fc7e0158bc3969a6c1544 |
| Default Java | Red Hat, Inc.-1.8.0_312-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/testReport/ |
| Max. process+thread count | 585 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672508)
Time Spent: 1h (was: 50m)
> Make HDFS ownership tools cross platform
>
>
> Key: HDFS-16285
> URL: https://issues.apache.org/jira/browse/HDFS-16285
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The source files for *hdfs_chown*, *hdfs_chmod* and *hdfs_chgrp* uses getopt
> for parsing the command line arguments. getopt is available only on Linux and
> thus, isn't cross platform. We need to replace getopt with
> boos
[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log
[ https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=672514&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672514 ] ASF GitHub Bot logged work on HDFS-16266: - Author: ASF GitHub Bot Created on: 01/Nov/21 02:24 Start Date: 01/Nov/21 02:24 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3538: URL: https://github.com/apache/hadoop/pull/3538#issuecomment-955873323 @tomscut Thanks for your thoughts. That makes sense to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 672514) Time Spent: 6h 40m (was: 6.5h) > Add remote port information to HDFS audit log > - > > Key: HDFS-16266 > URL: https://issues.apache.org/jira/browse/HDFS-16266 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 6h 40m > Remaining Estimate: 0h > > In our production environment, we occasionally encounter a problem where a > user submits an abnormal computation task, causing a sudden flood of > requests, which causes the queueTime and processingTime of the Namenode to > rise very high, causing a large backlog of tasks. > We usually locate and kill specific Spark, Flink, or MapReduce tasks based on > metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but > there is no port information, so it is difficult to locate specific processes > sometimes. Therefore, I propose that we add the port information to the audit > log, so that we can easily track the upstream process. > Currently, some projects contain port information in audit logs, such as > Hbase and Alluxio. I think it is also necessary to add port information for > HDFS audit logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log
[ https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=672515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672515 ] ASF GitHub Bot logged work on HDFS-16266: - Author: ASF GitHub Bot Created on: 01/Nov/21 02:29 Start Date: 01/Nov/21 02:29 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3538: URL: https://github.com/apache/hadoop/pull/3538#issuecomment-955874908 > @tomscut Thanks for your thoughts. That makes sense to me. Thanks @tasanuma for your reply and review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 672515) Time Spent: 6h 50m (was: 6h 40m) > Add remote port information to HDFS audit log > - > > Key: HDFS-16266 > URL: https://issues.apache.org/jira/browse/HDFS-16266 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 6h 50m > Remaining Estimate: 0h > > In our production environment, we occasionally encounter a problem where a > user submits an abnormal computation task, causing a sudden flood of > requests, which causes the queueTime and processingTime of the Namenode to > rise very high, causing a large backlog of tasks. > We usually locate and kill specific Spark, Flink, or MapReduce tasks based on > metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but > there is no port information, so it is difficult to locate specific processes > sometimes. Therefore, I propose that we add the port information to the audit > log, so that we can easily track the upstream process. > Currently, some projects contain port information in audit logs, such as > Hbase and Alluxio. I think it is also necessary to add port information for > HDFS audit logs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16285) Make HDFS ownership tools cross platform
[
https://issues.apache.org/jira/browse/HDFS-16285?focusedWorklogId=672525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672525
]
ASF GitHub Bot logged work on HDFS-16285:
-
Author: ASF GitHub Bot
Created on: 01/Nov/21 03:56
Start Date: 01/Nov/21 03:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3588:
URL: https://github.com/apache/hadoop/pull/3588#issuecomment-955905296
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 12s | | trunk passed |
| +1 :green_heart: | compile | 3m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 66m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed |
| +1 :green_heart: | cc | 2m 48s | | the patch passed |
| +1 :green_heart: | golang | 2m 48s | | the patch passed |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 30m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 95m 57s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 199m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3588 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux 43a970c247a7 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 15677758e0447b99c25fc7e0158bc3969a6c1544 |
| Default Java | Debian-11.0.12+7-post-Debian-2deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3588/5/console |
| versions | git=2.20.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672525)
Time Spent: 1h 10m (was: 1h)
> Make HDFS ownership tools cross platform
>
>
> Key: HDFS-16285
> URL: https://issues.apache.org/jira/browse/HDFS-16285
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The source files for *hdfs_chown*, *hdfs_chmod* and *hdfs_chgrp* uses getopt
> for parsing the command line arguments. getopt is available only on Linux and
> thus, isn't cross platform. We need to replace getopt with
> b
[jira] [Work logged] (HDFS-16269) [Fix] Improve NNThroughputBenchmark#blockReport operation
[
https://issues.apache.org/jira/browse/HDFS-16269?focusedWorklogId=672541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672541
]
ASF GitHub Bot logged work on HDFS-16269:
-
Author: ASF GitHub Bot
Created on: 01/Nov/21 04:35
Start Date: 01/Nov/21 04:35
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on a change in pull request #3544:
URL: https://github.com/apache/hadoop/pull/3544#discussion_r739951306
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
##
@@ -166,4 +166,25 @@ public void testNNThroughputForAppendOp() throws Exception
{
}
}
}
+
+ /**
+ * This test runs {@link NNThroughputBenchmark} against a mini DFS cluster
+ * for block report operation.
+ */
+ @Test(timeout = 12)
+ public void testNNThroughputForBlockReportOp() throws Exception {
+final Configuration conf = new HdfsConfiguration();
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(3).build()) {
+ cluster.waitActive();
+ final Configuration benchConf = new HdfsConfiguration();
+ benchConf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+ benchConf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+ NNThroughputBenchmark.runBenchmark(benchConf,
+ new String[]{"-fs", cluster.getURI().toString(), "-op",
+ "blockReport", "-datanodes", "3", "-reports", "2"});
Review comment:
OK. I will fix some checkstyle formats, the updated format is as follows:
`
NNThroughputBenchmark.runBenchmark(benchConf,
new String[]{"-fs", cluster.getURI().toString(), "-op",
"blockReport", "-datanodes", "3", "-reports", "2"});
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672541)
Time Spent: 4h (was: 3h 50m)
> [Fix] Improve NNThroughputBenchmark#blockReport operation
> -
>
> Key: HDFS-16269
> URL: https://issues.apache.org/jira/browse/HDFS-16269
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, namenode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> When using NNThroughputBenchmark to verify the blockReport, you will get some
> exception information.
> Commands used:
> ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
> -op blockReport -datanodes 3 -reports 1
> The exception information:
> 21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark:
> blockReport
> 21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with
> 10 blocks each.
> 21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark:
> java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughp
[jira] [Work logged] (HDFS-16269) [Fix] Improve NNThroughputBenchmark#blockReport operation
[
https://issues.apache.org/jira/browse/HDFS-16269?focusedWorklogId=672543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672543
]
ASF GitHub Bot logged work on HDFS-16269:
-
Author: ASF GitHub Bot
Created on: 01/Nov/21 04:38
Start Date: 01/Nov/21 04:38
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on a change in pull request #3544:
URL: https://github.com/apache/hadoop/pull/3544#discussion_r739951306
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
##
@@ -166,4 +166,25 @@ public void testNNThroughputForAppendOp() throws Exception
{
}
}
}
+
+ /**
+ * This test runs {@link NNThroughputBenchmark} against a mini DFS cluster
+ * for block report operation.
+ */
+ @Test(timeout = 12)
+ public void testNNThroughputForBlockReportOp() throws Exception {
+final Configuration conf = new HdfsConfiguration();
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(3).build()) {
+ cluster.waitActive();
+ final Configuration benchConf = new HdfsConfiguration();
+ benchConf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+ benchConf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+ NNThroughputBenchmark.runBenchmark(benchConf,
+ new String[]{"-fs", cluster.getURI().toString(), "-op",
+ "blockReport", "-datanodes", "3", "-reports", "2"});
Review comment:
OK. I will fix some checkstyle formats, the updated format is as follows:
`
NNThroughputBenchmark.runBenchmark(benchConf,
new String[]{"-fs", cluster.getURI().toString(), "-op",
"blockReport", "-datanodes", "3", "-reports", "2"});
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672543)
Time Spent: 4h 10m (was: 4h)
> [Fix] Improve NNThroughputBenchmark#blockReport operation
> -
>
> Key: HDFS-16269
> URL: https://issues.apache.org/jira/browse/HDFS-16269
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, namenode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When using NNThroughputBenchmark to verify the blockReport, you will get some
> exception information.
> Commands used:
> ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
> -op blockReport -datanodes 3 -reports 1
> The exception information:
> 21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark:
> blockReport
> 21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with
> 10 blocks each.
> 21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark:
> java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenod
[jira] [Work logged] (HDFS-16269) [Fix] Improve NNThroughputBenchmark#blockReport operation
[
https://issues.apache.org/jira/browse/HDFS-16269?focusedWorklogId=672544&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672544
]
ASF GitHub Bot logged work on HDFS-16269:
-
Author: ASF GitHub Bot
Created on: 01/Nov/21 04:39
Start Date: 01/Nov/21 04:39
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on a change in pull request #3544:
URL: https://github.com/apache/hadoop/pull/3544#discussion_r739952380
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
##
@@ -166,4 +166,25 @@ public void testNNThroughputForAppendOp() throws Exception
{
}
}
}
+
+ /**
+ * This test runs {@link NNThroughputBenchmark} against a mini DFS cluster
+ * for block report operation.
+ */
+ @Test(timeout = 12)
+ public void testNNThroughputForBlockReportOp() throws Exception {
+final Configuration conf = new HdfsConfiguration();
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+numDataNodes(3).build()) {
+ cluster.waitActive();
+ final Configuration benchConf = new HdfsConfiguration();
+ benchConf.setInt(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, 16);
+ benchConf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 16);
+ NNThroughputBenchmark.runBenchmark(benchConf,
+ new String[]{"-fs", cluster.getURI().toString(), "-op",
+ "blockReport", "-datanodes", "3", "-reports", "2"});
Review comment:
Sorry, the information shown here cannot fully demonstrate my thoughts.
I will submit some updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 672544)
Time Spent: 4h 20m (was: 4h 10m)
> [Fix] Improve NNThroughputBenchmark#blockReport operation
> -
>
> Key: HDFS-16269
> URL: https://issues.apache.org/jira/browse/HDFS-16269
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: benchmarks, namenode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> When using NNThroughputBenchmark to verify the blockReport, you will get some
> exception information.
> Commands used:
> ./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs
> -op blockReport -datanodes 3 -reports 1
> The exception information:
> 21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark:
> blockReport
> 21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with
> 10 blocks each.
> 21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark:
> java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner
[jira] [Work logged] (HDFS-16291) Make the comment of INode#ReclaimContext more standardized
[ https://issues.apache.org/jira/browse/HDFS-16291?focusedWorklogId=672546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672546 ] ASF GitHub Bot logged work on HDFS-16291: - Author: ASF GitHub Bot Created on: 01/Nov/21 05:21 Start Date: 01/Nov/21 05:21 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3602: URL: https://github.com/apache/hadoop/pull/3602#issuecomment-955942472 It seems that Jenkins did not execute successfully. It seems that these have little to do with the code I submitted. @ayushtkn @virajjasani @tomscut, you are willing to spend some time reviewing this PR. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 672546) Time Spent: 0.5h (was: 20m) > Make the comment of INode#ReclaimContext more standardized > -- > > Key: HDFS-16291 > URL: https://issues.apache.org/jira/browse/HDFS-16291 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Affects Versions: 3.4.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-request-available > Attachments: image-2021-10-31-20-25-08-379.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > In the INode#ReclaimContext class, there are some comments that are not > standardized enough. > E.g: > !image-2021-10-31-20-25-08-379.png! > We should make comments more standardized. This will be more readable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
