[jira] [Commented] (HDFS-16044) Fix getListing call getLocatedBlocks even source is a directory
[
https://issues.apache.org/jira/browse/HDFS-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410945#comment-17410945
]
Hadoop QA commented on HDFS-16044:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
40s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 32s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 18m
29s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
42s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
32s{color} | {color
[jira] [Work logged] (HDFS-16197) Simplify getting NNStorage in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16197?focusedWorklogId=647183&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647183 ] ASF GitHub Bot logged work on HDFS-16197: - Author: ASF GitHub Bot Created on: 07/Sep/21 05:47 Start Date: 07/Sep/21 05:47 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3357: URL: https://github.com/apache/hadoop/pull/3357#issuecomment-914007894 Thanks @aajisaka for the comment. Do you think I need more jenkins running here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 647183) Time Spent: 2h 20m (was: 2h 10m) > Simplify getting NNStorage in FSNamesystem > -- > > Key: HDFS-16197 > URL: https://issues.apache.org/jira/browse/HDFS-16197 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > In FSNamesystem, there are many places where NNStorage needs to be used > (according to preliminary statistics, there are 15 times), and these places > are obtained using "getFSImage().getStorage()". We should try to use a > simpler way. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16197) Simplify getting NNStorage in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16197?focusedWorklogId=647182&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647182 ] ASF GitHub Bot logged work on HDFS-16197: - Author: ASF GitHub Bot Created on: 07/Sep/21 05:13 Start Date: 07/Sep/21 05:13 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #3357: URL: https://github.com/apache/hadoop/pull/3357#issuecomment-913996073 It seems that the Jenkins is down. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 647182) Time Spent: 2h 10m (was: 2h) > Simplify getting NNStorage in FSNamesystem > -- > > Key: HDFS-16197 > URL: https://issues.apache.org/jira/browse/HDFS-16197 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > In FSNamesystem, there are many places where NNStorage needs to be used > (according to preliminary statistics, there are 15 times), and these places > are obtained using "getFSImage().getStorage()". We should try to use a > simpler way. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16197) Simplify getting NNStorage in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16197?focusedWorklogId=647162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647162 ] ASF GitHub Bot logged work on HDFS-16197: - Author: ASF GitHub Bot Created on: 07/Sep/21 01:59 Start Date: 07/Sep/21 01:59 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3357: URL: https://github.com/apache/hadoop/pull/3357#issuecomment-913935189 Jenkins has been running for a long time (1d 20h), and it still seems to be running. Not sure when it can stop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 647162) Time Spent: 2h (was: 1h 50m) > Simplify getting NNStorage in FSNamesystem > -- > > Key: HDFS-16197 > URL: https://issues.apache.org/jira/browse/HDFS-16197 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > In FSNamesystem, there are many places where NNStorage needs to be used > (according to preliminary statistics, there are 15 times), and these places > are obtained using "getFSImage().getStorage()". We should try to use a > simpler way. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410871#comment-17410871 ] JiangHua Zhu commented on HDFS-14703: - Okay, I get it. Thanks [~prasad-acit] for the comment. [~weichiu], please pay attention to this comment, I hope it will help the fgl branch. > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16208) [FGL] Implement Delete API with FGL
[ https://issues.apache.org/jira/browse/HDFS-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410869#comment-17410869 ] JiangHua Zhu commented on HDFS-16208: - It looks good. Thanks [~prasad-acit] for the comment. > [FGL] Implement Delete API with FGL > --- > > Key: HDFS-16208 > URL: https://issues.apache.org/jira/browse/HDFS-16208 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Replace all global locks for file / directory deletion with FGL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16209) Add description for dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei updated HDFS-16209: --- Status: Patch Available (was: Open) > Add description for dfs.namenode.caching.enabled > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410702#comment-17410702 ] Renukaprasad C commented on HDFS-14703: --- [~jianghuazhu] Initially there are 2 commits done as part of POC in the beginning. INodeMap with PartitionedGSet and per-partition locking (This will map to Jira - HDFS-14734 & HDFS-14732). [FGL] Introduce INode key. (This will map to Jira - HDFS-14733) > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16208) [FGL] Implement Delete API with FGL
[ https://issues.apache.org/jira/browse/HDFS-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410693#comment-17410693 ] Renukaprasad C commented on HDFS-16208: --- Sure [~jianghuazhu]. I missed to attach the report for DELETE operation. And the query used - ./hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs file:/// -op delete -threads 200 -files 100 -filesPerDir 100 ||Itr||Base||Patch|| |1|36886|55126| |2|40783|52029| |3|39698|40950| |4|42247|55157| |5|38197|49285| |Avg|39562|50509| |Imp %| |27%| > [FGL] Implement Delete API with FGL > --- > > Key: HDFS-16208 > URL: https://issues.apache.org/jira/browse/HDFS-16208 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Replace all global locks for file / directory deletion with FGL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410606#comment-17410606 ] JiangHua Zhu commented on HDFS-14703: - I noticed that the file PartitionedGSet.java has some modifications, which were added by submitting "Add namespace key for INode. (shv)". https://github.com/apache/hadoop/commit/455e8c019184d5d3ae7bcff4d29d9baa7aff3663 The submission message does not have a jira id, and I cannot find a jira with the same abstract. It's a bit difficult to track current progress. I am here only as a reminder, if what I say here is wrong, I will correct it. > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, > 002-partitioned-inodeMap-POC.tar.gz, 003-partitioned-inodeMap-POC.tar.gz, > NameNode Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410605#comment-17410605 ] lei w commented on HDFS-16196: -- Hi [~hexiaoqiao] . I saw other requests on NameNode, those do not have the same case。Because router will send full path on other requests which without filedID. But fsync、addBlock、complete、abandonBlock has fileID and the path loged by method addBlock、complete、abandonBlock is resolved by fileID at first. Only complete method will use the path sent by router. So I think we don’t need to let the router send the full path in complete method, we can resolve path by fileID at first like addBlock. > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16196.001.patch > > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w reassigned HDFS-16196: Assignee: lei w > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16196.001.patch > > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w updated HDFS-16196: - Attachment: HDFS-16196.001.patch Status: Patch Available (was: Open) > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16196.001.patch > > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w updated HDFS-16196: - Attachment: HDFS-16196.001.patch > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Attachments: HDFS-16196.001.patch > > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w updated HDFS-16196: - Attachment: (was: HDFS-16196.001.patch) > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16196.001.patch > > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16208) [FGL] Implement Delete API with FGL
[ https://issues.apache.org/jira/browse/HDFS-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410558#comment-17410558 ] JiangHua Zhu commented on HDFS-16208: - I want to take the liberty to ask, are there any test reports that I can refer to here? Because we are also paying attention to this issue recently. > [FGL] Implement Delete API with FGL > --- > > Key: HDFS-16208 > URL: https://issues.apache.org/jira/browse/HDFS-16208 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Replace all global locks for file / directory deletion with FGL. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Add description for dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646872 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 11:10 Start Date: 06/Sep/21 11:10 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913562915 Hi @ayushtkn @ferhui , this PR seems not to trigger compilation, can you help trigger it. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646872) Time Spent: 2.5h (was: 2h 20m) > Add description for dfs.namenode.caching.enabled > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16204) Improve FSDirEncryptionZoneOp related parameter comments
[ https://issues.apache.org/jira/browse/HDFS-16204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HDFS-16204. Fix Version/s: 3.4.0 Resolution: Fixed > Improve FSDirEncryptionZoneOp related parameter comments > > > Key: HDFS-16204 > URL: https://issues.apache.org/jira/browse/HDFS-16204 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In FSDirEncryptionZoneOp, there are some parameter comments that are too > simple to understand. We can try to perfect them, this is the purpose of this > jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16204) Improve FSDirEncryptionZoneOp related parameter comments
[ https://issues.apache.org/jira/browse/HDFS-16204?focusedWorklogId=646832&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646832 ] ASF GitHub Bot logged work on HDFS-16204: - Author: ASF GitHub Bot Created on: 06/Sep/21 10:06 Start Date: 06/Sep/21 10:06 Worklog Time Spent: 10m Work Description: ferhui merged pull request #3368: URL: https://github.com/apache/hadoop/pull/3368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646832) Time Spent: 1h 20m (was: 1h 10m) > Improve FSDirEncryptionZoneOp related parameter comments > > > Key: HDFS-16204 > URL: https://issues.apache.org/jira/browse/HDFS-16204 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > In FSDirEncryptionZoneOp, there are some parameter comments that are too > simple to understand. We can try to perfect them, this is the purpose of this > jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16204) Improve FSDirEncryptionZoneOp related parameter comments
[ https://issues.apache.org/jira/browse/HDFS-16204?focusedWorklogId=646830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646830 ] ASF GitHub Bot logged work on HDFS-16204: - Author: ASF GitHub Bot Created on: 06/Sep/21 10:05 Start Date: 06/Sep/21 10:05 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3368: URL: https://github.com/apache/hadoop/pull/3368#issuecomment-913521252 @jianghuazhu Thanks for contribution. @ayushtkn @virajjasani Thanks for review! Merged to trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646830) Time Spent: 1h 10m (was: 1h) > Improve FSDirEncryptionZoneOp related parameter comments > > > Key: HDFS-16204 > URL: https://issues.apache.org/jira/browse/HDFS-16204 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In FSDirEncryptionZoneOp, there are some parameter comments that are too > simple to understand. We can try to perfect them, this is the purpose of this > jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Add description for dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646828 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 10:03 Start Date: 06/Sep/21 10:03 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913519756 > Thanks @tomscut, just one more thing. If you could add a `TODO` comment in DFSConfigKeys on top of `DFS_NAMENODE_CACHING_ENABLED_DEFAULT = true;` line stating that `TODO: Default value to be set false in 4.0.0 release onwards (HDFS-16209)`, that would be really great. Thanks @virajjasani for your suggestion, I added the todo description for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646828) Time Spent: 2h 20m (was: 2h 10m) > Add description for dfs.namenode.caching.enabled > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Add description for dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646819 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 09:39 Start Date: 06/Sep/21 09:39 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913503361 Thanks @tomscut, just one more thing. If you could add a `TODO` comment in DFSConfigKeys on top of `DFS_NAMENODE_CACHING_ENABLED_DEFAULT = true;` line stating that `TODO: Default value to be set false in 4.0.0 release onwards (HDFS-16209)`, that would be really great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646819) Time Spent: 2h 10m (was: 2h) > Add description for dfs.namenode.caching.enabled > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410501#comment-17410501 ] lei w commented on HDFS-16196: -- OK，Thanks for [~hexiaoqiao] comment. > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16210) Add the option of refreshCallQueue to RouterAdmin
[
https://issues.apache.org/jira/browse/HDFS-16210?focusedWorklogId=646815&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646815
]
ASF GitHub Bot logged work on HDFS-16210:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 09:29
Start Date: 06/Sep/21 09:29
Worklog Time Spent: 10m
Work Description: symious commented on a change in pull request #3379:
URL: https://github.com/apache/hadoop/pull/3379#discussion_r702751678
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1258,6 +1262,39 @@ public int genericRefresh(String[] argv, int i) throws
IOException {
}
}
+ /**
+ * Refresh Router's call Queue.
+ *
+ * @throws IOException if the operation was not successful.
+ */
+ private int refreshCallQueue() throws IOException {
+Configuration conf = getConf();
+String hostport = getConf().getTrimmed(
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
+
+// Create the client
+Class xface = RefreshCallQueueProtocolPB.class;
+InetSocketAddress address = NetUtils.createSocketAddr(hostport);
+UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+
+RPC.setProtocolEngine(conf, xface, ProtobufRpcEngine2.class);
+RefreshCallQueueProtocolPB proxy =
(RefreshCallQueueProtocolPB)RPC.getProxy(
+xface, RPC.getProtocolVersion(xface), address, ugi, conf,
+NetUtils.getDefaultSocketFactory(conf), 0);
+
+int returnCode = -1;
+try (RefreshCallQueueProtocolClientSideTranslatorPB xlator =
+new RefreshCallQueueProtocolClientSideTranslatorPB(proxy)) {
+ xlator.refreshCallQueue();
+ System.out.println("Refresh call queue successful for " + hostport);
Review comment:
Sure, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646815)
Time Spent: 1h 20m (was: 1h 10m)
> Add the option of refreshCallQueue to RouterAdmin
> -
>
> Key: HDFS-16210
> URL: https://issues.apache.org/jira/browse/HDFS-16210
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Janus Chow
>Assignee: Janus Chow
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We enabled FairCallQueue to RouterRpcServer, but Router can not
> refreshCallQueue as NameNode does.
> This ticket is to enable the refreshCallQueue for Router so that we don't
> have to restart the Routers when updating FairCallQueue configurations.
>
> The option is not to refreshCallQueue to NameNodes, just trying to refresh
> the callQueue of Router itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Add description for dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646814 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 09:26 Start Date: 06/Sep/21 09:26 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913495313 > @tomscut @virajjasani, Thanks. I think It's a good way to add description here. > BTW, change the title of JIRA and PR. Thanks @ferhui for your reply. I changed the title of JIRA and PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646814) Time Spent: 2h (was: 1h 50m) > Add description for dfs.namenode.caching.enabled > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16209) Add description for dfs.namenode.caching.enabled
[ https://issues.apache.org/jira/browse/HDFS-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16209: --- Summary: Add description for dfs.namenode.caching.enabled (was: Set dfs.namenode.caching.enabled to false as default) > Add description for dfs.namenode.caching.enabled > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[
https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410484#comment-17410484
]
Xiaoqiao He commented on HDFS-16196:
Thanks [~lei w] for your report. It is confused log for users. I think it
should be fixed no matter at NameNode or Router side. After checking I think It
is involved by the following `method` construction which give `new
RemoteParam()` without `src`.
{code:java}
@Override
public boolean complete(String src, String clientName, ExtendedBlock last,
long fileId) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
RemoteMethod method = new RemoteMethod("complete",
new Class[] {String.class, String.class, ExtendedBlock.class,
long.class},
new RemoteParam(), clientName, last, fileId);
if (last != null) {
return rpcClient.invokeSingle(last, method, Boolean.class);
}
final List locations =
rpcServer.getLocationsForPath(src, true);
// Complete can return true/false, so don't expect a result
return rpcClient.invokeSequential(locations, method, Boolean.class, null);
}
{code}
Maybe it is common case and some other requests (which use
`rpcClient.invokeSingle`)could have the same case. Would you like to improve it
together?
> Namesystem#completeFile method will log incorrect path information when
> router to access
>
>
> Key: HDFS-16196
> URL: https://issues.apache.org/jira/browse/HDFS-16196
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
>
> Router not send entire path information to namenode because
> ClientProtocol#complete method`s parameter with fileId. Then NameNode will
> log incorrect path information. This is very confusing, should we let the
> router pass the path information or modify the log path on namenode?
> completeFile log as fllow:
> StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16209) Set dfs.namenode.caching.enabled to false as default
[ https://issues.apache.org/jira/browse/HDFS-16209?focusedWorklogId=646808&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646808 ] ASF GitHub Bot logged work on HDFS-16209: - Author: ASF GitHub Bot Created on: 06/Sep/21 09:08 Start Date: 06/Sep/21 09:08 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3378: URL: https://github.com/apache/hadoop/pull/3378#issuecomment-913481069 @tomscut @virajjasani, Thanks. I think It's a good way to add description here. BTW, change the title of JIRA and PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 646808) Time Spent: 1h 50m (was: 1h 40m) > Set dfs.namenode.caching.enabled to false as default > > > Key: HDFS-16209 > URL: https://issues.apache.org/jira/browse/HDFS-16209 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Namenode config: > dfs.namenode.write-lock-reporting-threshold-ms=50ms > dfs.namenode.caching.enabled=true (default) > > In fact, the caching feature is not used in our cluster, but this switch is > turned on by default(dfs.namenode.caching.enabled=true), incurring some > additional write lock overhead. We count the number of write lock warnings in > a log file, and find that the number of rescan cache warnings reaches about > 32%, which greatly affects the performance of Namenode. > !namenode-write-lock.jpg! > > We should set 'dfs.namenode.caching.enabled' to false by default and turn it > on when we wants to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16210) Add the option of refreshCallQueue to RouterAdmin
[
https://issues.apache.org/jira/browse/HDFS-16210?focusedWorklogId=646801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646801
]
ASF GitHub Bot logged work on HDFS-16210:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 09:00
Start Date: 06/Sep/21 09:00
Worklog Time Spent: 10m
Work Description: ferhui commented on a change in pull request #3379:
URL: https://github.com/apache/hadoop/pull/3379#discussion_r702730385
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1258,6 +1262,39 @@ public int genericRefresh(String[] argv, int i) throws
IOException {
}
}
+ /**
+ * Refresh Router's call Queue.
+ *
+ * @throws IOException if the operation was not successful.
+ */
+ private int refreshCallQueue() throws IOException {
+Configuration conf = getConf();
+String hostport = getConf().getTrimmed(
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
+
+// Create the client
+Class xface = RefreshCallQueueProtocolPB.class;
+InetSocketAddress address = NetUtils.createSocketAddr(hostport);
+UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+
+RPC.setProtocolEngine(conf, xface, ProtobufRpcEngine2.class);
+RefreshCallQueueProtocolPB proxy =
(RefreshCallQueueProtocolPB)RPC.getProxy(
+xface, RPC.getProtocolVersion(xface), address, ugi, conf,
+NetUtils.getDefaultSocketFactory(conf), 0);
+
+int returnCode = -1;
+try (RefreshCallQueueProtocolClientSideTranslatorPB xlator =
+new RefreshCallQueueProtocolClientSideTranslatorPB(proxy)) {
+ xlator.refreshCallQueue();
+ System.out.println("Refresh call queue successful for " + hostport);
Review comment:
Uh, it's not my native language. Maybe adverb is more suitable here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646801)
Time Spent: 1h 10m (was: 1h)
> Add the option of refreshCallQueue to RouterAdmin
> -
>
> Key: HDFS-16210
> URL: https://issues.apache.org/jira/browse/HDFS-16210
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Janus Chow
>Assignee: Janus Chow
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We enabled FairCallQueue to RouterRpcServer, but Router can not
> refreshCallQueue as NameNode does.
> This ticket is to enable the refreshCallQueue for Router so that we don't
> have to restart the Routers when updating FairCallQueue configurations.
>
> The option is not to refreshCallQueue to NameNodes, just trying to refresh
> the callQueue of Router itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16210) Add the option of refreshCallQueue to RouterAdmin
[
https://issues.apache.org/jira/browse/HDFS-16210?focusedWorklogId=646799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646799
]
ASF GitHub Bot logged work on HDFS-16210:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 08:49
Start Date: 06/Sep/21 08:49
Worklog Time Spent: 10m
Work Description: symious commented on a change in pull request #3379:
URL: https://github.com/apache/hadoop/pull/3379#discussion_r702721915
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1258,6 +1262,39 @@ public int genericRefresh(String[] argv, int i) throws
IOException {
}
}
+ /**
+ * Refresh Router's call Queue.
+ *
+ * @throws IOException if the operation was not successful.
+ */
+ private int refreshCallQueue() throws IOException {
+Configuration conf = getConf();
+String hostport = getConf().getTrimmed(
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
+
+// Create the client
+Class xface = RefreshCallQueueProtocolPB.class;
+InetSocketAddress address = NetUtils.createSocketAddr(hostport);
+UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+
+RPC.setProtocolEngine(conf, xface, ProtobufRpcEngine2.class);
+RefreshCallQueueProtocolPB proxy =
(RefreshCallQueueProtocolPB)RPC.getProxy(
+xface, RPC.getProtocolVersion(xface), address, ugi, conf,
+NetUtils.getDefaultSocketFactory(conf), 0);
+
+int returnCode = -1;
+try (RefreshCallQueueProtocolClientSideTranslatorPB xlator =
+new RefreshCallQueueProtocolClientSideTranslatorPB(proxy)) {
+ xlator.refreshCallQueue();
+ System.out.println("Refresh call queue successful for " + hostport);
Review comment:
@ferhui Thanks for the review.
Just to be sure, do you mean changing "successful" to "successfully" and
"failed" to "unsuccessfully"?
Borrowed from DFSAdmin, but the changes are good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646799)
Time Spent: 1h (was: 50m)
> Add the option of refreshCallQueue to RouterAdmin
> -
>
> Key: HDFS-16210
> URL: https://issues.apache.org/jira/browse/HDFS-16210
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Janus Chow
>Assignee: Janus Chow
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We enabled FairCallQueue to RouterRpcServer, but Router can not
> refreshCallQueue as NameNode does.
> This ticket is to enable the refreshCallQueue for Router so that we don't
> have to restart the Routers when updating FairCallQueue configurations.
>
> The option is not to refreshCallQueue to NameNodes, just trying to refresh
> the callQueue of Router itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16210) Add the option of refreshCallQueue to RouterAdmin
[
https://issues.apache.org/jira/browse/HDFS-16210?focusedWorklogId=646797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-646797
]
ASF GitHub Bot logged work on HDFS-16210:
-
Author: ASF GitHub Bot
Created on: 06/Sep/21 08:41
Start Date: 06/Sep/21 08:41
Worklog Time Spent: 10m
Work Description: ferhui commented on a change in pull request #3379:
URL: https://github.com/apache/hadoop/pull/3379#discussion_r702715835
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##
@@ -1740,6 +1740,15 @@ public void testErrorFaultTolerant() throws Exception {
assertEquals(0, ToolRunner.run(admin, argv));
}
+ @Test
+ public void testRefreshCallQueue() throws Exception {
+
+System.setOut(new PrintStream(out));
+String[] argv = new String[]{"-refreshCallQueue"};
+assertEquals(0, ToolRunner.run(admin, argv));
+assertTrue(out.toString().contains("Refresh call queue successful"));
Review comment:
And here.
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##
@@ -1258,6 +1262,39 @@ public int genericRefresh(String[] argv, int i) throws
IOException {
}
}
+ /**
+ * Refresh Router's call Queue.
+ *
+ * @throws IOException if the operation was not successful.
+ */
+ private int refreshCallQueue() throws IOException {
+Configuration conf = getConf();
+String hostport = getConf().getTrimmed(
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
+RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
+
+// Create the client
+Class xface = RefreshCallQueueProtocolPB.class;
+InetSocketAddress address = NetUtils.createSocketAddr(hostport);
+UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+
+RPC.setProtocolEngine(conf, xface, ProtobufRpcEngine2.class);
+RefreshCallQueueProtocolPB proxy =
(RefreshCallQueueProtocolPB)RPC.getProxy(
+xface, RPC.getProtocolVersion(xface), address, ugi, conf,
+NetUtils.getDefaultSocketFactory(conf), 0);
+
+int returnCode = -1;
+try (RefreshCallQueueProtocolClientSideTranslatorPB xlator =
+new RefreshCallQueueProtocolClientSideTranslatorPB(proxy)) {
+ xlator.refreshCallQueue();
+ System.out.println("Refresh call queue successful for " + hostport);
Review comment:
here successfully？the bellow should be unsuccessfully?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 646797)
Time Spent: 50m (was: 40m)
> Add the option of refreshCallQueue to RouterAdmin
> -
>
> Key: HDFS-16210
> URL: https://issues.apache.org/jira/browse/HDFS-16210
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Janus Chow
>Assignee: Janus Chow
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We enabled FairCallQueue to RouterRpcServer, but Router can not
> refreshCallQueue as NameNode does.
> This ticket is to enable the refreshCallQueue for Router so that we don't
> have to restart the Routers when updating FairCallQueue configurations.
>
> The option is not to refreshCallQueue to NameNodes, just trying to refresh
> the callQueue of Router itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
