[jira] [Created] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
Shashikant Banerjee created HDFS-15568: -- Summary: namenode start failed to start when dfs.namenode.snapshot.max.limit set Key: HDFS-15568 URL: https://issues.apache.org/jira/browse/HDFS-15568 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Nilotpal Nandi Assignee: Shashikant Banerjee {code:java} 11:35:05.872 AM ERROR NameNode Failed to start namenode. org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot: there are already 20 snapshot(s) and the max snapshot limit is 20 at org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181) at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285) at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:960) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:933) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737) {code} Steps to reproduce: -- directory level snapshot limit set - 100 Created 100 snapshots deleted all 100 snapshots (in-oder) No snapshot exist Then, directory level snapshot limit set - 20 HDFS restart Namenode start failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[ https://issues.apache.org/jira/browse/HDFS-15568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15568: -- Labels: pull-request-available (was: ) > namenode start failed to start when dfs.namenode.snapshot.max.limit set > --- > > Key: HDFS-15568 > URL: https://issues.apache.org/jira/browse/HDFS-15568 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Nilotpal Nandi >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > {code:java} > 11:35:05.872 AM ERROR NameNode > Failed to start namenode. > org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot: > there are already 20 snapshot(s) and the max snapshot limit is 20 > at > org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285) > at > org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:960) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:933) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737) > {code} > Steps to reproduce: > -- > directory level snapshot limit set - 100 > Created 100 snapshots > deleted all 100 snapshots (in-oder) > No snapshot exist > Then, directory level snapshot limit set - 20 > HDFS restart > Namenode start failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[ https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481302&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481302 ] ASF GitHub Bot logged work on HDFS-15568: - Author: ASF GitHub Bot Created on: 10/Sep/20 08:19 Start Date: 10/Sep/20 08:19 Worklog Time Spent: 10m Work Description: bshashikant opened a new pull request #2296: URL: https://github.com/apache/hadoop/pull/2296 please see https://issues.apache.org/jira/browse/HDFS-15568 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481302) Remaining Estimate: 0h Time Spent: 10m > namenode start failed to start when dfs.namenode.snapshot.max.limit set > --- > > Key: HDFS-15568 > URL: https://issues.apache.org/jira/browse/HDFS-15568 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Nilotpal Nandi >Assignee: Shashikant Banerjee >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > {code:java} > 11:35:05.872 AM ERROR NameNode > Failed to start namenode. > org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot: > there are already 20 snapshot(s) and the max snapshot limit is 20 > at > org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285) > at > org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:960) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:933) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737) > {code} > Steps to reproduce: > -- > directory level snapshot limit set - 100 > Created 100 snapshots > deleted all 100 snapshots (in-oder) > No snapshot exist > Then, directory level snapshot limit set - 20 > HDFS restart > Namenode start failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15548) Allow configuring DISK/ARCHIVE storage types on same device mount
[ https://issues.apache.org/jira/browse/HDFS-15548?focusedWorklogId=481308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481308 ] ASF GitHub Bot logged work on HDFS-15548: - Author: ASF GitHub Bot Created on: 10/Sep/20 08:41 Start Date: 10/Sep/20 08:41 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2288: URL: https://github.com/apache/hadoop/pull/2288#issuecomment-690086118 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 31m 22s | trunk passed | | +1 :green_heart: | compile | 1m 18s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 9s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 56s | trunk passed | | +1 :green_heart: | mvnsite | 1m 18s | trunk passed | | +1 :green_heart: | shadedclient | 18m 7s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 52s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 20s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 8s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 11s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 1m 9s | hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 2 new + 602 unchanged - 0 fixed = 604 total (was 602) | | +1 :green_heart: | compile | 1m 2s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 1m 2s | hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 2 new + 586 unchanged - 0 fixed = 588 total (was 586) | | -0 :warning: | checkstyle | 0m 50s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 569 unchanged - 0 fixed = 576 total (was 569) | | +1 :green_heart: | mvnsite | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 15m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 15s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 13s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 113m 15s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 202m 26s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2288/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2288 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 64599a30402d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[jira] [Commented] (HDFS-14997) BPServiceActor processes commands from NameNode asynchronously
[ https://issues.apache.org/jira/browse/HDFS-14997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193490#comment-17193490 ] Stephen O'Donnell commented on HDFS-14997: -- [~Captainhzy] Do you have this patch included on your cluster and you see the heartbeat thread blocked for a long time? What type of command is holding the lock? Do you have any jstack traces showing the problem? > BPServiceActor processes commands from NameNode asynchronously > -- > > Key: HDFS-14997 > URL: https://issues.apache.org/jira/browse/HDFS-14997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Xiaoqiao He >Assignee: Xiaoqiao He >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14997.001.patch, HDFS-14997.002.patch, > HDFS-14997.003.patch, HDFS-14997.004.patch, HDFS-14997.005.patch, > HDFS-14997.addendum.patch, image-2019-12-26-16-15-44-814.png > > > There are two core functions, report(#sendHeartbeat, #blockReport, > #cacheReport) and #processCommand in #BPServiceActor main process flow. If > processCommand cost long time it will block send report flow. Meanwhile > processCommand could cost long time(over 1000s the worst case I meet) when IO > load of DataNode is very high. Since some IO operations are under > #datasetLock, So it has to wait to acquire #datasetLock long time when > process some of commands(such as #DNA_INVALIDATE). In such case, #heartbeat > will not send to NameNode in-time, and trigger other disasters. > I propose to improve #processCommand asynchronously and not block > #BPServiceActor to send heartbeat back to NameNode when meet high IO load. > Notes: > 1. Lifeline could be one effective solution, however some old branches are > not support this feature. > 2. IO operations under #datasetLock is another issue, I think we should solve > it at another JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[ https://issues.apache.org/jira/browse/HDFS-15563?focusedWorklogId=481356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481356 ] ASF GitHub Bot logged work on HDFS-15563: - Author: ASF GitHub Bot Created on: 10/Sep/20 10:25 Start Date: 10/Sep/20 10:25 Worklog Time Spent: 10m Work Description: bshashikant merged pull request #2295: URL: https://github.com/apache/hadoop/pull/2295 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481356) Time Spent: 0.5h (was: 20m) > Incorrect getTrashRoot return value when a non-snapshottable dir prefix > matches the path of a snapshottable dir > --- > > Key: HDFS-15563 > URL: https://issues.apache.org/jira/browse/HDFS-15563 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Affects Versions: 3.4.0 >Reporter: Nilotpal Nandi >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is > enabled. > Root cause analysis: > {{SnapshottableDirectoryStatus}} paths retrived inside > {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some > directories with the same path prefix to be mistakenly classified as > snapshottable directory. > Thanks [~shashikant] for the test case addition. > --- > Repro: > {code:java} > 1. snapshottable directory present in the cluster > hdfs lsSnapshottableDir > drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2 > drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 > /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable > directory > hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it > failed > hdfs dfs -rm -r /user/hrt_4/newdir/subdir2 > rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source > /user/hrt_4/newdir/subdir2 and dest > /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are > not under the same snapshot root. > {code} > "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be > "*/user/hrt_4/newdir/subdir/.Trash*" > as clear from the msg here: > {noformat} > rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source > /user/hrt_4/newdir/subdir2 and dest > /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are > not under the same snapshot root.{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[ https://issues.apache.org/jira/browse/HDFS-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15563: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Incorrect getTrashRoot return value when a non-snapshottable dir prefix > matches the path of a snapshottable dir > --- > > Key: HDFS-15563 > URL: https://issues.apache.org/jira/browse/HDFS-15563 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Affects Versions: 3.4.0 >Reporter: Nilotpal Nandi >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is > enabled. > Root cause analysis: > {{SnapshottableDirectoryStatus}} paths retrived inside > {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some > directories with the same path prefix to be mistakenly classified as > snapshottable directory. > Thanks [~shashikant] for the test case addition. > --- > Repro: > {code:java} > 1. snapshottable directory present in the cluster > hdfs lsSnapshottableDir > drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2 > drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 > /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable > directory > hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it > failed > hdfs dfs -rm -r /user/hrt_4/newdir/subdir2 > rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source > /user/hrt_4/newdir/subdir2 and dest > /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are > not under the same snapshot root. > {code} > "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be > "*/user/hrt_4/newdir/subdir/.Trash*" > as clear from the msg here: > {noformat} > rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source > /user/hrt_4/newdir/subdir2 and dest > /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are > not under the same snapshot root.{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15563) Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir
[ https://issues.apache.org/jira/browse/HDFS-15563?focusedWorklogId=481372&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481372 ] ASF GitHub Bot logged work on HDFS-15563: - Author: ASF GitHub Bot Created on: 10/Sep/20 11:08 Start Date: 10/Sep/20 11:08 Worklog Time Spent: 10m Work Description: bshashikant commented on pull request #2295: URL: https://github.com/apache/hadoop/pull/2295#issuecomment-690161198 Thanks @smengcl for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481372) Time Spent: 40m (was: 0.5h) > Incorrect getTrashRoot return value when a non-snapshottable dir prefix > matches the path of a snapshottable dir > --- > > Key: HDFS-15563 > URL: https://issues.apache.org/jira/browse/HDFS-15563 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Affects Versions: 3.4.0 >Reporter: Nilotpal Nandi >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Note: Only impacts a user if {{dfs.namenode.snapshot.trashroot.enabled}} is > enabled. > Root cause analysis: > {{SnapshottableDirectoryStatus}} paths retrived inside > {{DFSClient#getSnapshotRoot}} aren't appended with '/', causing some > directories with the same path prefix to be mistakenly classified as > snapshottable directory. > Thanks [~shashikant] for the test case addition. > --- > Repro: > {code:java} > 1. snapshottable directory present in the cluster > hdfs lsSnapshottableDir > drwx-x-x 0 hrt_2 hrt_2 0 2020-09-08 07:42 0 65536 /user/hrt_2 > drwxr-xr-x 0 hrt_4 hrt_4 0 2020-09-08 13:16 0 65536 > /user/hrt_4/newdir/subdir2. Created a new directory outside snapshottable > directory > hdfs dfs -mkdir /user/hrt_4/newdir/subdir23. Tried to delete subdir2 , it > failed > hdfs dfs -rm -r /user/hrt_4/newdir/subdir2 > rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source > /user/hrt_4/newdir/subdir2 and dest > /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are > not under the same snapshot root. > {code} > "*/user/hrt_4/newdir/subdir2*" , the trash root location is coming to be > "*/user/hrt_4/newdir/subdir/.Trash*" > as clear from the msg here: > {noformat} > rm: Failed to move to trash: hdfs://ns1/user/hrt_4/newdir/subdir2: Source > /user/hrt_4/newdir/subdir2 and dest > /user/hrt_4/newdir/subdir/.Trash/hdfs/Current/user/hrt_4/newdir/subdir2 are > not under the same snapshot root.{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[ https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481397 ] ASF GitHub Bot logged work on HDFS-15568: - Author: ASF GitHub Bot Created on: 10/Sep/20 11:49 Start Date: 10/Sep/20 11:49 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2296: URL: https://github.com/apache/hadoop/pull/2296#issuecomment-690215954 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 40m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 32m 7s | trunk passed | | +1 :green_heart: | compile | 1m 16s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 1m 8s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 47s | trunk passed | | +1 :green_heart: | mvnsite | 1m 15s | trunk passed | | +1 :green_heart: | shadedclient | 17m 51s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 50s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 19s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 11s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 8s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 1m 11s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 1m 1s | the patch passed | | -0 :warning: | checkstyle | 0m 40s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 89 unchanged - 0 fixed = 92 total (was 89) | | +1 :green_heart: | mvnsite | 1m 10s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 27s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 16s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 3m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 80m 10s | hadoop-hdfs in the patch passed. | | -1 :x: | asflicense | 0m 35s | The patch generated 4 ASF License warnings. | | | | 209m 18s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS | | | hadoop.fs.viewfs.TestViewFileSystemHdfs | | | hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash | | | hadoop.hdfs.TestSnapshotCommands | | | hadoop.fs.viewfs.TestViewFsAtHdfsRoot | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2296/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2296 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 94a92cfd8c29 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / e5fe3262702 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | checkstyle | https://ci-hadoop.apa
[jira] [Commented] (HDFS-15516) Add info for create flags in NameNode audit logs
[ https://issues.apache.org/jira/browse/HDFS-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193586#comment-17193586 ] jianghua zhu commented on HDFS-15516: - [~linyiqun], [~sunchao] , do you have other ideas to share? > Add info for create flags in NameNode audit logs > > > Key: HDFS-15516 > URL: https://issues.apache.org/jira/browse/HDFS-15516 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Shashikant Banerjee >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15516.001.patch, HDFS-15516.002.patch, > HDFS-15516.003.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, if file create happens with flags like overwrite , the audit logs > doesn't seem to contain the info regarding the flags in the audit logs. It > would be useful to add info regarding the create options in the audit logs > similar to Rename ops. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15516) Add info for create flags in NameNode audit logs
[ https://issues.apache.org/jira/browse/HDFS-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193586#comment-17193586 ] jianghua zhu edited comment on HDFS-15516 at 9/10/20, 12:35 PM: [~linyiqun], [~chaosun] , do you have other ideas to share? was (Author: jianghuazhu): [~linyiqun], [~sunchao] , do you have other ideas to share? > Add info for create flags in NameNode audit logs > > > Key: HDFS-15516 > URL: https://issues.apache.org/jira/browse/HDFS-15516 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Shashikant Banerjee >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15516.001.patch, HDFS-15516.002.patch, > HDFS-15516.003.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, if file create happens with flags like overwrite , the audit logs > doesn't seem to contain the info regarding the flags in the audit logs. It > would be useful to add info regarding the create options in the audit logs > similar to Rename ops. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
Hemanth Boyina created HDFS-15569: - Summary: Speed up the Storage#doRecover during datanode rolling upgrade Key: HDFS-15569 URL: https://issues.apache.org/jira/browse/HDFS-15569 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hemanth Boyina Assignee: Hemanth Boyina When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not having enough memory upgrade failed , Adjusted memory configurations and re upgraded datanode , Now datanode upgrade has taken more time , on analyzing found that Storage#deleteDir has taken more time in Recover_Upgrade state -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
[ https://issues.apache.org/jira/browse/HDFS-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193701#comment-17193701 ] Hemanth Boyina commented on HDFS-15569: --- One solution can be rename the current directory to current.tmp directory and delete the current.tmp in parallel , as the current directory doesn't exists now datanode upgrade can be proceeded > Speed up the Storage#doRecover during datanode rolling upgrade > --- > > Key: HDFS-15569 > URL: https://issues.apache.org/jira/browse/HDFS-15569 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Major > > When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not > having enough memory upgrade failed , Adjusted memory configurations and re > upgraded datanode , > Now datanode upgrade has taken more time , on analyzing found that > Storage#deleteDir has taken more time in Recover_Upgrade state -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
[ https://issues.apache.org/jira/browse/HDFS-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HDFS-15569: -- Description: When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not having enough memory upgrade failed , Adjusted memory configurations and re upgraded datanode , Now datanode upgrade has taken more time , on analyzing found that Storage#deleteDir has taken more time in RECOVER_UPGRADE state {code:java} "Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000]"Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000] java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.delete0(Native Method) at java.io.UnixFileSystem.delete(UnixFileSystem.java:265) at java.io.File.delete(File.java:1041) at org.apache.hadoop.fs.FileUtil.deleteImpl(FileUtil.java:229) at org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:270) at org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:153) at org.apache.hadoop.hdfs.server.common.Storage.deleteDir(Storage.java:1348) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.doRecover(Storage.java:782) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:174) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:224) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:253) at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:455) at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:389) - locked <0x7fdf08ec7548> (a org.apache.hadoop.hdfs.server.datanode.DataStorage) at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:557) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1761) - locked <0x7fdf08ec7598> (a org.apache.hadoop.hdfs.server.datanode.DataNode) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1697) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:822) at java.lang.Thread.run(Thread.java:748) {code} was: When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not having enough memory upgrade failed , Adjusted memory configurations and re upgraded datanode , Now datanode upgrade has taken more time , on analyzing found that Storage#deleteDir has taken more time in Recover_Upgrade state > Speed up the Storage#doRecover during datanode rolling upgrade > --- > > Key: HDFS-15569 > URL: https://issues.apache.org/jira/browse/HDFS-15569 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Major > > When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not > having enough memory upgrade failed , Adjusted memory configurations and re > upgraded datanode , > Now datanode upgrade has taken more time , on analyzing found that > Storage#deleteDir has taken more time in RECOVER_UPGRADE state > {code:java} > "Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c > runnable [0x7fdcdad2a000]"Thread-28" #270 daemon prio=5 os_prio=0 > tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000] > java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.delete0(Native > Method) at java.io.UnixFileSystem.delete(UnixFileSystem.java:265) at > java.io.File.delete(File.java:1041) at > org.apache.hadoop.fs.FileUtil.deleteImpl(FileUtil.java:229) at > org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:270) at > org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at > org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at > org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at > org.apache.hadoop.fs.FileUtil.fullyDelet
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481626&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481626 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:07 Start Date: 10/Sep/20 18:07 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486535639 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java ## @@ -34,28 +34,35 @@ @InterfaceStability.Unstable public enum StorageType { // sorted by the speed of the storage types, from fast to slow - RAM_DISK(true), - SSD(false), - DISK(false), - ARCHIVE(false), - PROVIDED(false); + RAM_DISK(true, true), + NVDIMM(false, true), + SSD(false, false), + DISK(false, false), + ARCHIVE(false, false), + PROVIDED(false, false); private final boolean isTransient; + private final boolean isRAM; public static final StorageType DEFAULT = DISK; public static final StorageType[] EMPTY_ARRAY = {}; private static final StorageType[] VALUES = values(); - StorageType(boolean isTransient) { + StorageType(boolean isTransient, boolean isRAM) { this.isTransient = isTransient; +this.isRAM = isRAM; } public boolean isTransient() { return isTransient; } + public boolean isRAM() { +return isRAM; + } Review comment: ok, By design if you dn't want to move. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481626) Time Spent: 3h 20m (was: 3h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[ https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-15565: - Fix Version/s: (was: 3.3.0) > Remove the invalid code in the Balancer#doBalance() method. > --- > > Key: HDFS-15565 > URL: https://issues.apache.org/jira/browse/HDFS-15565 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15565.001.patch > > Time Spent: 50m > Remaining Estimate: 0h > > In the Balancer#doBalance() method, an invalid line of code is added, as > follows: > static private int doBalance(Collection namenodes, > Collection nsIds, final BalancerParameters p, Configuration conf) > throws IOException, InterruptedException > { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes > Left To Move Bytes Being Moved"); ... } > > I think it was originally used for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Closed] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[ https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu closed HDFS-15565. > Remove the invalid code in the Balancer#doBalance() method. > --- > > Key: HDFS-15565 > URL: https://issues.apache.org/jira/browse/HDFS-15565 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15565.001.patch > > Time Spent: 50m > Remaining Estimate: 0h > > In the Balancer#doBalance() method, an invalid line of code is added, as > follows: > static private int doBalance(Collection namenodes, > Collection nsIds, final BalancerParameters p, Configuration conf) > throws IOException, InterruptedException > { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes > Left To Move Bytes Being Moved"); ... } > > I think it was originally used for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15565) Remove the invalid code in the Balancer#doBalance() method.
[ https://issues.apache.org/jira/browse/HDFS-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193774#comment-17193774 ] Mingliang Liu commented on HDFS-15565: -- Closing this issue and removing the fix versions. > Remove the invalid code in the Balancer#doBalance() method. > --- > > Key: HDFS-15565 > URL: https://issues.apache.org/jira/browse/HDFS-15565 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15565.001.patch > > Time Spent: 50m > Remaining Estimate: 0h > > In the Balancer#doBalance() method, an invalid line of code is added, as > follows: > static private int doBalance(Collection namenodes, > Collection nsIds, final BalancerParameters p, Configuration conf) > throws IOException, InterruptedException > { ... System.out.println("Time Stamp Iteration# Bytes Already Moved Bytes > Left To Move Bytes Being Moved"); ... } > > I think it was originally used for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481637&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481637 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:21 Start Date: 10/Sep/20 18:21 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486545111 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: Ok. Got it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481637) Time Spent: 3.5h (was: 3h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481640 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:25 Start Date: 10/Sep/20 18:25 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486548053 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: So, NVDIMM is peristent storage and RAM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481640) Time Spent: 3h 40m (was: 3.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481641 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 10/Sep/20 18:27 Start Date: 10/Sep/20 18:27 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486549682 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java ## @@ -34,28 +34,35 @@ @InterfaceStability.Unstable public enum StorageType { // sorted by the speed of the storage types, from fast to slow - RAM_DISK(true), - SSD(false), - DISK(false), - ARCHIVE(false), - PROVIDED(false); + RAM_DISK(true, true), + NVDIMM(false, true), + SSD(false, false), + DISK(false, false), + ARCHIVE(false, false), + PROVIDED(false, false); private final boolean isTransient; + private final boolean isRAM; public static final StorageType DEFAULT = DISK; public static final StorageType[] EMPTY_ARRAY = {}; private static final StorageType[] VALUES = values(); - StorageType(boolean isTransient) { + StorageType(boolean isTransient, boolean isRAM) { this.isTransient = isTransient; +this.isRAM = isRAM; } public boolean isTransient() { return isTransient; } + public boolean isRAM() { +return isRAM; + } Review comment: My final query then, why can't have one NVDIMM like one SSD as this also movable and peristent..? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481641) Time Spent: 3h 50m (was: 3h 40m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[ https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481660 ] ASF GitHub Bot logged work on HDFS-15568: - Author: ASF GitHub Bot Created on: 10/Sep/20 19:21 Start Date: 10/Sep/20 19:21 Worklog Time Spent: 10m Work Description: goiri commented on a change in pull request #2296: URL: https://github.com/apache/hadoop/pull/2296#discussion_r486579348 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java ## @@ -508,6 +508,10 @@ FSNamesystem getFSNamesystem() { return namesystem; } + public boolean isImageLoaded() { Review comment: Add javadoc ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java ## @@ -368,6 +368,13 @@ void assertFirstSnapshot(INodeDirectory dir, } } + boolean captureOpenFiles() { Review comment: javadoc ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java ## @@ -368,6 +368,13 @@ void assertFirstSnapshot(INodeDirectory dir, } } + boolean captureOpenFiles() { +return captureOpenFiles; + } + + int getMaxSnapshotLimit() { Review comment: VisibleForTesting ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java ## @@ -133,4 +137,68 @@ public void testValidateSnapshotIDWidth() throws Exception { getMaxSnapshotID() < Snapshot.CURRENT_STATE_ID); } + @Test + public void SnapshotLimitOnRestart() throws Exception { +final Configuration conf = new Configuration(); +final Path snapshottableDir += new Path("/" + getClass().getSimpleName()); +int numSnapshots = 5; +conf.setInt(DFSConfigKeys. +DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, numSnapshots); +conf.setInt(DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT, +numSnapshots * 2); +MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf). +numDataNodes(0).build(); +cluster.waitActive(); +DistributedFileSystem hdfs = cluster.getFileSystem(); +hdfs.mkdirs(snapshottableDir); +hdfs.allowSnapshot(snapshottableDir); +int i = 0; +for (; i < numSnapshots; i++) { + hdfs.createSnapshot(snapshottableDir, "s" + i); +} +try { + hdfs.createSnapshot(snapshottableDir, "s" + i); + Assert.fail("Expected SnapshotException not thrown"); +} catch (SnapshotException se) { + Assert.assertTrue( + StringUtils.toLowerCase(se.getMessage()).contains( + "max snapshot limit")); +} + +// now change max snapshot directory limit to 2 and restart namenode +cluster.getNameNode().getConf().setInt(DFSConfigKeys. +DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, 2); +cluster.restartNameNodes(); + +// make sure edits of all previous 5 create snapshots are replayed +Assert.assertEquals(numSnapshots, cluster.getNamesystem(). +getSnapshotManager().getNumSnapshots()); + +// make sure namenode has the new snapshot limit configured as 2 +Assert.assertEquals(2, +cluster.getNamesystem().getSnapshotManager().getMaxSnapshotLimit()); + +// Any new snapshot creation should still fail +try { + hdfs.createSnapshot(snapshottableDir, "s" + i); + Assert.fail("Expected SnapshotException not thrown"); +} catch (SnapshotException se) { + Assert.assertTrue( Review comment: LambdaTestUtils ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotManager.java ## @@ -133,4 +137,68 @@ public void testValidateSnapshotIDWidth() throws Exception { getMaxSnapshotID() < Snapshot.CURRENT_STATE_ID); } + @Test + public void SnapshotLimitOnRestart() throws Exception { +final Configuration conf = new Configuration(); +final Path snapshottableDir += new Path("/" + getClass().getSimpleName()); +int numSnapshots = 5; +conf.setInt(DFSConfigKeys. +DFS_NAMENODE_SNAPSHOT_MAX_LIMIT, numSnapshots); +conf.setInt(DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_FILESYSTEM_LIMIT, +numSnapshots * 2); +MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf). +numDataNodes(0).build(); +cluster.waitActive(); +DistributedFileSystem hdfs = cluster.getFileSystem(); +hdfs.mkdirs(snapshottableDir); +hdfs.allowSnapshot(snapshottableDir); +int i = 0; +for (; i < numSnapshots; i++) { + hdfs.createSnapshot(snapshottableDir, "s" + i); +} +try { + hdfs.createSnapshot(snapshottableDir, "s" + i); + Assert.fail("Expected SnapshotExce
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193808#comment-17193808 ] Íñigo Goiri commented on HDFS-13293: Let's try, but keep in mind that toucking something like CallerContext in common is usually controversial. > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Fei Hui >Priority: Major > Attachments: HDFS-13293.001.patch > > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15568) namenode start failed to start when dfs.namenode.snapshot.max.limit set
[ https://issues.apache.org/jira/browse/HDFS-15568?focusedWorklogId=481668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481668 ] ASF GitHub Bot logged work on HDFS-15568: - Author: ASF GitHub Bot Created on: 10/Sep/20 19:49 Start Date: 10/Sep/20 19:49 Worklog Time Spent: 10m Work Description: szetszwo commented on a change in pull request #2296: URL: https://github.com/apache/hadoop/pull/2296#discussion_r486594478 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java ## @@ -448,22 +455,31 @@ public String createSnapshot(final LeaseManager leaseManager, "snapshot IDs and ID rollover is not supported."); } int n = numSnapshots.get(); -if (n >= maxSnapshotFSLimit) { - // We have reached the maximum snapshot limit - throw new SnapshotException( - "Failed to create snapshot: there are already " + (n + 1) - + " snapshot(s) and the max snapshot limit is " - + maxSnapshotFSLimit); -} - -srcRoot.addSnapshot(snapshotCounter, snapshotName, leaseManager, -this.captureOpenFiles, maxSnapshotLimit, mtime); +checkSnapshotLimit(maxSnapshotFSLimit, n); +srcRoot.addSnapshot(this, snapshotName, leaseManager, mtime); //create success, update id snapshotCounter++; numSnapshots.getAndIncrement(); return Snapshot.getSnapshotPath(snapshotRoot, snapshotName); } + + void checkSnapshotLimit(int limit, int numSnapshots) Review comment: I suggest to add limit type to the error message as below. ``` diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java index 266c0a71241..7a47ab4000d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java @@ -190,8 +190,7 @@ public Snapshot addSnapshot(INodeDirectory snapshotRoot, + n + " snapshot(s) and the snapshot quota is " + snapshotQuota); } -snapshotManager.checkSnapshotLimit(snapshotManager. -getMaxSnapshotLimit(), n); +snapshotManager.checkPerDirectorySnapshotLimit(n); final Snapshot s = new Snapshot(id, name, snapshotRoot); final byte[] nameBytes = s.getRoot().getLocalNameBytes(); final int i = searchSnapshot(nameBytes); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java index 0a2e18c3dc3..7c482074486 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java @@ -455,7 +455,7 @@ public String createSnapshot(final LeaseManager leaseManager, "snapshot IDs and ID rollover is not supported."); } int n = numSnapshots.get(); -checkSnapshotLimit(maxSnapshotFSLimit, n); +checkFileSystemSnapshotLimit(n); srcRoot.addSnapshot(this, snapshotName, leaseManager, mtime); //create success, update id @@ -464,12 +464,19 @@ public String createSnapshot(final LeaseManager leaseManager, return Snapshot.getSnapshotPath(snapshotRoot, snapshotName); } - void checkSnapshotLimit(int limit, int numSnapshots) - throws SnapshotException { + void checkFileSystemSnapshotLimit(int n) throws SnapshotException { +checkSnapshotLimit(maxSnapshotFSLimit, n, "file system"); + } + + void checkPerDirectorySnapshotLimit(int n) throws SnapshotException { +checkSnapshotLimit(maxSnapshotLimit, n, "per directory"); + } + + private void checkSnapshotLimit(int limit, int numSnapshots, + String type) throws SnapshotException { if (numSnapshots >= limit) { - String msg = "there are already " + (numSnapshots + 1) - + " snapshot(s) and the max snapshot limit is " - + limit; + String msg = "There are already " + (numSnapshots + 1) + + " snapshot(s) and the " + type + " snapshot limit is " + limit; if (fsdir.isImageLoaded()) { // We have reached the maximum snapshot limit throw new SnapshotException(
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=481758&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481758 ] ASF GitHub Bot logged work on HDFS-15554: - Author: ASF GitHub Bot Created on: 10/Sep/20 22:32 Start Date: 10/Sep/20 22:32 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2266: URL: https://github.com/apache/hadoop/pull/2266#issuecomment-690766821 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 33m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 29m 21s | trunk passed | | +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | trunk passed | | +1 :green_heart: | shadedclient | 15m 26s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 54s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 10s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 31s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 0m 32s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 0 new + 30 unchanged - 2 fixed = 30 total (was 32) | | +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 0m 28s | hadoop-hdfs-project_hadoop-hdfs-rbf-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 0 new + 30 unchanged - 2 fixed = 30 total (was 32) | | +1 :green_heart: | checkstyle | 0m 17s | the patch passed | | +1 :green_heart: | mvnsite | 0m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 13m 56s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 1m 14s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 8m 39s | hadoop-hdfs-rbf in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 114m 33s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2266/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2266 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint | | uname | Linux 490356438ed4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 9960c01a25c | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[ https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481795 ] ASF GitHub Bot logged work on HDFS-15539: - Author: ASF GitHub Bot Created on: 10/Sep/20 23:51 Start Date: 10/Sep/20 23:51 Worklog Time Spent: 10m Work Description: smengcl commented on a change in pull request #2258: URL: https://github.com/apache/hadoop/pull/2258#discussion_r486694935 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java ## @@ -2442,4 +2442,38 @@ public void testGetTrashRootOnEZInSnapshottableDir() } } } + + @Test + public void testDisallowSnapshotShouldThrowWhenTrashRootExists() + throws IOException { +Configuration conf = getTestConfiguration(); +MiniDFSCluster cluster = +new MiniDFSCluster.Builder(conf).numDataNodes(1).build(); +try { + DistributedFileSystem dfs = cluster.getFileSystem(); + Path testDir = new Path("/disallowss/test1/"); + Path file0path = new Path(testDir, "file-0"); + dfs.create(file0path); + dfs.allowSnapshot(testDir); + // Create trash root manually + Path testDirTrashRoot = new Path(testDir, FileSystem.TRASH_PREFIX); + dfs.mkdirs(testDirTrashRoot); + // Try disallowing snapshot, should throw + try { +dfs.disallowSnapshot(testDir); +fail("Should have thrown IOException when trash root exists inside " Review comment: Thanks! I have updated accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481795) Time Spent: 50m (was: 40m) > When disallowing snapshot on a dir, throw exception if its trash root is not > empty > -- > > Key: HDFS-15539 > URL: https://issues.apache.org/jira/browse/HDFS-15539 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the > trash root in that dir anymore (if any). The risk is the trash root will be > left there forever. > We need to throw an exception there and prompt the user to clean up or rename > the trash root if it is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15554) RBF: force router check file existence in destinations before adding/updating mount points
[ https://issues.apache.org/jira/browse/HDFS-15554?focusedWorklogId=481828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481828 ] ASF GitHub Bot logged work on HDFS-15554: - Author: ASF GitHub Bot Created on: 11/Sep/20 00:59 Start Date: 11/Sep/20 00:59 Worklog Time Spent: 10m Work Description: fengnanli commented on pull request #2266: URL: https://github.com/apache/hadoop/pull/2266#issuecomment-690809794 The TestRouterRpcMultiDestination test passed locally. @goiri Can you help commit it? Thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481828) Time Spent: 3h 20m (was: 3h 10m) > RBF: force router check file existence in destinations before adding/updating > mount points > -- > > Key: HDFS-15554 > URL: https://issues.apache.org/jira/browse/HDFS-15554 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Adding/Updating mount points right now is only a router action without > validation in the downstream namenodes for the destination files/directories. > In practice we have set up the dangling mount points and when clients call > listStatus they would get the file returned, but then if they try to access > the file FileNotFoundException would be thrown out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481836 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 01:12 Start Date: 11/Sep/20 01:12 Worklog Time Spent: 10m Work Description: YaYun-Wang commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716471 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java ## @@ -34,28 +34,35 @@ @InterfaceStability.Unstable public enum StorageType { // sorted by the speed of the storage types, from fast to slow - RAM_DISK(true), - SSD(false), - DISK(false), - ARCHIVE(false), - PROVIDED(false); + RAM_DISK(true, true), + NVDIMM(false, true), + SSD(false, false), + DISK(false, false), + ARCHIVE(false, false), + PROVIDED(false, false); private final boolean isTransient; + private final boolean isRAM; public static final StorageType DEFAULT = DISK; public static final StorageType[] EMPTY_ARRAY = {}; private static final StorageType[] VALUES = values(); - StorageType(boolean isTransient) { + StorageType(boolean isTransient, boolean isRAM) { this.isTransient = isTransient; +this.isRAM = isRAM; } public boolean isTransient() { return isTransient; } + public boolean isRAM() { +return isRAM; + } Review comment: > My final query then, why can't have one NVDIMM like one SSD as this also movable and peristent..? Considering NVDIMM is faster, so NVDIMM does not use `FsDatasetCache()` which SSD needs in the design. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481836) Time Spent: 4h (was: 3h 50m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481837&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481837 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 01:14 Start Date: 11/Sep/20 01:14 Worklog Time Spent: 10m Work Description: YaYun-Wang commented on a change in pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716810 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java ## @@ -77,6 +77,9 @@ /** Returns true if the volume is NOT backed by persistent storage. */ boolean isTransientStorage(); Review comment: > So, NVDIMM is peristent storage and RAM. yes, that’s right. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481837) Time Spent: 4h 10m (was: 4h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481846 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 02:08 Start Date: 11/Sep/20 02:08 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690829884 @liuml07 and @brahmareddybattula So if all of the codes are ok, would you please to approve? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481846) Time Spent: 4h 20m (was: 4h 10m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481857 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 02:43 Start Date: 11/Sep/20 02:43 Worklog Time Spent: 10m Work Description: liuml07 commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476 Will check again later this week. Ideally we can get a clean QA. Could you check the test failures and make sure they are not related? Thanks, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481857) Time Spent: 4.5h (was: 4h 20m) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15532: --- Summary: listFiles on root/InternalDir will fail if fallback root has file (was: listFiles on root will fail if fallback root has file) > listFiles on root/InternalDir will fail if fallback root has file > - > > Key: HDFS-15532 > URL: https://issues.apache.org/jira/browse/HDFS-15532 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > listFiles implementation gets the RemoteIterator created in > InternalViewFSDirFs as the root is an InternalViewFSDir. > If there is a fallback and a file exist at root level, it would have > collected when collecting locatedStatuses. > When its iterating over to that fallbacks file from RemoteIterator (which > was returned from InternalViewFSDirFs ), iterator's next will will call > getFileBlockLocations if it's a file. > {code:java} > @Override > public LocatedFileStatus next() throws IOException { > System.out.println(this); > if (!hasNext()) { > throw new NoSuchElementException("No more entries in " + f); > } > FileStatus result = stats[i++]; > // for files, use getBlockLocations(FileStatus, int, int) to avoid > // calling getFileStatus(Path) to load the FileStatus again > BlockLocation[] locs = result.isFile() ? > getFileBlockLocations(result, 0, result.getLen()) : > null; > return new LocatedFileStatus(result, locs); > }{code} > > this getFileBlockLocations will be made on InternalViewFSDirFs, as that > Iterator created originally from that fs. > InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases. > It's always expecting "/", this means it always assuming the dir. > But with the fallback and returning Iterator from InternalViewFSDirFs, will > create problems. > Probably we need to handle fallback case in getFileBlockLocations as well.( > Fallback only should be the reason for call coming to InternalViewFSDirFs > with other than "/") > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?focusedWorklogId=481864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481864 ] ASF GitHub Bot logged work on HDFS-15532: - Author: ASF GitHub Bot Created on: 11/Sep/20 03:00 Start Date: 11/Sep/20 03:00 Worklog Time Spent: 10m Work Description: umamaheswararao opened a new pull request #2298: URL: https://github.com/apache/hadoop/pull/2298 https://issues.apache.org/jira/browse/HDFS-15532 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481864) Remaining Estimate: 0h Time Spent: 10m > listFiles on root/InternalDir will fail if fallback root has file > - > > Key: HDFS-15532 > URL: https://issues.apache.org/jira/browse/HDFS-15532 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > listFiles implementation gets the RemoteIterator created in > InternalViewFSDirFs as the root is an InternalViewFSDir. > If there is a fallback and a file exist at root level, it would have > collected when collecting locatedStatuses. > When its iterating over to that fallbacks file from RemoteIterator (which > was returned from InternalViewFSDirFs ), iterator's next will will call > getFileBlockLocations if it's a file. > {code:java} > @Override > public LocatedFileStatus next() throws IOException { > System.out.println(this); > if (!hasNext()) { > throw new NoSuchElementException("No more entries in " + f); > } > FileStatus result = stats[i++]; > // for files, use getBlockLocations(FileStatus, int, int) to avoid > // calling getFileStatus(Path) to load the FileStatus again > BlockLocation[] locs = result.isFile() ? > getFileBlockLocations(result, 0, result.getLen()) : > null; > return new LocatedFileStatus(result, locs); > }{code} > > this getFileBlockLocations will be made on InternalViewFSDirFs, as that > Iterator created originally from that fs. > InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases. > It's always expecting "/", this means it always assuming the dir. > But with the fallback and returning Iterator from InternalViewFSDirFs, will > create problems. > Probably we need to handle fallback case in getFileBlockLocations as well.( > Fallback only should be the reason for call coming to InternalViewFSDirFs > with other than "/") > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15532: -- Labels: pull-request-available (was: ) > listFiles on root/InternalDir will fail if fallback root has file > - > > Key: HDFS-15532 > URL: https://issues.apache.org/jira/browse/HDFS-15532 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > listFiles implementation gets the RemoteIterator created in > InternalViewFSDirFs as the root is an InternalViewFSDir. > If there is a fallback and a file exist at root level, it would have > collected when collecting locatedStatuses. > When its iterating over to that fallbacks file from RemoteIterator (which > was returned from InternalViewFSDirFs ), iterator's next will will call > getFileBlockLocations if it's a file. > {code:java} > @Override > public LocatedFileStatus next() throws IOException { > System.out.println(this); > if (!hasNext()) { > throw new NoSuchElementException("No more entries in " + f); > } > FileStatus result = stats[i++]; > // for files, use getBlockLocations(FileStatus, int, int) to avoid > // calling getFileStatus(Path) to load the FileStatus again > BlockLocation[] locs = result.isFile() ? > getFileBlockLocations(result, 0, result.getLen()) : > null; > return new LocatedFileStatus(result, locs); > }{code} > > this getFileBlockLocations will be made on InternalViewFSDirFs, as that > Iterator created originally from that fs. > InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases. > It's always expecting "/", this means it always assuming the dir. > But with the fallback and returning Iterator from InternalViewFSDirFs, will > create problems. > Probably we need to handle fallback case in getFileBlockLocations as well.( > Fallback only should be the reason for call coming to InternalViewFSDirFs > with other than "/") > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[ https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481869&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481869 ] ASF GitHub Bot logged work on HDFS-15539: - Author: ASF GitHub Bot Created on: 11/Sep/20 03:07 Start Date: 11/Sep/20 03:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2258: URL: https://github.com/apache/hadoop/pull/2258#issuecomment-690845948 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 49s | trunk passed | | +1 :green_heart: | compile | 4m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 4m 15s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 1m 2s | trunk passed | | +1 :green_heart: | mvnsite | 2m 14s | trunk passed | | +1 :green_heart: | shadedclient | 19m 0s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 56s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 28s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 38s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 56s | the patch passed | | +1 :green_heart: | compile | 4m 10s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 4m 10s | the patch passed | | +1 :green_heart: | compile | 3m 46s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 3m 46s | the patch passed | | +1 :green_heart: | checkstyle | 0m 53s | the patch passed | | +1 :green_heart: | mvnsite | 1m 59s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 50s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 20s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 47s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 7m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 77m 54s | hadoop-hdfs in the patch passed. | | +0 :ok: | asflicense | 0m 38s | ASF License check generated no output? | | | | 194m 25s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestReencryption | | | hadoop.hdfs.server.namenode.TestFSEditLogLoader | | | hadoop.hdfs.server.namenode.TestNameNodeReconfigure | | | hadoop.hdfs.server.namenode.TestNamenodeStorageDirectives | | | hadoop.hdfs.server.namenode.TestNameEditsConfigs | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.server.namenode.TestFileLimit | | | hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile | | | hadoop.hdfs.TestAppendDifferentChecksum | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.server.namenode.TestCheckpoint | | | hadoop.hdfs.server.namenode.TestMetadataVersionOutput | | | hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness | | | hadoop.hdfs.server.namenode.TestFSImageWithXAttr | | | hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocksWithRandomECPolicy | | | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier | | | hadoop.hdfs.TestDFSStripedInputStream | | | hadoop.hdfs.server.namenode.TestStoragePolicySatisfierWithHA | | | hado
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[ https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481871 ] ASF GitHub Bot logged work on HDFS-15539: - Author: ASF GitHub Bot Created on: 11/Sep/20 03:12 Start Date: 11/Sep/20 03:12 Worklog Time Spent: 10m Work Description: smengcl commented on pull request #2258: URL: https://github.com/apache/hadoop/pull/2258#issuecomment-690847462 > Thanks @smengcl for working on this,. The test failures like TestDistributedFileSystem#testGetTrashRoots look related. Can you plz verify? I'm checking. Probably need to add a line or two to clean up the old tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481871) Time Spent: 1h 10m (was: 1h) > When disallowing snapshot on a dir, throw exception if its trash root is not > empty > -- > > Key: HDFS-15539 > URL: https://issues.apache.org/jira/browse/HDFS-15539 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the > trash root in that dir anymore (if any). The risk is the trash root will be > left there forever. > We need to throw an exception there and prompt the user to clean up or rename > the trash root if it is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15532) listFiles on root/InternalDir will fail if fallback root has file
[ https://issues.apache.org/jira/browse/HDFS-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15532: --- Affects Version/s: 3.4.0 Status: Patch Available (was: Open) > listFiles on root/InternalDir will fail if fallback root has file > - > > Key: HDFS-15532 > URL: https://issues.apache.org/jira/browse/HDFS-15532 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > listFiles implementation gets the RemoteIterator created in > InternalViewFSDirFs as the root is an InternalViewFSDir. > If there is a fallback and a file exist at root level, it would have > collected when collecting locatedStatuses. > When its iterating over to that fallbacks file from RemoteIterator (which > was returned from InternalViewFSDirFs ), iterator's next will will call > getFileBlockLocations if it's a file. > {code:java} > @Override > public LocatedFileStatus next() throws IOException { > System.out.println(this); > if (!hasNext()) { > throw new NoSuchElementException("No more entries in " + f); > } > FileStatus result = stats[i++]; > // for files, use getBlockLocations(FileStatus, int, int) to avoid > // calling getFileStatus(Path) to load the FileStatus again > BlockLocation[] locs = result.isFile() ? > getFileBlockLocations(result, 0, result.getLen()) : > null; > return new LocatedFileStatus(result, locs); > }{code} > > this getFileBlockLocations will be made on InternalViewFSDirFs, as that > Iterator created originally from that fs. > InternalViewFSDirFs#getFileBlockLocations does not handle fallback cases. > It's always expecting "/", this means it always assuming the dir. > But with the fallback and returning Iterator from InternalViewFSDirFs, will > create problems. > Probably we need to handle fallback case in getFileBlockLocations as well.( > Fallback only should be the reason for call coming to InternalViewFSDirFs > with other than "/") > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao reassigned HDFS-15456: --- Assignee: Leon Gao > TestExternalStoragePolicySatisfier fails intermittently > --- > > Key: HDFS-15456 > URL: https://issues.apache.org/jira/browse/HDFS-15456 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Leon Gao >Priority: Major > Labels: test > > {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk > {code:bash} > [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 421.443 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier > [ERROR] > testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier) > Time elapsed: 43.983 s <<< ERROR! > java.util.concurrent.TimeoutException: > Timed out waiting for condition. Thread diagnostics: > Timestamp: 2020-07-07 07:51:10,267 > "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918) > "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait() > java.lang.Thread.State: WAITING (on object monitor) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636" > daemon prio=5 tid=1217 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205) > at java.lang.Thread.run(Thread.java:748) > "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > at > org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273) > at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252) > "pool-90-thread-1" prio=5 tid=1069 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > "IPC Server handler 2 on default port 37995" daemon prio=5 tid=1169 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:3
[jira] [Work logged] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-15456?focusedWorklogId=481891&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481891 ] ASF GitHub Bot logged work on HDFS-15456: - Author: ASF GitHub Bot Created on: 11/Sep/20 05:07 Start Date: 11/Sep/20 05:07 Worklog Time Spent: 10m Work Description: LeonGao91 opened a new pull request #2299: URL: https://github.com/apache/hadoop/pull/2299 One-liner fix of this UT. Ignore datanode load so block placement can successfully pick fallback storage type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481891) Remaining Estimate: 0h Time Spent: 10m > TestExternalStoragePolicySatisfier fails intermittently > --- > > Key: HDFS-15456 > URL: https://issues.apache.org/jira/browse/HDFS-15456 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Leon Gao >Priority: Major > Labels: test > Time Spent: 10m > Remaining Estimate: 0h > > {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk > {code:bash} > [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 421.443 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier > [ERROR] > testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier) > Time elapsed: 43.983 s <<< ERROR! > java.util.concurrent.TimeoutException: > Timed out waiting for condition. Thread diagnostics: > Timestamp: 2020-07-07 07:51:10,267 > "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918) > "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait() > java.lang.Thread.State: WAITING (on object monitor) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636" > daemon prio=5 tid=1217 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205) > at java.lang.Thread.run(Thread.java:748) > "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > at > org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273) > at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252) > "pool-90-thread-1" prio=5 tid=1069 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThre
[jira] [Updated] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15456: -- Labels: pull-request-available test (was: test) > TestExternalStoragePolicySatisfier fails intermittently > --- > > Key: HDFS-15456 > URL: https://issues.apache.org/jira/browse/HDFS-15456 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Leon Gao >Priority: Major > Labels: pull-request-available, test > Time Spent: 10m > Remaining Estimate: 0h > > {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk > {code:bash} > [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 421.443 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier > [ERROR] > testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier) > Time elapsed: 43.983 s <<< ERROR! > java.util.concurrent.TimeoutException: > Timed out waiting for condition. Thread diagnostics: > Timestamp: 2020-07-07 07:51:10,267 > "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918) > "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait() > java.lang.Thread.State: WAITING (on object monitor) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636" > daemon prio=5 tid=1217 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205) > at java.lang.Thread.run(Thread.java:748) > "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > at > org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273) > at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252) > "pool-90-thread-1" prio=5 tid=1069 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > "IPC Server handler 2 on default port 37995" daemon prio=5 tid=1169 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(L
[jira] [Work logged] (HDFS-15555) RBF: Refresh cacheNS when SocketException occurs
[ https://issues.apache.org/jira/browse/HDFS-1?focusedWorklogId=481893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481893 ] ASF GitHub Bot logged work on HDFS-1: - Author: ASF GitHub Bot Created on: 11/Sep/20 05:13 Start Date: 11/Sep/20 05:13 Worklog Time Spent: 10m Work Description: hemanthboyina commented on pull request #2267: URL: https://github.com/apache/hadoop/pull/2267#issuecomment-690878265 any update here @aajisaka , as HDFS-15543 was modifying same part of code This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481893) Time Spent: 1h (was: 50m) > RBF: Refresh cacheNS when SocketException occurs > > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: HDFS 3.3.0, Java 11 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Problem: > When active NameNode is restarted and loading fsimage, DFSRouters > significantly slow down. > Investigation: > When active NameNode is restarted and loading fsimage, RouterRpcClient > receives SocketException. Since > RouterRpcClient#isUnavailableException(IOException) returns false when the > argument is SocketException, the MembershipNameNodeResolver#cacheNS is not > refreshed. That's why the order of the NameNodes returned by > MemberShipNameNodeResolver#getNamenodesForNameserviceId(String) is unchanged > and the active NameNode is still returned first. Therefore RouterRpcClient > still tries to connect to the NameNode that is loading fsimage. > After loading the fsimage, the NameNode throws StandbyException. The > exception is one of the 'Unavailable Exception' and the cacheNS is refreshed. > Workaround: > Stop NameNode and wait 1 minute before starting NameNode instead of > restarting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15555) RBF: Refresh cacheNS when SocketException occurs
[ https://issues.apache.org/jira/browse/HDFS-1?focusedWorklogId=481895&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481895 ] ASF GitHub Bot logged work on HDFS-1: - Author: ASF GitHub Bot Created on: 11/Sep/20 05:21 Start Date: 11/Sep/20 05:21 Worklog Time Spent: 10m Work Description: aajisaka commented on pull request #2267: URL: https://github.com/apache/hadoop/pull/2267#issuecomment-690880934 HI @hemanthboyina I don't have any updates here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481895) Time Spent: 1h 10m (was: 1h) > RBF: Refresh cacheNS when SocketException occurs > > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf > Environment: HDFS 3.3.0, Java 11 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Problem: > When active NameNode is restarted and loading fsimage, DFSRouters > significantly slow down. > Investigation: > When active NameNode is restarted and loading fsimage, RouterRpcClient > receives SocketException. Since > RouterRpcClient#isUnavailableException(IOException) returns false when the > argument is SocketException, the MembershipNameNodeResolver#cacheNS is not > refreshed. That's why the order of the NameNodes returned by > MemberShipNameNodeResolver#getNamenodesForNameserviceId(String) is unchanged > and the active NameNode is still returned first. Therefore RouterRpcClient > still tries to connect to the NameNode that is loading fsimage. > After loading the fsimage, the NameNode throws StandbyException. The > exception is one of the 'Unavailable Exception' and the cacheNS is refreshed. > Workaround: > Stop NameNode and wait 1 minute before starting NameNode instead of > restarting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14815) RBF: Update the quota in MountTable when calling setQuota on a MountTable src.
[ https://issues.apache.org/jira/browse/HDFS-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193997#comment-17193997 ] Hemanth Boyina commented on HDFS-14815: --- thanks for the explanation [~LiJinglun] after this change , through file system object we couldn't able to set quota on a mount point , before the change we could able to set quota so it looks like an incompatible change > RBF: Update the quota in MountTable when calling setQuota on a MountTable src. > -- > > Key: HDFS-14815 > URL: https://issues.apache.org/jira/browse/HDFS-14815 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14815.001.patch, HDFS-14815.002.patch, > HDFS-14815.003.patch, HDFS-14815.004.patch, HDFS-14815.005.patch > > > The method setQuota() can make the remote quota(the quota on real clusters) > inconsistent with the MountTable. I think we have 3 ways to fix it: > # Reject all the setQuota() rpcs if it trys to change the quota of a mount > table. > # Let setQuota() to change the mount table quota. Update the quota on zk > first and then update remote quotas. > # Do nothing. The RouterQuotaUpdateService will finally make all the remote > quota right. We can tolerate short-term inconsistencies. > I'd like option 1 because I want the RouterAdmin to be the only entrance to > update the MountTable. > Option 3 we don't need change anything, but the quota will be inconsistent > for a short-term. The remote quota will be effective immediately and > auto-changed back after a while. User might be confused about the behavior. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15456) TestExternalStoragePolicySatisfier fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-15456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194013#comment-17194013 ] Ayush Saxena commented on HDFS-15456: - Thanx [~LeonG] for taking this up, is there a way to reproduce this issue. Or can you share the analysis, why DN load leads to this trace > TestExternalStoragePolicySatisfier fails intermittently > --- > > Key: HDFS-15456 > URL: https://issues.apache.org/jira/browse/HDFS-15456 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Leon Gao >Priority: Major > Labels: pull-request-available, test > Time Spent: 10m > Remaining Estimate: 0h > > {{TestExternalStoragePolicySatisfier}} frequently times-out on hadoop trunk > {code:bash} > [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 421.443 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier > [ERROR] > testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace(org.apache.hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier) > Time elapsed: 43.983 s <<< ERROR! > java.util.concurrent.TimeoutException: > Timed out waiting for condition. Thread diagnostics: > Timestamp: 2020-07-07 07:51:10,267 > "IPC Server handler 4 on default port 44933" daemon prio=5 tid=1138 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:307) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2918) > "ForkJoinPool-2-worker-19" daemon prio=5 tid=235 in Object.wait() > java.lang.Thread.State: WAITING (on object monitor) > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) > "refreshUsed-/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/sourcedir/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1/current/BP-912129709-172.17.0.2-1594151429636" > daemon prio=5 tid=1217 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.fs.CachingGetSpaceUsed$RefreshThread.run(CachingGetSpaceUsed.java:205) > at java.lang.Thread.run(Thread.java:748) > "Socket Reader #1 for port 0" daemon prio=5 tid=1192 runnable > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > at > org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1273) > at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1252) > "pool-90-thread-1" prio=5 tid=1069 timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > "IPC Server handler 2 on default port 37995" daemon prio=5 tid=1169 > timed_waiting > java.lang.Thread.State: TIMED_WAITING > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedS
[jira] [Commented] (HDFS-14815) RBF: Update the quota in MountTable when calling setQuota on a MountTable src.
[ https://issues.apache.org/jira/browse/HDFS-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194019#comment-17194019 ] Ayush Saxena commented on HDFS-14815: - By using the filesystem object, when you were setting quota, was that staying up? [~LiJinglun] said it wasn't. So, is there a change in functionality, NO, then we just prevented communicating the user that things worked and prevented him from an illusion. If it was staying up, First comes is a design issue, What permissions are being checked the one at RouterAdmin or the destination, I think, in this case would be destination? Then the mount entry belongs to router admin, changes to it should check its permission? Something to think. But yes, if it was staying up(I couldn't follow the discussion again) and there were no permission/security issue, then that might land up being an incompatible change, else Not. For eg: if on setting quota the user was getting twice the quota if there were two NS, now he gets only from One. It changed the CLI output, and changing CLI output is an incompatible change. But with the above logic, I won't put this as well in the incompatible category. Second: For non-mount entries in present code, if you set quota as 10, and there are 2 NS, it earlier use to show 20 and follow as well 20, Now it shows 10, follows 20, Changed CLI output, Is that incompatible? That is something I don't know. > RBF: Update the quota in MountTable when calling setQuota on a MountTable src. > -- > > Key: HDFS-14815 > URL: https://issues.apache.org/jira/browse/HDFS-14815 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14815.001.patch, HDFS-14815.002.patch, > HDFS-14815.003.patch, HDFS-14815.004.patch, HDFS-14815.005.patch > > > The method setQuota() can make the remote quota(the quota on real clusters) > inconsistent with the MountTable. I think we have 3 ways to fix it: > # Reject all the setQuota() rpcs if it trys to change the quota of a mount > table. > # Let setQuota() to change the mount table quota. Update the quota on zk > first and then update remote quotas. > # Do nothing. The RouterQuotaUpdateService will finally make all the remote > quota right. We can tolerate short-term inconsistencies. > I'd like option 1 because I want the RouterAdmin to be the only entrance to > update the MountTable. > Option 3 we don't need change anything, but the quota will be inconsistent > for a short-term. The remote quota will be effective immediately and > auto-changed back after a while. User might be confused about the behavior. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15570) Hadoop Erasure Coding ISA-L Check Request..
isurika created HDFS-15570: -- Summary: Hadoop Erasure Coding ISA-L Check Request.. Key: HDFS-15570 URL: https://issues.apache.org/jira/browse/HDFS-15570 Project: Hadoop HDFS Issue Type: Test Components: erasure-coding, hadoop-client Affects Versions: 3.2.1 Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core) *-Hadoop Version* : hadoop-3.2.1 *-CPU INFO (namenode :2 / datanode : 5)* - namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz - *-hadoop checknative (After source build)* - Native library checking: hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 zstd : false snappy: true /lib64/libsnappy.so.1 lz4: true revision:10301 bzip2: false openssl: true /lib64/libcrypto.so {color:#0747a6}*ISA-L: true /lib64/libisal.so.2*{color} - *-hdfs ec -listCodecs* Erasure Coding Codecs: Codec [Coder List] RS [RS_NATIVE, RS_JAVA] RS-LEGACY [RS-LEGACY_JAVA] XOR [XOR_NATIVE, XOR_JAVA] Reporter: isurika Fix For: 3.2.1 I am testing the performance of erasure coding in Hadoop 3.2.1 version environment. Apply ISA-L by referring to the manual (https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html) *▶ JOB LIST (Proceed in manual order)* # ISA-L library installed on all servers ld ISA-L library : https://github.com/01org/isa-l # hadoop build source (hadoop-3.2.1-src.tar.gz) # mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy # Deploy the built hadoop-3.2.1 folder to all servers But In Test (File upload Time/Select Orc table in Hive) There is no improvement in performance when tested. Ask for advice on work *[Question 1]* Is there anything wrong with the installation? Are there any missing tasks? *[Question 2]* Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) *[Question 3]* Whether to use ISA-L in Hadoop, How to check? ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no ※ The warn log that occurs before applying ISA-L is It no longer occurs. -- WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable -- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15570) Hadoop Erasure Coding ISA-L Check Request..
[ https://issues.apache.org/jira/browse/HDFS-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] isurika updated HDFS-15570: --- Description: I am testing the performance of erasure coding in Hadoop 3.2.1 version environment. Apply ISA-L by referring to the manual ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]) *▶ JOB LIST (Proceed in manual order)* # ISA-L library installed on all servers ld ISA-L library : [https://github.com/01org/isa-l] # hadoop build source (hadoop-3.2.1-src.tar.gz) # mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy # Deploy the built hadoop-3.2.1 folder to all servers But In Test (File upload Time/Select Orc table in Hive) There is no improvement in performance when tested. Ask for advice on work *[Question 1]* Is there anything wrong with the installation? Are there any missing tasks? *[Question 2]* Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) *[Question 3]* Whether to use ISA-L in Hadoop, How to check? ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no ※ The warn log that occurs before applying ISA-L is It no longer occurs. -- WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable -- *▶Reference information* === *-CPU INFO (namenode :2 / datanode : 5)* - namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz - *-hadoop checknative (After source build)* - Native library checking: hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 zstd : false snappy: true /lib64/libsnappy.so.1 lz4: true revision:10301 bzip2: false openssl: true /lib64/libcrypto.so *ISA-L: true /lib64/libisal.so.2* - *-hdfs ec -listCodecs* Erasure Coding Codecs: Codec [Coder List] RS [RS_NATIVE, RS_JAVA] RS-LEGACY [RS-LEGACY_JAVA] XOR [XOR_NATIVE, XOR_JAVA] === was: I am testing the performance of erasure coding in Hadoop 3.2.1 version environment. Apply ISA-L by referring to the manual (https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html) *▶ JOB LIST (Proceed in manual order)* # ISA-L library installed on all servers ld ISA-L library : https://github.com/01org/isa-l # hadoop build source (hadoop-3.2.1-src.tar.gz) # mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy # Deploy the built hadoop-3.2.1 folder to all servers But In Test (File upload Time/Select Orc table in Hive) There is no improvement in performance when tested. Ask for advice on work *[Question 1]* Is there anything wrong with the installation? Are there any missing tasks? *[Question 2]* Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) *[Question 3]* Whether to use ISA-L in Hadoop, How to check? ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no ※ The warn log that occurs before applying ISA-L is It no longer occurs. -- WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable -- Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core) *-Hadoop Version* : hadoop-3.2.1 was: *-OS Version* : CentOS Linux release 8.0.1905 (Core) *-Hadoop Version* : hadoop-3.2.1 *-CPU INFO (namenode :2 / datanode : 5)* - namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz - *-hadoop checknative (After source build)* - Native library checking: hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 zstd : false snappy: true /lib64/libsnappy.so.1 lz4: true revision:10301 bzip2: false openssl: true /lib64/libcrypto.so {color:#0747a6}*ISA-L: true /lib64/libisal.so.2*{color} - *-hdfs ec -listCodecs* Erasure Coding Codecs: Codec [Coder List] RS [RS_NATIVE, RS_JAVA] RS-LEGACY [RS-LEGACY_JAVA] XOR [XOR_NATIVE, XOR_JAVA] > Hadoop Erasure Coding ISA-L Check Request.. > --- > > Key: HDFS-15570 > URL: https://issues.apache.org/jira/browse/HDFS-15570 >
[jira] [Updated] (HDFS-15570) Hadoop Erasure Coding ISA-L Check Request..
[ https://issues.apache.org/jira/browse/HDFS-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] isurika updated HDFS-15570: --- Description: I am testing the performance of erasure coding in Hadoop 3.2.1 version environment. Apply ISA-L by referring to the manual ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]) *▶ JOB LIST (Proceed in manual order)* # ISA-L library installed on all servers ld ISA-L library : [https://github.com/01org/isa-l] # hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy) # Deploy the built hadoop-3.2.1 folder to all servers But In Test (File upload Time/Select Orc table in Hive) There is no improvement in performance when tested. Ask for advice on work *[Question 1]* Is there anything wrong with the installation? Are there any missing tasks? *[Question 2]* Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) *[Question 3]* Whether to use ISA-L in Hadoop, How to check? ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no ※ The warn log that occurs before applying ISA-L is It no longer occurs. -- WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable -- *▶Reference information* === *-CPU INFO (namenode :2 / datanode : 5)* - namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz - *-hadoop checknative (After source build)* - Native library checking: hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 zstd : false snappy: true /lib64/libsnappy.so.1 lz4: true revision:10301 bzip2: false openssl: true /lib64/libcrypto.so *ISA-L: true /lib64/libisal.so.2* - *-hdfs ec -listCodecs* Erasure Coding Codecs: Codec [Coder List] RS [RS_NATIVE, RS_JAVA] RS-LEGACY [RS-LEGACY_JAVA] XOR [XOR_NATIVE, XOR_JAVA] === was: I am testing the performance of erasure coding in Hadoop 3.2.1 version environment. Apply ISA-L by referring to the manual ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]) *▶ JOB LIST (Proceed in manual order)* # ISA-L library installed on all servers ld ISA-L library : [https://github.com/01org/isa-l] # hadoop build source (hadoop-3.2.1-src.tar.gz) # mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy # Deploy the built hadoop-3.2.1 folder to all servers But In Test (File upload Time/Select Orc table in Hive) There is no improvement in performance when tested. Ask for advice on work *[Question 1]* Is there anything wrong with the installation? Are there any missing tasks? *[Question 2]* Why doesn't it speed up ? (File upload Time/Select Orc table in Hive) *[Question 3]* Whether to use ISA-L in Hadoop, How to check? ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no ※ The warn log that occurs before applying ISA-L is It no longer occurs. -- WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable -- *▶Reference information* === *-CPU INFO (namenode :2 / datanode : 5)* - namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz - *-hadoop checknative (After source build)* - Native library checking: hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 zstd : false snappy: true /lib64/libsnappy.so.1 lz4: true revision:10301 bzip2: false openssl: true /lib64/libcrypto.so *ISA-L: true /lib64/libisal.so.2* - *-hdfs ec -listCodecs* Erasure Coding Codecs: Codec [Coder List] RS [RS_NATIVE, RS_JAVA] RS-LEGACY [RS-LEGACY_JAVA] XOR [XOR_NATIVE, XOR_JAVA] === > Hadoop Erasure Coding ISA-L Check Request.. > --- > > Key: HDFS-15570 > URL: https://issues.apache.org/jira/browse/HDFS-15570 > Project: Hadoop HDFS > Issue Type: Test > Components: erasure-codi
[jira] [Created] (HDFS-15571) RBF: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per mount entry.
Harshakiran Reddy created HDFS-15571: Summary: RBF: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per mount entry. Key: HDFS-15571 URL: https://issues.apache.org/jira/browse/HDFS-15571 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.1.1 Environment: {code:java} 1. Create dir's in 4 NS. 2. Create Mount path with 4 destinations with order HASH_ALL {code} Reporter: Harshakiran Reddy When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per End user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15571) RBF: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per mount entry.
[ https://issues.apache.org/jira/browse/HDFS-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshakiran Reddy updated HDFS-15571: - Environment: (was: {code:java} 1. Create dir's in 4 NS. 2. Create Mount path with 4 destinations with order HASH_ALL 3. Setquota 10 on Mount path 4. Get quota on mount path {code} ) > RBF: When we set the quota on mount Path which is having 4 Destinations, > count command shows the 4 as dir count, it should be 1 as per mount entry. > --- > > Key: HDFS-15571 > URL: https://issues.apache.org/jira/browse/HDFS-15571 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Priority: Major > > When we set the quota on mount Path which is having 4 Destinations, count > command shows the 4 as dir count, it should be 1 as per End user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15571) RBF: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per mount entry.
[ https://issues.apache.org/jira/browse/HDFS-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshakiran Reddy updated HDFS-15571: - Environment: {code:java} 1. Create dir's in 4 NS. 2. Create Mount path with 4 destinations with order HASH_ALL 3. Setquota 10 on Mount path 4. Get quota on mount path {code} was: {code:java} 1. Create dir's in 4 NS. 2. Create Mount path with 4 destinations with order HASH_ALL {code} > RBF: When we set the quota on mount Path which is having 4 Destinations, > count command shows the 4 as dir count, it should be 1 as per mount entry. > --- > > Key: HDFS-15571 > URL: https://issues.apache.org/jira/browse/HDFS-15571 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 > Environment: {code:java} > 1. Create dir's in 4 NS. > 2. Create Mount path with 4 destinations with order HASH_ALL > 3. Setquota 10 on Mount path > 4. Get quota on mount path > {code} >Reporter: Harshakiran Reddy >Priority: Major > > When we set the quota on mount Path which is having 4 Destinations, count > command shows the 4 as dir count, it should be 1 as per End user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15571) RBF: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per mount entry.
[ https://issues.apache.org/jira/browse/HDFS-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshakiran Reddy updated HDFS-15571: - Description: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per End user. {code:java} 1. Create dir's in 4 NS. 2. Create Mount path with 4 destinations with order HASH_ALL 3. Setquota 10 on Mount path 4. Get quota on mount path {code} was:When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per End user. > RBF: When we set the quota on mount Path which is having 4 Destinations, > count command shows the 4 as dir count, it should be 1 as per mount entry. > --- > > Key: HDFS-15571 > URL: https://issues.apache.org/jira/browse/HDFS-15571 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Priority: Major > > When we set the quota on mount Path which is having 4 Destinations, count > command shows the 4 as dir count, it should be 1 as per End user. > {code:java} > 1. Create dir's in 4 NS. > 2. Create Mount path with 4 destinations with order HASH_ALL > 3. Setquota 10 on Mount path > 4. Get quota on mount path > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15571) RBF: When we set the quota on mount Path which is having 4 Destinations, count command shows the 4 as dir count, it should be 1 as per mount entry.
[ https://issues.apache.org/jira/browse/HDFS-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina reassigned HDFS-15571: - Assignee: Hemanth Boyina > RBF: When we set the quota on mount Path which is having 4 Destinations, > count command shows the 4 as dir count, it should be 1 as per mount entry. > --- > > Key: HDFS-15571 > URL: https://issues.apache.org/jira/browse/HDFS-15571 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > > When we set the quota on mount Path which is having 4 Destinations, count > command shows the 4 as dir count, it should be 1 as per End user. > {code:java} > 1. Create dir's in 4 NS. > 2. Create Mount path with 4 destinations with order HASH_ALL > 3. Setquota 10 on Mount path > 4. Get quota on mount path > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[ https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=481924&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481924 ] ASF GitHub Bot logged work on HDFS-15539: - Author: ASF GitHub Bot Created on: 11/Sep/20 06:52 Start Date: 11/Sep/20 06:52 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2258: URL: https://github.com/apache/hadoop/pull/2258#issuecomment-690911942 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 33s | Maven dependency ordering for branch | | -1 :x: | mvninstall | 6m 17s | root in trunk failed. | | -1 :x: | compile | 0m 26s | hadoop-hdfs-project in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 25s | hadoop-hdfs-project in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 23s | The patch fails to run checkstyle in hadoop-hdfs-project | | -1 :x: | mvnsite | 0m 17s | hadoop-hdfs-client in trunk failed. | | +1 :green_heart: | shadedclient | 3m 6s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 51s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 2m 19s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 3m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 7m 1s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 42s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 18s | the patch passed | | -1 :x: | compile | 6m 7s | hadoop-hdfs-project in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 6m 7s | hadoop-hdfs-project in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 22s | hadoop-hdfs-project in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 22s | hadoop-hdfs-project in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 23s | The patch fails to run checkstyle in hadoop-hdfs-project | | -1 :x: | mvnsite | 0m 26s | hadoop-hdfs-client in the patch failed. | | -1 :x: | mvnsite | 0m 25s | hadoop-hdfs in the patch failed. | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 0m 25s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 25s | hadoop-hdfs-client in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 27s | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javadoc | 0m 23s | hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | findbugs | 0m 28s | hadoop-hdfs-client in the patch failed. | | -1 :x: | findbugs | 0m 30s | hadoop-hdfs in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 11s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 129m 33s | hadoop-hdfs in the patch passed. | | -1 :x: | asflicense | 0m 33s | The patch generated 4 ASF License warnings. | | | | 179m 16s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.security.TestPermissionSymlinks | | | hadoop.fs.permission.TestStickyBit | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | | | hadoop.hdfs.TestGetFileChecksum | | | hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | | | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs | | | hadoop.hdfs.TestFileChecksum | | | hadoop.security.TestRefreshUserMappings | | | hadoop.hdfs.TestReadStripedFileWithMissingBlocks | | | hadoop.tools.TestJMXGet | |
[jira] [Work logged] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?focusedWorklogId=481925&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481925 ] ASF GitHub Bot logged work on HDFS-15025: - Author: ASF GitHub Bot Created on: 11/Sep/20 06:54 Start Date: 11/Sep/20 06:54 Worklog Time Spent: 10m Work Description: huangtianhua commented on pull request #2189: URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690912424 @liuml07 We have checked for several times, the failure tests are not related with this patch, and I found almost all of the HDFS-* patches are not pass the hadoop-hdfs unit tests, like https://github.com/apache/hadoop/pull/2275 https://github.com/apache/hadoop/pull/2205 So maybe the failure tests are not blocker? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 481925) Time Spent: 4h 40m (was: 4.5h) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: YaYun Wang >Assignee: YaYun Wang >Priority: Major > Labels: pull-request-available > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > HDFS-15025.002.patch, HDFS-15025.003.patch, HDFS-15025.004.patch, > HDFS-15025.005.patch, HDFS-15025.006.patch, NVDIMM_patch(WIP).patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org