[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845949#comment-17845949
]
ASF GitHub Bot commented on HDFS-17518:
---
Hexiaoqiao commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1598536353
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java:
##
@@ -626,7 +626,8 @@ private synchronized boolean checkLeases(Collection
leasesToCheck) {
}
}
// If a lease recovery happened, we need to sync later.
Review Comment:
> I would recommend you to change the return type of internalReleaseLease()
to ImmutablePair to include both completed and needSync
values.
needSync will be true in both cases of file closed and lease re-assignment.
+1. If we will plan to improve it, should fix it together.
BTW, what will it happen if not sync in time, LeaseManager.Monitor is one
asynchronous logic, it can not be ensure to sync edits in one certain order
right?
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16368) DFSAdmin supports refresh topology info without restarting namenode
[ https://issues.apache.org/jira/browse/HDFS-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845928#comment-17845928 ] ASF GitHub Bot commented on HDFS-16368: --- Hexiaoqiao commented on PR #3743: URL: https://github.com/apache/hadoop/pull/3743#issuecomment-2107611900 Not sure why this PR has been pick up again. IIRC we have discussed add tools to support refresh topology online many times. I would like to share my concern again. This is one high-risky operation, especially for one large cluster. Considering the case that change the cluster topology randomly then namenode failover or restart, there will be block transfer flood where the replications of one block should be going to have enough racks. Before all blocks have enough racks (which is cost long times) the cluster could enter high-load phase and couldn't process any requests from client. In one word, be careful to refresh topology online if there is no stability guarantee measures. > DFSAdmin supports refresh topology info without restarting namenode > > > Key: HDFS-16368 > URL: https://issues.apache.org/jira/browse/HDFS-16368 > Project: Hadoop HDFS > Issue Type: New Feature > Components: dfsadmin, namanode >Affects Versions: 2.7.7, 3.3.1 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: features, pull-request-available > Attachments: 0001.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Currently in HDFS, if we update the rack info for rack-awareness, we may need > to rolling restart namenodes to let it be effective. If cluster is large, the > cost time of rolling restart namenodes is very log. So, we develope a method > to refresh topology info without rolling restart namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845922#comment-17845922
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1598472326
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1239,26 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
+createFile(routerFS, existingFile, existingFileSize);
+String sameRouterEmptyFile =
+cluster.getFederatedTestDirectoryForNS(sameNameservice) +
+"_newemptyfile";
+createFile(routerFS, sameRouterEmptyFile, 0);
+// Concat in same namespaces, succeeds
+testConcat(existingFile, sameRouterEmptyFile, false);
+FileStatus mergedStatus = getFileStatus(routerFS, sameRouterEmptyFile);
+assertEquals(existingFileSize, mergedStatus.getLen());
+
+// Test when concat srclist has some empty file, namenode will throw
IOException.
+String srcEmptyFile =
cluster.getFederatedTestDirectoryForNS(sameNameservice) + "_srcEmptyFile";
+createFile(routerFS, srcEmptyFile, 0);
+String targetFile =
cluster.getFederatedTestDirectoryForNS(sameNameservice) + "_targetFile";
+createFile(routerFS, targetFile, existingFileSize);
+// Concat in same namespaces, succeeds
+testConcat(srcEmptyFile, targetFile, true, true,
Review Comment:
Done. Thanks
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845891#comment-17845891
]
ASF GitHub Bot commented on HDFS-17509:
---
hadoop-yetus commented on PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2107394624
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 26m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 108m 34s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.security.token.TestSQLDelegationTokenSecretManagerImpl
|
| | hadoop.hdfs.server.federation.store.driver.TestStateStoreMySQL |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f5796ff89d5e 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0c3ea013638be3e91a7972c80cf694a1447ea1a0 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/8/testReport/ |
| Max. process+thread count |
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845869#comment-17845869
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1598226242
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1239,26 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
+createFile(routerFS, existingFile, existingFileSize);
+String sameRouterEmptyFile =
+cluster.getFederatedTestDirectoryForNS(sameNameservice) +
+"_newemptyfile";
+createFile(routerFS, sameRouterEmptyFile, 0);
+// Concat in same namespaces, succeeds
+testConcat(existingFile, sameRouterEmptyFile, false);
+FileStatus mergedStatus = getFileStatus(routerFS, sameRouterEmptyFile);
+assertEquals(existingFileSize, mergedStatus.getLen());
+
+// Test when concat srclist has some empty file, namenode will throw
IOException.
+String srcEmptyFile =
cluster.getFederatedTestDirectoryForNS(sameNameservice) + "_srcEmptyFile";
+createFile(routerFS, srcEmptyFile, 0);
+String targetFile =
cluster.getFederatedTestDirectoryForNS(sameNameservice) + "_targetFile";
+createFile(routerFS, targetFile, existingFileSize);
+// Concat in same namespaces, succeeds
+testConcat(srcEmptyFile, targetFile, true, true,
Review Comment:
here please check if the exception is
`org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.HadoopIllegalArgumentException)`
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845866#comment-17845866
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1598204990
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -1009,6 +999,27 @@ public HdfsFileStatus getFileInfo(String src) throws
IOException {
return ret;
}
+ public RemoteLocation getFileRemoteLocation(String path) throws IOException {
+rpcServer.checkOperation(NameNode.OperationCategory.READ);
+
+final List locations = rpcServer.getLocationsForPath(path,
false, false);
+if (locations.size() == 1)
Review Comment:
```
if (locations.size() == 1) {
return locations.get(0);
}
```
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845848#comment-17845848
]
ASF GitHub Bot commented on HDFS-17509:
---
hadoop-yetus commented on PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2107054325
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 94m 50s | | trunk passed |
| +1 :green_heart: | compile | 5m 11s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 152m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 03s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed |
| +1 :green_heart: | javac | 2m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 07s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 165m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 43s | | The patch does not
generate ASF License warnings. |
| | | 436m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 91722a5bf48a 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 61d08c583e427df3340a7afcd984ad69943a1675 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6784/4/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6784/4/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845844#comment-17845844
]
ASF GitHub Bot commented on HDFS-17522:
---
hadoop-yetus commented on PR #6814:
URL: https://github.com/apache/hadoop/pull/6814#issuecomment-2107022090
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 88m 18s | | trunk passed |
| +1 :green_heart: | compile | 6m 10s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 146m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 29s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 152m 03s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 11s | | The patch does not
generate ASF License warnings. |
| | | 414m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6814 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9c2306291c59 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / c2bd098a86c05e239cc2a8de073ba8d25e83dbb4 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-17522
> URL: https://issues.apache.org/jira/browse/HDFS-17522
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node
>Affects Versions: 3.0.0-alpha1, 3.5.0
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
>
> [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added
> protection for NameNode and DataNode, but missing protection for JournalNode
> web interfaces.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17504) DN process should exit when BPServiceActor exit
[ https://issues.apache.org/jira/browse/HDFS-17504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845841#comment-17845841 ] ASF GitHub Bot commented on HDFS-17504: --- Hexiaoqiao commented on PR #6792: URL: https://github.com/apache/hadoop/pull/6792#issuecomment-2107018371 @zhuzilong2013 Thanks for your report and contribution! IMO, they are independent between different BPServiceActor, if exit DN process due to one BPServiceActor issue, it will increase number of Dead DataNode from the whole cluster view, where I don't think it is proper in Federation Arch. Another side, maybe we could add some BPServiceActor count metric to monitor if BPServiceActor works fine? Thanks again. > DN process should exit when BPServiceActor exit > --- > > Key: HDFS-17504 > URL: https://issues.apache.org/jira/browse/HDFS-17504 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zilong Zhu >Assignee: Zilong Zhu >Priority: Major > Labels: pull-request-available > > BPServiceActor is a very important thread. In a non-HA cluster, the exit of > the BPServiceActor thread will cause the DN process to exit. However, in a HA > cluster, this is not the case. > I found HDFS-15651 causes BPServiceActor thread to exit and sets the > "runningState" from "RunningState.FAILED" to "RunningState.EXITED", it can > be confusing during troubleshooting. > I believe that the DN process should exit when the flag of the BPServiceActor > is set to RunningState.FAILED because at this point, the DN is unable to > recover and establish a heartbeat connection with the ANN on its own. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17524) OIV: add Transformed processor which reconstructs an fsimage from another fsimage file
[
https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845817#comment-17845817
]
ASF GitHub Bot commented on HDFS-17524:
---
XbaoWu opened a new pull request, #6820:
URL: https://github.com/apache/hadoop/pull/6820
### Description of PR
Hadoop 3.2 introduced optimization features for HDFS StringTable
(https://github.com/apache/hadoop/commit/b60ca37914b22550e3630fa02742d40697decb31),
It resulted in lower versions of Hadoop upgraded to 3.2 and later versions not
supporting downgrade operations. Therefore, we have added Transformed Processor
to support downgrading. In the upgrade with large version span, it avoids the
low version can not read the fs-image generated by the 3.2 and later version,
which support for the 3.2 and later version to be downgraded to the low version.
For more information about this PR, please refer to the following issue：
[HDFS-17524](https://issues.apache.org/jira/browse/HDFS-17524) OIV: add
Transformed processor which reconstructs an fsimage from another fsimage file.
### How was this patch tested?
add TestOfflineImageViewer.testTransformedWriter()
add TestOfflineImageViewer.testTransformedWriterWrongLayoutVersion()
add TestOfflineImageViewer.testTransformedWriterReasonableLayoutVersion()
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> OIV: add Transformed processor which reconstructs an fsimage from another
> fsimage file
> --
>
> Key: HDFS-17524
> URL: https://issues.apache.org/jira/browse/HDFS-17524
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 3.2.0, 3.3.4
>Reporter: Xiaobao Wu
>Priority: Major
>
> *Background：*
> The Image file generated by the existing Hadoop 3.3.4 version cannot be
> forward compatible . In the high version of HDFS, the fsimage file conversion
> tool is provided to support the generation of forward compatible fsimage file
> to support the downgrade operation.
> {*}Description{*}：
> Because there are differences in the structure and loading methods of some
> Sections between high and low versions of fsimage files, especially the
> StringTable Section. This will make it impossible to downgrade to a lower
> version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because
> when the lower version of HDFS loads the fsimage file generated by the higher
> version of HDFS, there will be an ArrayIndexOutOfBoundsException.
>
> The code differences are as follows：
> {code:java}
> // 3.3.4
> static SerialNumberManager.StringTable loadStringTable(InputStream in)
> throws IOException {
> ··· ···
> SerialNumberManager.StringTable stringTable =
> SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> stringTable.put(e.getId(), e.getStr());
> }
> return stringTable;
> }
> // 3.1.1
> static String[] loadStringTable(InputStream in) throws IOException {
> ··· ···
> String[] stringTable = new String[s.getNumEntry() + 1];
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> // ArrayIndexOutOfBoundsException is triggered when loading a higher
> version of the fsimage file.
> stringTable[e.getId()] = e.getStr();
> }
> return stringTable;
> }{code}
> {*}Solution{*}：
> Solution Reference from HDFS-17463
> !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
> From the figure, it can be seen that the Id arrangement of StringTable in the
> fsimage file has changed from a compact arrangement t
[jira] [Updated] (HDFS-17524) OIV: add Transformed processor which reconstructs an fsimage from another fsimage file
[
https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17524:
--
Labels: pull-request-available (was: )
> OIV: add Transformed processor which reconstructs an fsimage from another
> fsimage file
> --
>
> Key: HDFS-17524
> URL: https://issues.apache.org/jira/browse/HDFS-17524
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 3.2.0, 3.3.4
>Reporter: Xiaobao Wu
>Priority: Major
> Labels: pull-request-available
>
> *Background：*
> The Image file generated by the existing Hadoop 3.3.4 version cannot be
> forward compatible . In the high version of HDFS, the fsimage file conversion
> tool is provided to support the generation of forward compatible fsimage file
> to support the downgrade operation.
> {*}Description{*}：
> Because there are differences in the structure and loading methods of some
> Sections between high and low versions of fsimage files, especially the
> StringTable Section. This will make it impossible to downgrade to a lower
> version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because
> when the lower version of HDFS loads the fsimage file generated by the higher
> version of HDFS, there will be an ArrayIndexOutOfBoundsException.
>
> The code differences are as follows：
> {code:java}
> // 3.3.4
> static SerialNumberManager.StringTable loadStringTable(InputStream in)
> throws IOException {
> ··· ···
> SerialNumberManager.StringTable stringTable =
> SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> stringTable.put(e.getId(), e.getStr());
> }
> return stringTable;
> }
> // 3.1.1
> static String[] loadStringTable(InputStream in) throws IOException {
> ··· ···
> String[] stringTable = new String[s.getNumEntry() + 1];
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> // ArrayIndexOutOfBoundsException is triggered when loading a higher
> version of the fsimage file.
> stringTable[e.getId()] = e.getStr();
> }
> return stringTable;
> }{code}
> {*}Solution{*}：
> Solution Reference from HDFS-17463
> !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
> From the figure, it can be seen that the Id arrangement of StringTable in the
> fsimage file has changed from a compact arrangement to a decentralized
> arrangement, that is, USER, GROUP and XATTR are no longer mixed. The
> arrangement is divided into different storage areas and arranged separately.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> * When saving fsimage files in high and low versions, the main difference is
> the arrangement of Entry（e.g., USER, GROUP, and XATTR ） in StringTable.
> * We will add a conversion tool to convert the Id arrangement of the high
> version fsimage file StringTable to a compact arrangement, so that the low
> version can be compatible with this format fsimage file.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845793#comment-17845793 ] ASF GitHub Bot commented on HDFS-17463: --- hiwangzhihui commented on PR #6736: URL: https://github.com/apache/hadoop/pull/6736#issuecomment-2106760458 @Hexiaoqiao I agree with your propose , We will proceed with work for it. Thank you for your guide. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17520) TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed
[
https://issues.apache.org/jira/browse/HDFS-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845787#comment-17845787
]
ASF GitHub Bot commented on HDFS-17520:
---
slfan1989 commented on PR #6812:
URL: https://github.com/apache/hadoop/pull/6812#issuecomment-2106716295
LGTM +1
> TestDFSAdmin.testAllDatanodesReconfig and
> TestDFSAdmin.testDecommissionDataNodesReconfig failed
> ---
>
> Key: HDFS-17520
> URL: https://issues.apache.org/jira/browse/HDFS-17520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
> [ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
> Time elapsed: 2.086 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting:
> <["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
> property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true"",
> "started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
> 13:02:51 UTC 2024."]>
> to contain subsequence:
> <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true""]>
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845777#comment-17845777
]
ASF GitHub Bot commented on HDFS-17509:
---
hadoop-yetus commented on PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2106577140
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 33s | | trunk passed |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 12s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 3
unchanged - 0 fixed = 4 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 29m 7s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 111m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 10b17be7049b 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 61d08c583e427df3340a7afcd984ad69943a1675 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/7/testReport/ |
| Max. process+thread count | 3163 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Conso
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845776#comment-17845776 ] ASF GitHub Bot commented on HDFS-17384: --- hadoop-yetus commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2106553639 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | -1 :x: | patch | 1m 02s | | https://github.com/apache/hadoop/pull/6762 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6762 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/3/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845771#comment-17845771 ] ASF GitHub Bot commented on HDFS-17506: --- ferhui commented on PR #6806: URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2106540855 Thanks for contribution. Merged. > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845768#comment-17845768
]
ASF GitHub Bot commented on HDFS-17518:
---
ThinkerLei commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597806803
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src,
INodesInPath iip,
NameNode.stateChangeLog.warn("BLOCK*" +
" internalReleaseLease: All existing blocks are COMPLETE," +
" lease removed, file " + src + " closed.");
- return true; // closed!
+ return false; // closed!
Review Comment:
@vinayakumarb Thank you for your review. Let me explain the current logic.
The logic I am modifying now is as follows: if the lease is recovered or the
lease is reassigned, it will return false, just like the previous logic. Then,
in the checkLeases method, if the return is false and needSync is false,
needSync will be reset to true. Subsequently, the edits log will be flushed by
leaseMonitor. This way, when RPCs such as recoverLease call the
internalReleaseLease method, they can remain consistent with the original
behavior.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845769#comment-17845769
]
ASF GitHub Bot commented on HDFS-17518:
---
ThinkerLei commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597807172
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java:
##
@@ -626,7 +626,8 @@ private synchronized boolean checkLeases(Collection
leasesToCheck) {
}
}
// If a lease recovery happened, we need to sync later.
Review Comment:
@vinayakumarb Thank you for your review. Let me explain the current logic.
The logic I am modifying now is as follows: if the lease is recovered or the
lease is reassigned, it will return false, just like the previous logic. Then,
in the checkLeases method, if the return is false and needSync is false,
needSync will be reset to true. Subsequently, the edits log will be flushed by
leaseMonitor. This way, when RPCs such as recoverLease call the
internalReleaseLease method, they can remain consistent with the original
behavior.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845767#comment-17845767
]
ASF GitHub Bot commented on HDFS-17518:
---
ThinkerLei commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597806803
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src,
INodesInPath iip,
NameNode.stateChangeLog.warn("BLOCK*" +
" internalReleaseLease: All existing blocks are COMPLETE," +
" lease removed, file " + src + " closed.");
- return true; // closed!
+ return false; // closed!
Review Comment:
@vinayakumarb Thank you for your review. Let me explain the current logic.
The logic I am modifying now is as follows: if the lease is recovered or the
lease is reassigned, it will return false, just like the previous logic. Then,
in the checkLeases method, if the return is false and needSync is false,
needSync will be reset to true. Subsequently, the edits log will be flushed by
leaseMonitor. This way, when RPCs such as recoverLease call the
internalReleaseLease method, they can remain consistent with the original
behavior.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845764#comment-17845764
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597789296
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
Review Comment:
Add test for a empty src file
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845763#comment-17845763
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597788935
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -1009,6 +1000,20 @@ public HdfsFileStatus getFileInfo(String src) throws
IOException {
return ret;
}
+ public RemoteResult
getFileRemoteResult(String path)
+ throws IOException {
+rpcServer.checkOperation(NameNode.OperationCategory.READ);
+
+final List locations = rpcServer.getLocationsForPath(path,
false, false);
+RemoteMethod method =
+new RemoteMethod("getFileInfo", new Class[] {String.class}, new
RemoteParam());
+// Check for file information sequentially
+RemoteResult result =
Review Comment:
OK
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17397) Choose another DN as soon as possible, when encountering network issues
[ https://issues.apache.org/jira/browse/HDFS-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845760#comment-17845760 ] ASF GitHub Bot commented on HDFS-17397: --- daragu commented on PR #6591: URL: https://github.com/apache/hadoop/pull/6591#issuecomment-2106461328 LGTM > Choose another DN as soon as possible, when encountering network issues > --- > > Key: HDFS-17397 > URL: https://issues.apache.org/jira/browse/HDFS-17397 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xleoken >Priority: Minor > Labels: pull-request-available > Attachments: hadoop.png > > > Choose another DN as soon as possible, when encountering network issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845742#comment-17845742
]
ASF GitHub Bot commented on HDFS-17518:
---
vinayakumarb commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597693783
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java:
##
@@ -626,7 +626,8 @@ private synchronized boolean checkLeases(Collection
leasesToCheck) {
}
}
// If a lease recovery happened, we need to sync later.
Review Comment:
This is a nice hack. But this will not handle the case, where actual
recovery of the file is triggered and lease is reassigned.
Lease re-assignment also will have a edit log. This also should be synced as
well.
I would recommend you to change the return type of `internalReleaseLease()`
to `ImmutablePair` to include both `completed` and `needSync`
values.
needSync will be true in both cases of file closed and lease re-assignment.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845739#comment-17845739
]
ASF GitHub Bot commented on HDFS-17518:
---
hadoop-yetus commented on PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#issuecomment-2106325014
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 110m 34s | | trunk passed |
| +1 :green_heart: | compile | 7m 49s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 181m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 01s | | the patch passed |
| +1 :green_heart: | compile | 4m 44s | | the patch passed |
| +1 :green_heart: | javac | 4m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 195m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 7m 19s | | The patch does not
generate ASF License warnings. |
| | | 523m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6809 |
| JIRA Issue | HDFS-17518 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 117457eea4e9 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 6c1e9a137d17034a7f2b2a36b3b9ce0f6afb5ac9 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[
https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845730#comment-17845730
]
ASF GitHub Bot commented on HDFS-17438:
---
goiri commented on code in PR #6655:
URL: https://github.com/apache/hadoop/pull/6655#discussion_r1597672487
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java:
##
@@ -404,6 +404,28 @@ public static void waitFor(final Supplier check,
}
}
+ /**
+ * Wait at least {@code atLeastWaitForMillis} from start to end of the test.
+ *
+ * @param check the test to perform.
+ * @param atLeastWaitForMillis the minimum waiting time from the beginning
+ * to the end of the test.
+ * @throws InterruptedException if the method is interrupted while waiting.
+ */
+ public static T atLeastWaitFor(final Supplier check,
+ long atLeastWaitForMillis) throws InterruptedException {
+if (atLeastWaitForMillis < 0) {
+ atLeastWaitForMillis = 0;
+}
+long st = Time.monotonicNow();
+T result = check.get();
Review Comment:
I don't think you are waiting at least, you are getting the value and then
waiting.
Either change the method name to execute and wait or actually wait after the
sleep.
> RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
> -
>
> Key: HDFS-17438
> URL: https://issues.apache.org/jira/browse/HDFS-17438
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17438.001.patch
>
>
> At present, when the status of all namenodes in an ns in the router is the
> same, the namenode which is the newest reported will be placed at the top of
> the cache. when the client accesses the ns through the router, it will first
> access the namenode.
> If multiple namenodes in this route are in an active state, or if there are
> namenodes with multiple observer states, the existing logic is not a problem,
> because the newest reported active or observer state namenode have a higher
> probability of being true active or observer compared to the namenode that
> reported active or observer state a long time ago.
> Similarly, the newest reported namenode with a status of standby or
> unavailable has a higher probability of being a standby or unavailable
> namenode compared to the namenode reported with a status of standby or
> unavailable a long time ago. Therefore, the newest nn reported as standby or
> unavailable status should have a lower priority for access, the oldest nn
> reported as standby or unavailable status should have a higher priority for
> access.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845727#comment-17845727
]
ASF GitHub Bot commented on HDFS-17518:
---
hadoop-yetus commented on PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#issuecomment-2106300111
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 17s | | trunk passed |
| +1 :green_heart: | compile | 5m 56s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 00s | | trunk passed |
| +1 :green_heart: | shadedclient | 142m 05s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 30s | | the patch passed |
| +1 :green_heart: | compile | 3m 22s | | the patch passed |
| +1 :green_heart: | javac | 3m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 01s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 155m 05s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 13s | | The patch does not
generate ASF License warnings. |
| | | 411m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6809 |
| JIRA Issue | HDFS-17518 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 b1f57066a6fa 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 6c1e9a137d17034a7f2b2a36b3b9ce0f6afb5ac9 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/3/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6809/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845706#comment-17845706
]
ASF GitHub Bot commented on HDFS-17522:
---
hadoop-yetus commented on PR #6814:
URL: https://github.com/apache/hadoop/pull/6814#issuecomment-2106252485
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 93m 21s | | trunk passed |
| +1 :green_heart: | compile | 6m 17s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 02s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 153m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 53s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| -1 :x: | blanks | 0m 00s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 2m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 165m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 42s | | The patch does not
generate ASF License warnings. |
| | | 440m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6814 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 53b71b4f2f76 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 213e3fc0e16b2e7b1c9d5e86ff411a9762da7061 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6814/1/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-17522
> URL: https://issues.apache.org/jira/browse/HDFS-17522
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node
>Affects Versions: 3.0.0-alpha1
>Reporter: wangzhihui
>Priority: Major
> Labels: pull-request-available
>
> [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added
> protection for NameNode and DataNode, but missing protection for JournalNode
> web interfaces.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845674#comment-17845674
]
ASF GitHub Bot commented on HDFS-17518:
---
ThinkerLei commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597594907
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src,
INodesInPath iip,
NameNode.stateChangeLog.warn("BLOCK*" +
" internalReleaseLease: All existing blocks are COMPLETE," +
" lease removed, file " + src + " closed.");
- return true; // closed!
+ return false; // closed!
Review Comment:
@vinayakumarb Thank you very much for your comment; it is very accurate. I
have modified. Please review again.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845673#comment-17845673 ] ASF GitHub Bot commented on HDFS-17518: --- hadoop-yetus commented on PR #6809: URL: https://github.com/apache/hadoop/pull/6809#issuecomment-2106176232 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 15s | | https://github.com/apache/hadoop/pull/6809 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6809 | | JIRA Issue | HDFS-17518 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6809/3/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845664#comment-17845664 ] ASF GitHub Bot commented on HDFS-17522: --- hiwangzhihui opened a new pull request, #6814: URL: https://github.com/apache/hadoop/pull/6814 …NS protection ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1 >Reporter: wangzhihui >Priority: Major > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17522) JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection
[ https://issues.apache.org/jira/browse/HDFS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17522: -- Labels: pull-request-available (was: ) > JournalNode web interfaces lack configs for X-FRAME-OPTIONS protection > -- > > Key: HDFS-17522 > URL: https://issues.apache.org/jira/browse/HDFS-17522 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node >Affects Versions: 3.0.0-alpha1 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > > [HDFS-10579 |https://issues.apache.org/jira/browse/HDFS-10579] has added > protection for NameNode and DataNode, but missing protection for JournalNode > web interfaces. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845662#comment-17845662
]
ASF GitHub Bot commented on HDFS-17518:
---
vinayakumarb commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597551434
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src,
INodesInPath iip,
NameNode.stateChangeLog.warn("BLOCK*" +
" internalReleaseLease: All existing blocks are COMPLETE," +
" lease removed, file " + src + " closed.");
- return true; // closed!
+ return false; // closed!
Review Comment:
As per javadoc of this method, return value indicates whether file was
closed or not.
Changing that value here, may solve the problem of logSync() particularly in
this case, but it will be problematic for other usages of this method.
For ex: recoverLease() RPC will always get false, even though file was
closed.
As per the javadoc, even if the return value is false, there are edits
logged (reassigning the lease, when blockrecovery is initiated.).
So calling the logSync() is required in both these cases.
That said, Cannot blindly call logSync() always.
So, more correct approach to fix this is to return a combination of these
values from this method (i.e. complerted and needsync )
And determine whether to call sync or not in the caller.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[
https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845663#comment-17845663
]
ASF GitHub Bot commented on HDFS-17518:
---
vinayakumarb commented on code in PR #6809:
URL: https://github.com/apache/hadoop/pull/6809#discussion_r1597551434
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3738,7 +3738,7 @@ boolean internalReleaseLease(Lease lease, String src,
INodesInPath iip,
NameNode.stateChangeLog.warn("BLOCK*" +
" internalReleaseLease: All existing blocks are COMPLETE," +
" lease removed, file " + src + " closed.");
- return true; // closed!
+ return false; // closed!
Review Comment:
As per javadoc of this method, return value indicates whether file was
closed or not.
Changing that value here, may solve the problem of logSync() particularly in
this case, but it will be problematic for other usages of this method.
For ex: recoverLease() RPC will get false, even though file was closed.
As per the javadoc, even if the return value is false, there are edits
logged (reassigning the lease, when blockrecovery is initiated.).
So calling the logSync() is required in both these cases.
That said, Cannot blindly call logSync() always.
So, more correct approach to fix this is to return a combination of these
values from this method (i.e. complerted and needsync )
And determine whether to call sync or not in the caller.
> In the lease monitor, if a file is closed, we should sync the editslog
> --
>
> Key: HDFS-17518
> URL: https://issues.apache.org/jira/browse/HDFS-17518
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> In the lease monitor, if a file is closed, method checklease will return
> true, and then the edits log will not be sync. In my opinion, we should sync
> the edits log to avoid not synchronizing the state to the standby NameNode
> for a long time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845632#comment-17845632
]
ASF GitHub Bot commented on HDFS-17509:
---
hadoop-yetus commented on PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2105966551
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 88m 01s | | trunk passed |
| +1 :green_heart: | compile | 5m 17s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 07s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 141m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 00s | | the patch passed |
| +1 :green_heart: | compile | 2m 20s | | the patch passed |
| +1 :green_heart: | javac | 2m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 00s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 07s | | the patch passed |
| +1 :green_heart: | shadedclient | 149m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 23s | | The patch does not
generate ASF License warnings. |
| | | 403m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 40ce0a78e9b0 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / acc81392df1ebeb5567c816ad549ce5e81313a99 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6784/3/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6784/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[
https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845624#comment-17845624
]
ASF GitHub Bot commented on HDFS-17506:
---
hadoop-yetus commented on PR #6806:
URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2105929542
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 01s | | The patch appears to
include 2 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 108m 02s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 7m 17s | | HDFS-17384 passed |
| +1 :green_heart: | checkstyle | 5m 38s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 8m 07s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 7m 02s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 177m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 46s | | the patch passed |
| +1 :green_heart: | compile | 4m 29s | | the patch passed |
| +1 :green_heart: | javac | 4m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 08s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 193m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 26s | | The patch does not
generate ASF License warnings. |
| | | 512m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6806 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 a4a169f40776 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / db92d267a8d598d36f9fbe59ec8c24ccf754c558 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/3/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [FGL] Performance for phase 1
> -
>
> Key: HDFS-17506
> URL: https://issues.apache.org/jira/browse/HDFS-17506
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Do some benchmark testing for phase 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17520) TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed
[
https://issues.apache.org/jira/browse/HDFS-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845622#comment-17845622
]
ASF GitHub Bot commented on HDFS-17520:
---
hadoop-yetus commented on PR #6812:
URL: https://github.com/apache/hadoop/pull/6812#issuecomment-2105927891
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 94m 54s | | trunk passed |
| +1 :green_heart: | compile | 6m 43s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 06s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 156m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 00s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 06s | | the patch passed |
| +1 :green_heart: | shadedclient | 170m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 17s | | The patch does not
generate ASF License warnings. |
| | | 450m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6812 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 17b2ed3cc898 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 2d929bda295e05d4b7a3194a7dc311944b662661 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6812/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6812/1/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> TestDFSAdmin.testAllDatanodesReconfig and
> TestDFSAdmin.testDecommissionDataNodesReconfig failed
> ---
>
> Key: HDFS-17520
> URL: https://issues.apache.org/jira/browse/HDFS-17520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
> [ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
> Time elapsed: 2.086 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting:
> <["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
> property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true"",
> "started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
> 13:02:51 UTC 2024."]>
> to contain subsequence:
> <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true""]>
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
>
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845615#comment-17845615
]
ASF GitHub Bot commented on HDFS-17509:
---
hadoop-yetus commented on PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2105783690
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 54s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/6/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 3
unchanged - 0 fixed = 4 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 34s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 176m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d96a40ec6336 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / acc81392df1ebeb5567c816ad549ce5e81313a99 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/6/testReport/ |
| Max. process+thread count | 3527 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Conso
[jira] [Commented] (HDFS-17521) Erasure Coding: Fix calculation errors caused by special index order
[ https://issues.apache.org/jira/browse/HDFS-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845611#comment-17845611 ] ASF GitHub Bot commented on HDFS-17521: --- zhengchenyu opened a new pull request, #6813: URL: https://github.com/apache/hadoop/pull/6813 ### Description of PR I found that if the erasedIndexes distribution is such that the parity index is in front of the data index, ec will produce wrong results when decoding. In fact, [HDFS-15186](https://issues.apache.org/jira/browse/HDFS-15186) has described this problem, but does not fundamentally solve it. The reason is that the code assumes that erasedIndexes is preceded by the data index and followed by parity index. If there is a parity index placed in front of the data index in the incoming code, a calculation error will occur. ### How was this patch tested? The TestErasureCodingEncodeAndDecode unit test and the erasure_code_test binary were executed on different machines. The test machines include those with isa-l installed and those without isa-l installed. ### For code changes: - Make erasedIndexes support arbitrary index distribution. > Erasure Coding: Fix calculation errors caused by special index order > > > Key: HDFS-17521 > URL: https://issues.apache.org/jira/browse/HDFS-17521 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chenyu Zheng >Assignee: Chenyu Zheng >Priority: Critical > > I found that if the erasedIndexes distribution is such that the parity index > is in front of the data index, ec will produce wrong results when decoding. > In fact, HDFS-15186 has described this problem, but does not fundamentally > solve it. > The reason is that the code assumes that erasedIndexes is preceded by the > data index and followed by parity index. If there is a parity index placed in > front of the data index in the incoming code, a calculation error will occur. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17521) Erasure Coding: Fix calculation errors caused by special index order
[ https://issues.apache.org/jira/browse/HDFS-17521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17521: -- Labels: pull-request-available (was: ) > Erasure Coding: Fix calculation errors caused by special index order > > > Key: HDFS-17521 > URL: https://issues.apache.org/jira/browse/HDFS-17521 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chenyu Zheng >Assignee: Chenyu Zheng >Priority: Critical > Labels: pull-request-available > > I found that if the erasedIndexes distribution is such that the parity index > is in front of the data index, ec will produce wrong results when decoding. > In fact, HDFS-15186 has described this problem, but does not fundamentally > solve it. > The reason is that the code assumes that erasedIndexes is preceded by the > data index and followed by parity index. If there is a parity index placed in > front of the data index in the incoming code, a calculation error will occur. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17520) TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed
[
https://issues.apache.org/jira/browse/HDFS-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845609#comment-17845609
]
ASF GitHub Bot commented on HDFS-17520:
---
slfan1989 commented on PR #6812:
URL: https://github.com/apache/hadoop/pull/6812#issuecomment-2105740105
> @slfan1989 Master, I see you are familiar with
`testDecommissionDataNodesReconfig`, please help me review it. Thanks
@ZanderXu Thank you for your contribution! I will reply later.
> TestDFSAdmin.testAllDatanodesReconfig and
> TestDFSAdmin.testDecommissionDataNodesReconfig failed
> ---
>
> Key: HDFS-17520
> URL: https://issues.apache.org/jira/browse/HDFS-17520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
> [ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
> Time elapsed: 2.086 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting:
> <["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
> property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true"",
> "started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
> 13:02:51 UTC 2024."]>
> to contain subsequence:
> <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true""]>
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[
https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845591#comment-17845591
]
ASF GitHub Bot commented on HDFS-17506:
---
hadoop-yetus commented on PR #6806:
URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2105680116
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 21s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 0m 43s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 43s | | HDFS-17384 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 0m 47s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 0m 44s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 7s | | HDFS-17384 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 53s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 21m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 213m 38s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 306m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6806 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6412d6991c5d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / db92d267a8d598d36f9fbe59ec8c24ccf754c558 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/3/testReport/ |
| Max. process+thread count | 4448 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845576#comment-17845576
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597413470
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
Review Comment:
Can you modify the UT to cover the case that one or more source files are
empty?
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -1009,6 +1000,20 @@ public HdfsFileStatus getFileInfo(String src) throws
IOException {
return ret;
}
+ public RemoteResult
getFileRemoteResult(String path)
+ throws IOException {
+rpcServer.checkOperation(NameNode.OperationCategory.READ);
+
+final List locations = rpcServer.getLocationsForPath(path,
false, false);
+RemoteMethod method =
+new RemoteMethod("getFileInfo", new Class[] {String.class}, new
RemoteParam());
+// Check for file information sequentially
+RemoteResult result =
Review Comment:
RBF is simply responsible for locating the downstream namespace and then
proxying the request.
So if the input path is only mounted to one namespace, RBF only needs to
proxy it directly. RBF does not need to check if the file exists in this only
one downstream namespace, right?
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845562#comment-17845562
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597405273
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
Review Comment:
When srclist has a empty file ,both namenode and dfsrouter will throw the
same IOException
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L153-L155
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845559#comment-17845559
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597404047
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -1009,6 +1000,20 @@ public HdfsFileStatus getFileInfo(String src) throws
IOException {
return ret;
}
+ public RemoteResult
getFileRemoteResult(String path)
+ throws IOException {
+rpcServer.checkOperation(NameNode.OperationCategory.READ);
+
+final List locations = rpcServer.getLocationsForPath(path,
false, false);
+RemoteMethod method =
+new RemoteMethod("getFileInfo", new Class[] {String.class}, new
RemoteParam());
+// Check for file information sequentially
+RemoteResult result =
Review Comment:
This may not be true. Even locations only contains one namespace, it still
cannot decide whether the file exists or not. So getFileInfo is better to
execute at least once. Or it will send to namenode and namenode throw file is
not found.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17497) Logic for committed blocks is mixed when computing file size
[
https://issues.apache.org/jira/browse/HDFS-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845556#comment-17845556
]
ASF GitHub Bot commented on HDFS-17497:
---
hfutatzhanghb commented on code in PR #6765:
URL: https://github.com/apache/hadoop/pull/6765#discussion_r1597392676
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:
##
@@ -1105,15 +1106,12 @@ static void unprotectedUpdateCount(INodesInPath
inodesInPath,
/**
* Update the cached quota space for a block that is being completed.
* Must only be called once, as the block is being completed.
- * @param completeBlk - Completed block for which to update space
- * @param inodes - INodes in path to file containing completeBlk; if null
- * this will be resolved internally
+ * @param commitBlock - Committed block for which to update space
+ * @param iip - INodes in path to file containing committedBlock
*/
- public void updateSpaceForCompleteBlock(BlockInfo completeBlk,
- INodesInPath inodes) throws IOException {
+ public void updateSpaceForCommittedBlock(Block commitBlock,
Review Comment:
Which could be better parameter name : commitBlock or committedBlock?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3887,7 +3887,11 @@ void commitOrCompleteLastBlock(
final Block commitBlock) throws IOException {
assert hasWriteLock();
Preconditions.checkArgument(fileINode.isUnderConstruction());
-blockManager.commitOrCompleteLastBlock(fileINode, commitBlock, iip);
+if (!blockManager.commitOrCompleteLastBlock(fileINode, commitBlock)) {
+ return;
+}
+// Updating QuotaUsage when committing block since block size will not be
changed
+getFSDirectory().updateSpaceForCommittedBlock(commitBlock, iip);
Review Comment:
Sir, How about
```java
if (blockManager.commitOrCompleteLastBlock(fileINode, commitBlock)) {
// Updating QuotaUsage when committing block since block size will
not be changed
getFSDirectory().updateSpaceForCommittedBlock(commitBlock, iip);
}
```
> Logic for committed blocks is mixed when computing file size
>
>
> Key: HDFS-17497
> URL: https://issues.apache.org/jira/browse/HDFS-17497
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> One in-writing HDFS file may contains multiple committed blocks, as follows
> (assume one file contains three blocks):
> || ||Block 1||Block 2||Block 3||
> |Case 1|Complete|Commit|UnderConstruction|
> |Case 2|Complete|Commit|Commit|
> |Case 3|Commit|Commit|Commit|
>
> But the logic for committed blocks is mixed when computing file size, it
> ignores the bytes of the last committed block and contains the bytes of other
> committed blocks.
> {code:java}
> public final long computeFileSize(boolean includesLastUcBlock,
> boolean usePreferredBlockSize4LastUcBlock) {
> if (blocks.length == 0) {
> return 0;
> }
> final int last = blocks.length - 1;
> //check if the last block is BlockInfoUnderConstruction
> BlockInfo lastBlk = blocks[last];
> long size = lastBlk.getNumBytes();
> // the last committed block is not complete, so it's bytes may be ignored.
> if (!lastBlk.isComplete()) {
> if (!includesLastUcBlock) {
>size = 0;
> } else if (usePreferredBlockSize4LastUcBlock) {
>size = isStriped()?
>getPreferredBlockSize() *
>((BlockInfoStriped)lastBlk).getDataBlockNum() :
>getPreferredBlockSize();
> }
> }
> // The bytes of other committed blocks are calculated into the file length.
> for (int i = 0; i < last; i++) {
> size += blocks[i].getNumBytes();
> }
> return size;
> } {code}
> The bytes of one committed block will not be changed, so the bytes of the
> last committed block should be calculated into the file length too.
>
> And the logic for committed blocks is mixed too when computing file length in
> DFSInputStream. Normally DFSInputStream does not need to get visible length
> for committed block regardless of whether the committed block is the last
> block or not.
>
> -HDFS-10843- encountered one bug which actually caused by the committed
> block, but -HDFS-10843- fixed that bug by updating quota usage when
> completing block. The num of bytes of the committed block will no longer
> change, so we should update the quota usage when the block is committed,
> which can reduce the delta quota usage in time.
>
> So there are somethings we need to do:
> * Unify the calculation logic for all committed blocks in
> {{compu
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1784#comment-1784
]
ASF GitHub Bot commented on HDFS-17488:
---
hadoop-yetus commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2105622801
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| -1 :x: | patch | 0m 55s | |
https://github.com/apache/hadoop/pull/6759 does not apply to trunk. Rebase
required? Wrong Branch? See
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
|
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6759 |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6759/7/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845551#comment-17845551
]
ASF GitHub Bot commented on HDFS-17488:
---
ZanderXu commented on PR #6759:
URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2105617264
Merged. Thanks @kokonguyen191 for your contribution and thanks @Hexiaoqiao
@haiyang1987 @hfutatzhanghb for your review.
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17488) DN can fail IBRs with NPE when a volume is removed
[
https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845550#comment-17845550
]
ASF GitHub Bot commented on HDFS-17488:
---
ZanderXu merged PR #6759:
URL: https://github.com/apache/hadoop/pull/6759
> DN can fail IBRs with NPE when a volume is removed
> --
>
> Key: HDFS-17488
> URL: https://issues.apache.org/jira/browse/HDFS-17488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Felix N
>Assignee: Felix N
>Priority: Major
> Labels: pull-request-available
>
>
> Error logs
> {code:java}
> 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830
> heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode
> (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool
> BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid
> 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246)
> at
> org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920)
> at java.lang.Thread.run(Thread.java:748) {code}
> The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's
> called on a block belonging to a volume already removed prior. Because the
> volume was already removed
>
> {code:java}
> private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status,
> String delHint, String storageUuid, boolean isOnTransientStorage) {
> checkBlock(block);
> final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo(
> block.getLocalBlock(), status, delHint);
> final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid);
>
> // storage == null here because it's already removed earlier.
> for (BPServiceActor actor : bpServices) {
> actor.getIbrManager().notifyNamenodeBlock(info, storage,
> isOnTransientStorage);
> }
> } {code}
> so IBRs with a null storage are now pending.
> The reason why notifyNamenodeBlock can trigger on such blocks is up in
> DirectoryScanner#reconcile
> {code:java}
> public void reconcile() throws IOException {
> LOG.debug("reconcile start DirectoryScanning");
> scan();
> // If a volume is removed here after scan() already finished running,
> // diffs is stale and checkAndUpdate will run on a removed volume
> // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too
> // long
> int loopCount = 0;
> synchronized (diffs) {
> for (final Map.Entry entry : diffs.getEntries()) {
> dataset.checkAndUpdate(entry.getKey(), entry.getValue());
> ...
> } {code}
> Inside checkAndUpdate, memBlockInfo is null because all the block meta in
> memory is removed during the volume removal, but diskFile still exists. Then
> DataNode#notifyNamenodeDeletedBlock (and further down the line,
> notifyNamenodeBlock) is called on this block.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845548#comment-17845548
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1597385087
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -1009,6 +1000,20 @@ public HdfsFileStatus getFileInfo(String src) throws
IOException {
return ret;
}
+ public RemoteResult
getFileRemoteResult(String path)
+ throws IOException {
+rpcServer.checkOperation(NameNode.OperationCategory.READ);
+
+final List locations = rpcServer.getLocationsForPath(path,
false, false);
+RemoteMethod method =
+new RemoteMethod("getFileInfo", new Class[] {String.class}, new
RemoteParam());
+// Check for file information sequentially
+RemoteResult result =
Review Comment:
If `locations` only contains one namespace, we can returns this namespace
directly instead of getting the namespace through `getFileInfo`, right?
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
Review Comment:
Do the namenode and rbf throw the same Exception?
Maybe RBF throws NPE, but NN throws
`org.apache.hadoop.HadoopIllegalArgumentException`.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17520) TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed
[
https://issues.apache.org/jira/browse/HDFS-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845545#comment-17845545
]
ASF GitHub Bot commented on HDFS-17520:
---
ZanderXu commented on PR #6812:
URL: https://github.com/apache/hadoop/pull/6812#issuecomment-2105610789
@slfan1989 Master, I see you are familiar with
`testDecommissionDataNodesReconfig`, please help me review it. Thanks
> TestDFSAdmin.testAllDatanodesReconfig and
> TestDFSAdmin.testDecommissionDataNodesReconfig failed
> ---
>
> Key: HDFS-17520
> URL: https://issues.apache.org/jira/browse/HDFS-17520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
> [ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
> Time elapsed: 2.086 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting:
> <["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
> property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true"",
> "started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
> 13:02:51 UTC 2024."]>
> to contain subsequence:
> <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true""]>
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17520) TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed
[
https://issues.apache.org/jira/browse/HDFS-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17520:
--
Labels: pull-request-available (was: )
> TestDFSAdmin.testAllDatanodesReconfig and
> TestDFSAdmin.testDecommissionDataNodesReconfig failed
> ---
>
> Key: HDFS-17520
> URL: https://issues.apache.org/jira/browse/HDFS-17520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> [ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
> [ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
> Time elapsed: 2.086 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting:
> <["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
> property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true"",
> "started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
> 13:02:51 UTC 2024."]>
> to contain subsequence:
> <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true""]>
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17520) TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed
[
https://issues.apache.org/jira/browse/HDFS-17520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845544#comment-17845544
]
ASF GitHub Bot commented on HDFS-17520:
---
ZanderXu opened a new pull request, #6812:
URL: https://github.com/apache/hadoop/pull/6812
TestDFSAdmin.testAllDatanodesReconfig and
TestDFSAdmin.testDecommissionDataNodesReconfig failed.
[HDFS-17506](https://github.com/apache/hadoop/pull/6806) encountered this
failed UT and the error message like:
```
[ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
[ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
Time elapsed: 2.086 s <<< FAILURE!
java.lang.AssertionError:
Expecting:
<["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
property dfs.datanode.peer.stats.enabled",
"From: "false"",
"To: "true"",
"started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
13:02:51 UTC 2024."]>
to contain subsequence:
<["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
"From: "false"",
"To: "true""]>
at
org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
```
`getReconfigurationStatusUtil` concurrently get reconfiguration status from
multiple DNs. It may cause messages in out is wrong, such as:
```
Line1: Reconfiguring status for node [127.0.0.1:65229]: Reconfiguring status
for node [127.0.0.1:65224]: started at Sat May 11 15:05:49 CST 2024started at
Sat May 11 15:05:49 CST 2024 and finished at Sat May 11 15:05:49 CST 2024.
Line2: and finished at Sat May 11 15:05:49 CST 2024.
```
> TestDFSAdmin.testAllDatanodesReconfig and
> TestDFSAdmin.testDecommissionDataNodesReconfig failed
> ---
>
> Key: HDFS-17520
> URL: https://issues.apache.org/jira/browse/HDFS-17520
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
>
> {code:java}
> [ERROR] Tests run: 21, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 44.521 s <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdmin
> [ERROR] testAllDatanodesReconfig(org.apache.hadoop.hdfs.tools.TestDFSAdmin)
> Time elapsed: 2.086 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting:
> <["Reconfiguring status for node [127.0.0.1:43731]: SUCCESS: Changed
> property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true"",
> "started at Fri May 10 13:02:51 UTC 2024 and finished at Fri May 10
> 13:02:51 UTC 2024."]>
> to contain subsequence:
> <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
> " From: "false"",
> " To: "true""]>
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1286)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild
[jira] [Commented] (HDFS-17486) VIO: dumpXattrs logic optimization
[ https://issues.apache.org/jira/browse/HDFS-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845541#comment-17845541 ] ASF GitHub Bot commented on HDFS-17486: --- YaAYadeer commented on PR #6797: URL: https://github.com/apache/hadoop/pull/6797#issuecomment-2105598177 The test class TestOfflineImageViewer already exists. TestOfflineImageViewer.testPBImageXmlWriter() method will call dumpXattrs when input parameter of dumpXattrs has xattrs. > VIO: dumpXattrs logic optimization > -- > > Key: HDFS-17486 > URL: https://issues.apache.org/jira/browse/HDFS-17486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.2.0, 3.3.3 >Reporter: wangzhihui >Priority: Minor > Labels: pull-request-available > > The dumpXattrs logic in VIO should use > FSImageFormatPBINode.Loader.loadXAttrs() to get the Xattrs attribute for easy > maintenance. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845530#comment-17845530
]
ASF GitHub Bot commented on HDFS-17514:
---
hadoop-yetus commented on PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#issuecomment-2105541751
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 00s | | trunk passed |
| +1 :green_heart: | compile | 4m 58s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 139m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 2m 19s | | the patch passed |
| +1 :green_heart: | javac | 2m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 08s | | the patch passed |
| +1 :green_heart: | shadedclient | 147m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 14s | | The patch does not
generate ASF License warnings. |
| | | 397m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6804 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 6520897aafc9 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / c2452922760c6ef50b574a5f4a7ec523445da702 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/4/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/4/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[
https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845517#comment-17845517
]
ASF GitHub Bot commented on HDFS-17506:
---
hadoop-yetus commented on PR #6806:
URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2105313781
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 88m 04s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 6m 18s | | HDFS-17384 passed |
| +1 :green_heart: | checkstyle | 4m 39s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 6m 24s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 5m 43s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 144m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 44s | | the patch passed |
| +1 :green_heart: | compile | 3m 26s | | the patch passed |
| +1 :green_heart: | javac | 3m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 155m 07s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 14s | | The patch does not
generate ASF License warnings. |
| | | 415m 09s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6806 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 45a322de9117 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / 3b26cc4461beb5090016838299add8be22d8dd97 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [FGL] Performance for phase 1
> -
>
> Key: HDFS-17506
> URL: https://issues.apache.org/jira/browse/HDFS-17506
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Do some benchmark testing for phase 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13603) Warmup NameNode EDEK thread retries continuously if there's an invalid key
[
https://issues.apache.org/jira/browse/HDFS-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845512#comment-17845512
]
ASF GitHub Bot commented on HDFS-13603:
---
CloudZY commented on code in PR #6774:
URL: https://github.com/apache/hadoop/pull/6774#discussion_r1597249343
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java:
##
@@ -601,18 +604,22 @@ public void run() {
NameNode.LOG.error("Cannot warm up EDEKs.", e);
throw e;
}
-try {
- Thread.sleep(retryInterval);
-} catch (InterruptedException ie) {
- NameNode.LOG.info("EDEKCacheLoader interrupted during retry.");
- break;
+
+if (!success) {
+ try {
+Thread.sleep(retryInterval);
+ } catch (InterruptedException ie) {
+NameNode.LOG.info("EDEKCacheLoader interrupted during retry.");
+break;
+ }
+ retryCount++;
}
-sinceLastLog += retryInterval;
Review Comment:
It does not get updated since initial settings. Shall we add it back or
remove its usages completely as Simba said?
> Warmup NameNode EDEK thread retries continuously if there's an invalid key
> ---
>
> Key: HDFS-13603
> URL: https://issues.apache.org/jira/browse/HDFS-13603
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.8.0
>Reporter: Antony Jay
>Priority: Major
> Labels: pull-request-available
>
> https://issues.apache.org/jira/browse/HDFS-9405 adds a background thread to
> pre-warm EDEK cache.
> However this fails and retries continuously if key retrieval fails for one
> encryption zone. In our usecase, we have temporarily removed keys for certain
> encryption zones. Currently namenode and kms log is filled up with errors
> related to background thread retrying warmup for ever .
> The pre-warm thread should
> * Continue to refresh other encryption zones even if it fails for one
> * Should retry only if it fails for all encryption zones, which will be the
> case when kms is down.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845465#comment-17845465
]
ASF GitHub Bot commented on HDFS-17514:
---
ctrezzo commented on code in PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#discussion_r1597137120
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/PoolAlignmentContext.java:
##
@@ -64,7 +64,11 @@ public void
updateResponseState(RpcHeaderProtos.RpcResponseHeaderProto.Builder h
*/
@Override
public void receiveResponseState(RpcHeaderProtos.RpcResponseHeaderProto
header) {
-sharedGlobalStateId.accumulate(header.getStateId());
+if (header.getStateId() == 0 && sharedGlobalStateId.get() > 0) {
+ sharedGlobalStateId.reset();
+} else {
+ sharedGlobalStateId.accumulate(header.getStateId());
Review Comment:
I have a naive question: What protects us here from the state where
`header.getStateId() > 0 && header.getStateId() < sharedGlobalStateId`?
It seems like if this case were to occur then sharedGlobalStateId would go
backwards.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/PoolAlignmentContext.java:
##
@@ -64,7 +64,11 @@ public void
updateResponseState(RpcHeaderProtos.RpcResponseHeaderProto.Builder h
*/
@Override
public void receiveResponseState(RpcHeaderProtos.RpcResponseHeaderProto
header) {
-sharedGlobalStateId.accumulate(header.getStateId());
+if (header.getStateId() == 0 && sharedGlobalStateId.get() > 0) {
+ sharedGlobalStateId.reset();
+} else {
+ sharedGlobalStateId.accumulate(header.getStateId());
Review Comment:
@simbadzina I have a naive question: What protects us here from the state
where `header.getStateId() > 0 && header.getStateId() < sharedGlobalStateId`?
It seems like if this case were to occur then sharedGlobalStateId would go
backwards.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13603) Warmup NameNode EDEK thread retries continuously if there's an invalid key
[
https://issues.apache.org/jira/browse/HDFS-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845461#comment-17845461
]
ASF GitHub Bot commented on HDFS-13603:
---
simbadzina commented on code in PR #6774:
URL: https://github.com/apache/hadoop/pull/6774#discussion_r1597092684
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/ValueQueue.java:
##
@@ -269,12 +269,23 @@ public ValueQueue(final int numValues, final float
lowWaterMark, long expiry,
* Initializes the Value Queues for the provided keys by calling the
* fill Method with "numInitValues" values
* @param keyNames Array of key Names
- * @throws ExecutionException executionException.
+ * @throws IOException if no successful initialization for any key
Review Comment:
The wording here is confusing. One way to read this is if any key. fails to
initialize, then an except will be thrown. But IIUC an exception will be thrown
if all keys fail to initialize.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java:
##
@@ -537,12 +537,12 @@ static boolean isInAnEZ(final FSDirectory fsd, final
INodesInPath iip)
* then launch up a separate thread to warm them up.
*/
static void warmUpEdekCache(final ExecutorService executor,
- final FSDirectory fsd, final int delay, final int interval) {
+ final FSDirectory fsd, final int delay, final int interval, final int
maxRetries) {
Review Comment:
Can you edit a comment in the function documentation to indicate that the
warm up is best effort.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java:
##
@@ -580,15 +583,15 @@ public void run() {
final int logCoolDown = 1; // periodically print error log (if any)
int sinceLastLog = logCoolDown; // always print the first failure
boolean success = false;
+ int retryCount = 0;
IOException lastSeenIOE = null;
long warmUpEDEKStartTime = monotonicNow();
- while (true) {
+
+ while (!success && retryCount < maxRetries) {
try {
kp.warmUpEncryptedKeys(keyNames);
- NameNode.LOG
- .info("Successfully warmed up {} EDEKs.", keyNames.length);
+ NameNode.LOG.info("Successfully warmed up {} EDEKs.",
keyNames.length);
success = true;
- break;
} catch (IOException ioe) {
lastSeenIOE = ioe;
if (sinceLastLog >= logCoolDown) {
Review Comment:
`sinceLastLog` is no longer really used now. You can just print the failure
since the retry count is limited.
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirEncryptionZoneOp.java:
##
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension;
+import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.NamenodeRole;
+
+import org.junit.Test;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+public class TestFSDirEncryptionZoneOp {
+
+ @Test
+ public void testWarmUpEdekCacheRetries() throws IOException {
+NameNode.initMetrics(new Configuration(), NamenodeRole.NAMENODE);
+
+final int initialDelay = 100;
+final int retryInterval = 100;
+final int maxRetries = 2;
+
+KeyProviderCryptoExtension kpMock = mock(KeyProviderCryptoExtension.class);
+
+doThrow(new IOException())
+.doThrow(new IOException())
+.doAnswer(invocation -> null)
+.when(kpMock).warmUpEncryptedKeys(any());
+
+FSDirEncryptionZoneOp.EDEKCacheLoader loader =
+new FSDirEncryptionZoneOp.EDEKCacheLoader(new String[] {"edek1",
"edek2"}, kpMoc
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[
https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845459#comment-17845459
]
ASF GitHub Bot commented on HDFS-17506:
---
hadoop-yetus commented on PR #6806:
URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2105113000
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 128m 34s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 10m 30s | | HDFS-17384 passed |
| +1 :green_heart: | checkstyle | 7m 37s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 10m 54s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 9m 03s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 218m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 7m 15s | | the patch passed |
| +1 :green_heart: | compile | 5m 24s | | the patch passed |
| +1 :green_heart: | javac | 5m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 6m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 224m 07s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 28s | | The patch does not
generate ASF License warnings. |
| | | 614m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6806 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 ab4e39a29189 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / 3b26cc4461beb5090016838299add8be22d8dd97 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6806/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [FGL] Performance for phase 1
> -
>
> Key: HDFS-17506
> URL: https://issues.apache.org/jira/browse/HDFS-17506
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Do some benchmark testing for phase 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845458#comment-17845458
]
ASF GitHub Bot commented on HDFS-17514:
---
ctrezzo commented on code in PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#discussion_r1596090988
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/PoolAlignmentContext.java:
##
@@ -64,7 +64,11 @@ public void
updateResponseState(RpcHeaderProtos.RpcResponseHeaderProto.Builder h
*/
@Override
public void receiveResponseState(RpcHeaderProtos.RpcResponseHeaderProto
header) {
-sharedGlobalStateId.accumulate(header.getStateId());
+if (header.getStateId() == 0 && sharedGlobalStateId.get() > 0) {
Review Comment:
+1 to fixing the tests and associated check in follow on PR.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845393#comment-17845393 ] ASF GitHub Bot commented on HDFS-17518: --- ThinkerLei opened a new pull request, #6809: URL: https://github.com/apache/hadoop/pull/6809 In the lease monitor, if a file is closed, method checklease will return true, and then the edits log will not be sync. In my opinion, we should sync the edits log to avoid not synchronizing the state to the standby NameNode for a long time. > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17518) In the lease monitor, if a file is closed, we should sync the editslog
[ https://issues.apache.org/jira/browse/HDFS-17518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17518: -- Labels: pull-request-available (was: ) > In the lease monitor, if a file is closed, we should sync the editslog > -- > > Key: HDFS-17518 > URL: https://issues.apache.org/jira/browse/HDFS-17518 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > In the lease monitor, if a file is closed, method checklease will return > true, and then the edits log will not be sync. In my opinion, we should sync > the edits log to avoid not synchronizing the state to the standby NameNode > for a long time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845355#comment-17845355
]
ASF GitHub Bot commented on HDFS-17514:
---
hadoop-yetus commented on PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#issuecomment-2104652320
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 108m 42s | | trunk passed |
| +1 :green_heart: | compile | 5m 55s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 174m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 29s | | the patch passed |
| +1 :green_heart: | compile | 2m 51s | | the patch passed |
| +1 :green_heart: | javac | 2m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 189m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 21s | | The patch does not
generate ASF License warnings. |
| | | 500m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6804 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 d80193c83cbe 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / c2452922760c6ef50b574a5f4a7ec523445da702 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/3/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845350#comment-17845350
]
ASF GitHub Bot commented on HDFS-17509:
---
hadoop-yetus commented on PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#issuecomment-2104604443
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 31m 50s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 177m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0b91b258fcfd 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1fea88ce11670322a73d6d20f1e61f93ba7783b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/5/testReport/ |
| Max. process+thread count | 3575 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6784/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Fix Cli
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[
https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845348#comment-17845348
]
ASF GitHub Bot commented on HDFS-17506:
---
hadoop-yetus commented on PR #6806:
URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2104599790
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| -1 :x: | mvninstall | 35m 34s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/artifact/out/branch-mvninstall-root.txt)
| root in HDFS-17384 failed. |
| +1 :green_heart: | compile | 0m 44s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 40s | | HDFS-17384 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 43s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 0m 45s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 0m 43s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 9s | | HDFS-17384 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 56s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 21m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 212m 26s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 306m 48s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6806/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6806 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3535f61db013 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / 3b26cc4461beb5090016838299add8be22d8dd97 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versi
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845287#comment-17845287
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1596579974
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
Review Comment:
When there is a empty source file, it will throw exception in namenode.
This behaiver is as same as with dfsrouter.
And when trg is a empty file, it is diffrent . Without dfsrouter ,it
success. And with dfsrouter, it will throw Exception.
So I think there is no need to check the empty source file. Or implement
it in another PR.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16368) DFSAdmin supports refresh topology info without restarting namenode
[
https://issues.apache.org/jira/browse/HDFS-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845259#comment-17845259
]
ASF GitHub Bot commented on HDFS-16368:
---
hadoop-yetus commented on PR #3743:
URL: https://github.com/apache/hadoop/pull/3743#issuecomment-2104271718
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 01s | | buf was not available. |
| +0 :ok: | buf | 0m 01s | | buf was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 94m 33s | | trunk passed |
| +1 :green_heart: | compile | 42m 11s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 27s | | trunk passed |
| -1 :x: | mvnsite | 4m 53s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/3/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 21m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 194m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 17m 39s | | the patch passed |
| +1 :green_heart: | compile | 40m 05s | | the patch passed |
| +1 :green_heart: | cc | 40m 05s | | the patch passed |
| +1 :green_heart: | javac | 40m 05s | | the patch passed |
| -1 :x: | blanks | 0m 00s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/3/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 6m 18s | | the patch passed |
| -1 :x: | mvnsite | 4m 41s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/3/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 21m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 202m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 14s | | The patch does not
generate ASF License warnings. |
| | | 609m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/3743 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | MINGW64_NT-10.0-17763 67fad41479c9 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 2e5165fd3cb5b9e8ed8dfdade8e2b4874033a182 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/3/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/3/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DFSAdmin supports refresh topology info without restarting namenode
>
>
> Key: HDFS-16368
> URL: https://issues.apache.org/jira/browse/HDFS-16368
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: dfsadmin, namanode
>Affects Versions:
[jira] [Commented] (HDFS-17486) VIO: dumpXattrs logic optimization
[
https://issues.apache.org/jira/browse/HDFS-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845248#comment-17845248
]
ASF GitHub Bot commented on HDFS-17486:
---
hadoop-yetus commented on PR #6797:
URL: https://github.com/apache/hadoop/pull/6797#issuecomment-2104197718
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 89m 30s | | trunk passed |
| +1 :green_heart: | compile | 5m 53s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 03s | | trunk passed |
| +1 :green_heart: | shadedclient | 151m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 39s | | the patch passed |
| +1 :green_heart: | compile | 3m 27s | | the patch passed |
| +1 :green_heart: | javac | 3m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 160m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 23s | | The patch does not
generate ASF License warnings. |
| | | 428m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6797 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 5ae641b16f00 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 2f2b0c95e9420e8fbed97ce41534b0f9576b35b7 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6797/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6797/2/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> VIO: dumpXattrs logic optimization
> --
>
> Key: HDFS-17486
> URL: https://issues.apache.org/jira/browse/HDFS-17486
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.2.0, 3.3.3
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> The dumpXattrs logic in VIO should use
> FSImageFormatPBINode.Loader.loadXAttrs() to get the Xattrs attribute for easy
> maintenance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845223#comment-17845223
]
ASF GitHub Bot commented on HDFS-17514:
---
hadoop-yetus commented on PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#issuecomment-2104068739
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 50s | | trunk passed |
| +1 :green_heart: | compile | 4m 53s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 141m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 47s | | the patch passed |
| +1 :green_heart: | compile | 2m 19s | | the patch passed |
| +1 :green_heart: | javac | 2m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 01s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 07s | | the patch passed |
| +1 :green_heart: | shadedclient | 152m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 47s | | The patch does not
generate ASF License warnings. |
| | | 405m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6804 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 faac0eb4a8ee 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / c2452922760c6ef50b574a5f4a7ec523445da702 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845215#comment-17845215
]
ASF GitHub Bot commented on HDFS-17514:
---
hadoop-yetus commented on PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#issuecomment-2103966017
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 85m 42s | | trunk passed |
| +1 :green_heart: | compile | 4m 46s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 141m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 47s | | the patch passed |
| +1 :green_heart: | compile | 2m 17s | | the patch passed |
| +1 :green_heart: | javac | 2m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 07s | | the patch passed |
| +1 :green_heart: | shadedclient | 148m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 16s | | The patch does not
generate ASF License warnings. |
| | | 399m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6804 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 505bedc9962d 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / c2452922760c6ef50b574a5f4a7ec523445da702 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6804/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17511) method storagespaceConsumedContiguous should use BlockInfo#getReplication to compute dsDelta
[ https://issues.apache.org/jira/browse/HDFS-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845210#comment-17845210 ] ASF GitHub Bot commented on HDFS-17511: --- skyskyhu commented on PR #6799: URL: https://github.com/apache/hadoop/pull/6799#issuecomment-2103926821 Hi @ChenSammi @jojochuang @ayushtkn , could you please help review this PR when you have free time~ Thanks a lot. > method storagespaceConsumedContiguous should use BlockInfo#getReplication to > compute dsDelta > > > Key: HDFS-17511 > URL: https://issues.apache.org/jira/browse/HDFS-17511 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > As title says, we should use BlockInfo#getReplication to compute storage > space in method INodeFile#storagespaceConsumedContiguous. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[
https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845185#comment-17845185
]
ASF GitHub Bot commented on HDFS-17506:
---
ferhui commented on code in PR #6806:
URL: https://github.com/apache/hadoop/pull/6806#discussion_r1596215333
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/fgl/FSNLockBenchmarkThroughput.java:
##
@@ -0,0 +1,269 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.namenode.fgl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdfs.HdfsConfiguration;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * This class benchmarks the throughput of NN for both global-lock and
fine-grained lock.
+ */
+public class FSNLockBenchmarkThroughput {
+
+ private final int readWriteRatio;
+ private final int testingCount;
+ private final ExecutorService executorService;
+ private final FileSystem fileSystem;
+
+ public FSNLockBenchmarkThroughput(FileSystem fileSystem,
+ int readWriteRatio, int testingCount, int concurrency) {
+this.fileSystem = fileSystem;
+this.readWriteRatio = readWriteRatio;
+this.testingCount = testingCount;
+this.executorService = Executors.newFixedThreadPool(concurrency);
+ }
+
+ public void benchmark(String lockName) throws Exception {
+System.out.println("Do benchmark for " + lockName);
+Path basePath = new Path("/tmp/fsnlock/benchmark/throughput");
Review Comment:
How about making the path as an input? if no input, can set a default path.
> [FGL] Performance for phase 1
> -
>
> Key: HDFS-17506
> URL: https://issues.apache.org/jira/browse/HDFS-17506
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Do some benchmark testing for phase 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845184#comment-17845184
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1596217290
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
we don't need to get NSId or BPId from the result of getFileInfo, you can
refer to `invokeSequential` to loop all namespaces one by one.
1. Get all namespaces of the input path.
2. Send getFileInfo to each namespace one by one
3. The first namespace that the result of getFileInfo is not null is the one
we need
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845182#comment-17845182 ] ASF GitHub Bot commented on HDFS-17506: --- ferhui commented on PR #6806: URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2103801948 > > Thanks. What is the purpose of this PR? Provide a benchmark tool and each one can test the HDFS cluster performance? or just test the namenode performance locally? > > `FSNLockBenchmarkThroughput` is a tool, each one can do the performance tests through this tool after deploying FGL. `TestFSNLockBenchmarkThroughput` is a UT, it mocks a `MiniQJMHACluster` and do some performance tests locally. Got it, thanks. > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17503) Unreleased volume references because of OOM
[ https://issues.apache.org/jira/browse/HDFS-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845179#comment-17845179 ] ASF GitHub Bot commented on HDFS-17503: --- zhuzilong2013 commented on PR #6782: URL: https://github.com/apache/hadoop/pull/6782#issuecomment-2103787150 Thanks @ZanderXu for your review and merge~ > Unreleased volume references because of OOM > --- > > Key: HDFS-17503 > URL: https://issues.apache.org/jira/browse/HDFS-17503 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zilong Zhu >Assignee: Zilong Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > When BlockSender throws an error because of OOM，the volume reference obtained > by the thread is not released，which causes the thread trying to remove the > volume to wait and fall into an infinite loop. > I found HDFS-15963 catched exception and release volume reference. But it did > not handle the case of throwing errors. I think "catch (Throwable t)" should > be used instead of "catch (IOException ioe)". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845174#comment-17845174
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1596189673
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
>May we can use getFileInfo instead of getBlockLocation to fix this bug.
BTW, getFileInfo is no needed if this path only mounts to one namespace.
At the beginning , I consider use getFileInfo . But the problem is that
HdfsFileStatus does not have any information about nameservices or blockpoolid,
only LocatedBlock has.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17503) Unreleased volume references because of OOM
[ https://issues.apache.org/jira/browse/HDFS-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845173#comment-17845173 ] ASF GitHub Bot commented on HDFS-17503: --- ZanderXu commented on PR #6782: URL: https://github.com/apache/hadoop/pull/6782#issuecomment-2103752898 Merged. Thanks @zhuzilong2013 for your contribution. > Unreleased volume references because of OOM > --- > > Key: HDFS-17503 > URL: https://issues.apache.org/jira/browse/HDFS-17503 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zilong Zhu >Assignee: Zilong Zhu >Priority: Major > Labels: pull-request-available > > When BlockSender throws an error because of OOM，the volume reference obtained > by the thread is not released，which causes the thread trying to remove the > volume to wait and fall into an infinite loop. > I found HDFS-15963 catched exception and release volume reference. But it did > not handle the case of throwing errors. I think "catch (Throwable t)" should > be used instead of "catch (IOException ioe)". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17503) Unreleased volume references because of OOM
[ https://issues.apache.org/jira/browse/HDFS-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845172#comment-17845172 ] ASF GitHub Bot commented on HDFS-17503: --- ZanderXu merged PR #6782: URL: https://github.com/apache/hadoop/pull/6782 > Unreleased volume references because of OOM > --- > > Key: HDFS-17503 > URL: https://issues.apache.org/jira/browse/HDFS-17503 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zilong Zhu >Assignee: Zilong Zhu >Priority: Major > Labels: pull-request-available > > When BlockSender throws an error because of OOM，the volume reference obtained > by the thread is not released，which causes the thread trying to remove the > volume to wait and fall into an infinite loop. > I found HDFS-15963 catched exception and release volume reference. But it did > not handle the case of throwing errors. I think "catch (Throwable t)" should > be used instead of "catch (IOException ioe)". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845171#comment-17845171
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1596183555
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
> For a file in Router view with more than one nameservices, I think should
thrown Exception for concat method.
For multiple namespaces contain one same file case, RBF just return the file
in the first namespace currently, such as: getBlockLocation, getFileInfo, etc.
So If you want to thrown Exception for concat, maybe you need to modify all
RPCs to throw Exception for this case.
May we can use getFileInfo instead of getBlockLocation to fix this bug. BTW,
getFileInfo is no needed if this path only mounts to one namespace.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845170#comment-17845170 ] ASF GitHub Bot commented on HDFS-17506: --- ZanderXu commented on PR #6806: URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2103741784 > Thanks. What is the purpose of this PR? Provide a benchmark tool and each one can test the HDFS cluster performance? or just test the namenode performance locally? `FSNLockBenchmarkThroughput` is a tool, each one can do the performance tests through this tool after deploying FGL. `TestFSNLockBenchmarkThroughput` is a UT, it mocks a `MiniQJMHACluster` and do some performance tests locally. > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845167#comment-17845167 ] ASF GitHub Bot commented on HDFS-17506: --- ferhui commented on PR #6806: URL: https://github.com/apache/hadoop/pull/6806#issuecomment-2103691546 Thanks. What is the purpose of this PR? Provide a benchmark tool and each one can test the HDFS cluster performance? or just test the namenode performance locally? > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[
https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845166#comment-17845166
]
ASF GitHub Bot commented on HDFS-17438:
---
hadoop-yetus commented on PR #6655:
URL: https://github.com/apache/hadoop/pull/6655#issuecomment-2103686670
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 88m 40s | | trunk passed |
| +1 :green_heart: | compile | 40m 29s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 51s | | trunk passed |
| -1 :x: | mvnsite | 4m 20s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/7/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 9m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 163m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 49s | | the patch passed |
| +1 :green_heart: | compile | 36m 52s | | the patch passed |
| +1 :green_heart: | javac | 36m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 49s | | the patch passed |
| -1 :x: | mvnsite | 4m 19s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/7/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 9m 04s | | the patch passed |
| +1 :green_heart: | shadedclient | 166m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 28s | | The patch does not
generate ASF License warnings. |
| | | 521m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6655 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 ae72a0e5a6ae 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 50d9f7c20216e78abe2b5c89282dff62375a |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/7/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/7/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
> -
>
> Key: HDFS-17438
> URL: https://issues.apache.org/jira/browse/HDFS-17438
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17438.001.patch
>
>
> At present, when the status of all namenodes in an ns in the router is the
> same, the namenode which is the newest reported will be placed at the top of
> the cache. when the client accesses the ns through the router, it will first
> access the namenode.
> If multiple namenodes in this route are in an active state, or if there are
> namenodes wi
[jira] [Commented] (HDFS-17476) fix: False positive "Observer Node is too far behind" due to long overflow.
[
https://issues.apache.org/jira/browse/HDFS-17476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845162#comment-17845162
]
ASF GitHub Bot commented on HDFS-17476:
---
hadoop-yetus commented on PR #6747:
URL: https://github.com/apache/hadoop/pull/6747#issuecomment-2103618886
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 35s | | trunk passed |
| +1 :green_heart: | compile | 6m 26s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 08s | | trunk passed |
| +1 :green_heart: | shadedclient | 144m 09s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 23s | | the patch passed |
| +1 :green_heart: | javac | 3m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 154m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 17s | | The patch does not
generate ASF License warnings. |
| | | 413m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6747 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 cdbf5294217d 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 5f6e761f92c501b59ae53a552e64b5d6f54f20c1 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6747/3/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6747/3/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> fix: False positive "Observer Node is too far behind" due to long overflow.
> ---
>
> Key: HDFS-17476
> URL: https://issues.apache.org/jira/browse/HDFS-17476
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Critical
> Labels: pull-request-available
> Attachments: HDFS-17476.patch, image-2024-04-18-10-57-10-481.png
>
>
> In the code GlobalStateIdContext#receiveRequestState(), if clientStateId is a
> small negative number, clientStateId-serverStateId may be greater than
> （ESTIMATED_TRANSACTIONS_PER_SECOND due to overflow
> * TimeUnit.MILLISECONDS.toSeconds(clientWaitTime)
> * ESTIMATED_SERVER_TIME_MULTIPLIER）,
> resulting in false positives that Observer Node is too far behind.
> !image-2024-04-18-10-57-10-481.png|width=742,height=110!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16368) DFSAdmin supports refresh topology info without restarting namenode
[
https://issues.apache.org/jira/browse/HDFS-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845156#comment-17845156
]
ASF GitHub Bot commented on HDFS-16368:
---
hadoop-yetus commented on PR #3743:
URL: https://github.com/apache/hadoop/pull/3743#issuecomment-2103598386
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 01s | | buf was not available. |
| +0 :ok: | buf | 0m 01s | | buf was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 93m 28s | | trunk passed |
| +1 :green_heart: | compile | 42m 01s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 08s | | trunk passed |
| -1 :x: | mvnsite | 4m 41s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/2/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 20m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 191m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 16m 39s | | the patch passed |
| +1 :green_heart: | compile | 38m 28s | | the patch passed |
| +1 :green_heart: | cc | 38m 28s | | the patch passed |
| +1 :green_heart: | javac | 38m 28s | | the patch passed |
| -1 :x: | blanks | 0m 01s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/2/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 6m 24s | | the patch passed |
| -1 :x: | mvnsite | 4m 46s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/2/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 21m 09s | | the patch passed |
| +1 :green_heart: | shadedclient | 200m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 44s | | The patch does not
generate ASF License warnings. |
| | | 600m 09s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/3743 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | MINGW64_NT-10.0-17763 29a0d9562666 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 2e5165fd3cb5b9e8ed8dfdade8e2b4874033a182 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/2/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-3743/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> DFSAdmin supports refresh topology info without restarting namenode
>
>
> Key: HDFS-16368
> URL: https://issues.apache.org/jira/browse/HDFS-16368
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: dfsadmin, namanode
>Affects Versions:
[jira] [Commented] (HDFS-17438) RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
[
https://issues.apache.org/jira/browse/HDFS-17438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845152#comment-17845152
]
ASF GitHub Bot commented on HDFS-17438:
---
hadoop-yetus commented on PR #6655:
URL: https://github.com/apache/hadoop/pull/6655#issuecomment-2103584427
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 93m 05s | | trunk passed |
| +1 :green_heart: | compile | 41m 31s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 27s | | trunk passed |
| -1 :x: | mvnsite | 4m 44s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/6/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 10m 05s | | trunk passed |
| +1 :green_heart: | shadedclient | 169m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 02s | | the patch passed |
| +1 :green_heart: | compile | 42m 17s | | the patch passed |
| +1 :green_heart: | javac | 42m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 6m 43s | | the patch passed |
| -1 :x: | mvnsite | 5m 00s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/6/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 10m 04s | | the patch passed |
| +1 :green_heart: | shadedclient | 179m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 04s | | The patch does not
generate ASF License warnings. |
| | | 553m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6655 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 d4e3488ee1e9 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 50d9f7c20216e78abe2b5c89282dff62375a |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/6/testReport/
|
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs-rbf U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6655/6/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: The newest STANDBY and UNAVAILABLE nn should be the lowest priority.
> -
>
> Key: HDFS-17438
> URL: https://issues.apache.org/jira/browse/HDFS-17438
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17438.001.patch
>
>
> At present, when the status of all namenodes in an ns in the router is the
> same, the namenode which is the newest reported will be placed at the top of
> the cache. when the client accesses the ns through the router, it will first
> access the namenode.
> If multiple namenodes in this route are in an active state, or if there are
> namenodes wi
[jira] [Commented] (HDFS-16993) Datanode supports configure TopN DatanodeNetworkCounts
[
https://issues.apache.org/jira/browse/HDFS-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845120#comment-17845120
]
ASF GitHub Bot commented on HDFS-16993:
---
hadoop-yetus commented on PR #5597:
URL: https://github.com/apache/hadoop/pull/5597#issuecomment-2103362606
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 111m 38s | | trunk passed |
| +1 :green_heart: | compile | 8m 16s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 02s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 190m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 57s | | the patch passed |
| +1 :green_heart: | compile | 4m 21s | | the patch passed |
| +1 :green_heart: | javac | 4m 21s | | the patch passed |
| -1 :x: | blanks | 0m 00s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-5597/2/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 2m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 194m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 32s | | The patch does not
generate ASF License warnings. |
| | | 530m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/5597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 f08ad7bac7e6 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / ecd02f00aa8adecb6f79a6422b6752d9e711fd60 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-5597/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-5597/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Datanode supports configure TopN DatanodeNetworkCounts
> --
>
> Key: HDFS-16993
> URL: https://issues.apache.org/jira/browse/HDFS-16993
> Project: Hadoop HDFS
> Issue Type: Wish
>Affects Versions: 3.3.5
>Reporter: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> In our prod environment, we try to collect datanode metrics every 15s through
> jmx_exporter. we found the datanodenetworkerror metric generates a lot.
> for example, if we have a cluster with 1000 datanodes, every datanode may
> generate 999 datanodenetworkerror metrics, and overall datanodes will
> generate 1000 multiple 999 = 999000 metrics. This is a very expensive
> operation. In most scenarios, we only need the topN of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17276) The nn fetch editlog forbidden in kerberos environment
[
https://issues.apache.org/jira/browse/HDFS-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845069#comment-17845069
]
ASF GitHub Bot commented on HDFS-17276:
---
hadoop-yetus commented on PR #6326:
URL: https://github.com/apache/hadoop/pull/6326#issuecomment-2103056076
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 02s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 127m 17s | | trunk passed |
| +1 :green_heart: | compile | 10m 24s | | trunk passed |
| +1 :green_heart: | checkstyle | 7m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 10m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 212m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 58s | | the patch passed |
| +1 :green_heart: | compile | 5m 19s | | the patch passed |
| +1 :green_heart: | javac | 5m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 6m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 226m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 8m 28s | | The patch does not
generate ASF License warnings. |
| | | 609m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6326 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 4cdb66d2f064 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 74263653dbc9a16564d41d2a8bcd975d47d5d93f |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6326/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6326/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> The nn fetch editlog forbidden in kerberos environment
> --
>
> Key: HDFS-17276
> URL: https://issues.apache.org/jira/browse/HDFS-17276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm, security
>Affects Versions: 3.3.5, 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-12-06-20-21-03-557.png,
> image-2023-12-06-20-21-46-825.png
>
>
> * In a Kerberos environment, the namenode cannot fetch editlog from
> journalnode because the request is rejected (403).
> !image-2023-12-06-20-21-03-557.png!
> * GetJournalEditServlet checks if the request's username meets the
> requirements through the isValidRequestor function. After HDFS-16686 is
> merged, remotePrincipal becomes ugi.getUserName().
> * In a Kerberos environment, ugi.getUserName() gets the
> request.getRemoteUser() via DfsServlet's getUGI to get the username, and this
> username is not a full name.
> * Therefore, the obtained username is similar to namenode01 instead of
> namenode01/hos...@realm.tld, which meansit fails to pass the isValidRequestor
> check. !image-2023-12-06-20-21-46-825.png!
> *reproduction*
> * In the TestGetJournalEditServlet add testSecurityRequestNameNode
> {code:java}
> @Test
> public void testSecurityRequestNameNode() throws IOException,
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844966#comment-17844966
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1595385718
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
Emmm, there is a scene .
1) If a file is already exist in two nameservices. And the add router mount.
NS1 /user/test/file
NS2/user/test/file
2) Add router mount.
hdfs dfsrouteradmin -add /user/test NS1,NS2 /user/test -order RANDOM
3) getDestination
hdfs dfsrouteradmin -getDestination /user/test/file
Will return NS1,NS2
For a file in Router view with more than one nameservices, I think should
thrown Exception for concat method.
Look forward to your guidance , thanks ! @ZanderXu
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844945#comment-17844945
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1595273510
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
DFSRouter with different order(HASH,LOCAL, RANDOM, HASH_ALL, SPACE) , a file
cannot be written into two or more nameservices.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844944#comment-17844944
]
ASF GitHub Bot commented on HDFS-17509:
---
LiuGuH commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1595269674
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
Thanks for review.
> you can refer to this case:
>
> * /path mounts to NS1, NS2 and NS3
> * NS2 and NS3 contains /path
> * `OrderedResolver` returns NS1, NS2 and NS3
>
> for this case, we should proxy this `concat` to NS2 instead of NS1, right?
This only happens when /path is a directory. And for a file ,it must only be
from one exactly nameservice via dfsrouter.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844943#comment-17844943
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1595242159
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
Normally a file only exits in one Namespace. But maybe multiple namespaces
contain this file and RBF returns the file in the first namespace to the client.
This first namespace is got by the `OrderedResolver`.
you can refer to this case:
- /path mounts to NS1, NS2 and NS3
- NS2 and NS3 contains /path
- `OrderedResolver` returns NS1, NS2 and NS3
for this case, we should proxy this `concat` to NS2 instead of NS1, right?
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17509) RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
[
https://issues.apache.org/jira/browse/HDFS-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844940#comment-17844940
]
ASF GitHub Bot commented on HDFS-17509:
---
ZanderXu commented on code in PR #6784:
URL: https://github.com/apache/hadoop/pull/6784#discussion_r1595242159
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
Normally a file only exits in one Namespace. But maybe multiple namespaces
contain this file and RBF returns the file in the first namespace to the client.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -667,39 +667,28 @@ public void rename2(final String src, final String dst,
public void concat(String trg, String[] src) throws IOException {
rpcServer.checkOperation(NameNode.OperationCategory.WRITE);
-// See if the src and target files are all in the same namespace
-LocatedBlocks targetBlocks = getBlockLocations(trg, 0, 1);
-if (targetBlocks == null) {
- throw new IOException("Cannot locate blocks for target file - " + trg);
-}
-LocatedBlock lastLocatedBlock = targetBlocks.getLastLocatedBlock();
-String targetBlockPoolId = lastLocatedBlock.getBlock().getBlockPoolId();
-for (String source : src) {
- LocatedBlocks sourceBlocks = getBlockLocations(source, 0, 1);
- if (sourceBlocks == null) {
-throw new IOException(
-"Cannot located blocks for source file " + source);
- }
- String sourceBlockPoolId =
- sourceBlocks.getLastLocatedBlock().getBlock().getBlockPoolId();
- if (!sourceBlockPoolId.equals(targetBlockPoolId)) {
-throw new IOException("Cannot concatenate source file " + source
-+ " because it is located in a different namespace"
-+ " with block pool id " + sourceBlockPoolId
-+ " from the target file with block pool id "
-+ targetBlockPoolId);
- }
-}
+// Concat only effects when all files in same namespace.
+// And in router view, a file only exists in one RemoteLocation.
Review Comment:
For the empty file, maybe we can use `getFileInfo` to get the namespace that
this trg belongs to.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -1224,6 +1224,17 @@ public void testProxyConcatFile() throws Exception {
String badPath = "/unknownlocation/unknowndir";
compareResponses(routerProtocol, nnProtocol, m,
new Object[] {badPath, new String[] {routerFile}});
+
+// Test when concat trg is a empty file
Review Comment:
we also need to check the empty source file.
> RBF: Fix ClientProtocol.concat will throw NPE if tgr is a empty file.
> --
>
> Key: HDFS-17509
> URL: https://issues.apache.org/jira/browse/HDFS-17509
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> hdfs dfs -concat /tmp/merge /tmp/t1 /tmp/t2
> When /tmp/merge is a empty file, this command will throw NPE via DFSRouter.
>
>
--
This message was sent by Atla
[jira] [Commented] (HDFS-17515) Erasure Coding: ErasureCodingWork is not effectively limited during a block reconstruction cycle.
[
https://issues.apache.org/jira/browse/HDFS-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844927#comment-17844927
]
ASF GitHub Bot commented on HDFS-17515:
---
zhengchenyu commented on PR #6805:
URL: https://github.com/apache/hadoop/pull/6805#issuecomment-2102237952
Not ready for review! Wait for HDFS-17516!
> Erasure Coding: ErasureCodingWork is not effectively limited during a block
> reconstruction cycle.
> -
>
> Key: HDFS-17515
> URL: https://issues.apache.org/jira/browse/HDFS-17515
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chenyu Zheng
>Assignee: Chenyu Zheng
>Priority: Major
> Labels: pull-request-available
>
> In a block reconstruction cycle, ErasureCodingWork is not effectively
> limited. I add some debug log, log when ecBlocksToBeReplicated is an integer
> multiple of 100.
> {code:java}
> 2024-05-09 10:46:06,986 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 100 blocks
> 2024-05-09 10:46:06,987 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 200 blocks
> ...
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2000 blocks
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2100 blocks {code}
> During a block reconstruction cycle, ecBlocksToBeReplicated increases from 0
> to 2100, This is much larger than replicationStreamsHardLimit. This brings
> unfairness and leads to a greater tendency to copy EC blocks.
> In fact, for non ec block, this is not a problem.
> pendingReplicationWithoutTargets increase when schedule work. When
> pendingReplicationWithoutTargets is too big, will not schedule work for this
> node.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17515) Erasure Coding: ErasureCodingWork is not effectively limited during a block reconstruction cycle.
[
https://issues.apache.org/jira/browse/HDFS-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844928#comment-17844928
]
ASF GitHub Bot commented on HDFS-17515:
---
zhengchenyu closed pull request #6805: HDFS-17515. Erasure Coding:
ErasureCodingWork is not effectively limi…
URL: https://github.com/apache/hadoop/pull/6805
> Erasure Coding: ErasureCodingWork is not effectively limited during a block
> reconstruction cycle.
> -
>
> Key: HDFS-17515
> URL: https://issues.apache.org/jira/browse/HDFS-17515
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chenyu Zheng
>Assignee: Chenyu Zheng
>Priority: Major
> Labels: pull-request-available
>
> In a block reconstruction cycle, ErasureCodingWork is not effectively
> limited. I add some debug log, log when ecBlocksToBeReplicated is an integer
> multiple of 100.
> {code:java}
> 2024-05-09 10:46:06,986 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 100 blocks
> 2024-05-09 10:46:06,987 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 200 blocks
> ...
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2000 blocks
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2100 blocks {code}
> During a block reconstruction cycle, ecBlocksToBeReplicated increases from 0
> to 2100, This is much larger than replicationStreamsHardLimit. This brings
> unfairness and leads to a greater tendency to copy EC blocks.
> In fact, for non ec block, this is not a problem.
> pendingReplicationWithoutTargets increase when schedule work. When
> pendingReplicationWithoutTargets is too big, will not schedule work for this
> node.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17506) [FGL] Performance for phase 1
[ https://issues.apache.org/jira/browse/HDFS-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17506: -- Labels: pull-request-available (was: ) > [FGL] Performance for phase 1 > - > > Key: HDFS-17506 > URL: https://issues.apache.org/jira/browse/HDFS-17506 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Priority: Major > Labels: pull-request-available > > Do some benchmark testing for phase 1. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17503) Unreleased volume references because of OOM
[ https://issues.apache.org/jira/browse/HDFS-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844895#comment-17844895 ] ASF GitHub Bot commented on HDFS-17503: --- zhuzilong2013 commented on PR #6782: URL: https://github.com/apache/hadoop/pull/6782#issuecomment-2102119137 @Hexiaoqiao Hi~ sir. Could you please help me review this PR when you are free? Thanks. > Unreleased volume references because of OOM > --- > > Key: HDFS-17503 > URL: https://issues.apache.org/jira/browse/HDFS-17503 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zilong Zhu >Assignee: Zilong Zhu >Priority: Major > Labels: pull-request-available > > When BlockSender throws an error because of OOM，the volume reference obtained > by the thread is not released，which causes the thread trying to remove the > volume to wait and fall into an infinite loop. > I found HDFS-15963 catched exception and release volume reference. But it did > not handle the case of throwing errors. I think "catch (Throwable t)" should > be used instead of "catch (IOException ioe)". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17504) DN process should exit when BPServiceActor exit
[ https://issues.apache.org/jira/browse/HDFS-17504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844894#comment-17844894 ] ASF GitHub Bot commented on HDFS-17504: --- zhuzilong2013 commented on PR #6792: URL: https://github.com/apache/hadoop/pull/6792#issuecomment-2102118561 @Hexiaoqiao Hi~ sir. Could you please help me review this PR when you are free? Thanks. > DN process should exit when BPServiceActor exit > --- > > Key: HDFS-17504 > URL: https://issues.apache.org/jira/browse/HDFS-17504 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zilong Zhu >Assignee: Zilong Zhu >Priority: Major > Labels: pull-request-available > > BPServiceActor is a very important thread. In a non-HA cluster, the exit of > the BPServiceActor thread will cause the DN process to exit. However, in a HA > cluster, this is not the case. > I found HDFS-15651 causes BPServiceActor thread to exit and sets the > "runningState" from "RunningState.FAILED" to "RunningState.EXITED", it can > be confusing during troubleshooting. > I believe that the DN process should exit when the flag of the BPServiceActor > is set to RunningState.FAILED because at this point, the DN is unable to > recover and establish a heartbeat connection with the ANN on its own. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17515) Erasure Coding: ErasureCodingWork is not effectively limited during a block reconstruction cycle.
[
https://issues.apache.org/jira/browse/HDFS-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844880#comment-17844880
]
ASF GitHub Bot commented on HDFS-17515:
---
zhengchenyu opened a new pull request, #6805:
URL: https://github.com/apache/hadoop/pull/6805
### Description of PR
https://issues.apache.org/jira/browse/HDFS-17515
### How was this patch tested?
unit test
### For code changes:
- Add pendingECBlockReplicationWithoutTargets for DatanodeDescriptor
- When construct ErasureCodingWork, increase
pendingECBlockReplicationWithoutTargets. When set targets, decrease
pendingECBlockReplicationWithoutTargets.
- Because we can not decide the real source datanode when
scheduleReconstruction, so update pendingECBlockReplicationWithoutTargets for
all source datanode. So when we calculate getNumberOfBlocksToBeReplicated, use
a factor to appropriately lower the value.
> Erasure Coding: ErasureCodingWork is not effectively limited during a block
> reconstruction cycle.
> -
>
> Key: HDFS-17515
> URL: https://issues.apache.org/jira/browse/HDFS-17515
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chenyu Zheng
>Assignee: Chenyu Zheng
>Priority: Major
>
> In a block reconstruction cycle, ErasureCodingWork is not effectively
> limited. I add some debug log, log when ecBlocksToBeReplicated is an integer
> multiple of 100.
> {code:java}
> 2024-05-09 10:46:06,986 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 100 blocks
> 2024-05-09 10:46:06,987 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 200 blocks
> ...
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2000 blocks
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2100 blocks {code}
> During a block reconstruction cycle, ecBlocksToBeReplicated increases from 0
> to 2100, This is much larger than replicationStreamsHardLimit. This brings
> unfairness and leads to a greater tendency to copy EC blocks.
> In fact, for non ec block, this is not a problem.
> pendingReplicationWithoutTargets increase when schedule work. When
> pendingReplicationWithoutTargets is too big, will not schedule work for this
> node.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17515) Erasure Coding: ErasureCodingWork is not effectively limited during a block reconstruction cycle.
[
https://issues.apache.org/jira/browse/HDFS-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17515:
--
Labels: pull-request-available (was: )
> Erasure Coding: ErasureCodingWork is not effectively limited during a block
> reconstruction cycle.
> -
>
> Key: HDFS-17515
> URL: https://issues.apache.org/jira/browse/HDFS-17515
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chenyu Zheng
>Assignee: Chenyu Zheng
>Priority: Major
> Labels: pull-request-available
>
> In a block reconstruction cycle, ErasureCodingWork is not effectively
> limited. I add some debug log, log when ecBlocksToBeReplicated is an integer
> multiple of 100.
> {code:java}
> 2024-05-09 10:46:06,986 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 100 blocks
> 2024-05-09 10:46:06,987 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 200 blocks
> ...
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2000 blocks
> 2024-05-09 10:46:06,992 DEBUG
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManagerZCY:
> ecBlocksToBeReplicated for IP:PORT already have 2100 blocks {code}
> During a block reconstruction cycle, ecBlocksToBeReplicated increases from 0
> to 2100, This is much larger than replicationStreamsHardLimit. This brings
> unfairness and leads to a greater tendency to copy EC blocks.
> In fact, for non ec block, this is not a problem.
> pendingReplicationWithoutTargets increase when schedule work. When
> pendingReplicationWithoutTargets is too big, will not schedule work for this
> node.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17514) RBF: Routers keep using cached stateID even when active NN returns unset header
[
https://issues.apache.org/jira/browse/HDFS-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844876#comment-17844876
]
ASF GitHub Bot commented on HDFS-17514:
---
simbadzina commented on code in PR #6804:
URL: https://github.com/apache/hadoop/pull/6804#discussion_r1594952346
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/PoolAlignmentContext.java:
##
@@ -64,7 +64,11 @@ public void
updateResponseState(RpcHeaderProtos.RpcResponseHeaderProto.Builder h
*/
@Override
public void receiveResponseState(RpcHeaderProtos.RpcResponseHeaderProto
header) {
-sharedGlobalStateId.accumulate(header.getStateId());
+if (header.getStateId() == 0 && sharedGlobalStateId.get() > 0) {
Review Comment:
The tests in TestNoNamenodesAvailableLongTime rely on the router allowing a
stateId of 0. So having `sharedGlobalStateId.get() > 0` allows this behavior
while guarding against when the sharedGlobalStateId has advances beyond zero.
The tests in `TestNoNamenodesAvailableLongTime` do need to be fixed but I
would like to limit the scope of this PR.
> RBF: Routers keep using cached stateID even when active NN returns unset
> header
> ---
>
> Key: HDFS-17514
> URL: https://issues.apache.org/jira/browse/HDFS-17514
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Minor
> Labels: pull-request-available
>
> When a namenode that had "dfs.namenode.state.context.enabled" set to true is
> restarted with the configuration set to false, routers will keep using a
> previously cached state ID.
> Without RBF
> * clients that fetched the old stateID could have stale reads even after
> msyncing
> * new clients will go to the active.
> With RBF
> * client that fetched the old stateID could have stale reads like above.
> * New clients will also fetch the stale stateID and potentially have stale
> reads
> New clients that are created after the restart should not fetch the stale
> state ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
