[jira] [Commented] (HDFS-17152) Fix the documentation of count command in FileSystemShell.md
[ https://issues.apache.org/jira/browse/HDFS-17152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795245#comment-17795245 ] ASF GitHub Bot commented on HDFS-17152: --- zhangshuyan0 merged PR #5939: URL: https://github.com/apache/hadoop/pull/5939 > Fix the documentation of count command in FileSystemShell.md > > > Key: HDFS-17152 > URL: https://issues.apache.org/jira/browse/HDFS-17152 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Trivial > Labels: pull-request-available > > count -q means show quotas and usage. > count -u means show quotas. > We should fix this minor documentation error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17152) Fix the documentation of count command in FileSystemShell.md
[ https://issues.apache.org/jira/browse/HDFS-17152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuyan Zhang resolved HDFS-17152. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix the documentation of count command in FileSystemShell.md > > > Key: HDFS-17152 > URL: https://issues.apache.org/jira/browse/HDFS-17152 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > count -q means show quotas and usage. > count -u means show quotas. > We should fix this minor documentation error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[
https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795249#comment-17795249
]
ASF GitHub Bot commented on HDFS-17281:
---
hadoop-yetus commented on PR #6337:
URL: https://github.com/apache/hadoop/pull/6337#issuecomment-1849601339
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 17s | | trunk passed |
| +1 :green_heart: | compile | 8m 15s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 37s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 7m 51s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 7m 51s | | the patch passed |
| +1 :green_heart: | javac | 7m 51s | | the patch passed |
| +1 :green_heart: | compile | 7m 32s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | cc | 7m 32s | | the patch passed |
| +1 :green_heart: | javac | 7m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 56s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6337/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 325 unchanged - 0 fixed = 326 total (was
325) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 15s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 19m 13s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 159m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6337/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6337 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | Linux 1b542689e491 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4736bb8926a381a5329e385511d5e4b06e9f4ff9 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-
[jira] [Commented] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections
[
https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795268#comment-17795268
]
ASF GitHub Bot commented on HDFS-15413:
---
Neilxzn commented on PR #5829:
URL: https://github.com/apache/hadoop/pull/5829#issuecomment-1849677715
> @Neilxzn I tried & it fails locally
>
> ```
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
17.64 s <<< FAILURE! - in
org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
> [ERROR]
testPreadTimeout(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout)
Time elapsed: 17.509 s <<< FAILURE!
> java.lang.AssertionError: It Should fail to read striped time out with 1
attempt .
> at org.junit.Assert.fail(Assert.java:89)
> at
org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout.testPreadTimeout(TestDFSStripedInputStreamWithTimeout.java:145)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:750)
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR] TestDFSStripedInputStreamWithTimeout.testPreadTimeout:145 It
Should fail to read striped time out with 1 attempt .
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> ```
>
> To reproduce: in the hadoop root directory there is file named
`start-build-env.sh`, run that `bash start-build-env.sh`, it will give you a
docker env, run the test inside that & it will fail
Thank you. I will check it again soon.
> DFSStripedInputStream throws exception when datanodes close idle connections
>
>
> Key: HDFS-15413
> URL: https://issues.apache.org/jira/browse/HDFS-15413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding, hdfs-client
>Affects Versions: 3.1.3
> Environment: - Hadoop 3.1.3
> - erasure coding with ISA-L and RS-3-2-1024k scheme
> - running in kubernetes
> - dfs.client.socket-timeout = 1
> - dfs.datanode.socket.write.timeout = 1
>Reporter: Andrey Elenskiy
>Priority: Critical
> Labels: pull-request-available
> Attachments: out.log
>
>
> We've run into an issue with compactions failing in HBase when erasure coding
> is enabled on a table directory. After digging further I was able to narrow
> it down to a seek + read logic and able to reproduce the issue with hdfs
> client only:
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> public class ReaderRaw {
> public static void main(final String[] args) throws Exception {
> Path p = new Path(args[0]);
> int bufLen = Integer.parseInt(args[1]);
> int sleepDuration = Integer.parseInt(args[2]);
> int countBeforeSleep = Integer.parseInt(args[3]);
> int countAfterSleep = Integer.parseInt(args[4]);
> Configuration conf = new Configuration();
> FSDataInputStream istream = FileSystem.get(conf).open(p);
> byte[] buf = new byte[bufLen];
> int readTotal = 0;
> int count = 0;
> try {
> while (true) {
> istream.seek(readTotal);
>
[jira] [Created] (HDFS-17284) Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery
Hualong Zhang created HDFS-17284: Summary: Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery Key: HDFS-17284 URL: https://issues.apache.org/jira/browse/HDFS-17284 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Hualong Zhang Assignee: Hualong Zhang Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16196) Namesystem#completeFile method will log incorrect path information when router to access
[ https://issues.apache.org/jira/browse/HDFS-16196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w updated HDFS-16196: - Resolution: Fixed Status: Resolved (was: Patch Available) > Namesystem#completeFile method will log incorrect path information when > router to access > > > Key: HDFS-16196 > URL: https://issues.apache.org/jira/browse/HDFS-16196 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16196.001.patch > > > Router not send entire path information to namenode because > ClientProtocol#complete method`s parameter with fileId. Then NameNode will > log incorrect path information. This is very confusing, should we let the > router pass the path information or modify the log path on namenode? > completeFile log as fllow: > StateChange: DIR* completeFile: / is closed by DFSClient_NONMAPREDUC_* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16020) DatanodeReportType should add LIVE_NOT_DECOMMISSIONING type
[ https://issues.apache.org/jira/browse/HDFS-16020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lei w resolved HDFS-16020. -- Resolution: Invalid > DatanodeReportType should add LIVE_NOT_DECOMMISSIONING type > --- > > Key: HDFS-16020 > URL: https://issues.apache.org/jira/browse/HDFS-16020 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16020.001.patch > > > Balancer builds cluster nodes by > getDatanodeStorageReport(DatanodeReportType.LIVE) method。If the user does not > specify the exclude node list, the balancer may migrate data to the DataNode > in the decommission state. Should we filter out nodes in the decommission > state by a new DatanodeReportType(LIVE_NOT_DECOMMISSIONING) regardless of > whether the user specifies the exclude node list ? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16102) Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to save time
[
https://issues.apache.org/jira/browse/HDFS-16102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lei w resolved HDFS-16102.
--
Resolution: Invalid
> Remove redundant iteration in BlockManager#removeBlocksAssociatedTo(...) to
> save time
> --
>
> Key: HDFS-16102
> URL: https://issues.apache.org/jira/browse/HDFS-16102
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Minor
> Attachments: HDFS-16102.001.patch
>
>
> The current logic in removeBlocksAssociatedTo(...) is as follows：
> {code:java}
> void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
> providedStorageMap.removeDatanode(node);
> for (DatanodeStorageInfo storage : node.getStorageInfos()) {
> final Iterator it = storage.getBlockIterator();
> //add the BlockInfos to a new collection as the
> //returned iterator is not modifiable.
> Collection toRemove = new ArrayList<>();
> while (it.hasNext()) {
> toRemove.add(it.next()); // First iteration : to put blocks to
> another collection
> }
> for (BlockInfo b : toRemove) {
> removeStoredBlock(b, node); // Another iteration : to remove blocks
> }
> }
> // ..
> }
> {code}
> In fact , we can use the first iteration to achieve this logic , so should
> we remove the redundant iteration to save time and memory?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16083: -- Labels: pull-request-available (was: ) > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.1.patch, > HDFS-16083.005.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795376#comment-17795376 ] ASF GitHub Bot commented on HDFS-16083: --- ThinkerLei opened a new pull request, #6343: URL: https://github.com/apache/hadoop/pull/6343 When the Observer NameNode is turned on in the cluster, the Active NameNode will receive rollEditLog RPC requests from the Standby NameNode and Observer NameNode in a short time. Observer NameNode's rollEditLog request is a repetitive operation, so should we forbid Observer NameNode trigger active namenode roll editsLog > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.1.patch, > HDFS-16083.005.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17267) Client send the smae packet multiple times when method markSlowNode throws IOException.
[
https://issues.apache.org/jira/browse/HDFS-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795409#comment-17795409
]
ASF GitHub Bot commented on HDFS-17267:
---
hadoop-yetus commented on PR #6311:
URL: https://github.com/apache/hadoop/pull/6311#issuecomment-1850280283
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 15s | | trunk passed |
| +1 :green_heart: | compile | 2m 56s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 46s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 49s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 189m 46s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 294m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 557e59a81cb3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 633bbb1717600d019cf22b44746087580b295aa2 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
[jira] [Commented] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
[
https://issues.apache.org/jira/browse/HDFS-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795431#comment-17795431
]
ASF GitHub Bot commented on HDFS-17280:
---
hadoop-yetus commented on PR #6336:
URL: https://github.com/apache/hadoop/pull/6336#issuecomment-1850426479
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 30s | | trunk passed |
| +1 :green_heart: | compile | 5m 20s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 5m 14s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 16s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | | the patch passed |
| +1 :green_heart: | compile | 5m 14s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 5m 14s | | the patch passed |
| +1 :green_heart: | compile | 5m 4s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 5m 4s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/11/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 1m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 25s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 213m 51s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 382m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6336 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b2495fb968ee 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1ec6bae99071a71c5bda869d0a9ee7a52d5d6873 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd6
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[
https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795471#comment-17795471
]
ASF GitHub Bot commented on HDFS-16083:
---
hadoop-yetus commented on PR #6343:
URL: https://github.com/apache/hadoop/pull/6343#issuecomment-1850636348
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 130 unchanged
- 0 fixed = 139 total (was 130) |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 183m 6s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 269m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6343 |
| JIRA Issue | HDFS-16083 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6be53f9fa535 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a50b726807677ffec9f9a6a3e1734fcba070d9e1 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Tes
[jira] [Commented] (HDFS-17267) Client send the smae packet multiple times when method markSlowNode throws IOException.
[
https://issues.apache.org/jira/browse/HDFS-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795497#comment-17795497
]
ASF GitHub Bot commented on HDFS-17267:
---
hadoop-yetus commented on PR #6311:
URL: https://github.com/apache/hadoop/pull/6311#issuecomment-1850784164
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 43s | | trunk passed |
| +1 :green_heart: | compile | 3m 28s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 7s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 22s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 22s | | the patch passed |
| +1 :green_heart: | compile | 3m 2s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 3m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 53s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 185m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 297m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 64c64aaa4551 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh
[jira] [Commented] (HDFS-17267) Client send the smae packet multiple times when method markSlowNode throws IOException.
[
https://issues.apache.org/jira/browse/HDFS-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795501#comment-17795501
]
ASF GitHub Bot commented on HDFS-17267:
---
hadoop-yetus commented on PR #6311:
URL: https://github.com/apache/hadoop/pull/6311#issuecomment-1850809526
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 28s | | trunk passed |
| +1 :green_heart: | compile | 2m 52s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 3m 4s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 4s | | the patch passed |
| +1 :green_heart: | compile | 3m 23s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 3m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 47s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 206m 1s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 318m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestErasureCodingPolicies |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 11467317b2ab 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3c0d085e232b6d50a678879ddf3e4bdda60c07e5 |
| Default Java | Private Bui
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795570#comment-17795570 ] ASF GitHub Bot commented on HDFS-16083: --- ThinkerLei commented on PR #6343: URL: https://github.com/apache/hadoop/pull/6343#issuecomment-1851205116 @hadoop-yetus > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.1.patch, > HDFS-16083.005.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795575#comment-17795575 ] ASF GitHub Bot commented on HDFS-16083: --- tomscut commented on PR #6343: URL: https://github.com/apache/hadoop/pull/6343#issuecomment-1851237828 Thanks for reporting this. We talked about this before, see https://github.com/apache/hadoop/pull/4087. > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.1.patch, > HDFS-16083.005.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795580#comment-17795580 ] lei w commented on HDFS-16083: -- [~tomscut] Is this reasonable? > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.1.patch, > HDFS-16083.005.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17276) The nn fetch editlog forbidden in kerberos environment
[
https://issues.apache.org/jira/browse/HDFS-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795582#comment-17795582
]
ASF GitHub Bot commented on HDFS-17276:
---
gp1314 commented on PR #6326:
URL: https://github.com/apache/hadoop/pull/6326#issuecomment-1851244732
@sunchao , could you please make a cr for this modification? I would be very
grateful.
> The nn fetch editlog forbidden in kerberos environment
> --
>
> Key: HDFS-17276
> URL: https://issues.apache.org/jira/browse/HDFS-17276
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm, security
>Affects Versions: 3.3.5, 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-12-06-20-21-03-557.png,
> image-2023-12-06-20-21-46-825.png
>
>
> * In a Kerberos environment, the namenode cannot fetch editlog from
> journalnode because the request is rejected (403).
> !image-2023-12-06-20-21-03-557.png!
> * GetJournalEditServlet checks if the request's username meets the
> requirements through the isValidRequestor function. After HDFS-16686 is
> merged, remotePrincipal becomes ugi.getUserName().
> * In a Kerberos environment, ugi.getUserName() gets the
> request.getRemoteUser() via DfsServlet's getUGI to get the username, and this
> username is not a full name.
> * Therefore, the obtained username is similar to namenode01 instead of
> namenode01/hos...@realm.tld, which meansit fails to pass the isValidRequestor
> check. !image-2023-12-06-20-21-46-825.png!
> *reproduction*
> * In the TestGetJournalEditServlet add testSecurityRequestNameNode
> {code:java}
> @Test
> public void testSecurityRequestNameNode() throws IOException,
> ServletException {
> // Test: Make a request from a namenode
> CONF.set(HADOOP_SECURITY_AUTHENTICATION, "kerberos");
> UserGroupInformation.setConfiguration(CONF);
>
> HttpServletRequest request = mock(HttpServletRequest.class);
>
> when(request.getParameter(UserParam.NAME)).thenReturn("nn/localh...@realm.tld");
> when(request.getRemoteUser()).thenReturn("jn");
> boolean isValid = SERVLET.isValidRequestor(request, CONF);
>
> assertThat(isValid).isTrue();
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[ https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795583#comment-17795583 ] ASF GitHub Bot commented on HDFS-16083: --- ThinkerLei commented on PR #6343: URL: https://github.com/apache/hadoop/pull/6343#issuecomment-1851251616 > Thanks for reporting this. We talked about this before, see #4087. It's very funny. Please see HDFS-16083 > Forbid Observer NameNode trigger active namenode log roll > -- > > Key: HDFS-16083 > URL: https://issues.apache.org/jira/browse/HDFS-16083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Reporter: lei w >Assignee: lei w >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-16083.001.patch, HDFS-16083.002.patch, > HDFS-16083.003.patch, HDFS-16083.004.patch, HDFS-16083.005.1.patch, > HDFS-16083.005.patch, activeRollEdits.png > > > When the Observer NameNode is turned on in the cluster, the Active NameNode > will receive rollEditLog RPC requests from the Standby NameNode and Observer > NameNode in a short time. Observer NameNode's rollEditLog request is a > repetitive operation, so should we forbid Observer NameNode trigger active > namenode log roll ? We 'dfs.ha.log-roll.period' configured is 300( 5 > minutes) and active NameNode receives rollEditLog RPC as shown in > activeRollEdits.png -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17267) Client send the smae packet multiple times when method markSlowNode throws IOException.
[
https://issues.apache.org/jira/browse/HDFS-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795613#comment-17795613
]
ASF GitHub Bot commented on HDFS-17267:
---
hadoop-yetus commented on PR #6311:
URL: https://github.com/apache/hadoop/pull/6311#issuecomment-1851399396
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 28s | | trunk passed |
| +1 :green_heart: | compile | 2m 55s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 49s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 2m 43s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 48s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 187m 56s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 293m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a7737ef5daa9 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 318bb6cf9ee37248ef46887b2c17b0d961a01b9a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/10/testReport/ |
| Max. process+thread count | 4254 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[
https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795615#comment-17795615
]
ASF GitHub Bot commented on HDFS-16083:
---
hadoop-yetus commented on PR #6343:
URL: https://github.com/apache/hadoop/pull/6343#issuecomment-1851402685
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 2s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 130 unchanged
- 0 fixed = 139 total (was 130) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 167m 40s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 25s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/3/artifact/out/results-asflicense.txt)
| The patch generated 193 ASF License warnings. |
| | | 261m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestStoragePolicySummary |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.namenode.TestReencryptionHandler |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | ha
[jira] [Commented] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[
https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795617#comment-17795617
]
ASF GitHub Bot commented on HDFS-17281:
---
xinglin commented on PR #6337:
URL: https://github.com/apache/hadoop/pull/6337#issuecomment-1851408051
The following two assertions failed during one of the builds. Investigating.
```
assertCounter("RoundTripLockAndSleepNumOps", 1L, rttRB);
assertTrue(rttLockAndSleepAvgTime > serverSideLockAndSleepAvgTime);
```
```
[ERROR] testRpcRTTMetric(org.apache.hadoop.ipc.TestRPC) Time elapsed: 0.04
s <<< FAILURE!
java.lang.AssertionError: Bad value for metric RoundTripLockAndSleepNumOps
expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at
org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:230)
at org.apache.hadoop.ipc.TestRPC.testRpcRTTMetric(TestRPC.java:1518)
```
> Added support of reporting RPC round-trip time at NN.
> -
>
> Key: HDFS-17281
> URL: https://issues.apache.org/jira/browse/HDFS-17281
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png
>
>
> We have come across a few cases where the hdfs clients are reporting very bad
> latencies, while we don't see similar trends at NN-side. Instead, from
> NN-side, the latency metrics seem normal as usual. I attached a screenshot
> which we took during an internal investigation at LinkedIn. What was
> happening is a token management service was reporting an average latency of 1
> sec in fetching delegation tokens from our NN but at the NN-side, we did not
> see anything abnormal. The recent OverallRpcProcessingTime metric we added in
> HDFS-17042 did not seem to be sufficient to identify/signal such cases.
> We propose to extend the IPC header in hadoop, to communicate call create
> time at client-side to IPC servers, so that for each rpc call, the server can
> get its round-trip time.
>
> *Why is OverallRpcProcessingTime not sufficient?*
> OverallRpcProcessingTime captures the time starting from when the reader
> thread reads in the call from the socket to when the response is sent back to
> the client. As a result, it does not capture the time it takes to transmit
> the call from client to the server. Besides, we only have a couple of reader
> threads to monitor a large number of open connections. It is possible that
> many connections become ready to read at the same time. Then, the reader
> thread would need to read each call sequentially, leading to a wait time for
> many Rpc Calls. We have also hit the case where the callQueue becomes full
> (with a total of 25600 requests) and thus reader threads are blocked to add
> new Calls into the callQueue. This would lead to a longer latency for all
> connections/calls which are ready and wait to be read by reader threads.
> Ideally, we want to measure the time between when a socket/call is ready to
> read and when it is actually being read by the reader thread. This would give
> us the wait time that a call is taking to be read. However, after some Google
> search, we failed to find a way to get this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-17278) Detect order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module
[
https://issues.apache.org/jira/browse/HDFS-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reassigned HDFS-17278:
---
Assignee: Ruby
> Detect order dependent flakiness in TestViewfsWithNfs3.java under
> hadoop-hdfs-nfs module
>
>
> Key: HDFS-17278
> URL: https://issues.apache.org/jira/browse/HDFS-17278
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: openjdk version "17.0.9"
> Apache Maven 3.9.5
>Reporter: Ruby
>Assignee: Ruby
>Priority: Minor
> Labels: pull-request-available
> Attachments: failed-1.png, failed-2.png, success.png
>
>
> The order dependent flakiness was detected if the test class
> TestDFSClientCache.java runs before TestRpcProgramNfs3.java.
> The error message looks like below:
> {code:java}
> [ERROR] Failures:
> [ERROR] TestRpcProgramNfs3.testAccess:279 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testCommit:764 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testCreate:493 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR]
> TestRpcProgramNfs3.testEncryptedReadWrite:359->createFileUsingNfs:393
> Incorrect response: expected: but
> was:
> [ERROR] TestRpcProgramNfs3.testFsinfo:714 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testFsstat:696 Incorrect return code:
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testGetattr:205 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testLookup:249 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testMkdir:517 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testPathconf:738 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRead:341 Incorrect return code: expected:<0>
> but was:<13>
> [ERROR] TestRpcProgramNfs3.testReaddir:642 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReaddirplus:666 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReadlink:297 Incorrect return code:
> expected:<0> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRemove:570 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRename:618 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRmdir:594 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSetattr:225 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSymlink:546 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testWrite:468 Incorrect return code:
> expected:<13> but was:<5>
> [INFO]
> [ERROR] Tests run: 25, Failures: 20, Errors: 0, Skipped: 0
> [INFO]
> [ERROR] There are test failures. {code}
> The polluter that led to this flakiness was the test method
> testGetUserGroupInformationSecure() in TestDFSClientCache.java. There was a
> line
> {code:java}
> UserGroupInformation.setLoginUser(currentUserUgi);{code}
> which modifies some shared state and resource, something like pre-setup the
> config. To fix this issue, I added the cleanup methods in
> TestDFSClientCache.java to reset the UserGroupInformation to ensure the
> isolation among each test class.
> {code:java}
> @AfterClass
> public static void cleanup() {
> UserGroupInformation.reset();
> }{code}
> Including setting
> {code:java}
> authenticationMethod = null;
> conf = null; // set configuration to null
> setLoginUser(null); // reset login user to default null{code}
> ..., and so on. The reset() methods can be referred to
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java.
> After the fix, the error was no longer exist and the succeed message was:
> {code:java}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 18.457 s - in org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> {code}
> Here is the CustomTest.java file that I used to run these two tests in order,
> the error can be reproduce by running this CustomTest.java.
> {code:java}
> package org.apac
[jira] [Commented] (HDFS-17278) Detect order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module
[
https://issues.apache.org/jira/browse/HDFS-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795620#comment-17795620
]
Ayush Saxena commented on HDFS-17278:
-
Committed to trunk.
Thanx [~yijujt2] for the contribution!!!
> Detect order dependent flakiness in TestViewfsWithNfs3.java under
> hadoop-hdfs-nfs module
>
>
> Key: HDFS-17278
> URL: https://issues.apache.org/jira/browse/HDFS-17278
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: openjdk version "17.0.9"
> Apache Maven 3.9.5
>Reporter: Ruby
>Assignee: Ruby
>Priority: Minor
> Labels: pull-request-available
> Attachments: failed-1.png, failed-2.png, success.png
>
>
> The order dependent flakiness was detected if the test class
> TestDFSClientCache.java runs before TestRpcProgramNfs3.java.
> The error message looks like below:
> {code:java}
> [ERROR] Failures:
> [ERROR] TestRpcProgramNfs3.testAccess:279 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testCommit:764 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testCreate:493 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR]
> TestRpcProgramNfs3.testEncryptedReadWrite:359->createFileUsingNfs:393
> Incorrect response: expected: but
> was:
> [ERROR] TestRpcProgramNfs3.testFsinfo:714 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testFsstat:696 Incorrect return code:
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testGetattr:205 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testLookup:249 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testMkdir:517 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testPathconf:738 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRead:341 Incorrect return code: expected:<0>
> but was:<13>
> [ERROR] TestRpcProgramNfs3.testReaddir:642 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReaddirplus:666 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReadlink:297 Incorrect return code:
> expected:<0> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRemove:570 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRename:618 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRmdir:594 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSetattr:225 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSymlink:546 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testWrite:468 Incorrect return code:
> expected:<13> but was:<5>
> [INFO]
> [ERROR] Tests run: 25, Failures: 20, Errors: 0, Skipped: 0
> [INFO]
> [ERROR] There are test failures. {code}
> The polluter that led to this flakiness was the test method
> testGetUserGroupInformationSecure() in TestDFSClientCache.java. There was a
> line
> {code:java}
> UserGroupInformation.setLoginUser(currentUserUgi);{code}
> which modifies some shared state and resource, something like pre-setup the
> config. To fix this issue, I added the cleanup methods in
> TestDFSClientCache.java to reset the UserGroupInformation to ensure the
> isolation among each test class.
> {code:java}
> @AfterClass
> public static void cleanup() {
> UserGroupInformation.reset();
> }{code}
> Including setting
> {code:java}
> authenticationMethod = null;
> conf = null; // set configuration to null
> setLoginUser(null); // reset login user to default null{code}
> ..., and so on. The reset() methods can be referred to
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java.
> After the fix, the error was no longer exist and the succeed message was:
> {code:java}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 18.457 s - in org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> {code}
> Here is the CustomTest.java file that I used to run these two tests in order,
>
[jira] [Commented] (HDFS-17278) Detect order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module
[
https://issues.apache.org/jira/browse/HDFS-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795619#comment-17795619
]
ASF GitHub Bot commented on HDFS-17278:
---
ayushtkn merged PR #6329:
URL: https://github.com/apache/hadoop/pull/6329
> Detect order dependent flakiness in TestViewfsWithNfs3.java under
> hadoop-hdfs-nfs module
>
>
> Key: HDFS-17278
> URL: https://issues.apache.org/jira/browse/HDFS-17278
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: openjdk version "17.0.9"
> Apache Maven 3.9.5
>Reporter: Ruby
>Assignee: Ruby
>Priority: Minor
> Labels: pull-request-available
> Attachments: failed-1.png, failed-2.png, success.png
>
>
> The order dependent flakiness was detected if the test class
> TestDFSClientCache.java runs before TestRpcProgramNfs3.java.
> The error message looks like below:
> {code:java}
> [ERROR] Failures:
> [ERROR] TestRpcProgramNfs3.testAccess:279 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testCommit:764 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testCreate:493 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR]
> TestRpcProgramNfs3.testEncryptedReadWrite:359->createFileUsingNfs:393
> Incorrect response: expected: but
> was:
> [ERROR] TestRpcProgramNfs3.testFsinfo:714 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testFsstat:696 Incorrect return code:
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testGetattr:205 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testLookup:249 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testMkdir:517 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testPathconf:738 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRead:341 Incorrect return code: expected:<0>
> but was:<13>
> [ERROR] TestRpcProgramNfs3.testReaddir:642 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReaddirplus:666 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReadlink:297 Incorrect return code:
> expected:<0> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRemove:570 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRename:618 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRmdir:594 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSetattr:225 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSymlink:546 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testWrite:468 Incorrect return code:
> expected:<13> but was:<5>
> [INFO]
> [ERROR] Tests run: 25, Failures: 20, Errors: 0, Skipped: 0
> [INFO]
> [ERROR] There are test failures. {code}
> The polluter that led to this flakiness was the test method
> testGetUserGroupInformationSecure() in TestDFSClientCache.java. There was a
> line
> {code:java}
> UserGroupInformation.setLoginUser(currentUserUgi);{code}
> which modifies some shared state and resource, something like pre-setup the
> config. To fix this issue, I added the cleanup methods in
> TestDFSClientCache.java to reset the UserGroupInformation to ensure the
> isolation among each test class.
> {code:java}
> @AfterClass
> public static void cleanup() {
> UserGroupInformation.reset();
> }{code}
> Including setting
> {code:java}
> authenticationMethod = null;
> conf = null; // set configuration to null
> setLoginUser(null); // reset login user to default null{code}
> ..., and so on. The reset() methods can be referred to
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java.
> After the fix, the error was no longer exist and the succeed message was:
> {code:java}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 18.457 s - in org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> {code}
> Here is the CustomTest.java file that I used to run these two
[jira] [Resolved] (HDFS-17278) Detect order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module
[
https://issues.apache.org/jira/browse/HDFS-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HDFS-17278.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Detect order dependent flakiness in TestViewfsWithNfs3.java under
> hadoop-hdfs-nfs module
>
>
> Key: HDFS-17278
> URL: https://issues.apache.org/jira/browse/HDFS-17278
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: openjdk version "17.0.9"
> Apache Maven 3.9.5
>Reporter: Ruby
>Assignee: Ruby
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: failed-1.png, failed-2.png, success.png
>
>
> The order dependent flakiness was detected if the test class
> TestDFSClientCache.java runs before TestRpcProgramNfs3.java.
> The error message looks like below:
> {code:java}
> [ERROR] Failures:
> [ERROR] TestRpcProgramNfs3.testAccess:279 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testCommit:764 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testCreate:493 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR]
> TestRpcProgramNfs3.testEncryptedReadWrite:359->createFileUsingNfs:393
> Incorrect response: expected: but
> was:
> [ERROR] TestRpcProgramNfs3.testFsinfo:714 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testFsstat:696 Incorrect return code:
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testGetattr:205 Incorrect return code
> expected:<0> but was:<13>
> [ERROR] TestRpcProgramNfs3.testLookup:249 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testMkdir:517 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testPathconf:738 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRead:341 Incorrect return code: expected:<0>
> but was:<13>
> [ERROR] TestRpcProgramNfs3.testReaddir:642 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReaddirplus:666 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testReadlink:297 Incorrect return code:
> expected:<0> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRemove:570 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRename:618 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testRmdir:594 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSetattr:225 Incorrect return code
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testSymlink:546 Incorrect return code:
> expected:<13> but was:<5>
> [ERROR] TestRpcProgramNfs3.testWrite:468 Incorrect return code:
> expected:<13> but was:<5>
> [INFO]
> [ERROR] Tests run: 25, Failures: 20, Errors: 0, Skipped: 0
> [INFO]
> [ERROR] There are test failures. {code}
> The polluter that led to this flakiness was the test method
> testGetUserGroupInformationSecure() in TestDFSClientCache.java. There was a
> line
> {code:java}
> UserGroupInformation.setLoginUser(currentUserUgi);{code}
> which modifies some shared state and resource, something like pre-setup the
> config. To fix this issue, I added the cleanup methods in
> TestDFSClientCache.java to reset the UserGroupInformation to ensure the
> isolation among each test class.
> {code:java}
> @AfterClass
> public static void cleanup() {
> UserGroupInformation.reset();
> }{code}
> Including setting
> {code:java}
> authenticationMethod = null;
> conf = null; // set configuration to null
> setLoginUser(null); // reset login user to default null{code}
> ..., and so on. The reset() methods can be referred to
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java.
> After the fix, the error was no longer exist and the succeed message was:
> {code:java}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 18.457 s - in org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> {code}
> Here is the CustomTest.java file that I used to run these two tests in order,
> the erro
[jira] [Commented] (HDFS-16083) Forbid Observer NameNode trigger active namenode log roll
[
https://issues.apache.org/jira/browse/HDFS-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795625#comment-17795625
]
ASF GitHub Bot commented on HDFS-16083:
---
hadoop-yetus commented on PR #6343:
URL: https://github.com/apache/hadoop/pull/6343#issuecomment-1851432168
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 39s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 130 unchanged
- 0 fixed = 139 total (was 130) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 211m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6343/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +0 :ok: | asflicense | 0m 27s | | ASF License check generated no
output? |
| | | 302m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
| |
hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.blockmanagement.TestErasureCodingCorruption |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.datanode.TestDataNodeReconfiguration |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling |
| | hadoop.hdfs.server.blockmanagement.TestStorageBlockPoolUsageStdDev |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop
[jira] [Commented] (HDFS-17267) Client send the smae packet multiple times when method markSlowNode throws IOException.
[
https://issues.apache.org/jira/browse/HDFS-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17795631#comment-17795631
]
ASF GitHub Bot commented on HDFS-17267:
---
hadoop-yetus commented on PR #6311:
URL: https://github.com/apache/hadoop/pull/6311#issuecomment-1851450693
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 47s | | trunk passed |
| +1 :green_heart: | compile | 2m 59s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 54s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 2m 51s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 51s | | the patch passed |
| +1 :green_heart: | compile | 2m 46s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 51s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 184m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 290m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6311/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6311 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b0035275bc85 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 67870e486b3faeb19c39c3274c86045a8af4fc63 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-
[jira] [Created] (HDFS-17285) [RBF] Decrease dfsrouter safe mode check period.
liuguanghua created HDFS-17285: -- Summary: [RBF] Decrease dfsrouter safe mode check period. Key: HDFS-17285 URL: https://issues.apache.org/jira/browse/HDFS-17285 Project: Hadoop HDFS Issue Type: Improvement Reporter: liuguanghua When dfsrouter start, it enters safe mode. And it will cost 1min to leave. The log is blow: 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leave startup safe mode after 3 ms 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Enter safe mode after 18 ms without reaching the State Store 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Entering safe mode 14:35:24,996 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Delaying safemode exit for 28721 milliseconds... 14:36:25,037 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leaving safe mode after 61319 milliseconds It depends on these configs. DFS_ROUTER_SAFEMODE_EXTENSION 30s DFS_ROUTER_SAFEMODE_EXPIRATION 3min DFS_ROUTER_CACHE_TIME_TO_LIVE_MS 1min (it is the period for check safe mode) Because in safe mode dfsrouter will reject write requests, so it should be shorter in check period if refreshCaches is done. And we should be separted DFS_ROUTER_CACHE_TIME_TO_LIVE_MS form RouterSafemodeService. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17285) [RBF] Decrease dfsrouter safe mode check period.
[ https://issues.apache.org/jira/browse/HDFS-17285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuguanghua updated HDFS-17285: --- Description: When dfsrouter start, it enters safe mode. And it will cost 1min to leave. The log is blow: 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leave startup safe mode after 3 ms 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Enter safe mode after 18 ms without reaching the State Store 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Entering safe mode 14:35:24,996 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Delaying safemode exit for 28721 milliseconds... 14:36:25,037 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leaving safe mode after 61319 milliseconds It depends on these configs. DFS_ROUTER_SAFEMODE_EXTENSION 30s DFS_ROUTER_SAFEMODE_EXPIRATION 3min DFS_ROUTER_CACHE_TIME_TO_LIVE_MS 1min (it is the period for check safe mode) Because in safe mode dfsrouter will reject write requests, so it should be shorter in check period if refreshCaches is done. And we should remove DFS_ROUTER_CACHE_TIME_TO_LIVE_MS form RouterSafemodeService. was: When dfsrouter start, it enters safe mode. And it will cost 1min to leave. The log is blow: 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leave startup safe mode after 3 ms 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Enter safe mode after 18 ms without reaching the State Store 14:35:23,717 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Entering safe mode 14:35:24,996 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Delaying safemode exit for 28721 milliseconds... 14:36:25,037 INFO org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leaving safe mode after 61319 milliseconds It depends on these configs. DFS_ROUTER_SAFEMODE_EXTENSION 30s DFS_ROUTER_SAFEMODE_EXPIRATION 3min DFS_ROUTER_CACHE_TIME_TO_LIVE_MS 1min (it is the period for check safe mode) Because in safe mode dfsrouter will reject write requests, so it should be shorter in check period if refreshCaches is done. And we should be separted DFS_ROUTER_CACHE_TIME_TO_LIVE_MS form RouterSafemodeService. > [RBF] Decrease dfsrouter safe mode check period. > > > Key: HDFS-17285 > URL: https://issues.apache.org/jira/browse/HDFS-17285 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuguanghua >Priority: Minor > > When dfsrouter start, it enters safe mode. And it will cost 1min to leave. > The log is blow: > 14:35:23,717 INFO > org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Leave > startup safe mode after 3 ms > 14:35:23,717 INFO > org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: Enter > safe mode after 18 ms without reaching the State Store > 14:35:23,717 INFO > org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: > Entering safe mode > 14:35:24,996 INFO > org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: > Delaying safemode exit for 28721 milliseconds... > 14:36:25,037 INFO > org.apache.hadoop.hdfs.server.federation.router.RouterSafemodeService: > Leaving safe mode after 61319 milliseconds > It depends on these configs. > DFS_ROUTER_SAFEMODE_EXTENSION 30s > DFS_ROUTER_SAFEMODE_EXPIRATION 3min > DFS_ROUTER_CACHE_TIME_TO_LIVE_MS 1min (it is the period for check safe mode) > Because in safe mode dfsrouter will reject write requests, so it should be > shorter in check period if refreshCaches is done. And we should remove > DFS_ROUTER_CACHE_TIME_TO_LIVE_MS form RouterSafemodeService. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
