[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755372#comment-17755372
]
ASF GitHub Bot commented on HDFS-17156:
---
hadoop-yetus commented on PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1681656464
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 26s | | trunk passed |
| +1 :green_heart: | compile | 18m 17s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 17m 19s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 17m 19s | | the patch passed |
| +1 :green_heart: | compile | 16m 35s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 9s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5951/3/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 1 new + 127
unchanged - 0 fixed = 128 total (was 127) |
| +1 :green_heart: | mvnsite | 1m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 54s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 236m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5951/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5951 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3da3be0432d9 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 49cae565e0c1235e0bf279c776b2c38ba6c60a42 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5951/3/testReport/ |
| Max. process+thread count | 1263 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755363#comment-17755363
]
ASF GitHub Bot commented on HDFS-17087:
---
haiyang1987 commented on PR #5845:
URL: https://github.com/apache/hadoop/pull/5845#issuecomment-1681618415
Thanks Sir @tomscut @Hexiaoqiao @ayushtkn help me review and merge it.
> Add Throttler for datanode reading block
> -
>
> Key: HDFS-17087
> URL: https://issues.apache.org/jira/browse/HDFS-17087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> DataXceiver#readBlock code is:
> {code:java}
> ...
> read = blockSender.sendBlock(out, baseStream, null); // send data
> ..
> {code}
> Current DataXceiver#readBlock doesn't throttler, maybe we can support
> throttle for reading block satisfied catering for specific scenarios.
> Default read throttler value is still null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17161) Adding test for StripedBlockReader#createBlockReader leaks socket on IOException
[ https://issues.apache.org/jira/browse/HDFS-17161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755359#comment-17755359 ] ASF GitHub Bot commented on HDFS-17161: --- tonyPan123 opened a new pull request, #5956: URL: https://github.com/apache/hadoop/pull/5956 …s socket on IOException ### Description of PR In [HDFS-13039](https://issues.apache.org/jira/browse/HDFS-13039), it is proposed a situation in which an IOException during StripedBlockReader#createBlockReader has potential to leak the socket. However, in [HDFS-13039](https://issues.apache.org/jira/browse/HDFS-13039), the test patch is still missed. Then in this case, we add a unit test to reproduce that. From the test results, when the patch in HDFS-13039 is removed, the test would failed while the test could pass when the patch is added back. Any comments and suggestions would be appreciated. ### How was this patch tested? The case itself is about a unit test ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Adding test for StripedBlockReader#createBlockReader leaks socket on > IOException > > > Key: HDFS-17161 > URL: https://issues.apache.org/jira/browse/HDFS-17161 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, erasure-coding >Affects Versions: 3.3.6 >Reporter: Haoze Wu >Priority: Major > > In [HDFS-13039|https://issues.apache.org/jira/browse/HDFS-13039], it is > proposed a situation in which an IOException during > StripedBlockReader#createBlockReader has potential to leak the socket. > However, in [HDFS-13039|https://issues.apache.org/jira/browse/HDFS-13039], > the test patch is still missed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17161) Adding test for StripedBlockReader#createBlockReader leaks socket on IOException
[ https://issues.apache.org/jira/browse/HDFS-17161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17161: -- Labels: pull-request-available (was: ) > Adding test for StripedBlockReader#createBlockReader leaks socket on > IOException > > > Key: HDFS-17161 > URL: https://issues.apache.org/jira/browse/HDFS-17161 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, erasure-coding >Affects Versions: 3.3.6 >Reporter: Haoze Wu >Priority: Major > Labels: pull-request-available > > In [HDFS-13039|https://issues.apache.org/jira/browse/HDFS-13039], it is > proposed a situation in which an IOException during > StripedBlockReader#createBlockReader has potential to leak the socket. > However, in [HDFS-13039|https://issues.apache.org/jira/browse/HDFS-13039], > the test patch is still missed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14037) Fix SSLFactory truststore reloader thread leak in URLConnectionFactory
[
https://issues.apache.org/jira/browse/HDFS-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie reassigned HDFS-14037:
Assignee: Takanobu Asanuma
> Fix SSLFactory truststore reloader thread leak in URLConnectionFactory
> --
>
> Key: HDFS-14037
> URL: https://issues.apache.org/jira/browse/HDFS-14037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, webhdfs
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14037.1.patch, HDFS-14037.2.patch
>
>
> This is reported by [~yoshiata]. It is a similar issue as HADOOP-11368 and
> YARN-5309 in URLConnectionFactory.
> {quote}SSLFactory in newSslConnConfigurator and subsequently creates the
> ReloadingX509TrustManager instance which in turn starts a trust store
> reloader thread.
> However, the SSLFactory is never destroyed and hence the trust store reloader
> threads are not killed.
> {quote}
> We observed many leaked threads when we used swebhdfs via NiFi cluster.
> {noformat}
> "Truststore reloader thread" Id=221 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:189)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14037) Fix SSLFactory truststore reloader thread leak in URLConnectionFactory
[
https://issues.apache.org/jira/browse/HDFS-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie reassigned HDFS-14037:
Assignee: (was: lujie)
> Fix SSLFactory truststore reloader thread leak in URLConnectionFactory
> --
>
> Key: HDFS-14037
> URL: https://issues.apache.org/jira/browse/HDFS-14037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, webhdfs
>Reporter: Takanobu Asanuma
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14037.1.patch, HDFS-14037.2.patch
>
>
> This is reported by [~yoshiata]. It is a similar issue as HADOOP-11368 and
> YARN-5309 in URLConnectionFactory.
> {quote}SSLFactory in newSslConnConfigurator and subsequently creates the
> ReloadingX509TrustManager instance which in turn starts a trust store
> reloader thread.
> However, the SSLFactory is never destroyed and hence the trust store reloader
> threads are not killed.
> {quote}
> We observed many leaked threads when we used swebhdfs via NiFi cluster.
> {noformat}
> "Truststore reloader thread" Id=221 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:189)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14037) Fix SSLFactory truststore reloader thread leak in URLConnectionFactory
[
https://issues.apache.org/jira/browse/HDFS-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie reassigned HDFS-14037:
Assignee: lujie (was: Takanobu Asanuma)
> Fix SSLFactory truststore reloader thread leak in URLConnectionFactory
> --
>
> Key: HDFS-14037
> URL: https://issues.apache.org/jira/browse/HDFS-14037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, webhdfs
>Reporter: Takanobu Asanuma
>Assignee: lujie
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14037.1.patch, HDFS-14037.2.patch
>
>
> This is reported by [~yoshiata]. It is a similar issue as HADOOP-11368 and
> YARN-5309 in URLConnectionFactory.
> {quote}SSLFactory in newSslConnConfigurator and subsequently creates the
> ReloadingX509TrustManager instance which in turn starts a trust store
> reloader thread.
> However, the SSLFactory is never destroyed and hence the trust store reloader
> threads are not killed.
> {quote}
> We observed many leaked threads when we used swebhdfs via NiFi cluster.
> {noformat}
> "Truststore reloader thread" Id=221 TIMED_WAITING on null
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:189)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755318#comment-17755318
]
ASF GitHub Bot commented on HDFS-17156:
---
simbadzina commented on PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1681496761
> @tasanuma Thank your for the review and approval! @simbadzina Thanks for
your comments. I cherry-picked your unit test and it worked well in my local
environment. Thanks!
>
> Below is the test result in my local environment: with the fix:
>
> ```
> mvn test -Dtest=TestIPC.java
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.ipc.TestIPC
> [WARNING] Tests run: 46, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
112.083 s - in org.apache.hadoop.ipc.TestIPC
> [INFO]
> [INFO] Results:
> [INFO]
> [WARNING] Tests run: 46, Failures: 0, Errors: 0, Skipped: 1
> ```
>
> without the fix: failed as expected
>
> ```
> mvn test -Dtest=TestIPC.java
> [INFO] Running org.apache.hadoop.ipc.TestIPC
> [ERROR] Tests run: 46, Failures: 1, Errors: 0, Skipped: 1, Time elapsed:
113.739 s <<< FAILURE! - in org.apache.hadoop.ipc.TestIPC
> [ERROR]
testReceiveStateBeforeCallerNotification(org.apache.hadoop.ipc.TestIPC) Time
elapsed: 0.181 s <<< FAILURE!
> java.lang.AssertionError
>at org.junit.Assert.fail(Assert.java:87)
>at org.junit.Assert.assertTrue(Assert.java:42)
>at org.junit.Assert.assertTrue(Assert.java:53)
>at
org.apache.hadoop.ipc.TestIPC.testReceiveStateBeforeCallerNotification(TestIPC.java:1365)
>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
>at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
>at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>at java.base/java.lang.Thread.run(Thread.java:829)
> ```
For sure. Thanks for finding and fixing this bug.
> mapreduce job encounters java.io.IOException
>
>
> Key: HDFS-17156
> URL: https://issues.apache.org/jira/browse/HDFS-17156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Chunyi Yang
>Assignee: Chunyi Yang
>Priority: Minor
> Labels: Observer, RBF, pull-request-available
>
> While executing a mapreduce job in an environment utilizing Router-Based
> Federation with Observer read enabled, there is an estimated 1% chance of
> encountering the following error.
> {code:java}
> "java.io.IOException: Resource
> hdfs:///user//.staging/job_XX/.tez/application_XX/tez-conf.pb
> changed on src filesystem - expected: \"2023-07-07T12:41:16.801+0900\", was:
> \"2023-07-07T12:41:16.822+0900\", current time:
> \"2023-07-07T12:41:22.386+0900\"",
> {code}
> This error happens in function verifyAndCopy inside FSDownload.java when
> nodemanager tries to download a file right after the file has been written to
> the HDFS. The write operation runs on active namenode and read operation runs
> on observer namenode as expected.
> The edits file and hdfs-audit files indicate that the expected timestamp
> mentioned in the error message aligns with the OP_CLOSE MTIME of the
> 'tez-conf.pb' file (which is correct). However, the actual timestamp
> retrieved from the read operation corresponds to the OP_ADD MTIME of the
> target 'tez-conf.pf' file (which is incorrect). This inconsistency suggests
> that the observer namenode responds to the client before its edits file is
> updated with the latest stateId.
> Further troubleshooting has revealed that during write operations, the router
> responds to the client before receiving the latest stateId from the active
> namenode. Consequently, the outdated stateId is then used in the subsequent
> read operation on the observer namenode, leading
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755317#comment-17755317
]
ASF GitHub Bot commented on HDFS-17156:
---
chunyiyang commented on code in PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#discussion_r1296594414
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java:
##
@@ -1330,6 +1337,37 @@ public void run() {
server.stop();
}
}
+
+ /**
+ * Verify that stateID is received into call before
+ * call is notified.
Review Comment:
Fixed. Thanks!
> mapreduce job encounters java.io.IOException
>
>
> Key: HDFS-17156
> URL: https://issues.apache.org/jira/browse/HDFS-17156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Chunyi Yang
>Assignee: Chunyi Yang
>Priority: Minor
> Labels: Observer, RBF, pull-request-available
>
> While executing a mapreduce job in an environment utilizing Router-Based
> Federation with Observer read enabled, there is an estimated 1% chance of
> encountering the following error.
> {code:java}
> "java.io.IOException: Resource
> hdfs:///user//.staging/job_XX/.tez/application_XX/tez-conf.pb
> changed on src filesystem - expected: \"2023-07-07T12:41:16.801+0900\", was:
> \"2023-07-07T12:41:16.822+0900\", current time:
> \"2023-07-07T12:41:22.386+0900\"",
> {code}
> This error happens in function verifyAndCopy inside FSDownload.java when
> nodemanager tries to download a file right after the file has been written to
> the HDFS. The write operation runs on active namenode and read operation runs
> on observer namenode as expected.
> The edits file and hdfs-audit files indicate that the expected timestamp
> mentioned in the error message aligns with the OP_CLOSE MTIME of the
> 'tez-conf.pb' file (which is correct). However, the actual timestamp
> retrieved from the read operation corresponds to the OP_ADD MTIME of the
> target 'tez-conf.pf' file (which is incorrect). This inconsistency suggests
> that the observer namenode responds to the client before its edits file is
> updated with the latest stateId.
> Further troubleshooting has revealed that during write operations, the router
> responds to the client before receiving the latest stateId from the active
> namenode. Consequently, the outdated stateId is then used in the subsequent
> read operation on the observer namenode, leading to inaccuracies in the
> information provided by the observer namenode.
> To resolve this issue, it is essential to ensure that the router sends a
> response to the client only after receiving the latest stateId from the
> active namenode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755316#comment-17755316
]
ASF GitHub Bot commented on HDFS-17156:
---
chunyiyang commented on PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1681491223
@tasanuma Thank your for the review and approval!
@simbadzina Thanks for your comments. I cherry-picked your unit test and it
worked well in my local environment. Thanks!
Below is the test result in my local environment:
with the fix:
```
mvn test -Dtest=TestIPC.java
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.ipc.TestIPC
[WARNING] Tests run: 46, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
112.083 s - in org.apache.hadoop.ipc.TestIPC
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 46, Failures: 0, Errors: 0, Skipped: 1
```
without the fix: failed as expected
```
mvn test -Dtest=TestIPC.java
[INFO] Running org.apache.hadoop.ipc.TestIPC
[ERROR] Tests run: 46, Failures: 1, Errors: 0, Skipped: 1, Time elapsed:
113.739 s <<< FAILURE! - in org.apache.hadoop.ipc.TestIPC
[ERROR]
testReceiveStateBeforeCallerNotification(org.apache.hadoop.ipc.TestIPC) Time
elapsed: 0.181 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.ipc.TestIPC.testReceiveStateBeforeCallerNotification(TestIPC.java:1365)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:829)
```
> mapreduce job encounters java.io.IOException
>
>
> Key: HDFS-17156
> URL: https://issues.apache.org/jira/browse/HDFS-17156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Chunyi Yang
>Assignee: Chunyi Yang
>Priority: Minor
> Labels: Observer, RBF, pull-request-available
>
> While executing a mapreduce job in an environment utilizing Router-Based
> Federation with Observer read enabled, there is an estimated 1% chance of
> encountering the following error.
> {code:java}
> "java.io.IOException: Resource
> hdfs:///user//.staging/job_XX/.tez/application_XX/tez-conf.pb
> changed on src filesystem - expected: \"2023-07-07T12:41:16.801+0900\", was:
> \"2023-07-07T12:41:16.822+0900\", current time:
> \"2023-07-07T12:41:22.386+0900\"",
> {code}
> This error happens in function verifyAndCopy inside FSDownload.java when
> nodemanager tries to download a file right after the file has been written to
> the HDFS. The write operation runs on active namenode and read operation runs
> on observer namenode as expected.
> The edits file and hdfs-audit files indicate that the expected timestamp
> mentioned in the error message aligns with the OP_CLOSE MTIME of the
> 'tez-conf.pb' file (which is correct). However, the actual timestamp
> retrieved from the read operation corresponds to the OP_ADD MTIME of the
> target 'tez-conf.pf' file (which is incorrect). This inconsistency suggests
> that the observer namenode responds to the client before its edits file is
> updated with the latest stateId.
> Further troubleshooting has revealed that during write operations, the router
> responds to the client before receiving the latest stateId from the active
> namenode. Consequently, the outdated stateId is then used in the subsequent
> read operation on the observer namenode, leading to inaccuracies in the
> information provided by the observer namenode.
> To resolve this issue, it is
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755315#comment-17755315
]
ASF GitHub Bot commented on HDFS-17156:
---
simbadzina commented on code in PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#discussion_r1296591248
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java:
##
@@ -1330,6 +1337,37 @@ public void run() {
server.stop();
}
}
+
+ /**
+ * Verify that stateID is received into call before
+ * call is notified.
Review Comment:
Nit: Typo "caller" is notified.
> mapreduce job encounters java.io.IOException
>
>
> Key: HDFS-17156
> URL: https://issues.apache.org/jira/browse/HDFS-17156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Chunyi Yang
>Assignee: Chunyi Yang
>Priority: Minor
> Labels: Observer, RBF, pull-request-available
>
> While executing a mapreduce job in an environment utilizing Router-Based
> Federation with Observer read enabled, there is an estimated 1% chance of
> encountering the following error.
> {code:java}
> "java.io.IOException: Resource
> hdfs:///user//.staging/job_XX/.tez/application_XX/tez-conf.pb
> changed on src filesystem - expected: \"2023-07-07T12:41:16.801+0900\", was:
> \"2023-07-07T12:41:16.822+0900\", current time:
> \"2023-07-07T12:41:22.386+0900\"",
> {code}
> This error happens in function verifyAndCopy inside FSDownload.java when
> nodemanager tries to download a file right after the file has been written to
> the HDFS. The write operation runs on active namenode and read operation runs
> on observer namenode as expected.
> The edits file and hdfs-audit files indicate that the expected timestamp
> mentioned in the error message aligns with the OP_CLOSE MTIME of the
> 'tez-conf.pb' file (which is correct). However, the actual timestamp
> retrieved from the read operation corresponds to the OP_ADD MTIME of the
> target 'tez-conf.pf' file (which is incorrect). This inconsistency suggests
> that the observer namenode responds to the client before its edits file is
> updated with the latest stateId.
> Further troubleshooting has revealed that during write operations, the router
> responds to the client before receiving the latest stateId from the active
> namenode. Consequently, the outdated stateId is then used in the subsequent
> read operation on the observer namenode, leading to inaccuracies in the
> information provided by the observer namenode.
> To resolve this issue, it is essential to ensure that the router sends a
> response to the client only after receiving the latest stateId from the
> active namenode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li resolved HDFS-17087.
---
Fix Version/s: 3.4.0
Resolution: Fixed
> Add Throttler for datanode reading block
> -
>
> Key: HDFS-17087
> URL: https://issues.apache.org/jira/browse/HDFS-17087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> DataXceiver#readBlock code is:
> {code:java}
> ...
> read = blockSender.sendBlock(out, baseStream, null); // send data
> ..
> {code}
> Current DataXceiver#readBlock doesn't throttler, maybe we can support
> throttle for reading block satisfied catering for specific scenarios.
> Default read throttler value is still null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755310#comment-17755310
]
ASF GitHub Bot commented on HDFS-17087:
---
tomscut commented on PR #5845:
URL: https://github.com/apache/hadoop/pull/5845#issuecomment-1681478001
Thanks @haiyang1987 for your contribution! Thanks @ayushtkn and @Hexiaoqiao
for the review!
> Add Throttler for datanode reading block
> -
>
> Key: HDFS-17087
> URL: https://issues.apache.org/jira/browse/HDFS-17087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> DataXceiver#readBlock code is:
> {code:java}
> ...
> read = blockSender.sendBlock(out, baseStream, null); // send data
> ..
> {code}
> Current DataXceiver#readBlock doesn't throttler, maybe we can support
> throttle for reading block satisfied catering for specific scenarios.
> Default read throttler value is still null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755309#comment-17755309
]
ASF GitHub Bot commented on HDFS-17087:
---
tomscut merged PR #5845:
URL: https://github.com/apache/hadoop/pull/5845
> Add Throttler for datanode reading block
> -
>
> Key: HDFS-17087
> URL: https://issues.apache.org/jira/browse/HDFS-17087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> DataXceiver#readBlock code is:
> {code:java}
> ...
> read = blockSender.sendBlock(out, baseStream, null); // send data
> ..
> {code}
> Current DataXceiver#readBlock doesn't throttler, maybe we can support
> throttle for reading block satisfied catering for specific scenarios.
> Default read throttler value is still null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755308#comment-17755308
]
ASF GitHub Bot commented on HDFS-17087:
---
tomscut commented on PR #5845:
URL: https://github.com/apache/hadoop/pull/5845#issuecomment-1681476750
LGTM.
> Add Throttler for datanode reading block
> -
>
> Key: HDFS-17087
> URL: https://issues.apache.org/jira/browse/HDFS-17087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> DataXceiver#readBlock code is:
> {code:java}
> ...
> read = blockSender.sendBlock(out, baseStream, null); // send data
> ..
> {code}
> Current DataXceiver#readBlock doesn't throttler, maybe we can support
> throttle for reading block satisfied catering for specific scenarios.
> Default read throttler value is still null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17157) Transient network failure in lease recovery could be mitigated to ensure better consistency
[
https://issues.apache.org/jira/browse/HDFS-17157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755305#comment-17755305
]
ASF GitHub Bot commented on HDFS-17157:
---
hadoop-yetus commented on PR #5954:
URL: https://github.com/apache/hadoop/pull/5954#issuecomment-1681453746
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 199 unchanged
- 0 fixed = 201 total (was 199) |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 188m 15s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 286m 22s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5954 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 43ce060b5c99 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5e36c6ef1fb896a617233c4ccd4084c9551bfd0 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755265#comment-17755265
]
ASF GitHub Bot commented on HDFS-17156:
---
simbadzina commented on PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1681305358
I've created a unit test here:
https://github.com/simbadzina/hadoop/pull/1/commits/d216bfe8c5f506f08263cf490479a0398b097f05
```
/**
* Verify that stateID is received into call before
* call is notified.
* @throws IOException
*/
@Test(timeout=6)
public void testReceiveStateBeforeCallerNotification() throws IOException {
AtomicBoolean stateReceived = new AtomicBoolean(false);
AlignmentContext alignmentContext = Mockito.mock(AlignmentContext.class);
Mockito.doAnswer((Answer) invocation -> {
Thread.sleep(1000);
stateReceived.set(true);
return null;
}).when(alignmentContext)
.receiveResponseState(any(RpcHeaderProtos.RpcResponseHeaderProto.class));
final Client client = Mockito.spy(new Client(LongWritable.class, conf));
final TestServer server = new TestServer(1, false);
try {
InetSocketAddress addr = NetUtils.getConnectAddress(server);
server.start();
call(client, new LongWritable(RANDOM.nextLong()), addr,
0, conf, alignmentContext);
Assert.assertTrue(stateReceived.get());
} finally {
client.stop();
server.stop();
}
}
```
> mapreduce job encounters java.io.IOException
>
>
> Key: HDFS-17156
> URL: https://issues.apache.org/jira/browse/HDFS-17156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Chunyi Yang
>Assignee: Chunyi Yang
>Priority: Minor
> Labels: Observer, RBF, pull-request-available
>
> While executing a mapreduce job in an environment utilizing Router-Based
> Federation with Observer read enabled, there is an estimated 1% chance of
> encountering the following error.
> {code:java}
> "java.io.IOException: Resource
> hdfs:///user//.staging/job_XX/.tez/application_XX/tez-conf.pb
> changed on src filesystem - expected: \"2023-07-07T12:41:16.801+0900\", was:
> \"2023-07-07T12:41:16.822+0900\", current time:
> \"2023-07-07T12:41:22.386+0900\"",
> {code}
> This error happens in function verifyAndCopy inside FSDownload.java when
> nodemanager tries to download a file right after the file has been written to
> the HDFS. The write operation runs on active namenode and read operation runs
> on observer namenode as expected.
> The edits file and hdfs-audit files indicate that the expected timestamp
> mentioned in the error message aligns with the OP_CLOSE MTIME of the
> 'tez-conf.pb' file (which is correct). However, the actual timestamp
> retrieved from the read operation corresponds to the OP_ADD MTIME of the
> target 'tez-conf.pf' file (which is incorrect). This inconsistency suggests
> that the observer namenode responds to the client before its edits file is
> updated with the latest stateId.
> Further troubleshooting has revealed that during write operations, the router
> responds to the client before receiving the latest stateId from the active
> namenode. Consequently, the outdated stateId is then used in the subsequent
> read operation on the observer namenode, leading to inaccuracies in the
> information provided by the observer namenode.
> To resolve this issue, it is essential to ensure that the router sends a
> response to the client only after receiving the latest stateId from the
> active namenode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17161) Adding test for StripedBlockReader#createBlockReader leaks socket on IOException
Haoze Wu created HDFS-17161: --- Summary: Adding test for StripedBlockReader#createBlockReader leaks socket on IOException Key: HDFS-17161 URL: https://issues.apache.org/jira/browse/HDFS-17161 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, erasure-coding Affects Versions: 3.3.6 Reporter: Haoze Wu In [HDFS-13039|https://issues.apache.org/jira/browse/HDFS-13039], it is proposed a situation in which an IOException during StripedBlockReader#createBlockReader has potential to leak the socket. However, in [HDFS-13039|https://issues.apache.org/jira/browse/HDFS-13039], the test patch is still missed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17157) Transient network failure in lease recovery could be mitigated to ensure better consistency
[
https://issues.apache.org/jira/browse/HDFS-17157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17157:
--
Labels: pull-request-available (was: )
> Transient network failure in lease recovery could be mitigated to ensure
> better consistency
> ---
>
> Key: HDFS-17157
> URL: https://issues.apache.org/jira/browse/HDFS-17157
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha, 3.3.6
>Reporter: Haoze Wu
>Priority: Major
> Labels: pull-request-available
>
> This case is related to HDFS-12070.
> In HDFS-12070, we saw how a faulty drive at a certain datanode could lead to
> permanent block recovery failure and leaves the file open indefinitely. In
> the patch, instead of failing the whole lease recovery process when the
> second stage of block recovery is failed at one datanode, the whole lease
> recovery process is failed if only these are failed for all the datanodes.
> Attached is the code snippet for the second stage of the block recovery, in
> BlockRecoveryWorker#syncBlock:
> {code:java}
> ...
> final List successList = new ArrayList<>();
> for (BlockRecord r : participatingList) {
> try {
> r.updateReplicaUnderRecovery(bpid, recoveryId, blockId,
> newBlock.getNumBytes());
> successList.add(r);
> } catch (IOException e) {
> ...{code}
> However, because of transient network failure, the RPC in
> updateReplicaUnderRecovery initiated from the primary datanode to another
> datanode could return an EOFException while the other side does not process
> the RPC at all or throw an IOException when reading from the socket.
> {code:java}
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:788)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy29.updateReplicaUnderRecovery(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolTranslatorPB.updateReplicaUnderRecovery(InterDatanodeProtocolTranslatorPB.java:112)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.updateReplicaUnderRecovery(BlockRecoveryWorker.java:88)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.access$700(BlockRecoveryWorker.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.syncBlock(BlockRecoveryWorker.java:300)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:188)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:606)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1796)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1165)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1061)
> {code}
> Then if there is any other datanode in which the second stage of block
> recovery success, the lease recovery would be successful and close the file.
> However, the last block failed to be synced to that failed datanode and this
> inconsistency could potentially last for a very long time.
> To fix the issue, I propose adding a configurable retry of
> updateReplicaUnderRecovery RPC so that transient network failure could be
> mitigated.
> Any comments and suggestions would be appreciated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17157) Transient network failure in lease recovery could be mitigated to ensure better consistency
[
https://issues.apache.org/jira/browse/HDFS-17157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755244#comment-17755244
]
ASF GitHub Bot commented on HDFS-17157:
---
tonyPan123 opened a new pull request, #5954:
URL: https://github.com/apache/hadoop/pull/5954
### Description of PR
See [HDFS-17157](https://issues.apache.org/jira/browse/HDFS-17157) for
details on the symptom and diagnostic.
The fix of [HDFS-17157](https://issues.apache.org/jira/browse/HDFS-17157) is
implemented. In my local machine, it is able to pass all test cases.
I add some configurable retry mechanism to ZkDataBase#loadDataBase() to
tolerate EOFException caused by possible transient network failure.
Any comments and suggestions would be appreciated.
### How was this patch tested?
### For code changes:
- [Yes] Does the title or this PR starts with the corresponding JIRA issue
id (e.g. 'HADOOP-17799. Your PR title ...')?
- [Yes ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [Yes] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Transient network failure in lease recovery could be mitigated to ensure
> better consistency
> ---
>
> Key: HDFS-17157
> URL: https://issues.apache.org/jira/browse/HDFS-17157
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha, 3.3.6
>Reporter: Haoze Wu
>Priority: Major
>
> This case is related to HDFS-12070.
> In HDFS-12070, we saw how a faulty drive at a certain datanode could lead to
> permanent block recovery failure and leaves the file open indefinitely. In
> the patch, instead of failing the whole lease recovery process when the
> second stage of block recovery is failed at one datanode, the whole lease
> recovery process is failed if only these are failed for all the datanodes.
> Attached is the code snippet for the second stage of the block recovery, in
> BlockRecoveryWorker#syncBlock:
> {code:java}
> ...
> final List successList = new ArrayList<>();
> for (BlockRecord r : participatingList) {
> try {
> r.updateReplicaUnderRecovery(bpid, recoveryId, blockId,
> newBlock.getNumBytes());
> successList.add(r);
> } catch (IOException e) {
> ...{code}
> However, because of transient network failure, the RPC in
> updateReplicaUnderRecovery initiated from the primary datanode to another
> datanode could return an EOFException while the other side does not process
> the RPC at all or throw an IOException when reading from the socket.
> {code:java}
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:788)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy29.updateReplicaUnderRecovery(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolTranslatorPB.updateReplicaUnderRecovery(InterDatanodeProtocolTranslatorPB.java:112)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.updateReplicaUnderRecovery(BlockRecoveryWorker.java:88)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.access$700(BlockRecoveryWorker.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.syncBlock(BlockRecoveryWorker.java:300)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:188)
> at
>
[jira] [Commented] (HDFS-17160) Metrics fsyncCount should be computed correctly in flushOrSync
[
https://issues.apache.org/jira/browse/HDFS-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755230#comment-17755230
]
ASF GitHub Bot commented on HDFS-17160:
---
hadoop-yetus commented on PR #5952:
URL: https://github.com/apache/hadoop/pull/5952#issuecomment-1681147728
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 33s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 189m 24s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5952/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 283m 52s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5952/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5952 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1b6b7c3a2bd5 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f06cc7cfdf35ec94faf5702205b4bde035e1fc42 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5952/1/testReport/ |
| Max. process+thread count | 3721 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-17128) RBF: SQLDelegationTokenSecretManager should use version of tokens updated by other routers
[ https://issues.apache.org/jira/browse/HDFS-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755198#comment-17755198 ] Hector Sandoval Chaverri commented on HDFS-17128: - [~slfan1989] gentle ping on the request above :) The patch I provided should apply cleanly on branch-3.3. Thanks for all your help! > RBF: SQLDelegationTokenSecretManager should use version of tokens updated by > other routers > -- > > Key: HDFS-17128 > URL: https://issues.apache.org/jira/browse/HDFS-17128 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HDFS-17128-branch-3.3.patch > > > The SQLDelegationTokenSecretManager keeps tokens that it has interacted with > in a memory cache. This prevents routers from connecting to the SQL server > for each token operation, improving performance. > We've noticed issues with some tokens being loaded in one router's cache and > later renewed on a different one. If clients try to use the token in the > outdated router, it will throw an "Auth failed" error when the cached token's > expiration has passed. > This can also affect cancelation scenarios since a token can be removed from > one router's cache and still exist in another one. > A possible solution is already implemented on the > ZKDelegationTokenSecretManager, which consists of having an executor > refreshing each router's cache on a periodic basis. We should evaluate > whether this will work with the volume of tokens expected to be handled by > the SQLDelegationTokenSecretManager. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755094#comment-17755094
]
ASF GitHub Bot commented on HDFS-17087:
---
hadoop-yetus commented on PR #5845:
URL: https://github.com/apache/hadoop/pull/5845#issuecomment-1680746435
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 233m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 215m 24s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 559m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5845/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5845 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 6d18383bdabe 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / be6c037261133bde1ce562d687192f155156903a |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5845/6/testReport/ |
| Max. process+thread count | 3455 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5845/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-17160) Metrics fsyncCount should be computed correctly in flushOrSync
[ https://issues.apache.org/jira/browse/HDFS-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755086#comment-17755086 ] ASF GitHub Bot commented on HDFS-17160: --- hfutatzhanghb opened a new pull request, #5952: URL: https://github.com/apache/hadoop/pull/5952 ### Description of PR Currently in method flushOrSync, it will only increase fsyncCount once when `isSync == true`. But actually we fsync twice when `checksumOut != null` and `streams.getDataOut() != null`. So we should increase fsync count twice. > Metrics fsyncCount should be computed correctly in flushOrSync > -- > > Key: HDFS-17160 > URL: https://issues.apache.org/jira/browse/HDFS-17160 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17160) Metrics fsyncCount should be computed correctly in flushOrSync
[ https://issues.apache.org/jira/browse/HDFS-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17160: -- Labels: pull-request-available (was: ) > Metrics fsyncCount should be computed correctly in flushOrSync > -- > > Key: HDFS-17160 > URL: https://issues.apache.org/jira/browse/HDFS-17160 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-17160) Metrics fsyncCount should be computed correctly in flushOrSync
[ https://issues.apache.org/jira/browse/HDFS-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] farmmamba reassigned HDFS-17160: Assignee: farmmamba > Metrics fsyncCount should be computed correctly in flushOrSync > -- > > Key: HDFS-17160 > URL: https://issues.apache.org/jira/browse/HDFS-17160 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17160) Metrics fsyncCount should be computed correctly in flushOrSync
farmmamba created HDFS-17160: Summary: Metrics fsyncCount should be computed correctly in flushOrSync Key: HDFS-17160 URL: https://issues.apache.org/jira/browse/HDFS-17160 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.4.0 Reporter: farmmamba -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17154) EC: Fix bug in updateBlockForPipeline after failover
[ https://issues.apache.org/jira/browse/HDFS-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755032#comment-17755032 ] ASF GitHub Bot commented on HDFS-17154: --- Hexiaoqiao commented on PR #5941: URL: https://github.com/apache/hadoop/pull/5941#issuecomment-1680515885 Committed to trunk. Thanks @zhangshuyan0 for your works and @haiyang1987 for your reviews! Note: The failed unit test is not related to this changes. > EC: Fix bug in updateBlockForPipeline after failover > > > Key: HDFS-17154 > URL: https://issues.apache.org/jira/browse/HDFS-17154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > In the method `updateBlockForPipeline`, NameNode uses the > `BlockUnderConstructionFeature` of a BlockInfo to generate the member > `blockIndices` of `LocatedStripedBlock`. > And then, NameNode uses `blockIndices` to generate block tokens for client. > However, if there is a failover, the location info in > BlockUnderConstructionFeature may be incomplete, which results in the absence > of the corresponding block tokens. > When the client receives these incomplete block tokens, it will throw a NPE > because `updatedBlks[i]` is null. > NameNode should just return block tokens for all indices to the client. > Client can pick whichever it likes to use. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17154) EC: Fix bug in updateBlockForPipeline after failover
[ https://issues.apache.org/jira/browse/HDFS-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-17154. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > EC: Fix bug in updateBlockForPipeline after failover > > > Key: HDFS-17154 > URL: https://issues.apache.org/jira/browse/HDFS-17154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > In the method `updateBlockForPipeline`, NameNode uses the > `BlockUnderConstructionFeature` of a BlockInfo to generate the member > `blockIndices` of `LocatedStripedBlock`. > And then, NameNode uses `blockIndices` to generate block tokens for client. > However, if there is a failover, the location info in > BlockUnderConstructionFeature may be incomplete, which results in the absence > of the corresponding block tokens. > When the client receives these incomplete block tokens, it will throw a NPE > because `updatedBlks[i]` is null. > NameNode should just return block tokens for all indices to the client. > Client can pick whichever it likes to use. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17154) EC: Fix bug in updateBlockForPipeline after failover
[ https://issues.apache.org/jira/browse/HDFS-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755031#comment-17755031 ] ASF GitHub Bot commented on HDFS-17154: --- Hexiaoqiao merged PR #5941: URL: https://github.com/apache/hadoop/pull/5941 > EC: Fix bug in updateBlockForPipeline after failover > > > Key: HDFS-17154 > URL: https://issues.apache.org/jira/browse/HDFS-17154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > In the method `updateBlockForPipeline`, NameNode uses the > `BlockUnderConstructionFeature` of a BlockInfo to generate the member > `blockIndices` of `LocatedStripedBlock`. > And then, NameNode uses `blockIndices` to generate block tokens for client. > However, if there is a failover, the location info in > BlockUnderConstructionFeature may be incomplete, which results in the absence > of the corresponding block tokens. > When the client receives these incomplete block tokens, it will throw a NPE > because `updatedBlks[i]` is null. > NameNode should just return block tokens for all indices to the client. > Client can pick whichever it likes to use. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17154) EC: Fix bug in updateBlockForPipeline after failover
[ https://issues.apache.org/jira/browse/HDFS-17154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755015#comment-17755015 ] ASF GitHub Bot commented on HDFS-17154: --- haiyang1987 commented on PR #5941: URL: https://github.com/apache/hadoop/pull/5941#issuecomment-1680411872 LGTM. +1 > EC: Fix bug in updateBlockForPipeline after failover > > > Key: HDFS-17154 > URL: https://issues.apache.org/jira/browse/HDFS-17154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > In the method `updateBlockForPipeline`, NameNode uses the > `BlockUnderConstructionFeature` of a BlockInfo to generate the member > `blockIndices` of `LocatedStripedBlock`. > And then, NameNode uses `blockIndices` to generate block tokens for client. > However, if there is a failover, the location info in > BlockUnderConstructionFeature may be incomplete, which results in the absence > of the corresponding block tokens. > When the client receives these incomplete block tokens, it will throw a NPE > because `updatedBlks[i]` is null. > NameNode should just return block tokens for all indices to the client. > Client can pick whichever it likes to use. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[
https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754977#comment-17754977
]
ASF GitHub Bot commented on HDFS-17138:
---
hadoop-yetus commented on PR #5921:
URL: https://github.com/apache/hadoop/pull/5921#issuecomment-1680352528
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 43s | | trunk passed |
| +1 :green_heart: | compile | 18m 58s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 18m 17s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 6m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 12s | | the patch passed |
| +1 :green_heart: | compile | 18m 11s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 18m 11s | | the patch passed |
| +1 :green_heart: | compile | 17m 57s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 17m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 36s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 6m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 1s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 237m 14s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5921/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 524m 56s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5921/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5921 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b5a23c4b9d0f 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 05db88a35ba7ad994d4e9f9872f0808ed025c8c1 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
[jira] [Commented] (HDFS-17158) Show the rate of metrics in EC recovery task.
[
https://issues.apache.org/jira/browse/HDFS-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754962#comment-17754962
]
ASF GitHub Bot commented on HDFS-17158:
---
hadoop-yetus commented on PR #5947:
URL: https://github.com/apache/hadoop/pull/5947#issuecomment-1680296670
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 238m 29s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 393m 4s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5947 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6720bf461b1f 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3fac74d6f63ecd8c9e2d0abb46afa305c8989e7c |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5947/2/testReport/ |
| Max. process+thread count | 2247 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
|
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754938#comment-17754938
]
ASF GitHub Bot commented on HDFS-17156:
---
hadoop-yetus commented on PR #5951:
URL: https://github.com/apache/hadoop/pull/5951#issuecomment-1680234074
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 34s | | trunk passed |
| +1 :green_heart: | compile | 17m 23s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 16s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 16m 21s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 16m 21s | | the patch passed |
| +1 :green_heart: | compile | 16m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 18s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 232m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5951/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5951 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 84be26a3ecef 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c95258cbef20f04bb20ded485ff4c38dbcaf1d47 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5951/1/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5951/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
