[jira] [Commented] (HADOOP-19056) Highlight RBF features and improvements targeting version 3.4
[
https://issues.apache.org/jira/browse/HADOOP-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811762#comment-17811762
]
ASF GitHub Bot commented on HADOOP-19056:
-
hadoop-yetus commented on PR #6512:
URL: https://github.com/apache/hadoop/pull/6512#issuecomment-1914140028
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 76m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 114m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6512/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6512 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets |
| uname | Linux 7409fb68ea70 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62806a983ede08f694ca7830bdf7b3cb8abce17b |
| Max. process+thread count | 718 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6512/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Highlight RBF features and improvements targeting version 3.4
> -
>
> Key: HADOOP-19056
> URL: https://issues.apache.org/jira/browse/HADOOP-19056
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Labels: pull-request-available
>
> I want to hight recent RBF features and improvements.
> - Support observer node from Router-Based Federation
> - Enhanced IPC throughput between Router and NameNode
> - Improved isolation for downstream name nodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. [hadoop]
hadoop-yetus commented on PR #6512:
URL: https://github.com/apache/hadoop/pull/6512#issuecomment-1914140028
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 76m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 114m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6512/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6512 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets |
| uname | Linux 7409fb68ea70 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62806a983ede08f694ca7830bdf7b3cb8abce17b |
| Max. process+thread count | 718 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6512/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] MAPREDUCE-7469 NNBench createControlFiles should use thread pool to i… [hadoop]
hadoop-yetus commented on PR #6463:
URL: https://github.com/apache/hadoop/pull/6463#issuecomment-1914118917
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 28s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 131m 46s | |
hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 253m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6463/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6463 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 07ea3b5a8216 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 689fa7d93e9629b927215bb90f963caf82032068 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6463/5/testReport/ |
| Max. process+thread count | 1207 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6463/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[ https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811755#comment-17811755 ] ASF GitHub Bot commented on HADOOP-18883: - saxenapranav commented on PR #6511: URL: https://github.com/apache/hadoop/pull/6511#issuecomment-1914080478 @steveloughran , as advised, I have created backport pr on branch-3.4 for https://github.com/apache/hadoop/pull/6022. Requesting your kind review please. Thank you very much. > Expect-100 JDK bug resolution: prevent multiple server calls > > > Key: HADOOP-18883 > URL: https://issues.apache.org/jira/browse/HADOOP-18883 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978]. > > With the current implementation of HttpURLConnection if server rejects the > “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be > thrown from 'expect100Continue()' method. > After the exception thrown, If we call any other method on the same instance > (ex getHeaderField(), or getHeaderFields()). They will internally call > getOuputStream() which invokes writeRequests(), which make the actual server > call. > In the AbfsHttpOperation, after sendRequest() we call processResponse() > method from AbfsRestOperation. Even if the conn.getOutputStream() fails due > to expect-100 error, we consume the exception and let the code go ahead. So, > we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which > will be triggered after getOutputStream is failed. These invocation will lead > to server calls. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls [hadoop]
saxenapranav commented on PR #6511: URL: https://github.com/apache/hadoop/pull/6511#issuecomment-1914080478 @steveloughran , as advised, I have created backport pr on branch-3.4 for https://github.com/apache/hadoop/pull/6022. Requesting your kind review please. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[ https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811754#comment-17811754 ] ASF GitHub Bot commented on HADOOP-18883: - saxenapranav commented on PR #6484: URL: https://github.com/apache/hadoop/pull/6484#issuecomment-1914078990 @steveloughran , thank you very much for the review and the merge. I have created a pr https://github.com/apache/hadoop/pull/6511 on branch-3.4 Thank you very much. > Expect-100 JDK bug resolution: prevent multiple server calls > > > Key: HADOOP-18883 > URL: https://issues.apache.org/jira/browse/HADOOP-18883 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978]. > > With the current implementation of HttpURLConnection if server rejects the > “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be > thrown from 'expect100Continue()' method. > After the exception thrown, If we call any other method on the same instance > (ex getHeaderField(), or getHeaderFields()). They will internally call > getOuputStream() which invokes writeRequests(), which make the actual server > call. > In the AbfsHttpOperation, after sendRequest() we call processResponse() > method from AbfsRestOperation. Even if the conn.getOutputStream() fails due > to expect-100 error, we consume the exception and let the code go ahead. So, > we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which > will be triggered after getOutputStream is failed. These invocation will lead > to server calls. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple… [hadoop]
saxenapranav commented on PR #6484: URL: https://github.com/apache/hadoop/pull/6484#issuecomment-1914078990 @steveloughran , thank you very much for the review and the merge. I have created a pr https://github.com/apache/hadoop/pull/6511 on branch-3.4 Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[
https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811752#comment-17811752
]
ASF GitHub Bot commented on HADOOP-18883:
-
saxenapranav commented on PR #6511:
URL: https://github.com/apache/hadoop/pull/6511#issuecomment-1914076990
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64
The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:336 » TestTimedOut
test timed o...
[INFO]
[ERROR] Tests run: 567, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64
The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut
test timed o...
[ERROR]
ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:186->twoWriters:154
» TestTimedOut
[INFO]
[ERROR] Tests run: 560, Failures: 0, Errors: 2, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89
There should not be any network I/O (elapsedTimeMs=42).
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345
» TestTimedOut
[INFO]
[ERROR] Tests run: 551, Failures: 1, Errors: 1, Skipped: 264
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut
test timed o...
[INFO]
[ERROR] Tests run: 564, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
Time taken: 49 mins 32 secs.
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit 56109e0e6c8962deca87d004159742584e32c3cb (HEAD ->
branch-3.4_expect100resolution, origin/branch-3.4_expect100resolution)
Author: Pranav Saxena <108325433+saxenapra...@users.noreply.github.com>
Date: Sun Jan 21 11:14:54 2024 -0800
HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple
server calls (#6022)
> Expect-100 JDK bug resolution: prevent multiple server calls
>
>
> Key: HADOOP-18883
> URL: https://issues.apache.org/jira/browse/HADOOP-18883
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Pranav Saxena
>Assignee: Pranav Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978].
>
> With the current implementation of HttpURLConnection if server rejects the
> “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be
> thrown from 'expect100Continue()' method.
> After the exception thrown, If we call any other method on the same instance
> (ex getHeaderField(), or getHeaderFields()). They will internally call
> getOuputStream() which invokes writeRequests(), which make the actual server
> call.
> In the AbfsHttpOperation, after sendRequest() we call processResponse()
> method from AbfsRestOperation. Even if the conn.getOutputStream() fails due
> to expect-100 error, we consume the exception and let the code go ahead. So,
> we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which
> will be triggered after getOutputStream is failed.
Re: [PR] HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls [hadoop]
saxenapranav commented on PR #6511:
URL: https://github.com/apache/hadoop/pull/6511#issuecomment-1914076990
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64
The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:336 » TestTimedOut
test timed o...
[INFO]
[ERROR] Tests run: 567, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64
The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut
test timed o...
[ERROR]
ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:186->twoWriters:154
» TestTimedOut
[INFO]
[ERROR] Tests run: 560, Failures: 0, Errors: 2, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89
There should not be any network I/O (elapsedTimeMs=42).
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345
» TestTimedOut
[INFO]
[ERROR] Tests run: 551, Failures: 1, Errors: 1, Skipped: 264
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut
test timed o...
[INFO]
[ERROR] Tests run: 564, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41
Time taken: 49 mins 32 secs.
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit 56109e0e6c8962deca87d004159742584e32c3cb (HEAD ->
branch-3.4_expect100resolution, origin/branch-3.4_expect100resolution)
Author: Pranav Saxena <108325433+saxenapra...@users.noreply.github.com>
Date: Sun Jan 21 11:14:54 2024 -0800
HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple
server calls (#6022)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19056) Highlight RBF features and improvements targeting version 3.4
[ https://issues.apache.org/jira/browse/HADOOP-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-19056: -- Status: Patch Available (was: Open) > Highlight RBF features and improvements targeting version 3.4 > - > > Key: HADOOP-19056 > URL: https://issues.apache.org/jira/browse/HADOOP-19056 > Project: Hadoop Common > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > > I want to hight recent RBF features and improvements. > - Support observer node from Router-Based Federation > - Enhanced IPC throughput between Router and NameNode > - Improved isolation for downstream name nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-19056) Highlight RBF features and improvements targeting version 3.4
[ https://issues.apache.org/jira/browse/HADOOP-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811746#comment-17811746 ] ASF GitHub Bot commented on HADOOP-19056: - tasanuma commented on PR #6512: URL: https://github.com/apache/hadoop/pull/6512#issuecomment-1914022925 @slfan1989 I'm really sorry for being late, but if there is a chance to incorporate changes into 3.4.0, could you consider including this PR? If RC1 has already been created, perhaps it would be best for us to simply merge this into the trunk. > Highlight RBF features and improvements targeting version 3.4 > - > > Key: HADOOP-19056 > URL: https://issues.apache.org/jira/browse/HADOOP-19056 > Project: Hadoop Common > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > > I want to hight recent RBF features and improvements. > - Support observer node from Router-Based Federation > - Enhanced IPC throughput between Router and NameNode > - Improved isolation for downstream name nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. [hadoop]
tasanuma commented on PR #6512: URL: https://github.com/apache/hadoop/pull/6512#issuecomment-1914022925 @slfan1989 I'm really sorry for being late, but if there is a chance to incorporate changes into 3.4.0, could you consider including this PR? If RC1 has already been created, perhaps it would be best for us to simply merge this into the trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[
https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811745#comment-17811745
]
ASF GitHub Bot commented on HADOOP-18883:
-
hadoop-yetus commented on PR #6511:
URL: https://github.com/apache/hadoop/pull/6511#issuecomment-1914015509
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 27s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 21s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 18s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 18s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 22s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 24s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 41s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 19m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 9s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 41s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not
generate ASF License warnings. |
| | | 80m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6511/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6511 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7bdaba9722bb 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 56109e0e6c8962deca87d004159742584e32c3cb |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6511/1/testReport/ |
| Max. process+thread count | 638 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6511/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Expect-100 JDK bug resolution: prevent multiple
Re: [PR] HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls [hadoop]
hadoop-yetus commented on PR #6511:
URL: https://github.com/apache/hadoop/pull/6511#issuecomment-1914015509
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 27s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 21s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 18s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 18s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 22s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 24s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 41s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 19m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 9s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 41s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not
generate ASF License warnings. |
| | | 80m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6511/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6511 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7bdaba9722bb 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 56109e0e6c8962deca87d004159742584e32c3cb |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6511/1/testReport/ |
| Max. process+thread count | 638 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6511/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure
[jira] [Commented] (HADOOP-19056) Highlight RBF features and improvements targeting version 3.4
[ https://issues.apache.org/jira/browse/HADOOP-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811743#comment-17811743 ] ASF GitHub Bot commented on HADOOP-19056: - tasanuma opened a new pull request, #6512: URL: https://github.com/apache/hadoop/pull/6512 ### Description of PR I want to hight recent RBF features and improvements. - Support observer node from Router-Based Federation - Enhanced IPC throughput between Router and NameNode - Improved isolation for downstream name nodes. ### How was this patch tested? ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? > Highlight RBF features and improvements targeting version 3.4 > - > > Key: HADOOP-19056 > URL: https://issues.apache.org/jira/browse/HADOOP-19056 > Project: Hadoop Common > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > > I want to hight recent RBF features and improvements. > - Support observer node from Router-Based Federation > - Enhanced IPC throughput between Router and NameNode > - Improved isolation for downstream name nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-19056) Highlight RBF features and improvements targeting version 3.4
[ https://issues.apache.org/jira/browse/HADOOP-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19056: Labels: pull-request-available (was: ) > Highlight RBF features and improvements targeting version 3.4 > - > > Key: HADOOP-19056 > URL: https://issues.apache.org/jira/browse/HADOOP-19056 > Project: Hadoop Common > Issue Type: Task >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > > I want to hight recent RBF features and improvements. > - Support observer node from Router-Based Federation > - Enhanced IPC throughput between Router and NameNode > - Improved isolation for downstream name nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. [hadoop]
tasanuma opened a new pull request, #6512: URL: https://github.com/apache/hadoop/pull/6512 ### Description of PR I want to hight recent RBF features and improvements. - Support observer node from Router-Based Federation - Enhanced IPC throughput between Router and NameNode - Improved isolation for downstream name nodes. ### How was this patch tested? ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19056) Highlight RBF features and improvements targeting version 3.4
Takanobu Asanuma created HADOOP-19056: - Summary: Highlight RBF features and improvements targeting version 3.4 Key: HADOOP-19056 URL: https://issues.apache.org/jira/browse/HADOOP-19056 Project: Hadoop Common Issue Type: Task Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma I want to hight recent RBF features and improvements. - Support observer node from Router-Based Federation - Enhanced IPC throughput between Router and NameNode - Improved isolation for downstream name nodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11650. Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue [hadoop]
hadoop-yetus commented on PR #6490:
URL: https://github.com/apache/hadoop/pull/6490#issuecomment-1913991706
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 45s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 5s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 105m 18s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 250m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6490/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6490 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 16629d116fa6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8fd3663da83ada01238f6ae04b2a8cef9e731a09 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6490/4/testReport/ |
| Max. process+thread count | 897 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6490/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by |
Re: [PR] HDFS-17362. RBF: RouterObserverReadProxyProvider should use ConfiguredFailoverProxyProvider internally [hadoop]
hadoop-yetus commented on PR #6510:
URL: https://github.com/apache/hadoop/pull/6510#issuecomment-1913950036
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 12s | | trunk passed |
| +1 :green_heart: | compile | 5m 27s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 5m 19s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 5m 14s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 5m 14s | | the patch passed |
| +1 :green_heart: | compile | 5m 4s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 5m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 11s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6510/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 4 new + 0 unchanged - 0 fixed = 4
total (was 0) |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 26s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 9m 49s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 171m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6510/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6510 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 3434721bf1a2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 77d9bbc7ece50b49c9801318669b92ce96950d62 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6510/1/testReport/ |
| Max. process+thread count | 2605 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console
[jira] [Commented] (HADOOP-18883) Expect-100 JDK bug resolution: prevent multiple server calls
[ https://issues.apache.org/jira/browse/HADOOP-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811727#comment-17811727 ] ASF GitHub Bot commented on HADOOP-18883: - saxenapranav opened a new pull request, #6511: URL: https://github.com/apache/hadoop/pull/6511 Backport of https://github.com/apache/hadoop/commit/7dc166ddc7fa524d842c77c0d6e83dde6d13dc54 Trunk PR: https://github.com/apache/hadoop/pull/6022 Description: Address JDK bug JDK-8314978 related to handling of HTTP 100 responses. https://bugs.openjdk.org/browse/JDK-8314978 In the AbfsHttpOperation, after sendRequest() we call processResponse() method from AbfsRestOperation. Even if the conn.getOutputStream() fails due to expect-100 error, we consume the exception and let the code go ahead. This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong() after getOutputStream() has failed. These invocation all lead to server calls. This commit aims to prevent this. If connection.getOutputStream() fails due to an Expect-100 error, the ABFS client does not invoke getHeaderField(), getHeaderFields(), getHeaderFieldLong() or getInputStream(). getResponseCode() is safe as on the failure it sets the responseCode variable in HttpUrlConnection object. > Expect-100 JDK bug resolution: prevent multiple server calls > > > Key: HADOOP-18883 > URL: https://issues.apache.org/jira/browse/HADOOP-18883 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > This is inline to JDK bug: [https://bugs.openjdk.org/browse/JDK-8314978]. > > With the current implementation of HttpURLConnection if server rejects the > “Expect 100-continue” then there will be ‘java.net.ProtocolException’ will be > thrown from 'expect100Continue()' method. > After the exception thrown, If we call any other method on the same instance > (ex getHeaderField(), or getHeaderFields()). They will internally call > getOuputStream() which invokes writeRequests(), which make the actual server > call. > In the AbfsHttpOperation, after sendRequest() we call processResponse() > method from AbfsRestOperation. Even if the conn.getOutputStream() fails due > to expect-100 error, we consume the exception and let the code go ahead. So, > we can have getHeaderField() / getHeaderFields() / getHeaderFieldLong() which > will be triggered after getOutputStream is failed. These invocation will lead > to server calls. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls [hadoop]
saxenapranav opened a new pull request, #6511: URL: https://github.com/apache/hadoop/pull/6511 Backport of https://github.com/apache/hadoop/commit/7dc166ddc7fa524d842c77c0d6e83dde6d13dc54 Trunk PR: https://github.com/apache/hadoop/pull/6022 Description: Address JDK bug JDK-8314978 related to handling of HTTP 100 responses. https://bugs.openjdk.org/browse/JDK-8314978 In the AbfsHttpOperation, after sendRequest() we call processResponse() method from AbfsRestOperation. Even if the conn.getOutputStream() fails due to expect-100 error, we consume the exception and let the code go ahead. This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong() after getOutputStream() has failed. These invocation all lead to server calls. This commit aims to prevent this. If connection.getOutputStream() fails due to an Expect-100 error, the ABFS client does not invoke getHeaderField(), getHeaderFields(), getHeaderFieldLong() or getInputStream(). getResponseCode() is safe as on the failure it sets the responseCode variable in HttpUrlConnection object. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19055. Preparing for 1.3.0 development. [hadoop-thirdparty]
slfan1989 merged PR #28: URL: https://github.com/apache/hadoop-thirdparty/pull/28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19054. Update hadoop-thirdparty index.md.vm. [hadoop-thirdparty]
slfan1989 commented on PR #27: URL: https://github.com/apache/hadoop-thirdparty/pull/27#issuecomment-1913932566 @Hexiaoqiao Thanks for reviewing the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19054. Update hadoop-thirdparty index.md.vm. [hadoop-thirdparty]
slfan1989 merged PR #27: URL: https://github.com/apache/hadoop-thirdparty/pull/27 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17362. RBF: RouterObserverReadProxyProvider should use ConfiguredFailoverProxyProvider internally [hadoop]
tasanuma commented on code in PR #6510: URL: https://github.com/apache/hadoop/pull/6510#discussion_r1469042435 ## hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml: ## @@ -177,6 +177,11 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd;> junit-jupiter-params test + + org.junit.jupiter + junit-jupiter-engine + test + Review Comment: `mvn test -Dtest=TestObserverWithRouter` skips ParameterizedTest if we don't add junit-jupiter-engine here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] MAPREDUCE-7469 NNBench createControlFiles should use thread pool to i… [hadoop]
LiuGuH commented on code in PR #6463:
URL: https://github.com/apache/hadoop/pull/6463#discussion_r1469037519
##
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java:
##
@@ -134,27 +143,59 @@ private void cleanupBeforeTestrun() throws IOException {
*
* @throws IOException on error
*/
- private void createControlFiles() throws IOException {
+ private void createControlFiles() {
LOG.info("Creating " + numberOfMaps + " control files");
+List> list = new ArrayList<>();
for (int i = 0; i < numberOfMaps; i++) {
String strFileName = "NNBench_Controlfile_" + i;
Path filePath = new Path(new Path(baseDir, CONTROL_DIR_NAME),
strFileName);
+ Future future = executorService.submit(new
CreateControlFile(strFileName, filePath, i));
+ list.add(future);
+}
+
+for (int i = 0; i < list.size(); i++) {
+ try {
+list.get(i).get();
+ } catch (InterruptedException | ExecutionException e) {
+LOG.error("Creating control files Error.", e);
+ }
+}
+
+executorService.shutdown();
Review Comment:
Done. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19055. Preparing for 1.3.0 development. [hadoop-thirdparty]
slfan1989 commented on PR #28: URL: https://github.com/apache/hadoop-thirdparty/pull/28#issuecomment-1913839080 @Hexiaoqiao Can you help review this PR? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-19054. Update hadoop-thirdparty index.md.vm. [hadoop-thirdparty]
slfan1989 commented on PR #27: URL: https://github.com/apache/hadoop-thirdparty/pull/27#issuecomment-1913838940 @Hexiaoqiao Can you help review this PR? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17362. RBF: RouterObserverReadProxyProvider should use ConfiguredFailoverProxyProvider internally [hadoop]
tasanuma opened a new pull request, #6510: URL: https://github.com/apache/hadoop/pull/6510 ### Description of PR Currently, RouterObserverReadProxyProvider is using IPFailoverProxyProvider, while ObserverReadProxyProvider is using ConfiguredFailoverProxyProvider. If we are to align RouterObserverReadProxyProvider with ObserverReadProxyProvider, RouterObserverReadProxyProvider should internally use ConfiguredFailoverProxyProvider. Moreover, IPFailoverProxyProvider has an issue with resolving HA configurations. (For example, IPFailoverProxyProvider cannot resolve hdfs://router-service.) ### How was this patch tested? `mvn test -Dtest=TestObserverWithRouter` succeeded in my local computer. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17361. DiskBalancer: Query command support with multiple nodes [hadoop]
hadoop-yetus commented on PR #6508:
URL: https://github.com/apache/hadoop/pull/6508#issuecomment-1913714510
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 41m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 59s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6508/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 2 unchanged -
0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 261m 53s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6508/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 453m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6508/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6508 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 3213ddf32fed 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 11dc1860b7b57e710d68b1d0daa26dd396cc66b3 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6508/1/testReport/ |
| Max. process+thread count | 2610 (vs. ulimit of 5500) |
| modules | C:
Re: [PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
hadoop-yetus commented on PR #6509:
URL: https://github.com/apache/hadoop/pull/6509#issuecomment-1913701342
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 195m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 293m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6509 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d1e5494161f3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 30e3bdaa8262a0ee2ed2c00773d3420a0b096ac6 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/testReport/ |
| Max. process+thread count | 4210 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6509/1/console |
|
Re: [PR] HDFS-17348. Enhance Log when checkLocations in RecoveryTaskStriped. [hadoop]
hadoop-yetus commented on PR #6485:
URL: https://github.com/apache/hadoop/pull/6485#issuecomment-1913654980
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 43m 9s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6485/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 273m 52s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 428m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6485/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6485 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ccb8736929f8 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e211a9e63f433201420af3278adc1715ec179869 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6485/3/testReport/ |
| Max. process+thread count | 3459 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6485/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond
[jira] [Commented] (HADOOP-17351) hadoop: shaded client test is too aggressive
[
https://issues.apache.org/jira/browse/HADOOP-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811676#comment-17811676
]
ASF GitHub Bot commented on HADOOP-17351:
-
hadoop-yetus commented on PR #6507:
URL: https://github.com/apache/hadoop/pull/6507#issuecomment-1913651462
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 16m 17s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 14m 54s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 18m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 59s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 19s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 60m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 33s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | mvninstall | 28m 39s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 15m 23s |
[/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 15m 23s |
[/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 14m 45s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javac | 14m 45s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 58s | | the patch passed |
| -1 :x: | mvnsite | 4m 50s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 51s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 18s | | hadoop-project has no data from
spotbugs
Re: [PR] HADOOP-17351, Add Support for AWS S3 Access Grants [hadoop]
hadoop-yetus commented on PR #6507:
URL: https://github.com/apache/hadoop/pull/6507#issuecomment-1913651462
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 16m 17s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 14m 54s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 18m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 59s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 19s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 60m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 33s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | mvninstall | 28m 39s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 15m 23s |
[/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 15m 23s |
[/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 14m 45s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| -1 :x: | javac | 14m 45s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 58s | | the patch passed |
| -1 :x: | mvnsite | 4m 50s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 51s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 18s | | hadoop-project has no data from
spotbugs |
| -1 :x: | spotbugs | 0m 23s |
[/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6507/1/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch
[jira] [Updated] (HADOOP-19055) Preparing for 1.3.0 development
[ https://issues.apache.org/jira/browse/HADOOP-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19055: Labels: pull-request-available (was: ) > Preparing for 1.3.0 development > --- > > Key: HADOOP-19055 > URL: https://issues.apache.org/jira/browse/HADOOP-19055 > Project: Hadoop Common > Issue Type: Sub-task > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.3.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.3.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19055) Preparing for 1.3.0 development
Shilun Fan created HADOOP-19055: --- Summary: Preparing for 1.3.0 development Key: HADOOP-19055 URL: https://issues.apache.org/jira/browse/HADOOP-19055 Project: Hadoop Common Issue Type: Sub-task Components: hadoop-thirdparty Affects Versions: thirdparty-1.3.0 Reporter: Shilun Fan Assignee: Shilun Fan Fix For: thirdparty-1.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17350) hadoop: flag native changes w/no Dockerfile change
[
https://issues.apache.org/jira/browse/HADOOP-17350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811672#comment-17811672
]
ASF GitHub Bot commented on HADOOP-17350:
-
hadoop-yetus commented on PR #6506:
URL: https://github.com/apache/hadoop/pull/6506#issuecomment-1913631554
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 12s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 47s | | trunk passed |
| +1 :green_heart: | compile | 16m 9s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 14m 56s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 18m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 3s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 48m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 0s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 28m 21s | | the patch passed |
| +1 :green_heart: | compile | 15m 39s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 39s | | the patch passed |
| +1 :green_heart: | compile | 15m 1s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 15m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 13m 47s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 30s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 48m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 745m 28s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 33s | | The patch does not
generate ASF License warnings. |
| | | 1031m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6506 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux e627ea12d51c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ece1c87c1d08bac94639b834a5cab210deb5d23d |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/testReport/ |
| Max. process+thread count | 3337 (vs. ulimit of
Re: [PR] HADOOP-17350. Update AWS SDK to v2.23.7 [hadoop]
hadoop-yetus commented on PR #6506:
URL: https://github.com/apache/hadoop/pull/6506#issuecomment-1913631554
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 12s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 47s | | trunk passed |
| +1 :green_heart: | compile | 16m 9s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 14m 56s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 18m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 3s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 48m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 0s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 28m 21s | | the patch passed |
| +1 :green_heart: | compile | 15m 39s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 39s | | the patch passed |
| +1 :green_heart: | compile | 15m 1s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 15m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 13m 47s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 30s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 48m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 745m 28s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 33s | | The patch does not
generate ASF License warnings. |
| | | 1031m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6506 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux e627ea12d51c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ece1c87c1d08bac94639b834a5cab210deb5d23d |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/testReport/ |
| Max. process+thread count | 3337 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6506/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0
[jira] [Updated] (HADOOP-19054) Update hadoop-thirdparty index.md.vm
[ https://issues.apache.org/jira/browse/HADOOP-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-19054: Labels: pull-request-available (was: ) > Update hadoop-thirdparty index.md.vm > > > Key: HADOOP-19054 > URL: https://issues.apache.org/jira/browse/HADOOP-19054 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: thirdparty-1.2.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19054) Update hadoop-thirdparty index.md.vm
Shilun Fan created HADOOP-19054: --- Summary: Update hadoop-thirdparty index.md.vm Key: HADOOP-19054 URL: https://issues.apache.org/jira/browse/HADOOP-19054 Project: Hadoop Common Issue Type: Sub-task Affects Versions: thirdparty-1.2.0 Reporter: Shilun Fan Assignee: Shilun Fan Fix For: thirdparty-1.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero. [hadoop]
hfutatzhanghb opened a new pull request, #6509:
URL: https://github.com/apache/hadoop/pull/6509
### Description of PR
Refer to HDFS-17358.
Recently, there is a strange case happened on our ec production cluster.
The phenomenon is as below described: NameNode does infinite recovery lease
of some ec files(~80K+) and those files could never be closed.
After digging into logs and releated code, we found the root cause is below
codes in method `BlockRecoveryWorker$RecoveryTaskStriped#recover`:
```java
// we met info.getNumBytes==0 here!
if (info != null &&
info.getGenerationStamp() >= block.getGenerationStamp() &&
info.getNumBytes() > 0) {
final BlockRecord existing = syncBlocks.get(blockId);
if (existing == null ||
info.getNumBytes() > existing.rInfo.getNumBytes()) {
// if we have >1 replicas for the same internal block, we
// simply choose the one with larger length.
// TODO: better usage of redundant replicas
syncBlocks.put(blockId, new BlockRecord(id, proxyDN, info));
}
}
// throw exception here!
checkLocations(syncBlocks.size());
```
The related logs are as below:
>java.io.IOException:
BP-1157541496-10.104.10.198-1702548776421:blk_-9223372036808032688_2938828 has
no enough internal blocks, unable to start recovery. Locations=[...]
>2024-01-23 12:48:16,171 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
initReplicaRecovery: blk_-9223372036808032686_2938828, recoveryId=27615365,
replica=ReplicaUnderRecovery, blk_-9223372036808032686_2938828, RUR
getNumBytes() = 0 getBytesOnDisk() = 0 getVisibleLength()= -1 getVolume() =
/data25/hadoop/hdfs/datanode getBlockURI() =
file:/data25/hadoop/hdfs/datanode/current/BP-1157541496-x.x.x.x-1702548776421/current/rbw/blk_-9223372036808032686
recoveryId=27529675 original=ReplicaWaitingToBeRecovered,
blk_-9223372036808032686_2938828, RWR getNumBytes() = 0 getBytesOnDisk() = 0
getVisibleLength()= -1 getVolume() = /data25/hadoop/hdfs/datanode getBlockURI()
=
file:/data25/hadoop/hdfs/datanode/current/BP-1157541496-10.104.10.198-1702548776421/current/rbw/blk_-9223372036808032686
because the length of RWR is zero, the length of the returned object in
below codes is zero. We can't put it into syncBlocks.
So throw exception in checkLocations method.
>ReplicaRecoveryInfo info = callInitReplicaRecovery(proxyDN,new
RecoveringBlock(internalBlk, null, recoveryId));
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17361. DiskBalancer: Query command support with multiple nodes [hadoop]
haiyang1987 opened a new pull request, #6508: URL: https://github.com/apache/hadoop/pull/6508 ### Description of PR https://issues.apache.org/jira/browse/HDFS-17361 For: https://issues.apache.org/jira/browse/HDFS-10821 mentioned, Query command will support with multiple nodes. That means we can use command hdfs diskbalancer -query to print one or one more datanodes status of the diskbalancer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17348. Enhance Log when checkLocations in RecoveryTaskStriped. [hadoop]
hfutatzhanghb commented on PR #6485: URL: https://github.com/apache/hadoop/pull/6485#issuecomment-1913560613 @Hexiaoqiao @zhangshuyan0 @tomscut Hi, sir. Could you please take a look at this PR when you have free time? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17292. Show the number of times the slowPeerCollectorDaemon thread has collected SlowNodes. [hadoop]
hadoop-yetus commented on PR #6364:
URL: https://github.com/apache/hadoop/pull/6364#issuecomment-1913559858
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 47s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 58s | | trunk passed |
| +1 :green_heart: | compile | 5m 24s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 5m 10s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 39s | | the patch passed |
| +1 :green_heart: | compile | 5m 16s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 5m 16s | | the patch passed |
| +1 :green_heart: | compile | 5m 5s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 5m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 13s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6364/12/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 177 unchanged - 0 fixed =
178 total (was 177) |
| +1 :green_heart: | mvnsite | 1m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 58s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 222m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6364/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 22m 11s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 418m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6364/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6364 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0c25b2b539f7 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4d5d50144572e8eb3b6e356bde203d89f2fbe486 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
[jira] [Updated] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18197: Affects Version/s: thirdparty-1.2.0 > Update protobuf 3.7.1 to a version without CVE-2021-22569 > - > > Key: HADOOP-18197 > URL: https://issues.apache.org/jira/browse/HADOOP-18197 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: Ivan Viaznikov >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available, security > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The artifact `org.apache.hadoop:hadoop-common` brings in a dependency > `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version > released in 2013 and it contains a vulnerability > [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569]. > Therefore, requesting you to clarify if this library version is going to be > updated in the following releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18197. - Fix Version/s: thirdparty-1.2.0 Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0 Resolution: Fixed > Update protobuf 3.7.1 to a version without CVE-2021-22569 > - > > Key: HADOOP-18197 > URL: https://issues.apache.org/jira/browse/HADOOP-18197 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ivan Viaznikov >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available, security > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The artifact `org.apache.hadoop:hadoop-common` brings in a dependency > `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version > released in 2013 and it contains a vulnerability > [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569]. > Therefore, requesting you to clarify if this library version is going to be > updated in the following releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18197: Component/s: hadoop-thirdparty > Update protobuf 3.7.1 to a version without CVE-2021-22569 > - > > Key: HADOOP-18197 > URL: https://issues.apache.org/jira/browse/HADOOP-18197 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Reporter: Ivan Viaznikov >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available, security > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The artifact `org.apache.hadoop:hadoop-common` brings in a dependency > `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version > released in 2013 and it contains a vulnerability > [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569]. > Therefore, requesting you to clarify if this library version is going to be > updated in the following releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-18197: --- Assignee: PJ Fanning > Update protobuf 3.7.1 to a version without CVE-2021-22569 > - > > Key: HADOOP-18197 > URL: https://issues.apache.org/jira/browse/HADOOP-18197 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ivan Viaznikov >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available, security > Time Spent: 2h 20m > Remaining Estimate: 0h > > The artifact `org.apache.hadoop:hadoop-common` brings in a dependency > `com.google.protobuf:protobuf-java:2.5.0`, which is an outdated version > released in 2013 and it contains a vulnerability > [CVE-2021-22569|https://nvd.nist.gov/vuln/detail/CVE-2021-22569]. > Therefore, requesting you to clarify if this library version is going to be > updated in the following releases -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
[ https://issues.apache.org/jira/browse/HADOOP-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18843: Fix Version/s: thirdparty-1.2.0 (was: 3.4.0) > Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) > > > Key: HADOOP-18843 > URL: https://issues.apache.org/jira/browse/HADOOP-18843 > Project: Hadoop Common > Issue Type: Task > Components: hadoop-thirdparty >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
[ https://issues.apache.org/jira/browse/HADOOP-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18843: Target Version/s: thirdparty-1.2.0 (was: 3.4.0, 3.3.7) > Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) > > > Key: HADOOP-18843 > URL: https://issues.apache.org/jira/browse/HADOOP-18843 > Project: Hadoop Common > Issue Type: Task > Components: hadoop-thirdparty >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18843) Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23)
[ https://issues.apache.org/jira/browse/HADOOP-18843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18843: Affects Version/s: thirdparty-1.2.0 (was: 3.4.0) > Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) > > > Key: HADOOP-18843 > URL: https://issues.apache.org/jira/browse/HADOOP-18843 > Project: Hadoop Common > Issue Type: Task > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Create the corresponding jira for hadoop-thirdparty PR#23. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18921) upgrade avro in hadoop-thirdparty to 1.11.3
[ https://issues.apache.org/jira/browse/HADOOP-18921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18921: Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0 > upgrade avro in hadoop-thirdparty to 1.11.3 > --- > > Key: HADOOP-18921 > URL: https://issues.apache.org/jira/browse/HADOOP-18921 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: thirdparty-1.2.0 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > https://lists.apache.org/thread/wcj1747hvyl7qjhrfr6d6j1l62hvpr5l -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18921) upgrade avro in hadoop-thirdparty to 1.11.3
[ https://issues.apache.org/jira/browse/HADOOP-18921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18921: Component/s: hadoop-thirdparty (was: build) > upgrade avro in hadoop-thirdparty to 1.11.3 > --- > > Key: HADOOP-18921 > URL: https://issues.apache.org/jira/browse/HADOOP-18921 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > https://lists.apache.org/thread/wcj1747hvyl7qjhrfr6d6j1l62hvpr5l -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18721) Upgrade guava in hadoop-thirdparty to 31.1-jre
[ https://issues.apache.org/jira/browse/HADOOP-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18721: Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0 Affects Version/s: thirdparty-1.2.0 (was: 3.4.0) > Upgrade guava in hadoop-thirdparty to 31.1-jre > -- > > Key: HADOOP-18721 > URL: https://issues.apache.org/jira/browse/HADOOP-18721 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Upgrade guava in hadoop-thirdparty to 31.1-jre to be in sync with the > downstream project Tez -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18342) Upgrade to Avro 1.11.1
[ https://issues.apache.org/jira/browse/HADOOP-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18342: Component/s: hadoop-thirdparty > Upgrade to Avro 1.11.1 > -- > > Key: HADOOP-18342 > URL: https://issues.apache.org/jira/browse/HADOOP-18342 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Latest version of Avro. Aimed only at trunk as there is no security concern > addressed here. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18342) Upgrade to Avro 1.11.1
[ https://issues.apache.org/jira/browse/HADOOP-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18342: Affects Version/s: thirdparty-1.2.0 > Upgrade to Avro 1.11.1 > -- > > Key: HADOOP-18342 > URL: https://issues.apache.org/jira/browse/HADOOP-18342 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: thirdparty-1.2.0 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Latest version of Avro. Aimed only at trunk as there is no security concern > addressed here. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18342) Upgrade to Avro 1.11.1
[ https://issues.apache.org/jira/browse/HADOOP-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HADOOP-18342. - Fix Version/s: thirdparty-1.2.0 Hadoop Flags: Reviewed Target Version/s: thirdparty-1.2.0 Resolution: Fixed > Upgrade to Avro 1.11.1 > -- > > Key: HADOOP-18342 > URL: https://issues.apache.org/jira/browse/HADOOP-18342 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Latest version of Avro. Aimed only at trunk as there is no security concern > addressed here. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18342) Upgrade to Avro 1.11.1
[ https://issues.apache.org/jira/browse/HADOOP-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-18342: --- Assignee: PJ Fanning > Upgrade to Avro 1.11.1 > -- > > Key: HADOOP-18342 > URL: https://issues.apache.org/jira/browse/HADOOP-18342 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Latest version of Avro. Aimed only at trunk as there is no security concern > addressed here. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18282) Add .asf.yaml to hadoop-thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18282: Target Version/s: thirdparty-1.2.0 > Add .asf.yaml to hadoop-thirdparty > -- > > Key: HADOOP-18282 > URL: https://issues.apache.org/jira/browse/HADOOP-18282 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > > no yaml file in thirdparty, it is dropping mails to common-dev for everything. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18282) Add .asf.yaml to hadoop-thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18282: Component/s: hadoop-thirdparty > Add .asf.yaml to hadoop-thirdparty > -- > > Key: HADOOP-18282 > URL: https://issues.apache.org/jira/browse/HADOOP-18282 > Project: Hadoop Common > Issue Type: Improvement > Components: hadoop-thirdparty >Affects Versions: thirdparty-1.2.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > > no yaml file in thirdparty, it is dropping mails to common-dev for everything. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18282) Add .asf.yaml to hadoop-thirdparty
[ https://issues.apache.org/jira/browse/HADOOP-18282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18282: Affects Version/s: thirdparty-1.2.0 > Add .asf.yaml to hadoop-thirdparty > -- > > Key: HADOOP-18282 > URL: https://issues.apache.org/jira/browse/HADOOP-18282 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: thirdparty-1.2.0 >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: thirdparty-1.2.0 > > > no yaml file in thirdparty, it is dropping mails to common-dev for everything. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19053) Release Hadoop-Thirdparty 1.2.0
Shilun Fan created HADOOP-19053: --- Summary: Release Hadoop-Thirdparty 1.2.0 Key: HADOOP-19053 URL: https://issues.apache.org/jira/browse/HADOOP-19053 Project: Hadoop Common Issue Type: Task Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17360. Record the number of times a block is read during a certain time period. [hadoop]
huangzhaobo99 commented on PR #6505: URL: https://github.com/apache/hadoop/pull/6505#issuecomment-1913528137 Hi @tasanuma @Hexiaoqiao @zhangshuyan0, Please kindly review this PR as well if you have bandwidth, Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18183) s3a audit logs to publish range start/end of GET requests in audit header
[
https://issues.apache.org/jira/browse/HADOOP-18183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18183:
Hadoop Flags: Reviewed
Description:
we don't get the range of ranged get requests in s3 server logs, because the
AWS s3 log doesn't record that information. we can see it's a partial get from
the 206 response, but the length of data retrieved is lost.
LoggingAuditor.beforeExecution() would need to recognise a ranged GET and
determine the extra key-val pairs for range start and end (rs & re?)
we might need to modify {{HttpReferrerAuditHeader.buildHttpReferrer()}} to take
a map of so it can dynamically create a header for each
request; currently that is not in there.
was:
we don't get the range of ranged get requests in s3 server logs, because the
AWS s3 log doesn't record that information. we can see it's a partial get from
the 206 response, but the length of data retrieved is lost.
LoggingAuditor.beforeExecution() would need to recognise a ranged GET and
determine the extra key-val pairs for range start and end (rs & re?)
we might need to modify {{HttpReferrerAuditHeader.buildHttpReferrer()}} to take
a map of so it can dynamically create a header for each
request; currently that is not in there.
> s3a audit logs to publish range start/end of GET requests in audit header
> -
>
> Key: HADOOP-18183
> URL: https://issues.apache.org/jira/browse/HADOOP-18183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.2
>Reporter: Steve Loughran
>Assignee: Ankit Saurabh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> we don't get the range of ranged get requests in s3 server logs, because the
> AWS s3 log doesn't record that information. we can see it's a partial get
> from the 206 response, but the length of data retrieved is lost.
> LoggingAuditor.beforeExecution() would need to recognise a ranged GET and
> determine the extra key-val pairs for range start and end (rs & re?)
> we might need to modify {{HttpReferrerAuditHeader.buildHttpReferrer()}} to
> take a map of so it can dynamically create a header for each
> request; currently that is not in there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
[ https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18186: Hadoop Flags: Reviewed Target Version/s: 3.3.6, 3.4.0 > s3a prefetching to use SemaphoredDelegatingExecutor for submitting work > --- > > Key: HADOOP-18186 > URL: https://issues.apache.org/jira/browse/HADOOP-18186 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Use SemaphoredDelegatingExecutor for each to stream to submit work, if > possible, for better fairness in processes with many streams. > this also takes a DurationTrackerFactory to count how long was spent in the > queue, something we would want to know > if cherrypicking, follow with HADOOP-18466 and HADOOP-18455 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18190) Collect IOStatistics during S3A prefetching
[ https://issues.apache.org/jira/browse/HADOOP-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18190: Hadoop Flags: Reviewed Target Version/s: 3.3.6, 3.4.0 > Collect IOStatistics during S3A prefetching > > > Key: HADOOP-18190 > URL: https://issues.apache.org/jira/browse/HADOOP-18190 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > There is a lot more happening in reads, so there's a lot more data to collect > and publish in IO stats for us to view in a summary at the end of processes > as well as get from the stream while it is active. > Some useful ones would seem to be: > counters > * is in memory. using 0 or 1 here lets aggregation reports count total #of > memory cached files. > * prefetching operations executed > * errors during prefetching > gauges > * number of blocks in cache > * total size of blocks > * active prefetches > + active memory used > duration tracking count/min/max/ave > * time to fetch a block > * time queued before the actual fetch begins > * time a reader is blocked waiting for a block fetch to complete > and some info on cache use itself > * number of blocks discarded unread > * number of prefetched blocks later used > * number of backward seeks to a prefetched block > * number of forward seeks to a prefetched block > the key ones I care about are > # memory consumption > # can we determine if cache is working (reads with cache hit) and when it is > not (misses, wasted prefetches) > # time blocked on executors > The stats need to be accessible on a stream even when closed, and aggregated > into the FS. once we get per-thread stats contexts we can publish there too > and collect in worker threads for reporting in task commits -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18189) S3PrefetchingInputStream to support status probes when closed
[ https://issues.apache.org/jira/browse/HADOOP-18189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18189: Hadoop Flags: Reviewed Target Version/s: 3.3.6, 3.4.0 > S3PrefetchingInputStream to support status probes when closed > - > > Key: HADOOP-18189 > URL: https://issues.apache.org/jira/browse/HADOOP-18189 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > S3PrefetchingInputStream is a bit over aggressive on raising > exceptions/downgrading responses after a stream is closed > * MUST: getPos() to return last read location, or a least 0 (maybe we should > add this to filesystem spec) > * MUST: getIOStatistics(). critical for collecting stats in processes > * MUST: seekToNewSource() (it's a no op anyway) > * MAY S3AInputStreamStatistics and getStreamStatistics() though that is only > in used in testing...ioStatistics have given us a stable stats api. we may > want to tag the method @VisibleForTesting to discourage use. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18188) Support touch command for directory
[ https://issues.apache.org/jira/browse/HADOOP-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18188: Hadoop Flags: Reviewed > Support touch command for directory > --- > > Key: HADOOP-18188 > URL: https://issues.apache.org/jira/browse/HADOOP-18188 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0 >Reporter: Akira Ajisaka >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently hadoop fs -touch command cannot update the mtime and the atime of > directory. The feature would be useful when we check whether the filesystem > is ready to write or not without creating any file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18191) Log retry count while handling exceptions in RetryInvocationHandler
[ https://issues.apache.org/jira/browse/HADOOP-18191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18191: Hadoop Flags: Reviewed Target Version/s: 3.3.5, 3.2.4, 2.10.2, 3.4.0 (was: 3.4.0, 2.10.2, 3.2.4, 3.3.5) > Log retry count while handling exceptions in RetryInvocationHandler > --- > > Key: HADOOP-18191 > URL: https://issues.apache.org/jira/browse/HADOOP-18191 > Project: Hadoop Common > Issue Type: Task > Components: common, io >Affects Versions: 3.4.0, 2.10.2, 3.2.4, 3.3.5 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.5 > > Time Spent: 1h > Remaining Estimate: 0h > > As part of failure handling in RetryInvocationHandler, we log details of the > Exception details with which API was invoked, failover attempts, delay. > For the purpose of better debugging as well as fine-tuning of retry params, > it would be good to also log retry count that we already maintain in the > Counter object. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18193) Support nested mount points in INodeTree
[
https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18193:
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
> Support nested mount points in INodeTree
>
>
> Key: HADOOP-18193
> URL: https://issues.apache.org/jira/browse/HADOOP-18193
> Project: Hadoop Common
> Issue Type: Improvement
> Components: viewfs
>Affects Versions: 2.10.0
>Reporter: Lei Yang
>Assignee: Lei Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Nested Mount Point in ViewFs.pdf
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Defining following client mount table config is not supported in INodeTree
> and will throw FileAlreadyExistsException
>
> {code:java}
> fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar
> fs.viewfs.mounttable.link./foo=hdfs://nn02/foo
> {code}
> INodeTree has 2 methods that need change to support nested mount points.
> {code:java}
> createLink(): build INodeTree during fs init.
> resolve(): resolve path in INodeTree with viewfs apis.
> {code}
> ViewFileSystem and ViewFs maintains an INodeTree instance(fsState) in both
> classes and call fsState.resolve(..) to resolve path to specific mount point.
> INodeTree.resolve encapsulates the logic of nested mount point resolving. So
> no changes are expected in both classes.
> AC:
> # INodeTree.createlink should support creating nested mount
> points.(INodeTree is constructed during fs init)
> # INodeTree.resolve should support resolve path based on nested mount
> points. (INodeTree.resolve is used in viewfs apis)
> # No regression in existing ViewFileSystem and ViewFs apis.
> # Ensure some important apis are not broken with nested mount points.
> (Rename, getContentSummary, listStatus...)
>
> Spec:
> Please review attached pdf for spec about this feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18206: Component/s: common Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.4.0 >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18196) Remove replace-guava from replacer plugin
[
https://issues.apache.org/jira/browse/HADOOP-18196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18196:
Hadoop Flags: Reviewed
> Remove replace-guava from replacer plugin
> -
>
> Key: HADOOP-18196
> URL: https://issues.apache.org/jira/browse/HADOOP-18196
> Project: Hadoop Common
> Issue Type: Task
> Components: build
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> While running the build, realized that all replacer plugin executions run
> only after "banned-illegal-imports" enforcer plugin.
> For instance,
> {code:java}
> [INFO] --- maven-enforcer-plugin:3.0.0:enforce (banned-illegal-imports) @
> hadoop-cloud-storage ---
> [INFO]
> [INFO] --- replacer:1.5.3:replace (replace-generated-sources) @
> hadoop-cloud-storage ---
> [INFO] Skipping
> [INFO]
> [INFO] --- replacer:1.5.3:replace (replace-sources) @ hadoop-cloud-storage ---
> [INFO] Skipping
> [INFO]
> [INFO] --- replacer:1.5.3:replace (replace-guava) @ hadoop-cloud-storage ---
> [INFO] Replacement run on 0 file.
> [INFO] {code}
> Hence, if our source code uses com.google.common, banned-illegal-imports will
> cause the build failure and replacer plugin would not even get executed.
> We should remove it as it is only redundant execution step.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18202) create-release fails fatal: unsafe repository ('/build/source' is owned by someone else)
[
https://issues.apache.org/jira/browse/HADOOP-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18202:
Hadoop Flags: Reviewed
Target Version/s: 3.3.3, 3.4.0
Description:
The first git command in the create-release docker container is failing as git
doesn't consider /build/source to belong to the current user.
since CVE-2022-24765 and the april 2022 git releases to refuse to work in such
directories
https://github.blog/2022-04-12-git-security-vulnerability-announced/
{code}
$ /usr/bin/git clean -xdf -e /patchprocess
fatal: unsafe repository ('/build/source' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /build/source
{code}
was:
The first git command in the create-release docker container is failing as git
doesn't consider /build/source to belong to the current user.
since CVE-2022-24765 and the april 2022 git releases to refuse to work in such
directories
https://github.blog/2022-04-12-git-security-vulnerability-announced/
{code}
$ /usr/bin/git clean -xdf -e /patchprocess
fatal: unsafe repository ('/build/source' is owned by someone else)
To add an exception for this directory, call:
git config --global --add safe.directory /build/source
{code}
> create-release fails fatal: unsafe repository ('/build/source' is owned by
> someone else)
>
>
> Key: HADOOP-18202
> URL: https://issues.apache.org/jira/browse/HADOOP-18202
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.5, 3.3.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.3
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The first git command in the create-release docker container is failing as
> git doesn't consider /build/source to belong to the current user.
> since CVE-2022-24765 and the april 2022 git releases to refuse to work in
> such directories
> https://github.blog/2022-04-12-git-security-vulnerability-announced/
> {code}
> $ /usr/bin/git clean -xdf -e /patchprocess
> fatal: unsafe repository ('/build/source' is owned by someone else)
> To add an exception for this directory, call:
> git config --global --add safe.directory /build/source
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18214) Update BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18214: Hadoop Flags: Reviewed > Update BUILDING.txt > --- > > Key: HADOOP-18214 > URL: https://issues.apache.org/jira/browse/HADOOP-18214 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Affects Versions: 3.3.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 40m > Remaining Estimate: 0h > > update building.txt to match the docker build settings. > this patch has already gone in, just without a jira in its name > https://github.com/apache/hadoop/pull/3811 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18214) Update BUILDING.txt
[ https://issues.apache.org/jira/browse/HADOOP-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-18214: --- Assignee: Steve Loughran > Update BUILDING.txt > --- > > Key: HADOOP-18214 > URL: https://issues.apache.org/jira/browse/HADOOP-18214 > Project: Hadoop Common > Issue Type: Improvement > Components: build, documentation >Affects Versions: 3.3.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Time Spent: 40m > Remaining Estimate: 0h > > update building.txt to match the docker build settings. > this patch has already gone in, just without a jira in its name > https://github.com/apache/hadoop/pull/3811 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18215) Enhance WritableName to be able to return aliases for classes that use serializers
[
https://issues.apache.org/jira/browse/HADOOP-18215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18215:
Component/s: common
Hadoop Flags: Reviewed
Target Version/s: 3.3.6, 3.4.0
Affects Version/s: 3.3.6
3.4.0
> Enhance WritableName to be able to return aliases for classes that use
> serializers
> --
>
> Key: HADOOP-18215
> URL: https://issues.apache.org/jira/browse/HADOOP-18215
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.4.0, 3.3.6
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> WritableName allows users shim in aliases for writables, in the case where a
> SequenceFile was written with a Writable class that has since been renamed or
> moved to another package. However, this requires that the aliased class
> extend Writable.
> Separately it's possible to configure jobs with keys and values which don't
> actually extend Writable. Instead they are meant to be
> serialized/deserialized using the serialization classes defined in
> {{io.serializations}} config.
> Unfortunately, the current implementation does not support these key/value
> classes. All we need to do to support this is remove the
> {{.asSubclass(Writable.class)}} as is already the case for the default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18216) Document "io.file.buffer.size" must be greater than zero
[
https://issues.apache.org/jira/browse/HADOOP-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18216:
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Document "io.file.buffer.size" must be greater than zero
>
>
> Key: HADOOP-18216
> URL: https://issues.apache.org/jira/browse/HADOOP-18216
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 3.4.0
>Reporter: ECFuzz
>Assignee: ECFuzz
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> when the configuration file in the "io.file.buffer.size" field is set to a
> value less than or equal to zero, hdfs can start normally, but read and write
> data will have problems.
> When the value is less than zero, the shell will throw the following
> exception:
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> -cat: Fatal internal error
> java.lang.NegativeArraySizeException: -4096
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:93)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:101)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391){code}
> When the value is equal to zero, the shell command will always block
> {code:java}
> hadoop@ljq1:~/hadoop-3.1.3-work/bin$ ./hdfs dfs -cat mapred
> ^Z
> [2]+ Stopped ./hdfs dfs -cat mapred{code}
> The description of the configuration file is not clear enough, it may make
> people think that set to 0 to enter the non-blocking mode.
>
> {code:java}
>
> io.file.buffer.size
> 4096
> The size of buffer for use in sequence files.
> The size of this buffer should probably be a multiple of hardware
> page size (4096 on Intel x86), and it determines how much data is
> buffered during read and write operations.
> {code}
>
> Considering that this value is uesd by hdfs and mapreduce frequently, we
> should make this value must be a number greater than zero.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[ https://issues.apache.org/jira/browse/HADOOP-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18217: Hadoop Flags: Reviewed Target Version/s: 3.3.5, 3.4.0 > shutdownhookmanager should not be multithreaded (deadlock possible) > --- > > Key: HADOOP-18217 > URL: https://issues.apache.org/jira/browse/HADOOP-18217 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.10.1 > Environment: linux, windows, any version >Reporter: Catherinot Remi >Assignee: Catherinot Remi >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Attachments: wtf.java > > Time Spent: 4h 40m > Remaining Estimate: 0h > > the ShutdownHookManager class uses an executor to run hooks to have a > "timeout" notion around them. It does this using a single threaded executor. > It can leads to deadlock leaving a never-shutting-down JVM with this > execution flow: > * JVM need to exit (only daemon threads remaining or someone called > System.exit) > * ShutdowHookManager kicks in > * SHMngr executor start running some hooks > * SHMngr executor thread kicks in and, as a side effect, run some code from > one of the hook that calls System.exit (as a side effect from an external lib > for example) > * the executor thread is waiting for a lock because another thread already > entered System.exit and has its internal lock, so the executor never returns. > * SHMngr never returns > * 1st call to System.exit never returns > * JVM stuck > > using an executor with a single thread does "fake" timeouts (the task keeps > running, you can interrupt it but until it stumble upon some piece of code > that is interruptible (like an IO) it will keep running) especially since the > executor is a single threaded one. So it has this bug for example : > * caller submit 1st hook (bad one that would need 1 hour of runtime and that > cannot be interrupted) > * executor start 1st hook > * caller of the future 1st hook result timeout > * caller submit 2nd hook > * bug : 1 hook still running, 2nd hook triggers a timeout but never got the > chance to run anyway, so 1st faulty hook makes it impossible for any other > hook to have a chance to run, so running hooks in a single separate thread > does not allow to run other hooks in parallel to long ones. > > If we really really want to timeout the JVM shutdown, even accepting maybe > dirty shutdown, it should rather handle the hooks inside the initial thread > (not spawning new one(s) so not triggering the deadlock described on the 1st > place) and if a timeout was configured, only spawn a single parallel daemon > thread that sleeps the timeout delay, and then use Runtime.halt (which bypass > the hook system so should not trigger the deadlock). If the normal > System.exit ends before the timeout delay everything is fine. If the > System.exit took to much time, the JVM is killed and so the reason why this > multithreaded shutdown hook implementation was created is satisfied (avoding > having hanging JVMs) > > Had the bug with both oracle and open jdk builds, all in 1.8 major version. > hadoop 2.6 and 2.7 did not have the issue because they do not run hooks in > another thread > > Another solution is of course to configure the timeout AND to have as many > threads as needed to run the hooks so to have at least some gain to offset > the pain of the dealock scenario > > EDIT: added some logs and reproduced the problem. in fact it is located after > triggering all the hook entries and before shutting down the executor. > Current code, after running the hooks, creates a new Configuration object and > reads the configured timeout from it, applies this timeout to shutdown the > executor. I sometimes run with a classloader doing remote classloading, > Configuration loads its content using this classloader, so when shutting down > the JVM and some network error occurs the classloader fails to load the > ressources needed by Configuration. So the code crash before shutting down > the executor and ends up inside the thread's default uncaught throwable > handler, which was calling System.exit, so got stuck, so shutting down the > executor never returned, so does the JVM. > So, forget about the halt stuff (even if it is a last ressort very robust > safety net). Still I'll do a small adjustement to the final executor shutdown > code to be slightly more robust to even the strangest exceptions/errors it > encounters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[ https://issues.apache.org/jira/browse/HADOOP-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HADOOP-18217: --- Assignee: Catherinot Remi > shutdownhookmanager should not be multithreaded (deadlock possible) > --- > > Key: HADOOP-18217 > URL: https://issues.apache.org/jira/browse/HADOOP-18217 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.10.1 > Environment: linux, windows, any version >Reporter: Catherinot Remi >Assignee: Catherinot Remi >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Attachments: wtf.java > > Time Spent: 4h 40m > Remaining Estimate: 0h > > the ShutdownHookManager class uses an executor to run hooks to have a > "timeout" notion around them. It does this using a single threaded executor. > It can leads to deadlock leaving a never-shutting-down JVM with this > execution flow: > * JVM need to exit (only daemon threads remaining or someone called > System.exit) > * ShutdowHookManager kicks in > * SHMngr executor start running some hooks > * SHMngr executor thread kicks in and, as a side effect, run some code from > one of the hook that calls System.exit (as a side effect from an external lib > for example) > * the executor thread is waiting for a lock because another thread already > entered System.exit and has its internal lock, so the executor never returns. > * SHMngr never returns > * 1st call to System.exit never returns > * JVM stuck > > using an executor with a single thread does "fake" timeouts (the task keeps > running, you can interrupt it but until it stumble upon some piece of code > that is interruptible (like an IO) it will keep running) especially since the > executor is a single threaded one. So it has this bug for example : > * caller submit 1st hook (bad one that would need 1 hour of runtime and that > cannot be interrupted) > * executor start 1st hook > * caller of the future 1st hook result timeout > * caller submit 2nd hook > * bug : 1 hook still running, 2nd hook triggers a timeout but never got the > chance to run anyway, so 1st faulty hook makes it impossible for any other > hook to have a chance to run, so running hooks in a single separate thread > does not allow to run other hooks in parallel to long ones. > > If we really really want to timeout the JVM shutdown, even accepting maybe > dirty shutdown, it should rather handle the hooks inside the initial thread > (not spawning new one(s) so not triggering the deadlock described on the 1st > place) and if a timeout was configured, only spawn a single parallel daemon > thread that sleeps the timeout delay, and then use Runtime.halt (which bypass > the hook system so should not trigger the deadlock). If the normal > System.exit ends before the timeout delay everything is fine. If the > System.exit took to much time, the JVM is killed and so the reason why this > multithreaded shutdown hook implementation was created is satisfied (avoding > having hanging JVMs) > > Had the bug with both oracle and open jdk builds, all in 1.8 major version. > hadoop 2.6 and 2.7 did not have the issue because they do not run hooks in > another thread > > Another solution is of course to configure the timeout AND to have as many > threads as needed to run the hooks so to have at least some gain to offset > the pain of the dealock scenario > > EDIT: added some logs and reproduced the problem. in fact it is located after > triggering all the hook entries and before shutting down the executor. > Current code, after running the hooks, creates a new Configuration object and > reads the configured timeout from it, applies this timeout to shutdown the > executor. I sometimes run with a classloader doing remote classloading, > Configuration loads its content using this classloader, so when shutting down > the JVM and some network error occurs the classloader fails to load the > ressources needed by Configuration. So the code crash before shutting down > the executor and ends up inside the thread's default uncaught throwable > handler, which was calling System.exit, so got stuck, so shutting down the > executor never returned, so does the JVM. > So, forget about the halt stuff (even if it is a last ressort very robust > safety net). Still I'll do a small adjustement to the final executor shutdown > code to be slightly more robust to even the strangest exceptions/errors it > encounters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands,
[jira] [Updated] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18219:
Component/s: test
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by:
[jira] [Updated] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18219:
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
>
[jira] [Updated] (HADOOP-18222) Prevent DelegationTokenSecretManagerMetrics from registering multiple times
[
https://issues.apache.org/jira/browse/HADOOP-18222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18222:
Component/s: common
Hadoop Flags: Reviewed
Target Version/s: 3.3.5, 3.4.0
Affects Version/s: 3.3.5
3.4.0
> Prevent DelegationTokenSecretManagerMetrics from registering multiple times
>
>
> Key: HADOOP-18222
> URL: https://issues.apache.org/jira/browse/HADOOP-18222
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0, 3.3.5
>Reporter: Hector Sandoval Chaverri
>Assignee: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> After committing HADOOP-18167, we received reports of the following error
> when ResourceManager is initialized:
> {noformat}
> Caused by: java.io.IOException: Problem starting http server
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1389)
> at
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:475)
> ... 4 more
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> DelegationTokenSecretManagerMetrics already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.(AbstractDelegationTokenSecretManager.java:152)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.(DelegationTokenManager.java:72)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.(DelegationTokenManager.java:122)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180)
> at
> org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53){noformat}
> This can happen if MetricsSystemImpl#init is called and multiple metrics are
> registered with the same name. A proposed solution is to declare the metrics
> in AbstractDelegationTokenSecretManager as singleton, which would prevent
> multiple instances DelegationTokenSecretManagerMetrics from being registered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18229) Fix Hadoop Common Java Doc Errors
[
https://issues.apache.org/jira/browse/HADOOP-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18229:
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
> Fix Hadoop Common Java Doc Errors
> -
>
> Key: HADOOP-18229
> URL: https://issues.apache.org/jira/browse/HADOOP-18229
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, common
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 13h 20m
> Remaining Estimate: 0h
>
> I found that when hadoop-multibranch compiled PR-4266, some errors would pop
> up, I tried to solve it
> The wrong compilation information is as follows, I try to fix the Error
> information
> {code:java}
> [ERROR]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:432:
> error: exception not thrown: java.io.IOException
> [ERROR]* @throws IOException
> [ERROR] ^
> [ERROR]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:885:
> error: unknown tag: username
> [ERROR]* E.g. link: ^/user/(?\\w+) =>
> s3://$user.apache.com/_${user}
> [ERROR] ^
> [ERROR]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:885:
> error: bad use of '>'
> [ERROR]* E.g. link: ^/user/(?\\w+) =>
> s3://$user.apache.com/_${user}
> [ERROR]^
> [ERROR]
> /home/jenkins/jenkins-agent/workspace/hadoop-multibranch_PR-4266/ubuntu-focal/src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:910:
> error: unknown tag: username
> [ERROR]*
> .linkRegex.replaceresolveddstpath:_:-#.^/user/(?\w+)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18224) Upgrade maven compiler plugin to 3.10.1
[
https://issues.apache.org/jira/browse/HADOOP-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18224:
Hadoop Flags: Reviewed
> Upgrade maven compiler plugin to 3.10.1
> ---
>
> Key: HADOOP-18224
> URL: https://issues.apache.org/jira/browse/HADOOP-18224
> Project: Hadoop Common
> Issue Type: Task
> Components: build
>Affects Versions: 3.4.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Currently we are using maven-compiler-plugin 3.1 version, which is quite old
> (2013) and it's also pulling in vulnerable log4j dependency:
> {code:java}
> [INFO]
> org.apache.maven.plugins:maven-compiler-plugin:maven-plugin:3.1:runtime
> [INFO] org.apache.maven.plugins:maven-compiler-plugin:jar:3.1
> [INFO] org.apache.maven:maven-plugin-api:jar:2.0.9
> [INFO] org.apache.maven:maven-artifact:jar:2.0.9
> [INFO] org.codehaus.plexus:plexus-utils:jar:1.5.1
> [INFO] org.apache.maven:maven-core:jar:2.0.9
> [INFO] org.apache.maven:maven-settings:jar:2.0.9
> [INFO] org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.9
> ...
> ...
> ...
> [INFO] log4j:log4j:jar:1.2.12
> [INFO] commons-logging:commons-logging-api:jar:1.1
> [INFO] com.google.collections:google-collections:jar:1.0
> [INFO] junit:junit:jar:3.8.2
> {code}
>
> We should upgrade to 3.10.1 (latest Mar, 2022) version of
> maven-compiler-plugin.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18231) tests in ITestS3AInputStreamPerformance are failing
[ https://issues.apache.org/jira/browse/HADOOP-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18231: Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > tests in ITestS3AInputStreamPerformance are failing > > > Key: HADOOP-18231 > URL: https://issues.apache.org/jira/browse/HADOOP-18231 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Assignee: Ahmar Suhail >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6h > Remaining Estimate: 0h > > The following tests are failing when prefetching is enabled: > testRandomIORandomPolicy - expects stream to be opened 4 times (once for > every random read), but prefetching will only open twice. > testDecompressionSequential128K - expects stream to be opened once, but > prefetching will open once for each block the file has. landsat file used in > the test has size 42MB, prefetching block size = 8MB, expected open count is > 6. > testReadWithNormalPolicy - same as above. > testRandomIONormalPolicy - executes random IO, but with a normal policy. > S3AInputStream will abort the stream and change the policy, prefetching > handles random IO by caching blocks so doesn't do any of that. > testRandomReadOverBuffer - multiple assertions failing here, also depends a > lot on readAhead values, not very relevant for prefetching -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18233) Initialization race condition with TemporaryAWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18233:
Target Version/s: 3.3.5, 3.4.0
> Initialization race condition with TemporaryAWSCredentialsProvider
> --
>
> Key: HADOOP-18233
> URL: https://issues.apache.org/jira/browse/HADOOP-18233
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth, fs/s3
>Affects Versions: 3.3.1
> Environment: spark v3.2.0
> hadoop-aws v3.3.1
> java version 1.8.0_265 via zulu-8
>Reporter: Jason Sleight
>Assignee: Jimmy Wong
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> I'm in the process of upgrading spark+hadoop versions for my workflows and
> observing a weird behavior regression. I'm setting
> {code:java}
> spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
> spark.sql.catalogImplementation=hive
> spark.hadoop.aws.region=us-west-2
> ...many other things, I think these might be the relevant ones though...{code}
> in Spark config and I'm observing some non-fatal warnings/exceptions (see
> below for some examples). The warnings/exceptions randomly appear for some
> tasks, which causes them to fail, but then when Spark retries the task it
> will succeed. The initial tasks don't always fail either, just sometimes.
> I also found that if I switch to a SimpleAWSCredentials and use static keys,
> then I don't see any issues.
> My old setup was spark v3.0.2 with hadoop-aws v3.2.1 which also does not have
> these warnings/exceptions.
> From reading some other tickets I thought perhaps adding
> {code:java}
> spark.sql.hive.metastore.sharedPrefixes=com.amazonaws {code}
> would help, but it did not.
> Appreciate any suggestions for how to proceed or debug further :)
>
> Example stack traces:
> First one for an s3 read
> {code:java}
> WARN TaskSetManager: Lost task 27.0 in stage 4.0 (TID 29) ( executor
> 13): java.nio.file.AccessDeniedException:
> s3a://bucket/path/to/part.snappy.parquet:
> org.apache.hadoop.fs.s3a.CredentialInitializationException: Provider
> TemporaryAWSCredentialsProvider has no credentials
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:206)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3289)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
> at
> org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:268)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$1(ParquetFileFormat.scala:267)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(ParquetFileFormat.scala:270)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:164)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
> at
> org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:522)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.columnartorow_nextBatch_0$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
> at
> org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at
>
[jira] [Updated] (HADOOP-18233) Initialization race condition with TemporaryAWSCredentialsProvider
[
https://issues.apache.org/jira/browse/HADOOP-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-18233:
Hadoop Flags: Reviewed
Target Version/s: 3.3.5, 3.4.0 (was: 3.4.0, 3.3.5)
> Initialization race condition with TemporaryAWSCredentialsProvider
> --
>
> Key: HADOOP-18233
> URL: https://issues.apache.org/jira/browse/HADOOP-18233
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth, fs/s3
>Affects Versions: 3.3.1
> Environment: spark v3.2.0
> hadoop-aws v3.3.1
> java version 1.8.0_265 via zulu-8
>Reporter: Jason Sleight
>Assignee: Jimmy Wong
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> I'm in the process of upgrading spark+hadoop versions for my workflows and
> observing a weird behavior regression. I'm setting
> {code:java}
> spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
> spark.sql.catalogImplementation=hive
> spark.hadoop.aws.region=us-west-2
> ...many other things, I think these might be the relevant ones though...{code}
> in Spark config and I'm observing some non-fatal warnings/exceptions (see
> below for some examples). The warnings/exceptions randomly appear for some
> tasks, which causes them to fail, but then when Spark retries the task it
> will succeed. The initial tasks don't always fail either, just sometimes.
> I also found that if I switch to a SimpleAWSCredentials and use static keys,
> then I don't see any issues.
> My old setup was spark v3.0.2 with hadoop-aws v3.2.1 which also does not have
> these warnings/exceptions.
> From reading some other tickets I thought perhaps adding
> {code:java}
> spark.sql.hive.metastore.sharedPrefixes=com.amazonaws {code}
> would help, but it did not.
> Appreciate any suggestions for how to proceed or debug further :)
>
> Example stack traces:
> First one for an s3 read
> {code:java}
> WARN TaskSetManager: Lost task 27.0 in stage 4.0 (TID 29) ( executor
> 13): java.nio.file.AccessDeniedException:
> s3a://bucket/path/to/part.snappy.parquet:
> org.apache.hadoop.fs.s3a.CredentialInitializationException: Provider
> TemporaryAWSCredentialsProvider has no credentials
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:206)
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3289)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
> at
> org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:39)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:268)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.footerFileMetaData$1(ParquetFileFormat.scala:267)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(ParquetFileFormat.scala:270)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:164)
> at
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
> at
> org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:522)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.columnartorow_nextBatch_0$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage7.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
> at
> org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at
>
[jira] [Updated] (HADOOP-18237) Upgrade Apache Xerces Java to 2.12.2
[ https://issues.apache.org/jira/browse/HADOOP-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18237: Hadoop Flags: Reviewed Target Version/s: 3.3.4, 3.4.0 > Upgrade Apache Xerces Java to 2.12.2 > > > Key: HADOOP-18237 > URL: https://issues.apache.org/jira/browse/HADOOP-18237 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.3 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.4 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Description > https://github.com/advisories/GHSA-h65f-jvqw-m9fj > There's a vulnerability within the Apache Xerces Java (XercesJ) XML parser > when handling specially crafted XML document payloads. This causes, the > XercesJ XML parser to wait in an infinite loop, which may sometimes consume > system resources for prolonged duration. This vulnerability is present within > XercesJ version 2.12.1 and the previous versions. > References > [https://nvd.nist.gov/vuln/detail/CVE-2022-23437] > https://lists.apache.org/thread/6pjwm10bb69kq955fzr1n0nflnjd27dl > http://www.openwall.com/lists/oss-security/2022/01/24/3 > https://www.oracle.com/security-alerts/cpuapr2022.html -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
