Re: [PR] HDFS-17121. BPServiceActor to provide new thread to handle FBR [hadoop]
hadoop-yetus commented on PR #5888:
URL: https://github.com/apache/hadoop/pull/5888#issuecomment-1811963263
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/32/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/32/artifact/out/blanks-tabs.txt)
| The patch 260 line(s) with tabs. |
| -0 :warning: | checkstyle | 0m 32s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/32/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 264 unchanged
- 0 fixed = 268 total (was 264) |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 190m 20s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/32/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 27s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/32/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 279m 31s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStream |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5888/32/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5888 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint |
| uname | Linux 173499360b32 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 227f
[jira] [Commented] (HADOOP-18869) ABFS: Fixing Behavior of a File System APIs on root path
[
https://issues.apache.org/jira/browse/HADOOP-18869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786202#comment-17786202
]
ASF GitHub Bot commented on HADOOP-18869:
-
hadoop-yetus commented on PR #6274:
URL: https://github.com/apache/hadoop/pull/6274#issuecomment-1811942044
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | docker | 2m 21s | | Docker failed to build run-specific
yetus/hadoop:tp-6753}. |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6274 |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6274/1/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> ABFS: Fixing Behavior of a File System APIs on root path
>
>
> Key: HADOOP-18869
> URL: https://issues.apache.org/jira/browse/HADOOP-18869
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.6
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Following HDFS Apis are failing when called on a root path.
> {*}{*}{*}{*}{*}{*}
> |FS Call|Status|Error thrown to caller|
> |create()|Failing|Operation failed: "The request URI is invalid.", 400, PUT,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-02076119-21ed-4ada-bcd0-14afaae54013/?resource=file&timeout=90,
> InvalidUri, "The request URI is invalid.
> RequestId:1d23f8c2-d01f-0059-61b6-c60c2400
> Time:2023-08-04T09:29:55.4813818Z"|
> |createNonRecursive()|Failing|Runtime Exception:
> java.lang.IllegalArgumentException: null path (This is occuring because
> getParentPath is null and getFileStatus is called on null)|
> |setXAttr()|Failing|Operation failed: "The request URI is invalid.", 400,
> HEAD,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false&timeout=90|
> |getXAttr()|Failing|Operation failed: "The request URI is invalid.", 400,
> HEAD,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false&timeout=91|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17121. BPServiceActor to provide new thread to handle FBR [hadoop]
LiuGuH commented on PR #6005: URL: https://github.com/apache/hadoop/pull/6005#issuecomment-1811942122 @Hexiaoqiao hello, sir. This is a new clean build. Could you review it？Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18869: [Backport to branch-3.3] [ABFS] Fixing Behavior of a File System APIs on root path. [hadoop]
hadoop-yetus commented on PR #6274: URL: https://github.com/apache/hadoop/pull/6274#issuecomment-1811942044 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 2m 21s | | Docker failed to build run-specific yetus/hadoop:tp-6753}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6274 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6274/1/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18869) ABFS: Fixing Behavior of a File System APIs on root path
[
https://issues.apache.org/jira/browse/HADOOP-18869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786197#comment-17786197
]
ASF GitHub Bot commented on HADOOP-18869:
-
anujmodi2021 opened a new pull request, #6274:
URL: https://github.com/apache/hadoop/pull/6274
Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18869
Trunk PR: https://github.com/apache/hadoop/pull/6003
Cherry-picked Commit:
https://github.com/apache/hadoop/commit/594e9f29f5d89563bd9afc2e45553d624a6accc7
Test Results:
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[WARNING] Tests run: 590, Failures: 0, Errors: 0, Skipped: 274
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
Time taken: 25 mins 33 secs.
> ABFS: Fixing Behavior of a File System APIs on root path
>
>
> Key: HADOOP-18869
> URL: https://issues.apache.org/jira/browse/HADOOP-18869
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.6
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Following HDFS Apis are failing when called on a root path.
> {*}{*}{*}{*}{*}{*}
> |FS Call|Status|Error thrown to caller|
> |create()|Failing|Operation failed: "The request URI is invalid.", 400, PUT,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-02076119-21ed-4ada-bcd0-14afaae54013/?resource=file&timeout=90,
> InvalidUri, "The request URI is invalid.
> RequestId:1d23f8c2-d01f-0059-61b6-c60c2400
> Time:2023-08-04T09:29:55.4813818Z"|
> |createNonRecursive()|Failing|Runtime Exception:
> java.lang.IllegalArgumentException: null path (This is occuring because
> getParentPath is null and getFileStatus is called on null)|
> |setXAttr()|Failing|Operation failed: "The request URI is invalid.", 400,
> HEAD,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false&timeout=90|
> |getXAttr()|Failing|Operation failed: "The request URI is invalid.", 400,
> HEAD,
> https://anujtesthns.dfs.core.windows.net/abfs-testcontainer-491399b3-c3d0-4568-9d4a-a26e0aa8f000/?upn=false&timeout=91|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-18869: [Backport to branch-3.3] [ABFS] Fixing Behavior of a File System APIs on root path. [hadoop]
anujmodi2021 opened a new pull request, #6274:
URL: https://github.com/apache/hadoop/pull/6274
Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18869
Trunk PR: https://github.com/apache/hadoop/pull/6003
Cherry-picked Commit:
https://github.com/apache/hadoop/commit/594e9f29f5d89563bd9afc2e45553d624a6accc7
Test Results:
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[WARNING] Tests run: 590, Failures: 0, Errors: 0, Skipped: 274
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
Time taken: 25 mins 33 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17377) ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata Service
[
https://issues.apache.org/jira/browse/HADOOP-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786191#comment-17786191
]
ASF GitHub Bot commented on HADOOP-17377:
-
hadoop-yetus commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811906117
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 20s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in trunk failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -0 :warning: | checkstyle | 0m 18s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt)
| The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 20s |
[/branch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| -1 :x: | javadoc | 0m 20s |
[/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | javadoc | 4m 44s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 0m 47s |
[/branch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| +1 :green_heart: | shadedclient | 7m 10s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 7m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s |
[/patch-c
Re: [PR] HADOOP-17377: AABFS: MsiTokenProvider doesn't retry HTTP 429/410 from the Instance Metadata Service [hadoop]
hadoop-yetus commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811906117
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 20s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in trunk failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -0 :warning: | checkstyle | 0m 18s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt)
| The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 20s |
[/branch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| -1 :x: | javadoc | 0m 20s |
[/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | javadoc | 4m 44s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 0m 47s |
[/branch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| +1 :green_heart: | shadedclient | 7m 10s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 7m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/19/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
[jira] [Commented] (HADOOP-17377) ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata Service
[
https://issues.apache.org/jira/browse/HADOOP-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786189#comment-17786189
]
ASF GitHub Bot commented on HADOOP-17377:
-
hadoop-yetus commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811896245
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in trunk failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -0 :warning: | checkstyle | 0m 18s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt)
| The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 22s |
[/branch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| -1 :x: | javadoc | 0m 20s |
[/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 21s |
[/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in trunk failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | spotbugs | 0m 21s |
[/branch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 21s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 2m 42s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 20s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 20s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/had
Re: [PR] HADOOP-17377: AABFS: MsiTokenProvider doesn't retry HTTP 429/410 from the Instance Metadata Service [hadoop]
hadoop-yetus commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811896245
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in trunk failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -0 :warning: | checkstyle | 0m 18s |
[/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-azure.txt)
| The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 22s |
[/branch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-mvnsite-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| -1 :x: | javadoc | 0m 20s |
[/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 21s |
[/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-azure in trunk failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | spotbugs | 0m 21s |
[/branch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 21s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 2m 42s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt)
| hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 20s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 20s |
[/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5273/18/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-azure in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile |
[jira] [Commented] (HADOOP-17377) ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata Service
[
https://issues.apache.org/jira/browse/HADOOP-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786175#comment-17786175
]
ASF GitHub Bot commented on HADOOP-17377:
-
anmolanmol1234 commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811839555
> I think this this PR is great, however there's still one related open
problem: the default values (2) for
`fs.azure.oauth.token.fetch.retry.delta.backoff` is incorrect. The value of 2
is consistent with MS recommendation
(https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/how-to-use-vm-token#retry-guidance),
but it is assumed in **seconds**, but as this is used in Thread.sleep
[here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java#L326),
it will be measured in **milliseconds**. I think we should change the default
to 2000. @steveloughran @anmolanmol1234 do you think we can implement this
minimal change in this PR, or we should open a separate one?
Will update this change as an iteration of this PR, but will some time for
the mockito upgrade PR.
> ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata
> Service
>
>
> Key: HADOOP-17377
> URL: https://issues.apache.org/jira/browse/HADOOP-17377
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Brandon
>Priority: Major
> Labels: pull-request-available
>
> *Summary*
> The instance metadata service has its own guidance for error handling and
> retry which are different from the Blob store.
> [https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token#error-handling]
> In particular, it responds with HTTP 429 if request rate is too high. Whereas
> Blob store will respond with HTTP 503. The retry policy used only accounts
> for the latter as it will retry any status >=500. This can result in job
> instability when running multiple processes on the same host.
> *Environment*
> * Spark talking to an ABFS store
> * Hadoop 3.2.1
> * Running on an Azure VM with user-assigned identity, ABFS configured to use
> MsiTokenProvider
> * 6 executor processes on each VM
> *Example*
> Here's an example error message and stack trace. It's always the same stack
> trace. This appears in logs a few hundred to low thousands of times a day.
> It's luckily skating by since the download operation is wrapped in 3 retries.
> {noformat}
> AADToken: HTTP connection failed for getting token from AzureAD. Http
> response: 429 null
> Content-Type: application/json; charset=utf-8 Content-Length: 90 Request ID:
> Proxies: none
> First 1K of Body: {"error":"invalid_request","error_description":"Temporarily
> throttled, too many requests"}
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:190)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:125)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:506)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:489)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:208)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:473)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:437)
> at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1717)
> at org.apache.spark.util.Utils$.fetchHcfsFile(Utils.scala:747)
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:724)
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:496)
> at
> org.apache.spark.executor.Executor.$anonfun$updateDependencies$7(Executor.scala:812)
> at
> org.apache.spark.executor.Executor.$anonfun$updateDependencies$7$adapted(Executor.scala:803)
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792)
> at
> scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
> at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
> at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791)
>
Re: [PR] HADOOP-17377: AABFS: MsiTokenProvider doesn't retry HTTP 429/410 from the Instance Metadata Service [hadoop]
anmolanmol1234 commented on PR #5273: URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811839555 > I think this this PR is great, however there's still one related open problem: the default values (2) for `fs.azure.oauth.token.fetch.retry.delta.backoff` is incorrect. The value of 2 is consistent with MS recommendation (https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/how-to-use-vm-token#retry-guidance), but it is assumed in **seconds**, but as this is used in Thread.sleep [here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java#L326), it will be measured in **milliseconds**. I think we should change the default to 2000. @steveloughran @anmolanmol1234 do you think we can implement this minimal change in this PR, or we should open a separate one? Will update this change as an iteration of this PR, but will some time for the mockito upgrade PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17377) ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata Service
[
https://issues.apache.org/jira/browse/HADOOP-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786173#comment-17786173
]
ASF GitHub Bot commented on HADOOP-17377:
-
anmolanmol1234 commented on PR #5273:
URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811834929
> I'll go with whatever @saxenapranav thinks here...we have seen this
ourselves and need a fix.
>
> However, that PR to update mockito bounced, so either
>
> 1. another attempt is made to update mockito, including the shaded client
> 2. this PR can be done without updating mockito (easier)
The mockito upgrade was needed as part of this PR to mock static methods. So
would it be fine if we remove that test method or if not I will attempt to
upgrade mockito, including the shaded client.
> ABFS: MsiTokenProvider doesn't retry HTTP 429 from the Instance Metadata
> Service
>
>
> Key: HADOOP-17377
> URL: https://issues.apache.org/jira/browse/HADOOP-17377
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Brandon
>Priority: Major
> Labels: pull-request-available
>
> *Summary*
> The instance metadata service has its own guidance for error handling and
> retry which are different from the Blob store.
> [https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token#error-handling]
> In particular, it responds with HTTP 429 if request rate is too high. Whereas
> Blob store will respond with HTTP 503. The retry policy used only accounts
> for the latter as it will retry any status >=500. This can result in job
> instability when running multiple processes on the same host.
> *Environment*
> * Spark talking to an ABFS store
> * Hadoop 3.2.1
> * Running on an Azure VM with user-assigned identity, ABFS configured to use
> MsiTokenProvider
> * 6 executor processes on each VM
> *Example*
> Here's an example error message and stack trace. It's always the same stack
> trace. This appears in logs a few hundred to low thousands of times a day.
> It's luckily skating by since the download operation is wrapped in 3 retries.
> {noformat}
> AADToken: HTTP connection failed for getting token from AzureAD. Http
> response: 429 null
> Content-Type: application/json; charset=utf-8 Content-Length: 90 Request ID:
> Proxies: none
> First 1K of Body: {"error":"invalid_request","error_description":"Temporarily
> throttled, too many requests"}
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:190)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:125)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:506)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:489)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:208)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:473)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:437)
> at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1717)
> at org.apache.spark.util.Utils$.fetchHcfsFile(Utils.scala:747)
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:724)
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:496)
> at
> org.apache.spark.executor.Executor.$anonfun$updateDependencies$7(Executor.scala:812)
> at
> org.apache.spark.executor.Executor.$anonfun$updateDependencies$7$adapted(Executor.scala:803)
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792)
> at
> scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
> at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
> at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791)
> at
> org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:803)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:375)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
Re: [PR] HADOOP-17377: AABFS: MsiTokenProvider doesn't retry HTTP 429/410 from the Instance Metadata Service [hadoop]
anmolanmol1234 commented on PR #5273: URL: https://github.com/apache/hadoop/pull/5273#issuecomment-1811834929 > I'll go with whatever @saxenapranav thinks here...we have seen this ourselves and need a fix. > > However, that PR to update mockito bounced, so either > > 1. another attempt is made to update mockito, including the shaded client > 2. this PR can be done without updating mockito (easier) The mockito upgrade was needed as part of this PR to mock static methods. So would it be fine if we remove that test method or if not I will attempt to upgrade mockito, including the shaded client. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17129. mis-order of ibr and fbr on datanode [hadoop]
LiuGuH commented on PR #6244: URL: https://github.com/apache/hadoop/pull/6244#issuecomment-1811744058 @virajjasani could you reivew it ?Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18956) Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider
[
https://issues.apache.org/jira/browse/HADOOP-18956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786102#comment-17786102
]
ASF GitHub Bot commented on HADOOP-18956:
-
jojochuang commented on code in PR #6263:
URL: https://github.com/apache/hadoop/pull/6263#discussion_r1393407606
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java:
##
@@ -297,8 +298,8 @@ public List getAllOtherNodes() {
@Override
protected boolean isSSLEnabled() {
-return conf.getBoolean(
-DFSConfigKeys.ZK_CLIENT_SSL_ENABLED,
-DFSConfigKeys.DEFAULT_ZK_CLIENT_SSL_ENABLED);
+return conf.getBoolean(CommonConfigurationKeys.ZK_CLIENT_SSL_ENABLED,
+conf.getBoolean(DFSConfigKeys.ZK_CLIENT_SSL_ENABLED,
+DFSConfigKeys.DEFAULT_ZK_CLIENT_SSL_ENABLED));
Review Comment:
This would imply dfs.ha.zkfc.client.ssl.enabled becomes defunct. And perhaps
we should update hdfs-default.xml too.
Why is it different from YARN? In YARN,
yarn.resourcemanager.zk-client-ssl.enabled takes effect if
hadoop.zk.ssl.enabled is not set.
> Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and
> ZKSignerSecretProvider
> --
>
> Key: HADOOP-18956
> URL: https://issues.apache.org/jira/browse/HADOOP-18956
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Zita Dombi
>Assignee: István Fajth
>Priority: Major
> Labels: pull-request-available
>
> HADOOP-18709 added support for Zookeeper to communicate with SSL/TLS enabled
> in hadoop-common. With those changes we have the necessary parameters, that
> we need to set to enable SSL/TLS in a ZK Client. That change also did changes
> in ZKCuratorManager, so with that it is easy to set the SSL/TLS, for Yarn it
> was done in YARN-11468.
> In DelegationTokenAuthenticationFilter currently we are using
> CuratorFrameworkFactory, it'd be good to change it to use ZKCuratorManager
> and with that we should support SSL/TLS enablement.
> *UPDATE*
> So as I investigated this a bit more, it wouldn't be so easy to move to using
> ZKCuratorManager.
> DelegationTokenAuthenticationFilter uses ZK from two places: in
> ZKDelegationTokenSecretManager and in ZKSignerSecretProvider. In both places
> it uses CuratorFrameworkFactory, but the attributes and creation
> differentiates from ZKCuratorManager.
> In ZKDelegationTokenSecretManager it would be easy to add the new config and
> based on that create ZK with CuratorFrameworkFactory. But
> ZKSignerSecretProvider is in hadoop-auth module and with my change it would
> need hadoop-common, so it would introduce circular dependency between modules
> 'hadoop-auth' and 'hadoop-common'. I'm still working on a straightforward
> solution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider [hadoop]
jojochuang commented on code in PR #6263:
URL: https://github.com/apache/hadoop/pull/6263#discussion_r1393407606
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java:
##
@@ -297,8 +298,8 @@ public List getAllOtherNodes() {
@Override
protected boolean isSSLEnabled() {
-return conf.getBoolean(
-DFSConfigKeys.ZK_CLIENT_SSL_ENABLED,
-DFSConfigKeys.DEFAULT_ZK_CLIENT_SSL_ENABLED);
+return conf.getBoolean(CommonConfigurationKeys.ZK_CLIENT_SSL_ENABLED,
+conf.getBoolean(DFSConfigKeys.ZK_CLIENT_SSL_ENABLED,
+DFSConfigKeys.DEFAULT_ZK_CLIENT_SSL_ENABLED));
Review Comment:
This would imply dfs.ha.zkfc.client.ssl.enabled becomes defunct. And perhaps
we should update hdfs-default.xml too.
Why is it different from YARN? In YARN,
yarn.resourcemanager.zk-client-ssl.enabled takes effect if
hadoop.zk.ssl.enabled is not set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18972) Bug in SaslPropertiesResolver allows mutation of internal state
[
https://issues.apache.org/jira/browse/HADOOP-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786091#comment-17786091
]
ASF GitHub Bot commented on HADOOP-18972:
-
hadoop-yetus commented on PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#issuecomment-1811440780
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 42s | | trunk passed |
| +1 :green_heart: | compile | 9m 34s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 2s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 9m 3s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 9m 3s | | the patch passed |
| +1 :green_heart: | compile | 8m 53s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 8m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | |
hadoop-common-project/hadoop-common: The patch generated 0 new + 5 unchanged -
1 fixed = 5 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 1s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 139m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6272/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6272 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux da92ec0221f2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8a432bdf79bd43f05353df53f1d81717bee783d7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6272/2/testReport/ |
| Max. process+thread count | 1842 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6272/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This
Re: [PR] HADOOP-18972: Copy SASL properties map when returning from SaslPropertiesResolver [hadoop]
hadoop-yetus commented on PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#issuecomment-1811440780
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 42s | | trunk passed |
| +1 :green_heart: | compile | 9m 34s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 2s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 9m 3s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 9m 3s | | the patch passed |
| +1 :green_heart: | compile | 8m 53s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 8m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | |
hadoop-common-project/hadoop-common: The patch generated 0 new + 5 unchanged -
1 fixed = 5 total (was 6) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 1s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 139m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6272/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6272 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux da92ec0221f2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8a432bdf79bd43f05353df53f1d81717bee783d7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6272/2/testReport/ |
| Max. process+thread count | 1842 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6272/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr.
[jira] [Commented] (HADOOP-18972) Bug in SaslPropertiesResolver allows mutation of internal state
[
https://issues.apache.org/jira/browse/HADOOP-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786051#comment-17786051
]
ASF GitHub Bot commented on HADOOP-18972:
-
charlesconnell commented on code in PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#discussion_r1393188374
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
Review Comment:
I'm subclassing it now. I didn't see examples of other test classes
overriding its methods though, and it didn't seem to make sense to do that.
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
+
+ private static final SaslRpcServer.QualityOfProtection PRIVACY_QOP =
SaslRpcServer.QualityOfProtection.PRIVACY;
+ private static final SaslRpcServer.QualityOfProtection AUTHENTICATION_QOP =
SaslRpcServer.QualityOfProtection.AUTHENTICATION;
+ private static final InetAddress LOCALHOST = new
InetSocketAddress("127.0.0.1", 1).getAddress();
+
+ private SaslPropertiesResolver resolver;
+
+ @Before
+ public void setup() {
+Configuration conf = new Configuration();
+conf.set(HADOOP_RPC_PROTECTION, "privacy");
+resolver = new SaslPropertiesResolver();
+resolver.setConf(conf);
+ }
+
+ @Test
+ public void testResolverDoesNotMutate() {
+assertEquals(PRIVACY_QOP.getSaslQop(),
resolver.getDefaultProperties().get(Sasl.QOP));
Review Comment:
Removed a bunch of duplication.
> Bug in SaslPropertiesResolver allows mutation of internal state
> ---
>
> Key: HADOOP-18972
> URL: https://issues.apache.org/jira/browse/HADOOP-18972
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Charles Connell
>Priority: Minor
> Labels: pull-request-available
>
> When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a
> SASL properties map to do a handshake, they call
> {{SaslPropertiesResolver#getServerProperties()}} or
> {{SaslPropertiesResolver#getClientProperties()}}, and they get back a
> {{Map}}. Every call gets the same {{Map}} object back, and
> then the callers sometimes call
> [put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385]
> on it. This means that future users of {{SaslPropertiesResolver}} get back
> the wrong information.
> I propose that {{SaslPropertiesResolver}} should pass a copy of its internal
> map, so that users can safety modify them.
> I discovered this problem in my company's testing environment as we began to
> enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes
> were using {{IngressPortBasedResolver}} to give out block tokens with
> different QOPs depending on the port used. The our HDFS client applications
> became unable to read or write to HDFS because they could not find a QOP in
> common with the DataNodes during SASL handshake. With multiple threads
> executing SASL handshakes at the same time, the properties map used in
> {{SaslDataTransferServer}} in a DataNode could be clobbered during usage,
> since the same map was used by all threads. Also, future clients that do not
> have a QOP embedded in their block tokens would connect to a server with the
> wrong SASL properties map. I think that one or both of these issues explains
> the problem that I saw. I eliminated this unsafety and saw the problem go
> away.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18972: Copy SASL properties map when returning from SaslPropertiesResolver [hadoop]
charlesconnell commented on code in PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#discussion_r1393188374
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
Review Comment:
I'm subclassing it now. I didn't see examples of other test classes
overriding its methods though, and it didn't seem to make sense to do that.
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
+
+ private static final SaslRpcServer.QualityOfProtection PRIVACY_QOP =
SaslRpcServer.QualityOfProtection.PRIVACY;
+ private static final SaslRpcServer.QualityOfProtection AUTHENTICATION_QOP =
SaslRpcServer.QualityOfProtection.AUTHENTICATION;
+ private static final InetAddress LOCALHOST = new
InetSocketAddress("127.0.0.1", 1).getAddress();
+
+ private SaslPropertiesResolver resolver;
+
+ @Before
+ public void setup() {
+Configuration conf = new Configuration();
+conf.set(HADOOP_RPC_PROTECTION, "privacy");
+resolver = new SaslPropertiesResolver();
+resolver.setConf(conf);
+ }
+
+ @Test
+ public void testResolverDoesNotMutate() {
+assertEquals(PRIVACY_QOP.getSaslQop(),
resolver.getDefaultProperties().get(Sasl.QOP));
Review Comment:
Removed a bunch of duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18972) Bug in SaslPropertiesResolver allows mutation of internal state
[
https://issues.apache.org/jira/browse/HADOOP-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786049#comment-17786049
]
ASF GitHub Bot commented on HADOOP-18972:
-
charlesconnell commented on code in PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#discussion_r1393186082
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslPropertiesResolver.java:
##
@@ -82,7 +83,7 @@ public Configuration getConf() {
* @return sasl Properties
*/
public Map getDefaultProperties() {
-return properties;
+return new TreeMap<>(properties);
Review Comment:
done
> Bug in SaslPropertiesResolver allows mutation of internal state
> ---
>
> Key: HADOOP-18972
> URL: https://issues.apache.org/jira/browse/HADOOP-18972
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Charles Connell
>Priority: Minor
> Labels: pull-request-available
>
> When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a
> SASL properties map to do a handshake, they call
> {{SaslPropertiesResolver#getServerProperties()}} or
> {{SaslPropertiesResolver#getClientProperties()}}, and they get back a
> {{Map}}. Every call gets the same {{Map}} object back, and
> then the callers sometimes call
> [put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385]
> on it. This means that future users of {{SaslPropertiesResolver}} get back
> the wrong information.
> I propose that {{SaslPropertiesResolver}} should pass a copy of its internal
> map, so that users can safety modify them.
> I discovered this problem in my company's testing environment as we began to
> enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes
> were using {{IngressPortBasedResolver}} to give out block tokens with
> different QOPs depending on the port used. The our HDFS client applications
> became unable to read or write to HDFS because they could not find a QOP in
> common with the DataNodes during SASL handshake. With multiple threads
> executing SASL handshakes at the same time, the properties map used in
> {{SaslDataTransferServer}} in a DataNode could be clobbered during usage,
> since the same map was used by all threads. Also, future clients that do not
> have a QOP embedded in their block tokens would connect to a server with the
> wrong SASL properties map. I think that one or both of these issues explains
> the problem that I saw. I eliminated this unsafety and saw the problem go
> away.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18972: Copy SASL properties map when returning from SaslPropertiesResolver [hadoop]
charlesconnell commented on code in PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#discussion_r1393186082
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslPropertiesResolver.java:
##
@@ -82,7 +83,7 @@ public Configuration getConf() {
* @return sasl Properties
*/
public Map getDefaultProperties() {
-return properties;
+return new TreeMap<>(properties);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11498 backport jettison exclusions [hadoop]
hadoop-yetus commented on PR #6063:
URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1810808995
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 54s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 46s | | branch-3.3 passed |
| +1 :green_heart: | compile | 19m 40s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 8m 46s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 6m 44s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 120m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 23m 3s | | the patch passed |
| +1 :green_heart: | compile | 18m 43s | | the patch passed |
| +1 :green_heart: | javac | 18m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 8m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 51m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 15s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 5m 2s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 23m 19s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | unit | 4m 25s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 95m 47s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 1m 14s | |
hadoop-yarn-applications-catalog-webapp in the patch passed. |
| +1 :green_heart: | unit | 0m 56s | | hadoop-resourceestimator in the
patch passed. |
| +1 :green_heart: | unit | 0m 36s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 34s | | hadoop-client-minicluster in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 380m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| | hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6063 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 342cf66f4955 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / f3ecc9addfcf157c42514a377a885f21808cdce8 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/9/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-p
[jira] [Updated] (HADOOP-18973) Clean yum cache after installing which in hadoop2 docker image
[
https://issues.apache.org/jira/browse/HADOOP-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-18973:
Labels: pull-request-available (was: )
> Clean yum cache after installing which in hadoop2 docker image
> --
>
> Key: HADOOP-18973
> URL: https://issues.apache.org/jira/browse/HADOOP-18973
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
>
> Minor addendum to HADOOP-18970: clean yum cache after installing {{which}} to
> reduce image size by ~230MB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18973) Clean yum cache after installing which in hadoop2 docker image
[
https://issues.apache.org/jira/browse/HADOOP-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785999#comment-17785999
]
ASF GitHub Bot commented on HADOOP-18973:
-
adoroszlai opened a new pull request, #6273:
URL: https://github.com/apache/hadoop/pull/6273
## What changes were proposed in this pull request?
* Clean yum cache after installing `which` to reduce image size by ~230MB.
* Reorder installation of Hadoop and `which`. Hadoop release bits are not
going to change, but we may want to install other packages in the image. The
updated order reduces the number of image layers that have to be downloaded
after such future change.
https://issues.apache.org/jira/browse/HADOOP-18973
## How was this patch tested?
```
$ docker pull apache/hadoop:2
$ docker run -it --rm apache/hadoop:2 du -sh /var/cache/yum
234M /var/cache/yum
$ ./build.sh
...
Successfully built c88de0d4d5c2
Successfully tagged apache/hadoop:2
$ docker run -it --rm apache/hadoop:2 du -sh /var/cache/yum
64K /var/cache/yum
$ docker image ls | head -3
REPOSITORYTAG
IMAGE ID CREATED SIZE
apache/hadoop 2
c88de0d4d5c2 9 minutes ago1.18GB
apache/hadoop
ba716169c882 49 minutes ago 1.43GB
```
> Clean yum cache after installing which in hadoop2 docker image
> --
>
> Key: HADOOP-18973
> URL: https://issues.apache.org/jira/browse/HADOOP-18973
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
>
> Minor addendum to HADOOP-18970: clean yum cache after installing {{which}} to
> reduce image size by ~230MB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392945651
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -667,6 +700,9 @@ AsyncLoggerSet getLoggerSetForTests() {
@Override
public void doPreUpgrade() throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("doPreUpgrade() does not support enabling jn
maintenance mode");
Review Comment:
nit: doPreUpgrade() does not support enabling jn maintenance mode ->
doPreUpgrade() is not support while in jn maintenance mode
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -684,6 +720,9 @@ public void doPreUpgrade() throws IOException {
@Override
public void doUpgrade(Storage storage) throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("doUpgrade() does not support enabling jn
maintenance mode");
Review Comment:
same here.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##
@@ -6333,6 +6333,20 @@
+
+dfs.journalnode.maintenance.nodes
+
+
+ In the case of one out of three journal nodes being down, theoretically
the service can still
+ continue. However, in reality, the downed node may not recover quickly.
If the Namenode needs
+ to be restarted, it will try the downed journal node through the lengthy
RPC retry mechanism,
+ resulting in a long initialization time for the Namenode to provide
services. By adding the
+ downed journal node to the maintenance nodes, the initialization time of
the Namenode in such
+ scenarios can be accelerated.
Review Comment:
nit
->
In the case that one out of three journal nodes being down, theoretically
HDFS can still
function. However, in reality, the unavailable journal node may not
recover quickly. During this period, when we need to restart an Namenode, the
Namenode will try to connect to the unavailable journal node through the
lengthy RPC retry mechanism,
resulting in a long initialization time for the Namenode. By adding
these
unavailable journal nodes to the maintenance nodes, we will skip these
unavailable journal nodes during Namenode initialization and thus reduce
namenode startup time.
1-node example values: <>
2-node example values: <>
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java:
##
@@ -1137,4 +1139,24 @@ public void testAddTransferRateMetricForInvalidValue() {
DFSUtil.addTransferRateMetric(mockMetrics, 100, 0);
verify(mockMetrics, times(0)).addReadTransferRate(anyLong());
}
+
+ @Test
+ public void testGetHostSet() {
+String[] testAddrs = new String[] {NS1_NN_ADDR, NS1_NN1_ADDR};
Review Comment:
this test case is a bit confusing. Can we just use
"unreachable-host1.com:9000" instead?
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1171,6 +1176,21 @@ public void testSelectViaRpcAfterJNRestart() throws
Exception {
}
}
+ /**
+ * Tests to throw an exception if the jn maintenance nodes exceeds half of
the journalnode number.
+ */
+ @Test
+ public void testJNMaintenanceListViaRpcTwoJNsError() throws Exception {
+StringJoiner maintenanceListBuff = new StringJoiner(",");
+for (int i = 0; i < 2; i++) {
+ maintenanceListBuff.add(
+
NetUtils.getHostPortString(cluster.getJournalNode(i).getBoundIpcAddress()));
+}
+this.conf.set(DFS_JOURNALNODE_MAINTENANCE_NODES_KEY,
maintenanceListBuff.toString());
+assertThrows(IllegalArgumentException.class, () -> createSpyingQJM());
+ }
+
+
Review Comment:
I assume we also need a test case to to verify maintenance journal nodes are
indeed excluded? total JN=3, with 1 is excluded. after we initialize, verify
only 2 JNs are included.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -701,6 +740,9 @@ public void doUpgrade(Storage storage) throws IOException {
@Override
public void doFinalize() throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("doFinalize() does not support enabling jn
maintenance mode");
Review Comment:
same here. and for others functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---
[PR] HADOOP-18973. Clean yum cache after installing which in hadoop2 docker image [hadoop]
adoroszlai opened a new pull request, #6273: URL: https://github.com/apache/hadoop/pull/6273 ## What changes were proposed in this pull request? * Clean yum cache after installing `which` to reduce image size by ~230MB. * Reorder installation of Hadoop and `which`. Hadoop release bits are not going to change, but we may want to install other packages in the image. The updated order reduces the number of image layers that have to be downloaded after such future change. https://issues.apache.org/jira/browse/HADOOP-18973 ## How was this patch tested? ``` $ docker pull apache/hadoop:2 $ docker run -it --rm apache/hadoop:2 du -sh /var/cache/yum 234M /var/cache/yum $ ./build.sh ... Successfully built c88de0d4d5c2 Successfully tagged apache/hadoop:2 $ docker run -it --rm apache/hadoop:2 du -sh /var/cache/yum 64K /var/cache/yum $ docker image ls | head -3 REPOSITORYTAG IMAGE ID CREATED SIZE apache/hadoop 2 c88de0d4d5c2 9 minutes ago1.18GB apache/hadoop ba716169c882 49 minutes ago 1.43GB ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17221. Add NNThroughputBenchmark params to specify subcluster lo… [hadoop]
hadoop-yetus commented on PR #6230:
URL: https://github.com/apache/hadoop/pull/6230#issuecomment-1810770270
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 26s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 13m 59s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6230/2/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 123 unchanged - 3
fixed = 123 total (was 126) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 213m 49s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 333m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6230/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6230 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 92774763447d 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a702199b726fdb3f9a81c569b40643605c2199d4 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6230/2/testReport/ |
| Max. process+thread count | 3277 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6230/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the messag
[jira] [Created] (HADOOP-18973) Clean yum cache after installing which in hadoop2 docker image
Attila Doroszlai created HADOOP-18973:
-
Summary: Clean yum cache after installing which in hadoop2 docker
image
Key: HADOOP-18973
URL: https://issues.apache.org/jira/browse/HADOOP-18973
Project: Hadoop Common
Issue Type: Improvement
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Minor addendum to HADOOP-18970: clean yum cache after installing {{which}} to
reduce image size by ~230MB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785990#comment-17785990
]
ASF GitHub Bot commented on HADOOP-18910:
-
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810730410
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 27s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 46s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 1s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 125m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux a55e0fa5c1ba 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ff2d5abbf873d3b9bba7aebcb53ad02c913d3c82 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/17/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-m
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810730410
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 27s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 46s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 1s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 125m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux a55e0fa5c1ba 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ff2d5abbf873d3b9bba7aebcb53ad02c913d3c82 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/17/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392944771
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -406,21 +421,39 @@ private void recoverUnclosedSegment(long segmentTxId)
throws IOException {
logToSync.getStartTxId(),
logToSync.getEndTxId()));
}
-
- static List createLoggers(Configuration conf,
+
+ List createLoggers(Configuration conf,
+ URI uri,
+ NamespaceInfo nsInfo,
+ AsyncLogger.Factory factory,
+ String nameServiceId)
+ throws IOException {
+String[] skipNodesHostPort = conf.getTrimmedStrings(
+DFS_JOURNALNODE_MAINTENANCE_NODES_KEY,
DFS_JOURNALNODE_MAINTENANCE_NODES_DEFAULT);
+return createLoggers(conf, uri, nsInfo, factory, nameServiceId,
skipNodesHostPort);
+ }
+
+ private List createLoggers(Configuration conf,
URI uri,
NamespaceInfo nsInfo,
AsyncLogger.Factory factory,
- String nameServiceId)
+ String nameServiceId,
+ String[] skipNodesHostPort)
throws IOException {
List ret = Lists.newArrayList();
List addrs = Util.getAddressesList(uri, conf);
if (addrs.size() % 2 == 0) {
LOG.warn("Quorum journal URI '" + uri + "' has an even number " +
"of Journal Nodes specified. This is not recommended!");
}
+setQuorumJournalCount(addrs.size());
+HostSet skipSet = DFSUtil.getHostSet(skipNodesHostPort);
String jid = parseJournalId(uri);
for (InetSocketAddress addr : addrs) {
+ if(skipSet.match(addr)) {
+LOG.info("The node {} is a maintenance node and will skip
initialization.", addr);
Review Comment:
nit: "will skip initialization" -> "will be skipped"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392943712
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -250,6 +262,9 @@ Map
createNewUniqueEpoch()
@Override
public void format(NamespaceInfo nsInfo, boolean force) throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("format() does not support enabling jn maintenance
mode");
Review Comment:
nit:
`isEnableJnMaintenance(`) -> `isJNMainteanceEnabled()` or should it actually
be `IsJNInMaintenanceMode()` ?
"format() does not support enabling jn maintenance mode" -> "Formatting a
journal node is not supported while in JN maintenance mode"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
xinglin commented on code in PR #6183: URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392934193 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java: ## @@ -144,7 +149,14 @@ public QuorumJournalManager(Configuration conf, this.uri = uri; this.nsInfo = nsInfo; this.nameServiceId = nameServiceId; -this.loggers = new AsyncLoggerSet(createLoggers(loggerFactory)); +this.loggers = new AsyncLoggerSet(createLoggers(loggerFactory), this.quorumJournalCount); Review Comment: this.quorumJournalCount is set inside createLoggers(). Can we separate them as two separate steps, like the following? ``` // Keep this in comment: // createLoggers() will set quorumJournalCount to total number of journal nodes while return a list of healthy/good journal nodes. List loggers = createLoggers(loggerFactory); this.loggers = new AsyncLoggerSet(loggers, this.quorumJournalCount); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18970) Upgrade hadoop2 docker scripts to latest 2.10.2
[
https://issues.apache.org/jira/browse/HADOOP-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HADOOP-18970:
--
Resolution: Done
Status: Resolved (was: Patch Available)
> Upgrade hadoop2 docker scripts to latest 2.10.2
> ---
>
> Key: HADOOP-18970
> URL: https://issues.apache.org/jira/browse/HADOOP-18970
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Apply enhancements from {{docker-hadoop-3}} branch, and upgrade to latest
> Hadoop 2 release: 2.10.2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785977#comment-17785977
]
ASF GitHub Bot commented on HADOOP-18910:
-
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810687703
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 0s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 18s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 125m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 05a1b198dbf2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 45ccd087497855112c8cdb2e1bb8f880b73ba038 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/16/testReport/ |
| Max. process+thread count | 727 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-m
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810687703
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 55s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 0s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 18s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 125m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 05a1b198dbf2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 45ccd087497855112c8cdb2e1bb8f880b73ba038 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/16/testReport/ |
| Max. process+thread count | 727 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392918855
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1982,4 +1983,32 @@ public static void addTransferRateMetric(final
DataNodeMetrics metrics, final lo
LOG.warn("Unexpected value for data transfer bytes={} duration={}",
read, duration);
}
}
+
+ /**
+ * Construct a HostSet from an array of "ip:port" strings.
+ * @param nodesHostPort ip port string array.
+ * @return HostSet of InetSocketAddress.
+ */
+ public static HostSet getHostSet(String[] nodesHostPort) {
+HostSet retSet = new HostSet();
+for (String hostPort : nodesHostPort) {
+ try {
+URI uri = new URI("dummy", hostPort, null, null, null);
+int port = uri.getPort();
+if (port == -1 || port == 0) {
Review Comment:
port=0 is valid. I guess we should probably just check for -1 here?
```
/**
* Returns the port number of this URI.
*
* The port component of a URI, if defined, is a non-negative
* integer.
*
* @return The port component of this URI,
* or {@code -1} if the port is undefined
*/
public int getPort() {
return port;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18970. Upgrade hadoop2 docker scripts to latest 2.10.2 [hadoop]
adoroszlai commented on PR #6269: URL: https://github.com/apache/hadoop/pull/6269#issuecomment-1810617071 Thanks @ayushtkn for the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18970. Upgrade hadoop2 docker scripts to latest 2.10.2 [hadoop]
adoroszlai merged PR #6269: URL: https://github.com/apache/hadoop/pull/6269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17218. NameNode should process time out excess redundancy blocks [hadoop]
hadoop-yetus commented on PR #6176:
URL: https://github.com/apache/hadoop/pull/6176#issuecomment-1810518324
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 34s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 185m 50s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 282m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6176 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux e449eb1c7786 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 59e36ccd62f34783f9141ddd343aef6b774272de |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/11/testReport/ |
| Max. process+thread count | 4426 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.
[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785937#comment-17785937 ] ASF GitHub Bot commented on HADOOP-18708: - steveloughran commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1810466203 ...oh and cloudstore storediag is some implicit validation. > AWS SDK V2 - Implement CSE > -- > > Key: HADOOP-18708 > URL: https://issues.apache.org/jira/browse/HADOOP-18708 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > S3 Encryption client for SDK V2 is now available, so add client side > encryption back in. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]
steveloughran commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1810466203 ...oh and cloudstore storediag is some implicit validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1810342572
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 31s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/9/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 207m 31s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 305m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 8d6598f059e3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 711aa42d79832c191747b2be15e2985577a1d358 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-ope
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1810339618
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/8/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 206m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 305m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestDFSInputStreamBlockLocations |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux be68a6bc0f70 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 711aa42d79832c191747b2be15e2985577a1d358 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
|
Re: [PR] HDFS-17121. BPServiceActor to provide new thread to handle FBR [hadoop]
hadoop-yetus commented on PR #6005:
URL: https://github.com/apache/hadoop/pull/6005#issuecomment-1810271066
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 56s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 57s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6005/9/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 264 unchanged
- 0 fixed = 268 total (was 264) |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 265m 52s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 418m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6005/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux bca88202f866 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 43b514c36e2ccb5891899a08eb60c17cca67efa7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6005/9/testReport/ |
| Max. process+thread count | 3071 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6005/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785890#comment-17785890
]
ASF GitHub Bot commented on HADOOP-18910:
-
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810198092
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 53s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/14/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 7 unchanged - 0
fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 141m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 5bc3c2154a64 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7caa63f505acb7ca66c94e5c2e95452b2134d07e |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apac
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810198092
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 53s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/14/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 2 new + 7 unchanged - 0
fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 3s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 141m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 5bc3c2154a64 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7caa63f505acb7ca66c94e5c2e95452b2134d07e |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/14/testReport/ |
| Max. process+thread count | 556 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-mult
[jira] [Commented] (HADOOP-18965) ITestS3AHugeFilesEncryption failure
[
https://issues.apache.org/jira/browse/HADOOP-18965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785885#comment-17785885
]
ASF GitHub Bot commented on HADOOP-18965:
-
hadoop-yetus commented on PR #6261:
URL: https://github.com/apache/hadoop/pull/6261#issuecomment-1810178210
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 8m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 17s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 90m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6261 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ed16bcfcde7b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 584c0a509fb92f9a8bf369c44814a7e46621d5c5 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/console |
|
Re: [PR] HADOOP-18965. ITestS3AHugeFilesEncryption failure [hadoop]
hadoop-yetus commented on PR #6261:
URL: https://github.com/apache/hadoop/pull/6261#issuecomment-1810178210
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 8m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 17s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt)
| hadoop-tools/hadoop-aws: The patch generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 12s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 90m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6261 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ed16bcfcde7b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 584c0a509fb92f9a8bf369c44814a7e46621d5c5 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6261/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the me
[jira] [Commented] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785884#comment-17785884 ] ASF GitHub Bot commented on HADOOP-18957: - pjfanning commented on PR #6231: URL: https://github.com/apache/hadoop/pull/6231#issuecomment-1810173668 > Quick Pass & mostly looks good, can we have some restrictions to prevent future usage of these? Something like this or better? https://github.com/apache/hadoop/blob/trunk/pom.xml#L194 Good idea. I've committed a banned import for the Guava Charsets class. > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18957. Use StandardCharsets.UTF_8 [hadoop]
pjfanning commented on PR #6231: URL: https://github.com/apache/hadoop/pull/6231#issuecomment-1810173668 > Quick Pass & mostly looks good, can we have some restrictions to prevent future usage of these? Something like this or better? https://github.com/apache/hadoop/blob/trunk/pom.xml#L194 Good idea. I've committed a banned import for the Guava Charsets class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785875#comment-17785875
]
ASF GitHub Bot commented on HADOOP-18910:
-
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810129241
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 29m 55s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 2s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 80m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 1c600ab6efd6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5e0ae6d47cbdae32c91e5706c2ca71ceac0ab8a |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/testReport/ |
| Max. process+thread count | 556 (vs. ulimit of 5500) |
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
hadoop-yetus commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1810129241
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 29m 55s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 2s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 80m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 1c600ab6efd6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5e0ae6d47cbdae32c91e5706c2ca71ceac0ab8a |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/testReport/ |
| Max. process+thread count | 556 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6069/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0
[jira] [Commented] (HADOOP-18971) ABFS: Enable Footer Read Optimizations with Appropriate Footer Read Buffer Size
[
https://issues.apache.org/jira/browse/HADOOP-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785857#comment-17785857
]
ASF GitHub Bot commented on HADOOP-18971:
-
steveloughran commented on code in PR #6270:
URL: https://github.com/apache/hadoop/pull/6270#discussion_r1392446821
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
Review Comment:
this is roughly the same for ORC, isn't it?
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
+ * First the read is done of last 8 bytes of parquet file to get the postion
of metadta and next read
+ * is done for reading that metadata. With this optimizations these two
reads can be combined into 1.
+ * Value: {@value}
+ */
public static final String AZURE_READ_OPTIMIZE_FOOTER_READ =
"fs.azure.read.optimizefooterread";
+ /**
+ * In case of footer reads it was not required to read full buffer size.
+ * Most of the metadata information required was within 256KB and it will be
more performant to read lesser.
Review Comment:
"read less"
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
+ * First the read is done of last 8 bytes of parquet file to get the postion
of metadta and next read
+ * is done for reading that metadata. With this optimizations these two
reads can be combined into 1.
Review Comment:
nit "optimization"
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##
@@ -59,7 +59,8 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
false;
public static final int DEFAULT_READ_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
public static final boolean DEFAULT_READ_SMALL_FILES_COMPLETELY = false;
- public static final boolean DEFAULT_OPTIMIZE_FOOTER_READ = false;
+ public static final boolean DEFAULT_OPTIMIZE_FOOTER_READ = true;
+ public static final int DEFAULT_FOOTER_READ_BUFFER_SIZE = 512 * ONE_KB;
Review Comment:
this is 512k; docs in file above say 265K.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
+ * First the read is done of last 8 bytes of parquet file to get the postion
of metadta and next read
+ * is done for reading that metadata. With this optimizations these two
reads can be combined into 1.
+ * Value: {@value}
+ */
public static final String AZURE_READ_OPTIMIZE_FOOTER_READ =
"fs.azure.read.optimizefooterread";
Review Comment:
i'd prefer "." between words, but it's too late now
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java:
##
@@ -190,7 +193,8 @@ private void seekReadAndTest(final FileSystem f
Re: [PR] HADOOP-18971: [ABFS] Enable Footer Read Optimizations with Appropriate Footer Read Buffer Size [hadoop]
steveloughran commented on code in PR #6270:
URL: https://github.com/apache/hadoop/pull/6270#discussion_r1392446821
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
Review Comment:
this is roughly the same for ORC, isn't it?
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
+ * First the read is done of last 8 bytes of parquet file to get the postion
of metadta and next read
+ * is done for reading that metadata. With this optimizations these two
reads can be combined into 1.
+ * Value: {@value}
+ */
public static final String AZURE_READ_OPTIMIZE_FOOTER_READ =
"fs.azure.read.optimizefooterread";
+ /**
+ * In case of footer reads it was not required to read full buffer size.
+ * Most of the metadata information required was within 256KB and it will be
more performant to read lesser.
Review Comment:
"read less"
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
+ * First the read is done of last 8 bytes of parquet file to get the postion
of metadta and next read
+ * is done for reading that metadata. With this optimizations these two
reads can be combined into 1.
Review Comment:
nit "optimization"
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##
@@ -59,7 +59,8 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
false;
public static final int DEFAULT_READ_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
public static final boolean DEFAULT_READ_SMALL_FILES_COMPLETELY = false;
- public static final boolean DEFAULT_OPTIMIZE_FOOTER_READ = false;
+ public static final boolean DEFAULT_OPTIMIZE_FOOTER_READ = true;
+ public static final int DEFAULT_FOOTER_READ_BUFFER_SIZE = 512 * ONE_KB;
Review Comment:
this is 512k; docs in file above say 265K.
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##
@@ -104,7 +104,20 @@ public final class ConfigurationKeys {
public static final String AZURE_ENABLE_SMALL_WRITE_OPTIMIZATION =
"fs.azure.write.enableappendwithflush";
public static final String AZURE_READ_BUFFER_SIZE =
"fs.azure.read.request.size";
public static final String AZURE_READ_SMALL_FILES_COMPLETELY =
"fs.azure.read.smallfilescompletely";
+ /**
+ * When parquet files are read, first few read are metadata reads before
reading the actual data.
+ * First the read is done of last 8 bytes of parquet file to get the postion
of metadta and next read
+ * is done for reading that metadata. With this optimizations these two
reads can be combined into 1.
+ * Value: {@value}
+ */
public static final String AZURE_READ_OPTIMIZE_FOOTER_READ =
"fs.azure.read.optimizefooterread";
Review Comment:
i'd prefer "." between words, but it's too late now
##
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java:
##
@@ -190,7 +193,8 @@ private void seekReadAndTest(final FileSystem fs, final
Path testFilePath,
try (FSDataInputStream iStream = fs.open(testFilePath)) {
AbfsInputStream abfsInputStream = (AbfsInputStream) iStream
.getWrappedStream();
- long bufferSize = abfsInputStream.getBufferSize();
+ long
[jira] [Commented] (HADOOP-18972) Bug in SaslPropertiesResolver allows mutation of internal state
[
https://issues.apache.org/jira/browse/HADOOP-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785852#comment-17785852
]
ASF GitHub Bot commented on HADOOP-18972:
-
steveloughran commented on code in PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#discussion_r139243
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslPropertiesResolver.java:
##
@@ -82,7 +83,7 @@ public Configuration getConf() {
* @return sasl Properties
*/
public Map getDefaultProperties() {
-return properties;
+return new TreeMap<>(properties);
Review Comment:
I'd propose having a private cloneProperties() method used in all calls,
which includes a `requireNonNull()` on the properties...it is null until
setConf() is called.
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
Review Comment:
subclass AbstractHadoopTestBase; override it's setup/teardown methods, which
do things like set the thread name
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
+
+ private static final SaslRpcServer.QualityOfProtection PRIVACY_QOP =
SaslRpcServer.QualityOfProtection.PRIVACY;
+ private static final SaslRpcServer.QualityOfProtection AUTHENTICATION_QOP =
SaslRpcServer.QualityOfProtection.AUTHENTICATION;
+ private static final InetAddress LOCALHOST = new
InetSocketAddress("127.0.0.1", 1).getAddress();
+
+ private SaslPropertiesResolver resolver;
+
+ @Before
+ public void setup() {
+Configuration conf = new Configuration();
+conf.set(HADOOP_RPC_PROTECTION, "privacy");
+resolver = new SaslPropertiesResolver();
+resolver.setConf(conf);
+ }
+
+ @Test
+ public void testResolverDoesNotMutate() {
+assertEquals(PRIVACY_QOP.getSaslQop(),
resolver.getDefaultProperties().get(Sasl.QOP));
Review Comment:
lot of duplication here;
is it possible to factor these out into new assertions? at the very least
some assertPrivacyQop() which is called all the way down.
> Bug in SaslPropertiesResolver allows mutation of internal state
> ---
>
> Key: HADOOP-18972
> URL: https://issues.apache.org/jira/browse/HADOOP-18972
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Charles Connell
>Priority: Minor
> Labels: pull-request-available
>
> When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a
> SASL properties map to do a handshake, they call
> {{SaslPropertiesResolver#getServerProperties()}} or
> {{SaslPropertiesResolver#getClientProperties()}}, and they get back a
> {{Map}}. Every call gets the same {{Map}} object back, and
> then the callers sometimes call
> [put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385]
> on it. This means that future users of {{SaslPropertiesResolver}} get back
> the wrong information.
> I propose that {{SaslPropertiesResolver}} should pass a copy of its internal
> map, so that users can safety modify them.
> I discovered this problem in my company's testing environment as we began to
> enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes
> were using {{IngressPortBasedResolver}} to give out block tokens with
> different QOPs depending on the port used. The our HDFS client applications
> became unable to read or write to HDFS because they could not find a QOP in
> common with the DataNodes during SASL handshake. With multiple threads
> executing SASL handshakes at the same time, the properties map used in
> {{SaslDataTransferServer}} in a DataNode could be clobbered during usage,
> since the same map was used by all threads. Also, future client
Re: [PR] HADOOP-18972: Copy SASL properties map when returning from SaslPropertiesResolver [hadoop]
steveloughran commented on code in PR #6272:
URL: https://github.com/apache/hadoop/pull/6272#discussion_r139243
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslPropertiesResolver.java:
##
@@ -82,7 +83,7 @@ public Configuration getConf() {
* @return sasl Properties
*/
public Map getDefaultProperties() {
-return properties;
+return new TreeMap<>(properties);
Review Comment:
I'd propose having a private cloneProperties() method used in all calls,
which includes a `requireNonNull()` on the properties...it is null until
setConf() is called.
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
Review Comment:
subclass AbstractHadoopTestBase; override it's setup/teardown methods, which
do things like set the thread name
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSaslPropertiesResolver.java:
##
@@ -0,0 +1,49 @@
+package org.apache.hadoop.security;
+
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import javax.security.sasl.Sasl;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_RPC_PROTECTION;
+import static org.junit.Assert.assertEquals;
+
+public class TestSaslPropertiesResolver {
+
+ private static final SaslRpcServer.QualityOfProtection PRIVACY_QOP =
SaslRpcServer.QualityOfProtection.PRIVACY;
+ private static final SaslRpcServer.QualityOfProtection AUTHENTICATION_QOP =
SaslRpcServer.QualityOfProtection.AUTHENTICATION;
+ private static final InetAddress LOCALHOST = new
InetSocketAddress("127.0.0.1", 1).getAddress();
+
+ private SaslPropertiesResolver resolver;
+
+ @Before
+ public void setup() {
+Configuration conf = new Configuration();
+conf.set(HADOOP_RPC_PROTECTION, "privacy");
+resolver = new SaslPropertiesResolver();
+resolver.setConf(conf);
+ }
+
+ @Test
+ public void testResolverDoesNotMutate() {
+assertEquals(PRIVACY_QOP.getSaslQop(),
resolver.getDefaultProperties().get(Sasl.QOP));
Review Comment:
lot of duplication here;
is it possible to factor these out into new assertions? at the very least
some assertPrivacyQop() which is called all the way down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785843#comment-17785843
]
ASF GitHub Bot commented on HADOOP-18910:
-
anujmodi2021 commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1809989042
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[WARNING] Tests run: 595, Failures: 0, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
Time taken: 26 mins 28 secs.
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[ https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785842#comment-17785842 ] ASF GitHub Bot commented on HADOOP-18910: - anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1809988857 Regarding the exception handling, I am in agreement to with what @steveloughran has suggested. The whole catch block will work only for the case of AbfsRestOperationException and will blow up if it somehow ends up being not AbfsRestOperationException. It's better to catch the AbfsRestOperationException only for that processing and handle AzureBlobFileSystemEception (which are not AbfsRestOperationException) separately. I have made code changes in this line. Let me know if this looks good @saxenapranav Thanks for review > ABFS: Adding Support for MD5 Hash based integrity verification of the request > content during transport > --- > > Key: HADOOP-18910 > URL: https://issues.apache.org/jira/browse/HADOOP-18910 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Azure Storage Supports Content-MD5 Request Headers in Both Read and Append > APIs. > Read: [Path - Read - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read] > Append: [Path - Update - REST API (Azure Storage Services) | Microsoft > Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update] > This change is to make client-side changes to support them. In Read request, > we will send the appropriate header in response to which server will return > the MD5 Hash of the data it sends back. On Client we will tally this with the > MD5 hash computed from the data received. > In Append request, we will compute the MD5 Hash of the data that we are > sending to the server and specify that in appropriate header. Server on > finding that header will tally this with the MD5 hash it will compute on the > data received. > This whole Checksum Validation Support is guarded behind a config, Config is > by default disabled because with the use of "https" integrity of data is > preserved anyways. This is introduced as an additional data integrity check > which will have a performance impact as well. > Users can decide if they want to enable this or not by setting the following > config to *"true"* or *"false"* respectively. *Config: > "fs.azure.enable.checksum.validation"* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1809988857 Regarding the exception handling, I am in agreement to with what @steveloughran has suggested. The whole catch block will work only for the case of AbfsRestOperationException and will blow up if it somehow ends up being not AbfsRestOperationException. It's better to catch the AbfsRestOperationException only for that processing and handle AzureBlobFileSystemEception (which are not AbfsRestOperationException) separately. I have made code changes in this line. Let me know if this looks good @saxenapranav Thanks for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1809989042
AGGREGATED TEST RESULT
HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[WARNING] Tests run: 595, Failures: 0, Errors: 0, Skipped: 277
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
Time taken: 26 mins 28 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11498 backport jettison exclusions [hadoop]
pjfanning commented on PR #6063: URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1809987070 In trunk, hadoop-client-runtime does not have the jettison classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785840#comment-17785840
]
ASF GitHub Bot commented on HADOOP-18910:
-
anujmodi2021 commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1392403224
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -879,9 +880,8 @@ private boolean checkUserError(int responseStatusCode) {
* @return boolean whether exception is due to MD5Mismatch or not
*/
protected boolean isMd5ChecksumError(final AzureBlobFileSystemException e) {
-return ((AbfsRestOperationException) e).getStatusCode()
-== HttpURLConnection.HTTP_BAD_REQUEST
-&& e.getMessage().contains(MD5_ERROR_SERVER_MESSAGE);
+AzureServiceErrorCode storageErrorCode = ((AbfsRestOperationException)
e).getErrorCode();
Review Comment:
In line with steve's suggestion here.
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport
[
https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785839#comment-17785839
]
ASF GitHub Bot commented on HADOOP-18910:
-
anujmodi2021 commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1392402611
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -798,6 +806,11 @@ public AbfsRestOperation append(final String path, final
byte[] buffer,
if (!op.hasResult()) {
throw e;
}
+
Review Comment:
Inline with this
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -861,6 +874,16 @@ private boolean checkUserError(int responseStatusCode) {
&& responseStatusCode < HttpURLConnection.HTTP_INTERNAL_ERROR);
}
+ /**
+ * To check if the failure exception returned by server is due to MD5
Mismatch
+ * @param e Exception returned by AbfsRestOperation
+ * @return boolean whether exception is due to MD5Mismatch or not
+ */
+ protected boolean isMd5ChecksumError(final AzureBlobFileSystemException e) {
Review Comment:
Taken
> ABFS: Adding Support for MD5 Hash based integrity verification of the request
> content during transport
> ---
>
> Key: HADOOP-18910
> URL: https://issues.apache.org/jira/browse/HADOOP-18910
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Azure Storage Supports Content-MD5 Request Headers in Both Read and Append
> APIs.
> Read: [Path - Read - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read]
> Append: [Path - Update - REST API (Azure Storage Services) | Microsoft
> Learn|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update]
> This change is to make client-side changes to support them. In Read request,
> we will send the appropriate header in response to which server will return
> the MD5 Hash of the data it sends back. On Client we will tally this with the
> MD5 hash computed from the data received.
> In Append request, we will compute the MD5 Hash of the data that we are
> sending to the server and specify that in appropriate header. Server on
> finding that header will tally this with the MD5 hash it will compute on the
> data received.
> This whole Checksum Validation Support is guarded behind a config, Config is
> by default disabled because with the use of "https" integrity of data is
> preserved anyways. This is introduced as an additional data integrity check
> which will have a performance impact as well.
> Users can decide if they want to enable this or not by setting the following
> config to *"true"* or *"false"* respectively. *Config:
> "fs.azure.enable.checksum.validation"*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1392403224
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -879,9 +880,8 @@ private boolean checkUserError(int responseStatusCode) {
* @return boolean whether exception is due to MD5Mismatch or not
*/
protected boolean isMd5ChecksumError(final AzureBlobFileSystemException e) {
-return ((AbfsRestOperationException) e).getStatusCode()
-== HttpURLConnection.HTTP_BAD_REQUEST
-&& e.getMessage().contains(MD5_ERROR_SERVER_MESSAGE);
+AzureServiceErrorCode storageErrorCode = ((AbfsRestOperationException)
e).getErrorCode();
Review Comment:
In line with steve's suggestion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]
anujmodi2021 commented on code in PR #6069:
URL: https://github.com/apache/hadoop/pull/6069#discussion_r1392402611
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -798,6 +806,11 @@ public AbfsRestOperation append(final String path, final
byte[] buffer,
if (!op.hasResult()) {
throw e;
}
+
Review Comment:
Inline with this
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##
@@ -861,6 +874,16 @@ private boolean checkUserError(int responseStatusCode) {
&& responseStatusCode < HttpURLConnection.HTTP_INTERNAL_ERROR);
}
+ /**
+ * To check if the failure exception returned by server is due to MD5
Mismatch
+ * @param e Exception returned by AbfsRestOperation
+ * @return boolean whether exception is due to MD5Mismatch or not
+ */
+ protected boolean isMd5ChecksumError(final AzureBlobFileSystemException e) {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18830) Cut S3 Select
[
https://issues.apache.org/jira/browse/HADOOP-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785832#comment-17785832
]
ASF GitHub Bot commented on HADOOP-18830:
-
hadoop-yetus commented on PR #6144:
URL: https://github.com/apache/hadoop/pull/6144#issuecomment-1809917406
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 9m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 13 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 24s | | trunk passed |
| +1 :green_heart: | compile | 11m 28s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 13m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 47s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 18s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 37m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 17m 46s | | the patch passed |
| +1 :green_heart: | compile | 8m 59s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 59s | | the patch passed |
| +1 :green_heart: | compile | 8m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 8m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 17s | | root: The patch generated
0 new + 5 unchanged - 8 fixed = 5 total (was 13) |
| +1 :green_heart: | mvnsite | 8m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 5m 38s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 56s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 17s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 36m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 648m 0s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 880m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6144/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6144 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint shellcheck shelldocs |
| uname | Linux 1086dca886f0 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 88f202d35a049030876dabf54ac2f252b85ee689 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1
Re: [PR] HADOOP-18830. Cut S3 Select [hadoop]
hadoop-yetus commented on PR #6144:
URL: https://github.com/apache/hadoop/pull/6144#issuecomment-1809917406
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 9m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 13 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 24s | | trunk passed |
| +1 :green_heart: | compile | 11m 28s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 13m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 47s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 18s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 37m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 17m 46s | | the patch passed |
| +1 :green_heart: | compile | 8m 59s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 59s | | the patch passed |
| +1 :green_heart: | compile | 8m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 8m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 17s | | root: The patch generated
0 new + 5 unchanged - 8 fixed = 5 total (was 13) |
| +1 :green_heart: | mvnsite | 8m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 5m 38s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 56s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 17s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 36m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 648m 0s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 880m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6144/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6144 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint shellcheck shelldocs |
| uname | Linux 1086dca886f0 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 88f202d35a049030876dabf54ac2f252b85ee689 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6144/5/testReport/ |
| Max. process+thread count | 3372 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws . U: .
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1809879806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 36s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 396m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux cabf21baac90 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5f415d73c212b0313a558ed06457c7e09a1b0e09 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/testReport/ |
| Max. process+thread count | 2380 (vs. ul
Re: [PR] YARN-11498 backport jettison exclusions [hadoop]
pjfanning commented on PR #6063: URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1809859313 The build issue is this: ``` Duplicate classes found: Found in: org.apache.hadoop:hadoop-client-minicluster:jar:3.3.9-SNAPSHOT:compile org.apache.hadoop:hadoop-client-runtime:jar:3.3.9-SNAPSHOT:compile Duplicate classes: org/apache/hadoop/shaded/org/codehaus/jettison/json/JSONWriter.class org/apache/hadoop/shaded/org/codehaus/jettison/json/JSONException.class ... ``` I'm not yet sure why this happens in branch-3.3 but not in trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. [hadoop]
anujmodi2021 commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1809858121 Thanks for your approval @steveloughran I have resolved the conflicts and raised the iteration. Please merge this PR whenever you can. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
