[jira] [Created] (HADOOP-17583) Enable shelldoc check in GitHub PR
Akira Ajisaka created HADOOP-17583: -- Summary: Enable shelldoc check in GitHub PR Key: HADOOP-17583 URL: https://issues.apache.org/jira/browse/HADOOP-17583 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Akira Ajisaka After HADOOP-17570, we can enable shelldoc check again because the commit hash of Yetus includes YETUS-1099. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17582) Replace GitHub App Token with GitHub OAuth token
[ https://issues.apache.org/jira/browse/HADOOP-17582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17582: --- Target Version/s: 3.3.1, 3.4.0, 3.2.3 > Replace GitHub App Token with GitHub OAuth token > > > Key: HADOOP-17582 > URL: https://issues.apache.org/jira/browse/HADOOP-17582 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > > GitHub App Token expires within 1 hour, so Yetus fails to write GitHub > comments in most cases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17582) Replace GitHub App Token with GitHub OAuth token
[ https://issues.apache.org/jira/browse/HADOOP-17582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300109#comment-17300109 ] Akira Ajisaka commented on HADOOP-17582: PR: https://github.com/apache/hadoop/pull/2766 > Replace GitHub App Token with GitHub OAuth token > > > Key: HADOOP-17582 > URL: https://issues.apache.org/jira/browse/HADOOP-17582 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > > GitHub App Token expires within 1 hour, so Yetus fails to write GitHub > comments in most cases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?focusedWorklogId=565157&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565157 ] ASF GitHub Bot logged work on HADOOP-17570: --- Author: ASF GitHub Bot Created on: 12/Mar/21 07:32 Start Date: 12/Mar/21 07:32 Worklog Time Spent: 10m Work Description: aajisaka closed pull request #2765: URL: https://github.com/apache/hadoop/pull/2765 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565157) Time Spent: 2h 50m (was: 2h 40m) > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka closed pull request #2765: Test PR to check HADOOP-17570
aajisaka closed pull request #2765: URL: https://github.com/apache/hadoop/pull/2765 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16748) Migrate to Python 3 and upgrade Yetus to 0.13.0
[ https://issues.apache.org/jira/browse/HADOOP-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300106#comment-17300106 ] Akira Ajisaka commented on HADOOP-16748: GitHub OAuth token seems good: [https://github.com/apache/hadoop/pull/2766] Filed HADOOP-17582 to fix. > Migrate to Python 3 and upgrade Yetus to 0.13.0 > --- > > Key: HADOOP-16748 > URL: https://issues.apache.org/jira/browse/HADOOP-16748 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17582) Replace GitHub App Token with GitHub OAuth token
Akira Ajisaka created HADOOP-17582: -- Summary: Replace GitHub App Token with GitHub OAuth token Key: HADOOP-17582 URL: https://issues.apache.org/jira/browse/HADOOP-17582 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Akira Ajisaka Assignee: Akira Ajisaka GitHub App Token expires within 1 hour, so Yetus fails to write GitHub comments in most cases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[
https://issues.apache.org/jira/browse/HADOOP-17570?focusedWorklogId=565148&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565148
]
ASF GitHub Bot logged work on HADOOP-17570:
---
Author: ASF GitHub Bot
Created on: 12/Mar/21 07:19
Start Date: 12/Mar/21 07:19
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2766:
URL: https://github.com/apache/hadoop/pull/2766#issuecomment-797289889
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 1m 22s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | shadedclient | 15m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 75m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2766/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2766 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs mvnsite
markdownlint |
| uname | Linux 594e246bee35 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 516b5b8d345e538269a0be9034c386ab6608fe61 |
| Max. process+thread count | 612 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2766/2/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 565148)
Time Spent: 2h 40m (was: 2.5h)
> Apply YETUS-1102 to re-enable GitHub comments
> -
>
> Key: HADOOP-17570
> URL: https://issues.apache.org/jira/browse/HADOOP-17570
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Yetus 0.13.0 enabled updating GitHub status instead of commenting the report,
> however, the report comments are still useful for some cases. Let's apply
> YETUS-1102 to re-enable the comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2766: Test PR to check HADOOP-17570 with GitHub OAuth Token (instead of GitHub App Token)
hadoop-yetus commented on pull request #2766:
URL: https://github.com/apache/hadoop/pull/2766#issuecomment-797289889
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 1m 22s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | shadedclient | 15m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 75m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2766/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2766 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs mvnsite
markdownlint |
| uname | Linux 594e246bee35 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 516b5b8d345e538269a0be9034c386ab6608fe61 |
| Max. process+thread count | 612 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2766/2/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17402) Add GCS FS impl reference to core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-17402?focusedWorklogId=565132&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565132
]
ASF GitHub Bot logged work on HADOOP-17402:
---
Author: ASF GitHub Bot
Created on: 12/Mar/21 06:52
Start Date: 12/Mar/21 06:52
Worklog Time Spent: 10m
Work Description: ravwojdyla commented on a change in pull request #2638:
URL: https://github.com/apache/hadoop/pull/2638#discussion_r592946720
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
##
@@ -142,6 +141,7 @@ public void initializeMemberVariables() {
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.webhdfs.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.wasb.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.swift.impl");
+xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.gs.impl");
Review comment:
@steveloughran done, also fixed the `swift` order.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 565132)
Time Spent: 2h 40m (was: 2.5h)
> Add GCS FS impl reference to core-default.xml
> -
>
> Key: HADOOP-17402
> URL: https://issues.apache.org/jira/browse/HADOOP-17402
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Rafal Wojdyla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Akin to current S3 default configuration add GCS configuration, specifically
> to declare the GCS implementation. [GCS
> connector|https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage].
> Has this not been done since the GCS connector is not part of the hadoop/ASF
> codebase, or is there any other blocker?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ravwojdyla commented on a change in pull request #2638: HADOOP-17402. Add GCS config to the core-site
ravwojdyla commented on a change in pull request #2638:
URL: https://github.com/apache/hadoop/pull/2638#discussion_r592946720
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
##
@@ -142,6 +141,7 @@ public void initializeMemberVariables() {
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.webhdfs.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.wasb.impl");
xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.swift.impl");
+xmlPropsToSkipCompare.add("fs.viewfs.overload.scheme.target.gs.impl");
Review comment:
@steveloughran done, also fixed the `swift` order.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb commented on pull request #2767: HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist
vinayakumarb commented on pull request #2767: URL: https://github.com/apache/hadoop/pull/2767#issuecomment-797277002 Generated test code (for 2.5.0 protobuf) has been added for 'arm' since 2.5.0 protobuf executable is not available for aarch64 platform. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] vinayakumarb opened a new pull request #2767: HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist
vinayakumarb opened a new pull request #2767: URL: https://github.com/apache/hadoop/pull/2767 ## Existing ClientSide available Protobuf RPC Engines. * ProtobufRpcEngine: Supports existing implementations based in protobuf 2.5.0 in both client side and server side. No code changes required in downstreams use this. * ProtobufRpcEngine2: Uses shaded protobuf of 3.7.1 version and supports client side and server side implementations based on shaded protobuf 3.7.1 # Whats Changed? ## Server side Protobuf RPC Engines `ProtobufRpcEngine2` itself will handle both versions (2.5.0 and 3.7.1) of requests for RpcKind.PROTOCOL_BUFFERS. `ProtobufRpcEngine2` will handover the processing to Legacy `ProtobufRpcEngine` if server side protocol implemenation found to be using older version of protobuf (2.5.0). ## How conflict arised? Both `ProtobufRpcEngine` and `ProtobufRpcEngine2` tried to register their corresponding RPC `Server` implementations to handle requests of kind `RpcKind.PROTOCOL_BUFFERS`, although this registering is of no-use in client-JVM. Only Server side this registration is required. Now, since `ProtobufRpcEngine2` itself can handle both versions of protobuf requests at server side, registering only `ProtobufRpcEngine2` will be sufficient. So no conflict is raised for co-existence. ## How old clients (before 3.3.0) are affected.? Since client JVM side registering of `ProtobufRpcEngine` will not affect server-JVM (running with latest version) and server can efficiently handle both type of implementations, there will not be any issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17579) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300072#comment-17300072 ] Brahma Reddy Battula commented on HADOOP-17579: --- looks this is duplicate to HADOOP-17580..? > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17579 > URL: https://issues.apache.org/jira/browse/HADOOP-17579 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: wenzhen >Priority: Minor > Fix For: 3.3.0 > > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?focusedWorklogId=565123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565123 ] ASF GitHub Bot logged work on HADOOP-17570: --- Author: ASF GitHub Bot Created on: 12/Mar/21 05:54 Start Date: 12/Mar/21 05:54 Worklog Time Spent: 10m Work Description: aajisaka opened a new pull request #2766: URL: https://github.com/apache/hadoop/pull/2766 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565123) Time Spent: 2.5h (was: 2h 20m) > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #2766: Test PR to check HADOOP-17570 with GitHub OAuth Token (instead of GitHub App Token)
aajisaka opened a new pull request #2766: URL: https://github.com/apache/hadoop/pull/2766 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16748) Migrate to Python 3 and upgrade Yetus to 0.13.0
[ https://issues.apache.org/jira/browse/HADOOP-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300062#comment-17300062 ] Akira Ajisaka commented on HADOOP-16748: The GitHub App token expires in only 1 hour, and most of the builds are affected in Hadoop. [https://docs.github.com/en/developers/apps/differences-between-github-apps-and-oauth-apps#token-based-identification] I think there are 2 ways to fix it: * Add comments in GitHub Status recovery tool (YETUS-1059) * Use the GitHub OAuth token instead. (However, it will disable updating GitHub Statuses.) I'll check if we can use the GitHub OAuth token. > Migrate to Python 3 and upgrade Yetus to 0.13.0 > --- > > Key: HADOOP-16748 > URL: https://issues.apache.org/jira/browse/HADOOP-16748 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16748) Migrate to Python 3 and upgrade Yetus to 0.13.0
[ https://issues.apache.org/jira/browse/HADOOP-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300053#comment-17300053 ] Akira Ajisaka commented on HADOOP-16748: Maybe the GitHub token is expired. Created a draft PR to check: https://github.com/apache/hadoop/pull/2765 > Migrate to Python 3 and upgrade Yetus to 0.13.0 > --- > > Key: HADOOP-16748 > URL: https://issues.apache.org/jira/browse/HADOOP-16748 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?focusedWorklogId=565117&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565117 ] ASF GitHub Bot logged work on HADOOP-17570: --- Author: ASF GitHub Bot Created on: 12/Mar/21 05:23 Start Date: 12/Mar/21 05:23 Worklog Time Spent: 10m Work Description: aajisaka opened a new pull request #2765: URL: https://github.com/apache/hadoop/pull/2765 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565117) Time Spent: 2h 20m (was: 2h 10m) > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request #2765: Test PR to check HADOOP-17570
aajisaka opened a new pull request #2765: URL: https://github.com/apache/hadoop/pull/2765 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16748) Migrate to Python 3 and upgrade Yetus to 0.13.0
[ https://issues.apache.org/jira/browse/HADOOP-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300048#comment-17300048 ] Ayush Saxena commented on HADOOP-16748: --- Hey [~aajisaka], Seems the comment stuff isn't working, I checked a couple of recent PR's [https://github.com/apache/hadoop/pull/2763] [https://github.com/apache/hadoop/pull/2762] Do we need to pass the parameter somewhere in the job as well? > Migrate to Python 3 and upgrade Yetus to 0.13.0 > --- > > Key: HADOOP-16748 > URL: https://issues.apache.org/jira/browse/HADOOP-16748 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17580?focusedWorklogId=565104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565104 ] ASF GitHub Bot logged work on HADOOP-17580: --- Author: ASF GitHub Bot Created on: 12/Mar/21 04:20 Start Date: 12/Mar/21 04:20 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592900180 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. Review comment: can you update `overview` section also like below..?  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565104) Time Spent: 50m (was: 40m) > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17580 > URL: https://issues.apache.org/jira/browse/HADOOP-17580 > Project: Hadoop Common > Issue Type: Improvement >Reporter: wenzhen >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2763: #HADOOP-17580, refresh freatures description in index.md
brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592900180 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. Review comment: can you update `overview` section also like below..?  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17580?focusedWorklogId=565102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565102 ] ASF GitHub Bot logged work on HADOOP-17580: --- Author: ASF GitHub Bot Created on: 12/Mar/21 04:19 Start Date: 12/Mar/21 04:19 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592899831 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. ## Features -* Read and write data stored in a HuaweiCloud OBS account. -* Reference file system paths using URLs using the `obs` scheme. -* Present a hierarchical file system view by implementing the standard Hadoop `FileSystem` interface. -* Support multipart upload for a large file. +* Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. +* In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. Review comment: Let it be similar to the azure like following..? `Read and write data stored in an Azure Data Lake Storage account.` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565102) Time Spent: 40m (was: 0.5h) > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17580 > URL: https://issues.apache.org/jira/browse/HADOOP-17580 > Project: Hadoop Common > Issue Type: Improvement >Reporter: wenzhen >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2754: HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcS…
tomscut commented on pull request #2754: URL: https://github.com/apache/hadoop/pull/2754#issuecomment-797223891 Failed junit tests hadoop.hdfs.server.federation.router.TestConnectionManager This failed unit tests is unrelated to the change, and it works fine locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2763: #HADOOP-17580, refresh freatures description in index.md
brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592899831 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. ## Features -* Read and write data stored in a HuaweiCloud OBS account. -* Reference file system paths using URLs using the `obs` scheme. -* Present a hierarchical file system view by implementing the standard Hadoop `FileSystem` interface. -* Support multipart upload for a large file. +* Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. +* In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. Review comment: Let it be similar to the azure like following..? `Read and write data stored in an Azure Data Lake Storage account.` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17580?focusedWorklogId=565100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565100 ] ASF GitHub Bot logged work on HADOOP-17580: --- Author: ASF GitHub Bot Created on: 12/Mar/21 04:09 Start Date: 12/Mar/21 04:09 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592897162 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. ## Features -* Read and write data stored in a HuaweiCloud OBS account. -* Reference file system paths using URLs using the `obs` scheme. -* Present a hierarchical file system view by implementing the standard Hadoop `FileSystem` interface. -* Support multipart upload for a large file. +* Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. +* In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. * Can act as a source of data in a MapReduce job, or a sink. +* Support multipart upload for a large file. +* Reference file system paths using URLs using the obs scheme. Review comment: `obs` can be highlighted here.? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565100) Time Spent: 0.5h (was: 20m) > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17580 > URL: https://issues.apache.org/jira/browse/HADOOP-17580 > Project: Hadoop Common > Issue Type: Improvement >Reporter: wenzhen >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2763: #HADOOP-17580, refresh freatures description in index.md
brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592897162 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. ## Features -* Read and write data stored in a HuaweiCloud OBS account. -* Reference file system paths using URLs using the `obs` scheme. -* Present a hierarchical file system view by implementing the standard Hadoop `FileSystem` interface. -* Support multipart upload for a large file. +* Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. +* In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. * Can act as a source of data in a MapReduce job, or a sink. +* Support multipart upload for a large file. +* Reference file system paths using URLs using the obs scheme. Review comment: `obs` can be highlighted here.? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17580?focusedWorklogId=565098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565098 ] ASF GitHub Bot logged work on HADOOP-17580: --- Author: ASF GitHub Bot Created on: 12/Mar/21 04:04 Start Date: 12/Mar/21 04:04 Worklog Time Spent: 10m Work Description: brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592895646 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. ## Features -* Read and write data stored in a HuaweiCloud OBS account. -* Reference file system paths using URLs using the `obs` scheme. -* Present a hierarchical file system view by implementing the standard Hadoop `FileSystem` interface. -* Support multipart upload for a large file. +* Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. +* In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. * Can act as a source of data in a MapReduce job, or a sink. +* Support multipart upload for a large file. +* Reference file system paths using URLs using the obs scheme. * Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and authentication schemes. +* For OBS 'Parallel file system bucket'(Posix), support rename with atomic semantics, Append, hflush&hsync. +* For OBS 'Parallel file system bucket'(Posix), provide trash mechanism and quickly delete, which using rename and lifecircle of OBS bucket. Review comment: can you mention about `lifecircle`.? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565098) Time Spent: 20m (was: 10m) > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17580 > URL: https://issues.apache.org/jira/browse/HADOOP-17580 > Project: Hadoop Common > Issue Type: Improvement >Reporter: wenzhen >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2763: #HADOOP-17580, refresh freatures description in index.md
brahmareddybattula commented on a change in pull request #2763: URL: https://github.com/apache/hadoop/pull/2763#discussion_r592895646 ## File path: hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md ## @@ -24,12 +24,14 @@ This support comes via the JAR file `hadoop-huaweicloud.jar`. ## Features -* Read and write data stored in a HuaweiCloud OBS account. -* Reference file system paths using URLs using the `obs` scheme. -* Present a hierarchical file system view by implementing the standard Hadoop `FileSystem` interface. -* Support multipart upload for a large file. +* Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. +* In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. * Can act as a source of data in a MapReduce job, or a sink. +* Support multipart upload for a large file. +* Reference file system paths using URLs using the obs scheme. * Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and authentication schemes. +* For OBS 'Parallel file system bucket'(Posix), support rename with atomic semantics, Append, hflush&hsync. +* For OBS 'Parallel file system bucket'(Posix), provide trash mechanism and quickly delete, which using rename and lifecircle of OBS bucket. Review comment: can you mention about `lifecircle`.? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17581) Fix reference to LOG is ambiguous after HADOOP-17482
[
https://issues.apache.org/jira/browse/HADOOP-17581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-17581:
Status: Patch Available (was: Open)
> Fix reference to LOG is ambiguous after HADOOP-17482
>
>
> Key: HADOOP-17581
> URL: https://issues.apache.org/jira/browse/HADOOP-17581
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HADOOP-17482 changes to have two slf4j LOG instances for FileSystem.class.
> This seems to breaks the Hadoop CI/Jenkins as some tests using this LOG
> directly are hitting the ambiguity issue between two slf4j Logger instances
> to the same FileSystem.class failed the build. This ticket is opened to fix
> those tests to unblock CI.
>
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java:[1424,25]
> error: reference to LOG is ambiguous
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java:[102,25]
> error: reference to LOG is ambiguous
> [INFO] 2 errors
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[ https://issues.apache.org/jira/browse/HADOOP-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300011#comment-17300011 ] Xiaoyu Yao commented on HADOOP-17482: - I opened HADOOP-17581 to unblock CI by changing the affected tests not using FileSystem.LOG directly. > Remove Commons Logger from FileSystem Class > --- > > Key: HADOOP-17482 > URL: https://issues.apache.org/jira/browse/HADOOP-17482 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Remove reference to Commons Logger in FileSystem, it already has SLF4J, so > it's a bit weird to be mixing and matching and interweaving loggers in this > way. Also, my hope is to eventually migrate everything to SLF4J to simplify > things for downstream consumers of the common library. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17581) Fix reference to LOG is ambiguous after HADOOP-17482
[
https://issues.apache.org/jira/browse/HADOOP-17581?focusedWorklogId=565091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565091
]
ASF GitHub Bot logged work on HADOOP-17581:
---
Author: ASF GitHub Bot
Created on: 12/Mar/21 03:10
Start Date: 12/Mar/21 03:10
Worklog Time Spent: 10m
Work Description: xiaoyuyao opened a new pull request #2764:
URL: https://github.com/apache/hadoop/pull/2764
JIRA: https://issues.apache.org/jira/browse/HADOOP-17581
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 565091)
Remaining Estimate: 0h
Time Spent: 10m
> Fix reference to LOG is ambiguous after HADOOP-17482
>
>
> Key: HADOOP-17581
> URL: https://issues.apache.org/jira/browse/HADOOP-17581
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HADOOP-17482 changes to have two slf4j LOG instances for FileSystem.class.
> This seems to breaks the Hadoop CI/Jenkins as some tests using this LOG
> directly are hitting the ambiguity issue between two slf4j Logger instances
> to the same FileSystem.class failed the build. This ticket is opened to fix
> those tests to unblock CI.
>
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java:[1424,25]
> error: reference to LOG is ambiguous
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java:[102,25]
> error: reference to LOG is ambiguous
> [INFO] 2 errors
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17581) Fix reference to LOG is ambiguous after HADOOP-17482
[
https://issues.apache.org/jira/browse/HADOOP-17581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17581:
Labels: pull-request-available (was: )
> Fix reference to LOG is ambiguous after HADOOP-17482
>
>
> Key: HADOOP-17581
> URL: https://issues.apache.org/jira/browse/HADOOP-17581
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HADOOP-17482 changes to have two slf4j LOG instances for FileSystem.class.
> This seems to breaks the Hadoop CI/Jenkins as some tests using this LOG
> directly are hitting the ambiguity issue between two slf4j Logger instances
> to the same FileSystem.class failed the build. This ticket is opened to fix
> those tests to unblock CI.
>
> {code:java}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java:[1424,25]
> error: reference to LOG is ambiguous
> [ERROR]
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java:[102,25]
> error: reference to LOG is ambiguous
> [INFO] 2 errors
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao opened a new pull request #2764: HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482.
xiaoyuyao opened a new pull request #2764: URL: https://github.com/apache/hadoop/pull/2764 JIRA: https://issues.apache.org/jira/browse/HADOOP-17581 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17581) Fix reference to LOG is ambiguous after HADOOP-17482
Xiaoyu Yao created HADOOP-17581:
---
Summary: Fix reference to LOG is ambiguous after HADOOP-17482
Key: HADOOP-17581
URL: https://issues.apache.org/jira/browse/HADOOP-17581
Project: Hadoop Common
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
HADOOP-17482 changes to have two slf4j LOG instances for FileSystem.class.
This seems to breaks the Hadoop CI/Jenkins as some tests using this LOG
directly are hitting the ambiguity issue between two slf4j Logger instances to
the same FileSystem.class failed the build. This ticket is opened to fix those
tests to unblock CI.
{code:java}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java:[1424,25]
error: reference to LOG is ambiguous
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java:[102,25]
error: reference to LOG is ambiguous
[INFO] 2 errors
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17482) Remove Commons Logger from FileSystem Class
[
https://issues.apache.org/jira/browse/HADOOP-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1730#comment-1730
]
Xiaoyu Yao commented on HADOOP-17482:
-
Agree with [~ste...@apache.org] mentioned above. The Jira is open but PR has
been merged.
The merged change breaks the Hadoop CI/Jenkins as now the ambiguity issue
between two slf4j Logger instances to the same FileSystem.class failed the
build.
Previously log instances seem hacky with as one from common logger and one from
slf4j.
{code:java}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java:[1424,25]
error: reference to LOG is ambiguous
[ERROR]
/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-2762/src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java:[102,25]
error: reference to LOG is ambiguous
[INFO] 2 errors
{code}
> Remove Commons Logger from FileSystem Class
> ---
>
> Key: HADOOP-17482
> URL: https://issues.apache.org/jira/browse/HADOOP-17482
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Remove reference to Commons Logger in FileSystem, it already has SLF4J, so
> it's a bit weird to be mixing and matching and interweaving loggers in this
> way. Also, my hope is to eventually migrate everything to SLF4J to simplify
> things for downstream consumers of the common library.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] karthikhw edited a comment on pull request #2728: HDFS-15865 Interrupt DataStreamer thread if no ack
karthikhw edited a comment on pull request #2728: URL: https://github.com/apache/hadoop/pull/2728#issuecomment-797195961 @mukul1987 Not exactly found when nodes come null but appears when client couldn't reachable datenode (during middle of its write). Looks the next retry comes with null. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] karthikhw commented on pull request #2728: HDFS-15865 Interrupt DataStreamer thread if no ack
karthikhw commented on pull request #2728: URL: https://github.com/apache/hadoop/pull/2728#issuecomment-797195961 @mukul1987 Not exactly found when nodes come null but appears when client couldn't reach datenode (during middle of its write). Looks the next retry comes with null. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17580: Labels: pull-request-available (was: ) > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17580 > URL: https://issues.apache.org/jira/browse/HADOOP-17580 > Project: Hadoop Common > Issue Type: Improvement >Reporter: wenzhen >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
[ https://issues.apache.org/jira/browse/HADOOP-17580?focusedWorklogId=565068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565068 ] ASF GitHub Bot logged work on HADOOP-17580: --- Author: ASF GitHub Bot Created on: 12/Mar/21 01:49 Start Date: 12/Mar/21 01:49 Worklog Time Spent: 10m Work Description: yuchen79 opened a new pull request #2763: URL: https://github.com/apache/hadoop/pull/2763 ## NOTICE refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support #HADOOP-17580 Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565068) Remaining Estimate: 0h Time Spent: 10m > refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support > --- > > Key: HADOOP-17580 > URL: https://issues.apache.org/jira/browse/HADOOP-17580 > Project: Hadoop Common > Issue Type: Improvement >Reporter: wenzhen >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, > such as : > rename, Append , hflush&hsync support. > link : > hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md > Features > Present a hierarchical HDFS file system by implementing the standard Hadoop > FileSystem interface. > In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. > Can act as a source of data in a MapReduce job, or a sink. > Support multipart upload for a large file. > Reference file system paths using URLs using the obs scheme. > Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and > authentication schemes. > For OBS 'Parallel file system bucket'(Posix), support rename with atomic > semantics, Append, hflush&hsync. > For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and > quickly delete, which using rename and lifecircle of OBS bucket. > Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yuchen79 opened a new pull request #2763: #HADOOP-17580, refresh freatures description in index.md
yuchen79 opened a new pull request #2763: URL: https://github.com/apache/hadoop/pull/2763 ## NOTICE refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support #HADOOP-17580 Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17580) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
wenzhen created HADOOP-17580: Summary: refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support Key: HADOOP-17580 URL: https://issues.apache.org/jira/browse/HADOOP-17580 Project: Hadoop Common Issue Type: Improvement Reporter: wenzhen Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, such as : rename, Append , hflush&hsync support. link : hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md Features Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. Can act as a source of data in a MapReduce job, or a sink. Support multipart upload for a large file. Reference file system paths using URLs using the obs scheme. Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and authentication schemes. For OBS 'Parallel file system bucket'(Posix), support rename with atomic semantics, Append, hflush&hsync. For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and quickly delete, which using rename and lifecircle of OBS bucket. Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17579) refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support
wenzhen created HADOOP-17579: Summary: refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support Key: HADOOP-17579 URL: https://issues.apache.org/jira/browse/HADOOP-17579 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.3.0 Reporter: wenzhen Fix For: 3.3.0 Refresh freatures description of HuaweiCloud OBS Adapter for Hadoop Support, such as : rename, Append , hflush&hsync support. link : hadoop/hadoop-cloud-storage-project/hadoop-huaweicloud/src/site/markdown/index.md Features Present a hierarchical HDFS file system by implementing the standard Hadoop FileSystem interface. In hadoop, Read and write data stored in a HuaweiCloud OBS bucket. Can act as a source of data in a MapReduce job, or a sink. Support multipart upload for a large file. Reference file system paths using URLs using the obs scheme. Uses HuaweiCloud OBS’s Java SDK with support for latest OBS features and authentication schemes. For OBS 'Parallel file system bucket'(Posix), support rename with atomic semantics, Append, hflush&hsync. For OBS 'Parallel file system bucket'(Posix), Provide trash mechanism and quickly delete, which using rename and lifecircle of OBS bucket. Tested for scale. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17514) Remove trace subcommand from hadoop CLI
[ https://issues.apache.org/jira/browse/HADOOP-17514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-17514: -- Fix Version/s: 3.4.0 Hadoop Flags: Incompatible change,Reviewed Release Note: `trace` subcommand of hadoop CLI was removed as a follow-up of removal of TraceAdmin protocol. Resolution: Fixed Status: Resolved (was: Patch Available) > Remove trace subcommand from hadoop CLI > --- > > Key: HADOOP-17514 > URL: https://issues.apache.org/jira/browse/HADOOP-17514 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > TraceAdmin protocol and utility class were removed by HADOOP-17424. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17514) Remove trace subcommand from hadoop CLI
[ https://issues.apache.org/jira/browse/HADOOP-17514?focusedWorklogId=565061&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565061 ] ASF GitHub Bot logged work on HADOOP-17514: --- Author: ASF GitHub Bot Created on: 12/Mar/21 01:27 Start Date: 12/Mar/21 01:27 Worklog Time Spent: 10m Work Description: iwasakims merged pull request #2680: URL: https://github.com/apache/hadoop/pull/2680 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565061) Time Spent: 1h 40m (was: 1.5h) > Remove trace subcommand from hadoop CLI > --- > > Key: HADOOP-17514 > URL: https://issues.apache.org/jira/browse/HADOOP-17514 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > TraceAdmin protocol and utility class were removed by HADOOP-17424. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims merged pull request #2680: HADOOP-17514. Remove trace subcommand from hadoop CLI.
iwasakims merged pull request #2680: URL: https://github.com/apache/hadoop/pull/2680 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #2754: HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcS…
tomscut commented on pull request #2754: URL: https://github.com/apache/hadoop/pull/2754#issuecomment-797168111 > Should be yes, if it isn't used anywhere, The build isn't happy, can you rebase and push again? > If things are good post that we can push this Thanks @ayushtkn for the review, I rebased and pushed again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17514) Remove trace subcommand from hadoop CLI
[ https://issues.apache.org/jira/browse/HADOOP-17514?focusedWorklogId=565058&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565058 ] ASF GitHub Bot logged work on HADOOP-17514: --- Author: ASF GitHub Bot Created on: 12/Mar/21 01:20 Start Date: 12/Mar/21 01:20 Worklog Time Spent: 10m Work Description: iwasakims edited a comment on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-784053049 @jojochuang TraceAdmin protocol was removed by @smengcl because it is not needed for alternative tracer impl. https://github.com/apache/hadoop/pull/2645#issuecomment-768251870 If it turned out to be needed, we can add it back again. Since trace subcommand was actually dynamic reconfiguration functionality, adding generic protocol which can be used for other purposes could be an option. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565058) Time Spent: 1h 20m (was: 1h 10m) > Remove trace subcommand from hadoop CLI > --- > > Key: HADOOP-17514 > URL: https://issues.apache.org/jira/browse/HADOOP-17514 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > TraceAdmin protocol and utility class were removed by HADOOP-17424. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17514) Remove trace subcommand from hadoop CLI
[ https://issues.apache.org/jira/browse/HADOOP-17514?focusedWorklogId=565059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565059 ] ASF GitHub Bot logged work on HADOOP-17514: --- Author: ASF GitHub Bot Created on: 12/Mar/21 01:20 Start Date: 12/Mar/21 01:20 Worklog Time Spent: 10m Work Description: iwasakims commented on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-797166266 Thanks, @smengcl. I'm merging this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 565059) Time Spent: 1.5h (was: 1h 20m) > Remove trace subcommand from hadoop CLI > --- > > Key: HADOOP-17514 > URL: https://issues.apache.org/jira/browse/HADOOP-17514 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > TraceAdmin protocol and utility class were removed by HADOOP-17424. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #2680: HADOOP-17514. Remove trace subcommand from hadoop CLI.
iwasakims commented on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-797166266 Thanks, @smengcl. I'm merging this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] iwasakims edited a comment on pull request #2680: HADOOP-17514. Remove trace subcommand from hadoop CLI.
iwasakims edited a comment on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-784053049 @jojochuang TraceAdmin protocol was removed by @smengcl because it is not needed for alternative tracer impl. https://github.com/apache/hadoop/pull/2645#issuecomment-768251870 If it turned out to be needed, we can add it back again. Since trace subcommand was actually dynamic reconfiguration functionality, adding generic protocol which can be used for other purposes could be an option. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #2585: HDFS-15759. EC: Verify EC reconstruction correctness on DataNode
ferhui commented on pull request #2585: URL: https://github.com/apache/hadoop/pull/2585#issuecomment-797163497 @touchida Thanks. I will review this. I check the CI results firstly This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17578) Improve UGI debug log to help troubleshooting TokenCache related issues
[ https://issues.apache.org/jira/browse/HADOOP-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-17578: Status: Patch Available (was: Open) > Improve UGI debug log to help troubleshooting TokenCache related issues > --- > > Key: HADOOP-17578 > URL: https://issues.apache.org/jira/browse/HADOOP-17578 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We have seen some issues around TokenCache getDelegationToken failures even > though the UGI already has a valid token. The tricky part is the token map is > keyed by the canonical service name, which can be different from the actual > service field in the token, e.g. KMS token in HA case. The current UGI log > dumps all the tokens but not the keys of the token map. This ticket is opened > to include the complete token map information in the debug log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17578) Improve UGI debug log to help troubleshooting TokenCache related issues
[ https://issues.apache.org/jira/browse/HADOOP-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17578: Labels: pull-request-available (was: ) > Improve UGI debug log to help troubleshooting TokenCache related issues > --- > > Key: HADOOP-17578 > URL: https://issues.apache.org/jira/browse/HADOOP-17578 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We have seen some issues around TokenCache getDelegationToken failures even > though the UGI already has a valid token. The tricky part is the token map is > keyed by the canonical service name, which can be different from the actual > service field in the token, e.g. KMS token in HA case. The current UGI log > dumps all the tokens but not the keys of the token map. This ticket is opened > to include the complete token map information in the debug log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17578) Improve UGI debug log to help troubleshooting TokenCache related issues
[ https://issues.apache.org/jira/browse/HADOOP-17578?focusedWorklogId=564983&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-564983 ] ASF GitHub Bot logged work on HADOOP-17578: --- Author: ASF GitHub Bot Created on: 11/Mar/21 22:10 Start Date: 11/Mar/21 22:10 Worklog Time Spent: 10m Work Description: xiaoyuyao opened a new pull request #2762: URL: https://github.com/apache/hadoop/pull/2762 https://issues.apache.org/jira/browse/HADOOP-17578 Additional Debug Log for troubleshooting TokenCache related issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 564983) Remaining Estimate: 0h Time Spent: 10m > Improve UGI debug log to help troubleshooting TokenCache related issues > --- > > Key: HADOOP-17578 > URL: https://issues.apache.org/jira/browse/HADOOP-17578 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We have seen some issues around TokenCache getDelegationToken failures even > though the UGI already has a valid token. The tricky part is the token map is > keyed by the canonical service name, which can be different from the actual > service field in the token, e.g. KMS token in HA case. The current UGI log > dumps all the tokens but not the keys of the token map. This ticket is opened > to include the complete token map information in the debug log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao opened a new pull request #2762: HADOOP-17578. Improve UGI debug log to help troubleshooting TokenCach…
xiaoyuyao opened a new pull request #2762: URL: https://github.com/apache/hadoop/pull/2762 https://issues.apache.org/jira/browse/HADOOP-17578 Additional Debug Log for troubleshooting TokenCache related issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17578) Improve UGI debug log to help troubleshooting TokenCache related issues
[ https://issues.apache.org/jira/browse/HADOOP-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-17578: Summary: Improve UGI debug log to help troubleshooting TokenCache related issues (was: Improve UGI debug log to help troubleshoot TokenCache related issues) > Improve UGI debug log to help troubleshooting TokenCache related issues > --- > > Key: HADOOP-17578 > URL: https://issues.apache.org/jira/browse/HADOOP-17578 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.2.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > > We have seen some issues around TokenCache getDelegationToken failures even > though the UGI already has a valid token. The tricky part is the token map is > keyed by the canonical service name, which can be different from the actual > service field in the token, e.g. KMS token in HA case. The current UGI log > dumps all the tokens but not the keys of the token map. This ticket is opened > to include the complete token map information in the debug log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17578) Improve UGI debug log to help troubleshoot TokenCache related issues
Xiaoyu Yao created HADOOP-17578: --- Summary: Improve UGI debug log to help troubleshoot TokenCache related issues Key: HADOOP-17578 URL: https://issues.apache.org/jira/browse/HADOOP-17578 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.2.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao We have seen some issues around TokenCache getDelegationToken failures even though the UGI already has a valid token. The tricky part is the token map is keyed by the canonical service name, which can be different from the actual service field in the token, e.g. KMS token in HA case. The current UGI log dumps all the tokens but not the keys of the token map. This ticket is opened to include the complete token map information in the debug log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17571) Upgrade com.fasterxml.woodstox:woodstox-core for security reasons
[ https://issues.apache.org/jira/browse/HADOOP-17571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-17571: --- Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Committed to all target branches (2.10.2+). Thank you [~vjasani] for your contribution. Thank you [~aajisaka] for your review. > Upgrade com.fasterxml.woodstox:woodstox-core for security reasons > - > > Key: HADOOP-17571 > URL: https://issues.apache.org/jira/browse/HADOOP-17571 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3 > > Time Spent: 1h > Remaining Estimate: 0h > > Due to security concerns (CVE: sonatype-2018-0624), we should bump up > woodstox-core to 5.3.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17571) Upgrade com.fasterxml.woodstox:woodstox-core for security reasons
[ https://issues.apache.org/jira/browse/HADOOP-17571?focusedWorklogId=564887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-564887 ] ASF GitHub Bot logged work on HADOOP-17571: --- Author: ASF GitHub Bot Created on: 11/Mar/21 19:51 Start Date: 11/Mar/21 19:51 Worklog Time Spent: 10m Work Description: liuml07 merged pull request #2757: URL: https://github.com/apache/hadoop/pull/2757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 564887) Time Spent: 1h (was: 50m) > Upgrade com.fasterxml.woodstox:woodstox-core for security reasons > - > > Key: HADOOP-17571 > URL: https://issues.apache.org/jira/browse/HADOOP-17571 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3 > > Time Spent: 1h > Remaining Estimate: 0h > > Due to security concerns (CVE: sonatype-2018-0624), we should bump up > woodstox-core to 5.3.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liuml07 merged pull request #2757: HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns
liuml07 merged pull request #2757: URL: https://github.com/apache/hadoop/pull/2757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17514) Remove trace subcommand from hadoop CLI
[ https://issues.apache.org/jira/browse/HADOOP-17514?focusedWorklogId=564848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-564848 ] ASF GitHub Bot logged work on HADOOP-17514: --- Author: ASF GitHub Bot Created on: 11/Mar/21 18:54 Start Date: 11/Mar/21 18:54 Worklog Time Spent: 10m Work Description: smengcl edited a comment on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-796956673 Good catch. Actually because `TraceAdmin` is removed, the `trace` subcommand is referencing a non-existent class now: ``` $ ./bin/hadoop trace Error: Could not find or load main class org.apache.hadoop.tracing.TraceAdmin ``` I'm +1 on removing this right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 564848) Time Spent: 1h 10m (was: 1h) > Remove trace subcommand from hadoop CLI > --- > > Key: HADOOP-17514 > URL: https://issues.apache.org/jira/browse/HADOOP-17514 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > TraceAdmin protocol and utility class were removed by HADOOP-17424. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl edited a comment on pull request #2680: HADOOP-17514. Remove trace subcommand from hadoop CLI.
smengcl edited a comment on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-796956673 Good catch. Actually because `TraceAdmin` is removed, the `trace` subcommand is referencing a non-existent class now: ``` $ ./bin/hadoop trace Error: Could not find or load main class org.apache.hadoop.tracing.TraceAdmin ``` I'm +1 on removing this right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299807#comment-17299807 ] Steve Loughran commented on HADOOP-17372: - adding to HADOOP-17511 as auditing plugins are the kind of thing which breaks > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17372 started by Steve Loughran. --- > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-17372: --- Assignee: Steve Loughran > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16829) Über-jira: S3A Hadoop 3.3.1 features
[ https://issues.apache.org/jira/browse/HADOOP-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299806#comment-17299806 ] Steve Loughran commented on HADOOP-16829: - Linking to HADOOP-17566 where everything not in 3.3.1 is being moved > Über-jira: S3A Hadoop 3.3.1 features > > > Key: HADOOP-16829 > URL: https://issues.apache.org/jira/browse/HADOOP-16829 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Über-jira: S3A features/fixes for Hadoop 3.4 > As usual, this will clutter up with everything which hasn't gone in: don't > interpret presence on this list as a commitment to implement. > And for anyone wanting to add patches > MUST > # reviews via github PRs > # *no declaration of AWS S3 endpoint (or other S3 impl) -no review* > SHOULD > # have a setup for testing SSE-KMS, DDB/S3Guard > # including an assumed role we can use for AssumedRole Delegation Tokens > If you are going near those bits of code, they uprate from SHOULD to MUST. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16572) S3A DT support to warn when loading expired token
[ https://issues.apache.org/jira/browse/HADOOP-16572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16572: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > S3A DT support to warn when loading expired token > - > > Key: HADOOP-16572 > URL: https://issues.apache.org/jira/browse/HADOOP-16572 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 > Environment: CLI with HADOOP_TOKEN_PATH pointing at a file from the > day before, containing an assumed Role DT, which was being loaded ahead of > any setting in the XML file >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > (This just cost me half an hour as somehow as working CLI command stopped > working since the day before, and I've been playing with endpoints and > signing before I realised it) > _If the DT provider code loads a token from a file, it doesn't check or warn > for an expired token -all you get is a 400 Bad request failure_ > This not at all obvious. > Proposed > * WARN if now > expiry > * extra entry in troubleshooting for 400 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17514) Remove trace subcommand from hadoop CLI
[ https://issues.apache.org/jira/browse/HADOOP-17514?focusedWorklogId=564828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-564828 ] ASF GitHub Bot logged work on HADOOP-17514: --- Author: ASF GitHub Bot Created on: 11/Mar/21 18:40 Start Date: 11/Mar/21 18:40 Worklog Time Spent: 10m Work Description: smengcl commented on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-796956673 Good catch. Actually because I removed `TraceAdmin`, the `trace` subcommand is referencing a non-existent class now (class not found exception when invoked). I'm +1 on removing this right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 564828) Time Spent: 1h (was: 50m) > Remove trace subcommand from hadoop CLI > --- > > Key: HADOOP-17514 > URL: https://issues.apache.org/jira/browse/HADOOP-17514 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > TraceAdmin protocol and utility class were removed by HADOOP-17424. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] smengcl commented on pull request #2680: HADOOP-17514. Remove trace subcommand from hadoop CLI.
smengcl commented on pull request #2680: URL: https://github.com/apache/hadoop/pull/2680#issuecomment-796956673 Good catch. Actually because I removed `TraceAdmin`, the `trace` subcommand is referencing a non-existent class now (class not found exception when invoked). I'm +1 on removing this right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299796#comment-17299796
]
Hadoop QA commented on HADOOP-16819:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 47s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m
34s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
36s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
49s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 17s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
[jira] [Updated] (HADOOP-16574) ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled
[ https://issues.apache.org/jira/browse/HADOOP-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16574: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled > - > > Key: HADOOP-16574 > URL: https://issues.apache.org/jira/browse/HADOOP-16574 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > If you enable DTs on a bucket, then those tests which force failures from bad > credential providers fail -the IOE they look for is wrapped in a > ServiceStateException > Proposed: catch those and rethrow the nested IOE -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14237: Resolution: Won't Fix Status: Resolved (was: Patch Available) > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura >Priority: Minor > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299787#comment-17299787 ] Steve Loughran commented on HADOOP-14661: - HADOOP-17511 is lining up for this patch -all S3 requests will be from a request factory...this is where the option will be set > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14661) S3A to support Requester Pays Buckets
[ https://issues.apache.org/jira/browse/HADOOP-14661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14661: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > S3A to support Requester Pays Buckets > - > > Key: HADOOP-14661 > URL: https://issues.apache.org/jira/browse/HADOOP-14661 > Project: Hadoop Common > Issue Type: Sub-task > Components: common, util >Affects Versions: 3.0.0-alpha3 >Reporter: Mandus Momberg >Assignee: Mandus Momberg >Priority: Minor > Attachments: HADOOP-14661.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > Amazon S3 has the ability to charge the requester for the cost of accessing > S3. This is called Requester Pays Buckets. > In order to access these buckets, each request needs to be signed with a > specific header. > http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16203) ITestS3AContractGetFileStatusV1List may have consistency issues
[
https://issues.apache.org/jira/browse/HADOOP-16203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16203:
Parent Issue: HADOOP-17566 (was: HADOOP-16829)
> ITestS3AContractGetFileStatusV1List may have consistency issues
> ---
>
> Key: HADOOP-16203
> URL: https://issues.apache.org/jira/browse/HADOOP-16203
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> Seeing in a failure in the listing tests which looks like it could suffer
> from some consistency/concurrency issues: the path used is chosen from the
> method name, but with two subclasses of the
> {{AbstractContractGetFileStatusTest}} suite, the S3A tests could be
> interfering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15347) S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy
[ https://issues.apache.org/jira/browse/HADOOP-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15347: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with > the throttle backoff policy > --- > > Key: HADOOP-15347 > URL: https://issues.apache.org/jira/browse/HADOOP-15347 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Minor > > FLINK-9061 implies that some 500 responses are caused by server-side overload > of some form. > That means they should really have the throttle retry policy applied, not the > connectivity one -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299783#comment-17299783 ] Steve Loughran commented on HADOOP-15870: - where are we with this to get it into 3.3.1? > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch, HADOOP-15870-006.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16538) S3AFilesystem trash handling should respect the current UGI
[ https://issues.apache.org/jira/browse/HADOOP-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299779#comment-17299779 ] Steve Loughran commented on HADOOP-16538: - FWIW, I think we should do it in FileSystem > S3AFilesystem trash handling should respect the current UGI > --- > > Key: HADOOP-16538 > URL: https://issues.apache.org/jira/browse/HADOOP-16538 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Siddharth Seth >Priority: Major > > S3 move to trash currently relies upon System.getProperty(user.name). > Instead, it should be relying on the current UGI to figure out the username. > getHomeDirectory needs to be overridden to use UGI instead of > System.getProperty -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16538) S3AFilesystem trash handling should respect the current UGI
[ https://issues.apache.org/jira/browse/HADOOP-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299778#comment-17299778 ] Steve Loughran commented on HADOOP-16538: - This has been somewhat covered in HADOOP-16026, which uses UGI, but it uses the user of the caller, when it should be the cached UGI.currentUser at time of FS.initialize -the way we do for DTs and other auth bits > S3AFilesystem trash handling should respect the current UGI > --- > > Key: HADOOP-16538 > URL: https://issues.apache.org/jira/browse/HADOOP-16538 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Siddharth Seth >Priority: Major > > S3 move to trash currently relies upon System.getProperty(user.name). > Instead, it should be relying on the current UGI to figure out the username. > getHomeDirectory needs to be overridden to use UGI instead of > System.getProperty -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16259) Distcp to set S3 Storage Class
[ https://issues.apache.org/jira/browse/HADOOP-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16259: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > Distcp to set S3 Storage Class > -- > > Key: HADOOP-16259 > URL: https://issues.apache.org/jira/browse/HADOOP-16259 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, tools/distcp >Affects Versions: 2.8.4 >Reporter: Prakash Gopalsamy >Priority: Minor > Attachments: ENHANCE_HADOOP_DISTCP_FOR_CUSTOM_S3_STORAGE_CLASS.docx > > Original Estimate: 168h > Remaining Estimate: 168h > > Hadoop distcp implementation doesn’t have properties to override Storage > class while transferring data to Amazon S3 storage. Hadoop distcp doesn’t set > any storage class while transferring data to Amazon S3 storage. Due to this > all the objects moved from cluster to S3 using Hadoop Distcp are been stored > in the default storage class “STANDARD”. By providing a new feature to > override the default S3 storage class through configuration properties will > be helpful to upload objects in other storage classes. I have come up with a > design to implement this feature in a design document and uploaded the same > in the JIRA. Kindly review and let me know for your suggestions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16370) S3AFileSystem copyFile to propagate etag/version from getObjectMetadata to copy request
[ https://issues.apache.org/jira/browse/HADOOP-16370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16370: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > S3AFileSystem copyFile to propagate etag/version from getObjectMetadata to > copy request > --- > > Key: HADOOP-16370 > URL: https://issues.apache.org/jira/browse/HADOOP-16370 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > Something to consider if we want: should the etag and version from the > initial getObjectMetadata call be propagated to the actual CopyRequest *if > they are not already known* > That way, if we rename() a file and its etag/version is not known, we can fix > them for the next stage of the operation. Relevant given we are copying > metadata over, and for resilience to changes while the copy is taking place -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17541) Yetus does not run qbt-trunk
[
https://issues.apache.org/jira/browse/HADOOP-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HADOOP-17541.
Resolution: Done
Closing.
> Yetus does not run qbt-trunk
>
>
> Key: HADOOP-17541
> URL: https://issues.apache.org/jira/browse/HADOOP-17541
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin, build, yetus
>Reporter: Ahmed Hussein
>Priority: Major
>
> On Feb20th, qbt-reports started to generate empty reports
> {code:bash}
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/424/
> ERROR: File 'out/email-report.txt' does not exist
> {code}
> On Jenkins, the job fails with the following error:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/425/console
> {code:bash}
> ERROR:
> /home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64//dev-support/bin/hadoop.sh
> does not exist.
> Build step 'Execute shell' marked build as failure
> Archiving artifacts
> [Fast Archiver] No prior successful build to compare, so performing full copy
> of artifacts
> Recording test results
> ERROR: Step ?Publish JUnit test result report? failed: No test report files
> were found. Configuration error?
> {code}
> [~aajisaka], I think this would be caused by HADOOP-16748 . I noticed that
> the PR of that HADOOP-16748 ceased from showing any reports, but for some
> reason I forgot about that while reviewing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16748) Migrate to Python 3 and upgrade Yetus to 0.13.0
[
https://issues.apache.org/jira/browse/HADOOP-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299693#comment-17299693
]
Akira Ajisaka commented on HADOOP-16748:
{quote}Can we get that behaviour back?
{quote}
Fixed by HADOOP-17570. Thanks.
> Migrate to Python 3 and upgrade Yetus to 0.13.0
> ---
>
> Key: HADOOP-16748
> URL: https://issues.apache.org/jira/browse/HADOOP-16748
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17570) Apply YETUS-1102 to re-enable GitHub comments
[ https://issues.apache.org/jira/browse/HADOOP-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17570: --- Fix Version/s: 3.2.3 Backported to branch-3.2. > Apply YETUS-1102 to re-enable GitHub comments > - > > Key: HADOOP-17570 > URL: https://issues.apache.org/jira/browse/HADOOP-17570 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Yetus 0.13.0 enabled updating GitHub status instead of commenting the report, > however, the report comments are still useful for some cases. Let's apply > YETUS-1102 to re-enable the comments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16870) Use spotbugs-maven-plugin instead of findbugs-maven-plugin
[ https://issues.apache.org/jira/browse/HADOOP-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-16870: --- Fix Version/s: 3.2.3 3.3.1 Backported to branch-3.3 and branch-3.2. > Use spotbugs-maven-plugin instead of findbugs-maven-plugin > -- > > Key: HADOOP-16870 > URL: https://issues.apache.org/jira/browse/HADOOP-16870 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > findbugs-maven-plugin is no longer maintained. Use spotbugs-maven-plugin > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16308) ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup
[ https://issues.apache.org/jira/browse/HADOOP-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16308: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > ITestS3AContractSeek teardown closes test FS before superclass can do its > cleanup > - > > Key: HADOOP-16308 > URL: https://issues.apache.org/jira/browse/HADOOP-16308 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0, 3.1.2, 3.2.1 >Reporter: Steve Loughran >Priority: Minor > > the cleanup for the ITestS3AContractSeek now adds a stack trace to the logs > warning that the getFileSystem() Fs has been closed. Looks like it came from > the parquet seek fix patch -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16176) Add some tests about S3 timestamp tracking
[ https://issues.apache.org/jira/browse/HADOOP-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16176. - Resolution: Won't Fix > Add some tests about S3 timestamp tracking > -- > > Key: HADOOP-16176 > URL: https://issues.apache.org/jira/browse/HADOOP-16176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > We need to make sure we understand timestamp behaviour across S3 > implementations, for both single part and multipart uploads > tests to verify that the timestamp of files is consistent > * individual files's timestamps don't appear to change > * if you upload two small files in sequence, the timestamp of file 2> file 1 > MPU uploads are different: their timestamp may be that of the upload start, > rater than upload delete > A sequence of > # init MPU to file1 > # PUT to file 2 > # complete MPU to file1 > may result in timestamp(file1) < timestamp(file2) > but if you add a PUT to file 0 before step 1, then we require that > timestamp(file0) < timestamp(file1). That is: there's still an ordering on > initialization. > these tests are probably relevant to all MPU implementations, though they'll > offer different timing outcomes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?focusedWorklogId=564669&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-564669
]
ASF GitHub Bot logged work on HADOOP-16819:
---
Author: ASF GitHub Bot
Created on: 11/Mar/21 15:22
Start Date: 11/Mar/21 15:22
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #1894:
URL: https://github.com/apache/hadoop/pull/1894#issuecomment-796815264
looking at this with a goal of getting it in to 3.3.1.
Is the race condition in generating the key IDs? If so, isn't that enough to
sync and we can leave the rest alone?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 564669)
Time Spent: 20m (was: 10m)
> Possible inconsistent state of AbstractDelegationTokenSecretManager
> ---
>
> Key: HADOOP-16819
> URL: https://issues.apache.org/jira/browse/HADOOP-16819
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, security
>Affects Versions: 3.3.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-16819.001.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> [AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360]
> increments the current key id and creates the new delegation key in two
> distinct synchronized blocks.
> This means that other threads can see the class in an *inconsistent state,
> where the key for the current key id doesn't exist (yet)*.
> For example the following method sometimes returns null when the token
> remover thread is between the two synchronized blocks:
> {noformat}
> @Override
> public DelegationKey getCurrentKey() {
> return getDelegationKey(getCurrentKeyId());
> }{noformat}
>
> Also it is possible that updateCurrentKey is called from multiple threads at
> the same time so *distinct keys can be generated with the same key id*.
>
> This issue is suspected to be the cause of the intermittent failure of
> [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195]
> - HIVE-22621.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #1894: HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager
steveloughran commented on pull request #1894: URL: https://github.com/apache/hadoop/pull/1894#issuecomment-796815264 looking at this with a goal of getting it in to 3.3.1. Is the race condition in generating the key IDs? If so, isn't that enough to sync and we can leave the rest alone? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?focusedWorklogId=564668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-564668
]
ASF GitHub Bot logged work on HADOOP-16819:
---
Author: ASF GitHub Bot
Created on: 11/Mar/21 15:21
Start Date: 11/Mar/21 15:21
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#1894:
URL: https://github.com/apache/hadoop/pull/1894#discussion_r592451348
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##
@@ -356,16 +356,14 @@ private void updateCurrentKey() throws IOException {
int newCurrentId;
synchronized (this) {
newCurrentId = incrementCurrentKeyId();
-}
-DelegationKey newKey = new DelegationKey(newCurrentId, System
-.currentTimeMillis()
-+ keyUpdateInterval + tokenMaxLifetime, generateSecret());
-//Log must be invoked outside the lock on 'this'
-logUpdateMasterKey(newKey);
-synchronized (this) {
- currentKey = newKey;
+ currentKey = new DelegationKey(newCurrentId, System
+ .currentTimeMillis()
+ + keyUpdateInterval + tokenMaxLifetime, generateSecret());
+
storeDelegationKey(currentKey);
}
+//Log must be invoked outside the lock on 'this'
+logUpdateMasterKey(currentKey);
Review comment:
so this is now happening after the store? And both generateSecret and
store are now synchronized?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 564668)
Remaining Estimate: 0h
Time Spent: 10m
> Possible inconsistent state of AbstractDelegationTokenSecretManager
> ---
>
> Key: HADOOP-16819
> URL: https://issues.apache.org/jira/browse/HADOOP-16819
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, security
>Affects Versions: 3.3.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Attachments: HADOOP-16819.001.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360]
> increments the current key id and creates the new delegation key in two
> distinct synchronized blocks.
> This means that other threads can see the class in an *inconsistent state,
> where the key for the current key id doesn't exist (yet)*.
> For example the following method sometimes returns null when the token
> remover thread is between the two synchronized blocks:
> {noformat}
> @Override
> public DelegationKey getCurrentKey() {
> return getDelegationKey(getCurrentKeyId());
> }{noformat}
>
> Also it is possible that updateCurrentKey is called from multiple threads at
> the same time so *distinct keys can be generated with the same key id*.
>
> This issue is suspected to be the cause of the intermittent failure of
> [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195]
> - HIVE-22621.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16819) Possible inconsistent state of AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HADOOP-16819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-16819:
Labels: pull-request-available (was: )
> Possible inconsistent state of AbstractDelegationTokenSecretManager
> ---
>
> Key: HADOOP-16819
> URL: https://issues.apache.org/jira/browse/HADOOP-16819
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, security
>Affects Versions: 3.3.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-16819.001.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [AbstractDelegationTokenSecretManager.updateCurrentKey|https://github.com/apache/hadoop/blob/581072a8f04f7568d3560f105fd1988d3acc9e54/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java#L360]
> increments the current key id and creates the new delegation key in two
> distinct synchronized blocks.
> This means that other threads can see the class in an *inconsistent state,
> where the key for the current key id doesn't exist (yet)*.
> For example the following method sometimes returns null when the token
> remover thread is between the two synchronized blocks:
> {noformat}
> @Override
> public DelegationKey getCurrentKey() {
> return getDelegationKey(getCurrentKeyId());
> }{noformat}
>
> Also it is possible that updateCurrentKey is called from multiple threads at
> the same time so *distinct keys can be generated with the same key id*.
>
> This issue is suspected to be the cause of the intermittent failure of
> [TestLlapSignerImpl.testSigning|https://github.com/apache/hive/blob/3c0705eaf5121c7b61f2dbe9db9545c3926f26f1/llap-server/src/test/org/apache/hadoop/hive/llap/security/TestLlapSignerImpl.java#L195]
> - HIVE-22621.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1894: HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager
steveloughran commented on a change in pull request #1894:
URL: https://github.com/apache/hadoop/pull/1894#discussion_r592451348
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
##
@@ -356,16 +356,14 @@ private void updateCurrentKey() throws IOException {
int newCurrentId;
synchronized (this) {
newCurrentId = incrementCurrentKeyId();
-}
-DelegationKey newKey = new DelegationKey(newCurrentId, System
-.currentTimeMillis()
-+ keyUpdateInterval + tokenMaxLifetime, generateSecret());
-//Log must be invoked outside the lock on 'this'
-logUpdateMasterKey(newKey);
-synchronized (this) {
- currentKey = newKey;
+ currentKey = new DelegationKey(newCurrentId, System
+ .currentTimeMillis()
+ + keyUpdateInterval + tokenMaxLifetime, generateSecret());
+
storeDelegationKey(currentKey);
}
+//Log must be invoked outside the lock on 'this'
+logUpdateMasterKey(currentKey);
Review comment:
so this is now happening after the store? And both generateSecret and
store are now synchronized?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14937) initial part uploads seem to block unnecessarily in S3ABlockOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14937.
-
Resolution: Cannot Reproduce
> initial part uploads seem to block unnecessarily in S3ABlockOutputStream
>
>
> Key: HADOOP-14937
> URL: https://issues.apache.org/jira/browse/HADOOP-14937
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steven Rand
>Assignee: Steven Rand
>Priority: Major
> Attachments: yjp_threads.png
>
>
> From looking at a YourKit snapshot of an FsShell process running a {{hadoop
> fs -put file:///... s3a://...}}, it seems that the first part in the
> multipart upload doesn't begin to upload until n of the
> {{s3a-transfer-shared-pool}} threads are able to start uploading, where n is
> the value of {{fs.s3a.fast.upload.active.blocks}}.
> To hopefully clarify a bit, the series of events that I expected to see with
> {{fs.s3a.fast.upload.active.blocks}} set to 4 is:
> 1. An amount of data equal to {{fs.s3a.multipart.size}} is buffered into
> off-heap memory (I have {{fs.s3a.fast.upload.buffer = bytebuffer}}).
> 2. As soon as that happens, a thread begins to upload that part. Meanwhile,
> the main thread continues to buffer data into off-heap memory.
> 3. Once another part has been buffered into off-heap memory, a separate
> thread uploads that part, and so on.
> Whereas what I think the YK snapshot shows happening is:
> 1. An amount of data equal to {{fs.s3a.multipart.size}} * 4 is buffered into
> off-heap memory.
> 2. Four threads start to upload one part each at the same time.
> I've attached a picture of the "Threads" tab to show what I mean. Basically
> the times at which the first four {{s3a-transfer-shared-pool}} threads start
> to upload are roughly the same, whereas I would've expected them to be more
> staggered.
> I'm actually not sure whether this is the expected behavior or not, so feel
> free to close if this doesn't come as a surprise to anyone.
> For some context, I've been trying to get a sense for roughly which values of
> {{fs.s3a.multipart.size}} perform the best at different file sizes. One thing
> that I found confusing is that a part size of 5 MB seems to outperform a part
> size of 64 MB up until files that are upwards of about 500 MB in size. This
> seems odd, since each {{uploadPart}} call is its own HTTP request, and I
> would've expected the overhead of those to become costly at small part sizes.
> My suspicion is that with 4 concurrent part uploads and 64 MB blocks, we have
> to wait until 256 MB are buffered before we can start uploading, while with 5
> MB blocks we can start uploading as soon as we buffer 20 MB, and that's what
> gives the smaller parts the advantage for smaller files.
> I'm happy to submit a patch if this is in fact a problem, but wanted to check
> to make sure I'm not just misunderstanding something.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14510) Use error code detail in AWS server responses for finer grained exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14510.
-
Resolution: Won't Fix
> Use error code detail in AWS server responses for finer grained exceptions
> --
>
> Key: HADOOP-14510
> URL: https://issues.apache.org/jira/browse/HADOOP-14510
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Major
>
> {{S3Utils.translateException()}} maps HTTP status code to exceptions. We
> aren't looking at the body of the reponses though, except when handling a 301
> redirect.
> We should use the exit code to fine tune responses, especially 400 & 401/403.
> Right now I'm not sure we are even getting that error code into the text.
> see: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14070) S3a: Failed to reset the request input stream/make S3A uploadPart() retriable
[
https://issues.apache.org/jira/browse/HADOOP-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14070:
Parent Issue: HADOOP-17566 (was: HADOOP-16829)
> S3a: Failed to reset the request input stream/make S3A uploadPart() retriable
> -
>
> Key: HADOOP-14070
> URL: https://issues.apache.org/jira/browse/HADOOP-14070
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha2
>Reporter: Seth Fitzsimmons
>Priority: Major
>
> {code}
> Feb 07, 2017 8:05:46 AM
> com.google.common.util.concurrent.Futures$CombinedFuture
> setExceptionAndMaybeLog
> SEVERE: input future failed.
> com.amazonaws.ResetException: Failed to reset the request input stream; If
> the request involves an input stream, the maximum stream buffer size can be
> configured via request.getRequestClientOptions().setReadLimit(int)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1221)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1042)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4041)
> at
> com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3041)
> at
> com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3026)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.uploadPart(S3AFileSystem.java:1114)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload$1.call(S3ABlockOutputStream.java:501)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload$1.call(S3ABlockOutputStream.java:492)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1219)
> at
> org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Resetting to invalid mark
> at java.io.BufferedInputStream.reset(BufferedInputStream.java:448)
> at
> com.amazonaws.internal.SdkBufferedInputStream.reset(SdkBufferedInputStream.java:106)
> at
> com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
> at com.amazonaws.event.ProgressInputStream.reset(ProgressInputStream.java:169)
> at
> com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:102)
> at
> org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
> ... 20 more
> 2017-02-07 08:05:46 WARN S3AInstrumentation:777 - Closing output stream
> statistics while data is still marked as pending upload in
> OutputStreamStatistics{blocksSubmitted=519, blocksInQueue=0, blocksActive=1,
> blockUploadsCompleted=518, blockUploadsFailed=2, bytesPendingUpload=82528300,
> bytesUploaded=54316236800, blocksAllocated=519, blocksReleased=519,
> blocksActivelyAllocated=0, exceptionsInMultipartFinalize=0,
> transferDuration=2637812 ms, queueDuration=839 ms, averageQueueTime=1 ms,
> totalUploadDuration=2638651 ms, effectiveBandwidth=2.05848506680118E7 bytes/s}
> Exception in thread "main" org.apache.hadoop.fs.s3a.AWSClientIOException:
> Multi-part upload with id
> 'uDonLgtsyeToSmhyZuNb7YrubCDiyXCCQy4mdVc5ZmYWPPHyZ3H3ZlFZzKktaPUiYb7uT4.oM.lcyoazHF7W8pK4xWmXV4RWmIYGYYhN6m25nWRrBEE9DcJHcgIhFD8xd7EKIjijEd1k4S5JY1HQvA--'
> to 2017/history-170130.orc on 2017/history-170130.orc:
> com.amazonaws.ResetException: Failed to reset the request input stream; If
> the request involves an input stream, the maximum stream buffer size can be
> configured via request.getRequestClientOptions().setReadLimit(int): Failed to
> reset the request input stream; If the request involves an input stream, the
> maximum stream buffer size can be configured via
> requ
[jira] [Updated] (HADOOP-16522) Encrypt S3A buffered data on disk
[
https://issues.apache.org/jira/browse/HADOOP-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16522:
Parent Issue: HADOOP-17566 (was: HADOOP-16829)
> Encrypt S3A buffered data on disk
> -
>
> Key: HADOOP-16522
> URL: https://issues.apache.org/jira/browse/HADOOP-16522
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Mike Yoder
>Priority: Major
>
> This came out of discussions with [~ste...@apache.org], [~irashid] and
> [~vanzin].
> Imran:
> {quote}
> Steve pointed out to me that the s3 libraries buffer data to disk. This is
> pretty much arbitrary user data.
>
> Spark has some settings to encrypt data that it writes to local disk (shuffle
> files etc.). Spark never has control of what arbitrary libraries are doing
> with data, so it doesn't guarantee that nothing ever ends up on disk -- but
> to the end user, they'd view those s3 libraries as part of the same system.
> So if a user is turning on spark's local-disk encryption, the users would be
> pretty surprised to find out that the data they're writing to S3 ends up on
> local-disk, unencrypted.
> {quote}
> Me:
> {quote}
> ... Regardless, this is still an s3a bug.
> {quote}
>
> Steve:
> {quote}
> I disagree
> we need to save intermediate data "somewhere" -people get a choice of disk or
> memory.
> encrypting data on disk was never considered as needed, on the basis that
> anyone malicious with read access under your home dir could lift the hadoop
> token file which YARN provides and so have full R/W access to all your data
> in the cluster filesystems until those tokens expire. If you don't have a
> good story there then the buffering of a few tens of MB of data during upload
> is a detail.
> There's also the extra complication that when uploading file blocks, we pass
> in the filename to the AWS SDK and let it do the uploads, rather than create
> the output stream; the SDK code has, in the past, been better at recovering
> failures there than output stream + mark and reset. that was a while back;
> things may change. But it is why I'd prefer any encrypted temp store as a new
> buffer option, rather than just silently change the "disk" buffer option to
> encrypt
> Be interesting to see where else in the code this needs to be addressed; I'd
> recommend looking at all uses if org.apache.hadoop.fs.LocalDirAllocator and
> making sure that Spark YARN launch+execute didn't use this indirectly
> JIRAs under HADOOP-15620 welcome; do look at the test policy in the
> hadoop-aws docs; we'd need a new subclass of AbstractSTestS3AHugeFiles for
> integration testing a different buffering option, plus whatever unit tests
> the encryption itself needed.
> {quote}
> Me:
> {quote}
> I get it. But ... there are a couple of subtleties here. One is that the
> tokens expire, while the data is still data. (This might or might not matter,
> depending on the threat...) Another is that customer policies in this area do
> not always align well with common sense. There are blanket policies like
> "data shall never be written to disk unencrypted" which we have come up
> against, which we'd like to be able to honestly answer in the affirmative.
> We have encrypted MR shuffle as one historical example, and encrypted impala
> memory spills as another.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14132) Filesystem discovery to stop loading implementation classes
[ https://issues.apache.org/jira/browse/HADOOP-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14132: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > Filesystem discovery to stop loading implementation classes > --- > > Key: HADOOP-14132 > URL: https://issues.apache.org/jira/browse/HADOOP-14132 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Priority: Major > > Integration testing of Hadoop with the HADOOP-14040 has shown up that the > move to a shaded AWS JAR is slowing all hadoop client code down. > I believe this is due to how we use service discovery to identify FS > implementations: the implementation classes themselves are instantiated. > This has known problems today with classloading, but clearly impacts > performance too, especially with complex transitive dependencies unique to > the loaded class. > Proposed: have lightweight service declaration classes which implement an > interface declaring > # schema > # classname of FileSystem impl > # classname of AbstractFS impl > # homepage (for third party code, support, etc) > These are what we register and scan in the FS to look for services. > This will leave the question about what to do for existing filesystems? I > think we'll need to retain the old code for external ones, while moving the > hadoop modules to the new ones -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15631) Remove transient dependency on hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HADOOP-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15631: Parent Issue: HADOOP-17566 (was: HADOOP-16829) > Remove transient dependency on hadoop-hdfs-client > - > > Key: HADOOP-15631 > URL: https://issues.apache.org/jira/browse/HADOOP-15631 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0, 3.2.0 >Reporter: Steve Loughran >Priority: Major > > When HADOOP-13786 included hadoop-mapreduce-client-core as provided, it > inadvertently added hadoop-hdfs-client as a transient dependency. Cut. > > This will require the classes required by the S3AMultipartUploader to be > moved from HDFS client into hadoop-common. Otherwise you can't use it unless > the hdfs lib is on the CP. Which I don't think HD/I deployments have -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
