[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778196#comment-17778196
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6180:
URL: https://github.com/apache/hadoop/pull/6180#issuecomment-1773895787
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 39m 10s | | trunk passed |
| +1 :green_heart: | compile | 20m 28s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 17m 56s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 19m 30s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 19m 30s | | the patch passed |
| +1 :green_heart: | compile | 18m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 18m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 58s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6180/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 2m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 12s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 278m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6180/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6180 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 0c4ee99aaa89 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3009d67f3595dc0a33afd949959415b9f0d5c076 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778118#comment-17778118 ] ASF GitHub Bot commented on HADOOP-18889: - hadoop-yetus commented on PR #6186: URL: https://github.com/apache/hadoop/pull/6186#issuecomment-1773809861 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/6186 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6186 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6186/1/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776498#comment-17776498
]
ASF GitHub Bot commented on HADOOP-18889:
-
virajjasani commented on code in PR #6180:
URL: https://github.com/apache/hadoop/pull/6180#discussion_r1363235836
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java:
##
@@ -378,17 +388,39 @@ private static void initSigner(Configuration conf,
*/
private static void initRequestTimeout(Configuration conf,
ClientOverrideConfiguration.Builder clientConfig) {
-long requestTimeoutMillis = conf.getTimeDuration(REQUEST_TIMEOUT,
-DEFAULT_REQUEST_TIMEOUT, TimeUnit.SECONDS, TimeUnit.MILLISECONDS);
+final Duration requestTimeoutMillis = getDuration(conf, REQUEST_TIMEOUT,
+DEFAULT_REQUEST_TIMEOUT, TimeUnit.SECONDS);
-if (requestTimeoutMillis > Integer.MAX_VALUE) {
- LOG.debug("Request timeout is too high({} ms). Setting to {} ms instead",
- requestTimeoutMillis, Integer.MAX_VALUE);
- requestTimeoutMillis = Integer.MAX_VALUE;
+if (requestTimeoutMillis.toMillis() > 0) {
+ clientConfig.apiCallAttemptTimeout(requestTimeoutMillis);
}
+ }
-if(requestTimeoutMillis > 0) {
-
clientConfig.apiCallAttemptTimeout(Duration.ofMillis(requestTimeoutMillis));
+ /**
+ * Get duration. This may be negative; callers must check.
+ * If the config option is greater than {@code Integer.MAX_VALUE}
milliseconds,
+ * it is set to that max.
+ * Logs the value for diagnostics.
+ * @param conf config
+ * @param name option name
+ * @param defVal default value
+ * @param defaultUnit unit of default value
+ * @return duration. may be negative.
+ */
+ private static Duration getDuration(final Configuration conf,
Review Comment:
nit: `getDurationInMillis` or `getDurationWithMillis` might be better?
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java:
##
@@ -378,17 +388,39 @@ private static void initSigner(Configuration conf,
*/
private static void initRequestTimeout(Configuration conf,
ClientOverrideConfiguration.Builder clientConfig) {
-long requestTimeoutMillis = conf.getTimeDuration(REQUEST_TIMEOUT,
-DEFAULT_REQUEST_TIMEOUT, TimeUnit.SECONDS, TimeUnit.MILLISECONDS);
+final Duration requestTimeoutMillis = getDuration(conf, REQUEST_TIMEOUT,
+DEFAULT_REQUEST_TIMEOUT, TimeUnit.SECONDS);
-if (requestTimeoutMillis > Integer.MAX_VALUE) {
- LOG.debug("Request timeout is too high({} ms). Setting to {} ms instead",
- requestTimeoutMillis, Integer.MAX_VALUE);
- requestTimeoutMillis = Integer.MAX_VALUE;
+if (requestTimeoutMillis.toMillis() > 0) {
+ clientConfig.apiCallAttemptTimeout(requestTimeoutMillis);
}
+ }
-if(requestTimeoutMillis > 0) {
-
clientConfig.apiCallAttemptTimeout(Duration.ofMillis(requestTimeoutMillis));
+ /**
+ * Get duration. This may be negative; callers must check.
+ * If the config option is greater than {@code Integer.MAX_VALUE}
milliseconds,
+ * it is set to that max.
+ * Logs the value for diagnostics.
+ * @param conf config
+ * @param name option name
+ * @param defVal default value
+ * @param defaultUnit unit of default value
+ * @return duration. may be negative.
Review Comment:
nit: `Get duration` -> `Get duration object representing value in
milliseconds`?
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776488#comment-17776488 ] ASF GitHub Bot commented on HADOOP-18889: - virajjasani commented on code in PR #6180: URL: https://github.com/apache/hadoop/pull/6180#discussion_r1363203939 ## hadoop-common-project/hadoop-common/src/main/resources/core-default.xml: ## @@ -1596,14 +1596,15 @@ fs.s3a.connection.establish.timeout - 5000 - Socket connection setup timeout in milliseconds. + 5s + Socket connection setup timeout in milliseconds; this will be retried +more than once. fs.s3a.connection.timeout - 20 - Socket connection timeout in milliseconds. + 200s + Socket connection timeout. Review Comment: For both of them, is it valuable to mention in the description that milliseconds is the default unit unless user specifies any other unit explicitly? > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775014#comment-17775014
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6186:
URL: https://github.com/apache/hadoop/pull/6186#issuecomment-1761976145
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 22s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 133m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6186/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6186 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 18ab96580819 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20
08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fb7044d4031ba8eb9f6a398db7031d25d074fe19 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6186/1/testReport/ |
| Max. process+thread count | 750 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6186/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> S3A: V2 SDK client does not work with third-party store
> -
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774980#comment-17774980 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6186: URL: https://github.com/apache/hadoop/pull/6186#issuecomment-1761781131 test failure was against s3 london with ` -Dparallel-tests -DtestsThreadCount=10 ` but: the verifyCost call creates and initialises a span before calling innerGetFileStatus(); and the test works in an ide. so unless something is happening with setup() not binding span to the right span force *for precisely one test case*, there's no obvious source of this > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774976#comment-17774976
]
ASF GitHub Bot commented on HADOOP-18889:
-
steveloughran commented on PR #6186:
URL: https://github.com/apache/hadoop/pull/6186#issuecomment-1761765834
got a test failure: unaudited operation
```
[ERROR] Tests run: 42, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
164.026 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[ERROR]
testCostOfGetFileStatusOnFile[keep-markers](org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost)
Time elapsed: 3.971 s <<< ERROR!
java.nio.file.AccessDeniedException:
s3a://stevel-london/fork-0003/test/testCostOfGetFileStatusOnFile[keep-markers]:
org.apache.hadoop.fs.s3a.audit.AuditFailureException:
f5b8d4a0-f3a9-451d-bd0f-46e044313e61-0114 unaudited operation executing a
request outside an audit span {action_http_head_request
'fork-0003/test/testCostOfGetFileStatusOnFile[keep-markers]' size=0,
mutating=false}
at
org.apache.hadoop.fs.s3a.audit.AuditIntegration.translateAuditException(AuditIntegration.java:161)
at
org.apache.hadoop.fs.s3a.audit.AuditIntegration.maybeTranslateAuditException(AuditIntegration.java:175)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:188)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:152)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3956)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3859)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.innerGetFileStatus(S3ATestUtils.java:1476)
at
org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.lambda$verifyInnerGetFileStatus$4(AbstractS3ACostTest.java:403)
at
org.apache.hadoop.fs.s3a.performance.OperationCostValidator.exec(OperationCostValidator.java:167)
at
org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.verifyMetrics(AbstractS3ACostTest.java:295)
at
org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.verify(AbstractS3ACostTest.java:383)
at
org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest.verifyInnerGetFileStatus(AbstractS3ACostTest.java:402)
at
org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testCostOfGetFileStatusOnFile(ITestS3AFileOperationCost.java:199)
```
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774701#comment-17774701
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6180:
URL: https://github.com/apache/hadoop/pull/6180#issuecomment-1760389977
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 23s | | trunk passed |
| +1 :green_heart: | compile | 18m 17s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 47s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 17m 37s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 17m 37s | | the patch passed |
| +1 :green_heart: | compile | 16m 43s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 30s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6180/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 2m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 12s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 39s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 265m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6180/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6180 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux e52452bd0f81 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3009d67f3595dc0a33afd949959415b9f0d5c076 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd6
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774690#comment-17774690
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1760334728
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 26 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 52s | | trunk passed |
| +1 :green_heart: | compile | 17m 11s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 15m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 35s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 33m 0s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 15m 18s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 18s | | the patch passed |
| +1 :green_heart: | compile | 14m 53s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 14m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 5s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 26 unchanged - 2 fixed = 29 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 21s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 50s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not
generate ASF License warnings. |
| | | 236m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux fb58909410e9 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20
08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97992b0ed722122360a4a27b2bccc6a8b2fd7261 |
| Default Java | Private Build-1.8
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774638#comment-17774638
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1760090881
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 26 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 0s | | trunk passed |
| +1 :green_heart: | compile | 18m 39s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 15m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 47s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 16m 31s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 31s | | the patch passed |
| +1 :green_heart: | compile | 16m 8s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 25s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/11/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 26 unchanged - 2 fixed = 29 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 3s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 14s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 9s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 250m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 24f73ce3741a 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 13fbaf84645e3e0bfb08e0ce0e8e6172d7d70148 |
| Default Java | Private Build-1
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774591#comment-17774591 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1759929273 going to cut out the bits related to connection settings; needs its own review. specifically, i want the settings ```xml fs.s3a.connection.timeout 25000 fs.s3a.connection.establish.timeout 5000 ``` to map to millis, and i want tests to show this > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774537#comment-17774537
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1759668331
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 26 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 32s | | trunk passed |
| +1 :green_heart: | compile | 16m 11s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 14m 42s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 34s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 32m 59s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 15m 18s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 18s | | the patch passed |
| +1 :green_heart: | compile | 14m 46s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 14m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 5s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/10/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 26 unchanged - 2 fixed = 29 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 4s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 55s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 241m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 01792a3cec59 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20
08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 97992b0ed722122360a4a27b2bccc6a8b2fd7261 |
| Default Java | Private Build-1.8
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774247#comment-17774247
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1758604805
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 26 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 54s | | trunk passed |
| +1 :green_heart: | compile | 16m 48s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 15m 55s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 2s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 24s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 52s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 15m 51s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 51s | | the patch passed |
| +1 :green_heart: | compile | 15m 26s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 15m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 15s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/9/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 26 unchanged - 2 fixed = 29 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 8s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 6s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 245m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 674e50616597 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 970d1bf1be047a01226671e41d266b5a97a3284a |
| Default Java | Private Build-1.8
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774196#comment-17774196 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1758298518 once javadocs is happy i will merge this > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774146#comment-17774146
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1758142645
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 26 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 46s | | trunk passed |
| +1 :green_heart: | compile | 16m 55s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 17m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 5m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 59s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 27s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 16m 16s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 16s | | the patch passed |
| +1 :green_heart: | compile | 15m 55s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 15m 55s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/8/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 27s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 26 unchanged - 2 fixed = 29 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javadoc | 0m 54s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/8/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 4m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 20s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not
generate ASF License warnings. |
| | | 247m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranc
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773833#comment-17773833
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1756322339
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 26 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 39s | | trunk passed |
| +1 :green_heart: | compile | 17m 19s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 15m 19s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 8s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 29s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 58s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 15m 56s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 56s | | the patch passed |
| +1 :green_heart: | compile | 15m 56s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 15m 56s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/7/artifact/out/blanks-eol.txt)
| The patch has 9 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 16s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 27 unchanged - 1 fixed = 29 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javadoc | 0m 54s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/7/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 4m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 21s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 246m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranc
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773799#comment-17773799
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1756124797
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 48s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 46m 30s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/6/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 10m 49s |
[/branch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/6/artifact/out/branch-compile-root-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | compile | 16m 4s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 19s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 49s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 21m 26s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 21m 26s | | the patch passed |
| +1 :green_heart: | compile | 19m 4s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 19m 4s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/6/artifact/out/blanks-eol.txt)
| The patch has 9 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 19s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/6/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 27 unchanged - 1 fixed = 28 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| -1 :x: | javadoc | 0m 53s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/6/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| -1 :x: | spotbugs | 0m 30s |
[/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/6/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | shadedclient | 8m 25s | | patch has errors when building
and testing our client artifact
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773765#comment-17773765 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1755940107 fixed the test cleanup; one failure is ``` [INFO] [ERROR] Failures: [ERROR] ITestS3AContractUnbuffer>AbstractContractUnbufferTest.testUnbufferMultipleReads:114->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 failed to read expected number of bytes from stream. This may be transient expected:<256> but was:<124> [ERROR] Errors: [ERROR] ITestS3APrefetchingLruEviction.testSeeksWithLruEviction:162 » TestTimedOut tes... ``` > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773751#comment-17773751 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1755812653 failure of test run with -Dparallel-tests -DtestsThreadCount=10 -Dscale normally I run with count=8; wonder if this is a factor. but the huge stuff is done at the end, sequentially. or its a leftover of me cancelling a previous run. in which case i should add a new cleanup ``` [ERROR] Failures: [ERROR] ITestS3AHugeMagicCommits.test_030_postCreationAssertions:150 [Pending uploads] Expected size:<1> but was:<2> in: <["Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID GU0jUb2jczQjP9D0HPT1pbctBiRNC6iXXok03zCW7UsBsqOpwXQfamXwj_E.cMVRiUcmVXxbIUqDYUftzCADiTu6ReSjverMlkx9yWGPpwEF6ZXgD..FCiXE5q02zeCO; initiated 2023-10-10 16:46:50", "Upload to tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin with ID UctmfmGnoyg.F0CvKsScj5.PyFChQdYdiIaes5AMEADRrbLOgZMFoh53P1osgA5hmBR89mSaWl6lqkmLct._3wizqiHYAmpFZvEE9FSUxm.vRSsCb2WcBleQOGYObM4s; initiated 2023-10-10 17:00:53"]> [ERROR] Errors: [ERROR] ITestS3AHugeMagicCommits>AbstractSTestS3AHugeFiles.test_045_vectoredIOHugeFile:549->AbstractSTestS3AHugeFiles.assumeHugeFileExists:461->AbstractSTestS3AHugeFiles.assumeFileExists:471 » FileNotFound ``` > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773692#comment-17773692
]
ASF GitHub Bot commented on HADOOP-18889:
-
steveloughran commented on code in PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#discussion_r1352549885
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestDirectoryMarkerListing.java:
##
@@ -610,15 +611,18 @@ private void put(final String key, final String content)
throws Exception {
RequestBody.fromString(content)));
}
/**
- * Delete an object.
+ * Delete an object; exceptions are swallowed.
* @param key key
- * @param content string
*/
private void deleteObject(final String key) throws Exception {
-exec("DELETE " + key, () -> {
- s3client.deleteObject(b -> b.bucket(bucket).key(key));
- return "deleted " + key;
-});
+try {
+ exec("DELETE " + key, () -> {
+s3client.deleteObject(b -> b.bucket(bucket).key(key));
+return "deleted " + key;
+ });
+} catch (IOException ignored) {
Review Comment:
note we didn't see this running against the v1 sdk, but that may be test
coverage rather than sdk changes
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773346#comment-17773346
]
ASF GitHub Bot commented on HADOOP-18889:
-
steveloughran commented on code in PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#discussion_r1350383710
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestDirectoryMarkerListing.java:
##
@@ -610,15 +611,18 @@ private void put(final String key, final String content)
throws Exception {
RequestBody.fromString(content)));
}
/**
- * Delete an object.
+ * Delete an object; exceptions are swallowed.
* @param key key
- * @param content string
*/
private void deleteObject(final String key) throws Exception {
-exec("DELETE " + key, () -> {
- s3client.deleteObject(b -> b.bucket(bucket).key(key));
- return "deleted " + key;
-});
+try {
+ exec("DELETE " + key, () -> {
+s3client.deleteObject(b -> b.bucket(bucket).key(key));
+return "deleted " + key;
+ });
+} catch (IOException ignored) {
Review Comment:
google gcs raises 404 if the object doesn't exist.
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773235#comment-17773235
]
ASF GitHub Bot commented on HADOOP-18889:
-
ahmarsuhail commented on code in PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#discussion_r1350071549
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABucketExistence.java:
##
@@ -69,8 +72,14 @@ public void testNoBucketProbing() throws Exception {
assertTrue("getFileStatus on root should always return a directory",
fs.getFileStatus(root).isDirectory());
-expectUnknownStore(
-() -> fs.listStatus(root));
+try {
Review Comment:
I didn't understand why we're doing this..why will this listStatus call fail
with a third party store?
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestDirectoryMarkerListing.java:
##
@@ -610,15 +611,18 @@ private void put(final String key, final String content)
throws Exception {
RequestBody.fromString(content)));
}
/**
- * Delete an object.
+ * Delete an object; exceptions are swallowed.
* @param key key
- * @param content string
*/
private void deleteObject(final String key) throws Exception {
-exec("DELETE " + key, () -> {
- s3client.deleteObject(b -> b.bucket(bucket).key(key));
- return "deleted " + key;
-});
+try {
+ exec("DELETE " + key, () -> {
+s3client.deleteObject(b -> b.bucket(bucket).key(key));
+return "deleted " + key;
+ });
+} catch (IOException ignored) {
Review Comment:
why are we swallowing exceptions here?
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772705#comment-17772705
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1751458973
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 25 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 17s | | trunk passed |
| +1 :green_heart: | compile | 17m 14s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 12s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 8s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 7s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 35s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 16m 29s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 16m 29s | | the patch passed |
| +1 :green_heart: | compile | 15m 56s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 15m 56s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/5/artifact/out/blanks-eol.txt)
| The patch has 9 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 18s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 27 unchanged - 1 fixed = 28 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 246m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 5313599553ca 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772659#comment-17772659 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1751215211 latest iteration tested: s3 london at -Dscale and -Dprefetch *Error translation uses reflection to create IOE of the right type. * All IOEs at the bottom of an AWS stack chain are regenerated. * Troubleshooting doc updated with the new classnames; cutting out stack traces rather than bother recreating them. Pulled the multipart param stuff, which can be addressed in a separate JIRA. Where I also think we should move listUploads() to S3AInternals as it is very internal. But did tweak ITestS3AMultipartUtils while debugging this... leaving that in as better assertions are always good. > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772593#comment-17772593
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1750787286
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 23 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 50s | | trunk passed |
| +1 :green_heart: | compile | 17m 21s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 9s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 7s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 3s | | trunk passed |
| +1 :green_heart: | shadedclient | 30m 47s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 31m 16s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 16m 43s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 16m 43s | | the patch passed |
| +1 :green_heart: | compile | 15m 55s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 15m 55s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/4/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 24s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was
28) |
| +1 :green_heart: | mvnsite | 2m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 58s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 55s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 23s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 8s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 242m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 14967e03ae5e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772581#comment-17772581
]
Steve Loughran commented on HADOOP-18889:
-
update: i don't see connection refusal failing fast through the s3a connector,
not how it is being raised. I think it is because it is coming in as a
suppressed exception.
{code}
2023-10-06 14:54:40,515 [main] DEBUG shell.Command
(Command.java:displayError(476)) - ls failure
java.net.ConnectException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: Connect to localhost:443 [localhost/127.0.0.1] failed: Connection
refused (Connection refused): Connection refused (Connection refused)
at
org.apache.hadoop.fs.s3a.impl.ErrorHandling.maybeTranslateNetworkException(ErrorHandling.java:68)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:197)
at org.apache.hadoop.fs.s3a.Invoker.onceInTheFuture(Invoker.java:190)
at
org.apache.hadoop.fs.s3a.Listing$ObjectListingIterator.next(Listing.java:652)
at
org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator.requestNextBatch(Listing.java:431)
at
org.apache.hadoop.fs.s3a.Listing$FileStatusListingIterator.(Listing.java:373)
at
org.apache.hadoop.fs.s3a.Listing.createFileStatusListingIterator(Listing.java:144)
at
org.apache.hadoop.fs.s3a.Listing.getFileStatusesAssumingNonEmptyDir(Listing.java:265)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3572)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$21(S3AFileSystem.java:3530)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatusIterator$22(S3AFileSystem.java:3529)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2648)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2667)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatusIterator(S3AFileSystem.java:3527)
at
org.apache.hadoop.fs.shell.PathData.getDirectoryContentsIterator(PathData.java:296)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:456)
at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:242)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:300)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:284)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
at org.apache.hadoop.fs.shell.Command.run(Command.java:191)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to
execute HTTP request: Connect to localhost:443 [localhost/127.0.0.1] failed:
Connection refused (Connection refused)
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772328#comment-17772328 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on PR #6141: URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1749420283 new commit now works properly with GCS except for some known test failures! Docs cover the details...the key thing is s3a can handle it if you don't have the gcs-connector. No idea about performance diffs. I know gcs does some things with caching parquet footers etc, but s3a offers vector io. also i've somehow got multipart playing up. will look at. > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772322#comment-17772322 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on code in PR #6141: URL: https://github.com/apache/hadoop/pull/6141#discussion_r1347798899 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ErrorHandling.java: ## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a.impl; + +import java.io.IOException; +import java.net.ConnectException; +import java.net.NoRouteToHostException; +import java.net.UnknownHostException; + +/** Review Comment: update: already had a class ErrorTranslation. thats's where things are going > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772315#comment-17772315
]
ASF GitHub Bot commented on HADOOP-18889:
-
steveloughran commented on code in PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#discussion_r1347793691
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java:
##
@@ -552,6 +554,7 @@ public void testS3SpecificSignerOverride() throws Exception
{
config.set(SIGNING_ALGORITHM_STS, "CustomSTSSigner");
config.set(AWS_REGION, EU_WEST_1);
+disableFilesystemCaching(config);
Review Comment:
i was having problems which seemed to be recycling failures from previous
attempts, so I was adding in places where I had a problem
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17772240#comment-17772240
]
Steve Loughran commented on HADOOP-18889:
-
actually, the stack is
{code}
s3a://stevel-gcs/test/testDeleteOnExitDir/outOfOrderFile
2023-10-05 15:39:15,146 [JUnit-testDeleteOnExit] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:processDeleteOnExit(4234)) - The exception for deleteOnExit
is {}
org.apache.hadoop.fs.PathIOException: `s3a://stevel-gcs': FileSystem is closed!
at
org.apache.hadoop.fs.s3a.S3AFileSystem.checkNotClosed(S3AFileSystem.java:4313)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$13(S3AFileSystem.java:2901)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2895)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3977)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3Exists(S3AFileSystem.java:4026)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectoryIfNecessary(S3AFileSystem.java:3500)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.maybeCreateFakeParentDirectory(S3AFileSystem.java:3518)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteWithoutCloseCheck(S3AFileSystem.java:3469)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.processDeleteOnExit(S3AFileSystem.java:4231)
at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2706)
at org.apache.hadoop.fs.s3a.S3AFileSystem.close(S3AFileSystem.java:4254)
at
org.apache.hadoop.fs.s3a.ITestS3ADeleteOnExit.testDeleteOnExit(ITestS3ADeleteOnExit.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
{code}
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771587#comment-17771587
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1745482175
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 16 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 47s | | trunk passed |
| +1 :green_heart: | compile | 19m 18s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 1s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 19m 9s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 19m 9s | | the patch passed |
| +1 :green_heart: | compile | 17m 23s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 17m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 5m 1s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 9 new + 22 unchanged - 0 fixed = 31 total (was
22) |
| +1 :green_heart: | mvnsite | 2m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 56s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 4s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 257m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 843a19528357 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c0dbd40e2b434dd2ff9ba95bc6d7da6d116f081 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_3
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771585#comment-17771585
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1745476536
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 16 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 18m 48s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 53s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 5s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 18m 38s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 18m 38s | | the patch passed |
| +1 :green_heart: | compile | 17m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 17m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 29s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 9 new + 22 unchanged - 0 fixed = 31 total (was
22) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 50s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 54s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 1s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
generate ASF License warnings. |
| | | 254m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
markdownlint |
| uname | Linux 3ff33b973b65 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c0dbd40e2b434dd2ff9ba95bc6d7da6d116f081 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_3
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771579#comment-17771579 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran commented on code in PR #6141: URL: https://github.com/apache/hadoop/pull/6141#discussion_r1344497948 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ErrorHandling.java: ## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a.impl; + +import java.io.IOException; +import java.net.ConnectException; +import java.net.NoRouteToHostException; +import java.net.UnknownHostException; + +/** Review Comment: yes...a big chunk of S3AUtils is now just that mapping, fault extraction etc and isolation is good > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > Labels: pull-request-available > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771577#comment-17771577
]
ASF GitHub Bot commented on HADOOP-18889:
-
steveloughran commented on code in PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#discussion_r1344494324
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -2888,6 +2898,7 @@ protected S3ListResult listObjects(S3ListRequest request,
trackDurationOfOperation(trackerFactory,
OBJECT_LIST_REQUEST,
() -> {
+checkNotClosed(); // this listing is done in the thread pool,
it may actually be closed
Review Comment:
exactly: i managed to create a stack trace here when closing an fs client
spinning on UnknownHost
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Labels: pull-request-available
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771483#comment-17771483
]
ASF GitHub Bot commented on HADOOP-18889:
-
ahmarsuhail commented on code in PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#discussion_r1343755412
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/MultipartUtils.java:
##
@@ -122,9 +122,17 @@ static class ListingIterator implements
this.s3 = s3;
this.requestFactory = storeContext.getRequestFactory();
this.maxKeys = maxKeys;
- this.prefix = prefix;
this.invoker = storeContext.getInvoker();
this.auditSpan = storeContext.getActiveAuditSpan();
+ String p;
+ if (prefix == null) {
Review Comment:
would prefer to move this logic out of the constructor. We do something
similar in
[S3AFileSystem.listMultipartUploads](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L5231).
Maybe move to a method in S3AUtils and use that in both places.
Do you know why we were doing the prefix logic in
S3AFileSystem.listMultipartUploads and not here?
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -4293,7 +4304,7 @@ protected synchronized void stopAllServices() {
*/
private void checkNotClosed() throws IOException {
if (isClosed) {
- throw new IOException(uri + ": " + E_FS_CLOSED);
+ throw new PathIOException(uri.toString(), E_FS_CLOSED);
Review Comment:
nit: noticed that the java doc for this method is wrong, `Verify that the
input stream is open` .. can be updated to verify that the FS is open
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java:
##
@@ -228,8 +229,16 @@ protected Map, RetryPolicy>
createExceptionMap() {
// Treated as an immediate failure
policyMap.put(AWSBadRequestException.class, fail);
-// Status 500 error code is also treated as a connectivity problem
-policyMap.put(AWSStatus500Exception.class, connectivityFailure);
+// Status 5xx error code is an immediate failure
+// this is sign of a server-side problem, and while
+// rare in AWS S3, it does happen on third party stores.
+// (out of disk space, etc).
+// by the time we get here, the aws sdk will have
+// already retried.
+policyMap.put(AWSStatus500Exception.class, fail);
Review Comment:
nit: Can we mention here that this does not include throttling 503s, and
those are handled separately. the `Status 5xx` makes it slightly ambiguous
##
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md:
##
@@ -0,0 +1,388 @@
+
+
+# Working with Third-party S3 Stores
+
+The S3A connector works well worth a third-party S3 stores if the following
requirements are met:
+
+* It correctly implements the core S3 REST API, including support for uploads
clothes and the V2 listing API.
Review Comment:
typo on this line, remove `clothes` ?
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSStatus500Exception.java:
##
@@ -21,13 +21,13 @@
import software.amazon.awssdk.awscore.exception.AwsServiceException;
/**
- * A 500 response came back from a service.
- * This is considered probably retriable, That is, we assume
- *
- * whatever error happened in the service itself to have happened
- *before the infrastructure committed the operation.
- *Nothing else got through either.
- *
+ * A 5xx response came back from a service.
Review Comment:
would be good to mention throttling and 501s are handled separately
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java:
##
@@ -552,6 +554,7 @@ public void testS3SpecificSignerOverride() throws Exception
{
config.set(SIGNING_ALGORITHM_STS, "CustomSTSSigner");
config.set(AWS_REGION, EU_WEST_1);
+disableFilesystemCaching(config);
Review Comment:
why do we need to do this here and not in other cases?
##
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/third_party_stores.md:
##
@@ -0,0 +1,388 @@
+
+
+# Working with Third-party S3 Stores
+
+The S3A connector works well worth a third-party S3 stores if the following
requirements are met:
Review Comment:
nit: typo, works well with
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -2888,6 +2898,7 @@ protected S3ListResult listObjects(S3ListRequest request,
trackDurationOfOperation(trackerFactory,
OBJECT_LIST_REQUEST,
() -> {
+checkNotClosed(); // this listing is done in the thread pool,
it may actually be closed
Review Comment:
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771208#comment-17771208
]
ASF GitHub Bot commented on HADOOP-18889:
-
hadoop-yetus commented on PR #6141:
URL: https://github.com/apache/hadoop/pull/6141#issuecomment-1743418221
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 16 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 22s | | trunk passed |
| +1 :green_heart: | compile | 17m 40s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 41s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 5m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 16m 33s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 16m 33s | | the patch passed |
| +1 :green_heart: | compile | 16m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 18s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/1/artifact/out/blanks-eol.txt)
| The patch has 14 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 21s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 9 new + 22 unchanged - 0 fixed = 31 total (was
22) |
| +1 :green_heart: | mvnsite | 2m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| -1 :x: | javadoc | 0m 55s |
[/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 4m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 40s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 58s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 249m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6141/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6141 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771078#comment-17771078 ] ASF GitHub Bot commented on HADOOP-18889: - steveloughran opened a new pull request, #6141: URL: https://github.com/apache/hadoop/pull/6141 1. handle getBucketLocation() unsupported 2. append / to marker tool searches if the first call fails. 3. downgrade change control errors a bit better. TODO: useful error text/docs 4. handle unknown storage classes. 5. Docs on how to bind to and test third party stores, including google gcs. Not doing the region logic; this is other test failures. Exception translation/resilience * new AWSUnsupportedFeatureException ex for unsupported/unavailable errors * handle 501 method unimplemented the same * all ex > 500 mapped to the AWSStatus500Exception * precondition errors handled a bit better * GCS throttle exception also recognised. * New class ErrorHandling to move our error handling logic too, initially somewhere for new stuff * ErrorHandling.maybeTranslateNetworkException() scans down an exception chain and if the inner one is of a known type - UnknownHostException - NoRouteToHostException - ConnectException Then a new exception of that specific type is created, with the top level ex its cause. This is done to retain the whole stack chain. * reduce the number of retries within the AWS SDK * and those of our S3ARetryPolicy The reduction in retries is because its clear when you try to create a bucket which doesn't resolve that the time for even an UnknownHostException to eventually fail over 90s, which then hit the s3a retry code. - Reducing the sdk retries means these escalate to our code better. - Cutting back on our own retries makes it a bit more responsive for most real deployments. - maybeTranslateNetworkException() and s3a retry policy means that UnknownHostException is recognised and fails fast. Lots of changes to the test code: * more switches to turn off test suites (acls, v1list, ...) * more resilience to different test failures * pull up skip() checks into config create phase, rather than waiting for fs to be instantiated (faster; keeps the fs cache clean) * tuning for new exceptions ### How was this patch tested? * AWS S3 london: no regressions observed. * a third party endpoint: showed troublespots, especially how handling of UnknownHostException needed to be radically improved. Gave me the stack traces for the docs too Docs cover google gcs binding; I've not tested that yet, but the settings came from a colleague who has connected the v1 sdk to it. A full test run against GCS should be something we require here as it supports a different subset of the s3 apis (no bulk delete, v1 list only...) ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768039#comment-17768039
]
Steve Loughran commented on HADOOP-18889:
-
npe will at least be fixed
{code}
^C^C2023-09-22 15:30:08,505 [main] DEBUG shell.Command
(Command.java:displayError(476)) - ls failure
org.apache.hadoop.fs.PathIOException: `s3a://nonexistent-bucket-example':
FileSystem is closed!
at
org.apache.hadoop.fs.s3a.S3AFileSystem.checkNotClosed(S3AFileSystem.java:4307)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$13(S3AFileSystem.java:2901)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2895)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$ListingOperationCallbacksImpl.lambda$listObjectsAsync$0(S3AFileSystem.java:2514)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
ls: `s3a://nonexistent-bucket-example': FileSystem is closed!
{code}
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768038#comment-17768038
]
Steve Loughran commented on HADOOP-18889:
-
+if the hostname is valid but no service there, sdk will retry a lot, as will
s3a (going to cut back on the sdk retries).
if you do this in a {{fs -list)) then try to interrupt the operation you get an
npe
{code}
^C^C^C^C^C^C^C^C^C^C^C^C2023-09-22 15:09:31,237 [main] DEBUG fs.FsShell
(FsShell.java:run(344)) - Error
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listObjects$13(S3AFileSystem.java:2904)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:2895)
at
org.apache.hadoop.fs.s3a.S3AFileSystem$ListingOperationCallbacksImpl.lambda$listObjectsAsync$0(S3AFileSystem.java:2514)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
-ls: Fatal internal error
java.lang.NullPointerException
{code}
this is happening because the connection refused retry logic is happening in
the async list, which doesn't check for the fs closed.
the process has effectively hung. even cutting back on retries its a slow
failure
{code}
minimums=((object_list_request.failures.min=0)
(op_get_file_status.min=1)
(op_glob_status.min=5)
(op_list_status.failures.min=35624));
maximums=((object_list_request.failures.max=140)
(op_get_file_status.max=1)
(op_glob_status.max=5)
(op_list_status.failures.max=35624));
means=((object_list_request.failures.mean=(samples=6, sum=164, mean=27.))
(op_get_file_status.mean=(samples=1, sum=1, mean=1.))
(op_glob_status.mean=(samples=1, sum=5, mean=5.))
(op_list_status.failures.mean=(samples=1, sum=35624, mean=35624.)));
{code}
conclusion: need to review retry and timeouts, and make sure there's no
excessive retry-round-retry stuff
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768032#comment-17768032 ] Steve Loughran commented on HADOOP-18889: - * pulling in HADOOP-18857 as its related * if I ask for a nonexistent bucket right now and i've set a region, then the operation fails fast ``` fs.s3a.bucket.nonexistent-bucket-example.access.key 0E264A0F-7A53-4102-9B2D-8E5F4A277CBC fs.s3a.bucket.nonexistent-bucket-example.secret.key BF8CD5FC-945A-42C1-ABDF-42915EE0DCE1 fs.s3a.bucket.nonexistent-bucket-example.endpoint.region internal ``` but remove that region and odd things happen > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767666#comment-17767666
]
Steve Loughran commented on HADOOP-18889:
-
[~ahmarsu] thanks; will look at tomorrow.
one little regression i've hit is that 3rd party stores may return 404 an abort
MPU request if the request has already been aborted...it's no longer idempotent
{code}
[ERROR]
testMultipartUploadAbort(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader)
Time elapsed: 11.787 s <<< ERROR!
java.io.FileNotFoundException: Aborting multipart upload ID
AQAAAYq4pVYYAJ8Ebo0GR4a3z_Yzdv14njmie8U on
fork-0005/test/testMultipartUploadAbort:
software.amazon.awssdk.services.s3.model.NoSuchUploadException: The specified
multipart upload does not exist. The upload ID might be invalid, or the
multipart upload might have been aborted or completed. (Service: S3, Status
Code: 404, Request ID: 0c07c871:18a4aedd1ab:35b98:1c16, Extended Request ID:
9578ba52d8520581c0645b2eb9ee12ccf205b96fbbe8f335826ea72de53a8fa7):NoSuchUpload
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:248)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.abortMultipartUpload(WriteOperationHelper.java:395)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.abortMultipartCommit(WriteOperationHelper.java:471)
at
org.apache.hadoop.fs.s3a.impl.S3AMultipartUploader.lambda$abort$3(S3AMultipartUploader.java:243)
at org.apache.hadoop.util.LambdaUtils.eval(LambdaUtils.java:52)
at
org.apache.hadoop.fs.s3a.impl.StoreContext.lambda$submit$0(StoreContext.java:374)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java
{code}
I don't intend to make any changes to production code here. the test in
question actually has a flag to indicate whether aborts are repeatable; hdfs
says no and so the FNFE raised there is accepted.
Options
# add another test switch to change behaviour
# document it's a known quirk and not worry about it. This is not a
production-time issue after all.
looking at option #2, personally.
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767567#comment-17767567 ] Ahmar Suhail commented on HADOOP-18889: --- [https://github.com/apache/hadoop/pull/6106] (still WIP), removes the region check. > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767563#comment-17767563 ] Steve Loughran commented on HADOOP-18889: - region checking of aa 3rd party bucket with no s3 equivalent raises unknown host exception which is then wrapped multiple times and lost stack trace. while there's nothing we can do about that directly, in ITestS3AEndpointRegion we can enable path style access for a different failure > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767265#comment-17767265 ] Steve Loughran commented on HADOOP-18889: - +500 errors are retried in the aws sdk with the usual backoff; if a store impl is rejecting an s3 call then s3a retry policy currently treat as a comms error and retries too (see https://github.com/aws/aws-sdk-java-v2/issues/3483 ). plan: failfast. +reject precondition (512) and unsupported media type. should probably look at the other 500x calls too and treat all as unrecoverable and map to our 500 error. simpler than adding new ones. but we should really have server-side change tracking to downgrade on a 512 after a warn once, as it would avoid people having to turn it off explicitly on ozone etc. Or at least we report the failure better, even though it'll surface on the first get in a stream (oh, and what about prefetching? we should just give up fetching at that point) > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766859#comment-17766859
]
Steve Loughran commented on HADOOP-18889:
-
and a few more to skip. one finding is that those exploring options
(encryption, storage class) only fail after the fs is created. Move the checks
into createConfiguration() and all will be good. or maybe we add an override
point in S3ABaseT
h2. TestS3ACannedACLs
{code}
[ERROR] testCreatedObjectsHaveACLs(org.apache.hadoop.fs.s3a.ITestS3ACannedACLs)
Time elapsed: 2.574 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: PUT 0-byte object on test:
The XML you provided was not well-formed or did not validate against our
published schema. (Service: Amazon S3; Status Code: 400; Error Code:
MalformedACLError; Request ID: 0c07c874:18a4af9cc87:2069a:2d; S3 Extended
Request ID: null; Proxy: null), S3 Extended Request ID: null:MalformedACLError:
The XML you provided was not well-formed or did not validate against our
published schema. (Service: Amazon S3; Status Code: 400; Error Code:
MalformedACLError; Request ID: 0c07c874:18a4af9cc87:2069a:2d; S3 Extended
Request ID: null; Proxy: null)
{code}
h2. ITestXAttrCost
{code}
[ERROR] testXAttrRoot(org.apache.hadoop.fs.s3a.impl.ITestXAttrCost) Time
elapsed: 1.959 s <<< FAILURE!
java.lang.AssertionError:
[Headers on root object]
Expecting:
<["header.Content-Length", "header.x-amz-storage-class"]>
to contain only:
<["header.Content-Length", "header.Content-Type"]>
elements not found:
<["header.Content-Type"]>
and elements not expected:
<["header.x-amz-storage-class"]>
{code}
think we should just make that test not worry about it.
h2. ITestS3AEncryptionSSEKMSDefaultKey
{code}
[ERROR]
testEncryptionSettingPropagation(org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey)
Time elapsed: 2.122 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: PUT 0-byte object on
fork-0001/test: Invalid Request (Service: Amazon S3; Status Code: 400; Error
Code: InvalidRequest; Request ID: 0c07c870:18a4aea24fc:1ffdb:323; S3 Extended
Request ID: null; Proxy: null), S3 Extended Request ID: null:InvalidRequest:
Invalid Request (Service: Amazon S3; Status Code: 400; Error Code:
InvalidRequest; Request ID:
{code}
h2. ITestS3AConcurrentOps timeouts
just a slow long haul test run against a store not being that responsive. just
need to mark as scale/increase timeouts.
{code}
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
1,733.925 s <<< FAILURE! - in
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[ERROR]
testThreadPoolCoolDown(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)
Time elapsed: 960.026 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 96
milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.testThreadPoolCoolDown(ITestS3AConcurrentOps.java:199)
at java.lang.Thread.run(Thread.java:750)
{code}
> S3A: V2 SDK client does not work with third-party store
> ---
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Ahmar Suhail
>Priority: Critical
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766751#comment-17766751 ] Steve Loughran commented on HADOOP-18889: - HADOOP-18908 to cover region stuff, this is for everything else > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[ https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764664#comment-17764664 ] Ahmar Suhail commented on HADOOP-18889: --- we can probably get rid of the region query now..as SDK V2 has cross region support as of v2.20.99, it wasn't there when we started this work. > S3A: V2 SDK client does not work with third-party store > --- > > Key: HADOOP-18889 > URL: https://issues.apache.org/jira/browse/HADOOP-18889 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Critical > > testing against an external store without specifying region now blows up > because the region is queried off eu-west-1. > What are we do to here? require the region setting *which wasn't needed > before? what even region do we provide for third party stores? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18889) S3A: V2 SDK client does not work with third-party store
[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764238#comment-17764238
]
Steve Loughran commented on HADOOP-18889:
-
{code}
2023-09-12 15:51:20,237 [main] INFO diag.StoreDiag
(StoreDurationInfo.java:close(137)) - Duration of Creating filesystem for
s3a://external-store/: 0:01:172
java.nio.file.AccessDeniedException: external-store: getS3Region on
external-store: software.amazon.awssdk.services.s3.model.S3Exception: null
(Service: S3, Status Code: 403, Request ID: B4X3E3K7JJMW31HT, Extended Request
ID:
11O4OGPp95JlbmEszl7NiiMBBL73AVpgO1XdjkSvZoyjslpWj8nATQJ/5SkzXw8W1Puz/bPZ0fg=):null
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:235)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getS3Region$4(S3AFileSystem.java:1039)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2631)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getS3Region(S3AFileSystem.java:1038)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:982)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:622)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3510)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:575)
at
org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:755)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:241)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:176)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:1171)
at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:1180)
at storediag.main(storediag.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: null (Service:
S3, Status Code: 403, Request ID: B4X3E3K7JJMW31HT, Extended Request ID:
11O4OGPp95JlbmEszl7NiiMBBL73AVpgO1XdjkSvZoyjslpWj8nATQJ/5SkzXw8W1Puz/bPZ0fg=)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
at
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95)
at
software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBui
