steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2477006979
noticed this isn't in branch-3.4 (some merge issues)
where are we with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please lo
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2284327249
that or we abandon s3 transfer manager. that has been discussed *more than
once*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
singhpk234 commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2257342895
@steveloughran thanks for bringing this up, Thanks @adnanhemani for
connecting us.
For S3Access Grants we need SDK version 2.25.70 atleast for this as called
out here (they added Cop
singhpk234 commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2257341062
@steveloughran thanks for bringing this up, Thanks @adnanhemani for
connecting us.
For S3Access Grants we need SDK version 2.25.70 atleast for this as called
out here (they added Cop
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2257332416
Unfortunately, I think there are further changes that are still missing from
the AWS SDK to unblock all code paths. But I've moved off of this work recently
- I'll let the new owner co
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2242904433
@adnanhemani hadoop branch-3.4 AWS SDK is now at 2.25.53. That has
everything you need, doesn't it?
If you can do a request of this patch back part branch 3.4 I will merge it
steveloughran merged PR #6544:
URL: https://github.com/apache/hadoop/pull/6544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2007791751
Yup, this makes sense. I will ensure that Hadoop gets the SDK changes as
soon as the SDK updates are complete. Thank you again for all your time
reviewing this!
--
This is an automa
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2007781626
hey, if yetus is unhappy, rebase is the right thing to do, so don't worry
too much. just trying to remember where i was, that's all
--
This is an automated message from the Apache
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2007775835
Sorry! Yetus was complaining that it could not apply the changes on top of
`trunk` so I instinctively rebased and force pushed - will keep this in mind!
--
This is an automated messa
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-2007753756
please, please please, no force push once reviews have started unless
there's merge problems or its been neglected for too long...makes it harder to
seee what's changed between revie
hadoop-yetus commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1995867156
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reex
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1995387593
> You're right there is no rename; copy is all there is. So that is not
available yet? Hmmm. This isn't ready for production yet is it? Let us keep it
in trunk for now. The other strat
hadoop-yetus commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1995387553
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexe
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1523757683
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software F
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1523715465
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##
@@ -401,4 +406,22 @@ private static Region getS3RegionFromEndpoint(
steveloughran commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1522069895
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##
@@ -401,4 +406,22 @@ private static Region getS3RegionFromEndpoin
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1992515204
You're right there is no rename; copy is all there is. So that is not
available yet? Hmmm. This isn't ready for production yet is it? Let us keep it
in trunk for now. The other strat
hadoop-yetus commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1985634498
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reex
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1985378218
Hi Steve, thanks for reviewing this as you finished up with the Hadoop
release work, I really appreciate you taking the time to look at this. I'm
re-running the integration and unit te
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1517420760
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software F
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1517361600
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software F
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1517130806
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##
@@ -1624,4 +1624,21 @@ private Constants() {
* Value: {@value}.
*/
pu
steveloughran commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1516477473
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##
@@ -1624,4 +1624,21 @@ private Constants() {
* Value: {@value}.
*/
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1983988376
Test results. thanks for these. you only need failures. (we don't care about
successes unless something has got very slow)
* rebase on trunk; you need "HADOOP-19057. S3A: Lands
steveloughran commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1957375191
really focused purely on 3.4.0 shipping right now, not looking at stuff it
doesn't need. sorry
--
This is an automated message from the Apache Git Service.
To respond to the messag
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1947538335
Hi @steveloughran, any thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1936795463
Hi @ahmarsuhail, I've made all changes requested. Please take a look when
you can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
hadoop-yetus commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1936789090
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reex
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484871939
##
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:
##
@@ -614,6 +614,38 @@ If the following property is not set or set to `true`, the
following
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484857819
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software F
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484852608
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AccessGrantConfiguration.java:
##
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software F
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484851522
##
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:
##
@@ -614,6 +614,38 @@ If the following property is not set or set to `true`, the
following
adnanhemani commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484848907
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##
@@ -401,4 +411,19 @@ private static Region getS3RegionFromEndpoint(
ahmarsuhail commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1936082879
@adnanhemani re test failures, just updating the core-site won't be enough
for some of them, you'll also need the code changes in Steve's PR
https://github.com/apache/hadoop/pull/6515
ahmarsuhail commented on code in PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#discussion_r1484099683
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##
@@ -401,4 +411,19 @@ private static Region getS3RegionFromEndpoint(
hadoop-yetus commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1935575917
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexe
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1935468938
WRT ITest failures, this is the `auth-keys.xml` file I'm using as per the
suggestions on #6515. But I'm still getting the above failures - should this
not be the case?
Tested on
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1935462396
Hi @ahmarsuhail and @steveloughran - I've opened this with the new code that
has the S3 Access Grants plugin as part of the AWS Java SDK bundle (this is an
evolution of #6507). We shou
adnanhemani commented on PR #6544:
URL: https://github.com/apache/hadoop/pull/6544#issuecomment-1935460439
Unit tests and Tests results below, per my understanding all failures are
due to various other known reasons:
```
^C88665a0afd7e:hadoop-aws ahemani$ mvn verify -Dparallel-tests -D
adnanhemani opened a new pull request, #6544:
URL: https://github.com/apache/hadoop/pull/6544
### Description of PR
This adds S3 Access Grants support in S3A using the S3 Access Grants plugin,
which is a part of the AWS Java SDK bundle starting in v2.23.19. As part of
this PR, we
42 matches
Mail list logo
