mukund-thakur commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1277902721
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md:
##########
@@ -49,14 +137,67 @@ has been replaced by
[software.amazon.awssdk.auth.credentials.AwsCredentialsProv
changed.
The change in interface will mean that custom credential providers will need
to be updated to now
-implement `AwsCredentialsProvider` instead of `AWSCredentialProvider`.
+implement `software.amazon.awssdk.auth.credentials.AwsCredentialsProvider`
instead of
+`com.amazonaws.auth.AWSCredentialsProvider`.
+
+#### Original v1 `AWSCredentialsProvider` interface
+
+Note how the interface begins with the capitalized "AWS" acronym.
+The v2 interface starts with "Aws". This is a very subtle change
+for developers to spot.
+Compilers _will_ detect and report the type mismatch.
+
Review Comment:
this is repeated below.
##########
hadoop-project/pom.xml:
##########
@@ -1132,18 +1133,29 @@
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>${aws-java-sdk.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
<version>${aws-java-sdk-v2.version}</version>
<exclusions>
<exclusion>
- <groupId>io.netty</groupId>
+ <groupId>*</groupId>
Review Comment:
We are excluding everything as this is a big shaded jar correct?
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md:
##########
@@ -663,6 +679,24 @@ to the `get()` call: do it.
## Troubleshooting
+### `NoClassDefFoundError: software/amazon/eventstream/MessageDecoder`
+
+Select operation failing with a missing evenstream class.
+
+```
+java.io.IOException: java.lang.NoClassDefFoundError:
software/amazon/eventstream/MessageDecoder
Review Comment:
the eventstream dependency has been added in the hadoop-aws pom. Do we still
get this error? I have seen this too.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -70,14 +70,45 @@ These are Hadoop filesystem client classes, found in the
`hadoop-aws` JAR.
An exception reporting this class as missing means that this JAR is not on
the classpath.
-### `ClassNotFoundException: com.amazonaws.services.s3.AmazonS3Client`
-(or other `com.amazonaws` class.)
+### `NoClassDefFoundError: software/amazon/awssdk/crt/s3/S3MetaRequest`
+
+The library `aws-crt.jar` is not on the classpath. Its classes
+are not in the AWS `bundle.jar` file, yet are needed for uploading
+and renaming objects.
+
+Fix: add.
+
+```
+java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
software/amazon/awssdk/crt/s3/S3MetaRequest
+at
software.amazon.awssdk.services.s3.internal.crt.S3MetaRequestPauseObservable.<init>(S3MetaRequestPauseObservable.java:33)
+at
software.amazon.awssdk.transfer.s3.internal.DefaultS3TransferManager.uploadFile(DefaultS3TransferManager.java:205)
+at org.apache.hadoop.fs.s3a.S3AFileSystem.putObject(S3AFileSystem.java:3064)
+at org.apache.hadoop.fs.s3a.S3AFileSystem.executePut(S3AFileSystem.java:4054)
+
+```
+### `ClassNotFoundException: software.amazon.awssdk.services.s3.S3Client`
-This means that the `aws-java-sdk-bundle.jar` JAR is not on the classpath:
+(or other `software.amazon` class.)
+
+This means that the AWS V2 SDK `bundle.jar` JAR is not on the classpath:
add it.
Review Comment:
these jars are now added in the hadoop-aws pom by default. Do we still need
this documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
