[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1297436980
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/V2Migration.java:
##
@@ -28,83 +28,47 @@
/**
* This class provides utility methods required for migrating S3A to AWS Java
SDK V2.
* For more information on the upgrade, see HADOOP-18073.
+ *
+ * in HADOOP-18382. Upgrade AWS SDK to V2 - Prerequisites,
+ * this class contained a series of `LogExactlyOnce` loggers to warn on
+ * the first use of a feature which would change incompatibly; this shipped in
Hadoop 3.3.5.
+ *
+ * With the move to v2 completed, attempts to use the v1 classes, will fail
+ * -except for the special case of support for v1 credential providers.
+ *
+ * The warning methods are still present, where appropriate, but downgraded to
debug
+ * and only retained for debugging migration issues.
*/
public final class V2Migration {
private V2Migration() { }
public static final Logger SDK_V2_UPGRADE_LOG =
LoggerFactory.getLogger(SDK_V2_UPGRADE_LOG_NAME);
- private static final LogExactlyOnce WARN_ON_DELEGATION_TOKENS =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
- private static final LogExactlyOnce WARN_ON_GET_S3_CLIENT =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
private static final LogExactlyOnce
WARN_OF_DIRECTLY_REFERENCED_CREDENTIAL_PROVIDER =
new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
- private static final LogExactlyOnce WARN_OF_CUSTOM_SIGNER =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
private static final LogExactlyOnce WARN_OF_REQUEST_HANDLERS =
new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
- private static final LogExactlyOnce WARN_ON_GET_OBJECT_METADATA =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
/**
- * Warns on an AWS V1 credential provider being referenced directly.
+ * Notes an AWS V1 credential provider being referenced directly.
* @param name name of the credential provider
*/
public static void v1ProviderReferenced(String name) {
Review Comment:
cut it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1297276732
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/V2Migration.java:
##
@@ -28,83 +28,47 @@
/**
* This class provides utility methods required for migrating S3A to AWS Java
SDK V2.
* For more information on the upgrade, see HADOOP-18073.
+ *
+ * in HADOOP-18382. Upgrade AWS SDK to V2 - Prerequisites,
+ * this class contained a series of `LogExactlyOnce` loggers to warn on
+ * the first use of a feature which would change incompatibly; this shipped in
Hadoop 3.3.5.
+ *
+ * With the move to v2 completed, attempts to use the v1 classes, will fail
+ * -except for the special case of support for v1 credential providers.
+ *
+ * The warning methods are still present, where appropriate, but downgraded to
debug
+ * and only retained for debugging migration issues.
*/
public final class V2Migration {
private V2Migration() { }
public static final Logger SDK_V2_UPGRADE_LOG =
LoggerFactory.getLogger(SDK_V2_UPGRADE_LOG_NAME);
- private static final LogExactlyOnce WARN_ON_DELEGATION_TOKENS =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
- private static final LogExactlyOnce WARN_ON_GET_S3_CLIENT =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
private static final LogExactlyOnce
WARN_OF_DIRECTLY_REFERENCED_CREDENTIAL_PROVIDER =
new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
- private static final LogExactlyOnce WARN_OF_CUSTOM_SIGNER =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
private static final LogExactlyOnce WARN_OF_REQUEST_HANDLERS =
new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
- private static final LogExactlyOnce WARN_ON_GET_OBJECT_METADATA =
- new LogExactlyOnce(SDK_V2_UPGRADE_LOG);
-
/**
- * Warns on an AWS V1 credential provider being referenced directly.
+ * Notes an AWS V1 credential provider being referenced directly.
* @param name name of the credential provider
*/
public static void v1ProviderReferenced(String name) {
Review Comment:
i'll look at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1297275400
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##
@@ -998,7 +999,7 @@ private void bindAWSClient(URI name, boolean dtEnabled)
throws IOException {
* @param parameters parameter object
* @throws IOException on any IO problem
*/
- private synchronized void createS3AsyncClient(S3ClientFactory clientFactory,
Review Comment:
i don't know WTF is up with spotbugs. it works on the command line for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1288761982
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java:
##
@@ -562,15 +563,65 @@ public static long dateToLong(final Date date) {
* @return instance of the specified class
* @throws IOException on any problem
*/
- @SuppressWarnings("unchecked")
- public static InstanceT getInstanceFromReflection(Class
instanceClass,
- Configuration conf, @Nullable URI uri, Class interfaceImplemented,
String methodName,
+ public static InstanceT getInstanceFromReflection(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1283235604 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md: ## @@ -65,7 +201,7 @@ The s3 client is an instance of `S3Client` in V2 rather than `AmazonS3`. For this reason, the `S3ClientFactory` will be deprecated and replaced by one that creates a V2 `S3Client`. -The `getAmazonS3ClientForTesting()` method will also be updated to return the `S3Client`. +The `getAmazonS3ClientForTesting()` method has been updated to return the `S3Client`. ### Signers Review Comment: will do. Also, I'm thinking of renaming this method S3ClientV2ForTesting(). Why so? you get a more meaningful "no such method" error rather than one about signature mismatch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1279497298 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md: ## @@ -49,14 +137,67 @@ has been replaced by [software.amazon.awssdk.auth.credentials.AwsCredentialsProv changed. The change in interface will mean that custom credential providers will need to be updated to now -implement `AwsCredentialsProvider` instead of `AWSCredentialProvider`. +implement `software.amazon.awssdk.auth.credentials.AwsCredentialsProvider` instead of +`com.amazonaws.auth.AWSCredentialsProvider`. + + Original v1 `AWSCredentialsProvider` interface + +Note how the interface begins with the capitalized "AWS" acronym. +The v2 interface starts with "Aws". This is a very subtle change +for developers to spot. +Compilers _will_ detect and report the type mismatch. + Review Comment: fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1279495750 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md: ## @@ -663,6 +679,24 @@ to the `get()` call: do it. ## Troubleshooting +### `NoClassDefFoundError: software/amazon/eventstream/MessageDecoder` + +Select operation failing with a missing evenstream class. + +``` +java.io.IOException: java.lang.NoClassDefFoundError: software/amazon/eventstream/MessageDecoder Review Comment: scoped to test; added a JIRA about whether to cut entirely. If people want it now, they need a new jar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1279492499
##
hadoop-project/pom.xml:
##
@@ -1132,18 +1133,29 @@
com.amazonaws
aws-java-sdk-core
${aws-java-sdk.version}
+
+
+*
+*
+
+
software.amazon.awssdk
bundle
${aws-java-sdk-v2.version}
-io.netty
+*
Review Comment:
correct. anything it declares a dependency on *should* be a false
dependency. except the aws-crt is required (transfer manager) and evenstream
for s3 select. both are bugs, IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on code in PR #5872: URL: https://github.com/apache/hadoop/pull/5872#discussion_r1279491510 ## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md: ## @@ -70,14 +70,45 @@ These are Hadoop filesystem client classes, found in the `hadoop-aws` JAR. An exception reporting this class as missing means that this JAR is not on the classpath. -### `ClassNotFoundException: com.amazonaws.services.s3.AmazonS3Client` -(or other `com.amazonaws` class.) +### `NoClassDefFoundError: software/amazon/awssdk/crt/s3/S3MetaRequest` + +The library `aws-crt.jar` is not on the classpath. Its classes +are not in the AWS `bundle.jar` file, yet are needed for uploading +and renaming objects. + +Fix: add. + +``` +java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: software/amazon/awssdk/crt/s3/S3MetaRequest +at software.amazon.awssdk.services.s3.internal.crt.S3MetaRequestPauseObservable.(S3MetaRequestPauseObservable.java:33) +at software.amazon.awssdk.transfer.s3.internal.DefaultS3TransferManager.uploadFile(DefaultS3TransferManager.java:205) +at org.apache.hadoop.fs.s3a.S3AFileSystem.putObject(S3AFileSystem.java:3064) +at org.apache.hadoop.fs.s3a.S3AFileSystem.executePut(S3AFileSystem.java:4054) + +``` +### `ClassNotFoundException: software.amazon.awssdk.services.s3.S3Client` -This means that the `aws-java-sdk-bundle.jar` JAR is not on the classpath: +(or other `software.amazon` class.) + +This means that the AWS V2 SDK `bundle.jar` JAR is not on the classpath: add it. Review Comment: yes, because people forget to include them when manually setting up their spark installations, where they just drop in a random pair of hadoop-aws and aws-sdk JARs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
