wombatu-kun commented on code in PR #18524:
URL: https://github.com/apache/hudi/pull/18524#discussion_r3861355927
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,45 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# The opt-in analytics stream type of hadoop-aws 3.4.2+ also needs
analyticsaccelerator-s3,
Review Comment:
hadoop-aws 3.5.0 flipped `StreamIntegration.DEFAULT_STREAM_TYPE` to
`Analytics`, so for the 4.2 arm and the 4+ fallback the analytics stream is the
default rather than opt-in, and `spark_base` never fetches
`analyticsaccelerator-s3`. Worth pulling that jar in the SDK v2 branch (2.4 MB,
no transitive deps) or pinning `fs.s3a.input.stream.type=classic`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
