voonhous commented on code in PR #18524:
URL: https://github.com/apache/hudi/pull/18524#discussion_r3861422812
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,45 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# The opt-in analytics stream type of hadoop-aws 3.4.2+ also needs
analyticsaccelerator-s3,
Review Comment:
Good catch, thanks. Fixed in e66b619: spark_base takes a third build arg,
`ANALYTICS_ACCELERATOR_VERSION`, and fetches the jar when it is set. The script
pins 1.2.1 for hadoop-aws 3.4.2 (still opt-in there) and 1.3.1 for 3.5.0 and
the 4+ fallback, the versions the matching hadoop-project poms declare; Spark
4.0.x (3.4.1) and 3.x leave it empty since the stream type does not exist
there. Went with the jar rather than pinning `classic` so the default stream
actually works out of the box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
