voonhous commented on code in PR #18524:
URL: https://github.com/apache/hudi/pull/18524#discussion_r3860944545
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,44 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# The opt-in analytics stream type of hadoop-aws 3.4.2+ also needs
analyticsaccelerator-s3,
+# not shipped here.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ;;
+ 4.1)
+ # Spark 4.1.x bundles Hadoop 3.4.2
+ HADOOP_AWS_VERSION="3.4.2"
+ AWS_SDK_VERSION="2.29.52"
+ ;;
+ 4.2)
Review Comment:
Refreshed: the mapping runs through 4.2 with the 4+ fallback, the 19888 line
is gone, the Kafka swap covers both files, and Impact is now 169-291 MB with
bundle 2.35.4 at 654 MB.
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,44 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# The opt-in analytics stream type of hadoop-aws 3.4.2+ also needs
analyticsaccelerator-s3,
+# not shipped here.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ;;
+ 4.1)
+ # Spark 4.1.x bundles Hadoop 3.4.2
+ HADOOP_AWS_VERSION="3.4.2"
+ AWS_SDK_VERSION="2.29.52"
+ ;;
+ 4.2)
+ # Spark 4.2.x bundles Hadoop 3.5.0
+ HADOOP_AWS_VERSION="3.5.0"
+ AWS_SDK_VERSION="2.35.4"
+ ;;
+ 4.*)
+ # Unmapped 4.x line: fall back to the newest mapped pairing and say so,
rather than
+ # silently shipping hadoop-aws from an older Hadoop line than the one
Spark bundles.
+ echo "Warning: no hadoop-aws mapping for Spark ${SPARK_VERSION}; using the
Spark 4.2 pairing" >&2
+ HADOOP_AWS_VERSION="3.5.0"
+ AWS_SDK_VERSION="2.35.4"
+ ;;
+ *)
Review Comment:
Good catch. The catch-all now branches on `SPARK_MAJOR >= 4` in 9b3a396: 4.3
and 5.0 warn and take the 4.2 pairing, 3.x keeps 3.3.4 / SDK v1. Dry-run for
3.5.3, 4.0.1, 4.1.3, 4.2.0, 4.3.0 and 5.0.0.
##########
docker/hoodie/hadoop/base_java17/Dockerfile:
##########
@@ -47,7 +47,9 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get -yq update \
&& mkdir /hadoop-data
COPY --from=hadoop-builder /opt/hadoop-${HADOOP_VERSION}
/opt/hadoop-${HADOOP_VERSION}
-RUN ln -s /opt/hadoop-${HADOOP_VERSION}/etc/hadoop /etc/hadoop
+# Hadoop 3.4.x ships mapred-site.xml directly; older releases only ship the
.template.
Review Comment:
Moved in 9b3a396: `base_java17` is back to master's plain symlink and
`base_java11` carries the guarded copy, so the README's `--hadoop-version
3.3.4` build no longer trips on the missing template. Checked
`hadoop-mapreduce-project/conf` at `rel/release-2.8.4` (template only) and
3.3.4 / 3.4.0 (`mapred-site.xml` only).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
