LuciferYang commented on code in PR #12902:
URL: https://github.com/apache/gluten/pull/12902#discussion_r3864070847
##########
dev/docker/ubuntu/Dockerfile.ubuntu22-m2-cache:
##########
@@ -42,7 +42,7 @@ RUN set -ex; \
dependency:go-offline -DskipTests || true; \
# Spark 4.0
- $MVN_CMD -Pjava-17,scala-2.13,spark-4,backends-velox,hadoop-3.3,spark-ut \
+ $MVN_CMD -Pjava-17,scala-2.13,spark-4.0,backends-velox,hadoop-3.3,spark-ut
\
Review Comment:
Drive-by fix, unrelated to Spark 3.3, flagged here because it sits in the
same `RUN` as the continuation fix below.
`spark-4` is not a profile id. The root pom defines `spark-4.0` and
`spark-4.1`, so Maven printed `The requested profile "spark-4" could not be
activated because it does not exist.` and carried on with the root defaults.
`enforce-spark-profile` does not catch it either: `dependency:go-offline`
invokes the plugin goal directly rather than running a lifecycle, and that
execution binds to `validate`, so it never fires. I confirmed the command exits
0 with only the warning.
Falling back to the defaults gives a mix that is mostly 3.5:
`sparkshim.artifactId` resolves to `spark-sql-columnar-shims-spark35`,
`sparkbundle.version` to 3.5, `delta.version` to 3.3.2. Only `spark.version`
happens to match, since the root default is already 4.0.2, so the Spark 4.0
jars did land in the cache while the Delta 4.0.1 and spark40 shim coordinates
did not. That is the whole cost: no red build, just a cache image that does not
hold what this stanza was meant to pre-populate for the 4.0 jobs.
Introduced in #11655. Happy to pull it into its own PR if you would rather
keep this one to the removal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
