This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3175d83 [SPARK-38394][BUILD] Upgrade `scala-maven-plugin` to 4.4.0
for Hadoop 3 profile
3175d83 is described below
commit 3175d830cb029d41909de8960aa790d4272aa188
Author: Steve Loughran <ste...@cloudera.com>
AuthorDate: Sun Mar 6 19:23:31 2022 -0600
[SPARK-38394][BUILD] Upgrade `scala-maven-plugin` to 4.4.0 for Hadoop 3
profile
### What changes were proposed in this pull request?
This sets scala-maven-plugin.version to 4.4.0 except when the hadoop-2.7
profile is used, because SPARK-36547 shows that only 4.3.0 works there.
### Why are the changes needed?
1. If you try to build against a local snapshot of hadoop trunk with
`-Dhadoop.version=3.4.0-SNAPSHOT` the build failes with the error shown in the
JIRA.
2. upgrading the scala plugin version fixes this. It is a plugin issue.
3. the version is made configurable so the hadoop 2.7 profile can switch
back to the one which works there.
As to why this only surfaces when compiling hadoop trunk, or why hadoop-2.7
requires the new one -who knows. they both look certificate related, which is
interesting. maybe something related to signed JARs?
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
by successfully building spark against a local build of hadoop
3.4.0-SNAPSHOT
Closes #35725 from steveloughran/SPARK-38394-compiler-version.
Authored-by: Steve Loughran <ste...@cloudera.com>
Signed-off-by: Sean Owen <sro...@gmail.com>
---
pom.xml | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 176d3af..8e03167 100644
--- a/pom.xml
+++ b/pom.xml
@@ -163,6 +163,10 @@
<scala.version>2.12.15</scala.version>
<scala.binary.version>2.12</scala.binary.version>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
+ <!-- SPARK-36547: This needs to be managed in different profiles to avoid
+ errors building different Hadoop versions -->
+
+ <scala-maven-plugin.version>4.4.0</scala-maven-plugin.version>
<scalafmt.parameters>--test</scalafmt.parameters>
<!-- for now, not running scalafmt as part of default verify pipeline -->
<scalafmt.skip>true</scalafmt.skip>
@@ -2775,8 +2779,7 @@
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
- <!-- SPARK-36547: Please don't upgrade the version below, otherwise
there will be an error on building Hadoop 2.7 package -->
- <version>4.3.0</version>
+ <version>${scala-maven-plugin.version}</version>
<executions>
<execution>
<id>eclipse-add-source</id>
@@ -3430,6 +3433,7 @@
<hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
<hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
<hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ <scala-maven-plugin.version>4.3.0</scala-maven-plugin.version>
</properties>
</profile>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
