Arsen Gumin created SPARK-49875:
-----------------------------------
Summary: error build spark 3.5.3 with
Key: SPARK-49875
URL: https://issues.apache.org/jira/browse/SPARK-49875
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 3.5.3
Environment: docker image in description
Reporter: Arsen Gumin
i try create make-distribution.sh and get error. maybe I did something wrong?
what did you try to do:
# Update scala-maven-plugin verison (all upper 4.8)
# Downgrade scala-maven-plugin (4.7.*)
# Build without my patch (patch -p1 < $SPARK_HOME/spark-42411.patch)
# change scala and java version(8,11,17)
{code:java}
// docker image with build
FROM eclipse-temurin:17
ENV MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"
ENV SCALA_VERSION=2.13
ENV SPARK_VERSION=3.5.3
ENV SPARK_HOME=/opt/spark
ENV
SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}.tgz
WORKDIR /opt
ARG spark_uid=185
RUN groupadd --system --gid=${spark_uid} spark && \
useradd --system --uid=${spark_uid} --gid=spark spark
RUN apt-get update; \
apt-get install -y wget patch gettext-base gnupg2 bash tini libc6
libpam-modules krb5-user libnss3 procps net-tools gosu libnss-wrapper; \
rm -rf /var/lib/apt/lists/*
RUN set -ex; \
mkdir -p $SPARK_HOME; \
wget -nv -O /opt/spark.tgz "$SPARK_TGZ_URL"; \
tar -zxf /opt/spark.tgz --strip-components=1 --directory=$SPARK_HOME; \
chown -R spark:spark .;
WORKDIR $SPARK_HOME
COPY mtls/spark-42411.patch .
RUN patch -p1 < $SPARK_HOME/spark-42411.patch
RUN ./dev/change-scala-version.sh ${SCALA_VERSION}
RUN ./dev/make-distribution.sh \
--name spark-mtls \
--pip \
-P"scala-${SCALA_VERSION}" \
-Pconnect \
-Pkubernetes
{code}
error
{code:java}
//
[INFO] Compiler bridge file:
/root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.9.5-bin_2.13.8__61.0-1.9.5_20230915T135648.jar
[INFO] Compiler bridge file is not installed yet
Downloading from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/scala-sbt/compiler-bridge_2.13/1.9.5/compiler-bridge_2.13-1.9.5-sources.jar
Downloaded from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/scala-sbt/compiler-bridge_2.13/1.9.5/compiler-bridge_2.13-1.9.5-sources.jar
(53 kB at 228 kB/s)
Downloading from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.pom
Downloaded from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.pom
(18 kB at 99 kB/s)
Downloading from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline-parent/3.22.0/jline-parent-3.22.0.pom
Downloaded from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline-parent/3.22.0/jline-parent-3.22.0.pom
(28 kB at 137 kB/s)
Downloading from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.jar
Downloaded from gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.jar
(1.1 MB at 1.2 MB/s)
warning: 4 deprecations (since 2.12.0)
warning: 1 deprecation (since 2.12.5)
warning: 2 deprecations (since 2.12.9)
warning: 2 deprecations (since 2.13)
warning: 8 deprecations (since 2.13.0)
warning: 1 deprecation (since 2.13.4)
warning: 18 deprecations in total; re-run with -deprecation for details
warning: 2 feature warnings; re-run with -feature for details
8 warnings
[INFO] Compiler bridge installed
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.5.3:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [03:47 min]
[INFO] Spark Project Sketch ............................... FAILURE [ 20.618 s]
[INFO] Spark Project Local DB ............................. SKIPPED
[INFO] Spark Project Networking ........................... SKIPPED
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Common Utils ......................... SKIPPED
[INFO] Spark Project Tags ................................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project SQL API .............................. SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] Spark Project Connect Server ....................... SKIPPED
[INFO] Spark Project Connect Common ....................... SKIPPED
[INFO] Spark Project Connect Client ....................... SKIPPED
[INFO] Spark Protobuf ..................................... SKIPPED
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:09 min
[INFO] Finished at: 2024-10-04T10:16:17Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:4.9.0:compile (scala-compile-first) on
project spark-sketch_2.12: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:4.9.0:compile failed:
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit
value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :spark-sketch_2.12
Error: building at STEP "RUN ./dev/make-distribution.sh --name spark-mtls
--pip -P"scala-${SCALA_VERSION}" -Pconnect -Pkubernetes
-Dscala-maven-plugin.version=4.9.0": while running runtime: exit status 1
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
