Hi,
I am trying to build sparc from the source and not using Hive. I am getting [error] Required file not found: scala-compiler-2.10.4.jar [error] See zinc -help for information about locating necessary files I have to run this as root otherwise build does not progress. Any help is appreciated. -bash-3.2# ./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.6,parquet-provided" +++ dirname ./make-distribution.sh ++ cd . ++ pwd + SPARK_HOME=/usr/lib/spark + DISTDIR=/usr/lib/spark/dist + SPARK_TACHYON=false + TACHYON_VERSION=0.7.1 + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz + TACHYON_URL=https://github.com/amplab/tachyon/releases/download/v0.7.1/tachy on-0.7.1-bin.tar.gz + MAKE_TGZ=false + NAME=none + MVN=/usr/lib/spark/build/mvn + (( 4 )) + case $1 in + NAME=hadoop2-without-hive + shift + shift + (( 2 )) + case $1 in + MAKE_TGZ=true + shift + (( 1 )) + case $1 in + break + '[' -z /usr/java/latest ']' + '[' -z /usr/java/latest ']' ++ command -v git + '[' ']' ++ command -v /usr/lib/spark/build/mvn + '[' '!' /usr/lib/spark/build/mvn ']' ++ /usr/lib/spark/build/mvn help:evaluate -Dexpression=project.version -Pyarn,hadoop-provided,hadoop-2.6,parquet-provided ++ grep -v INFO ++ tail -n 1 + VERSION=1.5.2 ++ /usr/lib/spark/build/mvn help:evaluate -Dexpression=scala.binary.version -Pyarn,hadoop-provided,hadoop-2.6,parquet-provided ++ grep -v INFO ++ tail -n 1 + SCALA_VERSION=2.10 ++ /usr/lib/spark/build/mvn help:evaluate -Dexpression=hadoop.version -Pyarn,hadoop-provided,hadoop-2.6,parquet-provided ++ grep -v INFO ++ tail -n 1 + SPARK_HADOOP_VERSION=2.6.0 ++ /usr/lib/spark/build/mvn help:evaluate -Dexpression=project.activeProfiles -pl sql/hive -Pyarn,hadoop-provided,hadoop-2.6,parquet-provided ++ grep -v INFO ++ fgrep --count '<id>hive</id>' ++ echo -n + SPARK_HIVE=0 + '[' hadoop2-without-hive == none ']' + echo 'Spark version is 1.5.2' Spark version is 1.5.2 + '[' true == true ']' + echo 'Making spark-1.5.2-bin-hadoop2-without-hive.tgz' Making spark-1.5.2-bin-hadoop2-without-hive.tgz + '[' false == true ']' + echo 'Tachyon Disabled' Tachyon Disabled + cd /usr/lib/spark + export 'MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m' + MAVEN_OPTS='-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m' + BUILD_COMMAND=("$MVN" clean package -DskipTests $@) + echo -e '\nBuilding with...' Building with... + echo -e '$ /usr/lib/spark/build/mvn' clean package -DskipTests '-Pyarn,hadoop-provided,hadoop-2.6,parquet-provided\n' $ /usr/lib/spark/build/mvn clean package -DskipTests -Pyarn,hadoop-provided,hadoop-2.6,parquet-provided + /usr/lib/spark/build/mvn clean package -DskipTests -Pyarn,hadoop-provided,hadoop-2.6,parquet-provided Using `mvn` from path: /usr/lib/spark/build/apache-maven-3.3.3/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] Spark Project Parent POM [INFO] Spark Project Launcher [INFO] Spark Project Networking [INFO] Spark Project Shuffle Streaming Service [INFO] Spark Project Unsafe [INFO] Spark Project Core [INFO] Spark Project Bagel [INFO] Spark Project GraphX [INFO] Spark Project Streaming [INFO] Spark Project Catalyst [INFO] Spark Project SQL [INFO] Spark Project ML Library [INFO] Spark Project Tools [INFO] Spark Project Hive [INFO] Spark Project REPL [INFO] Spark Project YARN [INFO] Spark Project Assembly [INFO] Spark Project External Twitter [INFO] Spark Project External Flume Sink [INFO] Spark Project External Flume [INFO] Spark Project External Flume Assembly [INFO] Spark Project External MQTT [INFO] Spark Project External MQTT Assembly [INFO] Spark Project External ZeroMQ [INFO] Spark Project External Kafka [INFO] Spark Project Examples [INFO] Spark Project External Kafka Assembly [INFO] Spark Project YARN Shuffle Service [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Spark Project Parent POM 1.5.2 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ spark-parent_2.10 --- [INFO] Deleting /usr/lib/spark/target [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ spark-parent_2.10 --- [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-parent_2.10 --- [INFO] Add Source directory: /usr/lib/spark/src/main/scala [INFO] Add Test Source directory: /usr/lib/spark/src/test/scala [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent_2.10 --- [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-parent_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) @ spark-parent_2.10 --- [INFO] Executing tasks main: [mkdir] Created dir: /usr/lib/spark/target/tmp [INFO] Executed tasks [INFO] [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spark-parent_2.10 --- [INFO] No sources to compile [INFO] [INFO] --- maven-dependency-plugin:2.10:build-classpath (default) @ spark-parent_2.10 --- [INFO] [INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-parent_2.10 --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.6:test-jar (prepare-test-jar) @ spark-parent_2.10 --- [INFO] Building jar: /usr/lib/spark/target/spark-parent_2.10-1.5.2-tests.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-parent_2.10 --- [INFO] [INFO] --- maven-shade-plugin:2.4.1:shade (default) @ spark-parent_2.10 --- [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. [INFO] Replacing original artifact with shaded artifact. [INFO] [INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-parent_2.10 --- [INFO] [INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-parent_2.10 --- [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Spark Project Launcher 1.5.2 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ spark-launcher_2.10 --- [INFO] Deleting /usr/lib/spark/launcher/target [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ spark-launcher_2.10 --- [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-launcher_2.10 --- [INFO] Add Source directory: /usr/lib/spark/launcher/src/main/scala [INFO] Add Test Source directory: /usr/lib/spark/launcher/src/test/scala [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-launcher_2.10 --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-launcher_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /usr/lib/spark/launcher/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-launcher_2.10 --- [INFO] Using zinc server for incremental compilation [error] Required file not found: scala-compiler-2.10.4.jar [error] See zinc -help for information about locating necessary files [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 2.816 s] [INFO] Spark Project Launcher ............................. FAILURE [ 2.885 s] [INFO] Spark Project Networking ........................... SKIPPED [INFO] Spark Project Shuffle Streaming Service ............ SKIPPED [INFO] Spark Project Unsafe ............................... SKIPPED [INFO] Spark Project Core ................................. SKIPPED [INFO] Spark Project Bagel ................................ SKIPPED [INFO] Spark Project GraphX ............................... SKIPPED [INFO] Spark Project Streaming ............................ SKIPPED [INFO] Spark Project Catalyst ............................. SKIPPED [INFO] Spark Project SQL .................................. SKIPPED [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project YARN ................................. SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Project External Twitter ..................... SKIPPED [INFO] Spark Project External Flume Sink .................. SKIPPED [INFO] Spark Project External Flume ....................... SKIPPED [INFO] Spark Project External Flume Assembly .............. SKIPPED [INFO] Spark Project External MQTT ........................ SKIPPED [INFO] Spark Project External MQTT Assembly ............... SKIPPED [INFO] Spark Project External ZeroMQ ...................... SKIPPED [INFO] Spark Project External Kafka ....................... SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Project External Kafka Assembly .............. SKIPPED [INFO] Spark Project YARN Shuffle Service ................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 6.643 s [INFO] Finished at: 2015-11-25T18:20:09+00:00 [INFO] Final Memory: 66M/896M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-launcher_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-launcher_2.10 NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
