I Recently downloaded spark package 1.4.0:
A build of Spark with "sbt/sbt clean assembly" failed with message "Error:
Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar"
Upon investigation I figured out that "sbt-launch-0.13.7.jar" is downloaded
at build time and that it contained the the following:
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
which is an HTML error message to the effect that the file is missing (from
the web server).
The script sbt-launch-lib.bash contains the following lines which determine
where the file sbt-launch.jar is downloaded from:
acquire_sbt_jar () {
SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}'
./project/build.properties`
URL1=
http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
URL2=
http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
JAR=build/sbt-launch-${SBT_VERSION}.jar
The script sbt-launch.bash downloads $URL1 first, and incorrectly concludes
that it succeeded on the basis that the file sbt-launch-0.13.7.jar exists
(though it contains HTML).
I succeeded in building Spark by:
(1) Downloading the file sbt-launch-0.13.7.jar from $URL2 and placing it
in the build directory.
(2) Modifying sbt-launch-lib.bash to prevent the download of that file.
(3) Restarting the download as I usually would, with "SPARK_HIVE=true
SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly"
I think a lot of people will be confused by this. Probably someone should
do some of the following:
(1) Delete $URL1 and all references, or replace it with the
correct/current URL which points to the sbt-launch.jar(s).
(2) Modify sbt-launch-lib.bash, so that it will not conclude that the
download of sbt-launch.jar succeeded, when the data returned is an HTML
error message.
Let me know if this is not clear, I will gladly explain in more detail or
with more clarity, if needed.
-Bryce Lobdell
A transcript of my console is below:
xxxxxxxx@ip-xx-xxx-xx-xxx:~/spark/spark-1.4.0$ SPARK_HIVE=true
SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly
NOTE: The sbt/sbt script has been relocated to build/sbt.
Please update references to point to the new location.
Invoking 'build/sbt clean assembly' now ...
Using /usr/lib/jvm/java-7-openjdk-amd64/ as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Attempting to fetch sbt
Launching sbt from build/sbt-launch-0.13.7.jar
*Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar*
inquidia@ip-10-102-69-107:~/spark/spark-1.4.0$ cd build/
inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
mvn sbt sbt-launch-0.13.7.jar sbt-launch-lib.bash
*inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ unzip -l
sbt-launch-0.13.7.jar*
*Archive: sbt-launch-0.13.7.jar*
* End-of-central-directory signature not found. Either this file is not*
* a zipfile, or it constitutes one disk of a multi-part archive. In the*
* latter case the central directory and zipfile comment will be found on*
* the last disk(s) of this archive.*
unzip: cannot find zipfile directory in one of sbt-launch-0.13.7.jar or
sbt-launch-0.13.7.jar.zip, and cannot find
sbt-launch-0.13.7.jar.ZIP, period.
inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
mvn sbt sbt-launch-0.13.7.jar sbt-launch-lib.bash
inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
total 28
-rwxr-xr-x 1 inquidia inquidia 5384 Jun 3 01:07 mvn
-rwxr-xr-x 1 inquidia inquidia 5395 Jun 3 01:07 sbt
-rw-rw-r-- 1 inquidia inquidia 162 Aug 7 20:24 sbt-launch-0.13.7.jar
-rwxr-xr-x 1 inquidia inquidia 5285 Jun 3 01:07 sbt-launch-lib.bash
inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
total 28
-rwxr-xr-x 1 inquidia inquidia 5384 Jun 3 01:07 mvn
-rwxr-xr-x 1 inquidia inquidia 5395 Jun 3 01:07 sbt
-rw-rw-r-- 1 inquidia inquidia *162 *Aug 7 20:24 sbt-launch-0.13.7.jar
-rwxr-xr-x 1 inquidia inquidia 5285 Jun 3 01:07 sbt-launch-lib.bash
inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ cat
sbt-launch-0.13.7.jar
*<html>*
*<head><title>404 Not Found</title></head>*
*<body bgcolor="white">*
*<center><h1>404 Not Found</h1></center>*
*<hr><center>nginx</center>*
*</body>*
*</html>*
