I downloaded the Spark 1.6.1 artifacts from the Apache mirror network and re-uploaded them to the spark-related-packages S3 bucket, so hopefully these packages should be fixed now.
On Mon, Apr 4, 2016 at 3:37 PM Nicholas Chammas <nicholas.cham...@gmail.com> wrote: > Thanks, that was the command. :thumbsup: > > On Mon, Apr 4, 2016 at 6:28 PM Jakob Odersky <ja...@odersky.com> wrote: > >> I just found out how the hash is calculated: >> >> gpg --print-md sha512 <spark-archive>.tgz >> >> you can use that to check if the resulting output matches the contents >> of <spark-archive>.tgz.sha >> >> On Mon, Apr 4, 2016 at 3:19 PM, Jakob Odersky <ja...@odersky.com> wrote: >> > The published hash is a SHA512. >> > >> > You can verify the integrity of the packages by running `sha512sum` on >> > the archive and comparing the computed hash with the published one. >> > Unfortunately however, I don't know what tool is used to generate the >> > hash and I can't reproduce the format, so I ended up manually >> > comparing the hashes. >> > >> > On Mon, Apr 4, 2016 at 2:39 PM, Nicholas Chammas >> > <nicholas.cham...@gmail.com> wrote: >> >> An additional note: The Spark packages being served off of CloudFront >> (i.e. >> >> the “direct download” option on spark.apache.org) are also corrupt. >> >> >> >> Btw what’s the correct way to verify the SHA of a Spark package? I’ve >> tried >> >> a few commands on working packages downloaded from Apache mirrors, but >> I >> >> can’t seem to reproduce the published SHA for >> spark-1.6.1-bin-hadoop2.6.tgz. >> >> >> >> >> >> On Mon, Apr 4, 2016 at 11:45 AM Ted Yu <yuzhih...@gmail.com> wrote: >> >>> >> >>> Maybe temporarily take out the artifacts on S3 before the root cause >> is >> >>> found. >> >>> >> >>> On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas >> >>> <nicholas.cham...@gmail.com> wrote: >> >>>> >> >>>> Just checking in on this again as the builds on S3 are still broken. >> :/ >> >>>> >> >>>> Could it have something to do with us moving release-build.sh? >> >>>> >> >>>> >> >>>> On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas >> >>>> <nicholas.cham...@gmail.com> wrote: >> >>>>> >> >>>>> Is someone going to retry fixing these packages? It's still a >> problem. >> >>>>> >> >>>>> Also, it would be good to understand why this is happening. >> >>>>> >> >>>>> On Fri, Mar 18, 2016 at 6:49 PM Jakob Odersky <ja...@odersky.com> >> wrote: >> >>>>>> >> >>>>>> I just realized you're using a different download site. Sorry for >> the >> >>>>>> confusion, the link I get for a direct download of Spark 1.6.1 / >> >>>>>> Hadoop 2.6 is >> >>>>>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >>>>>> On Fri, Mar 18, 2016 at 3:20 PM, Nicholas Chammas >> >>>>>> <nicholas.cham...@gmail.com> wrote: >> >>>>>> > I just retried the Spark 1.6.1 / Hadoop 2.6 download and got a >> >>>>>> > corrupt ZIP >> >>>>>> > file. >> >>>>>> > >> >>>>>> > Jakob, are you sure the ZIP unpacks correctly for you? Is it the >> same >> >>>>>> > Spark >> >>>>>> > 1.6.1/Hadoop 2.6 package you had a success with? >> >>>>>> > >> >>>>>> > On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky <ja...@odersky.com >> > >> >>>>>> > wrote: >> >>>>>> >> >> >>>>>> >> I just experienced the issue, however retrying the download a >> second >> >>>>>> >> time worked. Could it be that there is some load balancer/cache >> in >> >>>>>> >> front of the archive and some nodes still serve the corrupt >> >>>>>> >> packages? >> >>>>>> >> >> >>>>>> >> On Fri, Mar 18, 2016 at 8:00 AM, Nicholas Chammas >> >>>>>> >> <nicholas.cham...@gmail.com> wrote: >> >>>>>> >> > I'm seeing the same. :( >> >>>>>> >> > >> >>>>>> >> > On Fri, Mar 18, 2016 at 10:57 AM Ted Yu <yuzhih...@gmail.com> >> >>>>>> >> > wrote: >> >>>>>> >> >> >> >>>>>> >> >> I tried again this morning : >> >>>>>> >> >> >> >>>>>> >> >> $ wget >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >> --2016-03-18 07:55:30-- >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >> Resolving s3.amazonaws.com... 54.231.19.163 >> >>>>>> >> >> ... >> >>>>>> >> >> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >> >> >>>>>> >> >> gzip: stdin: unexpected end of file >> >>>>>> >> >> tar: Unexpected EOF in archive >> >>>>>> >> >> tar: Unexpected EOF in archive >> >>>>>> >> >> tar: Error is not recoverable: exiting now >> >>>>>> >> >> >> >>>>>> >> >> On Thu, Mar 17, 2016 at 8:57 AM, Michael Armbrust >> >>>>>> >> >> <mich...@databricks.com> >> >>>>>> >> >> wrote: >> >>>>>> >> >>> >> >>>>>> >> >>> Patrick reuploaded the artifacts, so it should be fixed now. >> >>>>>> >> >>> >> >>>>>> >> >>> On Mar 16, 2016 5:48 PM, "Nicholas Chammas" >> >>>>>> >> >>> <nicholas.cham...@gmail.com> >> >>>>>> >> >>> wrote: >> >>>>>> >> >>>> >> >>>>>> >> >>>> Looks like the other packages may also be corrupt. I’m >> getting >> >>>>>> >> >>>> the >> >>>>>> >> >>>> same >> >>>>>> >> >>>> error for the Spark 1.6.1 / Hadoop 2.4 package. >> >>>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>>> >> >>>> >> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz >> >>>>>> >> >>>> >> >>>>>> >> >>>> Nick >> >>>>>> >> >>>> >> >>>>>> >> >>>> >> >>>>>> >> >>>> On Wed, Mar 16, 2016 at 8:28 PM Ted Yu < >> yuzhih...@gmail.com> >> >>>>>> >> >>>> wrote: >> >>>>>> >> >>>>> >> >>>>>> >> >>>>> On Linux, I got: >> >>>>>> >> >>>>> >> >>>>>> >> >>>>> $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >>>>> >> >>>>>> >> >>>>> gzip: stdin: unexpected end of file >> >>>>>> >> >>>>> tar: Unexpected EOF in archive >> >>>>>> >> >>>>> tar: Unexpected EOF in archive >> >>>>>> >> >>>>> tar: Error is not recoverable: exiting now >> >>>>>> >> >>>>> >> >>>>>> >> >>>>> On Wed, Mar 16, 2016 at 5:15 PM, Nicholas Chammas >> >>>>>> >> >>>>> <nicholas.cham...@gmail.com> wrote: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> Does anyone else have trouble unzipping this? How did >> this >> >>>>>> >> >>>>>> happen? >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> What I get is: >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> $ gzip -t spark-1.6.1-bin-hadoop2.6.tgz >> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz: unexpected end of >> file >> >>>>>> >> >>>>>> gzip: spark-1.6.1-bin-hadoop2.6.tgz: uncompress failed >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> Seems like a strange type of problem to come across. >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> Nick >> >>>>>> >> >>>>> >> >>>>>> >> >>>>> >> >>>>>> >> >> >> >>>>>> >> > >> >>> >> >>> >> >> >> >
