Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Gentle ping: spark-1.6.1-bin-hadoop2.4.tgz from S3 is still corrupt. On Wed, Apr 6, 2016 at 12:55 PM, Josh Rosen wrote: > Sure, I'll take a look. Planning to do full verification in a bit. > > On Wed, Apr 6, 2016 at 12:54 PM Ted Yu wrote: > >>

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Sure, I'll take a look. Planning to do full verification in a bit. On Wed, Apr 6, 2016 at 12:54 PM Ted Yu wrote: > Josh: > Can you check spark-1.6.1-bin-hadoop2.4.tgz ? > > $ tar zxf spark-1.6.1-bin-hadoop2.4.tgz > > gzip: stdin: not in gzip format > tar: Child returned

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Josh: Can you check spark-1.6.1-bin-hadoop2.4.tgz ? $ tar zxf spark-1.6.1-bin-hadoop2.4.tgz gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now $ ls -l !$ ls -l spark-1.6.1-bin-hadoop2.4.tgz -rw-r--r--. 1 hbase hadoop 323614720 Apr 5 19:25

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Thank you Josh! I confirmed that the Spark 1.6.1 / Hadoop 2.6 package on S3 is now working, and the SHA512 checks out. On Wed, Apr 6, 2016 at 3:19 PM Josh Rosen wrote: > I downloaded the Spark 1.6.1 artifacts from the Apache mirror network and > re-uploaded them to the

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I downloaded the Spark 1.6.1 artifacts from the Apache mirror network and re-uploaded them to the spark-related-packages S3 bucket, so hopefully these packages should be fixed now. On Mon, Apr 4, 2016 at 3:37 PM Nicholas Chammas wrote: > Thanks, that was the command.

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Thanks, that was the command. :thumbsup: On Mon, Apr 4, 2016 at 6:28 PM Jakob Odersky wrote: > I just found out how the hash is calculated: > > gpg --print-md sha512 .tgz > > you can use that to check if the resulting output matches the contents > of .tgz.sha > > On Mon, Apr

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I just found out how the hash is calculated: gpg --print-md sha512 .tgz you can use that to check if the resulting output matches the contents of .tgz.sha On Mon, Apr 4, 2016 at 3:19 PM, Jakob Odersky wrote: > The published hash is a SHA512. > > You can verify the integrity

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

The published hash is a SHA512. You can verify the integrity of the packages by running `sha512sum` on the archive and comparing the computed hash with the published one. Unfortunately however, I don't know what tool is used to generate the hash and I can't reproduce the format, so I ended up

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

An additional note: The Spark packages being served off of CloudFront (i.e. the “direct download” option on spark.apache.org) are also corrupt. Btw what’s the correct way to verify the SHA of a Spark package? I’ve tried a few commands on working packages downloaded from Apache mirrors, but I

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Maybe temporarily take out the artifacts on S3 before the root cause is found. On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Just checking in on this again as the builds on S3 are still broken. :/ > > Could it have something to do with us moving

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Thanks. Of course, I verified checksum and it didn't matched. Kousuke On 2016/04/05 0:39, Jitendra Shelar wrote: We can think of using checksum for this kind of issues. On Mon, Apr 4, 2016 at 8:32 PM, Kousuke Saruta > wrote:

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

We can think of using checksum for this kind of issues. On Mon, Apr 4, 2016 at 8:32 PM, Kousuke Saruta wrote: > Oh, I overlooked that. Thanks. > > Kousuke > > > On 2016/04/04 22:58, Nicholas Chammas wrote: > > This is still an issue. The Spark 1.6.1 packages on S3

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Oh, I overlooked that. Thanks. Kousuke On 2016/04/04 22:58, Nicholas Chammas wrote: This is still an issue. The Spark 1.6.1 packages on S3 are corrupt. Is anyone looking into this issue? Is there anything contributors can do to help solve this problem? Nick On Sun, Mar 27, 2016 at 8:49

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

This is still an issue. The Spark 1.6.1 packages on S3 are corrupt. Is anyone looking into this issue? Is there anything contributors can do to help solve this problem? Nick On Sun, Mar 27, 2016 at 8:49 PM Nicholas Chammas wrote: > Pingity-ping-pong since this is

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Pingity-ping-pong since this is still a problem. On Thu, Mar 24, 2016 at 4:08 PM Michael Armbrust wrote: > Patrick is investigating. > > On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Just checking in on this again as the

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Patrick is investigating. On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Just checking in on this again as the builds on S3 are still broken. :/ > > Could it have something to do with us moving release-build.sh >

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Just checking in on this again as the builds on S3 are still broken. :/ Could it have something to do with us moving release-build.sh ? ​ On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Is someone going to retry fixing these packages? It's still a problem. Also, it would be good to understand why this is happening. On Fri, Mar 18, 2016 at 6:49 PM Jakob Odersky wrote: > I just realized you're using a different download site. Sorry for the > confusion, the

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I'm seeing the same. :( On Fri, Mar 18, 2016 at 10:57 AM Ted Yu wrote: > I tried again this morning : > > $ wget > https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz > --2016-03-18 07:55:30-- >

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I tried again this morning : $ wget https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz --2016-03-18 07:55:30-- https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz Resolving s3.amazonaws.com... 54.231.19.163 ... $ tar zxf

Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.6.tgz Does anyone else have trouble unzipping this? How did this happen? What I get is: $ gzip -t spark-1.6.1-bin-hadoop2.6.tgz gzip: spark-1.6.1-bin-hadoop2.6.tgz: unexpected end of file gzip:

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

On Linux, I got: $ tar zxf spark-1.6.1-bin-hadoop2.6.tgz gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now On Wed, Mar 16, 2016 at 5:15 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > >

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

OK cool. I'll test the hadoop-2.6 package and check back here if it's still broken. Just curious: How did those packages all get corrupted (if we know)? Seems like a strange thing to happen. 2016년 3월 17일 (목) 오전 11:57, Michael Armbrust 님이 작성: > Patrick reuploaded the

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Looks like the other packages may also be corrupt. I’m getting the same error for the Spark 1.6.1 / Hadoop 2.4 package. https://s3.amazonaws.com/spark-related-packages/spark-1.6.1-bin-hadoop2.4.tgz Nick ​ On Wed, Mar 16, 2016 at 8:28 PM Ted Yu wrote: > On Linux, I got: >

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Same with hadoop 2.3 tar ball: $ tar zxf spark-1.6.1-bin-hadoop2.3.tgz gzip: stdin: unexpected end of file tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now On Wed, Mar 16, 2016 at 5:47 PM, Nicholas Chammas < nicholas.cham...@gmail.com>

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Patrick reuploaded the artifacts, so it should be fixed now. On Mar 16, 2016 5:48 PM, "Nicholas Chammas" wrote: > Looks like the other packages may also be corrupt. I’m getting the same > error for the Spark 1.6.1 / Hadoop 2.4 package. > > >

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I just experienced the issue, however retrying the download a second time worked. Could it be that there is some load balancer/cache in front of the archive and some nodes still serve the corrupt packages? On Fri, Mar 18, 2016 at 8:00 AM, Nicholas Chammas wrote: > I'm

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I just retried the Spark 1.6.1 / Hadoop 2.6 download and got a corrupt ZIP file. Jakob, are you sure the ZIP unpacks correctly for you? Is it the same Spark 1.6.1/Hadoop 2.6 package you had a success with? On Fri, Mar 18, 2016 at 6:11 PM Jakob Odersky wrote: > I just

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

I just realized you're using a different download site. Sorry for the confusion, the link I get for a direct download of Spark 1.6.1 / Hadoop 2.6 is http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz On Fri, Mar 18, 2016 at 3:20 PM, Nicholas Chammas