Problems saving a large RDD (1 TB) to S3 as a sequence file

I've tried various ideas, but I'm really just shooting in the dark. I have an 8 node cluster of r3.8xlarge machines. The RDD (with 1024 partitions) I'm trying to save off to S3 is approximately 1TB in size (with the partitions pretty evenly distributed in size). I just tried a test to dial

Re: Problems saving a large RDD (1 TB) to S3 as a sequence file

Hey Darin, Are you running this over EMR or as a standalone cluster? I've had occasional success in similar cases by digging through all executor logs and trying to find exceptions that are not caused by the application shutdown (but the logs remain my main pain point with Spark). That aside,

Re: Problems saving a large RDD (1 TB) to S3 as a sequence file

Thanks for the ideas Sven. I'm using stand-alone cluster (Spark 1.2). FWIW, I was able to get this running (just now). This is the first time it's worked in probably my last 10 attempts. In addition to limiting the executors to only 50% of the cluster. In the settings below, I additionally

Re: Problems saving a large RDD (1 TB) to S3 as a sequence file

Can you also try increasing the akka framesize? .set(spark.akka.frameSize,50) // Set it to a higher number Thanks Best Regards On Sat, Jan 24, 2015 at 3:58 AM, Darin McBeath ddmcbe...@yahoo.com.invalid wrote: Thanks for the ideas Sven. I'm using stand-alone cluster (Spark 1.2). FWIW, I