We have some data that we are exporting from our HDFS cluster to S3 with
some help from Spark. The final RDD command we run is:
csvData.saveAsTextFile("s3n://data/mess/2014/11/dump-oct-30-to-nov-5-gzip",
classOf[GzipCodec])
We have our 'spark.local.dir' set to our large ephemeral partition on
e
Hi Gary,
Could you create a Spark JIRA ticket for this so that it doesn't fall
through the cracks? Thanks!
On Thu, Nov 6, 2014 at 2:10 PM, Gary Malouf wrote:
> We have some data that we are exporting from our HDFS cluster to S3 with
> some help from Spark. The final RDD command we run is:
>
>
