Re: I can't save DataFrame from running Spark locally

2018-01-23 Thread Toy
azon libraries must match the versions against which the Hadoop > binaries were built.* > > > > > https://hortonworks.github.io/hdp-aws/s3-s3aclient/index.html#using-the-s3a-filesystem-client > > > > > > > > > > *From: *Toy <noppani...@gmail.com>

Re: I can't save DataFrame from running Spark locally

2018-01-23 Thread Patrick Alwell
;user@spark.apache.org> Subject: I can't save DataFrame from running Spark locally Hi, First of all, my Spark application runs fine in AWS EMR. However, I'm trying to run it locally to debug some issue. My application is just to parse log files and convert to DataFrame then convert to ORC and

I can't save DataFrame from running Spark locally

2018-01-23 Thread Toy
Hi, First of all, my Spark application runs fine in AWS EMR. However, I'm trying to run it locally to debug some issue. My application is just to parse log files and convert to DataFrame then convert to ORC and save to S3. However, when I run locally I get this error java.io.IOException: