Re: I can't save DataFrame from running Spark locally

2018-01-23 Thread Toy
h the versions against which the Hadoop > binaries were built.* > > > > > https://hortonworks.github.io/hdp-aws/s3-s3aclient/index.html#using-the-s3a-filesystem-client > > > > > > > > > > *From: *Toy > *Date: *Tuesday, January 23, 2018 at 11:33 AM > *To

Re: I can't save DataFrame from running Spark locally

2018-01-23 Thread Patrick Alwell
ies must match the versions against which the Hadoop binaries were built. https://hortonworks.github.io/hdp-aws/s3-s3aclient/index.html#using-the-s3a-filesystem-client From: Toy Date: Tuesday, January 23, 2018 at 11:33 AM To: "user@spark.apache.org" Subject: I can't save Data

I can't save DataFrame from running Spark locally

2018-01-23 Thread Toy
Hi, First of all, my Spark application runs fine in AWS EMR. However, I'm trying to run it locally to debug some issue. My application is just to parse log files and convert to DataFrame then convert to ORC and save to S3. However, when I run locally I get this error java.io.IOException: /orc/dt=