Re: write.df is failing on Spark Cluster

2016-09-20 Thread Divya Gehlot
Spark version plz ? On 21 September 2016 at 09:46, Sankar Mittapally < sankar.mittapa...@creditvidya.com> wrote: > Yeah I can do all operations on that folder > > On Sep 21, 2016 12:15 AM, "Kevin Mellott" > wrote: > >> Are you able to manually delete the folder below?

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Sankar Mittapally
Yeah I can do all operations on that folder On Sep 21, 2016 12:15 AM, "Kevin Mellott" wrote: > Are you able to manually delete the folder below? I'm wondering if there > is some sort of non-Spark factor involved (permissions, etc). > >

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Kevin Mellott
Are you able to manually delete the folder below? I'm wondering if there is some sort of non-Spark factor involved (permissions, etc). /nfspartition/sankar/banking_l1_v2.csv On Tue, Sep 20, 2016 at 12:19 PM, Sankar Mittapally < sankar.mittapa...@creditvidya.com> wrote: > I used that one also >

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Sankar Mittapally
I used that one also On Sep 20, 2016 10:44 PM, "Kevin Mellott" wrote: > Instead of *mode="append"*, try *mode="overwrite"* > > On Tue, Sep 20, 2016 at 11:30 AM, Sankar Mittapally creditvidya.com> wrote: > >> Please find the code below. >> >>

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Kevin Mellott
Instead of *mode="append"*, try *mode="overwrite"* On Tue, Sep 20, 2016 at 11:30 AM, Sankar Mittapally < sankar.mittapa...@creditvidya.com> wrote: > Please find the code below. > > sankar2 <- read.df("/nfspartition/sankar/test/2016/08/test.json") > > I tried these two commands. >

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Sankar Mittapally
Please find the code below. sankar2 <- read.df("/nfspartition/sankar/test/2016/08/test.json") I tried these two commands. write.df(sankar2,"/nfspartition/sankar/test/test.csv","csv",header="true") saveDF(sankar2,"sankartest.csv",source="csv",mode="append",schema="true") On Tue, Sep 20, 2016

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Kevin Mellott
Can you please post the line of code that is doing the df.write command? On Tue, Sep 20, 2016 at 9:29 AM, Sankar Mittapally < sankar.mittapa...@creditvidya.com> wrote: > Hey Kevin, > > It is a empty directory, It is able to write part files to the directory > but while merging those part files

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Sankar Mittapally
Hey Kevin, It is a empty directory, It is able to write part files to the directory but while merging those part files we are getting above error. Regards On Tue, Sep 20, 2016 at 7:46 PM, Kevin Mellott wrote: > Have you checked to see if any files already exist at

Re: write.df is failing on Spark Cluster

2016-09-20 Thread Kevin Mellott
Have you checked to see if any files already exist at /nfspartition/sankar/ banking_l1_v2.csv? If so, you will need to delete them before attempting to save your DataFrame to that location. Alternatively, you may be able to specify the "mode" setting of the df.write operation to "overwrite",

write.df is failing on Spark Cluster

2016-09-19 Thread sankarmittapally
We have setup a spark cluster which is on NFS shared storage, there is no permission issues with NFS storage, all the users are able to write to NFS storage. When I fired write.df command in SparkR, I am getting below. Can some one please help me to fix this issue. 16/09/17 08:03:28 ERROR