Re: Does saveAsHadoopFile depend on master?

Hi guys! I solved my issue, that was indeed a permission problem. Thanks for your help. 2016-06-22 9:27 GMT+02:00 Spico Florin : > Hi! > I had a similar issue when the user that submit the job to the spark > cluster didn't have permission to write into the hdfs. If you

Re: Does saveAsHadoopFile depend on master?

Hi! I had a similar issue when the user that submit the job to the spark cluster didn't have permission to write into the hdfs. If you have the hdfs GUI then you can check which users are and what permissions. Also can in hdfs browser:(

Re: Does saveAsHadoopFile depend on master?

Please check the driver and executor log, there should be logs about where the data is written. On Wed, Jun 22, 2016 at 2:03 AM, Pierre Villard wrote: > Hi, > > I have a Spark job writing files to HDFS using .saveAsHadoopFile method. > > If I run my job in

Does saveAsHadoopFile depend on master?

Hi, I have a Spark job writing files to HDFS using .saveAsHadoopFile method. If I run my job in local/client mode, it works as expected and I get all my files written in HDFS. However if I change to yarn/cluster mode, I don't see any error logs (the job is successful) and there is no files