subject:"Does Spark uses its own HDFS client\?"

Re: Does Spark uses its own HDFS client?

2017-04-07 Thread Jörn Franke

Maybe using ranger or sentry would be the better choice to intercept those calls? > On 7. Apr 2017, at 16:32, Alvaro Brandon wrote: > > I was going through the SparkContext.textFile() and I was wondering at that > point does Spark communicates with HDFS. Since when

Re: Does Spark uses its own HDFS client?

2017-04-07 Thread Steve Loughran

On 7 Apr 2017, at 15:32, Alvaro Brandon > wrote: I was going through the SparkContext.textFile() and I was wondering at that point does Spark communicates with HDFS. Since when you download Spark binaries you also specify the Hadoop

Does Spark uses its own HDFS client?

2017-04-07 Thread Alvaro Brandon

I was going through the SparkContext.textFile() and I was wondering at that point does Spark communicates with HDFS. Since when you download Spark binaries you also specify the Hadoop version you will use, I'm guessing it has its own client that calls HDFS wherever you specify it in the