Re: How to authenticate to ADLS from within spark job on the fly

Hi Steve, I don't think I fully understand your answer. Please pardon my naiveness regarding the subject. From what I understand, the actual read will happen in the executor so executor needs access to data lake. In that sense, how do I make sure that I can programmatically pass azure credentials

Re: How to authenticate to ADLS from within spark job on the fly

This might help; I’ve built a REST API with livyServer: https://livy.incubator.apache.org/ From: Steve Loughran Date: Saturday, August 19, 2017 at 7:05 AM To: Imtiaz Ahmed Cc: "user@spark.apache.org" Subject: Re: How to

Re: How to authenticate to ADLS from within spark job on the fly

On 19 Aug 2017, at 02:42, Imtiaz Ahmed > wrote: Hi All, I am building a spark library which developers will use when writing their spark jobs to get access to data on Azure Data Lake. But the authentication will depend on the dataset they

Re: How to authenticate to ADLS from within spark job on the fly

It may not be as easy as you think. The rest call will happen in driver but the reads will be in the executors. On Sat, 19 Aug 2017 at 11:42 am, Imtiaz Ahmed wrote: > Hi All, > > I am building a spark library which developers will use when writing their > spark jobs to

How to authenticate to ADLS from within spark job on the fly

Hi All, I am building a spark library which developers will use when writing their spark jobs to get access to data on Azure Data Lake. But the authentication will depend on the dataset they ask for. I need to call a rest API from within spark job to get credentials and authenticate to read data