Re: Kerberos setup in Apache spark connecting to remote HDFS/Yarn

2016-06-17 Thread Sudarshan Rangarajan
, Sudarshan On Fri, Jun 17, 2016 at 12:01 PM, akhandeshi <ami.khande...@gmail.com> wrote: > Little more progress... > > I add few enviornment variables, not I get following error message: > > InvocationTargetException: Can't get Master Kerberos principal for use as

User impersonation with Kerberos and Delegation tokens

2016-05-25 Thread Sudarshan Rangarajan
park/pull/11358> is also relevant to this scenario. Thanks, Sudarshan

Re:Adding Kafka topics to a running streaming context

2015-08-27 Thread Sudarshan Kadambi (BLOOMBERG/ 731 LEX)
This is something we have been needing for a while too. We are restarting the streaming context to handle new topic subscriptions unsubscriptions which affects latency of update handling. I think this is something that needs to be addressed in core Spark Streaming (I can't think of any

Re: Adding Kafka topics to a running streaming context

2015-08-27 Thread Sudarshan Kadambi (BLOOMBERG/ 731 LEX)
I put up https://issues.apache.org/jira/browse/SPARK-10320. Let's continue this conversation there. From: c...@koeninger.org At: Aug 27 2015 10:30:12 To: Sudarshan Kadambi (BLOOMBERG/ 731 LEX) Cc: user@spark.apache.org Subject: Re: Adding Kafka topics to a running streaming context If you all

Re: saveAsTextFile() to save output of Spark program to HDFS

2015-05-05 Thread Sudarshan Murty
Another thing - could it be a permission problem ? It creates all the directory structure (in red)/tmp/wordcount/ _temporary/0/_temporary/attempt_201505051439_0001_m_01_3/part-1 so I am guessing not. On Tue, May 5, 2015 at 7:27 PM, Sudarshan Murty njmu...@gmail.com wrote: You

Re: saveAsTextFile() to save output of Spark program to HDFS

2015-05-05 Thread Sudarshan Murty
5, 2015 at 6:39 PM, ayan guha guha.a...@gmail.com wrote: What happens when you try to put files to your hdfs from local filesystem? Looks like its a hdfs issue rather than spark thing. On 6 May 2015 05:04, Sudarshan njmu...@gmail.com wrote: I have searched all replies to this question

saveAsTextFile() to save output of Spark program to HDFS

2015-05-05 Thread Sudarshan
I have searched all replies to this question not found an answer.I am running standalone Spark 1.3.1 and Hortonwork's HDP 2.2 VM, side by side, on the same machine and trying to write output of wordcount program into HDFS (works fine writing to a local file, /tmp/wordcount).Only line I added to