Re: Exception while reading from kafka stream

Thanks a lot , it worked for me. I'm using single direct stream which retrieves data from all the topic. *Thanks*, On Mon, Nov 2, 2015 at 8:13 PM, Cody Koeninger wrote: > combine topicsSet_1 and topicsSet_2 in a single

Re: Exception while reading from kafka stream

combine topicsSet_1 and topicsSet_2 in a single createDirectStream call. Then you can use hasOffsetRanges to see what the topic for a given partition is. On Mon, Nov 2, 2015 at 7:26 AM, Ramkumar V wrote: > if i try like below code snippet , it shows exception , how to

Re: Exception while reading from kafka stream

if i try like below code snippet , it shows exception , how to avoid this exception ? how to switch processing based on topic ? JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(30)); HashSet topicsSet_1 = new HashSet(Arrays.asList(topics.split(","))); HashSet

Re: Exception while reading from kafka stream

What Spark version are you using, also a small code snippet of how you use Spark Streaming would be greatly helpful. On Fri, Oct 30, 2015 at 3:57 PM, Ramkumar V wrote: > I can able to read and print few lines. Afterthat i'm getting this > exception. Any idea for this ?

Re: Exception while reading from kafka stream

spark version - spark 1.4.1 my code snippet: String brokers = "ip:port,ip:port"; String topics = "x,y,z"; HashSet TopicsSet = new HashSet(Arrays.asList(topics.split(","))); HashMap kafkaParams = new HashMap(); kafkaParams.put("metadata.broker.list", brokers);

Re: Exception while reading from kafka stream

I can able to read and print few lines. Afterthat i'm getting this exception. Any idea for this ? *Thanks*, On Thu, Oct 29, 2015 at 6:14 PM, Ramkumar V wrote: > Hi, > > I'm trying to read from kafka stream and printing it

Re: Exception while reading from kafka stream

Do you have any special settings, from your code, I don't think it will incur NPE at that place. On Fri, Oct 30, 2015 at 4:32 PM, Ramkumar V wrote: > spark version - spark 1.4.1 > > my code snippet: > > String brokers = "ip:port,ip:port"; > String topics = "x,y,z"; >

Re: Exception while reading from kafka stream

I just did a local test with your code, seems everything is fine, the only difference is that I use the master branch, but I don't think it changes a lot in this part. Do you met any other exceptions or errors beside this one? Probably this is due to other exceptions that makes this system

Re: Exception while reading from kafka stream

I don't think Spark Streaming supports multiple streaming context in one jvm, you cannot use in such way. Instead you could run multiple streaming applications, since you're using Yarn. 2015年10月30日星期五，Ramkumar V 写道： > I found NPE is mainly because of im using the same

Re: Exception while reading from kafka stream

No. this is the only exception that im getting multiple times in my log. Also i was reading some other topics earlier but im not faced this NPE. *Thanks*, On Fri, Oct 30, 2015 at 2:50 PM, Saisai Shao wrote: > I just did a local

Re: Exception while reading from kafka stream

>From the code, I think this field "rememberDuration" shouldn't be null, it will be verified at the start, unless some place changes it's value in the runtime that makes it null, but I cannot image how this happened. Maybe you could add some logs around the place where exception happens if you

Re: Exception while reading from kafka stream

I found NPE is mainly because of im using the same JavaStreamingContext for some other kafka stream. if i change the name , its running successfully. how to run multiple JavaStreamingContext in a program ? I'm getting following exception if i run multiple JavaStreamingContext in single file.

Re: Exception while reading from kafka stream

Just put them all in one stream and switch processing based on the topic On Fri, Oct 30, 2015 at 6:29 AM, Ramkumar V wrote: > i want to join all those logs in some manner. That's what i'm trying to do. > > *Thanks*, > > > > On

Re: Exception while reading from kafka stream

No, i dont have any special settings. if i keep only reading line in my code, it's throwing NPE. *Thanks*, On Fri, Oct 30, 2015 at 2:14 PM, Saisai Shao wrote: > Do you have any special settings, from your code, I don't think it

Re: Exception while reading from kafka stream

In general , i need to consume five different type of logs from kafka in spark. I have different set of topics for each log. How to start five different stream in spark ? *Thanks*, On Fri, Oct 30, 2015 at 4:40 PM, Ramkumar V

Re: Exception while reading from kafka stream

i want to join all those logs in some manner. That's what i'm trying to do. *Thanks*, On Fri, Oct 30, 2015 at 4:57 PM, Saisai Shao wrote: > I don't think Spark Streaming supports multiple streaming context in one > jvm, you

Exception while reading from kafka stream

Hi, I'm trying to read from kafka stream and printing it textfile. I'm using java over spark. I dont know why i'm getting the following exception. Also exception message is very abstract. can anyone please help me ? Log Trace : 15/10/29 12:15:09 ERROR scheduler.JobScheduler: Error in job