Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-30 Thread swetha kasireddy
got problems with kafka falling >>>>> over for half an hour, I'd look at fixing that before worrying about spark >>>>> monitoring... >>>>> >>>>> >>>>> On Mon, Nov 9, 2015 at 12:26 PM, swetha <swethakasire...@gmail.com>

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-30 Thread Cody Koeninger
ack, hoping that kafka has those offsets. >>> >>> -adrian >>> >>> From: swetha kasireddy >>> Date: Monday, November 9, 2015 at 10:40 PM >>> To: Cody Koeninger >>> Cc: "user@spark.apache.org" >>> Subject: Re: Kafka Dire

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-30 Thread swetha kasireddy
>>>> >>>>>>> The question of what kind of monitoring you need is more a question >>>>>>> for your particular infrastructure and what you're already using for

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-12 Thread Cody Koeninger
int folder to help the job > recover? E.g. Go 2 steps back, hoping that kafka has those offsets. > > -adrian > > From: swetha kasireddy > Date: Monday, November 9, 2015 at 10:40 PM > To: Cody Koeninger > Cc: "user@spark.apache.org" > Subject: Re: Kafka Direct does

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-10 Thread Adrian Tanase
;Thanks, >Swetha > > > >-- >View this message in context: >http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-Direct-does-not-recover-automatically-when-the-Kafka-Stream-gets-messed-up-tp25331.html >S

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-10 Thread Adrian Tanase
e that Kafka Direct recovers automatically when the broker fails for sometime say 30 minutes? What kind of monitors should be in place to recover the job? Thanks, Swetha -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-Direct-does-not-recover-automatic

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-09 Thread Cody Koeninger
for sometime say 30 minutes? What kind of monitors > should be in place to recover the job? > > Thanks, > Swetha > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-Direct-does-not-recover-automatically-when-the-Kafka-

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-09 Thread Cody Koeninger
gets messed up and the entire >>> Streaming job blows up unlike some other consumers which do recover >>> automatically. How can I make sure that Kafka Direct recovers >>> automatically >>> when the broker fails for sometime say 30 minutes? What kind of monitors

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-09 Thread swetha kasireddy
>> >>>> How to recover Kafka Direct automatically when the there is a problem >>>> with >>>> Kafka brokers? Sometimes our Kafka Brokers gets messed up and the entire >>>> Streaming job blows up unlike some other consumers which do recover >&g

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-09 Thread Cody Koeninger
;> How to recover Kafka Direct automatically when the there is a problem >>>>> with >>>>> Kafka brokers? Sometimes our Kafka Brokers gets messed up and the >>>>> entire >>>>>

Re: Kafka Direct does not recover automatically when the Kafka Stream gets messed up?

2015-11-09 Thread swetha kasireddy
before worrying about spark >>>>> monitoring... >>>>> >>>>> >>>>> On Mon, Nov 9, 2015 at 12:26 PM, swetha <swethakasire...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>&