Re: Spark streaming multiple kafka topic doesn't work at-least-once

2017-01-24 Thread Hakan İlter
; see any significant deltas.", what were you comparing it against? The > offset ranges printed in spark logs? > > On Tue, Jan 24, 2017 at 2:11 PM, Hakan İlter wrote: > > First of all, I can both see the "Input Rate" from Spark job's statistics > > page

Re: Spark streaming multiple kafka topic doesn't work at-least-once

2017-01-24 Thread Hakan İlter
confused, if you don't see any difference between the offsets the > job is processing and the offsets available in kafka, then how do you > know it's processing less than all of the data? > > On Tue, Jan 24, 2017 at 12:35 AM, Hakan İlter > wrote: > > I'm using Di

Re: Spark streaming multiple kafka topic doesn't work at-least-once

2017-01-23 Thread Hakan İlter
I'm using DirectStream as one stream for all topics. I check the offset ranges from Kafka Manager and don't see any significant deltas. On Tue, Jan 24, 2017 at 4:42 AM, Cody Koeninger wrote: > Are you using receiver-based or direct stream? > > Are you doing 1 stream per topic, or 1 stream for al