I'm using DirectStream as one stream for all topics. I check the offset ranges from Kafka Manager and don't see any significant deltas.
On Tue, Jan 24, 2017 at 4:42 AM, Cody Koeninger <c...@koeninger.org> wrote: > Are you using receiver-based or direct stream? > > Are you doing 1 stream per topic, or 1 stream for all topics? > > If you're using the direct stream, the actual topics and offset ranges > should be visible in the logs, so you should be able to see more > detail about what's happening (e.g. all topics are still being > processed but offsets are significantly behind, vs only certain topics > being processed but keeping up with latest offsets) > > On Mon, Jan 23, 2017 at 3:14 PM, hakanilter <hakanil...@gmail.com> wrote: > > Hi everyone, > > > > I have a spark (1.6.0-cdh5.7.1) streaming job which receives data from > > multiple kafka topics. After starting the job, everything works fine > first > > (like 700 req/sec) but after a while (couples of days or a week) it > starts > > processing only some part of the data (like 350 req/sec). When I check > the > > kafka topics, I can see that there are still 700 req/sec coming to the > > topics. I don't see any errors, exceptions or any other problem. The job > > works fine when I start the same code with just single kafka topic. > > > > Do you have any idea or a clue to understand the problem? > > > > Thanks. > > > > > > > > -- > > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Spark-streaming-multiple-kafka- > topic-doesn-t-work-at-least-once-tp28334.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > >
