leadership election timing on kafka broker failure

2020-03-25 Thread Larry Hemenway
All, We're experimenting with what happens in the event of a Kafka broker failure and we're seeing it take up to ~10 seconds for leadership to switch over. We've been unable to figure out if there are some parameters to tighten this timing. Are there broker config parameters that affect this timi

Get after put in stateStore returns null

2020-03-25 Thread Jan Bols
Hi all, I'm trying to aggregate a stream of messages and return a stream of aggregated results using kafka streams. At some point, depending on the incoming message, the old aggregate needs to be closed and a new aggregate needs to be created, just like a session that is closed due to some close ev

Re: Max poll interval and timeouts

2020-03-25 Thread Steve Tian
Hi Ryan, Have you tried Consumer's pause/resume methods? Steve On Wed, Mar 25, 2020, 17:13 Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > With group coordination protocol, you only have to increase the ` > max.poll.interval.ms` / `max.poll.records`. > Ignore the above messages.

Re: Max poll interval and timeouts

2020-03-25 Thread Kamal Chandraprakash
With group coordination protocol, you only have to increase the ` max.poll.interval.ms` / `max.poll.records`. Ignore the above messages. Consumer heartbeats are processed in a separate thread. On Wed, Mar 25, 2020 at 2:35 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Yes, wit

Re: Max poll interval and timeouts

2020-03-25 Thread Kamal Chandraprakash
Yes, with `assign` you'll lose the group coordination. You can still use the `subscribe` mode, update the above mentioned configs. You're ask is kind of Delay Queue. Kafka Consumer doesn't support that feature. You've to manually `sleep` in between the poll calls. On Tue, Mar 24, 2020 at 11:56 PM