Re: kafka-streams / window expiration because of shuffling

Hi John Thanks for your message. Indeed there's a potential feature here. Should I log it on Jira ? Mathieu Le ven. 20 nov. 2020 à 21:53, John Roesler a écrit : > Hi Mathieu, > > Ah, that is unfortunate. I believe your analysis is correct. In general, > we have no good solution to the problem

GlobalKTable restoration - Unexplained performance penalty

Hey all, We have been running a kafka-stream based service in production for the last couple of years (we have 4 brokers on this specific cluster). Inside this service, which does a lot of stuff - there are 2 GlobalKTables (which are based on 2 compacted topics). When I restart my app and clean th

Kafka issue (Leaders not elected)

Hi There, we are facing following issue in kafka. We have gone through some of the references and found it as a bug from kafka side. Please let us know if there are any solutions. │ 47] [PID:1] Producer send message failed, err:kafka server: In the middle of a leadership election, there is cur

[VOTE] 2.6.1 RC1

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 2.6.1. Since RC0, the following JIRAs have been fixed: KAFKA-10705, KAFKA-10755 and KAFKA-10706 Release notes for the 2.6.1 release: https://home.apache.org/~mimaison/kafka-2.6.1-rc1/REL

Re: multi-threaded consumer configuration like stream threads?

Thanks Liam! We don't have a requirement to maintain order of processing for events even within a partition. Essentially, these are events for various accounts (customers) that we want to support and do necessary database provisioning for those in our database. So they can be processed in parallel.

Re: multi-threaded consumer configuration like stream threads?

Hi Pushkar. Just for your information, https://github.com/line/decaton is a Kafka consumer framework that supports parallel processing per single partition. It manages committable (i.e. the offset that all preceding offsets have been processed) offset internally so that preserves at-least-once se

Re: multi-threaded consumer configuration like stream threads?

Thanks Haruki... right now the max of such types of events that we would have is 100 since we would be supporting those many customers (accounts) for now, for which we are considering a simple approach of a single consumer and a thread pool with around 10 threads. So the question was regarding how

Re: EXTERNAL: Re: Backup Apache KAFKA

Hey Ben, I think the distinction between your and Matthias' points is analogous to, borrowing from the OSI model, transport- vs presentation/application-level concerns. When configured appropriately, Kafka durably provides transport for messages from point A to point B, with such optional goodies

Re: GlobalKTable restoration - Unexplained performance penalty

Hello Nitay, Thanks for letting us know about your observations. When you restart the application from empty local state Kafka Streams will try to restore all the records up to the current log end for those global KTables, and during this period of time there should be no processing. Do you mind

Bytes-consumed-rate JMX Mbeans

Hello I try take metrics for bytes-consumed-rate . Bu t I have the following problem. I write the command JMX_PORT= bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.consumer:type=consumer-fetch-manager-metrics,client-id="([-.w]+)",name=bytes-consumed-rate and I take the error No

Re: Kafka issue (Leaders not elected)

Hi Mohammad, Leader elections are generally seldom and usually quick. If a leader is unable to elected, this could point to a) replication being set too low or b) inter-broker communication issues, plus a bunch of other possible problems. You'd have to look into your Kafka broker's logs to figur

Bytes-consumed-rate JMX Mbeans

Hello I try to take bytes-consumed-metrics metrics via JMX Mbeans but i have the following issue. I write the command JMX_PORT= bin/kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.consumer:type=consumer-fetch-manager-metrics,client-id="([-.w]+)",name=bytes-consumed-rate But I take t

Re: multi-threaded consumer configuration like stream threads?

I see. Then I think the appropriate approach depends on your delivery latency requirements. Just retrying until success is simpler but it could block subsequent messages to get processed. (also depends on thread pool size though) Then another concern when using dead letter topic would be retrying

Re: Unable to list Kafka topics

Hi Sunil, Thanks for your reply. all four words commands where giving connection refused. I moved files (log.1,log.103,log.142,log.16a,log.17d - all where having oct timestamp) present inside `/var/data/logs/zookeeper/version-2/` to some other directory and started zookeeper and then kafka

Re: multi-threaded consumer configuration like stream threads?

I think we can handle the failures selectively e.g. if there are issues with downstream database server then all the events will fail to process so it will be worth to keep retrying. Else if there is issue only while processing a particular event, then we can keep retry timeout and after that timeo