Re: Truncation

2019-07-22 Thread Guozhang Wang
Hi Jamie, The most relevant materials I can think of would be in KIP-101: https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation Although it is a bit out-dated it still contains most significant design k

Re: Rebalancing algorithm is extremely suboptimal for long processing

2019-07-22 Thread Guozhang Wang
Hello Raman, since you are using Consumer and you are concerning about the member-failure triggered rebalance, I think KIP-429 is most relevant to your scenario. As Matthias mentioned we are working on getting it in to the next release 2.4. Guozhang On Sat, Jul 20, 2019 at 6:36 PM Matthias J. Sa

Re: Questions about KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

2019-07-22 Thread Guozhang Wang
Hello Shurong, What you bumped into seems to be the same issue as is reported tracked by Luying Liu here: https://issues.apache.org/jira/browse/KAFKA-8676. Guozhang On Thu, Jun 27, 2019 at 10:48 AM srlin wrote: > Hi, team, > > We are using Kafka Connect at present and have encountered a probl

SchemaException: Error reading field 'topics'

2019-07-22 Thread Kyla Palos
I have a producer that would send messages to a topic with 4 partitions. Then, 4 consumers would consume from that topic. Most of the time, it works properly. But then, I would sometimes experience this error: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topics': Err

Strimzi

2019-07-22 Thread Csenge VirĂ¡g Maruzsi
Hi all, I recently started getting to know Strimzi and I have several problems with configuring it. My biggest problem so far that after creating a Cluster Operator and Kafka cluster, Kafka brokers can not reach zookeeper servers. However I did everything according to the documentation. Do you have

Re: Question regarding pattern subscriptions

2019-07-22 Thread Chris Hampson
Thanks Alok, I'd come across that and begun playing with different values for that. Tweaking it has done exactly what I was after. I'm now getting the desired behaviour. Thanks again for the speedy response, Cheers, Chris > On 22 Jul 2019, at 09:36, Alok Dwivedi wrote: > > Hi Chris > Since

Re: Question regarding pattern subscriptions

2019-07-22 Thread Alok Dwivedi
Hi Chris Since you are only missing the very first message sent to that topic, I wonder its related to " auto.offset.reset" consumer setting. This setting controls what happens when no offset is present for a topic (most commonly the case when there has been no consumer yet for that topic) Fro