How Kafka client identify the new leader when leader fail over happens.

2019-09-24 Thread Isuru Boyagane
Hi, Can anyone clarify the procedure of a client getting to know about a Kafka leader failover and identify the new leader? Thanks.

Re: [VOTE] 2.3.1 RC0

2019-09-24 Thread Jason Gustafson
Hi David, Thanks for running the release. I think we should consider getting this bug fixed: https://issues.apache.org/jira/browse/KAFKA-8896. The impact of this bug is that consumer groups cannot commit offsets or rebalance. The patch should be ready shortly. Thanks, Jason On Fri, Sep 13, 201

Event time semantics across windows

2019-09-24 Thread Parthasarathy, Mohan
Hi, Here is a simple example: Application is reading a topic where messages are being received from various clients identified by “client_id”. The messages are grouped by “client_id” and windowedBy 10 seconds with grace period of 5 seconds. The event time for the stream progresses when we rece

Cleaning up offsets stored outside Kafka cluster (e.g. DB)

2019-09-24 Thread M. Manna
Hello, I am trying to understand how to cleanup offsets stored outside kafka. What I understand is that using ConsumerRebalanceListener, we can utilise storing/retrieving consumer offsets from external storage. https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanc

Re: Shipping Kafka logs

2019-09-24 Thread Eva Sheeva
Thanks Liam, I'm trying with Filebeat but i'm unsure what filter should i use for kafka logs. I need all the controller.log, server.log and state-change.log to be shipped. I tried using grok filter but that is throwing error. Do you have any info on the filter to be used for Kafka logs? Regards O

Fwd: Shipping Kafka logs

2019-09-24 Thread Liam Clarke
Cc list -- Forwarded message - From: Liam Clarke Date: Tue, 24 Sep. 2019, 7:40 pm Subject: Re: Shipping Kafka logs To: Eva Sheeva Ah, for that you need something like Filebeat - it's an agent that forwards monitored log files to ELK. On Tue, 24 Sep. 2019, 7:17 pm Eva Sheeva,

Re: Shipping Kafka logs

2019-09-24 Thread Eva Sheeva
Thank you Jacob and Liam! I 'm looking for the steps on shipping Kafka application logs. I want the controller.log, state-change.log and server.log to be shipped to elk. Can you pls help! Regards! On Tue, Sep 24, 2019 at 8:08 AM Botuck, Jacob (STL) < jbot...@express-scripts.com> wrote: > You ca