Kakfa consumer times out as Kakfa-server responds to Fetch requests too late

2023-09-29 Thread Neha Rawat
Hello, I am using kafka server 3.4.0 along with flink. Kafka server and Flink are installed on a 48 core , 252GB box. My use case is as follows - 8 Kafka producers writing events at 200K per second to kafka topic "Event" with 20 partitions, source for flink --> Flink processing rules that

[VOTE] 3.6.0 RC2

2023-09-29 Thread Satish Duggana
Hello Kafka users, developers and client-developers, This is the third candidate for the release of Apache Kafka 3.6.0. Some of the major features include: * KIP-405 : Kafka Tiered Storage * KIP-868 : KRaft Metadata Transactions * KIP-875: First-class offsets support in Kafka Connect * KIP-898:

Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers

2023-09-29 Thread Matthias J. Sax
In general, Kafka Streams should keep running. Can you inspect the logs to figure out why it's going into ERROR state to begin with? Maybe you need to increase/change some timeouts/retries configs. The stack trace you shared, is a symptom, but not the root cause. -Matthias On 9/21/23 12:56

Re: Can a message avoid loss occur in Kafka

2023-09-29 Thread Matthias J. Sax
For the config you provide, data loss should not happen (as long as you don't allow for unclean leader election, which is disabled by default). But you might be subject to unavailability for some partitions if a broker fails. -Matthias On 9/17/23 7:49 AM, ι™ˆθΏ‘ε— wrote: Hello, Can a

Re: Kafka protocol ApiVersions request/response

2023-09-29 Thread Neeraj Vaidya
I think I have found the answer to my question about how many ApiKey elements are in the response. The array type for ApiKey is probably a COMPACT_ARRAY but the BNF grammar does not say so in the Documentation. Is this a documentation defect/gap ? ApiVersions Response (Version: 3) =>

Re: Apache Kafka Installation and Env Set up

2023-09-29 Thread sunil chaudhari
Hi Venkat, are you planning to use Open source Apache Kafka Or Confluent? what is your use case apart from streaming? regards, Sunil. On Fri, 29 Sep 2023 at 12:27 PM, ANANTHA VENKATA GANJAM wrote: > TCS Confidential > > Hi Team, > > We are planning to set up Lab environment for Kafka in TCS.

Apache Kafka Installation and Env Set up

2023-09-29 Thread ANANTHA VENKATA GANJAM
TCS Confidential Hi Team, We are planning to set up Lab environment for Kafka in TCS. Please guide us on the next steps. Thanks & Regards, Vijaya Sri TCS Confidential =-=-= Notice: The information contained in this e-mail message and/or attachments to it may contain