Starting out with Kafka

Hi All, I have a basic question: I tried importing kafka source code into intellij but there are bunch of imports that IDE cannot find like these: import kafka.api.ApiVersion; import kafka.log.CleanerConfig; import kafka.log.LogConfig; import kafka.log.LogManager; TIA. Thanks, Vinay

[jira] [Created] (KAFKA-14382) StreamThreads can miss rebalance events when processing records during a rebalance

A. Sophie Blee-Goldman created KAFKA-14382: -- Summary: StreamThreads can miss rebalance events when processing records during a rebalance Key: KAFKA-14382 URL: https://issues.apache.org/jira/browse/KAFKA-1

Re: Jackson CVE's in Mirror Maker 2.13-2.8.2

Hi Andrew, Kafka community will only do bug fix release for last 3 releases based on the wiki. So, there will be no newer 2.8 patch release. https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy ? Thank you. Luke On Thu, Nov 10, 2022 at

[jira] [Resolved] (KAFKA-14380) consumer should refresh preferred read replica on metadata update

[ https://issues.apache.org/jira/browse/KAFKA-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-14380. - Resolution: Duplicate duplicated with KAFKA-14379 > consumer should refresh preferred read repl

[jira] [Resolved] (KAFKA-14378) consumer should refresh preferred read replica on update metadata

[ https://issues.apache.org/jira/browse/KAFKA-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deng Ziming resolved KAFKA-14378. - Resolution: Duplicate duplicated with KAFKA-14379 > consumer should refresh preferred read repl

Re: [DISCUSS] KIP-886 Add Client Producer and Consumer Builders

This would be helpful. For our own client library wrappers we implemented this functionality for any type with defaults for and consumers/producers. On Thu, Nov 10, 2022, 6:35 PM Dan S wrote: > Hello all, > > I think that adding builders for the producer and the consumer in kafka > client woul

[jira] [Created] (KAFKA-14381) Support listing all partitions being reassigned in a cluster

Lucas Wang created KAFKA-14381: -- Summary: Support listing all partitions being reassigned in a cluster Key: KAFKA-14381 URL: https://issues.apache.org/jira/browse/KAFKA-14381 Project: Kafka Iss

[jira] [Created] (KAFKA-14380) consumer should refresh preferred read replica on metadata update

Jeff Kim created KAFKA-14380: Summary: consumer should refresh preferred read replica on metadata update Key: KAFKA-14380 URL: https://issues.apache.org/jira/browse/KAFKA-14380 Project: Kafka Is

[jira] [Created] (KAFKA-14378) consumer should refresh preferred read replica on update metadata

Jeff Kim created KAFKA-14378: Summary: consumer should refresh preferred read replica on update metadata Key: KAFKA-14378 URL: https://issues.apache.org/jira/browse/KAFKA-14378 Project: Kafka Is

[jira] [Created] (KAFKA-14379) consumer should refresh preferred read replica on update metadata

Jeff Kim created KAFKA-14379: Summary: consumer should refresh preferred read replica on update metadata Key: KAFKA-14379 URL: https://issues.apache.org/jira/browse/KAFKA-14379 Project: Kafka Is

[DISCUSS] KIP-886 Add Client Producer and Consumer Builders

Hello all, I think that adding builders for the producer and the consumer in kafka client would make it much easier for developers to instantiate new producers and consumers, especially if they are using an IDE with intellisense, and using the IDE to navigate to the documentation which could be ad

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

Hi, Divij, Thanks for the reply. It's true that the data in Kafka could be kept longer with KIP-405. How much data do you envision to have per broker? For 100TB data per broker, with 1GB segment and segment metadata of 100 bytes, it requires 100TB/1GB*100 = 10MB, which should fit in memory. Remo

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

*Jun,* *"the default implementation of RLMM does local caching, right?"* Yes, Jun. The default implementation of RLMM does indeed cache the segment metadata today, hence, it won't work for use cases when the number of segments in remote storage is large enough to exceed the size of cache. As part

[DISCUSS] KIP-885: Expose Broker's Name and Version to Clients Skip to end of metadata

Hi all, I've written a KIP to expose the BrokerSoftwareName and BrokerSoftwareVersion to clients: https://cwiki.apache.org/confluence/display/KAFKA/KIP-885%3A+Expose+Broker%27s+Name+and+Version+to+Clients If we agree this is useful, it would be great to have this in by 3.4. Thank you, - Travis

[jira] [Created] (KAFKA-14377) Expose Broker's Name and Version to Clients

Travis Bischel created KAFKA-14377: -- Summary: Expose Broker's Name and Version to Clients Key: KAFKA-14377 URL: https://issues.apache.org/jira/browse/KAFKA-14377 Project: Kafka Issue Type: I

[jira] [Created] (KAFKA-14376) Add ConfigProvider to make use of environment variables

Roman Schmitz created KAFKA-14376: - Summary: Add ConfigProvider to make use of environment variables Key: KAFKA-14376 URL: https://issues.apache.org/jira/browse/KAFKA-14376 Project: Kafka Iss

Re: Kafka adopt two measurement systems

Hi, Zhanchang, We started with Yammer metrics on the broker side. When we developed java clients, we wanted to reduce external dependencies. So, we added Kafka metric and used it in the clients. Eventually, we could consider consolidating the broker side metrics with Kafka metrics too. Currently,

PR 12343 review/merge

Hi All, I have a PR https://github.com/apache/kafka/pull/12343 that was approved in June but has not been merged yet. The PR fixes some issues with running on IBM Semeru JDKs. I've rebased to trunk and the Java 17/Scala 2.13 and Java 8/Scala 2.12 tests have all passed. There were two test fail

Re: [VOTE] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

+1 (non-binding) Thanks for the KIP, this is an important improvement. Greg On Thu, Nov 10, 2022 at 7:21 AM John Roesler wrote: > Thanks for the KIP, Daniel! > > I'm no MM expert, but I've read over the KIP and discussion, and it seems > reasonable to me. > > I'm +1 (binding). > > Thanks, > -J

[jira] [Created] (KAFKA-14375) Remove use of "authorizer-properties" n EndToEndAuthorizationTest.scala

Proven Provenzano created KAFKA-14375: - Summary: Remove use of "authorizer-properties" n EndToEndAuthorizationTest.scala Key: KAFKA-14375 URL: https://issues.apache.org/jira/browse/KAFKA-14375 Pro

Re: [DISCUSS] Apache Kafka 3.4.0 release

Hello again, This is a reminder that the KIP freeze deadline is approaching, all KIPs must be voted and accepted by *next Wednesday* *(the 16th)* Keep in mind that to allow for the full voting period, this means you must kick off the vote for your KIP no later than* next Monday* (*the 14th*). Th

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1348

See

Jackson CVE's in Mirror Maker 2.13-2.8.2

Hello Kafka Developers, I was wondering if there are any plans to back port fixes for certain CVE’s found in Mirror Maker 2.13-2.8.2. Scans of the code found the following unpatched CVE’s: * CVE-2022-42004 * CVE-2022-42003 * CVE-2020-36518 It’s my understanding that there’s going

Re: [VOTE] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

Thanks for the KIP, Daniel! I'm no MM expert, but I've read over the KIP and discussion, and it seems reasonable to me. I'm +1 (binding). Thanks, -John On 2022/10/22 07:38:38 Urbán Dániel wrote: > Hi everyone, > > I would like to start a vote on KIP-710 which aims to support running a > dedi

[jira] [Created] (KAFKA-14374) Kafka streams losing messages in State Store during first launch of app

Youcef Sebiat created KAFKA-14374: - Summary: Kafka streams losing messages in State Store during first launch of app Key: KAFKA-14374 URL: https://issues.apache.org/jira/browse/KAFKA-14374 Project: Ka

Recall: Review request - PR#12753

Sunnappugunta, Satheesh would like to recall the message, "[External] Review request - PR#12753".

RE: Review request - PR#12753

Sure Sarbik Thanks & Regards, Satheesh S LTS Integration - TIBCO Email: ss...@allstate.com Ph.: +91-8105295295 -Original Message- From: Dan S Sent: Thursday, November 10, 2022 1:17 PM To: dev@kafka.apache.org Subject: [External] Review request - PR#12753 Hello, I would really appreci