Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Hao, Thanks for the KIP! Overall, I think this is a great idea. I always wanted to circle back after the Smooth Scaling KIP to put a proper optimization algorithm into place. I think this has the promise to really improve the quality of the balanced assignments we produce. Thanks for

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1859

See

Re: [VOTE] 3.4.1 RC1

Also I'll point it out here so that everyone is aware, but I don't think it necessarily warrants a new RC on it's own: This patch: https://github.com/apache/kafka/pull/13005 is present on trunk/3.5.0, but is not present in 3.4.0/3.4.1-RC1. It is not a regression strictly, since it was already a

Re: [VOTE] 3.4.1 RC1

Hi Luke, I performed a test upgrade of MM2 from 3.4.0 to 3.4.1-RC1, and verified that the new offset translation logic worked as intended. Steps I took to verify: - Downloaded 3.4.0_2.13 from the Kafka website - Formatted and started two 3.4.0 1-node clusters, and configured MM2 to mirror data

Re: Feature Discussion: Enhancing Kafka Connect's transformation interface to 1 -> Many / Many -> Many

Supporting 1 Kafka record -> many rows in sink connectors would be really useful to properly support some data formats without incurring additional overhead. As an example, when consuming OpenTelmetry OTLP protobuf data, each record represents multiple data points, which significantly reduces

Re: [VOTE] 3.4.1 RC1

Hi Luke, I can confirm the *org.apache.kafka.common.utils.UtilsTest.testToLogDateTimeFormat* consistent failure when not running on a system set on UTC-0 timezone. I verified: - Built from source with Java 11 and Scala 2.13 - Checked signatures and hashes - Check Javadoc and checked links to JDK

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

Hi Daniel, Thanks for the KIP! A few thoughts: 1. This change is similar to when we added connector contexts to Connect in KIPs 449 [1] and 721 [2]. I wonder if we can follow a similar approach (combining both KIPs' changes into this single one), by doing the following: - Disabling the changes

Re: [VOTE] 3.4.1 RC1

Hi Luke, Thanks for running the release! Steps I took to verify: - Built from source with Java 11 - Checked signatures and checksums - Ran the quickstart with Java 11 in KRaft mode - Ran all unit tests - - The org.apache.kafka.common.utils.UtilsTest.testToLogDateTimeDormat test case failed

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

Hi Colt, Thanks for the comments. > and I struggle to see how the algorithm isn't at least O(N) where N is the number of Tasks...? For O(E^2 * (CU)) complexity, C and U can be viewed as constant. Number of edges E is T * N where T is the number of clients and N is the number of Tasks. This is

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1858

See Changes: -- [...truncated 478560 lines...] [2023-05-22T16:23:18.552Z] [2023-05-22T16:23:18.552Z] > Task :streams:javadoc [2023-05-22T16:23:19.371Z] > Task

[DISCUSS] KIP-934: Add DeleteTopicPolicy

Hi everyone, I'd like to start a discussion for KIP-934 < https://cwiki.apache.org/confluence/display/KAFKA/KIP-934%3A+Add+DeleteTopicPolicy> which proposes adding a new policy for when deleting topics. There have been related KIPs in the past that haven't been accepted and seem

[DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

Hi everyone, I'd like to start a discussion for KIP-935 < https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations> which proposes extend AlterConfigPolicy with existing configuration to enable more complex policies. There have been

[jira] [Created] (KAFKA-15016) LICENSE-binary file contains dependencies not included anymore

Mickael Maison created KAFKA-15016: -- Summary: LICENSE-binary file contains dependencies not included anymore Key: KAFKA-15016 URL: https://issues.apache.org/jira/browse/KAFKA-15016 Project: Kafka

[jira] [Created] (KAFKA-15015) Binaries contain 2 versions of reload4j

Mickael Maison created KAFKA-15015: -- Summary: Binaries contain 2 versions of reload4j Key: KAFKA-15015 URL: https://issues.apache.org/jira/browse/KAFKA-15015 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-15014) KIP-935: Extend AlterConfigPolicy with existing configurations

Jorge Esteban Quilcate Otoya created KAFKA-15014: Summary: KIP-935: Extend AlterConfigPolicy with existing configurations Key: KAFKA-15014 URL: https://issues.apache.org/jira/browse/KAFKA-15014

[jira] [Created] (KAFKA-15013) KIP-934: Add DeleteTopicPolicy

Jorge Esteban Quilcate Otoya created KAFKA-15013: Summary: KIP-934: Add DeleteTopicPolicy Key: KAFKA-15013 URL: https://issues.apache.org/jira/browse/KAFKA-15013 Project: Kafka

[VOTE] 3.5.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.5.0. Some of the major features include: - KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters - KIP-881: Rack-aware Partition Assignment for Kafka Consumers

Re: [VOTE] 3.4.1 RC1

Hi Luke, - Source signature and checksum - Build from source with Java 17 and Scala 2.13 - Full unit and integration test suite - Java app with staging Maven artifacts +1 (non binding) Thanks Fede PS: Links still point to RC0, but I checked and RC1 artifacts are there, including Maven. The

[GitHub] [kafka-site] mimaison opened a new pull request, #513: MINOR: Add Kafka 3.5 documentation and javadoc

mimaison opened a new pull request, #513: URL: https://github.com/apache/kafka-site/pull/513 Generated while running the 3.5.0 RC0 build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Created] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field

Ranjan Rao created KAFKA-15012: -- Summary: JsonConverter fails when there are leading Zeros in a field Key: KAFKA-15012 URL: https://issues.apache.org/jira/browse/KAFKA-15012 Project: Kafka

RE: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

Hello Igor! I have been working on a KIP to extend the functionality of JBOD broker disk failures (https://cwiki.apache.org/confluence/display/KAFKA/KIP-928%3A+Making+Kafka+resilient+to+log+directories+becoming+full) and was wondering what is the state of this KIP - were you planning on

Re: [DISCUSS] KIP-905: Broker interceptors

Hello David, Thank you for the proposal - it is an interesting read! I have a few questions about it. 1. Can you take a stance on whether you are proposing the feature just for producers or for consumers as well? If it is just for producers can we remove references to consumers? If it is for

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

Hi Yash, I've been following the discussion and have some thoughts. Ultimately I'm still in favor of this KIP and would hate to see it go dormant, though we may end up settling for a less-invasive option. On the topic of abstraction and inter-plugin interactions: First, there already are

Re: [DISCUSS] Apache Kafka 3.5.0 release

Makes sense Mickael. Thanks! Ismael On Mon, May 22, 2023 at 4:56 AM Mickael Maison wrote: > Hi Ismael, > > Yes we're getting a bit behind schedule. KAFKA-14980 was a significant > regression that would have broken a lot of MirrorMaker deployments. > Chris and I were at Kafka Summit last week,

[VOTE] 3.4.1 RC1

Hello Kafka users, developers and client-developers, This is the 2nd candidate for release of Apache Kafka 3.4.1. This is a bugfix release with several fixes since the release of 3.4.0. A few of the major issues include: - core KAFKA-14644

Re: [VOTE] 3.4.1 RC0

Thanks a lot! -Matthias On 5/21/23 7:27 PM, Luke Chen wrote: Hi Matthias, Yes, I agree we should get this hotfix into 3.4.1. I've backported into the 3.4 branch. I'll create a new RC for 3.4.1. Thanks. Luke On Mon, May 22, 2023 at 5:13 AM Matthias J. Sax wrote: Hi Luke, RC0 for 3.4.1

Re: [DISCUSS] KIP-932: Queues for Kafka

Wow looks very cool. One Q. In the Future Work session: > The concept can be extended to give key-based ordering so that partial ordering and fine-grained sharing can be achieved at the same time. I think this will be a pretty important feature needed to make shared consumer groups useful.

Re: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

Hello! The Wikimedia Foundation would be very interested in this feature. It would make deployment of multi DC/region streaming apps much simpler. We'd like to investigate more the feasibility of using a Kafka stretch cluster to

Re: [DISCUSS] Apache Kafka 3.5.0 release

Hi Ismael, Yes we're getting a bit behind schedule. KAFKA-14980 was a significant regression that would have broken a lot of MirrorMaker deployments. Chris and I were at Kafka Summit last week, so it took a bit longer to fix. The remaining issues are all about migration and at this point I'm not

Re: [DISCUSS] KIP-932: Queues for Kafka

Hey Andrew! Kudos on the proposal. It is greatly written - a joy to read. It is definitely an interesting solution to the queueing problem - I would not have guessed we could solve it like this. Thank you for working on this. Happy to get the discussion started - I have a few comments/questions

[DISCUSS] KIP-928: Making Kafka resilient to log directories becoming full

Hello all! I would like to start a discussion on KIP-928: Making Kafka resilient to log directories becoming full which can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-928%3A+Making+Kafka+resilient+to+log+directories+becoming+full . In summary, I frequently run into