Re: [VOTE] 3.4.0 RC2
I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the Javadocs. I am +1 (binding). Thanks for running the release! Best, David On Tue, Jan 31, 2023 at 8:00 PM David Arthur wrote: > Hey folks, we found a couple of blockers with RC1 and have fixed them in > the latest release candidate, RC2. > > The major features of this release include: > > * KIP-881: Rack-aware Partition Assignment for Kafka Consumers > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers > > > > * KIP-876: Time based cluster metadata snapshots > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-876%3A+Time+based+cluster+metadata+snapshots > > > > * KIP-787: MM2 manage Kafka resources with custom Admin implementation. > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335620 > > > > * KIP-866 ZooKeeper to KRaft Migration > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration > > > (Early > Access) > > > > Release notes for the 3.4.0 release: > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/RELEASE_NOTES.html > > > Please download, test and vote by Friday, February 3, 5pm PT > > > --- > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/ > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > * Javadoc: > > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/javadoc/ > > > * Tag to be voted upon (off 3.4 branch) is the 3.4.0 tag: > > https://github.com/apache/kafka/releases/tag/3.4.0-rc2 > > > * Documentation: > > https://kafka.apache.org/34/documentation.html > > > * Protocol: > > https://kafka.apache.org/34/protocol.html > > > --- > > > Test results: > > > We haven't had a 100% passing build, but the latest system test run looks > pretty good: > > http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.4/2023-01-31--001.system-test-kafka-3.4--1675184554--confluentinc--3.4--ef3f5bd834/report.html > > > Here are the Jenkins test runs for 3.4: > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/. We will > continue > trying to diagnose the flaky test failures as the release continues. I do > not expect that any of these test failures are blockers for the release. > > > Thanks! > > David Arthur >
Re: [VOTE] 3.3.1 RC0
Hey, I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the Javadocs. I am +1 (binding), assuming that the system tests look good. Thanks for running the release. Best, David On Fri, Sep 30, 2022 at 2:23 AM José Armando García Sancio wrote: > > On Thu, Sep 29, 2022 at 2:39 PM José Armando García Sancio > wrote: > > Please download, test and vote by Tuesday, October 4, 9am PT. > > The vote will be open for 72 hours. Please vote by Sunday, October 2nd, 3 PM > PT. > > Thanks! > -- > -José
Re: [VOTE] 3.3.0 RC2
Thanks for running the release, José and David. I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the Javadocs. I have also noticed that the doc is malformed but we can fix this separately. I am +1 (binding), assuming that the system tests look good. It would be great if you could share them. Best, David On Mon, Sep 26, 2022 at 5:36 PM John Roesler wrote: > > Thanks for running this, David! > > I've verified the signatures, looked at the docs, and run the quickstart (ZK > and KRaft). I also ran the unit tests, as well as all the tests for Streams > locally. > > The docs look a little malformed (the "collapse/expand" button floats over > the text, the collapsed doc tree is only halfway collapsed, and there's a > weird empty panel on the right). > > We can fix the docs site independent of this release, so I'm +1 (binding). > > Thanks, > -John > > On Tue, Sep 20, 2022, at 18:17, David Arthur wrote: > > Hello Kafka users, developers and client-developers, > > > > This is the second release candidate for Apache Kafka 3.3.0. Many new > > features and bug fixes are included in this major release of Kafka. A > > significant number of the issues in this release are related to KRaft, > > which will be considered "production ready" as part of this release > > (KIP-833) > > > > KRaft improvements: > > * KIP-778: Online KRaft to KRaft Upgrades > > * KIP-833: Mark KRaft as Production Ready > > * KIP-835: Monitor Quorum health (many new KRaft metrics) > > * KIP-836: Expose voter lag via kafka-metadata-quorum.sh > > * KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft > > * KIP-859: Add Metadata Log Processing Error Related Metrics > > > > Other major improvements include: > > * KIP-618: Exactly-Once Support for Source Connectors > > * KIP-831: Add metric for log recovery progress > > * KIP-827: Expose logdirs total and usable space via Kafka API > > * KIP-834: Add ability to Pause / Resume KafkaStreams Topologies > > > > The full release notes are available here: > > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/RELEASE_NOTES.html > > > > Please download, test and vote by Monday, Sep 26 at 5pm EDT > > > > Also, huge thanks to José for running the release so far. He has done > > the vast majority of the work to prepare this rather large release :) > > > > - > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: https://home.apache.org/~davidarthur/kafka-3.3.0-rc2/javadoc/ > > > > * Tag to be voted upon (off 3.3 branch) is the 3.3.0 tag: > > https://github.com/apache/kafka/releases/tag/3.3.0-rc2 > > > > * Documentation: https://kafka.apache.org/33/documentation.html > > > > * Protocol: https://kafka.apache.org/33/protocol.html > > > > > > > > > > Successful Jenkins builds to follow in a future update to this email. > > > > > > Thanks! > > David Arthur
Re: [VOTE] 3.2.0 RC1
Thanks for running the release, Bruno. I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the doc and the Javadocs. +1 (binding) Best, David On Thu, May 5, 2022 at 10:36 AM Jakub Scholz wrote: > > +1 (non-binding). > > I used the Scala 2.13 binaries and the staged Maven artifacts and ran > various tests with them. Thanks for doing the release. > > Jakub > > On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the second candidate for release of Apache Kafka 3.2.0. > > > > * log4j 1.x is replaced with reload4j (KAFKA-9366) > > * StandardAuthorizer for KRaft (KIP-801) > > * Send a hint to the partition leader to recover the partition (KIP-704) > > * Top-level error code field in DescribeLogDirsResponse (KIP-784) > > * kafka-console-producer writes headers and null values (KIP-798 and > > KIP-810) > > * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800) > > * Static membership protocol lets the leader skip assignment (KIP-814) > > * Rack-aware standby task assignment in Kafka Streams (KIP-708) > > * Interactive Query v2 (KIP-796, KIP-805, and KIP-806) > > * Connect APIs list all connector plugins and retrieve their > > configuration (KIP-769) > > * TimestampConverter SMT supports different unix time precisions (KIP-808) > > * Connect source tasks handle producer exceptions (KIP-779) > > > > > > Release notes for the 3.2.0 release: > > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html > > > > *** Please download, test and vote by Tuesday, May 10th, 9am PDT > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: > > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/ > > > > * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag: > > https://github.com/apache/kafka/releases/tag/3.2.0-rc1 > > > > * Documentation: > > https://kafka.apache.org/32/documentation.html > > > > * Protocol: > > https://kafka.apache.org/32/protocol.html > > > > * Successful Jenkins builds for the 3.2 branch: > > Unit/integration tests: I'll share a link once the builds complete > > System tests: > > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/ > > > > /** > > > > Thanks, > > Bruno > >
Re: [VOTE] 3.1.1 RC0
Hi Tom, Thanks for running the release. I wonder if we should include: https://github.com/apache/kafka/commit/134c432d6452de1bfb99d0f6b455a58c16bc626a. This is a follow up of KAFKA-13794. What do you think? Best, David On Fri, Apr 8, 2022 at 6:18 PM Tom Bentley wrote: > > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 3.1.1. > > Apache Kafka 3.1.1 is a bugfix release and 29 issues have been fixed > since 3.1.0. > > Release notes for the 3.1.1 release: > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html > > *** Please download, test and vote by Friday 15 April, 12:00 UTC > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/ > > * Maven artifacts to be voted upon: > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > * Javadoc: > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/ > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag: > https://github.com/apache/kafka/releases/tag/3.1.1-rc0 > > * Documentation: > https://kafka.apache.org/31/documentation.html > > * Protocol: > https://kafka.apache.org/31/protocol.html > > * Successful Jenkins builds for the 3.1 branch: > I will share a link one the build is complete. > > /** > > Thanks, > > Tom
[ANNOUNCE] Apache Kafka 3.1.0
The Apache Kafka community is pleased to announce the release for Apache Kafka 3.1.0. It is a major release that includes many new features, including: * Apache Kafka supports Java 17 * The FetchRequest supports Topic IDs (KIP-516) * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768) * Add broker count metrics (KIP-748) * Differentiate consistently metric latency measured in millis and nanos (KIP-773) * The eager rebalance protocol is deprecated (KAFKA-13439) * Add TaskId field to StreamsException (KIP-783) * Custom partitioners in foreign-key joins (KIP-775) * Fetch/findSessions queries with open endpoints for SessionStore/WindowStore (KIP-766) * Range queries with open endpoints (KIP-763) * Add total blocked time metric to Streams (KIP-761) * Add additional configuration to control MirrorMaker2 internal topics naming convention (KIP-690) You may read a more detailed list of features in the 3.1.0 blog post: https://blogs.apache.org/kafka/ All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.1.0/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#3.1.0 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 114 contributors to this release! A. Sophie Blee-Goldman, Alexander Iskuskov, Alexander Stohr, Almog Gavra, Andras Katona, Andrew Patterson, Andy Chambers, Andy Lapidas, Anna Sophie Blee-Goldman, Antony Stubbs, Arjun Satish, Bill Bejeck, Boyang Chen, Bruno Cadonna, CHUN-HAO TANG, Cheng Tan, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Colin P. McCabe, Cong Ding, Daniel Urban, David Arthur, David Jacot, David Mao, Dmitriy Fishman, Edoardo Comar, Ewen Cheslack-Postava, Greg Harris, Guozhang Wang, Igor Soarez, Ismael Juma, Israel Ekpo, Ivan Ponomarev, Jakub Scholz, James Galasyn, Jason Gustafson, Jeff Kim, Jim Galasyn, JoeCqupt, Joel Hamill, John Gray, John Roesler, Jongho Jeon, Jorge Esteban Quilcate Otoya, Jose Sancio, Josep Prat, José Armando García Sancio, Jun Rao, Justine Olshan, Kalpesh Patel, Kamal Chandraprakash, Kevin Zhang, Kirk True, Konstantine Karantasis, Kowshik Prakasam, Leah Thomas, Lee Dongjin, Lucas Bradstreet, Luke Chen, Manikumar Reddy, Matthew Wong, Matthias J. Sax, Michael Carter, Mickael Maison, Nigel Liang, Niket, Niket Goel, Oliver Hutchison, Omnia G H Ibrahim, Patrick Stuedi, Phil Hardwick, Prateek Agarwal, Rajini Sivaram, Randall Hauch, René Kerner, Richard Yu, Rohan, Ron Dagostino, Ryan Dielhenn, Sanjana Kaundinya, Satish Duggana, Sergio Peña, Sherzod Mamadaliev, Stanislav Vodetskyi, Ted Yu, Tom Bentley, Tomas Forsman, Tomer Wizman, Uwe Eisele, Victoria Xia, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson, Weisheng Yang, Xavier Léauté, Yanwen(Jason) Lin, Yi Ding, Zara Lim, andy0x01, dengziming, feyman2016, ik, ik.lim, jem, jiangyuan, kpatelatwork, leah, loboya~, lujiefsi, sebbASF, singingMan, vamossagar12, wenbingshen We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, David
Re: [VOTE] 3.1.0 RC1
Thanks for all the votes! I'm also +1 (binding) given that: * Ran the release and generated 3.1.0 RC1. * Verified all checksums and signatures. * Built from source and ran unit tests. * Installed 3.1.0 RC1 and performed the quickstart for both ZK and KRaft. * Spotchecked the Javadocs and the site doc. I will close the vote shortly. Best, David On Thu, Jan 20, 2022 at 1:21 PM Mickael Maison wrote: > > Thanks David for running this release! > > I checked signatures and keys, built from source and ran unit tests > with Java 11. I also ran the quickstart. > > +1 (binding) > > Thanks, > Mickael > > On Wed, Jan 19, 2022 at 9:20 PM Bill Bejeck wrote: > > > > Hi David, > > > > Thanks for running the release! > > > > I did the following: > > > >1. Validated all signatures and keys > >2. Built from source > >3. Ran all the unit tests > >4. Spot checked the java docs, and the site docs > >5. Ran a portion of the quick start > >6. Ran the Kafka Streams demo application > > > > I noticed one small error on the quick start page: the "Kafka Streams demo > > and > > the app development tutorial" links still reference the 2.5 release. > > This is a very minor issue and it doesn't require a new RC, we can simply > > update the links once released. > > > > +1(binding) > > > > Thanks, > > Bill > > > > > > On Wed, Jan 19, 2022 at 8:44 AM David Jacot > > wrote: > > > > > Thanks for all the votes! We are still missing two binding votes from > > > PMC members. > > > > > > Best, > > > David > > > > > > On Tue, Jan 18, 2022 at 5:00 PM Israel Ekpo wrote: > > > > > > > > Performed the following validations using the tools available here: > > > > > > > > https://github.com/izzyacademy/apache-kafka-release-party > > > > > > > >- Verified signatures, keys and hashes for release artifacts > > > >- Deployed Multi-Node Cluster in Legacy Mode (with Zookeeper) > > > >- Deployed Multi-Node Cluster in KRaft Mode (without Zookeeper) > > > >- Can confirm that KAFKA-13456 works as expected after switching to > > > 3.1.0 > > > >- Briefly Walked Through 3.1 Documentation, Javadocs and Protocol > > > Pages > > > > > > > > +1 (non-binding) for the release candidate > > > > > > > > Thanks for running this release > > > > > > > > Israel Ekpo > > > > Lead Instructor, IzzyAcademy.com > > > > https://www.youtube.com/c/izzyacademy > > > > https://izzyacademy.com/ > > > > > > > > > > > > On Wed, Jan 12, 2022 at 7:59 AM David Jacot wrote: > > > > > > > > > Hello Kafka users, developers and client-developers, > > > > > > > > > > This is the second candidate for release of Apache Kafka 3.1.0. > > > > > > > > > > * Apache Kafka supports Java 17 > > > > > * The FetchRequest supports Topic IDs (KIP-516) > > > > > * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768) > > > > > * Add broker count metrics (KIP-748) > > > > > * Differentiate consistently metric latency measured in millis and > > > > > nanos (KIP-773) > > > > > * The eager rebalance protocol is deprecated (KAFKA-13439) > > > > > * Add TaskId field to StreamsException (KIP-783) > > > > > * Custom partitioners in foreign-key joins (KIP-775) > > > > > * Fetch/findSessions queries with open endpoints for > > > > > SessionStore/WindowStore (KIP-766) > > > > > * Range queries with open endpoints (KIP-763) > > > > > * Add total blocked time metric to Streams (KIP-761) > > > > > * Add additional configuration to control MirrorMaker2 internal topics > > > > > naming convention (KIP-690) > > > > > > > > > > Release notes for the 3.1.0 release: > > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html > > > > > > > > > > *** Please download, test and vote by Monday, January 17, 9am PT > > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > > > > https://kafka.apache.org/KEYS > > > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/ > > > > > > > > > > * Maven artifacts to be voted upon: > > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > > > > > * Javadoc: > > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/ > > > > > > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag: > > > > > https://github.com/apache/kafka/releases/tag/3.1.0-rc1 > > > > > > > > > > * Documentation: > > > > > https://kafka.apache.org/31/documentation.html > > > > > > > > > > * Protocol: > > > > > https://kafka.apache.org/31/protocol.html > > > > > > > > > > * Successful Jenkins builds for the 3.1 branch: > > > > > Unit/integration tests: > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/ > > > > > System tests: > > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/ > > > > > > > > > > /** > > > > > > > > > > Thanks, > > > > > David > > > > > > > >
Re: [VOTE] 3.1.0 RC1
Thanks for all the votes! We are still missing two binding votes from PMC members. Best, David On Tue, Jan 18, 2022 at 5:00 PM Israel Ekpo wrote: > > Performed the following validations using the tools available here: > > https://github.com/izzyacademy/apache-kafka-release-party > >- Verified signatures, keys and hashes for release artifacts >- Deployed Multi-Node Cluster in Legacy Mode (with Zookeeper) >- Deployed Multi-Node Cluster in KRaft Mode (without Zookeeper) >- Can confirm that KAFKA-13456 works as expected after switching to 3.1.0 >- Briefly Walked Through 3.1 Documentation, Javadocs and Protocol Pages > > +1 (non-binding) for the release candidate > > Thanks for running this release > > Israel Ekpo > Lead Instructor, IzzyAcademy.com > https://www.youtube.com/c/izzyacademy > https://izzyacademy.com/ > > > On Wed, Jan 12, 2022 at 7:59 AM David Jacot wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the second candidate for release of Apache Kafka 3.1.0. > > > > * Apache Kafka supports Java 17 > > * The FetchRequest supports Topic IDs (KIP-516) > > * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768) > > * Add broker count metrics (KIP-748) > > * Differentiate consistently metric latency measured in millis and > > nanos (KIP-773) > > * The eager rebalance protocol is deprecated (KAFKA-13439) > > * Add TaskId field to StreamsException (KIP-783) > > * Custom partitioners in foreign-key joins (KIP-775) > > * Fetch/findSessions queries with open endpoints for > > SessionStore/WindowStore (KIP-766) > > * Range queries with open endpoints (KIP-763) > > * Add total blocked time metric to Streams (KIP-761) > > * Add additional configuration to control MirrorMaker2 internal topics > > naming convention (KIP-690) > > > > Release notes for the 3.1.0 release: > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html > > > > *** Please download, test and vote by Monday, January 17, 9am PT > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/ > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag: > > https://github.com/apache/kafka/releases/tag/3.1.0-rc1 > > > > * Documentation: > > https://kafka.apache.org/31/documentation.html > > > > * Protocol: > > https://kafka.apache.org/31/protocol.html > > > > * Successful Jenkins builds for the 3.1 branch: > > Unit/integration tests: > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/ > > System tests: > > https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/ > > > > /** > > > > Thanks, > > David > >
[VOTE] 3.1.0 RC1
Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 3.1.0. * Apache Kafka supports Java 17 * The FetchRequest supports Topic IDs (KIP-516) * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768) * Add broker count metrics (KIP-748) * Differentiate consistently metric latency measured in millis and nanos (KIP-773) * The eager rebalance protocol is deprecated (KAFKA-13439) * Add TaskId field to StreamsException (KIP-783) * Custom partitioners in foreign-key joins (KIP-775) * Fetch/findSessions queries with open endpoints for SessionStore/WindowStore (KIP-766) * Range queries with open endpoints (KIP-763) * Add total blocked time metric to Streams (KIP-761) * Add additional configuration to control MirrorMaker2 internal topics naming convention (KIP-690) Release notes for the 3.1.0 release: https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html *** Please download, test and vote by Monday, January 17, 9am PT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~dajac/kafka-3.1.0-rc1/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/ * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag: https://github.com/apache/kafka/releases/tag/3.1.0-rc1 * Documentation: https://kafka.apache.org/31/documentation.html * Protocol: https://kafka.apache.org/31/protocol.html * Successful Jenkins builds for the 3.1 branch: Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/ System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/ /** Thanks, David
[VOTE] 3.1.0 RC0
Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.1.0. * Apache Kafka supports Java 17 * The FetchRequest supports Topic IDs (KIP-516) * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768) * Add broker count metrics (KIP-748) * Differentiate consistently metric latency measured in millis and nanos (KIP-773) * The eager rebalance protocol is deprecated (KAFKA-13439) * Add TaskId field to StreamsException (KIP-783) * Custom partitioners in foreign-key joins (KIP-775) * Fetch/findSessions queries with open endpoints for SessionStore/WindowStore (KIP-766) * Range queries with open endpoints (KIP-763) * Add total blocked time metric to Streams (KIP-761) * Add additional configuration to control MirrorMaker2 internal topics naming convention (KIP-690) Release notes for the 3.1.0 release: https://home.apache.org/~dajac/kafka-3.1.0-rc0/RELEASE_NOTES.html *** Please download, test and vote by Friday, January 7, 9am PT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~dajac/kafka-3.1.0-rc0/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~dajac/kafka-3.1.0-rc0/javadoc/ * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag: https://github.com/apache/kafka/releases/tag/3.1.0-rc0 * Documentation: https://kafka.apache.org/31/documentation.html * Protocol: https://kafka.apache.org/31/protocol.html * Successful Jenkins builds for the 3.1 branch: Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/42/ System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.1/44/ /** Thanks, David
Re: [kafka-clients] Re: [DISCUSS] Before you Vote for Apache Kafka RCs, Watch This!
Israel, I am not aware of any apache docker repository. It seems preferable to keep it in your own repository at this stage. Otherwise, you would depend on committers/PMC members to update the image which is not ideal for your project, I suppose. Best, David On Sat, Sep 25, 2021 at 12:56 PM Israel Ekpo wrote: > David > > I have one more question: do we have a Docker Hub account where the Docker > images can be hosted? > > I am fine using my personal account but it would be better if it were > something like apache/kafka:3.1.0-rc2 or just kafka/3.1.0-rc2 instead of > izzyacademy/kafka:3.1.0-rc2 > > I can still use my personal account but it would look better and give > others opportunities to manage the images as well without depending on my > personal Docker Hub account > > Your initial feedback was great and I am going to trim the prep process by > prepping some base images to cut down the Docker image build elapsed time > > Thank you again > > Sincerely > Israel > > > > On Fri, Sep 24, 2021 at 6:45 PM Israel Ekpo wrote: > >> >> Thanks for the feedback David. I really appreciate it. >> >> I would work on eliminating some items in the docker container that are >> not needed and making this as a base image that already has >> those dependencies pre-baked. >> >> That could significantly trim the elapsed time between when they fire up >> the container and when they can get started with the validation. >> >> Let me update this and will share another update again soon. >> >> I appreciate this feedback >> >> Thanks. >> >> On Fri, Sep 24, 2021 at 10:51 AM 'David Jacot' via kafka-clients < >> kafka-clie...@googlegroups.com> wrote: >> >>> Hi Israel, >>> >>> Thank you for this initiative. >>> >>> The tool seems pretty cool and I think that it could be useful as well. I >>> haven't tried it yet though. >>> >>> I just watched the video and I found the part which requires to build >>> the docker container really long, perhaps even longer than the >>> validation itself (running the test apart). Could we simplify this? >>> >>> Automating the validation process is a great and risky thing at the >>> same time. Imagine if we would introduce a bug in the scripts. We >>> could all miss an issue in the RC. This is the advantage of our >>> current, boring and manual process. I feel like we would always >>> need some sort of manual sanity checks anyway. >>> >>> It is hard to say if this will really help to have more people validating >>> the release candidates. However, we could advertise it again when >>> we publish RCs for 3.1.0, by mentioning it in the thread, and see if >>> it helps folks. >>> >>> Cheers, >>> David >>> >>> On Fri, Sep 24, 2021 at 2:06 PM Israel Ekpo >>> wrote: >>> >>>> Hello Everyone, >>>> >>>> Please take a moment to review this and share your thoughts. It would be >>>> great to have more community involvement during release candidate >>>> validations >>>> >>>> I am wondering if this should be split up between the site docs repo and >>>> the core code repo or just in the code repo >>>> >>>> Should we include this in future release candidate voting notifications >>>> to >>>> the community? >>>> >>>> Also do you think we even need this at all? >>>> >>>> When you have a moment please let me know >>>> >>>> Thanks >>>> >>>> >>>> >>>> On Wed, Sep 15, 2021 at 7:18 PM Israel Ekpo >>>> wrote: >>>> >>>> > Before you Vote for Apache Kafka RCs, Watch This! >>>> > >>>> > https://youtu.be/T1VqFszLuQs >>>> > >>>> > Hello Kafka Community Members and welcome to the Apache Kafka Release >>>> > Party! >>>> > >>>> > As part of an effort (KAFKA-9861) to get more community participation >>>> > during release candidate (RC) validations, we have created the >>>> following >>>> > steps to hopefully allow more community members to participate >>>> thoroughly >>>> > in this validation process prior to casting their votes for specific >>>> RCs. >>>> > >>>> > It is also our hope that more community members will be able to >>>> > participate in th
Re: [DISCUSS] Before you Vote for Apache Kafka RCs, Watch This!
Hi Israel, Thank you for this initiative. The tool seems pretty cool and I think that it could be useful as well. I haven't tried it yet though. I just watched the video and I found the part which requires to build the docker container really long, perhaps even longer than the validation itself (running the test apart). Could we simplify this? Automating the validation process is a great and risky thing at the same time. Imagine if we would introduce a bug in the scripts. We could all miss an issue in the RC. This is the advantage of our current, boring and manual process. I feel like we would always need some sort of manual sanity checks anyway. It is hard to say if this will really help to have more people validating the release candidates. However, we could advertise it again when we publish RCs for 3.1.0, by mentioning it in the thread, and see if it helps folks. Cheers, David On Fri, Sep 24, 2021 at 2:06 PM Israel Ekpo wrote: > Hello Everyone, > > Please take a moment to review this and share your thoughts. It would be > great to have more community involvement during release candidate > validations > > I am wondering if this should be split up between the site docs repo and > the core code repo or just in the code repo > > Should we include this in future release candidate voting notifications to > the community? > > Also do you think we even need this at all? > > When you have a moment please let me know > > Thanks > > > > On Wed, Sep 15, 2021 at 7:18 PM Israel Ekpo wrote: > > > Before you Vote for Apache Kafka RCs, Watch This! > > > > https://youtu.be/T1VqFszLuQs > > > > Hello Kafka Community Members and welcome to the Apache Kafka Release > > Party! > > > > As part of an effort (KAFKA-9861) to get more community participation > > during release candidate (RC) validations, we have created the following > > steps to hopefully allow more community members to participate thoroughly > > in this validation process prior to casting their votes for specific RCs. > > > > It is also our hope that more community members will be able to > > participate in the process as a result. If you have any suggestions for > > improvement or would like to share changes, please feel free to send in a > > pull request or open issues against this github repo. > > > > I would like to solicit feedback from the users, community members, > > committers and PMC members on how this can be incorporated into the > release > > process and AK code base. > > > > My initial thoughts are to include the link to this resource or its > > modified form as part of the release candidate announcement to encourage > > users to participate in the RC validation process. > > > > Thanks in advance for your participation. > > > > *Process Simplification - Community Validation of Kafka Release > Candidates* > > https://issues.apache.org/jira/browse/KAFKA-9861 > > > > *Github Repository Accompanying this Tutorial* > > https://github.com/izzyacademy/apache-kafka-release-party > > > > *YouTube Video Explaining the Resources and How to Use It* > > https://youtu.be/T1VqFszLuQs > > >
[ANNOUNCE] Apache Kafka 2.8.1
The Apache Kafka community is pleased to announce the release for Apache Kafka 2.8.1 Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0 release. Please see the release notes for more information. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.8.1/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#2.8.1 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 35 contributors to this release! A. Sophie Blee-Goldman, Alexander Iskuskov, Andras Katona, Bill Bejeck, Bruno Cadonna, Chris Egerton, Colin Patrick McCabe, David Arthur, David Jacot, Davor Poldrugo, Dejan Stojadinović, Geordie, Guozhang Wang, Ismael Juma, Jason Gustafson, Jeff Kim, John Gray, John Roesler, Justine Olshan, Konstantine Karantasis, Lee Dongjin, Luke Chen, Matthias J. Sax, Michael Carter, Mickael Maison, Phil Hardwick, Rajini Sivaram, Randall Hauch, Shay Elkin, Stanislav Vodetskyi, Tom Bentley, vamossagar12, wenbingshen, YiDing-Duke We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, David
[RESULTS] [VOTE] Release Kafka version 2.8.1
This vote passes with 4 +1 votes (4 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Randall Hauch * Manikumar Reddy * Bill Bejeck * Mickael Maison Committers: * No votes Community: * No votes 0 votes * No votes -1 votes * No votes Vote thread: https://www.mail-archive.com/dev@kafka.apache.org/msg120966.html I'll continue with the release process and the release announcement will follow in the next few days. David
Re: [kafka-clients] Re: [VOTE] 2.8.1 RC1
Hi Israel, Thanks! That seems pretty cool. You should bring it up in an independent thread in order to give it more visibility. It might be lost in the middle of this vote thread. Best, David On Wed, Sep 15, 2021 at 3:39 PM David Jacot wrote: > Hi Chris, Josep, > > Thank you for reporting the issue and explaining the fix. JDK11 was > indeed used to generate the javadoc. I don't consider this as a blocker > for the release as we can fix it independently. > > Best, > David > > On Tue, Sep 14, 2021 at 11:37 PM Israel Ekpo wrote: > >> Hi Community Members, >> >> To encourage other community members to participate in the release >> candidate validations and voting, I have set up the following resource as >> part of the work for KAFKA-9861 >> >> It is a set of scripts and Docker images that allows community members to >> run local validations in a consistent manner. >> >> https://github.com/izzyacademy/apache-kafka-release-party >> >> Please take a look at the resource and share any feedback that you may >> have. I used it to perform my validation for the 3.0.0 Release Candidate. >> >> I plan to create a video tutorial that walks community members through how >> it can be used soon. Stay tuned. >> >> On Tue, Sep 14, 2021 at 3:13 PM 'Josep Prat' via kafka-clients < >> kafka-clie...@googlegroups.com> wrote: >> >> > Hi Ismael, >> > >> > Yes the 'fix' was merged in the 3.0.0 branch (trunk at that time). I >> also >> > checked the current 3.0.0 RC2 and the javadoc search works as expected. >> > >> > Best, >> > ——— >> > Josep Prat >> > >> > Aiven Deutschland GmbH >> > >> > Immanuelkirchstraße 26, 10405 Berlin >> > >> > Amtsgericht Charlottenburg, HRB 209739 B >> > >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> > >> > m: +491715557497 >> > >> > w: aiven.io >> > >> > e: josep.p...@aiven.io >> > >> > On Tue, Sep 14, 2021, 21:00 Ismael Juma wrote: >> > >> >> Josep, >> >> >> >> This is fixed in 3.0.0, right? >> >> >> >> Ismael >> >> >> >> On Tue, Sep 14, 2021 at 11:32 AM Josep Prat >> >> >> wrote: >> >> >> >> > Hi Chris and David, >> >> > >> >> > Regarding the API search, this is a known bug in JDK 11 and prior. It >> >> can't >> >> > correctly generate URLs for projects not using the module system >> (Kafka) >> >> > that link to projects that use it (Java Standard Library). The bug >> has >> >> been >> >> > fixed in JDK 12 and higher. >> >> > To have the search work as expected, javadoc must be generated with >> JDK >> >> 12 >> >> > or higher (for example JDK16 or the freshly released 17). For further >> >> info >> >> > check https://issues.apache.org/jira/browse/KAFKA-12782 or >> >> > https://github.com/apache/kafka/pull/10780 >> >> > >> >> > Best, >> >> > ——— >> >> > Josep Prat >> >> > >> >> > Aiven Deutschland GmbH >> >> > >> >> > Immanuelkirchstraße 26, 10405 Berlin >> >> > >> >> > Amtsgericht Charlottenburg, HRB 209739 B >> >> > >> >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> >> > >> >> > m: +491715557497 >> >> > >> >> > w: aiven.io >> >> > >> >> > e: josep.p...@aiven.io >> >> > >> >> > On Tue, Sep 14, 2021, 19:42 Manikumar >> >> wrote: >> >> > >> >> > > Hi, >> >> > > >> >> > > +1 (binding) >> >> > > >> >> > > - verified the signatures >> >> > > - ran the tests on the source archive >> >> > > - verified the quickstart with binary >> >> > > - verified the artifacts, javadoc >> >> > > >> >> > > Thanks for running the release! >> >> > > >> >> > > Thanks, >> >> > > Manikumar >> >> > > >> >> > > On Tue, Sep 14, 2021 at 10:36 PM Chris Egerton >> >> > > >> > > > >> >> > > wrote: >> >> > > &g
Re: [kafka-clients] Re: [VOTE] 2.8.1 RC1
l > >> > > > , > >> > > > which displayed a 404 "Not Found" error. > >> > > > > >> > > > This doesn't appear to be a regression as the same behavior occurs > >> for > >> > me > >> > > > when browsing the 2.8.0 Javadocs ( > >> > > > > >> https://kafka.apache.org/28/javadoc/index.html?overview-summary.html), > >> > > but > >> > > > I wanted to bring it up just in case. Normally I don't use the > >> search > >> > bar > >> > > > but with the removal of sidebar frames from Javadocs I've had to > >> start > >> > > > relying on it a little bit more. > >> > > > > >> > > > Cheers, > >> > > > > >> > > > Chris > >> > > > > >> > > > On Tue, Sep 14, 2021 at 12:56 PM Randall Hauch > > >> > > wrote: > >> > > > > >> > > > > Thanks for generating a new RC1 with the corrected site docs, > >> David. > >> > > > > > >> > > > > I was able to successfully complete the following: > >> > > > > > >> > > > > - Installed 2.8.1 RC0 and performed quickstart for broker and > >> Connect > >> > > > > - Verified signatures and checksums > >> > > > > - Verified the tag > >> > > > > - Manually compared the release notes to JIRA > >> > > > > - Build release archive from the tag, ran Connect tests, > installed > >> > > > locally, > >> > > > > and ran a portion of quickstart > >> > > > > - Manually spotchecked the Javadocs > >> > > > > - Verified the site docs at > >> > > > https://kafka.apache.org/28/documentation.html > >> > > > > has corrected version references, except for the tar and cd > >> commands > >> > in > >> > > > > step 1 of the > >> > > https://kafka.apache.org/28/documentation.html#quickstart. > >> > > > > > >> > > > > I think that last issue is minor and not worth another RC, since > >> the > >> > > > other > >> > > > > version references in > >> https://kafka.apache.org/28/documentation.html > >> > > do > >> > > > > reference 2.8.1 and we can easily fix it on the website, > >> optionally > >> > as > >> > > > part > >> > > > > of the other post-vote changes to the site. > >> > > > > > >> > > > > So I'm +1 (binding) > >> > > > > > >> > > > > Best regards, > >> > > > > > >> > > > > Randall > >> > > > > > >> > > > > On Tue, Sep 14, 2021 at 8:39 AM David Jacot > >> > > >> > > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Hello Kafka users, developers and client-developers, > >> > > > > > > >> > > > > > This is the second candidate for release of Apache Kafka > 2.8.1. > >> > > > > > > >> > > > > > Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues > since > >> > the > >> > > > > 2.8.0 > >> > > > > > release. Please see the release notes for more information. > >> > > > > > > >> > > > > > Release notes for the 2.8.1 release: > >> > > > > > > >> https://home.apache.org/~dajac/kafka-2.8.1-rc1/RELEASE_NOTES.html > >> > > > > > > >> > > > > > *** Please download, test and vote by Friday, September 17, > 9am > >> PT > >> > > > > > > >> > > > > > Kafka's KEYS file containing PGP keys we use to sign the > >> release: > >> > > > > > https://kafka.apache.org/KEYS > >> > > > > > > >> > > > > > * Release artifacts to be voted upon (source and binary): > >> > > > > > https://home.apache.org/~dajac/kafka-2.8.1-rc1/ > >> > > > > > > >> > > > > > * Maven artifacts to be voted upon: > >> > > > > > > >> > > > >> https://repository.apache.org/content/groups/staging/org/apache/kafka/ > >> > > > > > > >> > > > > > * Javadoc: > >> > > > > > https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/ > >> > > > > > > >> > > > > > * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag: > >> > > > > > https://github.com/apache/kafka/releases/tag/2.8.1-rc1 > >> > > > > > > >> > > > > > * Documentation: > >> > > > > > https://kafka.apache.org/28/documentation.html > >> > > > > > > >> > > > > > * Protocol: > >> > > > > > https://kafka.apache.org/28/protocol.html > >> > > > > > > >> > > > > > * Successful Jenkins builds for the 2.8 branch: > >> > > > > > Unit/integration tests: > >> > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/ > >> > > > > > System tests: > >> > > > > > > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/ > >> > > > > > > >> > > > > > /** > >> > > > > > > >> > > > > > Thanks, > >> > > > > > David > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > -- > > You received this message because you are subscribed to the Google Groups > > "kafka-clients" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to kafka-clients+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/kafka-clients/CAOJ18G7pukzdy1ZAtg1ar2TSFfz7de20xXoPgj4Qr5XRQt0yrA%40mail.gmail.com > > < > https://groups.google.com/d/msgid/kafka-clients/CAOJ18G7pukzdy1ZAtg1ar2TSFfz7de20xXoPgj4Qr5XRQt0yrA%40mail.gmail.com?utm_medium=email&utm_source=footer > > > > . > > >
Re: [VOTE] 2.8.1 RC0
I have started a new thread to vote for the RC1. Therefore, this vote is close. Thanks, David On Tue, Sep 14, 2021 at 11:14 AM David Jacot wrote: > Hi Randall, > > Thanks for your comments. > > I definitely missed updating > https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js. I > will fix this. I will cut a new RC to address this. > > Regarding https://github.com/apache/kafka-site/tree/asf-site/28, I did > not update the documentation > there yet. The issue is that it would update the current live > documentation (2.8) with references to an > unreleased version. We can't really do this, right? I could publish the > doc in a temporary directory > (e.g. 281) for the time being. What do you think? > > Best, > David > > On Tue, Sep 14, 2021 at 12:19 AM Randall Hauch wrote: > >> Thanks, David. >> >> I was able to successfully complete the following: >> >> - Build release archive from the tag, installed locally, and ran a portion >> of quickstart >> - Installed 2.8.1 RC0 and performed quickstart for broker and Connect >> - Verified signatures and checksums >> - Verified the tag >> - Compared the release notes to JIRA >> - Manually spotchecked the Javadocs >> >> However, generated docs at >> https://kafka.apache.org/28/documentation.html incorrectly >> reference the 2.8.0 version in the following sections: >> >> * https://kafka.apache.org/28/documentation.html#quickstart >> * https://kafka.apache.org/28/documentation.html#producerapi >> * https://kafka.apache.org/28/documentation.html#consumerapi >> * https://kafka.apache.org/28/documentation.html#streamsapi >> * https://kafka.apache.org/28/documentation.html#adminapi >> >> The way these are updated during the release process has changed recently. >> IIUC they are generated as part of the release build via the >> https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js >> file, which appears to have not been updated in the tag: >> https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js#L22 >> . >> Maybe >> the https://cwiki.apache.org/confluence/display/KAFKA/Release+Process >> needs >> to be tweaked to update this file even when cutting a patch release RC (it >> currently says to update this file in the section for major and minor >> releases). >> >> Also, the https://github.com/apache/kafka-site/tree/asf-site/28 history >> shows no updates since July. I guess that's possible, but might be worth >> double checking. >> >> Thanks! >> >> Randall >> >> On Fri, Sep 10, 2021 at 5:15 AM David Jacot >> wrote: >> >> > Hello Kafka users, developers and client-developers, >> > >> > This is the first candidate for release of Apache Kafka 2.8.1. >> > >> > Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the >> 2.8.0 >> > release. Please see the release notes for more information. >> > >> > Release notes for the 2.8.1 release: >> > https://home.apache.org/~dajac/kafka-2.8.1-rc0/RELEASE_NOTES.html >> > >> > *** Please download, test and vote by Friday, September 17, 9am PT >> > >> > Kafka's KEYS file containing PGP keys we use to sign the release: >> > https://kafka.apache.org/KEYS >> > >> > * Release artifacts to be voted upon (source and binary): >> > https://home.apache.org/~dajac/kafka-2.8.1-rc0/ >> > >> > * Maven artifacts to be voted upon: >> > https://repository.apache.org/content/groups/staging/org/apache/kafka/ >> > >> > * Javadoc: >> > https://home.apache.org/~dajac/kafka-2.8.1-rc0/javadoc/ >> > >> > * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag: >> > https://github.com/apache/kafka/releases/tag/2.8.1-rc0 >> > >> > * Documentation: >> > https://kafka.apache.org/28/documentation.html >> > >> > * Protocol: >> > https://kafka.apache.org/28/protocol.html >> > >> > * Successful Jenkins builds for the 2.8 branch: >> > Unit/integration tests: >> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/ >> > System tests: >> > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/ >> > >> > /** >> > >> > Thanks, >> > David >> > >> >
[VOTE] 2.8.1 RC1
Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 2.8.1. Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0 release. Please see the release notes for more information. Release notes for the 2.8.1 release: https://home.apache.org/~dajac/kafka-2.8.1-rc1/RELEASE_NOTES.html *** Please download, test and vote by Friday, September 17, 9am PT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~dajac/kafka-2.8.1-rc1/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/ * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag: https://github.com/apache/kafka/releases/tag/2.8.1-rc1 * Documentation: https://kafka.apache.org/28/documentation.html * Protocol: https://kafka.apache.org/28/protocol.html * Successful Jenkins builds for the 2.8 branch: Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/ System tests: https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/ /** Thanks, David
Re: [VOTE] 2.8.1 RC0
Hi Randall, Thanks for your comments. I definitely missed updating https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js. I will fix this. I will cut a new RC to address this. Regarding https://github.com/apache/kafka-site/tree/asf-site/28, I did not update the documentation there yet. The issue is that it would update the current live documentation (2.8) with references to an unreleased version. We can't really do this, right? I could publish the doc in a temporary directory (e.g. 281) for the time being. What do you think? Best, David On Tue, Sep 14, 2021 at 12:19 AM Randall Hauch wrote: > Thanks, David. > > I was able to successfully complete the following: > > - Build release archive from the tag, installed locally, and ran a portion > of quickstart > - Installed 2.8.1 RC0 and performed quickstart for broker and Connect > - Verified signatures and checksums > - Verified the tag > - Compared the release notes to JIRA > - Manually spotchecked the Javadocs > > However, generated docs at > https://kafka.apache.org/28/documentation.html incorrectly > reference the 2.8.0 version in the following sections: > > * https://kafka.apache.org/28/documentation.html#quickstart > * https://kafka.apache.org/28/documentation.html#producerapi > * https://kafka.apache.org/28/documentation.html#consumerapi > * https://kafka.apache.org/28/documentation.html#streamsapi > * https://kafka.apache.org/28/documentation.html#adminapi > > The way these are updated during the release process has changed recently. > IIUC they are generated as part of the release build via the > https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js > file, which appears to have not been updated in the tag: > https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js#L22 > . > Maybe > the https://cwiki.apache.org/confluence/display/KAFKA/Release+Process > needs > to be tweaked to update this file even when cutting a patch release RC (it > currently says to update this file in the section for major and minor > releases). > > Also, the https://github.com/apache/kafka-site/tree/asf-site/28 history > shows no updates since July. I guess that's possible, but might be worth > double checking. > > Thanks! > > Randall > > On Fri, Sep 10, 2021 at 5:15 AM David Jacot > wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of Apache Kafka 2.8.1. > > > > Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the > 2.8.0 > > release. Please see the release notes for more information. > > > > Release notes for the 2.8.1 release: > > https://home.apache.org/~dajac/kafka-2.8.1-rc0/RELEASE_NOTES.html > > > > *** Please download, test and vote by Friday, September 17, 9am PT > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~dajac/kafka-2.8.1-rc0/ > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: > > https://home.apache.org/~dajac/kafka-2.8.1-rc0/javadoc/ > > > > * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag: > > https://github.com/apache/kafka/releases/tag/2.8.1-rc0 > > > > * Documentation: > > https://kafka.apache.org/28/documentation.html > > > > * Protocol: > > https://kafka.apache.org/28/protocol.html > > > > * Successful Jenkins builds for the 2.8 branch: > > Unit/integration tests: > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/ > > System tests: > > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/ > > > > /** > > > > Thanks, > > David > > >
[VOTE] 2.8.1 RC0
Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.8.1. Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0 release. Please see the release notes for more information. Release notes for the 2.8.1 release: https://home.apache.org/~dajac/kafka-2.8.1-rc0/RELEASE_NOTES.html *** Please download, test and vote by Friday, September 17, 9am PT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~dajac/kafka-2.8.1-rc0/ * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~dajac/kafka-2.8.1-rc0/javadoc/ * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag: https://github.com/apache/kafka/releases/tag/2.8.1-rc0 * Documentation: https://kafka.apache.org/28/documentation.html * Protocol: https://kafka.apache.org/28/protocol.html * Successful Jenkins builds for the 2.8 branch: Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/ System tests: https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/ /** Thanks, David
Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis
Congrats, Konstantine. Well deserved! Best, David On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan wrote: > Congrats Konstantine > > On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison > wrote: > > > Hi, > > > > It's my pleasure to announce that Konstantine Karantasis is now a > > member of the Kafka PMC. > > > > Konstantine has been a Kafka committer since Feb 2020. He has remained > > active in the community since becoming a committer. > > > > Congratulations Konstantine! > > > > Mickael, on behalf of the Apache Kafka PMC > > >
Re: [ANNOUNCE] New Committer: Bruno Cadonna
Congrats Bruno! On Thu, Apr 8, 2021 at 9:53 AM Tom Bentley wrote: > Congratulations Bruno! > > On Thu, Apr 8, 2021 at 6:21 AM Konstantine Karantasis > wrote: > > > Congratulations Bruno! > > > > Konstantine > > > > On Wed, Apr 7, 2021 at 8:08 PM Sophie Blee-Goldman > > wrote: > > > > > Congrats! > > > > > > On Wed, Apr 7, 2021 at 6:32 PM Luke Chen wrote: > > > > > > > Congrats Bruno!! > > > > > > > > Luke > > > > > > > > On Thu, Apr 8, 2021 at 9:18 AM Matthias J. Sax > > wrote: > > > > > > > > > Congrats Bruno! Very well deserved! > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > On 4/7/21 3:51 PM, Bill Bejeck wrote: > > > > > > Congrats Bruno! Well deserved. > > > > > > > > > > > > Bill > > > > > > > > > > > > On Wed, Apr 7, 2021 at 6:34 PM Guozhang Wang > > > > > wrote: > > > > > > > > > > > >> Hello all, > > > > > >> > > > > > >> I'm happy to announce that Bruno Cadonna has accepted his > > invitation > > > > to > > > > > >> become an Apache Kafka committer. > > > > > >> > > > > > >> Bruno has been contributing to Kafka since Jan. 2019 and has > made > > 99 > > > > > >> commits and more than 80 PR reviews so far: > > > > > >> > > > > > >> https://github.com/apache/kafka/commits?author=cadonna > > > > > >> > > > > > >> He worked on a few key KIPs on Kafka Streams: > > > > > >> > > > > > >> * KIP-471: Expose RocksDB Metrics in Kafka Streams > > > > > >> * KIP-607: Add Metrics to Kafka Streams to Report Properties of > > > > RocksDB > > > > > >> * KIP-662: Throw Exception when Source Topics of a Streams App > are > > > > > Deleted > > > > > >> > > > > > >> Besides all the code contributions and reviews, he's also done a > > > > handful > > > > > >> for the community: multiple Kafka meetup talks in Berlin and > Kafka > > > > > Summit > > > > > >> talks, an introductory class to Kafka at Humboldt-Universität zu > > > > Berlin > > > > > >> seminars, and have co-authored a paper on Kafka's stream > > processing > > > > > >> semantics in this year's SIGMOD conference ( > > > > > >> https://en.wikipedia.org/wiki/SIGMOD). Bruno has also been > quite > > > > > active on > > > > > >> SO channels and AK mailings. > > > > > >> > > > > > >> Please join me to congratulate Bruno for all the contributions! > > > > > >> > > > > > >> -- Guozhang > > > > > >> > > > > > > > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck
Congrats, Bill! On Thu, Apr 8, 2021 at 9:54 AM Tom Bentley wrote: > Congratulations Bill! > > On Thu, Apr 8, 2021 at 2:36 AM Luke Chen wrote: > > > Congratulations Bill! > > > > Luke > > > > On Thu, Apr 8, 2021 at 9:17 AM Matthias J. Sax wrote: > > > > > Hi, > > > > > > It's my pleasure to announce that Bill Bejeck in now a member of the > > > Kafka PMC. > > > > > > Bill has been a Kafka committer since Feb 2019. He has remained > > > active in the community since becoming a committer. > > > > > > > > > > > > Congratulations Bill! > > > > > > -Matthias, on behalf of Apache Kafka PMC > > > > > >
Re: [ANNOUNCE] New committer: David Jacot
Thanks! On Mon, Oct 19, 2020 at 8:01 PM James Cheng wrote: > Congratulations, David! > > -James > > > On Oct 16, 2020, at 9:01 AM, Gwen Shapira wrote: > > > > The PMC for Apache Kafka has invited David Jacot as a committer, and > > we are excited to say that he accepted! > > > > David Jacot has been contributing to Apache Kafka since July 2015 (!) > > and has been very active since August 2019. He contributed several > > notable KIPs: > > > > KIP-511: Collect and Expose Client Name and Version in Brokers > > KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies: > > KIP-570: Add leader epoch in StopReplicaReques > > KIP-599: Throttle Create Topic, Create Partition and Delete Topic > Operations > > KIP-496 Added an API for the deletion of consumer offsets > > > > In addition, David Jacot reviewed many community contributions and > > showed great technical and architectural taste. Great reviews are hard > > and often thankless work - but this is what makes Kafka a great > > product and helps us grow our community. > > > > Thanks for all the contributions, David! Looking forward to more > > collaboration in the Apache Kafka community. > > > > -- > > Gwen Shapira > >
Re: [ANNOUNCE] New committer: Mickael Maison
Congrats Mickeal, well deserved! On Fri, Nov 8, 2019 at 8:56 AM Tom Bentley wrote: > Congratulations Mickael! > > On Fri, Nov 8, 2019 at 6:41 AM Vahid Hashemian > wrote: > > > Congrats Mickael, > > > > Well deserved! > > > > --Vahid > > > > On Thu, Nov 7, 2019 at 9:10 PM Maulin Vasavada < > maulin.vasav...@gmail.com> > > wrote: > > > > > Congratulations Mickael! > > > > > > On Thu, Nov 7, 2019 at 8:27 PM Manikumar > > > wrote: > > > > > > > Congrats Mickeal! > > > > > > > > On Fri, Nov 8, 2019 at 9:05 AM Dong Lin wrote: > > > > > > > > > Congratulations Mickael! > > > > > > > > > > On Thu, Nov 7, 2019 at 1:38 PM Jun Rao wrote: > > > > > > > > > > > Hi, Everyone, > > > > > > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka > > committer > > > > > > Mickael > > > > > > Maison. > > > > > > > > > > > > Mickael has been contributing to Kafka since 2016. He proposed > and > > > > > > implemented multiple KIPs. He has also been propomating Kafka > > through > > > > > blogs > > > > > > and public talks. > > > > > > > > > > > > Congratulations, Mickael! > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun (on behalf of the Apache Kafka PMC) > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks! > > --Vahid > > >
Re: A new way migrate replica from old broker to new one
Hello, Yes, that should be fine. If you move the data to a new machine and use the corresponding broker.id, it is basically the same broker but on different vm. Best, David On Fri, Aug 16, 2019 at 10:03 AM Xiangyuan LI wrote: > Hi, > Now our kafka cluster has 5 brokers that broker.id is 1,2,3,4,5. > > We want to decommission broker 1 and add new broker 6,we expect that all > replica in broker 1 could migrate to broker 6,and nothing change for 2~5. > > Now we have an idea.our servers are in aws,we plan to remove disk from > broker 1,and add it to broker 6,then modify broker6's broker.id to 1,and > then add it to cluster. > > now we have broker 2~6,but broker 6's broker.id is 1,i want to know whther > we could do so? > > thx.. >
Re: Multiple bootstrap clusters in kafka producer config
Hello, As of today, the producer is able to talk to only one Kafka cluster. *bootstrap.servers* is to provide a list of brokers belonging to the same cluster to ensure at least one is available. In your case, you have to replicate the data from the first cluster to the second cluster with mirror maker for instance. Best, David On Mon, Aug 12, 2019 at 11:13 AM lsroudi abdel wrote: > I guess it more about replication, you can push your data in one choosen > cluster and replicate your data to the second one by using mirror maker or > confluent replicator or the new open sourced project by LinkedIn, > "Brooklin" > > Le lun. 12 août 2019 à 10:19, Garvit Sharma a > écrit : > > > Thanks, Isroudi. But in my usecase there are two separate zookeeper as > > well. Let me know how would it handle. > > > > On Mon, Aug 12, 2019 at 1:45 PM lsroudi abdel wrote: > > > > > If it is two cluster it mean you have two zookeeper, I guess you can't > do > > > that, I f you have one zookeeper it will be ok, in the case you have > one > > > zookeeper it's just an architecture with replica across two data > center, > > I > > > hope it clear for you > > > > > > Le lun. 12 août 2019 à 09:22, Garvit Sharma a > > > écrit : > > > > > > > Hi, > > > > > > > > I have 2 kafka clusters in different data-centers so can I provide > the > > > dns > > > > hostnames of both the clusters separated by comma in > > *bootstrap.servers* > > > > key in producer config ? > > > > > > > > If I provide two different clusters in *bootstrap.servers *then how > > would > > > > the events get published ? > > > > Would events get published to both the clusters or either of them > > > randomly > > > > ? > > > > > > > > Looking forward to hearing from you. > > > > > > > > Thanks > > > > > > > > > >
Re: Kafka cluster metadata update question
Hello Emanuel, It does not work because Kafka advertises the IP it is configured with (the ones of the broker) in the metadata response. I think you have two options: 1. You configure the default listener of each broker to advertise the IP address of their proxy instance. It means that the inter-broker traffic will also go trough the proxies; or 2. You keep the default listener with the internal IP and add another one in the config which advertises the external IP (the proxy one). There is a good explanation available here: https://rmoff.net/2018/08/02/kafka-listeners-explained/ Best, David On Mon, Aug 12, 2019 at 5:32 PM Emanuel Muhr wrote: > Hi, > > we are using Kafka in an environment where we have restricted access to > the Kafka brokers. > The access needs to happen via TCP proxies. So, in my setup I have 3 Kafka > brokers (broker1-3) for which I created 3 proxy instances and I have set > these proxy instances as the bootstrap servers: > bootstrap.servers = [ssl://proxy1:9092, ssl://proxy2:9092, > ssl://proxy3:9092] > > Upon startup of a Kafka client I see that there is a successful metadata > update and the client receive the original kafka broker names: > [2019-08-12 16:09:56,921] DEBUG Updated cluster metadata version 2 to > Cluster(id = lFFAv0okS12Vg2rvVbsKdg, nodes = [broker1:9092 (id: 1 rack: > r1), broker3:9092 (id: 3 rack: r3), broker2:9092 (id: 2 rack: r2)], > partitions = [Partition(topic = mytopic, partition = 0, leader = 2, > replicas = [2], isr = [2], offlineReplicas = [])], controller = > broker1:9092 (id: 1 rack: r1)) (org.apache.kafka.clients.Metadata) > > So, the initial connection via the proxy instances does work correctly. > The Kafka documentation states that the bootstrap.servers are only used for > bootstrapping so I assume that if the hostnames do not match the ones from > the brokers they will be replaced. > Additionally I found this C++ library documentation ( > https://github.com/edenhill/librdkafka/wiki/FAQ#number-of-broker-tcp-connections) > which states the following: > The initial bootstrap broker connections will only be used for Metadata > queries, unless the hostname and port of a bootstrap broker exactly matches > the hostname and port of a broker returned in the Metadata response (this > is the advertised.listeners broker configuration property), in which case > the bootstrap broker connection will be associated with that broker's > broker id and used for the full protocol set (such as producing or > consuming). > > We do not use this library but Spring Kafka Java library instead. However, > the documentation makes sense with what I'm observing. > > I'm asking if there is any way to disable this behavior and let Kafka use > the hostnames from the bootstrap broker list for producing and consuming as > well. > I do not see any other way so that we can use Kafka in this restricted > scenario unless we introduce some DNS rerouting. > > I would appreciate any insights on this topic if I'm missing information > or if there is another way to achieve the successful connection under these > circumstances. > > Thanks > Emanuel > > > > >
