Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0
Hello Kafka Team, We have an environment where Kafka Broker can go down for whatever reason. Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to drop messages when brokers were NOT available. Now the issue is we get data loss due to METADATA not being available and get this exception “*Topic not present in metadata after 0 ms.”. *This is due to the fast metadata has expired and the next request to send an event does not have metadata. Why does Kafka have his design? Why can’t Kafka distinguish between Broker down vs metadata refresh not available? Is it reasonable to expect metadata would refresh BEFORE it expires so metadata refresh doesn’t need before it expires? Have Metadata ready before expires? Any particular reason send() has wait for metadata refresh vs background thread that automatically refreshes metadata before it expires, hence send() method never incur wait(). Let me know what suggestion you have to prevent the application thread from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs metadata is NOT available due to expiration. Let me know your suggestions and what you think about metadata refresh. Should Kafka Producer be proactively refreshing metadata intelligently rather than what the producer does today? Thanks, Bhavesh
CVE-2022-34917: Unauthenticated clients may cause OutOfMemoryError on Apache Kafka Brokers
Severity: High Description: A security vulnerability has been identified in Apache Kafka. It affects all releases since 2.8.0. The vulnerability allows malicious unauthenticated clients to allocate large amounts of memory on brokers. This can lead to brokers hitting OutOfMemoryException and causing denial of service. Example scenarios: - Kafka cluster without authentication: Any clients able to establish a network connection to a broker can trigger the issue. - Kafka cluster with SASL authentication: Any clients able to establish a network connection to a broker, without the need for valid SASL credentials, can trigger the issue. - Kafka cluster with TLS authentication: Only clients able to successfully authenticate via TLS can trigger the issue. We advise the users to upgrade the Kafka installations to one of the 3.2.3, 3.1.2, 3.0.2, 2.8.2 versions. Credit: Apache Kafka would like to thank Mickael Maison, Tom Bentley and Daniel Collins for reporting this issue. References: https://kafka.apache.org/cve-list
[ANNOUNCE] Apache Kafka 3.2.3
The Apache Kafka community is pleased to announce the release for Apache Kafka 3.2.3 Apache Kafka 3.2.3 is a bugfix release and it contains important security fixes. This also fixes 7 issues since the 3.2.1 release. Please see the release notes for more information. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.2.3/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#3.2.3 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 12 contributors to this release! Andrew Borley, Andrew Dean, Colin Patrick McCabe, David Arthur, Derek Troy-West, Divij Vaidya, Jason Gustafson, Manikumar Reddy, Mickael Maison, Philip Nee, Thomas Cooper, Tom Bentley Thanks to Mickael Maison and Tom Bentley for driving this release. We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Apache Kafka PMC
[ANNOUNCE] Apache Kafka 3.1.2
The Apache Kafka community is pleased to announce the release for Apache Kafka 3.1.2 Apache Kafka 3.1.2 is a bugfix release and it contains important security fixes. It also fixes 4 issues since the 3.1.1 release. Please see the release notes for more information. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.1.2/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#3.1.2 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 16 contributors to this release! Andrew Borley, Bruno Cadonna, Colin Patrick McCabe, David Jacot, Derek Troy-West, Divij Vaidya, Guozhang Wang, Ismael Juma, Jason Gustafson, Kirk True, Lucas Bradstreet, Manikumar Reddy, Mickael Maison, nicolasguyomar, Niket, Tom Bentley Thanks to Mickael Maison and Tom Bentley for driving this release. We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Apache Kafka PMC
[ANNOUNCE] Apache Kafka 3.0.2
The Apache Kafka community is pleased to announce the release for Apache Kafka 3.0.2 Apache Kafka 3.0.2 is a bugfix release and it contains important security fixes. It also fixes 10 issues since the 3.0.1 release. Please see the release notes for more information. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.0.2/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#3.0.2 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 22 contributors to this release! Andrew Borley, Bounkong Khamphousone, Colin Patrick McCabe, David Jacot, Derek Troy-West, Ismael Juma, Jason Gustafson, Jules Ivanic, Justine Olshan, Konstantine Karantasis, Lucas Bradstreet, Manikumar Reddy, Mickael Maison, nicolasguyomar, Niket, Philip Nee, Randall Hauch, Stanislav Vodetskyi, Tom Bentley, Vincent Jiang, Xiaoyue Xue, Yang Yu Thanks to Tom Bentley for driving this release. We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Apache Kafka PMC
[ANNOUNCE] Apache Kafka 2.8.2
The Apache Kafka community is pleased to announce the release for Apache Kafka 2.8.2 Apache Kafka 2.8.2 is a bugfix release and it contains important security fixes. It also fixes 11 issues since the 2.8.1 release. Please see the release notes for more information. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.8.2/RELEASE_NOTES.html You can download the source and binary release (Scala 2.12 and 2.13) from: https://kafka.apache.org/downloads#2.8.2 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 17 contributors to this release! A. Sophie Blee-Goldman, Andrew Borley, Bruno Cadonna, Colin Patrick McCabe, David Jacot, Jason Gustafson, jiangyuan, Justine Olshan, Luke Chen, Manikumar Reddy, Matthias J. Sax, Oliver Hutchison, Philip Nee, Prateek Agarwal, prince-mahajan, Randall Hauch, Stanislav Vodetskyi We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, Manikumar
