Re: [VOTE] 3.2.1 RC3

[Randall Hauch]

Thanks for the RC, David.

I was able to successfully complete the following:

- Installed 3.2.1 RC3 and performed quickstart for broker and
Connect (using Java 17)
- Verified signatures and checksums
- Verified the tag
- Manually compared the release notes to JIRA
- Build release archive from the tag, installed locally, and ran a portion
of quickstart
- Manually spotchecked the Javadocs and release notes linked above
- The site docs at https://kafka.apache.org/32/documentation.html still
reference the 3.2.0 version (as expected), but I verified that putting the
contents of
https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/kafka_2.12-3.2.1-site-docs.tgz
into the "32" directory of a local Apache server running
https://github.com/apache/kafka-site, and the proper 3.2.1 version was
referenced.

So I'm +1 (binding)

Best regards,

Randall

On Thu, Jul 21, 2022 at 8:15 PM David Arthur  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first release candidate of Apache Kafka 3.2.1.
>
> This is a bugfix release with several fixes since the release of 3.2.0. A
> few of the major issues include:
>
> * KAFKA-14062 OAuth client token refresh fails with SASL extensions
> * KAFKA-14079 Memory leak in connectors using errors.tolerance=all
> * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck
>
>
> Release notes for the 3.2.1 release:
> https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/RELEASE_NOTES.html
>
>
>
>  Please download, test and vote by Wednesday July 27, 2022 at 17:00 PT.
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/
>
> Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> Javadoc: https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/javadoc/
>
> Tag to be voted upon (off 3.2 branch) is the 3.2.1 tag:
> https://github.com/apache/kafka/releases/tag/3.2.1-rc3
>
> Documentation: https://kafka.apache.org/32/documentation.html
>
> Protocol: https://kafka.apache.org/32/protocol.html
>
>
> The past few builds have had flaky test failures. I will update this thread
> with passing build links soon.
>
> Unit/Integration test job:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/
> System test job:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/
>
>
> Thanks!
> David Arthur
>

Re: [ANNOUNCE] New committer: Luke Chen

[Randall Hauch]

Congratulations, Luke!

On Wed, Feb 9, 2022 at 11:02 PM Matthias J. Sax  wrote:

> Congratulations! Glad to have you onboard, Luke!
>
> -Matthias
>
> On 2/9/22 16:37, Bill Bejeck wrote:
> > Congrats Luke! Well deserved.
> >
> > -Bill
> >
> > On Wed, Feb 9, 2022 at 7:25 PM Israel Ekpo  wrote:
> >
> >> Congratulations Luke!
> >>
> >> Thank you for your service
> >>
> >> On Wed, Feb 9, 2022 at 6:22 PM Guozhang Wang 
> wrote:
> >>
> >>> The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer
> >> and
> >>> we are pleased to announce that he has accepted!
> >>>
> >>> Luke has been actively contributing to Kafka since early 2020. He has
> >>> made more than 120 commits on various components of Kafka, with notable
> >>> contributions to the rebalance protocol in Consumer and Streams
> (KIP-766,
> >>> KIP-726, KIP-591, KAFKA-12675 and KAFKA12464, to just name a few), as
> >> well
> >>> as making an impact on improving test stability of the project. Aside
> >> from
> >>> all his code contributions, Luke has been a great participant in
> >>> discussions across the board, a very active and helpful reviewer of
> other
> >>> contributors' works, all of which are super valuable and highly
> >> appreciated
> >>> by the community.
> >>>
> >>>
> >>> Thanks for all of your contributions Luke. Congratulations!
> >>>
> >>> -- Guozhang, on behalf of the Apache Kafka PMC
> >>>
> >> --
> >> Israel Ekpo
> >> Lead Instructor, IzzyAcademy.com
> >> https://www.youtube.com/c/izzyacademy
> >> https://izzyacademy.com/
> >>
> >
>

Re: [VOTE] 3.1.0 RC1

[Randall Hauch]

Thanks for the RC, David.

I was able to successfully complete the following:

- Installed 3.1.0 RC1 and performed quickstart for broker and Connect
(using Java 17)
- Verified signatures and checksums
- Verified the tag
- Manually compared the release notes to JIRA
- Build release archive from the tag, ran Connect tests, installed
locally, and ran a portion of quickstart
- Manually spotchecked the Javadocs linked above
- Verified the site docs at
https://kafka.apache.org/31/documentation.html has the correct version
references

So I'm +1 (binding)

Best regards,

Randall

On Wed, Jan 12, 2022 at 6:59 AM David Jacot  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.1.0.
>
> * Apache Kafka supports Java 17
> * The FetchRequest supports Topic IDs (KIP-516)
> * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> * Add broker count metrics (KIP-748)
> * Differentiate consistently metric latency measured in millis and
> nanos (KIP-773)
> * The eager rebalance protocol is deprecated (KAFKA-13439)
> * Add TaskId field to StreamsException (KIP-783)
> * Custom partitioners in foreign-key joins (KIP-775)
> * Fetch/findSessions queries with open endpoints for
> SessionStore/WindowStore (KIP-766)
> * Range queries with open endpoints (KIP-763)
> * Add total blocked time metric to Streams (KIP-761)
> * Add additional configuration to control MirrorMaker2 internal topics
> naming convention (KIP-690)
>
> Release notes for the 3.1.0 release:
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, January 17, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag:
> https://github.com/apache/kafka/releases/tag/3.1.0-rc1
>
> * Documentation:
> https://kafka.apache.org/31/documentation.html
>
> * Protocol:
> https://kafka.apache.org/31/protocol.html
>
> * Successful Jenkins builds for the 3.1 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/
>
> /**
>
> Thanks,
> David

CVE-2021-38153: Timing Attack Vulnerability for Apache Kafka Connect and Clients

[Randall Hauch]

Severity: moderate

Description:

Some components in Apache Kafka use `Arrays.equals` to validate a password or 
key, which is vulnerable to timing attacks that make brute force attacks for 
such credentials more likely to be successful. Users should upgrade to 2.8.1 or 
higher, or 3.0.0 or higher where this vulnerability has been fixed. The 
affected versions include Apache Kafka 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 
2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.6.0, 2.6.1, 2.6.2, 
2.7.0, 2.7.1, and 2.8.0.

Credit:

Apache Kafka would like to thank J. Santilli for reporting this issue.

References:

https://kafka.apache.org/cve-list

Re: [ANNOUNCE] Apache Kafka 2.8.1

[Randall Hauch]

Thank you, David, for serving as the release manager for 2.8.1. Great work!

And congratulations to the Apache Kafka community!

Best regards,

Randall

On Mon, Sep 20, 2021 at 2:48 AM David Jacot  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 2.8.1
>
> Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0
> release. Please see the release notes for more information.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.8.1/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.8.1
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 35 contributors to this release!
>
> A. Sophie Blee-Goldman, Alexander Iskuskov, Andras Katona, Bill Bejeck,
> Bruno Cadonna, Chris Egerton, Colin Patrick McCabe, David Arthur, David
> Jacot,
> Davor Poldrugo, Dejan Stojadinović, Geordie, Guozhang Wang, Ismael Juma,
> Jason Gustafson, Jeff Kim, John Gray, John Roesler, Justine Olshan,
> Konstantine Karantasis, Lee Dongjin, Luke Chen, Matthias J. Sax, Michael
> Carter,
> Mickael Maison, Phil Hardwick, Rajini Sivaram, Randall Hauch, Shay Elkin,
> Stanislav Vodetskyi, Tom Bentley, vamossagar12, wenbingshen, YiDing-Duke
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
> David
>

Re: [VOTE] 2.8.1 RC1

[Randall Hauch]

Thanks for generating a new RC1 with the corrected site docs, David.

I was able to successfully complete the following:

- Installed 2.8.1 RC0 and performed quickstart for broker and Connect
- Verified signatures and checksums
- Verified the tag
- Manually compared the release notes to JIRA
- Build release archive from the tag, ran Connect tests, installed locally,
and ran a portion of quickstart
- Manually spotchecked the Javadocs
- Verified the site docs at https://kafka.apache.org/28/documentation.html
has corrected version references, except for the tar and cd commands in
step 1 of the https://kafka.apache.org/28/documentation.html#quickstart.

I think that last issue is minor and not worth another RC, since the other
version references in https://kafka.apache.org/28/documentation.html do
reference 2.8.1 and we can easily fix it on the website, optionally as part
of the other post-vote changes to the site.

So I'm +1 (binding)

Best regards,

Randall

On Tue, Sep 14, 2021 at 8:39 AM David Jacot 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 2.8.1.
>
> Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0
> release. Please see the release notes for more information.
>
> Release notes for the 2.8.1 release:
> https://home.apache.org/~dajac/kafka-2.8.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, September 17, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~dajac/kafka-2.8.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/
>
> * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag:
> https://github.com/apache/kafka/releases/tag/2.8.1-rc1
>
> * Documentation:
> https://kafka.apache.org/28/documentation.html
>
> * Protocol:
> https://kafka.apache.org/28/protocol.html
>
> * Successful Jenkins builds for the 2.8 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/
>
> /**
>
> Thanks,
> David
>

Re: [VOTE] 2.8.1 RC0

[Randall Hauch]

Thanks, David.

I was able to successfully complete the following:

- Build release archive from the tag, installed locally, and ran a portion
of quickstart
- Installed 2.8.1 RC0 and performed quickstart for broker and Connect
- Verified signatures and checksums
- Verified the tag
- Compared the release notes to JIRA
- Manually spotchecked the Javadocs

However, generated docs at
https://kafka.apache.org/28/documentation.html incorrectly
reference the 2.8.0 version in the following sections:

* https://kafka.apache.org/28/documentation.html#quickstart
* https://kafka.apache.org/28/documentation.html#producerapi
* https://kafka.apache.org/28/documentation.html#consumerapi
* https://kafka.apache.org/28/documentation.html#streamsapi
* https://kafka.apache.org/28/documentation.html#adminapi

The way these are updated during the release process has changed recently.
IIUC they are generated as part of the release build via the
https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js
file, which appears to have not been updated in the tag:
https://github.com/apache/kafka/blob/2.8.1-rc0/docs/js/templateData.js#L22.
Maybe
the https://cwiki.apache.org/confluence/display/KAFKA/Release+Process needs
to be tweaked to update this file even when cutting a patch release RC (it
currently says to update this file in the section for major and minor
releases).

Also, the https://github.com/apache/kafka-site/tree/asf-site/28 history
shows no updates since July. I guess that's possible, but might be worth
double checking.

Thanks!

Randall

On Fri, Sep 10, 2021 at 5:15 AM David Jacot 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 2.8.1.
>
> Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0
> release. Please see the release notes for more information.
>
> Release notes for the 2.8.1 release:
> https://home.apache.org/~dajac/kafka-2.8.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, September 17, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~dajac/kafka-2.8.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~dajac/kafka-2.8.1-rc0/javadoc/
>
> * Tag to be voted upon (off 2.8 branch) is the 2.8.1 tag:
> https://github.com/apache/kafka/releases/tag/2.8.1-rc0
>
> * Documentation:
> https://kafka.apache.org/28/documentation.html
>
> * Protocol:
> https://kafka.apache.org/28/protocol.html
>
> * Successful Jenkins builds for the 2.8 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/80/
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.8/214/
>
> /**
>
> Thanks,
> David
>

Re: [ANNOUNCE] Apache Kafka 2.6.2

[Randall Hauch]

Than you for driving this release, Sophie! And thanks to everyone that
contributed. Congratulations!

Best regards,

Randall

On Wed, Apr 21, 2021 at 3:43 PM Sophie Blee-Goldman 
wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.6.2
>
> This is a bug fix release and it includes fixes and improvements from
> 35 JIRAs, including a few critical bugs.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.6.2/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12
> and 2.13) from:
> https://kafka.apache.org/downloads#2.6.2
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 26 contributors to this release!
>
> A. Sophie Blee-Goldman, Andy Wilkinson, Bob Barrett, Bruno Cadonna,
> Chia-Ping Tsai, Chris Egerton, dengziming, Greg Harris, Ismael Juma, Jim
> Galasyn, John Roesler, Julien Jean Paul Sirocchi, Lee Dongjin, Luke Chen,
> Manikumar Reddy, Matthias J. Sax, Michael G. Noll, Mickael Maison, Rajini
> Sivaram, Ramesh Krishnan M, Randall Hauch, Sharath Bhat, Stanislav
> Vodetskyi, Walker Carlson, Yilong Chang
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Cheers,
>
> Sophie
>

Re: [ANNOUNCE] Apache Kafka 2.7.0

[Randall Hauch]

Fantastic! Thanks for driving the release, Bill.

Congratulations to the whole Kafka community.

On Mon, Dec 21, 2020 at 5:55 PM Gwen Shapira  wrote:

> woooh!!!
>
> Great job on the release Bill and everyone!
>
> On Mon, Dec 21, 2020 at 8:01 AM Bill Bejeck  wrote:
> >
> > The Apache Kafka community is pleased to announce the release for Apache
> > Kafka 2.7.0
> >
> > * Configurable TCP connection timeout and improve the initial metadata
> fetch
> > * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> > part 1)
> > * Throttle Create Topic, Create Partition and Delete Topic Operations
> > * Add TRACE-level end-to-end latency metrics to Streams
> > * Add Broker-side SCRAM Config API
> > * Support PEM format for SSL certificates and private key
> > * Add RocksDB Memory Consumption to RocksDB Metrics
> > * Add Sliding-Window support for Aggregations
> >
> > This release also includes a few other features, 53 improvements, and 91
> > bug fixes.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/2.7.0/RELEASE_NOTES.html
> >
> > You can read about some of the more prominent changes in the Apache Kafka
> > blog:
> > https://blogs.apache.org/kafka/entry/what-s-new-in-apache4
> >
> > You can download the source and binary release (Scala 2.12, 2.13) from:
> > https://kafka.apache.org/downloads#2.7.0
> >
> >
> ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 117 contributors to this release!
> >
> > A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Adem Efe Gencer,
> > albert02lowis, Alex Diachenko, Andras Katona, Andre Araujo, Andrew Choi,
> > Andrew Egelhofer, Andy Coates, Ankit Kumar, Anna Povzner, Antony Stubbs,
> > Arjun Satish, Ashish Roy, Auston, Badai Aqrandista, Benoit Maggi, bill,
> > Bill Bejeck, Bob Barrett, Boyang Chen, Brian Byrne, Bruno Cadonna, Can
> > Cecen, Cheng Tan, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> > David Arthur, David Jacot, David Mao, Dhruvil Shah, Dima Reznik, Edoardo
> > Comar, Ego, Evelyn Bayes, feyman2016, Gal Margalit, gnkoshelev, Gokul
> > Srinivas, Gonzalo Muñoz, Greg Harris, Guozhang Wang, high.lee,
> huangyiming,
> > huxi, Igor Soarez, Ismael Juma, Ivan Yurchenko, Jason Gustafson, Jeff
> Kim,
> > jeff kim, Jesse Gorzinski, jiameixie, Jim Galasyn, JoelWee, John Roesler,
> > John Thomas, Jorge Esteban Quilcate Otoya, Julien Jean Paul Sirocchi,
> > Justine Olshan, khairy, Konstantine Karantasis, Kowshik Prakasam, leah,
> Lee
> > Dongjin, Leonard Ge, Levani Kokhreidze, Lucas Bradstreet, Lucent-Wong,
> Luke
> > Chen, Mandar Tillu, manijndl7, Manikumar Reddy, Mario Molina, Matthias J.
> > Sax, Micah Paul Ramos, Michael Bingham, Mickael Maison, Navina Ramesh,
> > Nikhil Bhatia, Nikolay, Nikolay Izhikov, Ning Zhang, Nitesh Mor, Noa
> > Resare, Rajini Sivaram, Raman Verma, Randall Hauch, Rens Groothuijsen,
> > Richard Fussenegger, Rob Meng, Rohan, Ron Dagostino, Sanjana Kaun

Re: [ANNOUNCE] Apache Kafka 2.5.1

[Randall Hauch]

Thanks, John, for driving this release!

Randall

On Tue, Aug 11, 2020 at 10:33 PM Ismael Juma  wrote:

> Thanks for driving the release John!
>
> Ismael
>
> On Tue, Aug 11, 2020 at 1:40 PM John Roesler  wrote:
>
> > The Apache Kafka community is pleased to announce the
> > release for Apache Kafka 2.5.1
> >
> > This is a bug fix release, and it includes fixes and
> > improvements for 72 issues, including some critical bugs.
> >
> > All of the changes in this release can be found in the
> > release notes:
> > https://www.apache.org/dist/kafka/2.5.1/RELEASE_NOTES.html
> >
> >
> > You can download the source and binary release (Scala 2.12
> > and 2.13) from:
> > https://kafka.apache.org/downloads#2.5.1
> >
> > 
> > ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four
> > core APIs:
> >
> >
> > ** The Producer API allows an application to publish a
> > stream records to one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to
> > one or more topics and process the stream of records
> > produced to them.
> >
> > ** The Streams API allows an application to act as a stream
> > processor, consuming an input stream from one or more topics
> > and producing an output stream to one or more output topics,
> > effectively transforming the input streams to output
> > streams.
> >
> > ** The Connector API allows building and running reusable
> > producers or consumers that connect Kafka topics to existing
> > applications or data systems. For example, a connector to a
> > relational database might capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of
> > application:
> >
> > ** Building real-time streaming data pipelines that reliably
> > get data between systems or applications.
> >
> > ** Building real-time streaming applications that transform
> > or react to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies
> > worldwide, including Capital One, Goldman Sachs, ING,
> > LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York
> > Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 42 contributors to this
> > release!
> >
> > Adam Bellemare, Andras Katona, Andy Coates, Anna Povzner, A.
> > Sophie Blee-Goldman, Auston, belugabehr, Bill Bejeck, Boyang
> > Chen, Bruno Cadonna, Chia-Ping Tsai, Chris Egerton, David
> > Arthur, David Jacot, Dezhi “Andy” Fang, Dima Reznik, Ego,
> > Evelyn Bayes, Ewen Cheslack-Postava, Greg Harris, Guozhang
> > Wang, Ismael Juma, Jason Gustafson, Jeff Widman, Jeremy
> > Custenborder, jiameixie, John Roesler, Jorge Esteban
> > Quilcate Otoya, Konstantine Karantasis, Lucent-Wong, Mario
> > Molina, Matthias J. Sax, Navinder Pal Singh Brar, Nikolay,
> > Rajini Sivaram, Randall Hauch, Sanjana Kaundinya, showuon,
> > Steve Rodrigues, Tom Bentley, Tu V. Tran, vinoth chandar
> >
> > We welcome your help and feedback. For more information on
> > how to report problems, and to get involved, visit the
> > project website at https://kafka.apache.org/
> >
> > Thank you!
> >
> >
> > Regards,
> >
> > John Roesler
> >
> >
>

[ANNOUNCE] Apache Kafka 2.6.0

[Randall Hauch]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.6.0

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Significant performance improvements, especially when the broker has
large numbers of partitions
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes other features, 74 improvements, 175 bug fixes,
plus other changes.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.6.0/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#2.6.0

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 127 contributors to this release!

17hao, A. Sophie Blee-Goldman, Aakash Shah, Adam Bellemare, Agam Brahma,
Alaa Zbair, Alexandra Rodoni, Andras Katona, Andrew Olson, Andy Coates,
Aneel Nazareth, Anna Povzner, Antony Stubbs, Arjun Satish, Auston, avalsa,
Badai Aqrandista, belugabehr, Bill Bejeck, Bob Barrett, Boyang Chen, Brian
Bushree, Brian Byrne, Bruno Cadonna, Charles Feduke, Chia-Ping Tsai, Chris
Egerton, Colin Patrick McCabe, Daniel, Daniel Beskin, David Arthur, David
Jacot, David Mao, dengziming, Dezhi “Andy” Fang, Dima Reznik, Dominic
Evans, Ego, Eric Bolinger, Evelyn Bayes, Ewen Cheslack-Postava, fantayeneh,
feyman2016, Florian Hussonnois, Gardner Vickers, Greg Harris, Gunnar
Morling, Guozhang Wang, high.lee, Hossein Torabi, huxi, Ismael Juma, Jason
Gustafson, Jeff Huang, jeff kim, Jeff Widman, Jeremy Custenborder, Jiamei
Xie, jiameixie, jiao, Jim Galasyn, Joel Hamill, John Roesler, Jorge Esteban
Quilcate Otoya, José Armando García Sancio, Konstantine Karantasis, Kowshik
Prakasam, Kun Song, Lee Dongjin, Leonard Ge, Lev Zemlyanov, Levani
Kokhreidze, Liam Clarke-Hutchinson, Lucas Bradstreet, Lucent-Wong, Magnus
Edenhill, Manikumar Reddy, Mario Molina, Matthew Wong, Matthias J. Sax,
maulin-vasavada, Michael Viamari, Michal T, Mickael Maison, Mitch, Navina
Ramesh, Navinder Pal Singh Brar, nicolasguyomar, Nigel Liang, Nikolay,
Okada Haruki, Paul, Piotr Fras, Radai Rosenblatt, Rajini Sivaram, Randall
Hauch, Rens Groothuijsen, Richard Yu, Rigel Bezerra de Melo, Rob Meng,
Rohan, Ron Dagostino, Sanjana Kaundinya, Scott, Scott Hendricks, sebwills,
Shailesh Panwar, showuon, SoontaekLim, Stanislav Kozlovski, Steve
Rodrigues, Svend Vanderveken, Sönke Liebau, THREE LEVEL HELMET, Tom
Bentley, Tu V. Tran, Valeria, Vikas Singh, Viktor Somogyi, vinoth chandar,
Vito Jeng, Xavier Léauté, xiaodongdu, Zach Zhang, zhaohaidao, zshuo, 阿洋

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Randall Hauch

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

The last blog post for 2.5.0 mentioned KIP-500, and I know we've made a lot
of progress on that and will have even more in upcoming releases. I'd like
to add the following paragraph to the opening section, just after the
paragraph mentioning JDK 14 and Scala 2.13:

"Finally, these accomplishments are only one part of a larger active
roadmap in the run up to Apache Kafka 3.0, which may be one of the most
significant releases in the project’s history. The work to replace
Zookeeper (link to KIP-500) with built-in RAFT-based consensus is well
underway with eight KIPs in active development. Kafka’s new RAFT protocol
for the metadata quorum is already available for review (link to
https://github.com/apache/kafka/pull/9130). Tiered Storage unlocks infinite
scaling and faster rebalance times via KIP-405 (link to KIP-405), and is up
and running in internal clusters at Uber."

I'ved incorporated this into the preview doc, but feedback is welcome.

Best regards,

Randall

On Wed, Aug 5, 2020 at 10:27 AM Randall Hauch  wrote:

> I've prepared a preliminary blog post about the upcoming Apache Kafka
>  2.6.0 release.
> Please take a look and let me know via this thread if you want to
> add/modify details.
> Thanks to all who contributed to this blog post.
>
> Unfortunately, the preview is not currently publicly visible at
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3 
> (I've
> logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> interest of time I created a Google doc with read privilege for everyone by
> pasting the preview content directly into:
>
>
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
>
> I will continue to update this doc with any changes to the draft blog post.
>
>
> Thanks,
> Randall
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

Hi, Maulin.

We have traditionally not mentioned *every* KIP in the project's blog posts
-- unless a release just happens to involve a small number of KIPs. This
means that we do have to choose a subset of the KIPs -- AK 2.6.0 had 30
KIPs that were implemented, and a blog summarizing each would be too long
and much less widely received. I tried to choose KIPs that clearly impacted
the broadest set of users, and finding a subset wasn't that easy.

I chose not to include KIP-519 in the blog simply because it requires
installing and integration into the broker components that are not included
in the official distribution. I hope that helps explain my thought process.

Best regards,

Randall

On Wed, Aug 5, 2020 at 12:52 PM Maulin Vasavada 
wrote:

> Hi Randall
>
> One question: Do we mention all KIPs/NewFeatures in the blog that are
> listed in the release notes document -
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html ?
>
> I see that [KAFKA-8890 <https://issues.apache.org/jira/browse/KAFKA-8890>]
> - KIP-519: Make SSL context/engine configuration extensible is missing from
> the google doc that you shared.
>
> Thanks
> Maulin
>
>
> On Wed, Aug 5, 2020 at 9:49 AM Randall Hauch  wrote:
>
> > Thanks, Jason. We haven't done that for a few releases, but I think it's
> a
> > great idea. I've updated the blog post draft and the Google doc to
> mention
> > the 127 contributors by name that will also be mentioned in the email
> > release announcement.
> >
> > I also linked to the project's downloads page in the opening sentence,
> and
> > tweaked the wording slightly in the first paragraph.
> >
> > Best regards,
> >
> > Randall
> >
> > On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson 
> > wrote:
> >
> > > Hey Randall,
> > >
> > > Thanks for putting this together. I think it would be great if the blog
> > > included the names of the release contributors. It's an easy way to
> give
> > > some recognition. I know we have done that in the past.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch 
> wrote:
> > >
> > > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > > 2.6.0
> > > > release.
> > > > Please take a look and let me know via this thread if you want to
> > > > add/modify details.
> > > > Thanks to all who contributed to this blog post.
> > > >
> > > > Unfortunately, the preview is not currently publicly visible at
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > > > (I've
> > > > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > > > interest of time I created a Google doc with read privilege for
> > everyone
> > > by
> > > > pasting the preview content directly into:
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> > > >
> > > > I will continue to update this doc with any changes to the draft blog
> > > post.
> > > >
> > > >
> > > > Thanks,
> > > > Randall
> > > >
> > >
> >
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

Thanks, Jason. We haven't done that for a few releases, but I think it's a
great idea. I've updated the blog post draft and the Google doc to mention
the 127 contributors by name that will also be mentioned in the email
release announcement.

I also linked to the project's downloads page in the opening sentence, and
tweaked the wording slightly in the first paragraph.

Best regards,

Randall

On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson  wrote:

> Hey Randall,
>
> Thanks for putting this together. I think it would be great if the blog
> included the names of the release contributors. It's an easy way to give
> some recognition. I know we have done that in the past.
>
> Thanks,
> Jason
>
> On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch  wrote:
>
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.6.0
> > release.
> > Please take a look and let me know via this thread if you want to
> > add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> > Unfortunately, the preview is not currently publicly visible at
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > (I've
> > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > interest of time I created a Google doc with read privilege for everyone
> by
> > pasting the preview content directly into:
> >
> >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> >
> > I will continue to update this doc with any changes to the draft blog
> post.
> >
> >
> > Thanks,
> > Randall
> >
>

Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

I've prepared a preliminary blog post about the upcoming Apache Kafka 2.6.0
release.
Please take a look and let me know via this thread if you want to
add/modify details.
Thanks to all who contributed to this blog post.

Unfortunately, the preview is not currently publicly visible at
https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
(I've
logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
interest of time I created a Google doc with read privilege for everyone by
pasting the preview content directly into:

https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing

I will continue to update this doc with any changes to the draft blog post.


Thanks,
Randall

Re: [VOTE] 2.6.0 RC2

[Randall Hauch]

+1 (non-binding)

I'm closing the vote since this has met the release criteria.

Randall

On Mon, Aug 3, 2020 at 2:57 AM Ismael Juma  wrote:

> +1 (binding), verified signatures, ran the tests on the source archive with
> Scala 2.13 and Java 14 and verified the quickstart with the source archive
> and Scala 2.13 binary archive.
>
> Thanks,
> Ismael
>
> On Tue, Jul 28, 2020, 2:52 PM Randall Hauch  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for release of Apache Kafka 2.6.0. This is a
> > major release that includes many new features, including:
> >
> > * TLSv1.3 has been enabled by default for Java 11 or newer.
> > * Smooth scaling out of Kafka Streams applications
> > * Kafka Streams support for emit on change
> > * New metrics for better operational insight
> > * Kafka Connect can automatically create topics for source connectors
> > * Improved error reporting options for sink connectors in Kafka Connect
> > * New Filter and conditional SMTs in Kafka Connect
> > * The default value for the `client.dns.lookup` configuration is
> > now `use_all_dns_ips`
> > * Upgrade Zookeeper to 3.5.8
> >
> > This release also includes a few other features, 74 improvements, 175 bug
> > fixes, plus other fixes.
> >
> > Release notes for the 2.6.0 release:
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, August 3, 9am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/2.6.0-rc2
> >
> > * Documentation:
> > https://kafka.apache.org/26/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/26/protocol.html
> >
> > * Successful Jenkins builds for the 2.6 branch:
> > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.6-jdk8/101/
> > System tests: (link to follow)
> >
> >
> > Thanks,
> > Randall Hauch
> >
>

Re: [VOTE] 2.6.0 RC2

[Randall Hauch]

We finally have a green system test using the same commit used for RC2:
https://jenkins.confluent.io/job/system-test-kafka/job/2.6/53/

Thanks to Rajini and Gwen for their testing, verification, and binding +1s.
Thanks also to Bill for is testing and non-binding +1.

That means we're looking for at least one more binding +1 vote from PMC
members. Please download, test and vote by Monday, August 3, 9am PT,

Best regards!

Randall

On Fri, Jul 31, 2020 at 11:44 AM Randall Hauch  wrote:

> Thanks, Rajini.
>
> Here's an update on the system tests. Unfortunately we've not yet had a
> fully-green system test run, but each of the system test runs since
> https://jenkins.confluent.io/job/system-test-kafka/job/2.6/49/ has had
> just one or two failures -- and no failure has been repeated. This suggests
> the failing tests appear to be somewhat flaky. I'll keep running more
> system tests and will reply here if something appears suspicious, but
> please holler if you think my analysis is incorrect.
>
> Best regards,
>
> Randall
>
> On Fri, Jul 31, 2020 at 11:00 AM Rajini Sivaram 
> wrote:
>
>> Thanks Randall, +1 (binding)
>>
>> Built from source and ran tests, had a quick look through some Javadoc
>> changes, ran quickstart and some tests with Java 11 TLSv1.3 on the binary.
>>
>> Regards,
>>
>> Rajini
>>
>>
>> On Tue, Jul 28, 2020 at 10:50 PM Randall Hauch  wrote:
>>
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the third candidate for release of Apache Kafka 2.6.0. This is a
>> > major release that includes many new features, including:
>> >
>> > * TLSv1.3 has been enabled by default for Java 11 or newer.
>> > * Smooth scaling out of Kafka Streams applications
>> > * Kafka Streams support for emit on change
>> > * New metrics for better operational insight
>> > * Kafka Connect can automatically create topics for source connectors
>> > * Improved error reporting options for sink connectors in Kafka Connect
>> > * New Filter and conditional SMTs in Kafka Connect
>> > * The default value for the `client.dns.lookup` configuration is
>> > now `use_all_dns_ips`
>> > * Upgrade Zookeeper to 3.5.8
>> >
>> > This release also includes a few other features, 74 improvements, 175
>> bug
>> > fixes, plus other fixes.
>> >
>> > Release notes for the 2.6.0 release:
>> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html
>> >
>> > *** Please download, test and vote by Monday, August 3, 9am PT
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > https://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> >
>> > * Javadoc:
>> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/
>> >
>> > * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
>> > https://github.com/apache/kafka/releases/tag/2.6.0-rc2
>> >
>> > * Documentation:
>> > https://kafka.apache.org/26/documentation.html
>> >
>> > * Protocol:
>> > https://kafka.apache.org/26/protocol.html
>> >
>> > * Successful Jenkins builds for the 2.6 branch:
>> > Unit/integration tests:
>> https://builds.apache.org/job/kafka-2.6-jdk8/101/
>> > System tests: (link to follow)
>> >
>> >
>> > Thanks,
>> > Randall Hauch
>> >
>>
>

Re: [VOTE] 2.6.0 RC2

[Randall Hauch]

Thanks, Gwen.

I'll log an issue to remove the kafka_2.12-2.6.0-site-docs.* files, and
look into whether AgentTest.testAgentGetStatus is flaky and if so log an
issue.

Randall

On Sat, Aug 1, 2020 at 12:16 AM Gwen Shapira  wrote:

> Thank you, Randall for driving this release.
>
> +1 (binding) after verifying signatures and hashes, building from sources,
> running unit/integration tests and some manual tests with the 2.13 build.
>
> Two minor things:
> 1. There were two sitedoc files - 2.12 and 2.13, we don't really need two
> sitedocs generated. Not a big deal, but maybe worth tracking and fixing.
> 2. I got one test failure locally:
>
> org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus failed, log
> available in
>
> /Users/gwenshap/releases/2.6.0-rc2/kafka-2.6.0-src/tools/build/reports/testOutput/org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus.test.stdout
>
> org.apache.kafka.trogdor.agent.AgentTest > testAgentGetStatus FAILED
> java.lang.RuntimeException:
> at
>
> org.apache.kafka.trogdor.rest.RestExceptionMapper.toException(RestExceptionMapper.java:69)
> at
>
> org.apache.kafka.trogdor.rest.JsonRestServer$HttpResponse.body(JsonRestServer.java:285)
> at
> org.apache.kafka.trogdor.agent.AgentClient.status(AgentClient.java:130)
> at
>
> org.apache.kafka.trogdor.agent.AgentTest.testAgentGetStatus(AgentTest.java:115)
>
> Gwen
>
> On Tue, Jul 28, 2020 at 2:50 PM Randall Hauch  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for release of Apache Kafka 2.6.0. This is a
> > major release that includes many new features, including:
> >
> > * TLSv1.3 has been enabled by default for Java 11 or newer.
> > * Smooth scaling out of Kafka Streams applications
> > * Kafka Streams support for emit on change
> > * New metrics for better operational insight
> > * Kafka Connect can automatically create topics for source connectors
> > * Improved error reporting options for sink connectors in Kafka Connect
> > * New Filter and conditional SMTs in Kafka Connect
> > * The default value for the `client.dns.lookup` configuration is
> > now `use_all_dns_ips`
> > * Upgrade Zookeeper to 3.5.8
> >
> > This release also includes a few other features, 74 improvements, 175 bug
> > fixes, plus other fixes.
> >
> > Release notes for the 2.6.0 release:
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, August 3, 9am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/2.6.0-rc2
> >
> > * Documentation:
> > https://kafka.apache.org/26/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/26/protocol.html
> >
> > * Successful Jenkins builds for the 2.6 branch:
> > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.6-jdk8/101/
> > System tests: (link to follow)
> >
> >
> > Thanks,
> > Randall Hauch
> >
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [VOTE] 2.6.0 RC2

[Randall Hauch]

Thanks, Rajini.

Here's an update on the system tests. Unfortunately we've not yet had a
fully-green system test run, but each of the system test runs since
https://jenkins.confluent.io/job/system-test-kafka/job/2.6/49/ has had just
one or two failures -- and no failure has been repeated. This suggests the
failing tests appear to be somewhat flaky. I'll keep running more system
tests and will reply here if something appears suspicious, but please
holler if you think my analysis is incorrect.

Best regards,

Randall

On Fri, Jul 31, 2020 at 11:00 AM Rajini Sivaram 
wrote:

> Thanks Randall, +1 (binding)
>
> Built from source and ran tests, had a quick look through some Javadoc
> changes, ran quickstart and some tests with Java 11 TLSv1.3 on the binary.
>
> Regards,
>
> Rajini
>
>
> On Tue, Jul 28, 2020 at 10:50 PM Randall Hauch  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for release of Apache Kafka 2.6.0. This is a
> > major release that includes many new features, including:
> >
> > * TLSv1.3 has been enabled by default for Java 11 or newer.
> > * Smooth scaling out of Kafka Streams applications
> > * Kafka Streams support for emit on change
> > * New metrics for better operational insight
> > * Kafka Connect can automatically create topics for source connectors
> > * Improved error reporting options for sink connectors in Kafka Connect
> > * New Filter and conditional SMTs in Kafka Connect
> > * The default value for the `client.dns.lookup` configuration is
> > now `use_all_dns_ips`
> > * Upgrade Zookeeper to 3.5.8
> >
> > This release also includes a few other features, 74 improvements, 175 bug
> > fixes, plus other fixes.
> >
> > Release notes for the 2.6.0 release:
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, August 3, 9am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/2.6.0-rc2
> >
> > * Documentation:
> > https://kafka.apache.org/26/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/26/protocol.html
> >
> > * Successful Jenkins builds for the 2.6 branch:
> > Unit/integration tests:
> https://builds.apache.org/job/kafka-2.6-jdk8/101/
> > System tests: (link to follow)
> >
> >
> > Thanks,
> > Randall Hauch
> >
>

[VOTE] 2.6.0 RC2

[Randall Hauch]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 2.6.0. This is a
major release that includes many new features, including:

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes a few other features, 74 improvements, 175 bug
fixes, plus other fixes.

Release notes for the 2.6.0 release:
https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Monday, August 3, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~rhauch/kafka-2.6.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~rhauch/kafka-2.6.0-rc2/javadoc/

* Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
https://github.com/apache/kafka/releases/tag/2.6.0-rc2

* Documentation:
https://kafka.apache.org/26/documentation.html

* Protocol:
https://kafka.apache.org/26/protocol.html

* Successful Jenkins builds for the 2.6 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/101/
System tests: (link to follow)


Thanks,
Randall Hauch

Re: [VOTE] 2.6.0 RC1

[Randall Hauch]

Any thoughts, Rajini?

On Mon, Jul 20, 2020 at 9:55 PM Randall Hauch  wrote:

>
> When I was checking the documentation for RC1 after the tag was pushed, I
> noticed that the fix Rajini mentioned in the RC0 vote thread (
> https://github.com/apache/kafka/pull/8979
> <https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21>)
> and merged to the `2.6` branch includes the following comment about being
> deprecated in 2.7:
> https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21
> .
>
> Rajini, can you please check the commits merged to the `2.6` do not have
> the reference to 2.7? Since these are JavaDocs, I'm assuming that we'll
> need to cut RC2.
>
> But it'd be good for everyone else to double check this release.
>
> Best regards,
>
> Randall Hauch
>
> On Mon, Jul 20, 2020 at 9:50 PM Randall Hauch  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate for release of Apache Kafka 2.6.0. This is a
>> major release that includes many new features, including:
>>
>> * TLSv1.3 has been enabled by default for Java 11 or newer.
>> * Smooth scaling out of Kafka Streams applications
>> * Kafka Streams support for emit on change
>> * New metrics for better operational insight
>> * Kafka Connect can automatically create topics for source connectors
>> * Improved error reporting options for sink connectors in Kafka Connect
>> * New Filter and conditional SMTs in Kafka Connect
>> * The default value for the `client.dns.lookup` configuration is
>> now `use_all_dns_ips`
>> * Upgrade Zookeeper to 3.5.8
>>
>> This release also includes a few other features, 76 improvements, and 165
>> bug fixes.
>>
>> Release notes for the 2.6.0 release:
>> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, July 20, 9am PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> https://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/javadoc/
>>
>> * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
>> https://github.com/apache/kafka/releases/tag/2.6.0-rc1
>>
>> * Documentation:
>> https://kafka.apache.org/26/documentation.html
>>
>> * Protocol:
>> https://kafka.apache.org/26/protocol.html
>>
>> * Successful Jenkins builds for the 2.6 branch:
>> Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/91/ (one
>> flaky test)
>> System tests: (link to follow)
>>
>> Thanks,
>> Randall Hauch
>>
>

[VOTE] 2.6.0 RC1

[Randall Hauch]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 2.6.0. This is a
major release that includes many new features, including:

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes a few other features, 76 improvements, and 165
bug fixes.

Release notes for the 2.6.0 release:
https://home.apache.org/~rhauch/kafka-2.6.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Monday, July 20, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~rhauch/kafka-2.6.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~rhauch/kafka-2.6.0-rc1/javadoc/

* Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
https://github.com/apache/kafka/releases/tag/2.6.0-rc1

* Documentation:
https://kafka.apache.org/26/documentation.html

* Protocol:
https://kafka.apache.org/26/protocol.html

* Successful Jenkins builds for the 2.6 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/91/ (one
flaky test)
System tests: (link to follow)

Thanks,
Randall Hauch

Re: [VOTE] 2.6.0 RC1

[Randall Hauch]

When I was checking the documentation for RC1 after the tag was pushed, I
noticed that the fix Rajini mentioned in the RC0 vote thread (
https://github.com/apache/kafka/pull/8979
<https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21>)
and merged to the `2.6` branch includes the following comment about being
deprecated in 2.7:
https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21
.

Rajini, can you please check the commits merged to the `2.6` do not have
the reference to 2.7? Since these are JavaDocs, I'm assuming that we'll
need to cut RC2.

But it'd be good for everyone else to double check this release.

Best regards,

Randall Hauch

On Mon, Jul 20, 2020 at 9:50 PM Randall Hauch  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 2.6.0. This is a
> major release that includes many new features, including:
>
> * TLSv1.3 has been enabled by default for Java 11 or newer.
> * Smooth scaling out of Kafka Streams applications
> * Kafka Streams support for emit on change
> * New metrics for better operational insight
> * Kafka Connect can automatically create topics for source connectors
> * Improved error reporting options for sink connectors in Kafka Connect
> * New Filter and conditional SMTs in Kafka Connect
> * The default value for the `client.dns.lookup` configuration is
> now `use_all_dns_ips`
> * Upgrade Zookeeper to 3.5.8
>
> This release also includes a few other features, 76 improvements, and 165
> bug fixes.
>
> Release notes for the 2.6.0 release:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, July 20, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/javadoc/
>
> * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> https://github.com/apache/kafka/releases/tag/2.6.0-rc1
>
> * Documentation:
> https://kafka.apache.org/26/documentation.html
>
> * Protocol:
> https://kafka.apache.org/26/protocol.html
>
> * Successful Jenkins builds for the 2.6 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/91/ (one
> flaky test)
> System tests: (link to follow)
>
> Thanks,
> Randall Hauch
>

[VOTE] 2.6.0 RC0

[Randall Hauch]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 2.6.0. This is a
major release that includes many new features, including:

* TLSv1.3 has been enabled by default for Java 11 or newer.
* Smooth scaling out of Kafka Streams applications
* Kafka Streams support for emit on change
* New metrics for better operational insight
* Kafka Connect can automatically create topics for source connectors
* Improved error reporting options for sink connectors in Kafka Connect
* New Filter and conditional SMTs in Kafka Connect
* The default value for the `client.dns.lookup` configuration is
now `use_all_dns_ips`
* Upgrade Zookeeper to 3.5.8

This release also includes a few other features, 76 improvements, and 165
bug fixes.

Release notes for the 2.6.0 release:
https://home.apache.org/~rhauch/kafka-2.6.0-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Monday, July 20, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~rhauch/kafka-2.6.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~rhauch/kafka-2.6.0-rc0/javadoc/

* Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
https://github.com/apache/kafka/releases/tag/2.6.0-rc0

* Documentation:
https://kafka.apache.org/26/documentation.html

* Protocol:
https://kafka.apache.org/26/protocol.html

* Successful Jenkins builds for the 2.6 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.6-jdk8/80/
System tests: (link to follow)

Thanks,
Randall Hauch

CVE-2019-12399: Apache Kafka Connect REST API may expose plaintext secrets in tasks endpoint

[Randall Hauch]

CVE-2019-12399: Apache Kafka Connect REST API may expose plaintext secrets
in tasks endpoint

Severity: Medium

Vendor: The Apache Software Foundation

Versions Affected:

Apache Kafka 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.3.0

Description:

When Connect workers in Apache Kafka 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0,
2.2.1, or 2.3.0 are configured with one or more config providers, and a
connector is created/updated on that Connect cluster to use an externalized
secret variable in a substring of a connector configuration property value
(the externalized secret variable is not the whole configuration property
value), then any client can issue a request to the same Connect cluster to
obtain the connector's task configurations and the response will contain
the plaintext secret rather than the externalized secrets variable.


Mitigation:

Apache Kafka Connect users should upgrade to one of the following versions
where this vulnerability has been fixed:
- 2.0.2 or higher
- 2.1.2 or higher
- 2.2.2 or higher
- 2.3.1 or higher

Acknowledgements:

This issue was first reported by Oleksandr Diachenko.


Regards,

Randall

Re: [VOTE] 2.4.0 RC2

[Randall Hauch]

Ah, I merged https://issues.apache.org/jira/browse/KAFKA-9258 last night
and forgot to send an email to this thread.

Thanks!

Randall

On Wed, Dec 4, 2019 at 10:42 AM Eric Lalonde  wrote:

> > I have also shared a pull request for the changes I have made to the test
> > script
> >
> > https://github.com/elalonde/kafka/pull/2
>
>
> Thanks, this has been merged.
>

Re: [ANNOUNCE] Apache Kafka 2.2.2

[Randall Hauch]

As mentioned above, the issues fixed in 2.2.2 are listed in the
https://www.apache.org/dist/kafka/2.2.2/RELEASE_NOTES.html.

On Tue, Dec 3, 2019 at 7:20 AM Sachin Mittal  wrote:

> Does this release has fix for that critical windows bug of not able to
> delete topics?
> If not then under which release we can expect the same.
>
> Thanks and Regards
> Sachin
>
>
> On Mon, Dec 2, 2019 at 5:55 PM Karolis Pocius
>  wrote:
>
> > 2.2.2 is a bugfix release, it contains some of the fixes from 2.3.0/1,
> but
> > does not backport new features. You can go to
> > https://kafka.apache.org/downloads and check the release notes of
> 2.3.0/1
> > to see what new features and improvements you'd be missing if you didn't
> > upgrade directly to those versions. As to whether it will make any
> > difference if you upgrade to 2.2.2 now and another version later -- it
> > shouldn't really matter.
> >
> > On Mon, Dec 2, 2019 at 2:15 PM Vinay Kumar  wrote:
> >
> > > Hi,
> > > May I know how would this version be different from the later versions
> > > 2.3.0, 2.3.1.
> > > I'm looking to upgrade my current Kafka 2.1.0, so can you please let me
> > > know what differences it would make for upgrading to 2.2.2 or 2.3.1?
> > >
> > > Thanks,
> > > Vinay
> > >
> > > On Monday, December 2, 2019, Vahid Hashemian <
> vahid.hashem...@gmail.com>
> > > wrote:
> > >
> > > > Awesome. Thanks for managing this release Randall!
> > > >
> > > > Regards,
> > > > --Vahid
> > > >
> > > > On Sun, Dec 1, 2019 at 5:45 PM Randall Hauch 
> > wrote:
> > > >
> > > > > The Apache Kafka community is pleased to announce the release for
> > > Apache
> > > > > Kafka 2.2.2
> > > > >
> > > > > This is a bugfix release for Apache Kafka 2.2.
> > > > > All of the changes in this release can be found in the release
> notes:
> > > > > https://www.apache.org/dist/kafka/2.2.2/RELEASE_NOTES.html
> > > > >
> > > > > You can download the source and binary release from:
> > > > > https://kafka.apache.org/downloads#2.2.2
> > > > >
> > > > >
> > > > > 
> > > > ---
> > > > >
> > > > >
> > > > > Apache Kafka is a distributed streaming platform with four core
> APIs:
> > > > >
> > > > >
> > > > > ** The Producer API allows an application to publish a stream
> records
> > > to
> > > > > one or more Kafka topics.
> > > > >
> > > > > ** The Consumer API allows an application to subscribe to one or
> more
> > > > > topics and process the stream of records produced to them.
> > > > >
> > > > > ** The Streams API allows an application to act as a stream
> > processor,
> > > > > consuming an input stream from one or more topics and producing an
> > > > > output stream to one or more output topics, effectively
> transforming
> > > the
> > > > > input streams to output streams.
> > > > >
> > > > > ** The Connector API allows building and running reusable producers
> > or
> > > > > consumers that connect Kafka topics to existing applications or
> data
> > > > > systems. For example, a connector to a relational database might
> > > > > capture every change to a table.
> > > > >
> > > > >
> > > > > With these APIs, Kafka can be used for two broad classes of
> > > application:
> > > > >
> > > > > ** Building real-time streaming data pipelines that reliably get
> data
> > > > > between systems or applications.
> > > > >
> > > > > ** Building real-time streaming applications that transform or
> react
> > > > > to the streams of data.
> > > > >
> > > > >
> > > > > Apache Kafka is in use at large and small companies worldwide,
> > > including
> > > > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> > > Rabobank,
> > > > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > > > >
> > > > > A big thank you for the following 41 contributors to this release!

[ANNOUNCE] Apache Kafka 2.2.2

[Randall Hauch]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.2.2

This is a bugfix release for Apache Kafka 2.2.
All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.2.2/RELEASE_NOTES.html

You can download the source and binary release from:
https://kafka.apache.org/downloads#2.2.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 41 contributors to this release!

A. Sophie Blee-Goldman, Matthias J. Sax, Bill Bejeck, Jason Gustafson,
Chris Egerton, Boyang Chen, Alex Diachenko, cpettitt-confluent, Magesh
Nandakumar, Randall Hauch, Ismael Juma, John Roesler, Konstantine
Karantasis, Mickael Maison, Nacho Muñoz Gómez, Nigel Liang, Paul, Rajini
Sivaram, Robert Yokota, Stanislav Kozlovski, Vahid Hashemian, Victoria
Bialas, cadonna, cwildman, mjarvie, sdreynolds, slim, vinoth chandar,
wenhoujx, Arjun Satish, Chia-Ping Tsai, Colin P. Mccabe, David Arthur,
Dhruvil Shah, Greg Harris, Gunnar Morling, Hai-Dang Dam, Lifei Chen, Lucas
Bradstreet, Manikumar Reddy, Michał Borowiecki

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,
Randall Hauch

Re: Subject: [VOTE] 2.2.2 RC2

[Randall Hauch]

Thanks, Mickael!

Anyone else get a chance to validate the 2.2.2 RC2 build? It'd be great to
get this out the door.

Randall

On Tue, Nov 5, 2019 at 6:34 AM Mickael Maison 
wrote:

> +1 (non binding)
> I verified signatures, built it from source, ran unit tests and quickstart
>
>
>
> On Fri, Oct 25, 2019 at 3:10 PM Randall Hauch  wrote:
> >
> > Hello all, we identified around three dozen bug fixes, including an
> update
> > of a third party dependency, and wanted to release a patch release for
> the
> > Apache Kafka 2.2.0 release.
> >
> > This is the *second* candidate for release of Apache Kafka 2.2.2. (RC1
> did
> > not include a fix for https://issues.apache.org/jira/browse/KAFKA-9053,
> but
> > the fix appeared before RC1 was announced so it was easier to just create
> > RC2.)
> >
> > Check out the release notes for a complete list of the changes in this
> > release candidate:
> > https://home.apache.org/~rhauch/kafka-2.2.2-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Wednesday, October 30, 9am PT>
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~rhauch/kafka-2.2.2-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~rhauch/kafka-2.2.2-rc2/javadoc/
> >
> > * Tag to be voted upon (off 2.2 branch) is the 2.2.2 tag:
> > https://github.com/apache/kafka/releases/tag/2.2.2-rc2
> >
> > * Documentation:
> > https://kafka.apache.org/22/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/22/protocol.html
> >
> > * Successful Jenkins builds for the 2.2 branch:
> > Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/1/
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/2.2/216/
> >
> > /**
> >
> > Thanks,
> >
> > Randall Hauch
>

Subject: [VOTE] 2.2.2 RC2

[Randall Hauch]

Hello all, we identified around three dozen bug fixes, including an update
of a third party dependency, and wanted to release a patch release for the
Apache Kafka 2.2.0 release.

This is the *second* candidate for release of Apache Kafka 2.2.2. (RC1 did
not include a fix for https://issues.apache.org/jira/browse/KAFKA-9053, but
the fix appeared before RC1 was announced so it was easier to just create
RC2.)

Check out the release notes for a complete list of the changes in this
release candidate:
https://home.apache.org/~rhauch/kafka-2.2.2-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Wednesday, October 30, 9am PT>

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~rhauch/kafka-2.2.2-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~rhauch/kafka-2.2.2-rc2/javadoc/

* Tag to be voted upon (off 2.2 branch) is the 2.2.2 tag:
https://github.com/apache/kafka/releases/tag/2.2.2-rc2

* Documentation:
https://kafka.apache.org/22/documentation.html

* Protocol:
https://kafka.apache.org/22/protocol.html

* Successful Jenkins builds for the 2.2 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/1/
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/2.2/216/

/**

Thanks,

Randall Hauch

Re: [ANNOUNCE] New Committer: Randall Hauch

[Randall Hauch]

Thanks, everyone!

On Sun, Feb 17, 2019 at 8:57 PM Becket Qin  wrote:

> Congratulations, Randall!
>
> On Sat, Feb 16, 2019 at 2:44 AM Matthias J. Sax 
> wrote:
>
> > Congrats Randall!
> >
> >
> > -Matthias
> >
> > On 2/14/19 6:16 PM, Guozhang Wang wrote:
> > > Hello all,
> > >
> > > The PMC of Apache Kafka is happy to announce another new committer
> > joining
> > > the project today: we have invited Randall Hauch as a project committer
> > and
> > > he has accepted.
> > >
> > > Randall has been participating in the Kafka community for the past 3
> > years,
> > > and is well known as the founder of the Debezium project, a popular
> > project
> > > for database change-capture streams using Kafka (https://debezium.io).
> > More
> > > recently he has become the main person keeping Kafka Connect moving
> > > forward, participated in nearly all KIP discussions and QAs on the
> > mailing
> > > list. He's authored 6 KIPs and authored 50 pull requests and conducted
> > over
> > > a hundred reviews around Kafka Connect, and has also been evangelizing
> > > Kafka Connect at several Kafka Summit venues.
> > >
> > >
> > > Thank you very much for your contributions to the Connect community
> > Randall
> > > ! And looking forward to many more :)
> > >
> > >
> > > Guozhang, on behalf of the Apache Kafka PMC
> > >
> >
> >
>

Re: Kafka CDC from Databases to S3

[Randall Hauch]

Kafka Connect was designed explicitly to support this kind of use case.
Debezium's connectors implement the Kafka Connect API, and they just
released a MS SQL Server source connector that captures the changes from
SQL Server and writes them to Kafka. Debezium's MySQL source connector may
work with MariaDB, but it's probably best to check with their community ()
about that.

There are other sink connectors for reading Kafka topics and writing to S3,
including Confluent's S3 sink connector (
https://docs.confluent.io/current/connect/kafka-connect-s3/index.html#connect-s3
).

Of course the connector's I've mentioned may not be the only options for
the systems you mentioned, and there are lots of source and sink connectors
for other systems.

Best regards,

Randall

On Thu, Feb 14, 2019 at 12:03 PM John Ziebro  wrote:

> Hi,
>
> We are planning on implementing Kafka to stream changes from a MariaDB as
> well as a MS SQL Server database into an S3 bucket and would like to stay
> with opensource solutions as much as possible. To simplify this process it
> looks like we might need to use something that leverages Kafka like
> Debezium.
>
> What are your suggestions for handling this scenario?
>
> Thank you,
> John Z
>
>

Re: Kafka Connect restarts while creating nw

[Randall Hauch]

This is expected behavior, though there is already an enhancement request
to improve it: https://issues.apache.org/jira/browse/KAFKA-5505

Best regards,

Randall

On Fri, Jun 1, 2018 at 2:26 AM, Chintan Patel 
wrote:

> Hello,
>
> I'm using Kafka connect in distributed mode. When I create new connector
> through Rest API, It restarts all the connectors.
>
> I don't know It's normal behavior or I have to configure something so It
> will not restarts all the connectors when I create new connector.
>
> Thanks.
> --
> CHINTAN PATEL
> *SR. SOFTWARE DEVELOPER*
>
> Calle 99 # 7a - 77
> Oficina 201 Ed. Advance
> (57-1) 605-88-80
>

Re: Artemis Source connector into apache kafka.

[Randall Hauch]

Connect is not only for sources that support reading from a specific point.
But unless the source system is tracking state for you, you connector's
likely to miss information in that source system if the connector stops.
This is probably okay in quite a few systems, so its good if you're fine
with that.

On Wed, May 30, 2018 at 6:57 PM, meow licous 
wrote:

> Hi,
>
>
> I am trying to write a source connector from Apache Artemis into Kafka.
> Here `Source Partition` doesn't make much sense since you can't keep track
> of position in Artemis. You either `Ack` or `Nack` a message.
>
> is connect only for sources that support reading from a specific point like
> a file position, dbtimestamp ect?
>
> Is is possible to copy these messages into kafka in a lossless manner.
>
> - Meow
>

Re: JSONSchema Kafka Connect Converter

[Randall Hauch]

Hi, Andrew.

The Converter is part of Connect's public API, so it certainly is
valid/encouraged to create new implementations when that makes sense for
users. I know of several Converter implementations that are outside of the
Apache Kafka project. The project's existing JSON converter is fairly
limited w/r/t schemas, and when configured with the default settings it
serializes Connect's own schema rather than another schema language (in
part because there is no truly standard schema language for JSON, and in
part because that's all that's necessary). This is convenient but also
results in very verbose messages, since the schema is included in every
message. Of course, you can disable the JSON converter's use of schemas
altogether, at which point the converter simply serializes/deserializes
simple JSON documents, arrays, and literals.

So it certainly makes sense to implement a Converter when you want or need
to use other representations. The whole raison d'être that Connect
defines the Converter interface is precisely because the Converter -- and
not the connectors -- is solely responsibility for serialization and
deserialization. Simply plug in a new Converter, and you can still use any
and all of the connectors that are properly implemented. Just be aware that
you may also want to implement Kafka's Serializer and Deserializer
interfaces that most JVM clients require. A common approach is to implement
those, and then to have the Converter implementation simply reuse those.

When implementing a Converter that deals with schemas, consider the
following questions. The answers will likely affect whether or how easily
others can reuse that Converter in their own environments.

1) Where will the schema be persisted? Within every record? Externally and
somehow only referenced from each message? Or only used to validate before
serialization? The JSON converter uses the former approach, but that has
significant disadvantages, including message size and the performance
degradation due to repeatedly serializing and deserializing the same
schema. Confluent's Schema Registry (which as you point out currently
supports only Avro schemas) takes the second approach by storing the schema
in a centralized service, and the AvroConverter then includes only small
message identifier within every message rather than the whole schema,
allowing the deserializer to know exactly which schema was used during
serialization. (This takes advantage of Avro's ability to deserialize using
a different but compatible schema than the serializer.)
2) How will the converter handle schema evolution? Will it constrain how
producers and consumers (whether or not those are connectors) can be
upgraded?
3) How will the converter handle schema variation? Is it possible to use
the converter on message streams that are a mixture of a finite number of
different schemas?

Finally, it's difficult to speculate whether the Apache Kafka project would
be interested in having a particular Converter, especially one that doesn't
exist. The project has a few general purpose ones, but note that the
project has also expressly avoided "owning" connector implementations
(other than a few example connectors). It may be that the project does feel
the Converter you describe is needed and would welcome any such
contribution. But even if that weren't the case and project decided it did
not want to "own" any more Converters, there probably is a community of
users out there that are willing to collaborate with you. For example, as
you mention people have suggested something similar in the Confluent Schema
Registry community, and you might find that a welcoming place to do this
work while also benefiting from the existing capabilities.

I hope this has been useful.

Randall


On Tue, Jan 23, 2018 at 1:17 PM, Andrew Otto  wrote:

> Hi all,
>
> I’ve been thinking a lot recently about JSON and Kafka.  Because JSON is
> not strongly typed, it isn’t treated as a first class citizen of the Kafka
> ecosystem.  At Wikimedia, we use JSONSchema validated JSON
>  for Kafka
> messages.  This makes it so easy for our many disparate teams and services
> to consume data from Kafka, without having to consult a remote schema
> registry to read data.  (Yes we have to worry about schema evolution, but
> we do this on the producer side by requiring that the only schema change
> allowed is adding optional fields.)
>
> There’s been discussion
>  about
> JSONSchema support in Confluent’s Schema registry, or perhaps even support
> to produce validated Avro JSON (not binary) from Kafka REST proxy.
>
> However, the more I think about this, I realize that I don’t really care
> about JSON support in Confluent products.  What I (and I betcha most of the
> folks who commented on the issue
> ) really want
> is 

Re: Custom converter with Kafka Connect ?

[Randall Hauch]

> On Oct 18, 2017, at 7:05 AM, Jehan Bruggeman <jehanbrugge...@gmail.com> wrote:
> 
> Hi all,
> 
> quick follow-up: thank you Philip, you were right ! Indeed, I got rid of
> this error by putting the converter in the connector's folder.
> 
> I also tried something else: create a custom connector and use that custom
> connector with the custom converter, both loaded as plugins. It also works.
> 
> Summary: converters are loaded by the connector. If your connector is a
> plugin, your converter should be as well. If you connector is not a plugin
> (bundled with your kafka connect distrib), your converter should not be
> either.

Yes, this strange behavior is due to the bug 
https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-6007. Once that is 
fixed, then this will not be a constraint. 

> 
> Jehan
> 
> On 17 October 2017 at 14:51, Jehan Bruggeman <jehanbrugge...@gmail.com>
> wrote:
> 
>> HI Philip,
>> 
>> thanks for the reply !
>> 
>> This issue looks like it might match my problem indeed. I'll look into it.
>> 
>> Jehan
>> 
>> On 16 October 2017 at 20:53, Philip Schmitt <philip.schm...@outlook.com>
>> wrote:
>> 
>>> Hi Jehan,
>>> 
>>> I've run into the same issue last week and also got a "class could not be
>>> found" error.
>>> 
>>> Konstantine Karantasis helpfully pointed me towards
>>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-6007
>>> 
>>> To test this, I simply copied my SMT jar to the folder of the connector I
>>> was using and adjusted the plugin.path property.
>>> 
>>> I haven't fully tested tested it, but in my very quick initial test, it
>>> seemed to have move past that "class could not be found" issue.
>>> 
>>> Regards,
>>> Philip
>>> 
>>> From: jeh...@gmail.com <jeh...@gmail.com> on behalf of Jehan Bruggeman <
>>> jehanbrugge...@gmail.com>
>>> Sent: Monday, October 16, 2017 8:17 AM
>>> To: users@kafka.apache.org
>>> Subject: Re: Custom converter with Kafka Connect ?
>>> 
>>> Hi Randall,
>>> 
>>> thanks for your reply. I'm not sure about this; what detail could I add
>>> that would help you figure it out ?
>>> 
>>> Concerning the classpath: as described in my original email, I'm pretty
>>> sure the jars are correctly added to the classpath since the classes in
>>> the
>>> jar are recognized by Kafka Connect when it starts (they are mentionned in
>>> the logs, at least).
>>> 
>>> (It's easier to read here, where I asked the same question:
>>> https://stackoverflow.com/questions/46712095/using-a-custom-
>>> converter-with-kafka-connect
>>> )
>>> 
>>> thanks for your help !
>>> 
>>> Jehan
>>> 
>>>> On 13 October 2017 at 16:07, Randall Hauch <rha...@gmail.com> wrote:
>>>> 
>>>> On Tue, Oct 10, 2017 at 8:31 AM, Jehan Bruggeman <
>>> jehanbrugge...@gmail.com
>>>>> 
>>>> wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> I'm trying to use a custom converter with Kafka Connect and I cannot
>>> seem
>>>>> to get it right. I'm hoping someone has experience with this and could
>>>> help
>>>>> me figure it out !
>>>>> 
>>>>> 
>>>>> Initial situation
>>>>> 
>>>>> 
>>>>> - my custom converter's class path is 'custom.CustomStringConverter'.
>>>>> 
>>>>> - to avoid any mistakes, my custom converter is currently just a
>>>> copy/paste
>>>>> of the pre-existing StringConverter (of course, this will change when
>>>> I'll
>>>>> get it to work).
>>>>> https://github.com/apache/kafka/blob/trunk/connect/api/
>>>>> src/main/java/org/apache/kafka/connect/storage/StringConverter.java
>>>>> 
>>>>> - I have a kafka connect cluster of 3 nodes, The nodes are running
>>>>> confluent's official docker images ( confluentinc/cp-kafka-connect:
>>> 3.3.0
>>>>> ).
>>>>> 
>>>>> - Each node is configured to load a jar with my converter in it
>>> (using a
>>>>> docker volume).
>>>>> 
>>>> 
>>>> Can you explain this in more detail? Make sure that you add the JAR to
>>>

Re: Configuration of Kafka Connect REST interface

[Randall Hauch]

On Wed, Oct 4, 2017 at 9:44 AM, Jakub Scholz  wrote:

> Hi,
>
> I have some questions about the configuration of the Kafka Connect REST
> interface.
>
> 1) What are the configuration options rest.advertised.host.name
> and rest.advertised.port useful for? The documentation says "If this is
> set, this is the hostname that will be given out to other workers to
> connect to." - does this mean that the workers actually talk with each
> other through the REST interface? I thought that they coordinate through
> Kafka brokers and the REST is only for users.
>

Yes, the workers do talk to each other. One of the workers will be
designated as the leader, and the other(s) as followers. When a REST
request is sent to a follower, the follower will forward the request to the
leader to ensure that the response has correct/consistent information.


>
> 2) Is there some way how to secure the REST interface? I would be thinking
> SSL encryption and possibly also SSL client authentication. In the docu,
> the SSL related configuration parameters seem to be there only to connect
> to Kafka brokers, but not for the REST interface. Unless there is something
> hidden (and undocumented), I would probably raise a KIP because it seem to
> me as valuable feature.
>

I see you found https://issues.apache.org/jira/browse/KAFKA-4029, but I
mention this in case others are looking for it.

>
> Thanks & Regards
> Jakub
>

Re: Custom converter with Kafka Connect ?

[Randall Hauch]

On Tue, Oct 10, 2017 at 8:31 AM, Jehan Bruggeman 
wrote:

> Hello,
>
> I'm trying to use a custom converter with Kafka Connect and I cannot seem
> to get it right. I'm hoping someone has experience with this and could help
> me figure it out !
>
>
> Initial situation
> 
>
> - my custom converter's class path is 'custom.CustomStringConverter'.
>
> - to avoid any mistakes, my custom converter is currently just a copy/paste
> of the pre-existing StringConverter (of course, this will change when I'll
> get it to work).
> https://github.com/apache/kafka/blob/trunk/connect/api/
> src/main/java/org/apache/kafka/connect/storage/StringConverter.java
>
> - I have a kafka connect cluster of 3 nodes, The nodes are running
> confluent's official docker images ( confluentinc/cp-kafka-connect:3.3.0
> ).
>
> - Each node is configured to load a jar with my converter in it (using a
> docker volume).
>

Can you explain this in more detail? Make sure that you add the JAR to the
classpath.


>
>
>
> What happens ?
> 
>
> When the connectors start, they correctly load the jars and find the custom
> converter. Indeed, this is what I see in the logs :
>
> [2017-10-10 13:06:46,274] INFO Registered loader:
> PluginClassLoader{pluginLocation=file:/opt/custom-connectors/custom-
> converter-1.0-SNAPSHOT.jar}
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:199)
> [2017-10-10 13:06:46,274] INFO Added plugin 'custom.CustomStringConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:132)
> [...]
> [2017-10-10 13:07:43,454] INFO Added aliases 'CustomStringConverter' and
> 'CustomString' to plugin 'custom.CustomStringConverter'
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:293)
>
> I then POST a JSON config to one of the connector nodes to create my
> connector :
>
> {
>   "name": "hdfsSinkCustom",
>   "config": {
> "topics": "yellow",
> "tasks.max": "1",
> "key.converter": "org.apache.kafka.connect.storage.StringConverter",
> "value.converter": "custom.CustomStringConverter",
> "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
> "hdfs.url": "hdfs://hdfs-namenode:8020/hdfs-sink",
> "topics.dir": "yellow_storage",
> "flush.size": "1",
> "rotate.interval.ms": "1000"
>   }
> }
>
> And receive the following reply :
>
> {
>"error_code": 400,
>"message": "Connector configuration is invalid and contains the
> following 1 error(s):\nInvalid value custom.CustomStringConverter for
> configuration value.converter: Class custom.CustomStringConverter could not
> be found.\nYou can also find the above list of errors at the endpoint
> `/{connectorType}/config/validate`"
> }
>
> 
>
> If I try running Kafka Connect stadnalone, the error message is the same.
>
> Has anybody faced this already ? What am I missing ?
>
> Many thanks to anybody reading this !
>
> Jehan
>

Re: Kafka Connect Transformations - tranformations for value broken on tombsone events

[Randall Hauch]

Would you mind logging an issue against the Kafka Connect component in our
JIRA? https://issues.apache.org/jira/projects/KAFKA

Thanks!

On Sun, Jun 25, 2017 at 8:21 AM, 男Danngo  wrote:

> Hi all,
>
> I'm trying to use the transformation for Kafka Connect and running into
> issues.
>
>
> The tranformation configuration is:
>
> -
>
> "transforms": "GetAfter",
>
> "transforms.GetAfter.type":
> "org.apache.kafka.connect.transforms.ExtractField$Value",
>
> "transforms.GetAfter.field": "after",
>
> -
>
> And I got the following errors occasionally：
>
> -
>
> org.apache.kafka.connect.errors.DataException: Only Map objects
> supported in absence of schema for [field extraction], found: null
> at org.apache.kafka.connect.transforms.util.Requirements.
> requireMap(Requirements.java:38)
> at org.apache.kafka.connect.transforms.ExtractField.apply(
> ExtractField.java:57)
> at org.apache.kafka.connect.runtime.TransformationChain.
> apply(TransformationChain.java:39)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.
> convertMessages(WorkerSinkTask.java:408)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(
> WorkerSinkTask.java:249)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.
> iteration(WorkerSinkTask.java:179)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.
> execute(WorkerSinkTask.java:148)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(
> WorkerTask.java:139)
> at org.apache.kafka.connect.runtime.WorkerTask.run(
> WorkerTask.java:182)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> -
>
> Seems that tombstone events break the transformation, but after
> checking the source code for Transformations on Value
> (ExtractField$Value, ValueToKey, MaskField$Value, ReplaceField$Value),
> none of them handles tombstone events explicitly, none of them work
> through Tombstone events neither.
>
> So, is this a bug? Or is there any configuration that handles
> tombstone events? Am I missing something?
>

Re: Kafka Connect fails to read offset

[Randall Hauch]

At any time, did your standalone worker config contain
`internal.value.converter.schemas.enable=false`?

On Thu, Jun 15, 2017 at 5:00 AM, Artem B.  wrote:

> Hi all,
>
> Each time I start my source connector it fails to read the offset stored in
> a file with the following error:
>
> 21:05:01:519 | ERROR | pool-1-thread-1 | o.a.k.c.s.OffsetStorageReaderImpl
> |
>  CRITICAL: Failed to deserialize offset data when getting offsets for tas
> k with namespace zohocrm-source-calls. No value for this data will be
> returned, which may break the task or cause it to skip some data. This
> could
> ei
> ther be due to an error in the connector implementation or incompatible
> schema.
> org.apache.kafka.connect.errors.DataException: JsonConverter with
> schemas.enable
> requires "schema" and "payload" fields and may not contain additiona
> l fields. If you are trying to deserialize plain JSON data, set schemas.
> enable=false in your converter configuration.
> at org.apache.kafka.connect.json.JsonConverter.toConnectData(Js
> onConverter.java:309)
>
>
> When the connector is working though, the offsets are committed
> successfully.
>
> 13:49:42:634 | INFO  | pool-5-thread-1 | o.a.k.c.r.WorkerSourceTask |
> Finished WorkerSourceTask{id=zohocrm-source-calls-0} commitOffsets
> successfully in 0 ms
>
> However, when I stop and re-start it, the same error appears.
>
> Here are my StandaloneConfig values:
>
> access.control.allow.methods =
> access.control.allow.origin =
> bootstrap.servers = [localhost:9092]
> internal.key.converter = class org.apache.kafka.connect.json.
> JsonConverter
> internal.value.converter = class org.apache.kafka.connect.json.
> JsonConverter
> key.converter = class io.confluent.connect.avro.AvroConverter
> offset.flush.interval.ms = 6
> offset.flush.timeout.ms = 5000
> offset.storage.file.filename = maxoptra-data.offset
> rest.advertised.host.name = null
> rest.advertised.port = null
> rest.host.name = null
> rest.port = 8083
> task.shutdown.graceful.timeout.ms = 5000
> value.converter = class io.confluent.connect.avro.AvroConverter
>
>
> Here is my connector config:
>
> connector.class = com.maxoptra.data.zoho.connect.
> ZohoCrmSourceConnector
> key.converter = null
> name = zohocrm-source-calls
> tasks.max = 1
> transforms = null
> value.converter = null
>
> Please advise.
>
> Thank you
>
> --
> With best regards,
> Artem.
>

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Randall Hauch]

I expect that this is because the Gradle file does defines the JSON
converter module as "test". See
https://github.com/apache/kafka/blob/trunk/build.gradle#L1082

This is likely because this ensures that the runtime source code will not
directly depend upon the JSON converter module. Any custom software
component (such as custom test modules or applications) that relies upon
these modules and tries to execute Connect workers is thus acting as its
own runtime and should declare the dependency on the JSON converter module
as well as any other modules needed by that environment.

On Thu, Jun 8, 2017 at 11:09 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Yes exactly ... I see that for the Connect "runtime" module, the "json"
> module is a dependency as "Test". I'm able to avoid the exception if I
> change the dependency to be "Compile" so that it will be available even in
> test and at runtime.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Randall Hauch <rha...@gmail.com>
> Sent: Thursday, June 8, 2017 4:05 PM
> To: users@kafka.apache.org
> Subject: Re: ConnectStandalone : not found JsonConverter running in
> IntelliJ
>
> Are you talking about importing the Apache Kafka project into IntelliJ? If
> so, IntelliJ should have the JSON converter's source code in the IDE's
> project. Is that correct? And you're still getting the ConfigException for
> the "org.apache.kafka.connect.json.JsonConverter"?
>
> On Thu, Jun 8, 2017 at 10:57 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
>
> > Hi Randall,
> >
> >
> > after running the "./gradlew idea" command I imported the project into
> > IntelliJ thanks to the Gradle support.
> >
> > I did no changes on that.
> >
> >
> > For now the way I'm debugging is just building the kafka binaries, then
> > running the connect example on the command line with debugging enabled
> > (DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote
> > session from IntelliJ.
> >
> > I hope there is a simpler way to do that ;)
> >
> >
> > Thanks,
> >
> > Paolo.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
> >
> > 
> > From: Randall Hauch <rha...@gmail.com>
> > Sent: Thursday, June 8, 2017 3:14 PM
> > To: users@kafka.apache.org
> > Subject: Re: ConnectStandalone : not found JsonConverter running in
> > IntelliJ
> >
> > Hi, Paolo.
> >
> > How are the Kafka Connect libraries loaded into your IntelliJ project? If
> > they are loaded as external libraries, where in the order of the external
> > libraries does the "org.apache.kafka:connector:connect-json" Maven
> module
> > appear? Or, is that module loaded as source?
> >
> > Best regards,
> >
> > Randall
> >
> > On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno <ppatie...@live.com>
> wrote:
> >
> > > Hi,
> > >
> > >
> > > I'm trying to run the ConnectStandalone application inside IntelliJ
> > > providing the worker and console source properties files from the
> config
> > > dir but I receive the following exception :
> > >
> > >
> > > Exception in thread "main" org.apache.kafka.common.
> > config.ConfigException:
> > > Invalid value org.apache.kafka.connect.json.JsonConverter for
> > > configuration internal.key.converter: Class
> > org.apache.kafka.connect.json.JsonConverter
> > > could not be found.
> > > at org.apache.kafka.common.config.ConfigDef.parseType(
> > ConfigDef.java:711)
> > > at org.apache.kafka.common.config.ConfigDef.parseValue(
> > ConfigDef.java:457)
> > > at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> > > at org.apache.kafka.common.config.AbstractConfig.(
> > > AbstractConfig.java:62)
> > > at org.

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Randall Hauch]

Are you talking about importing the Apache Kafka project into IntelliJ? If
so, IntelliJ should have the JSON converter's source code in the IDE's
project. Is that correct? And you're still getting the ConfigException for
the "org.apache.kafka.connect.json.JsonConverter"?

On Thu, Jun 8, 2017 at 10:57 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi Randall,
>
>
> after running the "./gradlew idea" command I imported the project into
> IntelliJ thanks to the Gradle support.
>
> I did no changes on that.
>
>
> For now the way I'm debugging is just building the kafka binaries, then
> running the connect example on the command line with debugging enabled
> (DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote
> session from IntelliJ.
>
> I hope there is a simpler way to do that ;)
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> 
> From: Randall Hauch <rha...@gmail.com>
> Sent: Thursday, June 8, 2017 3:14 PM
> To: users@kafka.apache.org
> Subject: Re: ConnectStandalone : not found JsonConverter running in
> IntelliJ
>
> Hi, Paolo.
>
> How are the Kafka Connect libraries loaded into your IntelliJ project? If
> they are loaded as external libraries, where in the order of the external
> libraries does the "org.apache.kafka:connector:connect-json" Maven module
> appear? Or, is that module loaded as source?
>
> Best regards,
>
> Randall
>
> On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno <ppatie...@live.com> wrote:
>
> > Hi,
> >
> >
> > I'm trying to run the ConnectStandalone application inside IntelliJ
> > providing the worker and console source properties files from the config
> > dir but I receive the following exception :
> >
> >
> > Exception in thread "main" org.apache.kafka.common.
> config.ConfigException:
> > Invalid value org.apache.kafka.connect.json.JsonConverter for
> > configuration internal.key.converter: Class
> org.apache.kafka.connect.json.JsonConverter
> > could not be found.
> > at org.apache.kafka.common.config.ConfigDef.parseType(
> ConfigDef.java:711)
> > at org.apache.kafka.common.config.ConfigDef.parseValue(
> ConfigDef.java:457)
> > at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> > at org.apache.kafka.common.config.AbstractConfig.(
> > AbstractConfig.java:62)
> > at org.apache.kafka.common.config.AbstractConfig.(
> > AbstractConfig.java:75)
> > at org.apache.kafka.connect.runtime.WorkerConfig.(
> > WorkerConfig.java:197)
> > at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(
> > StandaloneConfig.java:42)
> > at org.apache.kafka.connect.cli.ConnectStandalone.main(
> > ConnectStandalone.java:70)
> >
> >
> > The converter is available in the JSON module of course.
> >
> >
> > Any idea ?
> >
> >
> > Thanks,
> >
> > Paolo
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno<http://twitter.com/ppatierno>
> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> > Blog : DevExperience<http://paolopatierno.wordpress.com/>
> >
>

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Randall Hauch]

Hi, Paolo.

How are the Kafka Connect libraries loaded into your IntelliJ project? If
they are loaded as external libraries, where in the order of the external
libraries does the "org.apache.kafka:connector:connect-json" Maven module
appear? Or, is that module loaded as source?

Best regards,

Randall

On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno  wrote:

> Hi,
>
>
> I'm trying to run the ConnectStandalone application inside IntelliJ
> providing the worker and console source properties files from the config
> dir but I receive the following exception :
>
>
> Exception in thread "main" org.apache.kafka.common.config.ConfigException:
> Invalid value org.apache.kafka.connect.json.JsonConverter for
> configuration internal.key.converter: Class 
> org.apache.kafka.connect.json.JsonConverter
> could not be found.
> at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:711)
> at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:457)
> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> at org.apache.kafka.common.config.AbstractConfig.(
> AbstractConfig.java:62)
> at org.apache.kafka.common.config.AbstractConfig.(
> AbstractConfig.java:75)
> at org.apache.kafka.connect.runtime.WorkerConfig.(
> WorkerConfig.java:197)
> at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(
> StandaloneConfig.java:42)
> at org.apache.kafka.connect.cli.ConnectStandalone.main(
> ConnectStandalone.java:70)
>
>
> The converter is available in the JSON module of course.
>
>
> Any idea ?
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>

Re: Accumulating data in Kafka Connect source tasks

[Randall Hauch]

On January 28, 2016 at 7:07:02 PM, Ewen Cheslack-Postava (e...@confluent.io) 
wrote:
Randall, 

Great question. Ideally you wouldn't need this type of state since it 
should really be available in the source system. In your case, it might 
actually make sense to be able to grab that information from the DB itself, 
although that will also have issues if, for example, there have been 
multiple schema changes and you can no longer get a previous schema from 
the current state of the tables. 
I agree that ideally connectors would be stateless, or at least have no need 
for maintaining state across restarts. Unfortunately, that’s not always 
possible.

Reading the log but using the current schema does pose a problem if/when the 
schema has evolved since the point in the log that we’re currently reading. 
This is far more of an issue if you’re playing catch up and there’s been 
non-compatible schema changes.

Case in point: when MySQL inserts/updates/removes a row from a table, it writes 
an event in the log that includes (a) a table identifier and (b) the row values 
in column-order. There is no other information. Column renames might be okay, 
but adding or removing columns will likely result in mismatching the row values 
to the appropriate columns.

Luckily, MySQL includes the DDL statements in the log, so my connector parses 
these as part of processing and builds up the schema state as it goes along. 
This works beautifully, with the only issue being how to persist and recover 
this after restarts.


The offset storage is probably pretty close to what you're looking for, 
although we obviously structure that very narrowly. Adding in some sort of 
other state store is an interesting idea, though I'd be curious how many 
other systems encounter similar challenges. I think one way to do this 
without huge changes and in a way that properly handles offset commits 
would be to expose a small API for setting local state and have Connect 
store that state right in the same topic (and message) as offsets. To 
handle offset commits and reviving tasks that hit a fault, we would just 
grab the current state as part of the process of committing offsets. Then 
offsets would just be a special case of that more general state. 

However, I'm also wary of doing something like this. Right now every worker 
has to consume the entire offsets topic. This works fine because offsets, 
while capable of being pretty complex, are generally pretty small such that 
there's no concern having to tail it on all workers (and no concern for the 
load on brokers leading those partitions). Once you provide a generic state 
storage mechanism without clear constraints on how it should be used, 
someone will come along and abuse it. Also, with offsets it is very clear 
(to the connector developer) which task should write to which keys (where 
the key here is the partition of the source partitioned stream). With 
support for arbitrary state, ownership of different subsets of the key 
space is very unclear. I think you may not have that particular problem 
because you probably only have 1 partition anyway since you are reading a 
transaction log. 

In any case, you're right that this doesn't exist today. There is one very 
hacky way to get around this, which is to store that schema information in 
your "offsets". This may not be *too* bad -- it'll increase the size of 
offset data, but probably doesn't affect much else. The data size may not 
be that bad as long as offsets aren't committed too frequently. In terms of 
performance, I'm assuming these schema changes are relatively rare, and you 
can just include the same schema object in every offset you create during 
the periods between schema changes so you (and the GC) are probably only 
doing a negligible amount of extra work. 

Hmm, it sound like hammering accumulated state into the offsets could be pretty 
problematic and potentially risky, especially if the state has very different 
size and frequency characteristics than the offsets.

Re: creating a consumer, Connect doesn't provide any utilities to do that 
since the goal is to handle everything Kafka-related for the connector 
developer so they can just focus on getting the data from the other system! 
We could consider exposing some of the worker config though, which I 
imagine is all you really need -- it'd just be convenient to have the 
connection info for the Kafka brokers. 
Having a way to get the worker config would be awesome, and IMO it a nice 
minimalistic approach. If you think this is a good idea, I can log a JIRA and 
take it to the dev list. I’m willing to work on it, too. 

I’m starting to think that storing state on a separate dedicated topic is the 
best option, at least for me. First, connector tasks can easily record their 
state by simply adding more SourceRecord instances during polling. Second, that 
information might be useful for downstream consumers. And third, recording 
state this way requires no changes to 

Re: Accumulating data in Kafka Connect source tasks

[Randall Hauch]

Rather than leave this thread so open ended, perhaps I can narrow down to what 
I think is the best approach. These accumulations are really just additional 
information from the source that don’t get written to the normal topics. 
Instead, each change to the accumulated state can be emitted as source records 
on a dedicated topic. That is very straightforward with the existing Kafka 
Connect.

The challenge I’m struggling with is how a task can/should, upon startup, 
*consume* that stream to rebuild its state. I can set up my own Kafka consumer 
for that topic, but IIUC now my connector config has to include much of the 
same information included in the Kafka Connect workers configuration. 

Am I just missing how a connector can see the worker configuration properties? 
Or is there a way that Kafka Connect can help me create a Kafka consumer?

Best regards,

Randall Hauch

On January 28, 2016 at 12:11:07 AM, Randall Hauch (rha...@gmail.com) wrote:
I’m creating a custom Kafka Connect source connector, and I’m running into a 
situation for which Kafka Connect doesn’t seem to provide a solution out of the 
box. I thought I’d first post to the users list in case I’m just missing a 
feature that’s already there.

My connector’s SourceTask implementation is reading a relational database 
transaction log. That log contains schema changes and row changes, and the row 
changes include a reference to the table and the row values. Thus, as the task 
processes the log, it has to use any schema changes in the log to adjust how it 
converts subsequent row changes into Kafka source records. Should the task stop 
and be restarted elsewhere, it can continue reading the transaction log where 
it left off only if that new task instance can recover the schema state 
accumulated by an earlier task.

While I certainly can use a custom solution to store this state somewhere, it 
seems like other connectors might benefit from having Kafka Connect include 
something out of the box. And, this accumulated state (and its history with 
respect to the source offset at which the state changes) seems like a perfect 
fit for storing in a Kafka topic.

Does Kafka Connect already have a mechanism for tasks to store and recover 
arbitrary state? If not, then is there interest in adding this capability to 
Kafka Connect? (If there is interest, then perhaps the dev list is a better 
venue.)

Best regards,

Randall Hauch

Accumulating data in Kafka Connect source tasks

[Randall Hauch]

I’m creating a custom Kafka Connect source connector, and I’m running into a 
situation for which Kafka Connect doesn’t seem to provide a solution out of the 
box. I thought I’d first post to the users list in case I’m just missing a 
feature that’s already there.

My connector’s SourceTask implementation is reading a relational database 
transaction log. That log contains schema changes and row changes, and the row 
changes include a reference to the table and the row values. Thus, as the task 
processes the log, it has to use any schema changes in the log to adjust how it 
converts subsequent row changes into Kafka source records. Should the task stop 
and be restarted elsewhere, it can continue reading the transaction log where 
it left off only if that new task instance can recover the schema state 
accumulated by an earlier task.

While I certainly can use a custom solution to store this state somewhere, it 
seems like other connectors might benefit from having Kafka Connect include 
something out of the box. And, this accumulated state (and its history with 
respect to the source offset at which the state changes) seems like a perfect 
fit for storing in a Kafka topic.

Does Kafka Connect already have a mechanism for tasks to store and recover 
arbitrary state? If not, then is there interest in adding this capability to 
Kafka Connect? (If there is interest, then perhaps the dev list is a better 
venue.)

Best regards,

Randall Hauch