Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

[Guozhang Wang]

Awesome, thanks!

+1 from me as well.

On Thu, May 5, 2022 at 5:50 PM Artem Livshits
 wrote:

> Hi  Guozhang,
>
> The current DefaultStreamPartitioner behavior is not changed by this KIP,
> so I think we can track it separately.  I've created a ticket:
> https://issues.apache.org/jira/browse/KAFKA-13880.
>
> -Artem
>
>
>
> On Thu, May 5, 2022 at 10:24 AM Guozhang Wang  wrote:
>
> > Hello Artem,
> >
> > Thanks for proposing this KIP. I took a look at the current PR and also
> > thought about its implications on Kafka Streams. Here are some thoughts:
> >
> > Today Kafka Streams use an explicit Partitioner --- note it is not
> > implementing the Producer's Partitioner --- to determine the partition
> > number before calling `producer.send`. Also the record is serialized
> > outside the `send` call as well. That means, the record sent to the
> > producer is always in `` type and partition id is always
> > specified.
> >
> > The reason for serializing outside the producer is that the same producer
> > is used to send to various topics with different schema, and hence we
> > cannot specify a single serializer config. And the reason to determine
> the
> > partition id outside the producer is that we have different logic for
> > windowed record v.s. non-windowed record and hence cannot have a single
> > partitioner config.
> >
> > For the windowed partitioner it does not matter since the key would
> always
> > be specified and hence we would not leverage sticky partitioning. For the
> > non-windowed partitioner it's possible that the key is null, and inside
> the
> > non-windowed customized partitioner, the `DefaultPartitioner` is used
> > indeed. But with this KIP as the new logic is not encoded in the
> configured
> > partitioner it means Kafka Streams would not be able to leverage its
> > benefits.
> >
> > I think we can modify the non-windowed partitioner such that when the key
> > is null, we just set the partition to null, then inside the KafkaProducer
> > we could still leverage on the new sticky behavior. Since in Kafka
> Streams
> > only sink topics data may have null keys which would not be required
> state
> > metadata, relaxing this in the StreamsPartitioner should be fine.
> >
> >
> > Guozhang
> >
> >
> > On Sat, Mar 26, 2022 at 4:05 PM Lucas Bradstreet
> > 
> > wrote:
> >
> > > Hi Artem,
> > >
> > > Thank you for all the work on this. I think it'll be quite impactful.
> > >
> > > +1 non-binding from me.
> > >
> > > Lucas
> > >
> > > On Wed, Mar 23, 2022 at 8:27 PM Luke Chen  wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > Thanks for the KIP and the patience during discussion!
> > > > +1 binding from me.
> > > >
> > > > Luke
> > > >
> > > > On Thu, Mar 24, 2022 at 3:43 AM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks for the KIP and for taking the time to address all the
> > feedback.
> > > > +1
> > > > > (binding)
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Mar 21, 2022 at 4:52 PM Artem Livshits
> > > > >  wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start a vote on
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner
> > > > > > .
> > > > > >
> > > > > > -Artem
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

Re: The latest version of Kafka 3.1.0 does not support Ubuntu 14 04？ The Java client cannot send a message

[Luke Chen]

Hi zw,

I think Kafka should be able to run under any Linux environment.
Could you share the client/broker logs for troubleshooting?

Thank you.
Luke

On Thu, May 5, 2022 at 10:13 PM zw  wrote:

> Hello! My English is not very good, please forgive me.
> The questions are as follows：
> Install Kafka under Ubuntu 14.04.5 LTS_ 2.12-3.1.0. The service can be
> started normally under the default configuration, but when sending messages
> using java clients (Kafka clients), you can see that the server
> automatically creates a topic, but the client cannot send messages
> successfully.
>
>
>
>
> The same configuration is normal under Ubuntu 20.04.4 LTS. Does the latest
> version no longer support Ubuntu 14.04.5 lts?

Re: There is the mistake on kafka.apache.org

[Luke Chen]

Hi Emily,

Thanks for reporting the issue.
But the doc you referred to is for Kafka v0.8.
This issue has been fixed in later versions (PR:
https://github.com/apache/kafka/pull/2010)
So, you can just check the latest version of documentation.

Of course, if you're interested, welcome to submit a PR to fix the issue
for older versions of documents in this repo:
https://github.com/apache/kafka-site

Thank you.
Luke

On Thu, May 5, 2022 at 11:56 PM Emily Kelly  wrote:

> Hi!
>
> After looking at your website I have found a broken link that needs
> rectifying. The page the link is on is:
> kafka.apache.org/08/documentation.html
>
> The broken link is: westnet.com/~gsmith/content/linux-pdflush.htm
>
> The broken link needs to be replaced with:
>
> business-asset.com/eng/wiki-blog/varia/the-linux-page-cache-and-pdflush-theory-of-operation-and-tuning-for-write-heavy-loads-2855.html
>
> A poor website experience may deter users from your site and reflect
> negatively on your brand.
>
> Please change the broken link at your earliest availability or forward this
> email to the right person.
>
> Thank you.
>
> Kind regards, Emily Kelly
>
> Click here
> <
> https://h1.mltrk.io/unsubscribe/bEM2WkcxTGpjeHpMWGZ5U09XeGlCQ3NnU3JsMXp4blh3K2t6eE9adytyTGZ6TXM2L3dTNjdrQnl3MUk3OUJzUXJzdEplTUQvNlIzNjlPaHd0OVNRTlRzc3h4QVdJSWk0OGZoUG1PcHlIcUQxYUxRanJHYm5pZmxmaVFKUTNSRHItLVhucXdka01NbmZIMVc4WWtZMHRNZXc9PQ==--a5636350b4ff13b8ac9480722fbe0685c8b5ef3c
> >
> if you don't want to hear from me again.
>

[jira] [Resolved] (KAFKA-13854) Refactor ApiVersion to MetadataVersion

[dengziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dengziming resolved KAFKA-13854.

Resolution: Fixed

> Refactor ApiVersion to MetadataVersion
> --
>
> Key: KAFKA-13854
> URL: https://issues.apache.org/jira/browse/KAFKA-13854
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: Alyssa Huang
>Priority: Major
>
> In KRaft, we will have a value for {{metadata.version}} corresponding to each 
> IBP. In order to keep this association and make it obvious for developers, we 
> will consolidate the IBP and metadata version into a new MetadataVersion 
> enum. This new enum will replace the existing ApiVersion trait. 
> For IBPs that precede the first KRaft preview version (AK 3.0), we will use a 
> value of -1 for the metadata.version. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #909

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Hi  Guozhang,

The current DefaultStreamPartitioner behavior is not changed by this KIP,
so I think we can track it separately.  I've created a ticket:
https://issues.apache.org/jira/browse/KAFKA-13880.

-Artem



On Thu, May 5, 2022 at 10:24 AM Guozhang Wang  wrote:

> Hello Artem,
>
> Thanks for proposing this KIP. I took a look at the current PR and also
> thought about its implications on Kafka Streams. Here are some thoughts:
>
> Today Kafka Streams use an explicit Partitioner --- note it is not
> implementing the Producer's Partitioner --- to determine the partition
> number before calling `producer.send`. Also the record is serialized
> outside the `send` call as well. That means, the record sent to the
> producer is always in `` type and partition id is always
> specified.
>
> The reason for serializing outside the producer is that the same producer
> is used to send to various topics with different schema, and hence we
> cannot specify a single serializer config. And the reason to determine the
> partition id outside the producer is that we have different logic for
> windowed record v.s. non-windowed record and hence cannot have a single
> partitioner config.
>
> For the windowed partitioner it does not matter since the key would always
> be specified and hence we would not leverage sticky partitioning. For the
> non-windowed partitioner it's possible that the key is null, and inside the
> non-windowed customized partitioner, the `DefaultPartitioner` is used
> indeed. But with this KIP as the new logic is not encoded in the configured
> partitioner it means Kafka Streams would not be able to leverage its
> benefits.
>
> I think we can modify the non-windowed partitioner such that when the key
> is null, we just set the partition to null, then inside the KafkaProducer
> we could still leverage on the new sticky behavior. Since in Kafka Streams
> only sink topics data may have null keys which would not be required state
> metadata, relaxing this in the StreamsPartitioner should be fine.
>
>
> Guozhang
>
>
> On Sat, Mar 26, 2022 at 4:05 PM Lucas Bradstreet
> 
> wrote:
>
> > Hi Artem,
> >
> > Thank you for all the work on this. I think it'll be quite impactful.
> >
> > +1 non-binding from me.
> >
> > Lucas
> >
> > On Wed, Mar 23, 2022 at 8:27 PM Luke Chen  wrote:
> >
> > > Hi Artem,
> > >
> > > Thanks for the KIP and the patience during discussion!
> > > +1 binding from me.
> > >
> > > Luke
> > >
> > > On Thu, Mar 24, 2022 at 3:43 AM Ismael Juma  wrote:
> > >
> > > > Thanks for the KIP and for taking the time to address all the
> feedback.
> > > +1
> > > > (binding)
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Mar 21, 2022 at 4:52 PM Artem Livshits
> > > >  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start a vote on
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner
> > > > > .
> > > > >
> > > > > -Artem
> > > > >
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>

[jira] [Created] (KAFKA-13880) DefaultStreamPartitioner may get "stuck" to one partition for unkeyed messages

[Artem Livshits (Jira)]

Artem Livshits created KAFKA-13880:
--

 Summary: DefaultStreamPartitioner may get "stuck" to one partition 
for unkeyed messages
 Key: KAFKA-13880
 URL: https://issues.apache.org/jira/browse/KAFKA-13880
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.4.0
Reporter: Artem Livshits


While working on KIP-794, I noticed that DefaultStreamPartitioner does not call 
.onNewBatch.  The "sticky" DefaultStreamPartitioner introduced as a result of 
https://issues.apache.org/jira/browse/KAFKA-8601 requires .onNewBatch call in 
order to switch to a new partitions for unkeyed messages, just calling 
.partition would return the same "sticky" partition chosen during the first 
call to .partition.  The partition doesn't change even if the partition leader 
is unavailable.

Ideally, for unkeyed messages the DefaultStreamPartitioner should take 
advantage of the new built-in partitioning logic introduced in 
[https://github.com/apache/kafka/pull/12049.]  Perhaps, it could return null 
partition for unkeyed message, so that KafkaProducer could run built-in 
partitioning logic.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Israel Ekpo]

Thanks Colin.

I think we may need to update the KIP name to reflect the intent of the KIP
and convey everything it’s about if all the 3 action items will be covered
by the same KIP

It contains three parts:

Marking KRraft as Production Ready
Deprecating ZK Mode and
Removing Zookeeper Mode

Should this be broken up in three separate KIPs since it will be done in
multiple releases?

The current name of the KIP only conveys the first item and not the next
two

It is just a thought. I will like to get your perspective



On Thu, May 5, 2022 at 1:19 PM Colin McCabe  wrote:

> Hi all,
>
> Thanks for the comments. I agree that we should split out declaring KRaft
> going production for new clusters from deprecating ZK. We can do the former
> in the next release, 3.3, and the latter in the release after that, 3.4.
>
> I also talked offline with some of the people working on upgrade from ZK
> and it seems like 3.4 is a more realistic target for that work. Partly this
> is because 3.3 will be a bit earlier than I originally thought (for some
> reason I thought it might be October, but Ismael pointed out it's planned
> for August)
>
> I also agree that it will probably be useful to have a 3.5 release
> following the 3.4 one, which will also support ZK. Hopefully we will not
> need a 3.6, but we don't have to decide that now.
>
> I added a timeline section to the KIP to make this all clearer. To be
> clear, it is a preliminary timeline, which may change. It's difficult to
> fully plan out the next 1.5 years of Apache Kafka releases right now -- and
> obviously, there are things which may come up to change our plans. However,
> I think it is still helpful to have the section to give us a feeling for
> the general roadmap.
>
> When you read the KIP, please consider it all speculative except for the
> three proposed changes at the top:
>
> 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
> 3.3 release.
> 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
> 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
>
> best,
> Colin
>
>
> On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> > Yes, all features supported by zk mode will be available in kraft mode in
> > the 3.x series.
> >
> > Ismael
> >
> > On Wed, May 4, 2022, 5:28 PM Israel Ekpo  wrote:
> >
> >> Ismael,
> >>
> >> I like the timeline. However, does this or will this also account for
> >> features  users rely on today in Zookeeper mode being available when
> >> Zookeeper is dropped?
> >>
> >> That’s my main concern
> >>
> >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:
> >>
> >> > Hi Colin,
> >> >
> >> > Thanks for the KIP, this is exciting. Trying to balance progress and
> >> > compatibility, how about the following?
> >> >
> >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> >> > production ready and zk mode is deprecated
> >> > 3. 3.5 (~April 2023): buffer release
> >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
> removed
> >> >
> >> > This would mean about 1 year from kraft being production ready to zk
> >> > removal and 8 months from zk deprecation to zk removal.
> >> >
> >> > If necessary (due to important bugs or security issues), we can do a
> >> couple
> >> > of additional bug fix releases in the 3.5 series after 4.0 is
> released.
> >> >
> >> > Thoughts?
> >> >
> >> > Ismael
> >> >
> >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe  wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I've written a KIP for marking KRaft as production ready. Please
> take a
> >> > > look if you have a chance:
> >> > >
> >> > > https://cwiki.apache.org/confluence/x/8xKhD
> >> > >
> >> > > thanks,
> >> > > Colin
> >> > >
> >> >
> >> --
> >> Israel Ekpo
> >> Lead Instructor, IzzyAcademy.com
> >> https://www.youtube.com/c/izzyacademy
> >> https://izzyacademy.com/
> >>
>
-- 
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

Hello Bruno,

The KIP as been updated. Feel free to give more feedbacks and I will
complete accordingly.

Kr,

F.

Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a écrit :

> Hi Francois,
>
> Thanks for the KIP!
>
> Here my first feedback:
>
> 1. Could you please extend the motivation section, so that it is clear
> for a non-Spring dev why the change is needed? Usually, a motivation
> section benefits a lot from an actual example.
> Extending the motivation section would also make the KIP more
> self-contained which is important IMO since this is kind of a log of the
> major changes to Kafka. Descriptions of major changes should not
> completely depend on external links (which may become dead in future).
> Referencing external resources to point to more details or give context
> is useful, though.
>
> 2. Why do you only want to change/add the constructors that take the
> properties objects and de/serializers and you do not also want to
> add/change the constructors that take only the properties?
>
> 3. I found the following stalled KIP whose motivation is really similar
> to yours:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers
>
> That KIP is also the reason why Kafka Streams still has the constructors
> with the StreamsConfig parameter. Maybe you want to mention this KIP in
> yours or even incorporate the remaining topology test driver API changes
> in your KIP.
> Some related links:
> - https://github.com/apache/kafka/pull/5344#issuecomment-413350338
> - https://github.com/apache/kafka/pull/10484
> - https://issues.apache.org/jira/browse/KAFKA-6386
>
> Best,
> Bruno
>
>
> On 04.05.22 22:26, François Rosière wrote:
> > Hi all,
> >
> > KIP-832 has been created to allow implementing Spring managed
> interceptors
> > for Producers and Consumers.
> >
> > At the moment, interceptors are given as configuration classes to the
> > producer and consumer configurations. So, the idea here would be to
> create
> > 2 new constructors to allow using a Producer and Consumer configuration
> > instead of properties or a key value map of configurations entries.
> > Interceptors could then be given as instances by overriding a config
> method.
> > More details can be found in the Spring issue.
> > https://github.com/spring-projects/spring-kafka/issues/2244
> >
> > Any feedback, proposal, vote for this KIP would be more than welcome.
> >
> > Kind regards,
> >
> > Francois R.
> >
> > Le lun. 2 mai 2022 à 21:05, François Rosière 
> a
> > écrit :
> >
> >> Kip link:
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578
> >>
> >>
> >
>

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[Bruno Cadonna]

Hi Francois,

Thanks for the KIP!

Here my first feedback:

1. Could you please extend the motivation section, so that it is clear 
for a non-Spring dev why the change is needed? Usually, a motivation 
section benefits a lot from an actual example.
Extending the motivation section would also make the KIP more 
self-contained which is important IMO since this is kind of a log of the 
major changes to Kafka. Descriptions of major changes should not 
completely depend on external links (which may become dead in future). 
Referencing external resources to point to more details or give context 
is useful, though.


2. Why do you only want to change/add the constructors that take the 
properties objects and de/serializers and you do not also want to 
add/change the constructors that take only the properties?


3. I found the following stalled KIP whose motivation is really similar 
to yours:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers

That KIP is also the reason why Kafka Streams still has the constructors 
with the StreamsConfig parameter. Maybe you want to mention this KIP in 
yours or even incorporate the remaining topology test driver API changes 
in your KIP.

Some related links:
- https://github.com/apache/kafka/pull/5344#issuecomment-413350338
- https://github.com/apache/kafka/pull/10484
- https://issues.apache.org/jira/browse/KAFKA-6386

Best,
Bruno


On 04.05.22 22:26, François Rosière wrote:

Hi all,

KIP-832 has been created to allow implementing Spring managed interceptors
for Producers and Consumers.

At the moment, interceptors are given as configuration classes to the
producer and consumer configurations. So, the idea here would be to create
2 new constructors to allow using a Producer and Consumer configuration
instead of properties or a key value map of configurations entries.
Interceptors could then be given as instances by overriding a config method.
More details can be found in the Spring issue.
https://github.com/spring-projects/spring-kafka/issues/2244

Any feedback, proposal, vote for this KIP would be more than welcome.

Kind regards,

Francois R.

Le lun. 2 mai 2022 à 21:05, François Rosière  a
écrit :


Kip link:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #112

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13879) Exponential backoff for reconnect does not work

[Chern Yih Cheah (Jira)]

Chern Yih Cheah created KAFKA-13879:
---

 Summary: Exponential backoff for reconnect does not work
 Key: KAFKA-13879
 URL: https://issues.apache.org/jira/browse/KAFKA-13879
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 2.7.0
Reporter: Chern Yih Cheah


When a client connects to a SSL listener using PLAINTEXT security protocol, 
after the TCP connection is setup, the client considers the channel setup is 
complete (in reality the channel setup is not complete yet). The client issues 
API version request after that. When issuing API version request, reconnection 
exponential backoff is reset. Since the broker expects SSL handshake, client's 
API version request will cause the connection to disconnect. Reconnect will 
happen without exponential backoff since it has been reset.

[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java#L249.]
  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] 3.1.1 RC1

[Dongjoon Hyun]

+1 (non-binding)

RC1 was tested with Apache Spark tests

- https://github.com/apache/spark/pull/36135

Thanks,
Dongjoon.

On 2022/05/05 03:25:25 Luke Chen wrote:
> Hi Tom,
> 
> I did:
> 1. check the signature and checksums
> 2. ran quick start with java17 + sacla2.12
> 3. browse java docs/documentations
> 
> +1 (non-binding)
> 
> Thanks for running the release.
> 
> Luke
> 
> On Thu, May 5, 2022 at 12:43 AM Bill Bejeck  wrote:
> 
> > Hi Tom,
> >
> > Thanks for running the release!
> >
> > I did the following checks:
> >
> >1. Validated all the checksum and signatures
> >2. Built from source and ran the unit tests (Java 11)
> >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> >4. Did a quick scan of the Javadoc and the documentation
> >
> > I noticed the same issues as Mickael on the website, but otherwise, it
> > looks good.
> > +1 (binding)
> >
> > Thanks,
> > Bill
> >
> > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz  wrote:
> >
> > > +1 (non-binding). I used the Scala 2.13 binaries and staged Maven
> > artifacts
> > > and ran various tests with them. Thanks for doing the release.
> > >
> > > Jakub
> > >
> > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison 
> > > wrote:
> > >
> > > > Hi Tom,
> > > >
> > > > Thanks for running this release!
> > > >
> > > > I've done the following:
> > > > - Checked signatures and checksums
> > > > - Checked javadocs/maven artifacts
> > > > - Built from source and run all tests with Java 11
> > > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > > >
> > > > It looks like the website has not been updated yet, I still only see
> > > > 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in
> > > > the notable changes section.
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley 
> > > wrote:
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > > >
> > > > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> > > > > since 3.1.0.
> > > > >
> > > > > Release notes for the 3.1.1 release:
> > > > >
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/31/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/31/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > I will share a link one the build is complete.
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Tom
> > > >
> > >
> >
> 

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

[Guozhang Wang]

Hello Artem,

Thanks for proposing this KIP. I took a look at the current PR and also
thought about its implications on Kafka Streams. Here are some thoughts:

Today Kafka Streams use an explicit Partitioner --- note it is not
implementing the Producer's Partitioner --- to determine the partition
number before calling `producer.send`. Also the record is serialized
outside the `send` call as well. That means, the record sent to the
producer is always in `` type and partition id is always
specified.

The reason for serializing outside the producer is that the same producer
is used to send to various topics with different schema, and hence we
cannot specify a single serializer config. And the reason to determine the
partition id outside the producer is that we have different logic for
windowed record v.s. non-windowed record and hence cannot have a single
partitioner config.

For the windowed partitioner it does not matter since the key would always
be specified and hence we would not leverage sticky partitioning. For the
non-windowed partitioner it's possible that the key is null, and inside the
non-windowed customized partitioner, the `DefaultPartitioner` is used
indeed. But with this KIP as the new logic is not encoded in the configured
partitioner it means Kafka Streams would not be able to leverage its
benefits.

I think we can modify the non-windowed partitioner such that when the key
is null, we just set the partition to null, then inside the KafkaProducer
we could still leverage on the new sticky behavior. Since in Kafka Streams
only sink topics data may have null keys which would not be required state
metadata, relaxing this in the StreamsPartitioner should be fine.


Guozhang


On Sat, Mar 26, 2022 at 4:05 PM Lucas Bradstreet 
wrote:

> Hi Artem,
>
> Thank you for all the work on this. I think it'll be quite impactful.
>
> +1 non-binding from me.
>
> Lucas
>
> On Wed, Mar 23, 2022 at 8:27 PM Luke Chen  wrote:
>
> > Hi Artem,
> >
> > Thanks for the KIP and the patience during discussion!
> > +1 binding from me.
> >
> > Luke
> >
> > On Thu, Mar 24, 2022 at 3:43 AM Ismael Juma  wrote:
> >
> > > Thanks for the KIP and for taking the time to address all the feedback.
> > +1
> > > (binding)
> > >
> > > Ismael
> > >
> > > On Mon, Mar 21, 2022 at 4:52 PM Artem Livshits
> > >  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner
> > > > .
> > > >
> > > > -Artem
> > > >
> > >
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

Hi all,

Thanks for the comments. I agree that we should split out declaring KRaft going 
production for new clusters from deprecating ZK. We can do the former in the 
next release, 3.3, and the latter in the release after that, 3.4.

I also talked offline with some of the people working on upgrade from ZK and it 
seems like 3.4 is a more realistic target for that work. Partly this is because 
3.3 will be a bit earlier than I originally thought (for some reason I thought 
it might be October, but Ismael pointed out it's planned for August)

I also agree that it will probably be useful to have a 3.5 release following 
the 3.4 one, which will also support ZK. Hopefully we will not need a 3.6, but 
we don't have to decide that now.

I added a timeline section to the KIP to make this all clearer. To be clear, it 
is a preliminary timeline, which may change. It's difficult to fully plan out 
the next 1.5 years of Apache Kafka releases right now -- and obviously, there 
are things which may come up to change our plans. However, I think it is still 
helpful to have the section to give us a feeling for the general roadmap.

When you read the KIP, please consider it all speculative except for the three 
proposed changes at the top:

1. Mark KRaft as production-ready for new clusters in the upcoming Kafka 3.3 
release.
2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.

best,
Colin


On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> Yes, all features supported by zk mode will be available in kraft mode in
> the 3.x series.
>
> Ismael
>
> On Wed, May 4, 2022, 5:28 PM Israel Ekpo  wrote:
>
>> Ismael,
>>
>> I like the timeline. However, does this or will this also account for
>> features  users rely on today in Zookeeper mode being available when
>> Zookeeper is dropped?
>>
>> That’s my main concern
>>
>> On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:
>>
>> > Hi Colin,
>> >
>> > Thanks for the KIP, this is exciting. Trying to balance progress and
>> > compatibility, how about the following?
>> >
>> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
>> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
>> > production ready and zk mode is deprecated
>> > 3. 3.5 (~April 2023): buffer release
>> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
>> >
>> > This would mean about 1 year from kraft being production ready to zk
>> > removal and 8 months from zk deprecation to zk removal.
>> >
>> > If necessary (due to important bugs or security issues), we can do a
>> couple
>> > of additional bug fix releases in the 3.5 series after 4.0 is released.
>> >
>> > Thoughts?
>> >
>> > Ismael
>> >
>> > On Tue, May 3, 2022, 6:03 PM Colin McCabe  wrote:
>> >
>> > > Hi all,
>> > >
>> > > I've written a KIP for marking KRaft as production ready. Please take a
>> > > look if you have a chance:
>> > >
>> > > https://cwiki.apache.org/confluence/x/8xKhD
>> > >
>> > > thanks,
>> > > Colin
>> > >
>> >
>> --
>> Israel Ekpo
>> Lead Instructor, IzzyAcademy.com
>> https://www.youtube.com/c/izzyacademy
>> https://izzyacademy.com/
>>

[jira] [Created] (KAFKA-13878) Connect deadlock in WorkerConnector on desired state change

[Greg Harris (Jira)]

Greg Harris created KAFKA-13878:
---

 Summary: Connect deadlock in WorkerConnector on desired state 
change
 Key: KAFKA-13878
 URL: https://issues.apache.org/jira/browse/KAFKA-13878
 Project: Kafka
  Issue Type: Bug
Reporter: Greg Harris


We've experienced multiple instances of deadlocks where the connector thread is 
blocked indefinitely with the following stacktrace:
{noformat}
"connector-thread-myconnector" #2059323 prio=5 os_prio=0 cpu=123.43ms 
elapsed=147352.41s tid=0x7f5588160de0 nid=0x18be in Object.wait()  
[0x7f550e3fe000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@16.0.2/Native Method)
    - waiting on 
    at java.lang.Object.wait(java.base@16.0.2/Object.java:320)
    at 
org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:149)
    - locked <0x0005d2253818> (a 
org.apache.kafka.connect.runtime.WorkerConnector)
    at 
org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:118)
    at 
java.util.concurrent.Executors$RunnableAdapter.call(java.base@16.0.2/Executors.java:515)
    at java.util.concurrent.FutureTask.run(java.base@16.0.2/FutureTask.java:264)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@16.0.2/ThreadPoolExecutor.java:1130)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@16.0.2/ThreadPoolExecutor.java:630)
    at java.lang.Thread.run(java.base@16.0.2/Thread.java:831){noformat}
This appears to be caused by a race condition where a notify() is never 
delivered to a task, causing the connector thread to wait indefinitely for a 
notify() call that never comes.

This causes the connector to be unable to process further state changes, or 
generate task configurations.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13877) Flaky RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-13877:
-

 Summary: Flaky 
RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags
 Key: KAFKA-13877
 URL: https://issues.apache.org/jira/browse/KAFKA-13877
 Project: Kafka
  Issue Type: Bug
  Components: streams, unit tests
Reporter: Guozhang Wang


The following test fails on local testbeds about once per 10-15 runs:

{code}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at 
org.apache.kafka.streams.integration.RackAwarenessIntegrationTest.shouldDistributeStandbyReplicasOverMultipleClientTags(RackAwarenessIntegrationTest.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:53)
at 
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)


{code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

There is the mistake on kafka.apache.org

[Emily Kelly]

Hi!

After looking at your website I have found a broken link that needs
rectifying. The page the link is on is:
kafka.apache.org/08/documentation.html

The broken link is: westnet.com/~gsmith/content/linux-pdflush.htm

The broken link needs to be replaced with:
business-asset.com/eng/wiki-blog/varia/the-linux-page-cache-and-pdflush-theory-of-operation-and-tuning-for-write-heavy-loads-2855.html

A poor website experience may deter users from your site and reflect
negatively on your brand.

Please change the broken link at your earliest availability or forward this
email to the right person.

Thank you.

Kind regards, Emily Kelly

Click here

if you don't want to hear from me again.

[jira] [Created] (KAFKA-13876) Mirrormaker-2 consumer settings not working

[Christian Bates (Jira)]

Christian Bates created KAFKA-13876:
---

 Summary: Mirrormaker-2 consumer settings not working
 Key: KAFKA-13876
 URL: https://issues.apache.org/jira/browse/KAFKA-13876
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.6.1, 2.5.0
Reporter: Christian Bates


I am starting a mirrormaker connect cluster using ./bin/connect-mirror-maker.sh 
and a properties file.

I followed the guide at 
https://github.com/apache/kafka/tree/trunk/connect/mirror to attempt to 
configure the consumer properties, however, no matter what I set, the timeout 
for the consumer remains fixed at the default, confirmed both by kafka 
outputting its config in the log and by timing how long between disconnect 
messages I get.  e.g. I set:

{{CLOUD_EU.consumer.request.timeout.ms=12}}

In the properties that I start MM-2 with and it is ignored.  

 

based on various guides I have found while looking at this, I have also tried

{{CLOUD_EU.request.timeout.ms=12}}
{{CLOUD_EU.cluster.consumer.request.timeout.ms=12}}
{{CLOUD_EU.consumer.override.request.timeout.ms=12}}
{{CLOUD_EU.cluster.consumer.override.request.timeout.ms=12}}

None of which have worked.  

How can I change the consumer request.timeout setting?

 

Context:

I am attempting to use mirrormaker 2 to replicate data between AWS Managed 
Kafkas (MSK) in 2 different AWS regions - one in eu-west-1 (CLOUD_EU) and the 
other in us-west-2 (CLOUD_NA), both running Kafka 2.6.1.  For testing I am 
currently trying just to replicate topics 1 way, from EU -> NA.

This works fine for topics with small messages on them, but one of my topic has 
binary messages up to 20MB in size.  When I try to replicate that topic I get 
an error every 30 seconds 

{{[2022-04-21 13:47:05,268] INFO [Consumer clientId=consumer-29, groupId=null] 
Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 2: {}. 
(org.apache.kafka.clients.FetchSessionHandler:481)}}
{{org.apache.kafka.common.errors.DisconnectException}}

 

When logging in DEBUG to get more information we get

{{[2022-04-21 13:47:05,267] DEBUG [Consumer clientId=consumer-29, groupId=null] 
Disconnecting from node 2 due to request timeout. 
(org.apache.kafka.clients.NetworkClient:784)}}
{{[2022-04-21 13:47:05,268] DEBUG [Consumer clientId=consumer-29, groupId=null] 
Cancelled request with header RequestHeader(apiKey=FETCH, apiVersion=11, 
clientId=consumer-29, correlationId=35) due to node 2 being disconnected 
(org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient:593)}}

It gets stuck in a loop constantly disconnecting with request timeout every 30s 
and then trying again.

Looking at this, I suspected that the problem is the `request.timeout.ms` was 
on the default (30s) and timing out trying to read the topic with many large 
messages, hence I was trying to override the consumer settings to have a longer 
timeout.

 

Properties file I am using to start the cluster:

 

# specify any number of cluster aliases
clusters = CLOUD_EU, CLOUD_NA

# connection information for each cluster
CLOUD_EU.bootstrap.servers = kafka.eu-west-1.amazonaws.com:9092
CLOUD_NA.bootstrap.servers = kafka.us-west-2.amazonaws.com:9092

# enable and configure individual replication flows
CLOUD_EU->CLOUD_NA.enabled = true
CLOUD_EU->CLOUD_NA.topics = METRICS_ATTACHMENTS_OVERSIZE_EU
CLOUD_NA->CLOUD_EU.enabled = false

replication.factor=3
tasks.max = 1

# Internal Topic Settings  
#
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3

    Kafka Settings    
###

# CLOUD_EU cluster over writes
CLOUD_EU.consumer.request.timeout.ms=12
CLOUD_EU.consumer.session.timeout.ms=15



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

The latest version of Kafka 3.1.0 does not support Ubuntu 14 04？ The Java client cannot send a message

[zw]

Hello! My English is not very good, please forgive me.
The questions are as follows：
Install Kafka under Ubuntu 14.04.5 LTS_ 2.12-3.1.0. The service can be started 
normally under the default configuration, but when sending messages using java 
clients (Kafka clients), you can see that the server automatically creates a 
topic, but the client cannot send messages successfully.




The same configuration is normal under Ubuntu 20.04.4 LTS. Does the latest 
version no longer support Ubuntu 14.04.5 lts?

Re: [Discuss] KIP-581: Value of optional null field which has default value

[Cheng Pan]

Update PR links

[1] https://github.com/apache/kafka/pull/12126

On 2022/05/05 13:16:59 Cheng Pan wrote:
> Hi Mickael,
> 
> Thanks for ping me. 
> 
> I updated the KIP-581 description to narrow the change scope to address this 
> specific issue, and raised a WIP PR[1] to implement it.
> 
> Please let me know if you have any concerns.
> 
> [1] https://github.com/apache/kafka/pull/12126
> 
> Thanks,
> Cheng Pan
> 
> On 2022/05/02 12:57:25 Mickael Maison wrote:
> > Hi Cheng Pan,
> > 
> > Thanks for raising this KIP, this would be a useful improvement!
> > 
> > You never started a VOTE thread for this KIP. Are you interested in
> > finishing up this work?
> > If so, based on the discussion, I think you can open a VOTE thread as
> > described in 
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
> > If not, it's not a problem, someone else can volunteer to pick it up.
> > 
> > Please let us know.
> > 
> > Thanks,
> > Mickael
> > 
> > On Sat, Aug 8, 2020 at 11:05 AM Ruslan Gibaiev  wrote:
> > >
> > > Hello guys.
> > > Proposed PR seems to be fixing the issue in a backward-compatible way.
> > > Let's please move forward with it. Would be great to see it included into 
> > > next Kafka release
> > > Thank you
> > >
> > > On 2020/07/29 02:49:07, "379377...@qq.com" <379377...@qq.com> wrote:
> > > > Hi Chris,
> > > >
> > > > Thanks for your good suggestion, the KIP document and draft PR has been 
> > > > updated, please review again.
> > > >
> > > > And I found due to my misoperation, the mail thread has been broken, no 
> > > > idea how to fix it.
> > > >
> > > >
> > > >
> > > >
> > > > Thanks
> > > > Cheng Pan
> > > >
> > > > From: Christopher Egerton
> > > > Date: 2020-05-04 10:53
> > > > To: dev
> > > > Subject: Re: [Discuss] KIP-581: Value of optional null field which has 
> > > > default value
> > > > Hi Cheng,
> > > >
> > > > I think refactoring that method should be fine (if maybe a little 
> > > > painful);
> > > > the method itself is private and all places that it's invoked directly 
> > > > are
> > > > either package-private or non-static, so it shouldn't affect any of the
> > > > public methods of the JSON converter to change "convertToConnect" to be
> > > > non-static. Even if it did, the only parts of the JSON converter that 
> > > > are
> > > > public API (and therefore officially subject to concerns about
> > > > compatibility) are the methods it implements that satisfy the 
> > > > "Converter"
> > > > and "HeaderConverter" interfaces.
> > > >
> > > > Would you mind explicitly specifying in the KIP that the new property 
> > > > will
> > > > be added for the JSON converter only, and that it will affect both
> > > > serialization and deserialization?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Apr 28, 2020 at 10:52 AM 379377944 <379377...@qq.com> wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > >
> > > > > Thanks for your reminder, the original implement is deprecated, I just
> > > > > update the JIRA with the new
> > > > > PR link:  https://github.com/apache/kafka/pull/8575
> > > > >
> > > > >
> > > > > As question 2), I agree with you that we should consider both
> > > > > serialization and deserialization, and as you said, I only implement 
> > > > > the
> > > > > serialization now. This is  because the original serde implement is 
> > > > > not
> > > > > symmetrical, the convertToConnect is a static method and can’t access 
> > > > > the
> > > > > field in JsonConverter
> > > > > instance, maybe I should do some refactoring to implement the
> > > > > deserialization.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Cheng Pan
> > > > >  Original Message
> > > > > Sender: Christopher Egerton
> > > > > Recipient: dev
> > > > > Date: Wednesday, Apr 15, 2020 02:28
> > > > > Subject: Re: [Discuss] KIP-581: Value of optional null field which has
> > > > > default value
> > > > >
> > > > >
> > > > > Hi Cheng, Thanks for the KIP! I really appreciate the care that was 
> > > > > taken
> > > > > to ensure backwards compatibility for existing users, and the minimal
> > > > > changes to public interface that are suggested to address this. I 
> > > > > have two
> > > > > quick requests for clarification: 1) Where is the proposed
> > > > > "accept.optional.null" property going to apply? It's hinted that 
> > > > > it'll take
> > > > > effect on the JSON converter but not actually called out anywhere. 2)
> > > > > Assuming this takes effect on the JSON converter, is the intent to 
> > > > > alter
> > > > > the semantics for both serialization and deserialization? The code 
> > > > > snippet
> > > > > from the JSON converter that's included in the KIP comes from the
> > > > > "convertToJson" method, which is used for serialization. However, 
> > > > > based on
> > > > > 

Re: [Discuss] KIP-581: Value of optional null field which has default value

[Cheng Pan]

Hi Mickael,

Thanks for ping me. 

I updated the KIP-581 description to narrow the change scope to address this 
specific issue, and raised a WIP PR[1] to implement it.

Please let me know if you have any concerns.

[1] https://github.com/apache/kafka/pull/8575

Thanks,
Cheng Pan

On 2022/05/02 12:57:25 Mickael Maison wrote:
> Hi Cheng Pan,
> 
> Thanks for raising this KIP, this would be a useful improvement!
> 
> You never started a VOTE thread for this KIP. Are you interested in
> finishing up this work?
> If so, based on the discussion, I think you can open a VOTE thread as
> described in 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process
> If not, it's not a problem, someone else can volunteer to pick it up.
> 
> Please let us know.
> 
> Thanks,
> Mickael
> 
> On Sat, Aug 8, 2020 at 11:05 AM Ruslan Gibaiev  wrote:
> >
> > Hello guys.
> > Proposed PR seems to be fixing the issue in a backward-compatible way.
> > Let's please move forward with it. Would be great to see it included into 
> > next Kafka release
> > Thank you
> >
> > On 2020/07/29 02:49:07, "379377...@qq.com" <379377...@qq.com> wrote:
> > > Hi Chris,
> > >
> > > Thanks for your good suggestion, the KIP document and draft PR has been 
> > > updated, please review again.
> > >
> > > And I found due to my misoperation, the mail thread has been broken, no 
> > > idea how to fix it.
> > >
> > >
> > >
> > >
> > > Thanks
> > > Cheng Pan
> > >
> > > From: Christopher Egerton
> > > Date: 2020-05-04 10:53
> > > To: dev
> > > Subject: Re: [Discuss] KIP-581: Value of optional null field which has 
> > > default value
> > > Hi Cheng,
> > >
> > > I think refactoring that method should be fine (if maybe a little 
> > > painful);
> > > the method itself is private and all places that it's invoked directly are
> > > either package-private or non-static, so it shouldn't affect any of the
> > > public methods of the JSON converter to change "convertToConnect" to be
> > > non-static. Even if it did, the only parts of the JSON converter that are
> > > public API (and therefore officially subject to concerns about
> > > compatibility) are the methods it implements that satisfy the "Converter"
> > > and "HeaderConverter" interfaces.
> > >
> > > Would you mind explicitly specifying in the KIP that the new property will
> > > be added for the JSON converter only, and that it will affect both
> > > serialization and deserialization?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Apr 28, 2020 at 10:52 AM 379377944 <379377...@qq.com> wrote:
> > >
> > > > Hi Chris,
> > > >
> > > >
> > > > Thanks for your reminder, the original implement is deprecated, I just
> > > > update the JIRA with the new
> > > > PR link:  https://github.com/apache/kafka/pull/8575
> > > >
> > > >
> > > > As question 2), I agree with you that we should consider both
> > > > serialization and deserialization, and as you said, I only implement the
> > > > serialization now. This is  because the original serde implement is not
> > > > symmetrical, the convertToConnect is a static method and can’t access 
> > > > the
> > > > field in JsonConverter
> > > > instance, maybe I should do some refactoring to implement the
> > > > deserialization.
> > > >
> > > >
> > > > Thanks,
> > > > Cheng Pan
> > > >  Original Message
> > > > Sender: Christopher Egerton
> > > > Recipient: dev
> > > > Date: Wednesday, Apr 15, 2020 02:28
> > > > Subject: Re: [Discuss] KIP-581: Value of optional null field which has
> > > > default value
> > > >
> > > >
> > > > Hi Cheng, Thanks for the KIP! I really appreciate the care that was 
> > > > taken
> > > > to ensure backwards compatibility for existing users, and the minimal
> > > > changes to public interface that are suggested to address this. I have 
> > > > two
> > > > quick requests for clarification: 1) Where is the proposed
> > > > "accept.optional.null" property going to apply? It's hinted that it'll 
> > > > take
> > > > effect on the JSON converter but not actually called out anywhere. 2)
> > > > Assuming this takes effect on the JSON converter, is the intent to alter
> > > > the semantics for both serialization and deserialization? The code 
> > > > snippet
> > > > from the JSON converter that's included in the KIP comes from the
> > > > "convertToJson" method, which is used for serialization. However, based 
> > > > on
> > > > https://github.com/apache/kafka/blob/ea47a885b1fe47dfb87c1dc86db1b0e7eb8a273c/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L712-L713
> > > > it looks like the converter also inserts the default value for
> > > > optional-but-null data during deserialization. Thanks again for the KIP!
> > > > Cheers, Chris On Wed, Mar 18, 2020 at 12:00 AM Cheng Pan 
> > > > <379377...@qq.com>
> > > > wrote: > Hi all, > > I'd like to use this thread to discuss KIP-581: 
> > > > Value
> > > > of optional null > field which has default value, please see detail 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #111

[Apache Jenkins Server]

See 

Re: [VOTE] 3.2.0 RC1

[Jakub Scholz]

+1 (non-binding).

I used the Scala 2.13 binaries and the staged Maven artifacts and ran
various tests with them. Thanks for doing the release.

Jakub

On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.2.0.
>
> * log4j 1.x is replaced with reload4j (KAFKA-9366)
> * StandardAuthorizer for KRaft (KIP-801)
> * Send a hint to the partition leader to recover the partition (KIP-704)
> * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> * kafka-console-producer writes headers and null values (KIP-798 and
> KIP-810)
> * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
> * Static membership protocol lets the leader skip assignment (KIP-814)
> * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> * Connect APIs list all connector plugins and retrieve their
> configuration (KIP-769)
> * TimestampConverter SMT supports different unix time precisions (KIP-808)
> * Connect source tasks handle producer exceptions (KIP-779)
>
>
> Release notes for the 3.2.0 release:
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, May 10th, 9am PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> https://github.com/apache/kafka/releases/tag/3.2.0-rc1
>
> * Documentation:
> https://kafka.apache.org/32/documentation.html
>
> * Protocol:
> https://kafka.apache.org/32/protocol.html
>
> * Successful Jenkins builds for the 3.2 branch:
> Unit/integration tests: I'll share a link once the builds complete
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
>
> /**
>
> Thanks,
> Bruno
>