Re: [ANNOUNCE] New committer: Igor Soarez

[Tom Bentley]

Congratulations Igor!

On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai  wrote:

> Congratulations, Igor! you are one of the best Kafka developers!!!
>
> Mickael Maison  於 2024年4月25日 週四 上午2:16寫道：
>
> > Congratulations Igor!
> >
> > On Wed, Apr 24, 2024 at 8:06 PM Colin McCabe  wrote:
> > >
> > > Hi all,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Igor Soarez.
> > >
> > > Igor has been a Kafka contributor since 2019. In addition to being a
> > regular contributor and reviewer, he has made significant contributions
> to
> > improving Kafka's JBOD support in KRaft mode. He has also contributed to
> > discussing and reviewing many KIPs such as KIP-690, KIP-554, KIP-866, and
> > KIP-938.
> > >
> > > Congratulations, Igor!
> > >
> > > Thanks,
> > >
> > > Colin (on behalf of the Apache Kafka PMC)
> >
>

Re: [DISCUSS] KIP-932: Queues for Kafka

[Tom Bentley]

Hi Andrew (and Omnia),

Thanks for the KIP. I hope to provide some feedback on this KIP soon, but I
had a thought on the specific subject of group configs and MM2. If brokers
validate for known groups configs then doesn't this induce an ordering
requirement on upgrading clusters: Wouldn't you have to upgrade a
destination cluster first, in order that it knew about `group.type`,
otherwise it would reject attempts to configure an unknown group config
parameter? A similar issue arises wrt topic configs, but this is the first
instance (that I'm aware of) of a config being added during the MM2 era, so
perhaps this is a minor problem worth thinking about.

Cheers,

Tom

On Wed, 3 Apr 2024 at 05:31, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Omnia,
> Thanks for your questions.
>
> The DR angle on `group.type` is interesting and I had not considered it.
> The namespace of groups contains
> both consumer groups and share groups, so I was trying to ensure that
> which group type was used was
> deterministic rather than a race to create the first member. There are
> already other uses of the group protocol
> such as Kafka Connect, so it’s all a bit confusing even today.
>
> It is actually KIP-848 which introduces configurations for group resources
> and KIP-932 is just building on
> the idea. I think that MM2 will need to sync these configurations. The
> question of whether `group.type` is
> a sensible configuration I think is separate.
>
> Imagine that we do have `group.type` as a group configuration. How would
> we end up with groups with
> the same ID but different types on the two ends of MM2? Assuming that both
> ends have KIP-932 enabled,
> either the configuration was not set, and a consumer group was made on one
> end while a share group was
> made on the other, OR, the configuration was set but its value changed,
> and again we get a divergence.
>
> I think that on balance, having `group.type` as a configuration does at
> least mean there’s a better chance that
> the two ends of MM2 do agree on the type of group. I’m happy to consider
> other ways to do this better. The
> fact that we have different kinds of group in the same namespace is the
> tricky thing. I think this was possible
> before this KIP, but it’s much more likely now.
>
>
> Onto the question of memory. There are several different parts to this,
> all of which are distributed across
> the cluster.
>
> * For the group coordinator, the memory consumption will be affected by
> the number of groups,
> the number of members and the number of topic-partitions to be assigned to
> the members. The
> group coordinator is concerned with membership and assignment, so the
> memory per topic-partition
> will be small.
> * For the share coordinator, the memory consumption will be affected by
> the number of groups, the
> number of topic-partitions being consumed in the group, and the number of
> in-flight records, but not
> the number of members. We should be talking about no more than kilobytes
> per topic-partition.
> * For the share-partition leader, the memory consumption will be affected
> by the number of share group
> members assigned the topic-partition and the number of in-flight records.
> Again, we should be talking
> about no more than kilobytes per topic-partition.
>
> Of these, the factor that is not directly under control is the number of
> topic-partitions. The reason is that
> I wanted to avoid a situation where the number of partitions in a topic
> was increased and suddenly
> consumption in a share group hit a limit that was not anticipated.
>
> I could introduce a configuration for controlling the number of topics
> allowed to be subscribed in a share
> group. Personally, I think 1 would be a good starting point.
>
> Let me know what you think.
>
> Thanks,
> Andrew
>
>
> > On 2 Apr 2024, at 15:39, Omnia Ibrahim  wrote:
> >
> > Hi Andrew,
> > Thanks for the KIP it is definitely an interesting read. I have few
> questions
> > As the KIP proposing extending `AdminClient.incrementalAlterConfigs` to
> add an explicit `group.type` what would this means for DR feature in MM2
> offering?
> > Right now MM2 sync consumer group offsets from source to destination
> cluster. And it also offer sync ACLs which contribute to DR feature. Would
> this KIP means MM2 needs to also sync the type of groups to destination?
> > As `AdminClient.incrementalAlterConfigs` means "when a new group is
> created with this name, it must have this type”. What will happened if
> clusters on both ends of MM2 has same group id but with different types?
> > If this concern is out of the scope we might need to call this out
> somewhere in the KIP.
> > While the number of share-group and the number of consumers in
> share-group is limited by `group.share.max.groups`and
> `group.share.max.size` the total number of share-group state records that
> might need to be loaded in-memeory has another factor which is the number
> of partitions. In cases where 

Re: [ANNOUNCE] New committer: Christo Lolov

[Tom Bentley]

Congratulations!

On Wed, 27 Mar 2024 at 21:10, Matthias J. Sax  wrote:

> Congrats!
>
> On 3/26/24 9:39 PM, Christo Lolov wrote:
> > Thank you everyone!
> >
> > It wouldn't have been possible without quite a lot of reviews and
> extremely
> > helpful inputs from you and the rest of the community! I am looking
> forward
> > to working more closely with you going forward :)
> >
> > On Tue, 26 Mar 2024 at 14:31, Kirk True  wrote:
> >
> >> Congratulations Christo!
> >>
> >>> On Mar 26, 2024, at 7:27 AM, Satish Duggana 
> >> wrote:
> >>>
> >>> Congratulations Christo!
> >>>
> >>> On Tue, 26 Mar 2024 at 19:20, Ivan Yurchenko  wrote:
> 
>  Congrats!
> 
>  On Tue, Mar 26, 2024, at 14:48, Lucas Brutschy wrote:
> > Congrats!
> >
> > On Tue, Mar 26, 2024 at 2:44 PM Federico Valeri <
> fedeval...@gmail.com>
> >> wrote:
> >>
> >> Congrats!
> >>
> >> On Tue, Mar 26, 2024 at 2:27 PM Mickael Maison <
> >> mickael.mai...@gmail.com> wrote:
> >>>
> >>> Congratulations Christo!
> >>>
> >>> On Tue, Mar 26, 2024 at 2:26 PM Chia-Ping Tsai  >
> >> wrote:
> 
>  Congrats Christo!
> 
>  Chia-Ping
> >
> >>
> >>
> >
>
>

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

[Tom Bentley]

Hi Mickael,

+1 to getting KIP-390 merged with the per-codec level config.

Thanks,

Tom

On Fri, 9 Feb 2024 at 07:27, Diop, Assane  wrote:
>
> Hi Mickael
> I will take some time today to add some documentation to KIP-984. I actually 
> have a working implementation of the plugin and I have been running test. I 
> would be ready to do a PR anytime.
> As far as motivation, Divij did asked about that and I responded in an email  
> to the mailing list but I will update the KIP -984 accordingly as well.
>
> Assane
>
> -Original Message-
> From: Mickael Maison 
> Sent: Wednesday, February 7, 2024 6:40 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-390: Support Compression Level (rebooted)
>
> Hi Divij,
>
> Thanks for bringing that point. After reading KIP-984, I don't think it 
> supersedes KIP-390/KIP-780. Being able to tune the built-in codecs would 
> directly benefit many users. It may also cover some scenarios that motivated 
> KIP-984 without requiring users to write a custom codec.
> I've not commented in the KIP-984 thread yet but at the moment it seems very 
> light on details (no proposed API for codecs, no explanations of error 
> scenarios if clients or brokers don't have compatible codecs), including the 
> motivation which is important when exposing new APIs. On the other hand, 
> KIP-390/KIP-780 have much more details with benchmarks to support the 
> motivation.
>
> In my opinion starting with the compression level (KIP-390) is a good first 
> step and I think we should focus on that and deliver it. I believe one of the 
> reasons KIP-780 wasn't voted is because we never delivered KIP-390 and nobody 
> was keen on building a KIP on top of another undelivered KIP.
>
> Thanks,
> Mickael
>
>
> On Wed, Feb 7, 2024 at 12:27 PM Divij Vaidya  wrote:
> >
> > Hey Mickael
> >
> > Since this KIP was written, we have a new proposal to make the
> > compression completely pluggable
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-984%3A+Add+pluggable+compression+interface+to+Kafka.
> > If we implement that KIP, would it supersede the need for adding fine
> > grain compression controls in Kafka?
> >
> > It might be beneficial to have a joint proposal of these two KIPs
> > which may satisfy both use cases.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Wed, Feb 7, 2024 at 11:14 AM Mickael Maison
> > 
> > wrote:
> >
> > > Hi,
> > >
> > > I'm resurrecting this old thread as this KIP would be a nice
> > > improvement and almost 3 years later the PR for this KIP has still
> > > not been merged!
> > >
> > > The reason is that during reviews we noticed the proposed
> > > configuration, compression.level, was not easy to use as each codec
> > > has its own valid range of levels [0].
> > >
> > > As proposed by Jun in the PR [1], I updated the KIP to use
> > > compression..level configurations instead of a single
> > > compression.level setting. This syntax would also line up with the
> > > proposal to add per-codec configuration options from KIP-780 [2]
> > > (still to be voted). I moved the original proposal to the rejected
> > > section.
> > >
> > > I've put the original voters and KIP author on CC. Let me know if
> > > you have any feedback.
> > >
> > > 0: https://github.com/apache/kafka/pull/10826
> > > 1:
> > > https://github.com/apache/kafka/pull/10826#issuecomment-1795952612
> > > 2:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support
> > > +fine-grained+compression+options
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Fri, Jun 11, 2021 at 10:00 AM Dongjin Lee  wrote:
> > > >
> > > > This KIP is now passed with:
> > > >
> > > > - binding: +3 (Ismael, Tom, Konstantine)
> > > > - non-binding: +1 (Ryanne)
> > > >
> > > > Thanks again to all the supporters. I also updated the KIP by
> > > > moving the compression buffer option into the 'Future Works'
> > > > section, as Ismael proposed.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > >
> > > >
> > > > On Fri, Jun 11, 2021 at 3:03 AM Konstantine Karantasis
> > > >  wrote:
> > > >
> > > > > Makes sense. Looks like a good improvement. Thanks for including
> > > > > the evaluation in the proposal Dongjin.
> > > 

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

[Tom Bentley]

Hi Mickael,

You'll have seen that I left some comments on the discussion thread, but
they're minor enough that I'm happy to vote +1 here.

Thanks,

Tom

On Thu, 11 Jan 2024 at 06:14, Mickael Maison 
wrote:

> Bumping this thread since I've not seen any feedback.
>
> Thanks,
> Mickael
>
> On Tue, Dec 19, 2023 at 10:03 AM Mickael Maison
>  wrote:
> >
> > Hi,
> >
> > I'd like to start a vote on KIP-877:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugins+and+connectors+to+register+metrics
> >
> > Let me know if you have any feedback.
> >
> > Thanks,
> > Mickael
>
>

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Tom Bentley]

Hi Mickael,

Thanks for the KIP! I can tell a lot of thought went into this. I have a
few comments, but they're all pretty trivial and aimed at making the
correct use of this API clearer to implementors.

1. Configurable and Reconfigurable both use a verb in the imperative mood
for their method name. Monitorable doesn't, which initially seemed a bit
inconsistent to me, but I think your intention is to allow plugins to
merely retain a reference to the PluginMetrics, and allow registering
metrics at any later point? If that's the case you could add something like
"Plugins can register and unregister metrics using the given PluginMetrics
at any point in their lifecycle prior to their close method being called"
to the javadoc to make clear how this can be used.
2. I assume PluginMetrics will be thread-safe? We should document that as
part of the contract.
3. I don't think IAE is quite right for duplicate metrics. In this case the
arguments themselves are fine, it's the current state of the PluginMetrics
which causes the problem. If the earlier point about plugins being allowed
to register and unregister metrics at any point is correct then this
exception could be thrown after configuration time. That being the case I
think a new exception type might be clearer.
4. You define some semantics for PluginMetrics.close(): It might be a good
idea to override the inherited method and add that as javadoc.
5. You say "It will be the responsibility of the plugin that creates
metrics to call close() of the PluginMetrics instance they were given to
remove their metrics." But you don't provide any guidance to users about
when they need to do this. I guess that they should be doing this in their
plugin's close method (and that's why you're only adding Monitorable to
plugins which implement Closeable and AutoCloseable), but if that's the
case then let's say so in the Javadoc.

Thanks again,

Tom

On Wed, 13 Dec 2023 at 06:09, Mickael Maison 
wrote:

> Hi,
>
> I've not received any feedback since I updated the KIP.
> I'll wait a few more days and if there's no further feedback I'll start a
> vote.
>
> Thanks,
> Mickael
>
> On Tue, Nov 7, 2023 at 6:29 PM Mickael Maison 
> wrote:
> >
> > Hi,
> >
> > A bit later than initially planned I finally restarted looking at this
> KIP.
> >
> > I made a few significant changes to the proposed APIs.
> > I considered Chris' idea of automatically removing metrics but decided
> > to leave that responsibility to the plugins. All plugins that will
> > support this feature have close/stop methods and will need to close
> > their PluginMetrics instance. This simplifies the required changes a
> > lot and I think it's not a big ask on users implementing plugins.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 30, 2023 at 11:32 AM Mickael Maison
> >  wrote:
> > >
> > > Hi Jorge,
> > >
> > > There are a few issues with the current proposal. Once 3.5 is out, I
> > > plan to start looking at this again.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, May 15, 2023 at 3:19 PM Jorge Esteban Quilcate Otoya
> > >  wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Just to check the status of this KIP as it looks very useful. I can
> see how
> > > > new Tiered Storage interfaces and plugins may benefit from this.
> > > >
> > > > Cheers,
> > > > Jorge.
> > > >
> > > > On Mon, 6 Feb 2023 at 23:00, Chris Egerton 
> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I agree that adding a getter method for Monitorable isn't great. A
> few
> > > > > alternatives come to mind:
> > > > >
> > > > > 1. Introduce a new ConfiguredInstance (name subject to change)
> interface
> > > > > that wraps an instance of type T, but also contains a getter
> method for any
> > > > > PluginMetrics instances that the plugin was instantiated with
> (which may
> > > > > return null either if no PluginMetrics instance could be created
> for the
> > > > > plugin, or if it did not implement the Monitorable interface).
> This can be
> > > > > the return type of the new AbstractConfig::getConfiguredInstance
> variants.
> > > > > It would give us room to move forward with other
> plugin-for-your-plugin
> > > > > style interfaces without cluttering things up with getter methods.
> We could
> > > > > even add a close method to this interface which would handle
> cleanup of all
> > > > > extra resources allocated for the plugin by the runtime, and even
> possibly
> > > > > the plugin itself.
> > > > >
> > > > > 2. Break out the instantiation logic into two separate steps. The
> first
> > > > > step, creating a PluginMetrics instance, can be either private or
> public
> > > > > API. The second step, providing that PluginMetrics instance to a
> > > > > newly-created object, can be achieved with a small tweak of the
> proposed
> > > > > new methods for the AbstractConfig class; instead of accepting a
> Metrics
> > > > > instance, they would now accept a PluginMetrics instance. For the
> first
> > > > > step, we might even introduce a 

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

[Tom Bentley]

Congratulations!

On Thu, 28 Dec 2023 at 06:17, Philip Nee  wrote:

> congrats divij!
>
> On Wed, Dec 27, 2023 at 8:55 AM Justine Olshan
> 
> wrote:
>
> > Congratulations Divij!
> >
> > On Wed, Dec 27, 2023 at 4:20 AM Gaurav Narula  wrote:
> >
> > > Congratulations Divij!
> > >
> > > Regards,
> > > Gaurav
> > >
> > > > On 27-Dec-2023, at 17:44, Mickael Maison 
> > > wrote:
> > > >
> > > > Congratulations Divij!
> > > >
> > > >> On Wed, Dec 27, 2023 at 1:05 PM Sagar 
> > > wrote:
> > > >>
> > > >> Congrats Divij! Absolutely well deserved !
> > > >>
> > > >> Thanks!
> > > >> Sagar.
> > > >>
> > > >>> On Wed, Dec 27, 2023 at 5:15 PM Luke Chen 
> wrote:
> > > >>>
> > > >>> Hi, Everyone,
> > > >>>
> > > >>> Divij has been a Kafka committer since June, 2023. He has remained
> > very
> > > >>> active and instructive in the community since becoming a committer.
> > > It's my
> > > >>> pleasure to announce that Divij is now a member of Kafka PMC.
> > > >>>
> > > >>> Congratulations Divij!
> > > >>>
> > > >>> Luke
> > > >>> on behalf of Apache Kafka PMC
> > > >>>
> > >
> >
>

Re: [VOTE] 3.5.2 RC1

[Tom Bentley]

Hi,

I have validated signatures, checked the Java docs, built from source and
run tests. I had a few unit test failures, but I note that others saw them
pass and the CI was green too, so I think this is a problem with my system
rather than the release.

+1 (binding).

Thanks!

On Wed, 6 Dec 2023 at 12:17, Justine Olshan 
wrote:

> Hey all,
>
> I've built from source, ran unit tests, and ran a produce bench test on a
> running server.
> I've also scanned the various release components. Given the test results
> and the validations, +1 (binding) from me.
>
> Thanks,
> Justine
>
> On Tue, Dec 5, 2023 at 3:59 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > Thanks for helping validate the RC1 build.
> > I've got 1 binding, and 3 non-binding votes.
> > Please help validate it when available.
> >
> > Update for the system test results:
> >
> >
> https://drive.google.com/file/d/1gLt5hTFCVnpoKZ_I5KmUvnowVGtzfip_/view?usp=sharing
> >
> > The result failed at 2 groups of tests:
> > 1. quota_test test suite failed with "ValueError: max() arg is an empty
> > sequence".
> > This is a known issue and these tests can be passed after re-run.
> > 2. zookeeper_migration_test failed with
> >   2.1. "Kafka server didn't finish startup in 60 seconds" : This is
> because
> > we added a constraint to ZK migrating to KRaft that we don't support JBOD
> > in use. These system tests are fixed in this PR in trunk:
> >
> >
> https://github.com/apache/kafka/pull/14654/files#diff-17b8c06d37fe43a3bd6ba5b89e08ff8f988ad5f4e5f7eda87844d51f7e5a5b96R61
> >   2.2. "Zookeeper node failed to start": This is because the ZK is
> pointing
> > to 3.4.0 version, which should be 3.4.1. These system tests are fixed in
> > this PR in trunk:
> >
> >
> https://github.com/apache/kafka/pull/14208/files#diff-17b8c06d37fe43a3bd6ba5b89e08ff8f988ad5f4e5f7eda87844d51f7e5a5b96R143
> >
> > I've confirmed that after applying this patch, the system tests pass now.
> > The PR to backport this fix to 3.5 branch is opened:
> > https://github.com/apache/kafka/pull/14927
> > But that doesn't block 3.5.2 release because they are test problems only.
> >
> > Thank you.
> > Luke
> >
> > On Mon, Nov 27, 2023 at 2:13 AM Mickael Maison  >
> > wrote:
> >
> > > Hi Luke,
> > >
> > > I ran the following checks:
> > > - Verified signatures and checksums
> > > - Ran the KRaft and ZooKeeper quickstarts with the 2.13 binaries
> > > - Built sources and ran unit/integration tests with Java 17
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > On Fri, Nov 24, 2023 at 10:41 AM Jakub Scholz  wrote:
> > > >
> > > > +1 non-binding. I used the staged Scala 2.13 binaries and the staged
> > > Maven
> > > > repo to run my tests and all seems to work fine.
> > > >
> > > > Thanks & Regards
> > > > Jakub
> > > >
> > > > On Tue, Nov 21, 2023 at 11:09 AM Luke Chen 
> wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 3.5.2.
> > > > >
> > > > > This is a bugfix release with several fixes since the release of
> > 3.5.1,
> > > > > including dependency version bumps for CVEs.
> > > > >
> > > > > Release notes for the 3.5.2 release:
> > > > >
> https://home.apache.org/~showuon/kafka-3.5.2-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Nov. 28.
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~showuon/kafka-3.5.2-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~showuon/kafka-3.5.2-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.5 branch) is the 3.5.2 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.5.2-rc1
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/35/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/35/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.5 branch:
> > > > > Unit/integration tests:
> > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/98/
> > > > > There are some falky tests, including the testSingleIP test
> failure.
> > It
> > > > > failed because of some infra change and we fixed it
> > > > >  recently.
> > > > >
> > > > > System tests: running, will update the results later.
> > > > >
> > > > >
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Tom Bentley]

Congratulations!

On Sun, 29 Oct 2023 at 5:41 PM, Guozhang Wang 
wrote:

> Congratulations Satish!
>
> On Sat, Oct 28, 2023 at 12:59 AM Luke Chen  wrote:
> >
> > Congrats Satish!
> >
> > Luke
> >
> > On Sat, Oct 28, 2023 at 11:16 AM ziming deng 
> > wrote:
> >
> > > Congratulations Satish!
> > >
> > > > On Oct 27, 2023, at 23:03, Jun Rao  wrote:
> > > >
> > > > Hi, Everyone,
> > > >
> > > > Satish Duggana has been a Kafka committer since 2022. He has been
> very
> > > > instrumental to the community since becoming a committer. It's my
> > > pleasure
> > > > to announce that Satish is now a member of Kafka PMC.
> > > >
> > > > Congratulations Satish!
> > > >
> > > > Jun
> > > > on behalf of Apache Kafka PMC
> > >
> > >
>
>

Re: [DISCUSS] KIP-987: Connect Static Assignments

[Tom Bentley]

Hi Greg,

Many thanks for the KIP. Here are a few initial questions

1. Incomplete sentence: "But Connect is not situated to be able to manage
resources directly, as workers are given a fixed "
2. You explain how sessioned is now a subset of static, but what happens in
a cluster where some workers are using static and some are using either
eager or compatible?
3. "Assign each unassigned static job to a static worker which specifies
that job, choosing arbitrarily if there are multiple valid workers." I
think there might be less ambiguous words than "arbitrary" to specify this
behaviour. Hashing the task name would _appear_ pretty arbitrary to the
user, but would be deterministic. Picking at random would be
non-deterministic. Even better if you have a rationale.
4. You don't describe how a user, or automated system, starting with a set
of connectors, should find out the tasks that they want to run. This
relates to the contract of
org.apache.kafka.connect.connector.Connector#taskConfigs(int maxTasks).
AFAIK (please correct me if I'm wrong, because it's a long time since I
looked at this code) there's nothing that validates that the returned list
has at most the `maxTasks` and connectors can, of course, return fewer than
that many tasks. So without deeper knowledge of a particular connector it's
not clear to the user/operator how to configure their static workers and
static assignments.
5. Is there a lurking assumption that task indices are stable? E.g. that
the task with index 3 will always be the resource-intensive one. I can see
that that would often be a reliable assumption, but it's not clear to me
that it is always the case.

Thanks again,

Tom

On Fri, 6 Oct 2023 at 12:36, Greg Harris 
wrote:

> Hey everyone!
>
> I'd like to propose an improvement to Kafka Connect's scheduling
> algorithm, with the goal of improving the operability of connect
> clusters through resource isolation.
>
> Find the KIP here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-987%3A+Connect+Static+Assignments
>
> This feature is primarily intended to be consumed by cluster
> management systems, so I've opened a sister proposal in the Strimzi
> project that uses this feature to provide user-facing resource
> isolation. The base feature is generic, so it is usable manually and
> with cluster management systems other than Strimzi.
>
> Find the proposal here: https://github.com/strimzi/proposals/pull/96
>
> Thanks!
> Greg Harris
>
>

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

[Tom Bentley]

Hi Greg,

Thanks for this KIP! It is obviously very ambitious, but it's great to have
a conversation about it.

I'll start with some general points:

Do you have a plan in mind for how to proceed with elaborating this KIP?
While I like how you're involving the community in elaborating the KIP, I
think there is a danger, which is more likely with this inclusive approach,
in trying to attempt too much at once.

In my opinion someone needs to take the difficult decisions necessary to
limit the initial scope (and, just as importantly, communicate that
clearly) in order to maximise the chances of actually getting something
accepted and implemented. Can we assume that you're that person? Defining
the what and how of the metadata replication, and the log replication seem
to me to be the core of what you're trying to achieve here. We should make
anything that is not crucial to that (i.e. NAT punching) a non-goal of this
KIP. Future KIPs can easily add those features.

I also had a few specific points:

Motivation
M1. I don't find the "logical replication" vs "physical replication"
particularly helpful. I think one key property is "offset preserving",
which is also self-explanatory. Slightly more generally, we could define
the concept of "consumer transparency", i.e. a consumer could reconnect to
either cluster and observe the same sequences of records (same order, same
offsets, and same transaction visibility). Consumer transparency requires
synchronous replication, but offset preserving does not.
M2. In the motivation you mention that MM offers asynchronous replication,
but the Goals subsection doesn't mention support for synchronous
replication. We should be clear which (or both) we're aiming for.
M3. A Non-Goals section would be useful, especially for a KIP that's large
and ambitious like this one.
M4. It might also be worth having a list of Assumptions. Here we could list
all the things we want to assume in order to make the initial KIP feasible.
M5. For example we should be explicit about whether or not it is assumed
that the same people are operating (and thus have visibility into) both
clusters.
M6. One thing worth calling out is whether the clusters themselves are in a
leader/follower relationship (e.g. the DR scenario), or whether this is a
topic-level concern. I guess it's topic level from the topic and consumer
group regexes. But this has consequences we should explore. For example
what if a transaction includes records in topics X and Y, where X is
replicated but Y is not?
M7. I think you should be clear about whether this leader/follower
relationship can be reversed, and in what circumstances. In the user
interface section you talk about "disconnected", but not this kind of
fail-back.


User interface
U1. "Links can be temporarily or permanently disconnected." Are you
describing a fact about the network between the two clusters, or is this
disconnection something actively managed by the system, or by the operator?

Data semantics
D1. The KIP says "both cross-cluster topics and intra-cluster replicas:
Have the same configuration as their source" but you also say
"cross-cluster replicas: Have a separate topic-id", this seems like a
contradiction, on the face of it. It seems like there's a whole host of
devils in the detail behind this. It implies replication of (some of) the
__cluster_metadata, I think, but not all (since you say ACLs are not
replicated). If that's right, then what does it imply about referential
integrity between metadata records? i.e. what if metadata record A (which
is replicated) references record B (which is not)? Even if this is not
possible by design initially, how does it constrain the future evolution of
metadata record schemas? Is any such metadata replication going to be
transaction preserving? If the topic ids can differ then what is
responsible for the mapping and rewriting of metadata records which include
topic ids?
D2. "The network path between Kafka clusters is assumed to be less reliable
than the intra-cluster network," we should be explicit about whether or not
we're assuming similar network latencies and bandwidth for the
inter-cluster network links as for the in-cluster ones.
D3 "Are not eligible for fetch-from-follower on the source cluster" the
reason for this isn't immediately apparent to me.

Thanks again,

Tom

On Tue, 3 Oct 2023 at 09:37, Greg Harris 
wrote:

> Hi all,
>
> I've opened an extremely early draft of the Cross-Cluster Replication
> feature, and I would like to invite any and all co-authors to expand
> on it. Find the page here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication
>
> This is not strictly an invitation to "review" the KIP, as the
> document has much less detail than other KIPs of similar complexity.
> But if you are knowledgeable in this area, some early sanity checks
> would be greatly appreciated.
>
> I've included a "shopping list" of properties that appear to me to be
> desirable, but I 

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[Tom Bentley]

Congratulations!

On Sun, 24 Sept 2023 at 12:32, Satish Duggana 
wrote:

> Congratulations Justine!!
>
> On Sat, 23 Sept 2023 at 15:46, Bill Bejeck  wrote:
> >
> > Congrats Justine!
> >
> > -Bill
> >
> > On Sat, Sep 23, 2023 at 6:23 PM Greg Harris  >
> > wrote:
> >
> > > Congratulations Justine!
> > >
> > > On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
> > >  wrote:
> > > >
> > > > Congrats Justin !
> > > >
> > > > On Sat 23 Sep 2023 at 14:44, Randall Hauch  wrote:
> > > >
> > > > > Congratulations, Justine!
> > > > >
> > > > > On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
> > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > >
> > > > > > Congrats Justine!
> > > > > >
> > > > > > On Sat, Sep 23, 2023, 13:28 Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations Justine!
> > > > > > >
> > > > > > > On Sat 23. Sep 2023 at 07:06, Chris Egerton <
> > > fearthecel...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congrats Justine!
> > > > > > > > On Fri, Sep 22, 2023, 20:47 Guozhang Wang <
> > > > > guozhang.wang...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations!
> > > > > > > > >
> > > > > > > > > On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <
> > > > > > > tzuli...@apache.org
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Congratulations Justine!
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 22, 2023, 19:25 Philip Nee <
> philip...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congrats Justine!
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <
> > > show...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > Justine Olshan has been a Kafka committer since Dec.
> > > 2022.
> > > > > She
> > > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > very active and instrumental to the community since
> > > becoming
> > > > > a
> > > > > > > > > committer.
> > > > > > > > > > > > It's my pleasure to announce that Justine is now a
> > > member of
> > > > > > > Kafka
> > > > > > > > > PMC.
> > > > > > > > > > > >
> > > > > > > > > > > > Congratulations Justine!
> > > > > > > > > > > >
> > > > > > > > > > > > Luke
> > > > > > > > > > > > on behalf of Apache Kafka PMC
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>
>

Re: [ANNOUNCE] New committer: Yash Mayya

[Tom Bentley]

Congratulations!

On Fri, 22 Sept 2023 at 09:11, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Congratulations, Yash!
>
> On Thu 21. Sep 2023 at 21.57, Randall Hauch  wrote:
>
> > Congratulations, Yash!
> >
> > On Thu, Sep 21, 2023 at 12:31 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> >
> > > Congratulations Yash!!
> > >
> > > On Thu, 21 Sept 2023 at 22:57, Viktor Somogyi-Vass
> > >  wrote:
> > > >
> > > > Congrats Yash!
> > > >
> > > > On Thu, Sep 21, 2023 at 7:04 PM Josep Prat
>  > >
> > > > wrote:
> > > >
> > > > > Congrats Yash!
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Thu, Sep 21, 2023, 18:55 Raymond Ng 
> > > wrote:
> > > > >
> > > > > > Congrats Yash! Well-deserved!
> > > > > >
> > > > > > /Ray
> > > > > >
> > > > > > On Thu, Sep 21, 2023 at 9:40 AM Kamal Chandraprakash <
> > > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > >
> > > > > > > Congratulations Yash!
> > > > > > >
> > > > > > > On Thu, Sep 21, 2023, 22:03 Bill Bejeck 
> > > wrote:
> > > > > > >
> > > > > > > > Congrats Yash!
> > > > > > > >
> > > > > > > > On Thu, Sep 21, 2023 at 12:26 PM Divij Vaidya <
> > > > > divijvaidy...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Yash!
> > > > > > > > >
> > > > > > > > > Divij Vaidya
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 21, 2023 at 6:18 PM Sagar <
> > > sagarmeansoc...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Congrats Yash !
> > > > > > > > > > On Thu, 21 Sep 2023 at 9:38 PM, Ashwin
> > > > > >  > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Awesome ! Congratulations Yash !!
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 21, 2023 at 9:25 PM Edoardo Comar <
> > > > > > > edoardli...@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations Yash
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, 21 Sept 2023 at 16:28, Bruno Cadonna <
> > > > > > cado...@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The PMC of Apache Kafka is pleased to announce a
> new
> > > Kafka
> > > > > > > > > committer
> > > > > > > > > > > > > Yash Mayya.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yash's major contributions are around Connect.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yash authored the following KIPs:
> > > > > > > > > > > > >
> > > > > > > > > > > > > KIP-793: Allow sink connectors to be used with
> > > > > topic-mutating
> > > > > > > > SMTs
> > > > > > > > > > > > > KIP-882: Kafka Connect REST API configuration
> > > validation
> > > > > > > timeout
> > > > > > > > > > > > > improvements
> > > > > > > > > > > > > KIP-970: Deprecate and remove Connect's redundant
> > task
> > > > > > > > > configurations
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > KIP-980: Allow creating connectors in a stopped
> state
> > > > > > > > > > > > >
> > > > > > > > > > > > > Overall, Yash is known for insightful and friendly
> > > input to
> > > > > > > > > discussions
> > > > > > > > > > > > > and his high quality contributions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Yash!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Bruno (on behalf of the Apache Kafka PMC)
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Lucas Brutschy

[Tom Bentley]

Congratulations!

On Fri, 22 Sept 2023 at 09:11, Sophie Blee-Goldman 
wrote:

> Congrats Lucas!
>

Re: [DISCUSS] KIP-714: Client metrics and observability

[Tom Bentley]

Hi Andrew,

Thanks for picking this KIP up. I know you've started a vote, so these are
unhelpfully late... sorry about that, but hopefully they're still useful.

1. "The Kafka Java client provides an API for the application to read the
generated client instance id to assist in mapping and identification of the
client based on the collected metrics." In the multi-client, single-process
model perhaps it would be desirable to have the option of including this in
log messages emitted by the client library.

2. "Mapping the client instance id to an actual application instance
running on a (virtual) machine can be done by inspecting the metrics
resource labels, such as the client source address and source port, or
security principal, all of which are added by the receiving broker." The
specific example of correlation given here (source IP address) is
problematic in environments where there may be network proxies (e.g.
Kubernetes ingress) on the path between client and broker: The broker sees
the IP address of the proxy. This is a rough edge which could be smoothed
out if Kafka supported the PROXY protocol[1] which seems to have become
something of a defacto standard. I'm not suggesting this need to be part of
the KIP, but perhaps it could be added to Future Work?
[1]: http://www.haproxy.org/download/2.9/doc/proxy-protocol.txt

3. Compression... just an idle idea, but I wonder if a useful further
improvement in compression ratio could be achieve using zstd's support for
dictionary compression[2]. I.e. a client could initially use training mode
when sending metrics, but eventually include a dictionary to be used for
subsequent metric sends. It's probably not worth the effort (at least for
the initial implementation), but since you've gone to the effort of
providing some numbers anyway, maybe it's not much additional effort to at
least find out whether this makes a useful difference.
[2]: http://facebook.github.io/zstd/#small-data

4. Maybe I didn't spot it, but I assume the initial
GetTelemetrySubscriptionsRequest
happens after authentication?

5. Rogue clients -- There are some interesting things to consider if we're
trying to defend against a genuinely adversarial client.

a) Client sends profiling information to all brokers at the maximum rate.
Each broker forwards to the time series DB. Obviously this scales linearly
with number of brokers, but it's clear that the load on the tsdb could be
many times larger than users might expect.
b) Client sends crafted compressed data which decompresses to require more
memory that the broker can allocate.

6. Shadowing the OLTP and protobuf jars -- to be clear by this you mean
both bundling _and_ relocating?

7. "In case the cluster load induced from metrics requests becomes
unmanageable the remedy is to temporarily remove or limit configured
metrics subscriptions.  " How would a user know that the observed load was
due to handling metrics requests?

8. If I understand correctly, when the configured metrics reporter does not
implement the new interface the client would still follow the described
protocol only to have nowhere to send the metrics. Am I overlooking
something?

Thanks again,

Tom

On Fri, 11 Aug 2023 at 07:52, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Doguscan,
> Thanks for your question.
>
> If the target broker is unreachable, the client can send the metrics to
> another
> broker. It can select any of the other brokers for this purpose. What I
> expect in
> practice is that it loses connection to the broker it’s being using for
> metrics,
> chooses or establishes a connection to another broker, and then selects
> that
> broker for subsequent metrics pushes.
>
> Thanks,
> Andrew
>
> > On 8 Aug 2023, at 08:34, Doğuşcan Namal 
> wrote:
> >
> > Thanks for your answers Andrew. I share your pain that it took a while
> for
> > you to get this KIP approved and you want to reduce the scope of it, will
> > be happy to help you with the implementation :)
> >
> > Could you help me walk through what happens if the target broker is
> > unreachable? Is the client going to drop these metrics or is it going to
> > send it to the other brokers it is connected to? This information is
> > crucial to understand the client side impact on leadership failovers.
> > Moreover, in case of partial outages, such as only the network between
> the
> > client and the broker is partitioned whereas the network within the
> cluster
> > is healthy, practically there is no other way than the client side
> metrics
> > to identify this problem.
> >
> > Doguscan
> >
> > On Fri, 4 Aug 2023 at 15:33, Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Doguscan,
> >> Thanks for your comments. I’m glad to hear you’re interested in this
> KIP.
> >>
> >> 1) It is preferred that a client sends its metrics to the same broker
> >> connection
> >> but actually it is able to send them to any broker. As a result, if a
> >> broker becomes
> >> unhealthy, the 

Re: [ANNOUNCE] New committer: Greg Harris

[Tom Bentley]

Congratulations!

On Mon, 10 Jul 2023 at 17:32,  wrote:

> Congrats Greg!
>
> -
> Gaurav
>
> > On 10 Jul 2023, at 17:25, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> hgerald...@bloomberg.net> wrote:
> >
> > Congrats Greg! Well deserved
> >
> > From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:
> dev@kafka.apache.org
> > Subject: Re: [ANNOUNCE] New committer: Greg Harris
> >
> > Congratulations!
> >
> > On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch  wrote:
> >>
> >> Congratulations, Greg.
> >>
> >> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >>> Congratulations Greg!
> >>>
> >>> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck  >
> >>> wrote:
> 
>  Congrats Greg!
> 
>  -Bill
> 
>  On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya <
> divijvaidy...@gmail.com>
>  wrote:
> 
> > Congratulations Greg! I am going through a new committer teething
> >>> process
> > right now and would be happy to get you up to speed. Looking forward
> to
> > working with you in your new role.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat
>  
> > wrote:
> >
> >> Congrats Greg!
> >>
> >>
> >> ———
> >> Josep Prat
> >>
> >> Aiven Deutschland GmbH
> >>
> >> Alexanderufer 3-7, 10117 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> m: +491715557497
> >>
> >> w: aiven.io
> >>
> >> e: josep.p...@aiven.io
> >>
> >> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> >>> wrote:
> >>
> >>> Congrats!
> >>>
> >>> On 7/10/23 8:45 AM, Chris Egerton wrote:
>  Hi all,
> 
>  The PMC for Apache Kafka has invited Greg Harris to become a
> > committer,
> >>> and
>  we are happy to announce that he has accepted!
> 
>  Greg has been contributing to Kafka since 2019. He has made over
> >>> 50
> >>> commits
>  mostly around Kafka Connect and Mirror Maker 2. His most notable
>  contributions include KIP-898: "Modernize Connect plugin
> >>> discovery"
> >> and a
>  deep overhaul of the offset syncing logic in MM2 that addressed
> > several
>  technically-difficult, long-standing, high-impact issues.
> 
>  He has also been an active participant in discussions and
> >>> reviews on
> >> the
>  mailing lists and on GitHub.
> 
>  Thanks for all of your contributions, Greg. Congratulations!
> 
> >>>
> >>
> >
> >>>
> >
> >
>
>

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Tom Bentley]

Hi Colin,

Thanks for the KIP.

1. It mentions kafka-configs.sh as one of the affected tools, but doesn't
mention that ControllerApis doesn't currently support DESCRIBE_CONFIGS. I
think this is worth noting as it is, in effect, a change to the wire
protocol accepted by the controller, even if it's an existing RPC.
2. The diff you show for the MetadataRequest.son doesn't show a change to
the top-level "listeners" key, presumably this should add "controller"?
Similarly, per the above point, I guess we'd also be updating the JSON for
DescribeConfigs.
3. Do you have any timeline for calling a vote for this KIP?

Many thanks,

Tom

On Thu, 27 Apr 2023 at 18:51, Colin McCabe  wrote:

> On Wed, Apr 26, 2023, at 22:08, Luke Chen wrote:
> > Hi Colin,
> >
> > Some comments:
> > 1. I agree we should set "top-level" errors for metadata response
> >
> > 2. In the "brokers" field of metadata response from controller, it'll
> > respond with "Controller endpoint information as given in
> > controller.quorum.voters", instead of the "alive" controllers(voters).
> That
> > will break the existing admin client because in admin client, we'll rely
> on
> > the metadata response to build the "current alive brokers" list, and
> choose
> > one from them to connect (either least load or other criteria). That
> means,
> > if now, we return the value in `controller.quorum.voters`, but one of
> them
> > is down. We might choose it to connect and get connection errors. Should
> we
> > return the "alive" controllers(voters) to client?
>
> Hi Luke,
>
> Good question. When talking to the controllers directly, the AdminClient
> needs to always send its RPCs to the active controller. There is one
> exception: configuring ephemeral log4j settings with
> incrementalAlterConfigs must be done by sending them to the specified
> controller node.
>
> I will add this to a section called "AdminClient Implementation Notes" so
> that it's captured in the KIP.
>
> >
> > 3. In the KIP, we list the command-line tools will get a new
> > --bootstrap-controllers argument, but without explaining why these tools
> > need to talk to controller directly. Could we add some explanation about
> > them? I tried but cannot know why some tools are listed here:
> > - kafka-acls.sh -> Allow clients to update ACLs via controller before
> > brokers up
> >
> > - kafka-cluster.sh -> Reasonable to get/update cluster info via
> > controller
> >
> > - kafka-configs.sh -> Allow clients to dynamically update
> > configs/describe configs from controller. But in this script, client can
> > still set quota for users/clients/topics... is client also able to update
> > via controllers? Or we only allow partial actions in the script to talk
> to
> > controllers?
> >
> > - kafka-delegation-tokens.sh -> Reasonable to update
> delegation-tokens
> > via controllers
> >
> > - kafka-features.sh -> Reasonable
> > - kafka-metadata-quorum.sh -> Reasonable
> > - kafka-metadata-shell.sh -> Reasonable
> >
> > - kafka-reassign-partitions.sh -> Why should we allow clients to move
> > metadata log partitions in controller nodes? What's the use-case?
> >
>
> Yes, the common thread here is that all of these shell commands perform
> operations can be done without the broker. So it's reasonable to allow them
> to be done without going through the broker. I don't know if we need a
> separate note for each since the rationale is really the same for all (is
> it reasonable? if so allow it.)
>
> kafka-reassign-partitions.sh cannot be used to move the __cluster_metadata
> topic. However, it can be used to move partitions that reside on the
> brokers, even when using --bootstrap-controllers to talk directly to the
> quorum.
>
> Colin
>
> >
> > Thank you.
> > Luke
> >
> > On Thu, Apr 27, 2023 at 8:04 AM Colin McCabe  wrote:
> >
> >> On Tue, Apr 25, 2023, at 04:59, Divij Vaidya wrote:
> >> > Thank you for the KIP Colin.
> >> >
> >> > In general, I like the idea of having the ability to interact directly
> >> with
> >> > the controllers. I agree with your observation that it helps in
> >> situations
> >> > where you would want to get data directly from the controller instead
> of
> >> > going via a broker. I have some general comments but the main concern
> I
> >> > have is with the piggy-backing of error code with response of
> >> > __cluster_metadata topic.
> >> >
> >> > 1. With this change, how are we guarding against the possibility of
> >> > misbehaving client traffic from disrupting the controller (that you
> >> > mentioned as a motivation of earlier behaviour)? One solution could
> be to
> >> > have default values set for request throttling on the controller.
> >>
> >> Hi Divij,
> >>
> >> Thanks for the comments.
> >>
> >> Our guards against client misbehavior remain the same:
> >> 1. our recommendation to put the clients on a separate network
> >> 2. the fact that producers and consumers can't interact directly with
> the
> >> controller
> >> 3. the authorizer.
> >>
> >> 

Re: [DISCUSS] Apache Kafka 3.5.1 release

[Tom Bentley]

SGTM, thanks Divij

On Thu, 29 Jun 2023 at 03:16, Luke Chen  wrote:

> Hi Divij,
>
> Thanks for volunteering!
>
> Luke
>
> On Wed, Jun 28, 2023 at 11:54 PM Manyanda Chitimbo <
> manyanda.chiti...@gmail.com> wrote:
>
> > Thank you Divij for volunteering to perform the release.
> >
> > On Wed 28 Jun 2023 at 13:52, Divij Vaidya 
> wrote:
> >
> > > Hey folks
> > >
> > > Looks like we are ready to perform a release for 3.5.1 to provide a fix
> > for
> > > the vulnerability in snappy-java [1]
> > >
> > > I would like to volunteer as release manager for the 3.5.1 release.
> > >
> > > If there are no objections, I will start a release plan next Monday, on
> > 3rd
> > > July.
> > >
> > > [1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455
> > >
> > > --
> > > Divij Vaidya
> > >
> > --
> > Manyanda Chitimbo.
> >
>

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

[Tom Bentley]

Hi Edorado and Adrian,

Thanks for the KIP.

I think it would be good to elaborate on exactly how validate() gets
called, because I think there are a number of potential problems, or at
least things to consider.

>From the broker's point of view, validate() can do arbitrary things. It
might never return due to blocking or an infinite loop. It might cause an
OOME, or throw a StackOverflowException. These are not entirely unlikely
and the risks cannot always be easily avoided by a careful policy
implementation. For example, a plugin which performs schema validation
would typically be fetching schema information from a remote registry (and
caching it for future use), and so could block on the networking (there are
further questions about retry in the case of network error). Then, when
deserializing a format like JSON deserializers might be prone to SOE or
OOME (e.g. consider a naive recursive JSON parser with JSON input starting
"..."). More generally, incorrect
deserialization of untrusted input is a common kind of CVE. Similarly
validation might involve regular expression matching (for example
JSONSchema supports pattern constraints). The matcher implementation really
matters and common matchers, including Java's Pattern, expose you to the
possibility of nasty exponential time behaviour.

You mentioned LogValidator in the KIP. This executes on an IO thread and
gets called with the log lock held. So the consequences of the validation
blocking could actually be a real problem from a broker availability PoV if
this validation happens in the same place. In the worst case all the IO
threads get stuck because of bad input (perhaps from a single producer), or
network problems between the broker and the registry. I don't think simply
moving the validation to before the acquisition of the lock is an easy
solution either, because of the dependency on the compression validation.

Kind regards,

Tom

On Thu, 22 Jun 2023 at 04:16, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi Eduardo, Adrian.
>
> Thanks for the KIP. I agree that allowing custom validations on the
> broker-side addresses a real gap as you clearly stated on the motivation.
>
> Some initial thoughts from my side:
>
> 1. Similar to Kirk's first point, I'm also concerned on how would the
> plugin developers / operators be able to apply multiple policies and how
> configurations would be passed to each policy.
>
> Some approaches from other plugins we can get some inspiration from:
>
> - AlterConfig, CreateTopic policies are designed to be 1 policy
> implementing the different APIs. Up to the plugin developer to pull
> policies together and configure it on the broker side. I guess for Record
> Validation this may be cumbersome considering some Schema Registry
> providers may want to offer implementations for their own backend.
>
> - Connect Transforms: here there's a named set of plugins to apply per
> connector, and each transform has its own configuration defined by prefix.
> Personally, I'd consider this one an interesting approach if we decide to
> allow multiple record validations to be configured.
>
> - Tiered Storage (probably Connectors as well) have class-loader aware
> implementations with class path specific to the plugin. Not sure if this is
> something to discuss on the KIP or later on the PR, but we could mention
> something on how this plugin would deal with dependency conflicts (e.g.
> different jackson version between broker, plugin(s)).
>
> Also, by potentially supporting multiple plugins for record validation, it
> would be important to consider if it's an all or nothing relation, or
> posible to choose _some_ policies apply per topic.
> I see there's some preference for setting the validation policy name on the
> topic, though this could be cumbersome to operate: topic creation users may
> not be aware of the record validation (similar to CreateTopic/AlterConfig
> policies) and would impose additional coordination.
> Maybe a flag to whether apply policies or not would be a better approach?
>
> 2. Have you consider adding the record metadata to the API? It may be
> useful for logging purposes (e.g. if record validation fails, to log
> topic-partition), or some policies are interested on record metadata (e.g.
> compression, timestamp type, etc.)
>
> 3. A minor comment for consistency regarding the APIs. As far as I have
> seen, we tend to use the name of the object returned directly instead of
> getters notation, see `AlterConfigPolicy.RecordMetadata` [1]. We may rename
> some of the proposed APIs accordingly:
>
> - `RecordProxy#headers()|key()|value()`
> - `TopicMetadata#topicPartition()`
>
> 4. For the `RecordIntrospectionHints`, I'm struggling to see how this may
> be used by the policy developers. Would you mind adding some examples on
> how the policy in general may be used?
> Specifically, `long needKeyBytes|needKeyValue` are difficult to interpret
> to me.
> nit: maybe replace 

Re: [ANNOUNCE] Apache Kafka 3.5.0

[Tom Bentley]

Thanks Mickael for the release, and thanks too to all our contributors.

On Thu, 15 Jun 2023 at 16:22, Chia-Ping Tsai  wrote:

> This is a great release. Mickael!!!
>
> > Chris Egerton  於 2023年6月15日 下午11:09 寫道：
> >
> > Mickael
>
>

Re: [ANNOUNCE] New committer: Divij Vaidya

[Tom Bentley]

Congratulations Divij!

On Tue, 13 Jun 2023 at 17:21, Mickael Maison 
wrote:

> Congratulations!
>
> On Tue, Jun 13, 2023 at 6:05 PM Yash Mayya  wrote:
> >
> > Congratulations Divij!
> >
> > On Tue, Jun 13, 2023 at 9:20 PM Bruno Cadonna 
> wrote:
> >
> > > Hi all,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Divij Vaidya.
> > >
> > > Divij's major contributions are:
> > >
> > > GDPR compliance enforcement of kafka-site -
> > > https://issues.apache.org/jira/browse/KAFKA-13868
> > >
> > > Performance improvements:
> > >
> > > Improve performance of VarInt encoding and decoding -
> > > https://github.com/apache/kafka/pull/13312
> > >
> > > Reduce data copy & buffer allocation during decompression -
> > > https://github.com/apache/kafka/pull/13135
> > >
> > > He also was heavily involved in the migration to Mockito.
> > >
> > > Furthermore, Divij is very active on the mailing lists as well as in
> > > maintaining and reviewing pull requests.
> > >
> > > Congratulations, Divij!
> > >
> > > Thanks,
> > >
> > > Bruno (on behalf of the Apache Kafka PMC)
> > >
> > >
> > >
>
>

Re: [ANNOUNCE] Apache Kafka 3.4.1

[Tom Bentley]

Thanks Luke!

On Wed, 7 Jun 2023 at 09:11, Mickael Maison 
wrote:

> Thanks for running the release!
>
> On Wed, Jun 7, 2023 at 9:11 AM Bruno Cadonna  wrote:
> >
> > Thanks Luke!
> >
> > On 07.06.23 07:55, Federico Valeri wrote:
> > > Thanks Luke!
> > >
> > > On Wed, Jun 7, 2023 at 5:56 AM Kamal Chandraprakash
> > >  wrote:
> > >>
> > >> Thanks Luke for running this release!
> > >>
> > >> On Wed, Jun 7, 2023 at 8:08 AM Chia-Ping Tsai 
> wrote:
> > >>
> > >>> Thank Luke for this hard work!!!
> > >>>
> >  Chris Egerton  於 2023年6月7日 上午10:35 寫道：
> > 
> >  Thanks for running this release, Luke!
> > 
> >  On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:
> > 
> > > The Apache Kafka community is pleased to announce the release for
> > > Apache Kafka 3.4.1.
> > >
> > > This is a bug fix release and it includes fixes and improvements
> from
> > > 58 JIRAs, including a few critical bugs:
> > > - core
> > > KAFKA-14644 Process should stop after failure in raft IO thread
> > > KAFKA-14946 KRaft controller node shutting down while renouncing
> > >>> leadership
> > > KAFKA-14887 ZK session timeout can cause broker to shutdown
> > > - client
> > > KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns
> partition
> > > in one rebalance cycle
> > > - connect
> > > KAFKA-12558 MM2 may not sync partition offsets correctly
> > > KAFKA-14666 MM2 should translate consumer group offsets behind
> > >>> replication
> > > flow
> > > - stream
> > > KAFKA-14172 bug: State stores lose state when tasks are reassigned
> under
> > > EOS
> > >
> > > All of the changes in this release can be found in the release
> notes:
> > >
> > > https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
> > >
> > > You can download the source and binary release (Scala 2.12 and
> Scala
> > >>> 2.13)
> > > from:
> > >
> > > https://kafka.apache.org/downloads#3.4.1
> > >
> > >
> > >
> > >>>
> ---
> > >
> > > Apache Kafka is a distributed streaming platform with four core
> APIs:
> > >
> > > ** The Producer API allows an application to publish a stream
> records
> > > to one or more Kafka topics.
> > >
> > > ** The Consumer API allows an application to subscribe to one or
> more
> > > topics and process the stream of records produced to them.
> > >
> > > ** The Streams API allows an application to act as a stream
> processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively
> transforming
> > > the input streams to output streams.
> > >
> > > ** The Connector API allows building and running reusable
> producers or
> > > consumers that connect Kafka topics to existing applications or
> data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.
> > >
> > >
> > > With these APIs, Kafka can be used for two broad classes of
> application:
> > >
> > > ** Building real-time streaming data pipelines that reliably get
> data
> > > between systems or applications.
> > >
> > > ** Building real-time streaming applications that transform or
> react
> > > to the streams of data.
> > >
> > > Apache Kafka is in use at large and small companies worldwide,
> > > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > > Zalando, among others.
> > >
> > > A big thank you for the following 32 contributors to this release!
> > >
> > > atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> > > csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
> > > emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector
> Geraldino,
> > > hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya,
> José
> > > Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> > > Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan,
> Rajini
> > > Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass,
> Yash
> > > Mayya
> > >
> > > We welcome your help and feedback. For more information on how to
> > > report problems, and to get involved, visit the project website at
> > > https://kafka.apache.org/
> > >
> > >
> > > Thank you!
> > >
> > > Regards,
> > > Luke
> > >
> > >>>
> > >>>
>
>

[jira] [Created] (KAFKA-15050) Prompts in the quickstarts

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-15050:
---

 Summary: Prompts in the quickstarts
 Key: KAFKA-15050
 URL: https://issues.apache.org/jira/browse/KAFKA-15050
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: Tom Bentley


In the quickstarts [Steps 
1-5|https://kafka.apache.org/documentation/#quickstart] use {{$}} to indicate 
the command prompt. When we start to use Kafka Connect in [Step 
6|https://kafka.apache.org/documentation/#quickstart_kafkaconnect] we switch to 
{{{}>{}}}. The [Kafka Streams 
quickstart|https://kafka.apache.org/documentation/streams/quickstart] also uses 
{{{}>{}}}. I don't think there's a reason for this, but if there is one (root 
vs user account?) it should be explained.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.4.1 RC3

[Tom Bentley]

Hi Luke,

Thanks for running the release.

I've checked signatures, eyeballed the Javadocs, built from source and run
the unit and integration tests.
DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate fails for me
repeatedly. I opened https://issues.apache.org/jira/browse/KAFKA-15049 for
it since I couldn't find an existing issue for this one. I note that others
seem to have run the integration tests without problems, so I don't think
this is a blocker. I also did the Kafka, Connect and Streams quickstarts.

+1 binding.

Tom



On Thu, 1 Jun 2023 at 08:46, Luke Chen  wrote:

> Hi all,
>
> Thanks to everyone who has tested and voted for the RC3 so far!
> Currently, I've got 2 binding votes and 3 non-binding votes:
>
> Binding +1 PMC votes:
> * Chris Egerton
> * Mickael Maison
>
> Non-binding votes:
> * Federico Valeri
> * Jakub Scholz
> * Josep Prat
>
> If anyone is available (especially PMC members :)), please help verify the
> RC build.
>
> Thank you.
> Luke
>
> On Wed, May 31, 2023 at 1:53 AM Chris Egerton 
> wrote:
>
> > Hi Luke,
> >
> > Many thanks for your continued work on this release!
> >
> > To verify, I:
> > - Built from source using Java 11 with both:
> > - - the 3.4.1-rc3 tag on GitHub
> > - - the kafka-3.4.1-src.tgz artifact from
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> > - Checked signatures and checksums
> > - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/ with Java 11 and Scala
> > 13
> > in KRaft mode
> > - Ran all unit tests
> > - Ran all integration tests for Connect and MM2
> >
> > +1 (binding)
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, May 30, 2023 at 11:16 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Luke,
> > >
> > > I built from source with Java 11 and Scala 2.13 and ran the unit and
> > > integration tests. It took a few retries to get some of them to pass.
> > > I verified signatures and hashes and also ran the zookeeper quickstart.
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Sat, May 27, 2023 at 12:58 PM Jakub Scholz  wrote:
> > > >
> > > > +1 (non-binding) ... I used the staged binaries and Maven artifacts
> to
> > > run
> > > > my tests and all seems to work fine.
> > > >
> > > > Thanks for running the release.
> > > >
> > > > Jakub
> > > >
> > > > On Fri, May 26, 2023 at 9:34 AM Luke Chen  wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the 4th candidate for release of Apache Kafka 3.4.1.
> > > > >
> > > > > This is a bugfix release with several fixes since the release of
> > > 3.4.0. A
> > > > > few of the major issues include:
> > > > > - core
> > > > > KAFKA-14644 
> > > Process
> > > > > should stop after failure in raft IO thread
> > > > > KAFKA-14946 
> > KRaft
> > > > > controller node shutting down while renouncing leadership
> > > > > KAFKA-14887  ZK
> > > session
> > > > > timeout can cause broker to shutdown
> > > > > - client
> > > > > KAFKA-14639 
> > Kafka
> > > > > CooperativeStickyAssignor revokes/assigns partition in one
> rebalance
> > > cycle
> > > > > - connect
> > > > > KAFKA-12558 
> MM2
> > > may
> > > > > not
> > > > > sync partition offsets correctly
> > > > > KAFKA-14666 
> MM2
> > > should
> > > > > translate consumer group offsets behind replication flow
> > > > > - stream
> > > > > KAFKA-14172 
> bug:
> > > State
> > > > > stores lose state when tasks are reassigned under EOS
> > > > >
> > > > >
> > > > > Release notes for the 3.4.1 release:
> > > > >
> https://home.apache.org/~showuon/kafka-3.4.1-rc3/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Jun 2, 2023
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.4.1-rc3
> > > > >
> > > > > * Documentation: (will be updated after released)
> > > > > https://kafka.apache.org/34/documentation.html
> > > > >
> > > > > * Protocol: (will be updated after released)
> > > > > https://kafka.apache.org/34/protocol.html
> > > > >
> > > > > The 

[jira] [Created] (KAFKA-15049) Flaky test DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-15049:
---

 Summary: Flaky test 
DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate
 Key: KAFKA-15049
 URL: https://issues.apache.org/jira/browse/KAFKA-15049
 Project: Kafka
  Issue Type: Bug
Reporter: Tom Bentley


While testing 3.4.1RC3 
DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate failed repeatedly 
on my machine always with the following stacktrace.

{{org.opentest4j.AssertionFailedError: Unexpected exception type thrown, 
expected:  but was: 

 at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
 at app//org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:67)
 at app//org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:35)
 at app//org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3083)
 at 
app//kafka.server.DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate(DynamicBrokerReconfigurationTest.scala:1066)}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Tom Bentley]

Congratulations Mickael and thanks again Jun.

On Mon, 24 Apr 2023 at 19:31, Roman Schmitz  wrote:

> Congratulations Mickael!
>
>
> Am Mo., 24. Apr. 2023 um 19:26 Uhr schrieb José Armando García Sancio
> :
>
> > Congratulations Mickael and thank you Jun for performing this role for
> > the past 10 years!
> >
> > On Mon, Apr 24, 2023 at 10:15 AM Yash Mayya 
> wrote:
> > >
> > > Congratulations Mickael!
> > >
> > > On Fri, Apr 21, 2023 at 8:39 PM Jun Rao 
> > wrote:
> > >
> > > > Hi, everyone,
> > > >
> > > > After more than 10 years, I am stepping down as the PMC chair of
> Apache
> > > > Kafka. We now have a new chair Mickael Maison, who has been a PMC
> > member
> > > > since 2020. I plan to continue to contribute to Apache Kafka myself.
> > > >
> > > > Congratulations, Mickael!
> > > >
> > > > Jun
> > > >
> >
> >
> >
> > --
> > -José
> >
>

Re: [VOTE] KIP-906 Tools migration guidelines

[Tom Bentley]

Hi Federico,

Thanks for the KIP, +1 (binding).

Tom

On Tue, 4 Apr 2023 at 08:30, Federico Valeri  wrote:

> Hi, we have 2 binding an 1 non binding votes. Thanks for tacking the
> time to review the KIP. Can I get one more binding vote?
>
> Cheers
> Fede
>
> On Fri, Mar 24, 2023 at 3:05 PM John Roesler  wrote:
> >
> > Thanks Federico,
> >
> > I'm +1 (binding)
> >
> > -John
> >
> > On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> > > Bumping this thread for more votes.
> > >
> > > Thanks.
> > >
> > > On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez <
> alexandre.dupr...@gmail.com>
> > > wrote:
> > >
> > >> Hi, Frederico,
> > >>
> > >> Thanks for the KIP.
> > >>
> > >> Non-binding +1.
> > >>
> > >> Thanks,
> > >> Alexandre
> > >>
> > >> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> > >> >
> > >> > +1 from me.
> > >> >
> > >> > Luke
> > >> >
> > >> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri <
> fedeval...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi everyone,
> > >> > >
> > >> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> > >> > >
> > >> > >
> > >> > >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > >> > >
> > >> > > Discussion thread:
> > >> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> > >> > >
> > >> > > Thanks
> > >> > > Fede
> > >> > >
> > >>
>
>

Re: [VOTE] KIP-581: Value of optional null field which has default value

[Tom Bentley]

+1 (binding), thanks again Mickael.

On Fri, 17 Mar 2023 at 14:56, Chris Egerton  wrote:

> +1 (binding), thanks Mickael!
>
> On Fri, Mar 17, 2023, 10:11 Mickael Maison 
> wrote:
>
> > Hi all,
> >
> > I'd like to start a vote on KIP-581 to allow keeping null values
> > instead of always replacing with the default values:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> >
> > Thanks,
> > Mickael
> >
>

Re: [VOTE] KIP-911: Add source tag to MirrorSourceConnector metrics

[Tom Bentley]

+1 (binding). Thanks Mickael!

On Thu, 16 Mar 2023 at 12:55, Chris Egerton  wrote:

> +1 (binding). Thanks for the KIP!
>
> On Wed, Mar 15, 2023 at 7:08 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I'd like to start the vote on KIP-911 to add the source cluster alias
> > as a tag on the MirrorSourceConnector metrics:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-911%3A+Add+source+tag+to+MirrorSourceConnector+metrics
> >
> > Thanks,
> > Mickael
> >
>

Re: [Discuss] KIP-581: Value of optional null field which has default value

[Tom Bentley]

Hi Mickael,

Thanks to Cheng for the KIP and to you for picking it up.

My only comment (feel free to ignore) is about the names of the configs.
Personally I don't think I'd correctly guess what
"serialize.use.optional.null" meant. Something like
"serialize.map.null.to.default" is much clearer to me, for the cost of one
extra token.

Otherwise LGTM.

Thanks,

Tom

On Wed, 8 Mar 2023 at 15:55, Mickael Maison 
wrote:

> Hi,
>
> This KIP has been staled for a long time. Since it would be a useful
> feature, I pinged Cheng about a month ago asking if he was planning to
> work on it again. I've not received a reply, so I've allowed myself to
> update the KIP (hopefully preserving the initial requirements) and
> would like to restart a discussion.
>
> The DISCUSS thread was split in two, you can find the other part in
> https://lists.apache.org/thread/dc56k17zptzvbyc7gtscovzgzwf6yn9p
>
> Let me know if you have any feedback.
>
> Thanks,
> Mickael
>
> On Tue, Apr 14, 2020 at 8:28 PM Christopher Egerton 
> wrote:
> >
> > Hi Cheng,
> >
> > Thanks for the KIP! I really appreciate the care that was taken to ensure
> > backwards compatibility for existing users, and the minimal changes to
> > public interface that are suggested to address this.
> >
> > I have two quick requests for clarification:
> >
> > 1) Where is the proposed "accept.optional.null" property going to apply?
> > It's hinted that it'll take effect on the JSON converter but not actually
> > called out anywhere.
> >
> > 2) Assuming this takes effect on the JSON converter, is the intent to
> alter
> > the semantics for both serialization and deserialization? The code
> snippet
> > from the JSON converter that's included in the KIP comes from the
> > "convertToJson" method, which is used for serialization. However, based
> on
> >
> https://github.com/apache/kafka/blob/ea47a885b1fe47dfb87c1dc86db1b0e7eb8a273c/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L712-L713
> > it
> > looks like the converter also inserts the default value for
> > optional-but-null data during deserialization.
> >
> > Thanks again for the KIP!
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Mar 18, 2020 at 12:00 AM Cheng Pan <379377...@qq.com> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to use this thread to discuss KIP-581: Value of optional null
> > > field which has default value, please see detail at:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > >
> > >
> > > There are some previous discussion at:
> > > https://github.com/apache/kafka/pull/7112
> > >
> > >
> > > I'm a beginner for apache project, please let me know if I did any
> thing
> > > wrong.
> > >
> > >
> > > Best regards,
> > > Cheng Pan
>
>

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

[Tom Bentley]

Congratulations!

On Fri, 10 Mar 2023 at 03:36, John Roesler  wrote:

> Congratulations, David!
> -John
>
> On Thu, Mar 9, 2023, at 20:18, ziming deng wrote:
> > Congrats David!
> >
> > Ziming
> >
> >> On Mar 10, 2023, at 10:02, Luke Chen  wrote:
> >>
> >> Congratulations, David!
> >>
> >> On Fri, Mar 10, 2023 at 9:56 AM Yash Mayya 
> wrote:
> >>
> >>> Congrats David!
> >>>
> >>> On Thu, Mar 9, 2023, 23:42 Jun Rao  wrote:
> >>>
>  Hi, Everyone,
> 
>  David Arthur has been a Kafka committer since 2013. He has been very
>  instrumental to the community since becoming a committer. It's my
> >>> pleasure
>  to announce that David is now a member of Kafka PMC.
> 
>  Congratulations David!
> 
>  Jun
>  on behalf of Apache Kafka PMC
> 
> >>>
>
>

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

[Tom Bentley]

Congratulations!

On Fri, 10 Mar 2023 at 03:35, John Roesler  wrote:

> Congratulations, Chris!
> -John
>
> On Thu, Mar 9, 2023, at 20:02, Luke Chen wrote:
> > Congratulations, Chris!
> >
> > On Fri, Mar 10, 2023 at 9:57 AM Yash Mayya  wrote:
> >
> >> Congratulations Chris!
> >>
> >> On Thu, Mar 9, 2023, 23:42 Jun Rao  wrote:
> >>
> >> > Hi, Everyone,
> >> >
> >> > Chris Egerton has been a Kafka committer since July 2022. He has been
> >> very
> >> > instrumental to the community since becoming a committer. It's my
> >> pleasure
> >> > to announce that Chris is now a member of Kafka PMC.
> >> >
> >> > Congratulations Chris!
> >> >
> >> > Jun
> >> > on behalf of Apache Kafka PMC
> >> >
> >>
>
>

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

[Tom Bentley]

Hi Chris,

Thanks for the KIP.

+1 (binding).

Cheers,

Tom

On Wed, 15 Feb 2023 at 16:11, Chris Egerton  wrote:

> Hi all,
>
> Thanks to everyone who's voted so far! Just wanted to bump this thread and
> see if we could get a few more votes; currently we're at +3 non-binding
> and +1 binding. Hoping we can get this approved, reviewed, and merged in
> time for 3.5.0.
>
> Cheers,
>
> Chris
>
> On Tue, Jan 31, 2023 at 2:52 AM Mickael Maison 
> wrote:
>
> > Thanks Chris for the KIP, this is a much needed feature!
> >
> > +1 (binding)
> >
> >
> > On Tue, Jan 24, 2023 at 3:45 PM Knowles Atchison Jr
> >  wrote:
> > >
> > > +1 (non binding)
> > >
> > > On Tue, Jan 24, 2023 at 5:24 AM Yash Mayya 
> wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > I'm +1 (non-binding). Thanks again for proposing this extremely
> > > > valuable addition to Kafka Connect!
> > > >
> > > > Thanks,
> > > > Yash
> > > >
> > > > On Thu, Jan 19, 2023 at 12:11 AM Chris Egerton
>  > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-875, which adds support for
> > viewing
> > > > and
> > > > > manipulating the offsets of connectors to the Kafka Connect REST
> API.
> > > > >
> > > > > The KIP:
> > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > > >
> > > > > The discussion thread:
> > > > > https://lists.apache.org/thread/m5bklnh5w4mwr9nbzrmfk0pftpxfjd02
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> >
>

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Tom Bentley]

Hi Chia-Ping,

To be honest the stateful version, setting an input stream once using the
`readFrom(InputStream)` method and then repeatedly asking for the next
record using a parameterless `readRecord()`, seems a bit more natural to me
than `readRecord(InputStream inputStream)` being called repeatedly with (I
assume) the same input stream. I think the contract is simpler to describe
and understand.

It's worth thinking about how implementers might have to read bytes from
the stream to discover the end of one record and the start of the next.
Unless we've guaranteed that the input stream supports mark and reset then
they have to buffer the initial bytes of the next record that they've just
read from the stream so that they can use them when called next time. So I
think RecordReaders are (in general) inherently stateful and therefore it
seems harmless for them to also have the input stream itself as some of
that state.

Cheers,

Tom

On Sat, 18 Feb 2023 at 08:25, Chia-Ping Tsai  wrote:

>
>
> On 2023/02/17 06:47:18 Luke Chen wrote:
> > Hi Chia-Ping,
> >
> > Thanks for the KIP!
> >
> > Overall LGTM, just one minor comment:
> > Could we log warning messages to users when using deprecated
> MessageReader?
>
> Sure. I will address it when implementing the KIP.
>
>

Re: [ANNOUNCE] New committer: Lucas Bradstreet

[Tom Bentley]

Congratulations!

On Fri, 17 Feb 2023 at 05:26, Yash Mayya  wrote:

> Congratulations Lucas!
>
> On Fri, Feb 17, 2023 at 3:25 AM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> Lucas
> > Bradstreet.
> >
> > Lucas has been a long time Kafka contributor since Oct. 2018. He has been
> > extremely valuable for Kafka on both performance and correctness
> > improvements.
> >
> > The following are his performance related contributions.
> >
> > KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates batch
> > iterator which is not used
> > KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer
> > KAFKA-9729: avoid readLock in authorizer ACL lookups
> > KAFKA-9039: Optimize ReplicaFetcher fetch path
> > KAFKA-8841: Reduce overhead of ReplicaManager.updateFollowerFetchState
> >
> > The following are his correctness related contributions.
> >
> > KAFKA-13194: LogCleaner may clean past highwatermark
> > KAFKA-10432: LeaderEpochCache is incorrectly recovered on segment
> recovery
> > for epoch 0
> > KAFKA-9137: Fix incorrect FetchSessionCache eviction logic
> >
> > Congratulations, Lucas!
> >
> > Thanks,
> >
> > Jun (on behalf of the Apache Kafka PMC)
> >
>

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Tom Bentley]

Hi Chia-Ping,

Thanks for the KIP. I have just a few minor comments:

1. The Javadoc for the new interface says "an `InputStream` received via
`init`" but the interface doesn't have an init() method. I guess you meant
configure()?
2. The Javadoc should mention the need for implementations to have a public
nullary constructor.
3. On the matter of configure(): While it doesn't add any functionality, it
would be more consistent with other plugins if this interface extended
Configurable, and the InputStream was then passed via some other method
(`readFrom(InputStream)` perhaps). If nothing else it would make it harder
to overlook this interface when making changes which apply to all plugins.
To be honest, I'm not entirely convinced myself, but I thought we should at
least consider it and add it to the rejected alternatives if we decide
against it.

Thanks again,

Tom


On Thu, 16 Feb 2023 at 11:36, Federico Valeri  wrote:

> Hi Chia-Ping, thanks for updating the KIP.
>
> I would only add that we plan to remove the old trait in the next
> major release. I think it's better to be explicit about this.
>
> On Thu, Feb 16, 2023 at 11:34 AM Chia-Ping Tsai 
> wrote:
> >
> > Dear all,
> >
> > I have updated the KIP to address comments. PTAL
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
>
>

Re: [DISCUSS] KIP-898: Modernize Connect plugin discovery

[Tom Bentley]

Hi Greg,

Thanks for the KIP. Overall I think using service loading would be a
worthwhile improvement.

1. "Both of these are well-established Java interfaces for which public
documentation should be readily available." I see no harm in adding a well
chosen link or two just to save  readers unfamiliar with service loading
from having to search for documentation.

2. "plugin.path.discovery" is a bit of an odd name, since it's not
discovering plugin _paths_, rather it's discovering plugins. So perhaps
plugin.discovery, or plugin.discovery.policy would be a better name.

3. I wondered if we really needed "ONLY_SCAN". It seems like the sort of
thing where someone silences the warnings then forgets about it until one
day they (or worse, someone else) gets the pleasure of upgrading and
finding they can't find their plugins. I guess the proposed roll out, over
multiple major releases, makes this less likely. Non-the-less, logged
warnings seem like a small price to pay overall, and would make it harder
for users to ignore the problem. We should be encouraging users to use the
migration script to silence warnings and adopt SERVICE_LOAD. That way they
get the benefits sooner than Kafka 5.0.

4. As described, connect-plugin-path.sh can modify jar files in-place. Have
you tested/considered whether this will work for sealed jars? I know
they're not common, but they are part of the Java platform so we can't rule
out that they're being used. I _think_ service loading should still work
even if you wrote a new shim jar file which just contained the service
files needed for loading (alongside the original unmodified jar containing
the plugins); if that's right then there's no need to actually modify
existing jars.

Thanks again,

Tom

On Wed, 8 Feb 2023 at 21:45, Chris Egerton  wrote:

> Hi Greg,
>
> Thanks for the updates. This is looking great. A few minor questions:
>
> 1. The description for the list command in the plugin path CLI states that
> it'll show "Whether the class is discoverable via scanning". Are there any
> cases where a plugin found by the CLI won't be discoverable via scanning?
>
> 2. Why are we treating module info files differently than service loader
> manifests (where the latter will be removed for not-found plugins on an
> opt-out basis, but the former are left unmodified no matter what)? Basing
> this off of this sentence in the KIP: "If a plugin declares an
> implementation via a module-info.java which is not loadable, the
> module-info.java will not be modified."; let me know if I'm
> misinterpreting.
>
> Cheers,
>
> Chris
>
> On Tue, Feb 7, 2023 at 4:32 PM Greg Harris 
> wrote:
>
> > Chris,
> >
> > Thanks for the comments!
> >
> > 1. I've updated the script to sync-manifests [--keep-not-found] with your
> > described semantics.
> >
> > 2. I've pushed all of the deprecation decisions to a follow-up KIP that
> can
> > take place after an intervening release.
> > I hope that this feature has a high enough ROI that the direction of the
> > migration is obvious to users even without a formal deprecation notice.
> > I do think that without a formal deprecation notice that the migration
> > scripts may be in use for a long time, particularly for connector
> > implementations which are not receiving active maintenance.
> >
> > 3. Thanks for looking into the ServiceLoader error handling contract.
> > From my inspection of the ServiceLoader implementation, it seems to be
> able
> > to recover from most of the poor packaging scenarios that I'm aware of.
> > I think with or without the deprecation, operators are free to select the
> > ONLY_SCAN mode to work around any errors we don't discover before
> release.
> >
> > Thanks,
> > Greg
> >
> > On Mon, Feb 6, 2023 at 8:44 AM Chris Egerton 
> > wrote:
> >
> > > Hi Greg,
> > >
> > > Thanks for the updates. Unless stated below, I agree with your
> > responses. A
> > > few more thoughts:
> > >
> > > 1. IMO it's not worth it to have separate commands for adding/removing
> > > manifests, mostly because it adds complexity to the tool and might make
> > it
> > > harder for users to understand. I think a single "update-manifests" or
> > > "sync-manifests" command that both adds manifests for found plugins and
> > > removes manifests for unfound plugins by default, with a --keep-unfound
> > > flag to disable removal on an opt-in basis, would make more sense.
> > >
> > > 2. I agree with this point you raise about deprecation: "if we agree
> that
> > > the old mechanism has some sunset date that we just don't know yet, we
> > > should still communicate to users that the sunset date is somewhere in
> > the
> > > future." However, I'm not certain that releasing the features proposed
> in
> > > this KIP alone gives us enough confidence in them to sunset the legacy
> > > plugin loading logic, which is why I was hoping we could have at least
> > one
> > > successful release (preferably even two or three) with this feature in
> > > order to identify potential gaps in the 

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Tom Bentley]

Hi Igor,

Thanks for your replies on points 21-25. Those all LGTM. See below for a
further thought on the meta.properties upgrade.

Kind regards,

Tom


On Fri, 13 Jan 2023 at 18:09, Igor Soarez  wrote:

> Hi Tom,
>
> Thank you for having another look.
>
> 20. Upon a downgrade to a Kafka version that runs the current
> "version == 1" assertion, then yes — a downgrade would not be possible
> without first updating (manually) the meta.properties files back
> to the previous version.
>
> We could prevent this issue if we consider the new fields optional and
> not bump the version, but this takes away from the value of having
> a version in the first place. I'm not sure this would be a good trade-off.
>
> IIUC in KIP-866 this issue was addressed by delaying the writing of
> meta.properties under the new version until after the full transition to
> KRaft mode is finished and relaxing the expectations around versioning.
> We could take a similar approach by relaxing the current
> expectation that the version must strictly be 1.
>
> Currently, MetaProperties#parse() checks that the version is 1 but only
> requires that the cluster.id and node.id properties are present in the
> file.
> We can relax the requirement, ensuring only that the version is at least 1,
> and land that change before this KIP is implemented – at least one release
> beforehand. Then there will be some previous version for which a downgrade
> is possible, even after the version has been bumped to 2 and the two new
> properties are introduced. WDYT?
>

Doing that would delay JBOD support and mean existing users of JBOD must
upgrade via that specific version.

I think I favour a different way of relaxing the expectation around
versioning:
1. In the new broker, update the meta.properties with the new keys but
keeping version=1 (AFAICS this won't cause problems if we roll back to old
broker binaries at any point prior to the completion of KRaft upgrade,
because the code ignores unknown keys).
2. When the KRaft upgrade is complete (i.e. once a broker has written the
final ZkMigrationRecord), update meta.properties to version=2, where the
new keys are required.

That would mean that during the upgrade the additional validation of
meta.properties is missing. I think that's acceptable because the extra
protection provided isn't something supported by ZK-based JBOD today, so
there is no loss of functionality.

Wdty?


> 21. I think in this case we can reuse LOG_DIR_NOT_FOUND (code 57) and
> perhaps
> reword its description slightly to be a bit more generic.
> I've updated the KIP to mention this.
>
> 22. This check should be enforced by the broker, as the controller cannot
> know whether the current replica to logdir assignment is correct.
> Currently, the controller does not unfence a broker that does not "want"
> to be unfenced. The broker should not want to be unfenced while there
> are still mismatching (wrong or missing) replica to logdir assignments.
> I updated the KIP to clarify this.
>
> So a reason wouldn't be included in the heartbeat response but your point
> about diagnosing is interesting. So I've updated the KIP to propose a
> new broker metric reporting the number of mismatching replica assignments
> - i.e. replica assignments that are either missing or incorrect in the
> cluster metadata, as observed by the broker — until this metric is zero,
> the broker would not want to be unfenced. Let me know what you think.
>
> 23. I don't see why not. We should be able to extend the system tests
> proposed in KIP-866 to cover this.
>
> 24. Correct. Updated to remove the typo.
>
> 25. Good point. Updated to align the names.
>
>
> Thank you,
>
> --
> Igor
>
>
>

Re: [ANNOUNCE] New committer: Walker Carlson

[Tom Bentley]

Congratulations!

On Wed, 18 Jan 2023 at 01:26, John Roesler  wrote:

> Congratulations, Walker!
> -John
>
> On Tue, Jan 17, 2023, at 18:50, Guozhang Wang wrote:
> > Congrats, Walker!
> >
> > On Tue, Jan 17, 2023 at 2:20 PM Chris Egerton 
> > wrote:
> >
> >> Congrats, Walker!
> >>
> >> On Tue, Jan 17, 2023, 17:07 Bill Bejeck 
> wrote:
> >>
> >> > Congratulations, Walker!
> >> >
> >> > -Bill
> >> >
> >> > On Tue, Jan 17, 2023 at 4:57 PM Matthias J. Sax 
> >> wrote:
> >> >
> >> > > Dear community,
> >> > >
> >> > > I am pleased to announce Walker Carlson as a new Kafka committer.
> >> > >
> >> > > Walker has been contributing to Apache Kafka since November 2019. He
> >> > > made various contributions including the following KIPs.
> >> > >
> >> > > KIP-671: Introduce Kafka Streams Specific Uncaught Exception Handler
> >> > > KIP-696: Update Streams FSM to clarify ERROR state meaning
> >> > > KIP-715: Expose Committed offset in streams
> >> > >
> >> > >
> >> > > Congratulations Walker and welcome on board!
> >> > >
> >> > >
> >> > > Thanks,
> >> > >-Matthias (on behalf of the Apache Kafka PMC)
> >> > >
> >> >
> >>
>
>

Re: [ANNOUNCE] New committer: Stanislav Kozlovski

[Tom Bentley]

Congratulations!

On Tue, 17 Jan 2023 at 16:52, Bill Bejeck  wrote:

> Congratulations Stan!
>
> -Bill
>
> On Tue, Jan 17, 2023 at 11:37 AM Bruno Cadonna  wrote:
>
> > Congrats Stan!
> >
> > Well deserved!
> >
> > Best,
> > Bruno
> >
> > On 17.01.23 16:50, Jun Rao wrote:
> > > Hi, Everyone,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Stanislav Kozlovski.
> > >
> > > Stan has been contributing to Apache Kafka since June 2018. He made
> > various
> > > contributions including the following KIPs.
> > >
> > > KIP-455: Create an Administrative API for Replica Reassignment
> > > KIP-412: Extend Admin API to support dynamic application log levels
> > >
> > > Congratulations, Stan!
> > >
> > > Thanks,
> > >
> > > Jun (on behalf of the Apache Kafka PMC)
> > >
> >
>

[jira] [Resolved] (KAFKA-14566) Please Close This Ticket As It Was Inadvertently Created. See KAFKA-14565 For The Correct Ticket.

[Tom Bentley (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Bentley resolved KAFKA-14566.
-
Resolution: Invalid

> Please Close This Ticket As It Was Inadvertently Created.  See KAFKA-14565 
> For The Correct Ticket.
> --
>
> Key: KAFKA-14566
> URL: https://issues.apache.org/jira/browse/KAFKA-14566
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Terry Beard
>Priority: Major
>  Labels: needs-kip
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Tom Bentley]

Hi Igor,

20. The description of the changes to meta.properties says "If there any
meta.properties file is missing directory.id a new UUID is generated, and
assigned to that log directory by updating the file", and the
upgrade/migration section says "As the upgraded brokers come up, the
existing meta.properties  files in each broker are updated with a generated
directory.id  and directory.ids ." Currently MetaProperties#parse() checks
that the version is 1, so would the described behaviour prevent downgrade
of a broker to an older version of the software?

21. "If the indicated log directory UUID is not a registered log directory
then the call fails with an error" can you specify which error (is it a new
error code)?

22. "If multiple log directories are registered the broker will remain
fenced until the controller learns of all the partition to log directory
placements in that broker - i.e. no remaining replicas assigned to Uuid.ZERO ."
Is an error code used in the BrokerHeartbeatResponse to indicate this
state? (Or is the only way to diagnose the reason for a broker remaining
fenced for this reason to look at the controller logs?)

23. Will there be a system test to cover the upgrade of a ZK+JBOD cluster
to KRaft+JBOD cluster?

24. In the rejected alternatives: "However the broker is in a better
position to make a choice of log directory than the broker". I think that
should be "...than the controller", right?

25. I wonder about the inconsistency of the RPC names: We have the existing
AlterReplicaLogDirs (and log.dirs broker config), but the new RPC is
AssignReplicasToDirectories.

Many thanks!

Tom

On Tue, 3 Jan 2023 at 18:05, Igor Soarez  wrote:

> Hi Jun,
>
> Thank you for having another look.
>
> 11. That is correct. I have updated the KIP in an attempt to make this
> clearer.
> I think the goal should be to try to minimize the chance that a log
> directory
> may happen while the metadata is incorrect about the log directory
> assignment,
> but also have a fallback safety mechanism to indicate to the controller
> that
> some replica was missed in case of a bad race.
>
> 13. Ok, I think I have misunderstood this. Thank you for correcting me.
> In this case the broker can update the existing meta.properties and create
> new meta.properties in the new log directories.
> This also means that the update-directories subcommand in kafka-storage.sh
> is not necessary.
> I have updated the KIP to reflect this.
>
> Please have another look.
>
>
> Thank you,
>
> --
> Igor
>
>
> > On 22 Dec 2022, at 00:25, Jun Rao  wrote:
> >
> > Hi, Igor,
> >
> > Thanks for the reply.
> >
> > 11. Yes, your proposal could work. Once the broker receives confirmation
> of
> > the metadata change, I guess it needs to briefly block appends to the old
> > replica, make sure the future log fully catches up and then make the
> switch?
> >
> > 13 (b). The kafka-storage.sh is only required in KIP-631 for a brand new
> > KRaft cluster. If the cluster already exists and one just wants to add a
> > log dir, it seems inconvenient to have to run the kafka-storage.sh tool
> > again.
> >
> > Thanks,
> >
> > Jun
>
>

Re: [VOTE] 3.3.2 RC1

[Tom Bentley]

+1 binding.

I have checked signatures, given the javadocs a quick look, followed the
quickstart for KRaft and run the tests.

Thanks Chris for running the release.

Tom



On Mon, 9 Jan 2023 at 15:29, Chris Egerton  wrote:

> Hi all,
>
> Thanks to everyone who has tested and voted for the release so far! Since
> we only have two binding votes so far, I'll be keeping the vote thread open
> until we either discover a blocker that requires a new release candidate to
> be generated, or get a third binding vote.
>
> For anyone following along, it may appear that there have already been
> three binding votes, but Satish's vote is actually non-binding since he is
> a committer, but not on the PMC (see the "Product Release" action in the
> project bylaws [1]). We've already clarified this offline and I'm only
> reiterating here to help prevent any confusion over the number of binding
> votes that have been cast. And either way--thank you for testing out the
> release, Satish!
>
> Cheers,
>
> Chris
>
> [1] -
> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Actions
>
> On Thu, Jan 5, 2023 at 9:45 AM Mickael Maison 
> wrote:
>
> > +1 binding
> >
> > - Checked signatures
> > - Ran quickstart with Scala 2.13 binaries
> > - Ran unit and integration tests
> >
> > Thanks Chris for running this release
> >
> >
> > On Wed, Jan 4, 2023 at 11:37 AM Satish Duggana  >
> > wrote:
> > >
> > > Hi Chris,
> > > Thanks for running the release.
> > >
> > > +1 (binding)
> > >
> > > - Verified signatures and artifacts
> > > - Ran testAll/releaseTarGzAll successfully with no failures.
> > > - Ran through quickstart of core/streams on builds generated from the
> > tag.
> > > - Ran a few internal apps targeting to topics on 3 node cluster.
> > >
> > > On Wed, 4 Jan 2023 at 00:49, Manikumar 
> > wrote:
> > > >
> > > > Hi Chris,
> > > >
> > > > +1 (binding)
> > > >
> > > > - verified the signatures, artifacts
> > > > - ran the tests on the source archive
> > > > - verified the quickstarts
> > > >
> > > > Thanks for running the release!
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > >
> > > > On Fri, Dec 23, 2022 at 4:44 PM Federico Valeri <
> fedeval...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi, I did the following to validate the release:
> > > > >
> > > > > - Checksums and signatures ok
> > > > > - Build from source using Java 17 and Scala 2.13 ok
> > > > > - Unit and integration tests ok
> > > > > - Quickstart in both ZK and KRaft modes ok
> > > > > - Test app with staging Maven artifacts ok
> > > > >
> > > > > +1 (non binding)
> > > > >
> > > > > Thanks
> > > > > Fede
> > > > >
> > > > > On Wed, Dec 21, 2022 at 11:21 PM Chris Egerton
> >  wrote:
> > > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the second candidate for release of Apache Kafka 3.3.2.
> > > > > >
> > > > > > This is a bugfix release with several fixes since the release of
> > 3.3.1. A
> > > > > > few of the major issues include:
> > > > > >
> > > > > > * KAFKA-14358 Users should not be able to create a regular topic
> > name
> > > > > > __cluster_metadata
> > > > > > KAFKA-14379 Consumer should refresh preferred read replica on
> > update
> > > > > > metadata
> > > > > > * KAFKA-13586 Prevent exception thrown during connector update
> from
> > > > > > crashing distributed herder
> > > > > >
> > > > > >
> > > > > > Release notes for the 3.3.2 release:
> > > > > >
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Friday, January 6, 2023,
> > 10pm UTC
> > > > > > (this date is chosen to accommodate the various upcoming holidays
> > that
> > > > > > members of the community will be taking and give everyone enough
> > time to
> > > > > > test out the release candidate, without unduly delaying the
> > release)
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.3.2-rc1
> > > > > >
> > > > > > * Documentation:
> > > > > > https://kafka.apache.org/33/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > https://kafka.apache.org/33/protocol.html
> > > > > >
> > > > > > The most recent build has had test failures. These all appear to
> > be due to
> > > > > > flakiness, but it would be nice if someone more familiar with the
> > failed
> > > > > > tests could confirm this. I may update this 

Re: [ANNOUNCE] New committer: Edoardo Comar

[Tom Bentley]

Congratulations!

On Sun, 8 Jan 2023 at 01:14, Satish Duggana 
wrote:

> Congratulations, Edorado!
>
> On Sun, 8 Jan 2023 at 00:15, Viktor Somogyi-Vass
>  wrote:
> >
> > Congrats Edoardo!
> >
> > On Sat, Jan 7, 2023, 18:15 Bill Bejeck  wrote:
> >
> > > Congratulations, Edoardo!
> > >
> > > -Bill
> > >
> > > On Sat, Jan 7, 2023 at 12:11 PM John Roesler 
> wrote:
> > >
> > > > Congrats, Edoardo!
> > > > -John
> > > >
> > > > On Fri, Jan 6, 2023, at 20:47, Matthias J. Sax wrote:
> > > > > Congrats!
> > > > >
> > > > > On 1/6/23 5:15 PM, Luke Chen wrote:
> > > > >> Congratulations, Edoardo!
> > > > >>
> > > > >> Luke
> > > > >>
> > > > >> On Sat, Jan 7, 2023 at 7:58 AM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Congratulations Edo!
> > > > >>>
> > > > >>>
> > > > >>> On Sat, Jan 7, 2023 at 12:05 AM Jun Rao  >
> > > > wrote:
> > > > 
> > > >  Hi, Everyone,
> > > > 
> > > >  The PMC of Apache Kafka is pleased to announce a new Kafka
> committer
> > > > >>> Edoardo
> > > >  Comar.
> > > > 
> > > >  Edoardo has been a long time Kafka contributor since 2016. His
> major
> > > >  contributions are the following.
> > > > 
> > > >  KIP-302: Enable Kafka clients to use all DNS resolved IP
> addresses
> > > >  KIP-277: Fine Grained ACL for CreateTopics API
> > > >  KIP-136: Add Listener name to SelectorMetrics tags
> > > > 
> > > >  Congratulations, Edoardo!
> > > > 
> > > >  Thanks,
> > > > 
> > > >  Jun (on behalf of the Apache Kafka PMC)
> > > > >>>
> > > > >>
> > > >
> > >
>
>

Re: [ANNOUNCE] New committer: Justine Olshan

[Tom Bentley]

Congratulations!

On Mon, 2 Jan 2023 at 18:25, Rajini Sivaram  wrote:

> Congratulations, Justine, well deserved!
>
> Regards,
>
> Rajini
>
> On Mon, Jan 2, 2023 at 5:29 PM Ismael Juma  wrote:
>
> > Congratulations Justine!
> >
> > Ismael
> >
> > On Thu, Dec 29, 2022, 12:58 PM David Jacot  wrote:
> >
> > > Hi all,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Justine
> > > Olshan.
> > >
> > > Justine has been contributing to Kafka since June 2019. She contributed
> > 53
> > > PRs including the following KIPs.
> > >
> > > KIP-480: Sticky Partitioner
> > > KIP-516: Topic Identifiers & Topic Deletion State Improvements
> > > KIP-854: Separate configuration for producer ID expiry
> > > KIP-890: Transactions Server-Side Defense (in progress)
> > >
> > > Congratulations, Justine!
> > >
> > > Thanks,
> > >
> > > David (on behalf of the Apache Kafka PMC)
> > >
> >
>

Re: [ANNOUNCE] New committer: Satish Duggana

[Tom Bentley]

Congratulations!

On Sat, 24 Dec 2022 at 05:05, Luke Chen  wrote:

> Congratulations, Satish!
>
> On Sat, Dec 24, 2022 at 4:12 AM Federico Valeri 
> wrote:
>
> > Hi Satish, congrats!
> >
> > On Fri, Dec 23, 2022, 8:46 PM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Congrats Satish!
> > >
> > > On Fri, Dec 23, 2022, 19:38 Mickael Maison 
> > > wrote:
> > >
> > > > Congratulations Satish!
> > > >
> > > > On Fri, Dec 23, 2022 at 7:36 PM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Congratulations Satish! 
> > > > >
> > > > > On Fri 23. Dec 2022 at 19:32, Josep Prat
>  > >
> > > > > wrote:
> > > > >
> > > > > > Congrats Satish!
> > > > > >
> > > > > > ———
> > > > > > Josep Prat
> > > > > >
> > > > > > Aiven Deutschland GmbH
> > > > > >
> > > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > > > <
> > > >
> > >
> >
> https://www.google.com/maps/search/Immanuelkirchstra%C3%9Fe+26,+10405+Berlin?entry=gmail=g
> > > > >
> > > > > >
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > >
> > > > > > m: +491715557497
> > > > > >
> > > > > > w: aiven.io
> > > > > >
> > > > > > e: josep.p...@aiven.io
> > > > > >
> > > > > > On Fri, Dec 23, 2022, 19:23 Chris Egerton <
> fearthecel...@gmail.com
> > >
> > > > wrote:
> > > > > >
> > > > > > > Congrats, Satish!
> > > > > > >
> > > > > > > On Fri, Dec 23, 2022, 13:19 Arun Raju 
> > > wrote:
> > > > > > >
> > > > > > > > Congratulations 
> > > > > > > >
> > > > > > > > On Fri, Dec 23, 2022, 1:08 PM Jun Rao
>  > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Everyone,
> > > > > > > > >
> > > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > committer
> > > > > > > > Satish
> > > > > > > > > Duggana.
> > > > > > > > >
> > > > > > > > > Satish has been a long time Kafka contributor since 2017.
> He
> > is
> > > > the
> > > > > > > main
> > > > > > > > > driver behind KIP-405 that integrates Kafka with remote
> > > storage,
> > > > a
> > > > > > > > > significant and much anticipated feature in Kafka.
> > > > > > > > >
> > > > > > > > > Congratulations, Satish!
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > > Divij Vaidya
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Josep Prat

[Tom Bentley]

Congratulations!

On Wed, 21 Dec 2022 at 03:05, John Roesler  wrote:

> Congratulations, Josep!
> -John
>
> On Tue, Dec 20, 2022, at 20:02, Luke Chen wrote:
> > Congratulations, Josep!
> >
> > Luke
> >
> > On Wed, Dec 21, 2022 at 6:26 AM Viktor Somogyi-Vass
> >  wrote:
> >
> >> Congrats Josep!
> >>
> >> On Tue, Dec 20, 2022, 21:56 Matthias J. Sax  wrote:
> >>
> >> > Congrats!
> >> >
> >> > On 12/20/22 12:01 PM, Josep Prat wrote:
> >> > > Thank you all!
> >> > >
> >> > > ———
> >> > > Josep Prat
> >> > >
> >> > > Aiven Deutschland GmbH
> >> > >
> >> > > Immanuelkirchstraße 26, 10405 Berlin
> >> > >
> >> > > Amtsgericht Charlottenburg, HRB 209739 B
> >> > >
> >> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > >
> >> > > m: +491715557497
> >> > >
> >> > > w: aiven.io
> >> > >
> >> > > e: josep.p...@aiven.io
> >> > >
> >> > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck  wrote:
> >> > >
> >> > >> Congratulations Josep!
> >> > >>
> >> > >> -Bill
> >> > >>
> >> > >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison <
> >> > mickael.mai...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> Congratulations Josep!
> >> > >>>
> >> > >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna  >
> >> > >> wrote:
> >> > 
> >> >  Congrats, Josep!
> >> > 
> >> >  Well deserved!
> >> > 
> >> >  Best,
> >> >  Bruno
> >> > 
> >> >  On 20.12.22 18:40, Kirk True wrote:
> >> > > Congrats Josep!
> >> > >
> >> > > On Tue, Dec 20, 2022, at 9:33 AM, Jorge Esteban Quilcate Otoya
> >> wrote:
> >> > >> Congrats Josep!!
> >> > >>
> >> > >> On Tue, 20 Dec 2022, 17:31 Greg Harris,
> >> > >>  >> > 
> >> > >> wrote:
> >> > >>
> >> > >>> Congratulations Josep!
> >> > >>>
> >> > >>> On Tue, Dec 20, 2022 at 9:29 AM Chris Egerton <
> >> > >>> fearthecel...@gmail.com>
> >> > >>> wrote:
> >> > >>>
> >> >  Congrats Josep! Well-earned.
> >> > 
> >> >  On Tue, Dec 20, 2022, 12:26 Jun Rao  >
> >> > >>> wrote:
> >> > 
> >> > > Hi, Everyone,
> >> > >
> >> > > The PMC of Apache Kafka is pleased to announce a new Kafka
> >> > >>> committer
> >> >  Josep
> >> > >Prat.
> >> > >
> >> > > Josep has been contributing to Kafka since May 2021. He
> >> > >>> contributed 20
> >> >  PRs
> >> > > including the following 2 KIPs.
> >> > >
> >> > > KIP-773 Differentiate metric latency measured in ms and ns
> >> > > KIP-744: Migrate TaskMetadata and ThreadMetadata to an
> >> interface
> >> > >>> with
> >> > > internal implementation
> >> > >
> >> > > Congratulations, Josep!
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun (on behalf of the Apache Kafka PMC)
> >> > >
> >> > 
> >> > >>>
> >> > >>
> >> > >
> >> > >>>
> >> > >>
> >> > >
> >> >
> >>
>
>

Re: [ANNOUNCE] New Kafka PMC Member: Luke Chen

[Tom Bentley]

Congratulations!

On Fri, 16 Dec 2022 at 20:40, Viktor Somogyi-Vass
 wrote:

> Congrats Luke! :)
>
> On Fri, Dec 16, 2022, 21:26 Randall Hauch  wrote:
>
> > Congratulations, Luke!
> >
> > On Fri, Dec 16, 2022 at 2:08 PM Josep Prat 
> > wrote:
> >
> > > Congrats Luke!
> > >
> > > On Fri, Dec 16, 2022 at 8:55 PM Bill Bejeck  wrote:
> > >
> > > > Congratulations Luke!
> > > >
> > > > -Bill
> > > >
> > > > On Fri, Dec 16, 2022 at 2:47 PM Chris Egerton <
> fearthecel...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Congrats Luke, well-deserved!
> > > > >
> > > > > On Fri, Dec 16, 2022, 14:41 Jun Rao 
> > wrote:
> > > > >
> > > > > > Hi, Everyone,
> > > > > >
> > > > > > Luke Chen has been a Kafka committer since Feb. 9, 2022. He has
> > been
> > > > very
> > > > > > instrumental to the community since becoming a committer. It's my
> > > > > pleasure
> > > > > > to announce that Luke  is now a member of Kafka PMC.
> > > > > >
> > > > > > Congratulations Luke!
> > > > > >
> > > > > > Jun
> > > > > > on behalf of Apache Kafka PMC
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Immanuelkirchstraße 26, 10405 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [ANNOUNCE] New committer: Ron Dagostino

[Tom Bentley]

Congratulations!

On Thu, 15 Dec 2022 at 07:40, Satish Duggana 
wrote:

> Congratulations, Ron!!
>
> On Thu, 15 Dec 2022 at 07:48, ziming deng 
> wrote:
>
> > Congratulations, Ron!
> > Well deserved!
> >
> > --
> > Ziming
> >
> > > On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> > >
> > > Congratulations, Ron!
> > > Well deserved!
> > >
> > > Luke
> >
> >
>

Re: [DISCUSS] KIP-888: Batch describe ACLs and describe client quotas

[Tom Bentley]

Hi Mickael,

Thanks for the KIP. I can understand the motivation, but to be honest I
have a doubt about this.

Taking the ACLs first, by allowing multiple filters with the current
proposal isn't there the chance that the same ACL will match multiple
filters, and be returned multiple times in the response. In fact, in the
worst case couldn't the client (by intent or accident) just request all
ACLs be included in the response an arbitrary number of times? This could
result in some pretty large responses. One way to avoid inflating the
response like this would be for the broker to deduplicate the ACLs in the
response by assigning an id for each, serialising each once and then using
the id to enumerate the matches for each pattern. It's worth noting that
the AccessControlEntryRecord used for KRaft clusters already has a Uuid. So
for the KRaft case it might be possible to use this, rather than the broker
having to deduplicate when handing the request.

Another wrinkle is that if the broker calls
Authorizer.acls(AclBindingFilter) once for each pattern there's no
guarantee that the results are consistent (they could be modified between
calls). It could be made consistent with appropriate locking, but since in
practice this would be a very rare occurrence maybe we could just document
that as a limitation.

Thanks again,

Tom

On Fri, 18 Nov 2022 at 18:00, Mickael Maison 
wrote:

> Hi,
>
> I have opened KIP-888 to allow describing ACLs and Quotas in batches:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-888%3A+Batch+describe+ACLs+and+describe+client+quotas
>
> Let me know if you have any feedback or suggestions.
>
> Thanks,
> Mickael
>
>

Re: Request the account for Kafka Jira

[Tom Bentley]

Hi Simon,

Usernames have to be all lower case ascii letters and digits, so I used
simonzhang. You should have a password reset email where you can complete
the account creation.

Thanks for your interest in Apache Kafka.

On Tue, 15 Nov 2022 at 15:25, zsy112371  wrote:

> Hi team,
> I would like to request an account for the kafka jira.
>
> username: Simon Zhang
> display name: Simon Zhang
> email address: zsy112...@163.com
>
> Thank you
>
>

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

[Tom Bentley]

Congratulations!

On Wed, 2 Nov 2022 at 06:40, David Jacot  wrote:

> Congrats, Bruno! Well deserved.
>
> Le mer. 2 nov. 2022 à 06:12, Randall Hauch  a écrit :
>
> > Congratulations, Bruno!
> >
> > On Tue, Nov 1, 2022 at 11:20 PM Sagar  wrote:
> >
> > > Congrats Bruno!
> > >
> > > Sagar.
> > >
> > > On Wed, Nov 2, 2022 at 7:51 AM deng ziming 
> > > wrote:
> > >
> > > > Congrats!
> > > >
> > > > --
> > > > Ziming
> > > >
> > > > > On Nov 2, 2022, at 3:36 AM, Guozhang Wang 
> > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to introduce our new Kafka PMC member, Bruno.
> > > > >
> > > > > Bruno has been a committer since April. 2021 and has been very
> active
> > > in
> > > > > the community. He's a key contributor to Kafka Streams, and also
> > helped
> > > > > review a lot of horizontal improvements such as Mockito. It is my
> > > > pleasure
> > > > > to announce that Bruno has agreed to join the Kafka PMC.
> > > > >
> > > > > Congratulations, Bruno!
> > > > >
> > > > > -- Guozhang Wang, on behalf of Apache Kafka PMC
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Tom Bentley]

Hi Igor,

Thanks for the KIP, I've finally managed to take an initial look.

0. You mention the command line tools (which one?) in the public interfaces
section, but don't spell out how it changes -- what options are added.
Reading the proposed changes it suggests that there are no changes to the
supported options and that it is done automatically during initial
formatting based on the broker config. But what about the case where we're
upgrading an existing non-JBOD KRaft cluster where the meta.properties
already exists? Do we just run `./bin/kafka-storage.sh format -c
/tmp/server.properties` again? How would an operator remove an existing log
dir?

1. In the example for the storage format command I think it's worth
explaining it in a little more detail. i.e. that the `directory.ids` has
three items: two for the configured log.dirs and one for the configured
metadata.log.dir.

2. In 'Broker lifecycle management' you presumably want to check that the
directory ids for each log dir are actually unique.

3. I don't understand the motivation for having the controller decide the
log dir for new replicas. I think there are two cases we want to support:
a) Where the user specifies the log dir (likely based on some external
information). This is out of scope for this KIP.
b) If the user didn't specify, isn't the broker in a better position to
decide (for example, based on free storage), and the communicate back to
the controller the log dir that was chosen using the
ASSIGN_REPLICAS_TO_DIRECTORIES RPC?

4. Broker registration. I don't understand the intent behind the
optimization for the single log dir case (last bullet). "Brokers whose
registration indicate that multiple log directories are configured remain
FENCED until all log directory assignments for that broker are learnt by
the active controller and persisted into metadata." is something you've
committed to anyway.

5. AFAICS there's no way for the user to determine via the Kafka protocol
which directory id corresponds to which log dir path. I.e. you've not
changed any of the Admin APIs. Perhaps adding a Future Work section to
spell out the pieces we know are missing would be a good idea?

I would second Jason's idea for piggybacking on- and off-line state changes
on the BrokerHeartbeat RPC. I suspect the complexity of making this
incrementally isn't so great, given that both broker and controller need to
keep track of the on- and off-line directories anyway. i.e. We could add
LogDirsOfflined and LogDirsOnlined fields to both request and response and
have the broker keep including a log dir in requests until acknowledged in
the response, but otherwise they'd be empty.

Thanks again,

Tom

On Tue, 25 Oct 2022 at 19:59, Igor Soarez  wrote:

> Hello,
>
> There’s now a proposal to address ZK to KRaft migration — KIP-866 — but
> currently it doesn't address JBOD so I've decided to update this proposal
> to address that migration scenario.
>
> So given that:
>
> - When migrating from a ZK cluster running JBOD to KRaft, brokers
> registering in KRaft mode will need to be able to register all configured
> log directories.
> - As part of the migration, the mapping of partition to log directory will
> have to be learnt by the active controller and persisted into the cluster
> metadata.
> - It isn’t safe to allow for leadership from replicas without this
> mapping, as if the hosting log directory fails there will be no failover
> mechanism.
>
> I have updated the proposal to reflect that:
>
> - Multiple log directories may be indicated in the first broker
> registration referencing log directory UUIDs. We no longer require a single
> log directory to start with.
> - The controller must never assign leadership to a replica in a broker
> registered with multiple log directories, unless the partition to log
> directory assignment is already in the cluster metadata.
> - The broker should not be unfenced until all of its partition to log
> directory mapping is persisted into cluster metadata
>
> I've also added details as to how the ZK to KRaft migration can work in a
> cluster that is already operating with JBOD.
>
> Please have a look and share your thoughts.
>
> --
> Igor
>
>
>

Re: [ANNOUNCE] New committer: Deng Ziming

[Tom Bentley]

Congratulations!

On Mon, 10 Oct 2022 at 18:24, John Roesler  wrote:

> Congratulations, Ziming!
>
> On Mon, Oct 10, 2022, at 12:05, Justine Olshan wrote:
> > Congratulations Ziming! I'll always remember the help you provided with
> > topic IDs.
> > Very well deserved!
> >
> > On Mon, Oct 10, 2022 at 9:53 AM Matthew Benedict de Detrich
> >  wrote:
> >
> >> Congratulations!
> >>
> >> On Mon, 10 Oct 2022, 11:30 Jason Gustafson,  >
> >> wrote:
> >>
> >> > Hi All
> >> >
> >> > The PMC for Apache Kafka has invited Deng Ziming to become a
> committer,
> >> > and we are excited to announce that he has accepted!
> >> >
> >> > Ziming has been contributing to Kafka for about three years. He has
> >> > authored
> >> > more than 100 patches and helped to review nearly as many. In
> particular,
> >> > he made significant contributions to the KRaft project which had a big
> >> part
> >> > in reaching our production readiness goal in the 3.3 release:
> >> > https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in.
> >> >
> >> > Please join me in congratulating Ziming! Thanks for all of your
> >> > contributions!
> >> >
> >> > -- Jason, on behalf of the Apache Kafka PMC
> >> >
> >>
>
>

Re: PR for CVE-2022-34917

[Tom Bentley]

Hi Swathi,

In this case the PR reviews happened on a private repo because the CVE
wasn't public at that time. On the 3.3 branches you can look at/cherry-pick
commits 015d7aede6cbd350d56d75006930dd2bf89a4a5a and
b2b928338c7226b41a73786df27a2127eaa32ab2.

Kind regards,

Tom


On Mon, 26 Sept 2022 at 15:19, Swathi Mocharla 
wrote:

> Hi,
> CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-34917
> Could you please help with the PR that fixed this vulnerability? We are
> looking to apply the patch that fixes this and we are unable to find it.
> Thanks,
> Swathi
>

Re: [DISCUSS]: Including TLA+ in the repo

[Tom Bentley]

Hi Divij,

I don't think that really works for a couple of reasons:

1. I'm not sure every KIP author would necessarily know enough to be able
to answer that question; we don't want knowledge of TLA+ to be seen as
necessary for writing KIPs.
2. It's not just KIPs which could make breaking changes. So really this is
something we need to be aware of during PR review, rather than KIP review.

Perhaps a more fruitful line of thought is about how we can correlate bits
of a spec (such as [1]) with parts of the codebase (such as [2]). That
would make it easier for people (reviewer or otherwise) to study how the
one relates to the other. I think that gets us closer to the second point
you made in your first reply to this thread, because if it's obvious in the
code it makes it easier to know what needs to be asserted in tests of that
code. I'm not suggesting how we might enable this navigability
specifically, just that it seems like a worthwhile problem to solve
somehow. Maybe naming conventions or comments are enough, or perhaps there
are more technical/machine-readable solutions (e.g. annotations) which
could add extra benefits/assurances. Obviously this isn't a problem we have
to solve on day 1, either.

Kind regards,

Tom


[1]:
https://github.com/hachikuji/kafka-specification/blob/master/KafkaReplication.tla#L138
[2]:
https://github.com/apache/kafka/blob/b172a0a94f4ebb9fe638b064d825f0720e7d20ab/core/src/main/scala/kafka/controller/KafkaController.scala#L1066

On Wed, 10 Aug 2022 at 17:44, Divij Vaidya  wrote:

> On keeping the proofs up to date:
>
> Shall we add a template questionnaire to the KIP template that would force
> the author to think about potential change in proofs, e.g. “Does the change
> modify the proof of semantics described at XYZ? If yes, how?”
>
> On Tue, Aug 9, 2022 at 11:54 PM Guozhang Wang  wrote:
>
> > +1 as well. I think adding such proofs in the repo could encourage more
> > people reviewing and challenging it, helping to improve whenever we see
> > fit. Also it helps readers better understand the design patterns.
> >
> > One thing worth noting though is how to make sure the proofs are keep up
> to
> > date with the actual designs, for large refactoring that touches on
> related
> > modules, e.g. the replication protocol etc, I think we should include the
> > proofs as part of the scope.
> >
> >
> > Guozhang
> >
> > On Thu, Jul 28, 2022 at 9:52 AM Matthew Benedict de Detrich
> >  wrote:
> >
> > > +1 from me as well, having the formal TLA+ proofs in the main repo is
> > > hugely beneficial not only from understanding the high level protocol
> but
> > > also in terms of awareness/making sure the proof is not outdated.
> > >
> > > --
> > > Matthew de Detrich
> > > Aiven Deutschland GmbH
> > > Immanuelkirchstraße 26, 10405 Berlin
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > m: +491603708037
> > > w: aiven.io e: matthew.dedetr...@aiven.io
> > > On 26. Jul 2022, 16:58 +0200, Tom Bentley ,
> wrote:
> > > > Hi,
> > > >
> > > > I noticed that TLA+ has featured in the Test Plans of a couple of
> > recent
> > > > KIPs [1,2]. This is a good thing in my opinion. I'm aware that TLA+
> has
> > > > been used in the past to prove properties of various parts of the
> Kafka
> > > > protocol [3,4].
> > > >
> > > > The point I wanted to raise is that I think it would be beneficial to
> > the
> > > > community if these models could be part of the main Kafka repo. That
> > way
> > > > there are fewer hurdles to their discoverability and it makes it
> easier
> > > for
> > > > people to compare the implementation with the spec. Spreading
> > familiarity
> > > > with TLA+ within the community is also a potential side-benefit.
> > > >
> > > > I notice that the specs in [4] are MIT-licensed, but according to the
> > > > Apache 3rd party license policy [5] it should be OK to include.
> > > >
> > > > Thoughts?
> > > >
> > > > Kind regards,
> > > >
> > > > Tom
> > > >
> > > > [1]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-TestPlan
> > > > [2]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Voter+Changes#KIP853:KRaftVoterChanges-TestPlan
> > > > [3]: https://github.com/hachikuji/kafka-specification
> > > > [4]:
> > > >
> > >
> >
> https://github.com/Vanlightly/raft-tlaplus/tree/main/specifications/pull-raft
> > > > [5]: https://www.apache.org/legal/resolved.html
> > >
> >
> >
> > --
> > -- Guozhang
> >
> --
> Divij Vaidya
>

Re: [VOTE] KIP-854 Separate configuration for producer ID expiry

[Tom Bentley]

Hi Justine,

Thanks again for the KIP, +1 (binding).



On Tue, 9 Aug 2022 at 18:09, Jason Gustafson 
wrote:

> Thanks Justine, +1 from me.
>
> On Tue, Aug 9, 2022 at 1:12 AM Sagar  wrote:
>
> > Thanks for the KIP.
> >
> > +1(non-binding)
> >
> > Sagar.
> >
> > On Tue, Aug 9, 2022 at 1:13 PM David Jacot 
> > wrote:
> >
> > > Thanks for the KIP, Justine. The proposal makes sense to me. I am +1
> > > (binding).
> > >
> > > Cheers,
> > > David
> > >
> > > On Mon, Aug 8, 2022 at 6:18 PM Justine Olshan
> > >  wrote:
> > > >
> > > > Hi all,
> > > > I'd like to start a vote for KIP-854: Separate configuration for
> > producer
> > > > ID expiry.
> > > >
> > > > KIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-854+Separate+configuration+for+producer+ID+expiry
> > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-14097
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/cz9x90883on98k082qd0tskj6yjhox1t
> > > >
> > > > Thanks,
> > > > Justine
> > >
> >
>

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Tom Bentley]

Hi Justine,

That all seems reasonable to me, thanks!

On Wed, 3 Aug 2022 at 19:14, Justine Olshan 
wrote:

> Hi Tom and Ismael,
>
> 1. Yes, there are definitely many ways to improve this issue and I plan to
> write followup KIPs to address some of the larger changes.
> Just wanted to get this simple fix in as a short term measure to prevent
> issues with too many producer IDs in the cache. Stay tuned :)
>
> 2. I did have some offline discussion about informing the client. I think
> for this specific KIP the default behavior in practice should not change
> enough to require this information to go back to the client. In other
> words, a reasonable configuration should not regress behavior. However,
> with the further changes I mention in 1, perhaps this is something we want
> to do. And yes -- unfortunately the current state of Kafka is no longer
> totally consistent with KIP-98. This is something we probably want to
> clarify in the future.
>
> 3. I will update the config to mention it is not dynamic. I think since the
> transactional id configuration is read-only, this should be too.
>
> 4. I can update this wording.
>
> 5. I think there are definitely benefits to the name `
> idempotent.pid.expiration.ms` but there are other ways this could cause
> confusion. And to be clear -- the configuration can expire a producer ID
> for a transactional producer as long as there isn't an ongoing transaction.
>
> Let me know if you have any questions and thanks for taking a look!
>
> Justine
>
> On Wed, Aug 3, 2022 at 9:30 AM Ismael Juma  wrote:
>
> > Regarding 1, more can certainly be done, but I think it would be
> > complementary. As such, I think this KIP stands on its own and additional
> > improvements can be handled via future KIPs (unless Justine wants to
> > combine things, of course).
> >
> > Ismael
> >
> > On Wed, Aug 3, 2022 at 9:12 AM Tom Bentley  wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for the KIP! I can see that this is a pragmatic attempt to
> > address a
> > > nasty problem. I have a few questions:
> > >
> > > 1. The KIP makes the problem significantly harder to trigger, but
> doesn't
> > > eliminate it entirely. How confident are you that it will be sufficient
> > in
> > > practice? We can point to applications which are creating idempotent
> > > producers at a high rate and say they're broken, but that doesn't do
> > > anything to defend the broker from an interaction pattern that differs
> > only
> > > in rate from a "good application". Did you consider a new quota to
> limit
> > > the rate at which a (principal, clientId) can allocate new PIDs?
> > >
> > > 2. The KIP contains this sentence: "when an idempotent producer’s ID
> > > expires, it silently loses its idempotency guarantees." That's at odds
> > with
> > > my reading of "PID expiration" in the KIP-98 design[1], but it does
> seem
> > > consistent with a (brief!) look at the code. I accept that the risk
> > should
> > > be minimal so long as the expiration time is > the producer's delivery
> > > timeout, but it would still be nice if we could detect this situation
> and
> > > return an error to the client. Is there a reason for the apparent
> > deviation
> > > from KIP-98 (or am I misreading the code?)
> > >
> > > 3. Could the KIP be explicit on whether the new config will be
> > dynamically
> > > changeable?
> > >
> > > 4. The description of producer.id.expiration.ms mentions the
> > > ProducerStateManager, which will mean nothing to a normal user. We
> could
> > > probably change it to "a topic partition leader" without loss of
> meaning.
> > >
> > > 5. The description also says "Producer IDs will not expire while a
> > > transaction associated to them is still ongoing." To me this suggests
> > that
> > > a more intuitive name for this config (from the user PoV) would include
> > > "idempotent", since it does not cover the transactional case. (I would
> > > suggest "idempotent.pid.expiration.ms" (c.f.
> > > transactional.id.expiration.ms),
> > > but the distinction between "id" and "pid" is easily missed–even if
> it's
> > > technically correct–so I'm not sure it's better than what you're
> > > proposing). I only mention it in case it prompts someone else to find a
> > > better name.
> > >
> > > Thanks again,
> > >
> > > Tom
> > >
>

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Tom Bentley]

Hi Justine,

Thanks for the KIP! I can see that this is a pragmatic attempt to address a
nasty problem. I have a few questions:

1. The KIP makes the problem significantly harder to trigger, but doesn't
eliminate it entirely. How confident are you that it will be sufficient in
practice? We can point to applications which are creating idempotent
producers at a high rate and say they're broken, but that doesn't do
anything to defend the broker from an interaction pattern that differs only
in rate from a "good application". Did you consider a new quota to limit
the rate at which a (principal, clientId) can allocate new PIDs?

2. The KIP contains this sentence: "when an idempotent producer’s ID
expires, it silently loses its idempotency guarantees." That's at odds with
my reading of "PID expiration" in the KIP-98 design[1], but it does seem
consistent with a (brief!) look at the code. I accept that the risk should
be minimal so long as the expiration time is > the producer's delivery
timeout, but it would still be nice if we could detect this situation and
return an error to the client. Is there a reason for the apparent deviation
from KIP-98 (or am I misreading the code?)

3. Could the KIP be explicit on whether the new config will be dynamically
changeable?

4. The description of producer.id.expiration.ms mentions the
ProducerStateManager, which will mean nothing to a normal user. We could
probably change it to "a topic partition leader" without loss of meaning.

5. The description also says "Producer IDs will not expire while a
transaction associated to them is still ongoing." To me this suggests that
a more intuitive name for this config (from the user PoV) would include
"idempotent", since it does not cover the transactional case. (I would
suggest "idempotent.pid.expiration.ms" (c.f. transactional.id.expiration.ms),
but the distinction between "id" and "pid" is easily missed–even if it's
technically correct–so I'm not sure it's better than what you're
proposing). I only mention it in case it prompts someone else to find a
better name.

Thanks again,

Tom

[1]:
https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.loujdamc9ptj

On Tue, 2 Aug 2022 at 22:00, Justine Olshan 
wrote:

> I've updated the KIP to make the new minimum value 1 and remove the -1
> configuration.
> I've also added the low priority to the configuration and edited the
> description as Ismael mentioned.
>
> I'm thinking about bringing this KIP to a vote soon! Let me know if there
> are any other comments or questions.
>
> Thanks,
> Justine
>
> On Tue, Aug 2, 2022 at 9:02 AM Jason Gustafson  >
> wrote:
>
> > I agree with Ismael that we should remove -1. I think we tend to view the
> > coupling of these behaviors into a single configuration as a mistake, so
> > it's a little odd to keep it (even if in a weakened form).
> >
> > -Jason
> >
> > On Sat, Jul 30, 2022 at 7:37 AM Ismael Juma  wrote:
> >
> > > I would remove -1 altogether. Two more comments:
> > >
> > > 1. The current description kind of leads people towards aligning the
> > config
> > > with delivery.timeout.ms. Is that what we want? We could say it should
> > be
> > > higher than delivery.timeout.ms but indicate that the default is
> usually
> > > fine. The main reason to reduce it would be to save memory, I guess.
> > > 2. Each config has a priority, we should specify it for this one. I'm
> > > assuming it will be "low".
> > >
> > > Ismael
> > >
> > > On Fri, Jul 29, 2022 at 2:38 PM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've updated the KIP to include the new default of 1 day and
> > information
> > > > about -1 in the description of the config.
> > > > I wonder though if including -1 makes sense now that it is not the
> > > default
> > > > value. Is there a benefit for manually setting -1 vs manually setting
> > the
> > > > value that transactional.id.expiration.ms has?
> > > >
> > > > Let me know your thoughts.
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Fri, Jul 29, 2022 at 10:54 AM Ismael Juma 
> > wrote:
> > > >
> > > > > +1 for having 1 day as the default and for including this change in
> > the
> > > > > release notes.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson
> > > >  > > > > >
> > > > > wrote:
> > > > >
> > > > > > I don't think a major release is a requirement for a default
> change
> > > for
> > > > > > what it's worth. I do appreciate that there is a preference for
> not
> > > > > rocking
> > > > > > the boat though. For a little bit of background here, the problem
> > we
> > > > > > have encountered in production since the idempotent producer
> became
> > > the
> > > > > > default is OOM errors due to huge numbers of producerIds that had
> > to
> > > be
> > > > > > retained in the cache for 7 days. It is hard to prevent use cases
> > > from
> > > > > > emerging where producers are used and discarded rapidly. We will
> be
> > > > > 

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Tom Bentley]

Congratulations Sophie!

On Tue, 2 Aug 2022 at 11:54, Robin Moffatt 
wrote:

> Congrats Sophie, great news!
>
>
> --
>
> Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Tue, 2 Aug 2022 at 00:42, Guozhang Wang  wrote:
>
> > Hi everyone,
> >
> > I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> > committer since Oct. 2020 and has been contributing to the community
> > consistently, especially around Kafka Streams and Kafka java consumer.
> She
> > has also presented about Kafka Streams at Kafka Summit London this year.
> It
> > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> >
> > Congratulations, Sophie!
> >
> > -- Guozhang Wang, on behalf of Apache Kafka PMC
> >
>

Re: [DISCUSS]: Including TLA+ in the repo

[Tom Bentley]

Thanks Jason and Jack!

I count myself as a beginner with TLA+, but would like to take this as an
opportunity to learn.

Tom

On Thu, 28 Jul 2022 at 17:34, Jason Gustafson 
wrote:

> Yeah, good idea. I'm happy to submit the specs I wrote for normal kafka
> replication. It will make them more accessible and I have long been looking
> for help reviewing them. Hopefully it will also provide a better chance to
> keep them in sync with the codebase as we update protocols.
>
> -Jason
>
> On Wed, Jul 27, 2022 at 1:50 AM Jack Vanlightly
>  wrote:
>
> > +1 for me too. Once the KIP-853 is agreed I will make any necessary
> changes
> > and submit a PR to the apache/kafka repo.
> >
> > Jack
> >
> > On Tue, Jul 26, 2022 at 10:10 PM Ismael Juma  wrote:
> >
> > > I'm +1 for inclusion in the main repo and I was going to suggest the
> same
> > > in the KIP-853 discussion. The original authors of 3 and 4 are part of
> > the
> > > kafka community, so we can ask them to submit PRs.
> > >
> > > Ismael
> > >
> > > On Tue, Jul 26, 2022 at 7:58 AM Tom Bentley 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I noticed that TLA+ has featured in the Test Plans of a couple of
> > recent
> > > > KIPs [1,2]. This is a good thing in my opinion. I'm aware that TLA+
> has
> > > > been used in the past to prove properties of various parts of the
> Kafka
> > > > protocol [3,4].
> > > >
> > > > The point I wanted to raise is that I think it would be beneficial to
> > the
> > > > community if these models could be part of the main Kafka repo. That
> > way
> > > > there are fewer hurdles to their discoverability and it makes it
> easier
> > > for
> > > > people to compare the implementation with the spec. Spreading
> > familiarity
> > > > with TLA+ within the community is also a potential side-benefit.
> > > >
> > > > I notice that the specs in [4] are MIT-licensed, but according to the
> > > > Apache 3rd party license policy [5] it should be OK to include.
> > > >
> > > > Thoughts?
> > > >
> > > > Kind regards,
> > > >
> > > > Tom
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-TestPlan
> > > > [2]:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Voter+Changes#KIP853:KRaftVoterChanges-TestPlan
> > > > [3]: https://github.com/hachikuji/kafka-specification
> > > > [4]:
> > > >
> > > >
> > >
> >
> https://github.com/Vanlightly/raft-tlaplus/tree/main/specifications/pull-raft
> > > > [5]: https://www.apache.org/legal/resolved.html
> > > >
> > >
> >
>

[DISCUSS]: Including TLA+ in the repo

[Tom Bentley]

Hi,

I noticed that TLA+ has featured in the Test Plans of a couple of recent
KIPs [1,2]. This is a good thing in my opinion. I'm aware that TLA+ has
been used in the past to prove properties of various parts of the Kafka
protocol [3,4].

The point I wanted to raise is that I think it would be beneficial to the
community if these models could be part of the main Kafka repo. That way
there are fewer hurdles to their discoverability and it makes it easier for
people to compare the implementation with the spec. Spreading familiarity
with TLA+ within the community is also a potential side-benefit.

I notice that the specs in [4] are MIT-licensed, but according to the
Apache 3rd party license policy [5] it should be OK to include.

Thoughts?

Kind regards,

Tom

[1]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-TestPlan
[2]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Voter+Changes#KIP853:KRaftVoterChanges-TestPlan
[3]: https://github.com/hachikuji/kafka-specification
[4]:
https://github.com/Vanlightly/raft-tlaplus/tree/main/specifications/pull-raft
[5]: https://www.apache.org/legal/resolved.html

Re: [ANNOUNCE] New Committer: Chris Egerton

[Tom Bentley]

Congratulations Chris!

On Mon, 25 Jul 2022 at 18:40, Jeremy Custenborder 
wrote:

> Congrats Chris!
>
> On Mon, Jul 25, 2022 at 12:07 PM Sagar  wrote:
> >
> > Congratulations Chris !
> >
> > On Mon, 25 Jul 2022 at 10:32 PM, Viktor Somogyi-Vass
> >  wrote:
> >
> > > Congrats Chris!
> > >
> > > On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
> > >  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > --
> > > > Matthew de Detrich
> > > > Aiven Deutschland GmbH
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > m: +491603708037
> > > > w: aiven.io e: matthew.dedetr...@aiven.io
> > > > On 25. Jul 2022, 18:26 +0200, Mickael Maison ,
> > > wrote:
> > > > > Hi all,
> > > > >
> > > > > The PMC for Apache Kafka has invited Chris Egerton as a committer,
> and
> > > > > we are excited to announce that he accepted!
> > > > >
> > > > > Chris has been contributing to Kafka since 2017. He has made over
> 80
> > > > > commits mostly around Kafka Connect. His most notable contributions
> > > > > include KIP-507: Securing Internal Connect REST Endpoints and
> KIP-618:
> > > > > Exactly-Once Support for Source Connectors.
> > > > >
> > > > > He has been an active participant in discussions and reviews on the
> > > > > mailing lists and on Github.
> > > > >
> > > > > Thanks for all of your contributions Chris. Congratulations!
> > > > >
> > > > > -- Mickael, on behalf of the Apache Kafka PMC
> > > >
> > >
>
>

Re: [DISCUSS] KIP-853: KRaft Voters Change

[Tom Bentley]

Hi José,

Thanks for the KIP. As Justine mentioned, this KIP currently lacks a
motivation, and nor does the JIRA provide any context. Please could you
provide this context, otherwise it's impossible for people on this list to
understand the problem you're trying to solve here.

Many thanks,

Tom

On Fri, 22 Jul 2022 at 01:04, Colin McCabe  wrote:

> Hi José,
>
> Thanks for the KIP! I have not had time to fully digest it, but I had some
> initial questions:
>
> 1. It seems like the proposal is to have a UUID per partition directory on
> the voter. If I understand correctly, this is sometimes referred to as
> "VoterUUID" and sometimes as "ReplicaUUID." The latter seems more accurate,
> since a single voter could have multiple of these IDs, in a situation where
> we had multiple Raft topics. So it would be good to standardize on that.
> Also, I didn't see a description of how this would be stored in the log
> directory. That would be good to add.
>
> 2. When we originally did the Raft and Quorum Controller KIPs, one
> contentious topic was node IDs. We eventually settled on the idea that
> broker and controller IDs were in the same ID space. So you can't (for
> example) have a broker 3 that is in a separate JVM from controller 3. This
> is pretty easy to enforce with a static configuration, but it seems like it
> will be harder to do dynamically.
>
> I would like to keep this invariant. This probably requires us to reject
> attempts to add a new quorum voter which duplicates a broker ID (except in
> the special case of co-location!) Similarly, we should reject broker
> registrations that duplicate an unrelated controller ID. The broker's
> incarnation ID is the key to doing this, I think. But that requires us to
> send the incarnation ID in many of these RPCs.
>
> 3. Is it really necessary to put the endpoint information into the
> AddVoterRecord? It seems like that could be figured out at runtime, like we
> do today. If we do need it, it seems particularly weird for it to be
> per-partition (will we have a separate TCP port for each Raft partition?) I
> also don't know why we'd want multiple endpoints. We have that for the
> broker because the endpoints have different uses, but that isn't the case
> here.
>
> The original rationale for multiple endpoints on the controllers was to
> support migration from PLAINTEXT to SSL (or whatever). But that only
> requires multiple listeners to be active on the receive side, not send
> side. A single voter never needs more than one endpoint to contact a peer.
>
> Overall, I think we'd be better off keeping this as soft state rather than
> adding it to the log. Particularly if it's not in the log at all for the
> static configuration case...
>
> 4. How do you get from the static configuration situation to the dynamic
> one? Can it be done with a rolling restart? I think the answer is yes, but
> I wasn't quite sure on reading. Does a leader using the static
> configuration auto-remove voters that aren't in that static config, as well
> as auto-add? The adding behavior is spelled out, but not removing (or maybe
> I missed it).
>
> best,
> Colin
>
>
> On Thu, Jul 21, 2022, at 09:49, José Armando García Sancio wrote:
> > Hi all,
> >
> > I would like to start the discussion on my design to support
> > dynamically changing the set of voters in the KRaft cluster metadata
> > topic partition.
> >
> > KIP URL: https://cwiki.apache.org/confluence/x/nyH1D
> >
> > Thanks!
> > -José
>
>

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

[Tom Bentley]

Hi again Omnia,

I left a couple of really minor points in the discussion thread. Assuming
you're happy with those I am +1 (binding).

Thanks for your patience on this. Kind regards,

Tom

On Tue, 28 Jun 2022 at 10:14, Omnia Ibrahim  wrote:

> Hi,
> I did a small change to the KIP interface to address some discussions on
> the KIP. If all is good now can I get more votes on this, please?
>
> Thanks
> Omnia
>
> On Tue, Jun 21, 2022 at 10:34 AM Omnia Ibrahim 
> wrote:
>
> > Hi,
> > Can I get more votes on this, please?
> >
> > Thanks
> >
> > On Sun, Jun 12, 2022 at 2:40 PM Federico Valeri 
> > wrote:
> >
> >> Hi Omnia, this will be really useful, especially in cloud environment.
> >>
> >> +1 non binding
> >>
> >> Thanks
> >> Fede
> >>
> >> On Tue, May 31, 2022 at 5:28 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >> >
> >> > Hi Omnia,
> >> >
> >> > I think the approach you settled on is the best option, this will
> >> > allow integrating MirrorMaker in more environments.
> >> >
> >> > +1 binding
> >> >
> >> > Thanks for the KIP (and your persistence!)
> >> > Mickael
> >> >
> >> > On Mon, May 30, 2022 at 12:09 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com>
> >> wrote:
> >> > >
> >> > > Hi,
> >> > > Can I get a vote on this, please?
> >> > > Thanks
> >> > >
> >> > > On Wed, May 25, 2022 at 11:15 PM Omnia Ibrahim <
> >> o.g.h.ibra...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi,
> >> > > > I'd like to start a vote on KIP-787
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-787
> >> > > > %3A+MM2+Interface+to+manage+Kafka+resources
> >> > > > <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-787%3A+MM2+Interface+to+manage+Kafka+resources
> >> >
> >> > > >
> >> > > > Thanks
> >> > > > Omnia
> >> > > >
> >>
> >
>

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Tom Bentley]

Hi Omnia,

Please could you add to the KIP the documentation that forward.admin.class
config will have, and include that the signature of the class named by
forward.admin.class must have a constructor that takes a `Map`?

Many thanks,

Tom

On Wed, 22 Jun 2022 at 11:30,  wrote:

> Hi Tom
> My initial thought was that the constructor with config and delegator
> would be for testing while the one with config is the one that will be used
> by MM2.
> I can removed one of them and keep only one.
>
> Sent from my iPhone
>
> > On 22 Jun 2022, at 08:15, Tom Bentley  wrote:
> >
> > Hi Omnia,
> >
> > Thanks for those answers. I'm a bit confused by the changes you've made
> for
> > 1 though. It's MM2 that's going to instantiate the class named in
> > forwarding.admin.class, so it's MM2 that needs to know the constructor
> > signature. The easiest way of doing this is to specify the signature as
> > part of the contract for forwarding.admin.class. But ForwardingAdmin now
> > has two constructors, which is confusing for someone who wants to
> subclass
> > it. Surely only one is needed? It would also be a good idea to show the
> > documentation that forward.admin.class will have.
> >
> > Kind regards,
> >
> > Tom
> >
> >> On Tue, 21 Jun 2022 at 17:06, Omnia Ibrahim 
> wrote:
> >>
> >> Hi Tom
> >>
> >>> 1. I assume there's an undocumented requirement in the KIP that
> whatever
> >>> class is named for forwarding.admin.class it has a public single
> argument
> >>> constructor that takes an Admin instance?
> >>>
> >> you are right, I updated the KIP to reflect the missing one.
> >> 3. What if the implementation needs to distinguish between creating
> ACLs on
> >> the source cluster and creating them on the destination cluster? E.g.
> the
> >> one should be done one way, but the other using a different mechanism?
> >> Users will need to define 2 implementations, one for source and another
> for
> >> target and configure each using .forwarding.admin.class.
> for
> >> example
> >> - target.forwarding.admin.class = TargetAdmin
> >> - source.forwarding.admin.class = SourceAdmin
> >>
> >>> On Tue, Jun 21, 2022 at 2:14 PM Tom Bentley 
> wrote:
> >>>
> >>> Hi Omnia,
> >>>
> >>> Thanks for the KIP! I'm sorry for the delay in this response. I have a
> >> few
> >>> questions:
> >>>
> >>> 1. I assume there's an undocumented requirement in the KIP that
> whatever
> >>> class is named for forwarding.admin.class it has a public single
> argument
> >>> constructor that takes an Admin instance?
> >>> 2. If 1 is correct then what about an implementation that requires
> extra
> >>> configuration, e.g. for whatever infra-as-code API it needs to use
> >> (instead
> >>> of using an Admin instance directly) how does it learn about that
> >> non-Kafka
> >>> config when it's only receiving an Admin instance?
> >>> 3. What if the implementation needs to distinguish between creating
> ACLs
> >> on
> >>> the source cluster and creating them on the destination cluster? E.g.
> the
> >>> one should be done one way, but the other using a different mechanism?
> >>>
> >>> Kind regards,
> >>>
> >>> Tom
> >>>
> >>>
> >>> On Mon, 20 Jun 2022 at 17:13, Omnia Ibrahim 
> >>> wrote:
> >>>
> >>>> Hi Chris, sorry for the late reply.
> >>>> ..Hi,
> >>>>
> >>>>> 1. I might be missing something, but can you give a concrete Java
> >>> example
> >>>>> of how the proposed ForwardingAdmin class is more convenient than
> >>>>> subclassing the KafkaAdminClient class? AFAICT the two would be
> >>> virtually
> >>>>> identical.
> >>>>
> >>>> I might be misunderstanding exactly how that class will be used;
> >>>>> I'm envisioning it as the pluggable class that users will implement
> >> for
> >>>>> custom administration logic and specify as the value for the
> >>>>> "forwarding.admin.class" (or ".forwarding.admin.class")
> >>>> property,
> >>>>> and that it will be instantiated with a KafkaAdminClient instance
> >> that
> >>>> can
> >>>>> be used to get the 

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Tom Bentley]

Hi Omnia,

Thanks for those answers. I'm a bit confused by the changes you've made for
1 though. It's MM2 that's going to instantiate the class named in
forwarding.admin.class, so it's MM2 that needs to know the constructor
signature. The easiest way of doing this is to specify the signature as
part of the contract for forwarding.admin.class. But ForwardingAdmin now
has two constructors, which is confusing for someone who wants to subclass
it. Surely only one is needed? It would also be a good idea to show the
documentation that forward.admin.class will have.

Kind regards,

Tom

On Tue, 21 Jun 2022 at 17:06, Omnia Ibrahim  wrote:

> Hi Tom
>
> > 1. I assume there's an undocumented requirement in the KIP that whatever
> > class is named for forwarding.admin.class it has a public single argument
> > constructor that takes an Admin instance?
> >
> you are right, I updated the KIP to reflect the missing one.
> 3. What if the implementation needs to distinguish between creating ACLs on
> the source cluster and creating them on the destination cluster? E.g. the
> one should be done one way, but the other using a different mechanism?
> Users will need to define 2 implementations, one for source and another for
> target and configure each using .forwarding.admin.class. for
> example
> - target.forwarding.admin.class = TargetAdmin
> - source.forwarding.admin.class = SourceAdmin
>
> On Tue, Jun 21, 2022 at 2:14 PM Tom Bentley  wrote:
>
> > Hi Omnia,
> >
> > Thanks for the KIP! I'm sorry for the delay in this response. I have a
> few
> > questions:
> >
> > 1. I assume there's an undocumented requirement in the KIP that whatever
> > class is named for forwarding.admin.class it has a public single argument
> > constructor that takes an Admin instance?
> > 2. If 1 is correct then what about an implementation that requires extra
> > configuration, e.g. for whatever infra-as-code API it needs to use
> (instead
> > of using an Admin instance directly) how does it learn about that
> non-Kafka
> > config when it's only receiving an Admin instance?
> > 3. What if the implementation needs to distinguish between creating ACLs
> on
> > the source cluster and creating them on the destination cluster? E.g. the
> > one should be done one way, but the other using a different mechanism?
> >
> > Kind regards,
> >
> > Tom
> >
> >
> > On Mon, 20 Jun 2022 at 17:13, Omnia Ibrahim 
> > wrote:
> >
> > > Hi Chris, sorry for the late reply.
> > > ..Hi,
> > >
> > > > 1. I might be missing something, but can you give a concrete Java
> > example
> > > > of how the proposed ForwardingAdmin class is more convenient than
> > > > subclassing the KafkaAdminClient class? AFAICT the two would be
> > virtually
> > > > identical.
> > >
> > > I might be misunderstanding exactly how that class will be used;
> > > > I'm envisioning it as the pluggable class that users will implement
> for
> > > > custom administration logic and specify as the value for the
> > > > "forwarding.admin.class" (or ".forwarding.admin.class")
> > > property,
> > > > and that it will be instantiated with a KafkaAdminClient instance
> that
> > > can
> > > > be used to get the same logic that MM2 provides today. In the case
> you
> > > > mentioned (KafkaAdminClient for read/describe, custom Admin for
> > > > create/update), I'd imagine one could override the createTopics,
> > > > deleteTopics, createAcls, deleteAcls (maybe?), alterConfigs (maybe?),
> > > etc.
> > > > methods, and then leave other methods such as listTopics,
> > describeTopics,
> > > > describeCluster, etc. as they are.
> > > >
> > >  The Forwarding decorator is one alternative for inheritance so you are
> > > right to say that they look identical. However, I would like to point
> out
> > > few points
> > > 1. that Kafka codebase has few wrappers or decorators around
> AdminClient
> > > instead of inheritance so using decorator over inheritance isn't new
> > > proposal for example
> > >
> > >- -
> > `org.apache.kafka.streams.processor.internals.InternalTopicManager`
> > >which has AdminClient as parameter instead of inheatance
> > >- - `org.apache.kafka.connect.util.SharedTopicAdmin` and
> > >`org.apache.kafka.connect.util.TopicAdmin` don't inherit
> > > KafkaAdminClient
> > >instead initialize KafkaAdminClient.
> > >
> > > 2. Usi

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Tom Bentley]

Hi Omnia,

Thanks for the KIP! I'm sorry for the delay in this response. I have a few
questions:

1. I assume there's an undocumented requirement in the KIP that whatever
class is named for forwarding.admin.class it has a public single argument
constructor that takes an Admin instance?
2. If 1 is correct then what about an implementation that requires extra
configuration, e.g. for whatever infra-as-code API it needs to use (instead
of using an Admin instance directly) how does it learn about that non-Kafka
config when it's only receiving an Admin instance?
3. What if the implementation needs to distinguish between creating ACLs on
the source cluster and creating them on the destination cluster? E.g. the
one should be done one way, but the other using a different mechanism?

Kind regards,

Tom


On Mon, 20 Jun 2022 at 17:13, Omnia Ibrahim  wrote:

> Hi Chris, sorry for the late reply.
> ..Hi,
>
> > 1. I might be missing something, but can you give a concrete Java example
> > of how the proposed ForwardingAdmin class is more convenient than
> > subclassing the KafkaAdminClient class? AFAICT the two would be virtually
> > identical.
>
> I might be misunderstanding exactly how that class will be used;
> > I'm envisioning it as the pluggable class that users will implement for
> > custom administration logic and specify as the value for the
> > "forwarding.admin.class" (or ".forwarding.admin.class")
> property,
> > and that it will be instantiated with a KafkaAdminClient instance that
> can
> > be used to get the same logic that MM2 provides today. In the case you
> > mentioned (KafkaAdminClient for read/describe, custom Admin for
> > create/update), I'd imagine one could override the createTopics,
> > deleteTopics, createAcls, deleteAcls (maybe?), alterConfigs (maybe?),
> etc.
> > methods, and then leave other methods such as listTopics, describeTopics,
> > describeCluster, etc. as they are.
> >
>  The Forwarding decorator is one alternative for inheritance so you are
> right to say that they look identical. However, I would like to point out
> few points
> 1. that Kafka codebase has few wrappers or decorators around AdminClient
> instead of inheritance so using decorator over inheritance isn't new
> proposal for example
>
>- - `org.apache.kafka.streams.processor.internals.InternalTopicManager`
>which has AdminClient as parameter instead of inheatance
>- - `org.apache.kafka.connect.util.SharedTopicAdmin` and
>`org.apache.kafka.connect.util.TopicAdmin` don't inherit
> KafkaAdminClient
>instead initialize KafkaAdminClient.
>
> 2. Using ForwardingAdmin will make it easier to test which methods use
> KafkaAdminClient and which don't this make the test for any customized
> implementation easier. We can't have this with inheatcancs
> 3. Inhearting KafkaAdminClient has the following limitation
>a. KafkaAdminClient doesn't have a public default constructor, so
> isn't gonna be easy to have contractor that initialize both
> KafkaAdminClient (to be used for read/describe/list) and customized
> fedrated client (to be used for create/update). (Note I don't want to touch
> anything in KafkaAdminClient code base to change that)
>   b. the only way to initialize instance is by using`createInternal`
> which is static and can't be overridden to include creating customized
> fedrated client. This is the method used by `Admin.create` to
> initialize `KafkaAdminClient`in
> most MM2 and Kafka codebase.
>
> 3. KIP-158 deals with the topics that source connectors write to, not the
> > internal topics used by the Connect framework. IIUC this includes the
> > topics that MM2 mirrors. I think it's still fine if we want to leave
> > support for this out and say that this KIP only addresses code that lives
> > inside the connect/mirror (and possibly connect/mirror-client?) modules,
> I
> > just want to make sure that whatever behavior we settle on is specified
> > clearly in the KIP and user-facing documentation.
>
> MM2 deals with a large number of topics, creating the topic if not
> existing, adding new partitions if the number of partitions increases and
> syncing configs and ACLs, so while KIP-158 can stop the source connect from
> creating a topic, this wouldn't be a great solution for MM2 when it runs on
> a large scale as this will add too much operation headache on the teams
> that run MM2 to create/update large number of topics manually. Also, it
> prevents teams from enabling MM2 to sync configs and ACLs as this still
> will be at odds with the federated solution.
>
> 4. That's fine  Just out of curiosity, is the motivation for this
> > decision to simplify the implementation? I can imagine it'd be easier to
> > modify the MM2 codebase exclusively and not have to worry about touching
> > the Connect framework as well given the possibility for unintended
> > consequences for other connectors with the latter.
>
>  The motivation of the KIP is to have the option to let MM2 create topics,
> add 

Re: Log4j 1.2 update timeline?

[Tom Bentley]

Hi Michael,

KIP-653 seeks to update the log4jv1 dependency to log4jv2. Following a
discussion a month or so ago [1] I think the plan is to do this in Kafka
4.0. The discussion for KIP-833 [2] suggests we'll see Kafka 3.5 followed
by Kafka 4.0, which should be around August 2023 (assuming we stick to the
usual time-based releases).

If you're asking for log4shell reasons, Kafka 3.2.0 and 3.1.1 use reload4j
instead of the original log4jv1 implementation.

Hopefully this answers your question.

Kind regards,

Tom

[1]: https://lists.apache.org/thread/qo1y3249xldt4cpg6r8zkcq5m1q32bf1
[2]: https://lists.apache.org/thread/90zkqvmmw3y8j6tkgbg3md78m7hs4yn6

On Thu, 2 Jun 2022 at 18:03, Cleveland, Michael <
michael.clevel...@breadfinancial.com> wrote:

> Good morning,
>
> I was wondering if there was a timeline when the version of Log4j will be
> updated when using Kafka?
>
> Thank you,
> Michael Cleveland
>
> __
> The information contained in this e-mail message and any attachments may
> be privileged and confidential. If the reader of this message is not the
> intended recipient or an agent responsible for delivering it to the
> intended recipient, you are hereby notified that any review, dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify the sender
> immediately by replying to this e-mail and delete the message and any
> attachments from your computer.
> __

Re: [VOTE] KIP-830: Allow disabling JMX Reporter

[Tom Bentley]

Hi Mickael,

Thanks for the KIP. +1 (binding).

Kind regards,

Tom

On Thu, 19 May 2022 at 13:14, Mickael Maison 
wrote:

> Hi Ismael,
>
> That's a good idea, I've updated the KIP.
>
> Thanks,
> Mickael
>
> On Tue, May 17, 2022 at 4:17 PM Federico Valeri 
> wrote:
> >
> > +1 (non binding)
> >
> > Thanks.
> >
> > On Tue, May 17, 2022 at 3:47 PM Mickael Maison 
> wrote:
> > >
> > > Hi,
> > >
> > > I'd like to start a vote on KIP-830 which proposes a method for a
> > > method for disabling JMXReporter and making JMXReporter behave like
> > > other reporters in the next major release when it will need to be
> > > explicitly listed in metric.reporters to be enabled.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-830%3A+Allow+disabling+JMX+Reporter
> > >
> > > Let me know if you have feedback,
> > >
> > > Thanks,
> > > Mickael
>
>

Re: [VOTE] KIP-831: Add metric for log recovery progress

[Tom Bentley]

Hi Luke,

Thanks for the KIP. +1 (binding).

Kind regards,

Tom

On Fri, 20 May 2022 at 02:58, Raman Verma 
wrote:

> ^^  (non binding)
>
> On Thu, May 19, 2022 at 5:40 PM Raman Verma  wrote:
> >
> > Hello Luke,
> >
> > The proposal looks good to me. Thanks. +1
> >
> >
> > On Tue, May 17, 2022 at 9:26 PM James Cheng 
> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > -James
> > >
> > > Sent from my iPhone
> > >
> > > > On May 16, 2022, at 12:12 AM, Luke Chen  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on KIP to expose metrics for log recovery
> > > > progress. These metrics would let the admins have a way to monitor
> the log
> > > > recovery progress.
> > > >
> > > > Details can be found here:
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
> > > >
> > > > Any feedback is appreciated.
> > > >
> > > > Thank you.
> > > > Luke
> >
> >
> >
> > --
> > Best Regards,
> > Raman Verma
>
>
>
> --
> Best Regards,
> Raman Verma
>
>

Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API

[Tom Bentley]

Hi Mickael,

Thanks for the KIP! +1 (binding).

Kind regards,

Tom

On Thu, 19 May 2022 at 11:28, Federico Valeri  wrote:

> Thanks Mickael.
>
> +1 (non binding)
>
> On Wed, May 18, 2022 at 11:08 AM Divij Vaidya 
> wrote:
> >
> > +1 non binding.
> >
> > Divij Vaidya
> >
> >
> >
> > On Tue, May 17, 2022 at 6:16 PM Igor Soarez  wrote:
> >
> > > Thanks for this KIP Mickael.
> > >
> > > +1 non binding
> > >
> > > --
> > > Igor
> > >
> > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote:
> > > > Hi Mickael,
> > > >
> > > > +1 (binding) from me.
> > > > Thanks for the KIP!
> > > >
> > > > Luke
> > > >
> > > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison <
> mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I'd like to start a vote on KIP-827. It proposes exposing the total
> > > >> and usable space of logdirs
> > > >> via the DescribeLogDirs API:
> > > >>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > >
>
>

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Tom Bentley]

Thanks Luke! LGTM.

On Sun, 22 May 2022 at 05:18, Luke Chen  wrote:

> Hi Tom and Raman,
>
> Thanks for your comments.
>
> > 1. There's not a JIRA for this KIP (or the JIRA link needs updating).
> 2. Similarly the link to this discussion thread needs updating.
> > Please update the links to JIRA and the discussion thread.
>
> Yes, thanks for the reminder. I've updated the KIP.
>
> > 3. I wonder whether we need to keep these metrics (with value 0) once the
> broker enters the running state. Do you see it as valuable? A benefit of
> removing the metrics would be a reduction on storage required for metric
> stores which are recording these metrics.
>
> Yes, removing the metrics after log recovery completed is a good idea.
> Updated the KIP.
>
> > 4. I think the KIP's public interfaces section could be a bit clearer.
> Previous KIPs which added metrics usually used a table, with the MBean
> name, metric type and description. SeeKIP-551 for example (or KIP-748,
> KIP-608). Similarly you could use a table in the proposed changes section
> rather than describing the tree you'd see in an MBean console.
>
> Good point! Updated the KIP to use a table to list the MBean name, metric
> type and descriptions.
>
>
> Thank you.
> Luke
>
> On Fri, May 20, 2022 at 9:13 AM Raman Verma 
> wrote:
>
> > Hi Luke,
> >
> > The change is useful and simple. Thanks.
> > Please update the links to JIRA and the discussion thread.
> >
> > Best Regards,
> > Raman Verma
> >
> > On Thu, May 19, 2022 at 8:57 AM Tom Bentley  wrote:
> > >
> > > Hi Luke,
> > >
> > > Thanks for the KIP. I think the idea makes sense and would provide
> useful
> > > observability of log recovery. I have a few comments.
> > >
> > > 1. There's not a JIRA for this KIP (or the JIRA link needs updating).
> > > 2. Similarly the link to this discussion thread needs updating.
> > > 3. I wonder whether we need to keep these metrics (with value 0) once
> the
> > > broker enters the running state. Do you see it as valuable? A benefit
> of
> > > removing the metrics would be a reduction on storage required for
> metric
> > > stores which are recording these metrics.
> > > 4. I think the KIP's public interfaces section could be a bit clearer.
> > > Previous KIPs which added metrics usually used a table, with the MBean
> > > name, metric type and description. SeeKIP-551 for example (or KIP-748,
> > > KIP-608). Similarly you could use a table in the proposed changes
> section
> > > rather than describing the tree you'd see in an MBean console.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > > On Wed, 11 May 2022 at 09:08, Luke Chen  wrote:
> > >
> > > > > And if people start using RemainingLogs and RemainingSegments and
> > then
> > > > REALLY FEEL like they need RemainingBytes, then we can always add it
> > in the
> > > > future.
> > > >
> > > > +1
> > > >
> > > > Thanks James!
> > > > Luke
> > > >
> > > > On Wed, May 11, 2022 at 3:57 PM James Cheng 
> > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > Thanks for the detailed explanation. I agree that the current
> > proposal of
> > > > > RemainingLogs and RemainingSegments will greatly improve the
> > situation,
> > > > and
> > > > > that we can go ahead with the KIP as is.
> > > > >
> > > > > If RemainingBytes were straight-forward to implement, then I’d like
> > to
> > > > > have it. But we can live without it for now. And if people start
> > using
> > > > > RemainingLogs and RemainingSegments and then REALLY FEEL like they
> > need
> > > > > RemainingBytes, then we can always add it in the future.
> > > > >
> > > > > Thanks Luke, for the detailed explanation, and for responding to my
> > > > > feedback!
> > > > >
> > > > > -James
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On May 10, 2022, at 6:48 AM, Luke Chen 
> wrote:
> > > > > >
> > > > > > Hi James and all,
> > > > > >
> > > > > > I checked again and I can see when creating UnifiedLog, we
> > expected the
> > > > > > logs/indexes/snapshots are in good state.
> > > &

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Tom Bentley]

Hi Luke,

Thanks for the KIP. I think the idea makes sense and would provide useful
observability of log recovery. I have a few comments.

1. There's not a JIRA for this KIP (or the JIRA link needs updating).
2. Similarly the link to this discussion thread needs updating.
3. I wonder whether we need to keep these metrics (with value 0) once the
broker enters the running state. Do you see it as valuable? A benefit of
removing the metrics would be a reduction on storage required for metric
stores which are recording these metrics.
4. I think the KIP's public interfaces section could be a bit clearer.
Previous KIPs which added metrics usually used a table, with the MBean
name, metric type and description. SeeKIP-551 for example (or KIP-748,
KIP-608). Similarly you could use a table in the proposed changes section
rather than describing the tree you'd see in an MBean console.

Kind regards,

Tom

On Wed, 11 May 2022 at 09:08, Luke Chen  wrote:

> > And if people start using RemainingLogs and RemainingSegments and then
> REALLY FEEL like they need RemainingBytes, then we can always add it in the
> future.
>
> +1
>
> Thanks James!
> Luke
>
> On Wed, May 11, 2022 at 3:57 PM James Cheng  wrote:
>
> > Hi Luke,
> >
> > Thanks for the detailed explanation. I agree that the current proposal of
> > RemainingLogs and RemainingSegments will greatly improve the situation,
> and
> > that we can go ahead with the KIP as is.
> >
> > If RemainingBytes were straight-forward to implement, then I’d like to
> > have it. But we can live without it for now. And if people start using
> > RemainingLogs and RemainingSegments and then REALLY FEEL like they need
> > RemainingBytes, then we can always add it in the future.
> >
> > Thanks Luke, for the detailed explanation, and for responding to my
> > feedback!
> >
> > -James
> >
> > Sent from my iPhone
> >
> > > On May 10, 2022, at 6:48 AM, Luke Chen  wrote:
> > >
> > > Hi James and all,
> > >
> > > I checked again and I can see when creating UnifiedLog, we expected the
> > > logs/indexes/snapshots are in good state.
> > > So, I don't think we should break the current design to expose the
> > > `RemainingBytesToRecovery`
> > > metric.
> > >
> > > If there is no other comments, I'll start a vote within this week.
> > >
> > > Thank you.
> > > Luke
> > >
> > >> On Fri, May 6, 2022 at 6:00 PM Luke Chen  wrote:
> > >>
> > >> Hi James,
> > >>
> > >> Thanks for your input.
> > >>
> > >> For the `RemainingBytesToRecovery` metric proposal, I think there's
> one
> > >> thing I didn't make it clear.
> > >> Currently, when log manager start up, we'll try to load all logs
> > >> (segments), and during the log loading, we'll try to recover logs if
> > >> necessary.
> > >> And the logs loading is using "thread pool" as you thought.
> > >>
> > >> So, here's the problem:
> > >> All segments in each log folder (partition) will be loaded in each log
> > >> recovery thread, and until it's loaded, we can know how many segments
> > (or
> > >> how many Bytes) needed to recover.
> > >> That means, if we have 10 partition logs in one broker, and we have 2
> > log
> > >> recovery threads (num.recovery.threads.per.data.dir=2), before the
> > >> threads load the segments in each log, we only know how many logs
> > >> (partitions) we have in the broker (i.e. RemainingLogsToRecover
> metric).
> > >> We cannot know how many segments/Bytes needed to recover until each
> > thread
> > >> starts to load the segments under one log (partition).
> > >>
> > >> So, the example in the KIP, it shows:
> > >> Currently, there are still 5 logs (partitions) needed to recover under
> > >> /tmp/log1 dir. And there are 2 threads doing the jobs, where one
> thread
> > has
> > >> 1 segments needed to recover, and the other one has 3 segments
> > needed
> > >> to recover.
> > >>
> > >>   - kafka.log
> > >>  - LogManager
> > >> - RemainingLogsToRecover
> > >>- /tmp/log1 => 5← there are 5 logs under
> > >>/tmp/log1 needed to be recovered
> > >>- /tmp/log2 => 0
> > >> - RemainingSegmentsToRecover
> > >>- /tmp/log1 ← 2 threads are doing log
> > >>recovery for /tmp/log1
> > >>- 0 => 1 ← there are 1 segments needed to
> be
> > >>   recovered for thread 0
> > >>   - 1 => 3
> > >>   - /tmp/log2
> > >>   - 0 => 0
> > >>   - 1 => 0
> > >>
> > >>
> > >> So, after a while, the metrics might look like this:
> > >> It said, now, there are only 4 logs needed to recover in /tmp/log1,
> and
> > >> the thread 0 has 9000 segments left, and thread 1 has 5 segments left
> > >> (which should imply the thread already completed 2 logs recovery in
> the
> > >> period)
> > >>
> > >>   - kafka.log
> > >>  - LogManager
> > >> - RemainingLogsToRecover
> > >>- /tmp/log1 => 3← there are 3 logs under
> > >>/tmp/log1 

[jira] [Created] (KAFKA-13898) metrics.recording.level is underdocumented

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13898:
---

 Summary: metrics.recording.level is underdocumented
 Key: KAFKA-13898
 URL: https://issues.apache.org/jira/browse/KAFKA-13898
 Project: Kafka
  Issue Type: Improvement
  Components: docs
Reporter: Tom Bentley


metrics.recording.level is only briefly described in the documentation. In 
particular the recording level associated with each metric is not documented, 
which makes it difficult to know the effect of changing the level.





--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13897) Add 3.1.1 to system tests and streams upgrade tests

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13897:
---

 Summary: Add 3.1.1 to system tests and streams upgrade tests
 Key: KAFKA-13897
 URL: https://issues.apache.org/jira/browse/KAFKA-13897
 Project: Kafka
  Issue Type: Task
  Components: streams, system tests
Reporter: Tom Bentley
 Fix For: 3.3.0, 3.1.2, 3.2.1


Per the penultimate bullet on the [release 
checklist|https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses],
 Kafka v3.1.1 is released. We should add this version to the system tests.





--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[ANNOUNCE] Apache Kafka 3.1.1

[Tom Bentley]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.1.1

Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
since 3.1.0.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.1.1/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.13 and 2.12) from:
https://kafka.apache.org/downloads#3.1.1

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 28 contributors to this release!

Bounkong Khamphousone, Chris Egerton, David Jacot, dengziming, Ed B,
Edwin, Idan Kamara, Ismael Juma, Jason Gustafson, Jules Ivanic, Justin
Lee, Justine Olshan, Konstantine Karantasis, Kvicii, Luke Chen, Marc
Löhe, Matthias J. Sax, Mike Lothian, Philip Nee, prince-mahajan,
Randall Hauch, Stanislav Vodetskyi, sunshujie1990, Tom Bentley,
Vincent Jiang, Xiaobing Fang, Xiaoyue Xue, Yang Yu

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,
Tom

[jira] [Created] (KAFKA-13895) Fix javadocs build with JDK < 12

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13895:
---

 Summary: Fix javadocs build with JDK < 12
 Key: KAFKA-13895
 URL: https://issues.apache.org/jira/browse/KAFKA-13895
 Project: Kafka
  Issue Type: Task
  Components: docs
Reporter: Tom Bentley


While doing the "Website update process" in the 3.1.1 release I found that I'd 
broken the javadoc search functionality due to having build the Java docs with 
Java 11. Java < 12 [a bug|https://bugs.openjdk.java.net/browse/JDK-8215291] 
that means the javadoc search functionality adds /undefined/ in the URL path 
(even though links between pages otherwise work. 

We could fix the build.gradle to use {{-no-module-directories}} when running 
with javadoc < v12, but that will then break the links to the JDK classes 
javadocs from the Kafka javadoc, [as described 
here|https://github.com/spring-projects/spring-security/issues/10944].

Alternatively we could change the release process docs to require building with 
Java 17. While this would fix the problem for the Javadocs published on the 
website, anyone building the javadocs for themselves would still be affected.




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[RESULTS] [VOTE] Release Kafka version 3.1.1

[Tom Bentley]

This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Mickael Maison
* Bill Bejeck
* David Jacot

Committers:
* Luke Chen
* Dongjoon Hyun

Community:
* Jakub Scholz
* Michal Tóth

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://lists.apache.org/thread/jrs4ws0m69hn22lq2kq56cqg53glqgbm

I'll continue with the release process and the release announcement will
follow in the next few days.

Tom

Re: [VOTE] 3.1.1 RC1

[Tom Bentley]

Hi,

Not having heard from any of the existing voters, I will go ahead and
complete the release process.

Thanks Chris, I will do that and also open JIRAs for any other flaky tests
which don't already have them.

Dongjoon, thanks for your patience and sorry for this having been a longer
than usual release process.

Kind regards,

Tom

On Thu, 12 May 2022 at 06:28, Dongjoon Hyun  wrote:

> Thank you for sharing your assessment, Tom and Chris.
>
> Apache Spark community had a discussion thread on March.
>
> https://lists.apache.org/thread/yq6f5gv7gtdbk1ynwpd9hnc547bgz03m
> "[DISCUSS] Migration guide on upgrading Kafka to 3.1 in Spark 3.3"
>
> As a person who initiated that upgrade via SPARK-36837, I need to answer
> it before Apache Spark 3.3 RC2. In the worst case, Apache Spark community
> may decide to wait for next Kafka versions instead of upgrading.
>
> Thank you again all people to help this release!
>
> Bests,
> Dongjoon.
>
> On 2022/05/12 01:39:16 Chris Egerton wrote:
> > It's worth noting that one of the most frequent offenders
> > (RebalanceSourceConnectorsIntegrationTest.testDeleteConnector), which
> > failed in five of the nine runs from 111 through 119, is actually
> disabled
> > on 3.2 and trunk as it's caused by a known and not-yet-addressed bug in
> > rebalancing logic. Since it's not a regression we can safely ignore those
> > failures, but if we plan on putting out another 3.1 release we should
> > probably disable that test on 3.1 as well.
> >
> > On Wed, May 11, 2022 at 3:32 AM Tom Bentley  wrote:
> >
> > > Hi Dongjoon,
> > >
> > > I've been trying to get a green build of the 3.1 branch from Jenkins (
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/) since my
> > > original email promised to provide this. Individually each stage
> between
> > > builds 111 and 119 has passed in at least one run, but there have been
> no
> > > runs where all stages have passed.
> > >
> > > However, as you note, the vote has the necessary 3 binding votes, so
> > > assuming the voters don't wish to change their votes I can proceed
> with the
> > > release. If this is not the case please respond on the thread. If I
> hear
> > > nothing in the next 24 hours I will assume the voters are OK with this
> and
> > > proceed with the rest of the release process.
> > >
> > > Apologies for the delay,
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > >
> > > On Wed, 11 May 2022 at 08:15, Dongjoon Hyun 
> wrote:
> > >
> > > > Hi, Tom.
> > > >
> > > > Could you conclude this vote as a release manager? :)
> > > >
> > > > Dongjoon.
> > > >
> > > > On 2022/05/06 13:31:15 Michal Tóth wrote:
> > > > > Hello
> > > > >
> > > > > I have executed some produce/consume system tests which all passed.
> > > > > Also everything passed from
> > > > https://github.com/tombentley/kafka-verify-rc
> > > > > - checking signatures, checksums, with gradle unit & integration
> tests,
> > > > etc.
> > > > >
> > > > > Good from me (non-binding).
> > > > >
> > > > >
> > > > >
> > > > > pi 6. 5. 2022 o 14:30 David Jacot 
> > > > napísal(a):
> > > > >
> > > > > > Thanks for running the release, Tom.
> > > > > >
> > > > > > I performed the following validations:
> > > > > > * Verified all checksums and signatures.
> > > > > > * Built from source and ran unit tests.
> > > > > > * Ran the first quickstart steps for both ZK and KRaft.
> > > > > > * Spotchecked the Javadocs.
> > > > > >
> > > > > > I noticed the same issues as others on the website. I checked
> > > > > > the doc in git and it looks good.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun <
> dongj...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > RC1 was tested with Apache Spark tests
> > > > > > >
> > > > > > > - https://gith

Re: [VOTE] 3.1.1 RC1

[Tom Bentley]

Hi Dongjoon,

I've been trying to get a green build of the 3.1 branch from Jenkins (
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/) since my
original email promised to provide this. Individually each stage between
builds 111 and 119 has passed in at least one run, but there have been no
runs where all stages have passed.

However, as you note, the vote has the necessary 3 binding votes, so
assuming the voters don't wish to change their votes I can proceed with the
release. If this is not the case please respond on the thread. If I hear
nothing in the next 24 hours I will assume the voters are OK with this and
proceed with the rest of the release process.

Apologies for the delay,

Kind regards,

Tom


On Wed, 11 May 2022 at 08:15, Dongjoon Hyun  wrote:

> Hi, Tom.
>
> Could you conclude this vote as a release manager? :)
>
> Dongjoon.
>
> On 2022/05/06 13:31:15 Michal Tóth wrote:
> > Hello
> >
> > I have executed some produce/consume system tests which all passed.
> > Also everything passed from
> https://github.com/tombentley/kafka-verify-rc
> > - checking signatures, checksums, with gradle unit & integration tests,
> etc.
> >
> > Good from me (non-binding).
> >
> >
> >
> > pi 6. 5. 2022 o 14:30 David Jacot 
> napísal(a):
> >
> > > Thanks for running the release, Tom.
> > >
> > > I performed the following validations:
> > > * Verified all checksums and signatures.
> > > * Built from source and ran unit tests.
> > > * Ran the first quickstart steps for both ZK and KRaft.
> > > * Spotchecked the Javadocs.
> > >
> > > I noticed the same issues as others on the website. I checked
> > > the doc in git and it looks good.
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > David
> > >
> > > On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun 
> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > RC1 was tested with Apache Spark tests
> > > >
> > > > - https://github.com/apache/spark/pull/36135
> > > >
> > > > Thanks,
> > > > Dongjoon.
> > > >
> > > > On 2022/05/05 03:25:25 Luke Chen wrote:
> > > > > Hi Tom,
> > > > >
> > > > > I did:
> > > > > 1. check the signature and checksums
> > > > > 2. ran quick start with java17 + sacla2.12
> > > > > 3. browse java docs/documentations
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks for running the release.
> > > > >
> > > > > Luke
> > > > >
> > > > > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck 
> > > wrote:
> > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > Thanks for running the release!
> > > > > >
> > > > > > I did the following checks:
> > > > > >
> > > > > >1. Validated all the checksum and signatures
> > > > > >2. Built from source and ran the unit tests (Java 11)
> > > > > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > > > > >4. Did a quick scan of the Javadoc and the documentation
> > > > > >
> > > > > > I noticed the same issues as Mickael on the website, but
> otherwise,
> > > it
> > > > > > looks good.
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Bill
> > > > > >
> > > > > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz 
> > > wrote:
> > > > > >
> > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and staged
> Maven
> > > > > > artifacts
> > > > > > > and ran various tests with them. Thanks for doing the release.
> > > > > > >
> > > > > > > Jakub
> > > > > > >
> > > > > > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison <
> > > mimai...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > Thanks for running this release!
> > > > > > > >
> > > > > > > > I've done the following:
> > > > > > >

[jira] [Created] (KAFKA-13882) Dockerfile for previewing website

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13882:
---

 Summary: Dockerfile for previewing website
 Key: KAFKA-13882
 URL: https://issues.apache.org/jira/browse/KAFKA-13882
 Project: Kafka
  Issue Type: Task
  Components: docs, website
Reporter: Tom Bentley


Previewing changes to the website/documentation is rather difficult because you 
either have to [hack with the 
HTML|https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes#ContributingWebsiteDocumentationChanges-KafkaWebsiteRepository]
 or [install 
httpd|https://cwiki.apache.org/confluence/display/KAFKA/Setup+Kafka+Website+on+Local+Apache+Server].
 This is a barrier to contribution.

Having a Dockerfile for previewing the Kafka website (i.e. with httpd properly 
set up) would make it easier for people to contribute website/docs changes.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13881) Add package.java for public package javadoc

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13881:
---

 Summary: Add package.java for public package javadoc
 Key: KAFKA-13881
 URL: https://issues.apache.org/jira/browse/KAFKA-13881
 Project: Kafka
  Issue Type: Task
Reporter: Tom Bentley


Our public javadoc ([https://kafka.apache.org/31/javadoc/index.html)] doesn't 
have any package descriptions, which is a bit intimidating for anyone who is 
new to the project.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[VOTE] 3.1.1 RC1

[Tom Bentley]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.1.1.

Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
since 3.1.0.

Release notes for the 3.1.1 release:
https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html

*** Please download, test and vote by 09:00 UTC, Friday 6th May

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~tombentley/kafka-3.1.1-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/

* Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
https://github.com/apache/kafka/releases/tag/3.1.1-rc1

* Documentation:
https://kafka.apache.org/31/documentation.html

* Protocol:
https://kafka.apache.org/31/protocol.html

* Successful Jenkins builds for the 3.1 branch:
I will share a link one the build is complete.

/**

Thanks,
Tom

Re: [VOTE] 3.1.1 RC0

[Tom Bentley]

Hi Dongjoon,

I apologise, I should have been a bit more communicative. I was waiting for
a better fix to the issue previously highlighted by David. Ismael has
kindly provided a patch [1], so I will roll RC1 once this is merged and
cherry-picked.

Kind regards,

Tom

[1]: https://github.com/apache/kafka/pull/12096


On Thu, 28 Apr 2022 at 04:38, Luke Chen  wrote:

> Hi Dongjoon,
>
> The Apache Kafka community doesn't recommend users to use which version of
> Kafka.
> The 2 releases v3.1.1 and v3.2.0 are running in parallel, and we don't
> guarantee which version will be released earlier.
>
> Thank you.
> Luke
>
>
>
>
> On Thu, Apr 28, 2022 at 4:54 AM Dongjoon Hyun  wrote:
>
> > Hi, All.
> >
> > It seems that Apache Kafka 3.2.0 RC0 vote started already instead of
> > Apache Kafka 3.1.1 release.
> >
> > Does Apache Kafka community recommend to use Apache Kafka 3.2.0 instead
> of
> > Apache Kafka 3.1.1?
> >
> > Dongjoon.
> >
> > On 2022/04/14 01:00:40 Ismael Juma wrote:
> > > I added a comment to that PR. Let's figure out if we need an additional
> > > change before doing the next RC.
> > >
> > > Ismael
> > >
> > > On Tue, Apr 12, 2022 at 7:47 PM Luke Chen  wrote:
> > >
> > > > Thanks for pointing that out, David.
> > > > +1 to include this PR since we've already included the first fix for
> > > > KAFKA-13794, and this is a follow up fix for it.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Wed, Apr 13, 2022 at 2:31 AM David Jacot
> > 
> > > > wrote:
> > > >
> > > > > Hi Tom,
> > > > >
> > > > > Thanks for running the release. I wonder if we should include:
> > > > >
> > > > >
> > > >
> >
> https://github.com/apache/kafka/commit/134c432d6452de1bfb99d0f6b455a58c16bc626a
> > > > > .
> > > > >
> > > > > This is a follow up of KAFKA-13794. What do you think?
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Fri, Apr 8, 2022 at 6:18 PM Tom Bentley 
> > wrote:
> > > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > > > >
> > > > > > Apache Kafka 3.1.1 is a bugfix release and 29 issues have been
> > fixed
> > > > > > since 3.1.0.
> > > > > >
> > > > > > Release notes for the 3.1.1 release:
> > > > > >
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Friday 15 April, 12:00 UTC
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc0
> > > > > >
> > > > > > * Documentation:
> > > > > > https://kafka.apache.org/31/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > https://kafka.apache.org/31/protocol.html
> > > > > >
> > > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > > I will share a link one the build is complete.
> > > > > >
> > > > > > /**
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Tom
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-797: Accept duplicate listener on port for IPv4/IPv6

[Tom Bentley]

Hi Matthew,

Thanks for the KIP, +1 (binding).

Kind regards,

Tom

On Thu, 14 Apr 2022 at 12:15, Matthew de Detrich
 wrote:

> Hi David,
>
> Thanks for the response.
>
> > 1. In the public interface section, could we spell out
> the configurations that we are changing with this
> KIP? The name does not change but the semantic is
> so it is good to be clear.
>
> Done
>
> > 2. In the proposed changes section, I would rather
> mention the configuration that we need to change the
> validation for instead of saying "loosening the validation
> on listenerListToEndPoints in kafka.utils.CoreUtils.scala"
> as this is specific to the implementation.
>
> This is already done with examples later down in the same section, or am I
> missing something? Would you like me to just remove the
> kafka.utils.CoreUtils.scala reference so its not implying an implementation
> detail?
>
> > 3. For my understanding, using the same port with two
> different DNS entries would fail, right? e.g.
> "PLAINTEXT://foo:9092,PLAINTEXT://bar:9092"
>
> Correct, the idea is that it checks that the listener host is an IP address
> and if it's not then it doesn't even consider it (i.e. it short circuits to
> what is current behaviour). The proposed KIP changes only apply if
> hostnames in the listener are IP address's otherwise no change is
> observable.
>
> Regards
>
> On Mon, Feb 21, 2022 at 10:42 AM David Jacot 
> wrote:
>
> > Hi Matthew,
> >
> > Thanks for the KIP. I have a few minor comments:
> >
> > 1. In the public interface section, could we spell out
> > the configurations that we are changing with this
> > KIP? The name does not change but the semantic is
> > so it is good to be clear.
> >
> > 2. In the proposed changes section, I would rather
> > mention the configuration that we need to change the
> > validation for instead of saying "loosening the validation
> > on listenerListToEndPoints in kafka.utils.CoreUtils.scala"
> > as this is specific to the implementation.
> >
> > 3. For my understanding, using the same port with two
> > different DNS entries would fail, right? e.g.
> > "PLAINTEXT://foo:9092,PLAINTEXT://bar:9092"
> >
> > Best,
> > David
> >
> > On Fri, Feb 11, 2022 at 10:35 AM Luke Chen  wrote:
> > >
> > > Hi Matthew,
> > >
> > > Thanks for the update.
> > > I'm +1 (binding)
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Feb 11, 2022 at 3:32 PM Matthew de Detrich
> > >  wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > I have just updated the KIP with the changes you requested.
> > > >
> > > > Regards
> > > >
> > > > On Fri, Feb 11, 2022 at 4:47 AM Luke Chen  wrote:
> > > >
> > > > > Hi Matthew,
> > > > >
> > > > > I checked again the KIP, and it LGTM.
> > > > >
> > > > > Just a minor comment:
> > > > > Maybe add some examples into the KIP to show how users can set both
> > IPv4
> > > > > and IPv6 on the same port.
> > > > > And some examples to show how the validation will fail like you
> > listed in
> > > > > `Proposed Changes`.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > >
> > > > > On Fri, Feb 11, 2022 at 8:54 AM Matthew de Detrich
> > > > >  wrote:
> > > > >
> > > > > > Hello everyone
> > > > > >
> > > > > > I have just updated/rebased the PR against the latest Kafka
> trunk.
> > Let
> > > > me
> > > > > > know if anything else is required/missing.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > On Thu, Jan 13, 2022 at 10:28 AM Matthew de Detrich <
> > > > > > matthew.dedetr...@aiven.io> wrote:
> > > > > >
> > > > > > > Does anyone have any additional comments/regards to help get
> > this PR
> > > > > > voted
> > > > > > > through?
> > > > > > >
> > > > > > > On Tue, Nov 23, 2021 at 7:46 AM Josep Prat
> > > >  > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Matthew,
> > > > > > >>
> > > > > > >> Thank you for the PR.
> > > > > > >>
> > > > > > >> +1 (non binding) from my side.
> > > > > > >>
> > > > > > >>
> > > > > > >> Best,
> > > > > > >>
> > > > > > >> ———
> > > > > > >> Josep Prat
> > > > > > >>
> > > > > > >> Aiven Deutschland GmbH
> > > > > > >>
> > > > > > >> Immanuelkirchstraße 26, 10405 Berlin
> > > > > > >>
> > > > > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >>
> > > > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > >>
> > > > > > >> m: +491715557497
> > > > > > >>
> > > > > > >> w: aiven.io
> > > > > > >>
> > > > > > >> e: josep.p...@aiven.io
> > > > > > >>
> > > > > > >> On Tue, Nov 23, 2021, 07:11 Ivan Yurchenko <
> > > > ivan0yurche...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi,
> > > > > > >> >
> > > > > > >> > Thank you for the KIP.
> > > > > > >> >
> > > > > > >> > +1 (non-binding)
> > > > > > >> >
> > > > > > >> > Ivan
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, 23 Nov 2021 at 04:18, Luke Chen 
> > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Matthew,
> > > > > > >> > > Thanks for the KIP.
> > > > > > >> > > It makes sense to allow IPv4 and IPv6 listening on the
> same
> 

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Tom Bentley]

Hi Jorge,

Thanks for the KIP, especially for the examples which are super-clear.

100. The name `field.style` isn't so clear for something like ReplaceField:
it's not so obvious that field.style applies to `include` and `exclude`.

101. The permitted values for `field.style` don't seem terribly intuitive
(to me anyway): the meaning of `plain` isn't very guessable. Why not
`top-level` or `root` instead? Also `nested` could be misconstrued to mean
nested-but-not-top-level, so perhaps `recursive` or `cascading` might be
better?

102. I'm torn on whether making the interpretation of existing configs like
`field` be dependent on `field.style` is a good idea. I can see that it's
the simplest thing to do, but it just feels a bit odd that sometimes the
`field` would actually be a path and have different escaping rules. An
alternative would be to come up with a parallel set of config names (e.g.
as well as "field" an SMT might support "path") which were defined to
always take paths, thus avoiding the changeable interpretation of the
existing configs. The SMT's #configure() would need to throw in the case
that both configs were given. I can see that that would be more work in
implementation, but it feels cleaner.

103. I think in order to allow for supporting arrays in a later KIP (which
certainly seems like it could be useful), we'd want to specify the syntax
now, even if it wasn't implemented under this KIP. That's because I don't
think you can't exclude the possibility that characters such as `[` and `]`
appear in field names. So you'd have a compatibility problem if people
started using the features of this KIP to access such fields, only for
those characters to change their meaning under a later KIP.

104. I also wonder whether making paths into a public Java API, for use by
3rd party SMTs, would be valuable.

Thanks again,

Tom



On Wed, 20 Apr 2022 at 17:53, Chris Egerton  wrote:

>  Thanks Jorge, LGTM!
>
> On Wed, Apr 20, 2022, 12:40 Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Thank you, Chris! Not possible without your feedback.
> >
> > On Tue, 19 Apr 2022 at 23:04, Chris Egerton 
> > wrote:
> >
> > > Hi Jorge,
> > >
> > > Thank you for sticking through this. I have one small remark and one
> > small
> > > clarification; assuming you agree with me on them then I'm ready to
> vote
> > on
> > > the KIP.
> > >
> > > 1. InsertField: The "field.on.missing.parent" and
> > "field.on.existing.field"
> > > docs both mention a permitted value of "ingore"; this should be
> "ignore",
> > > right?
> > >
> >
> > Of course, one more typo :)
> >
> >
> > > 2. InsertField: The examples are still missing the "field.style"
> property
> > > from the configurations. They should all include the property
> > > "transforms.smt1.field.style": "nested", correct?
> > >
> >
> > Yes, it is there. I think I know what you mean now, seems that Confluence
> > is putting everything in one line when it's in separate lines in the
> > editor.
> > Hopefully, it's fixed now.
> >
> >
> > >
> > > Thanks again for working through this, and congratulations on a
> > > well-written KIP!
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Apr 19, 2022 at 2:06 PM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > > > Thank you, Chris! I apply these improvements to the KIP, let me know
> > how
> > > it
> > > > looks now.
> > > >
> > > > On Mon, 11 Apr 2022 at 23:43, Chris Egerton  >
> > > > wrote:
> > > >
> > > > > Hi Jorge,
> > > > >
> > > > > Wow, those examples are great! A few more remarks, but I think
> we're
> > > > > getting close:
> > > > >
> > > > > 1. The examples differ across SMTs with the name of the
> > > newly-introduced
> > > > > style property; some of them use "field.style", and some use
> > > > > "fields.style". I think for consistency's sake we should stick with
> > > just
> > > > > "field.style"; otherwise it could be painful for users to have to
> > > > remember
> > > > > which to use.
> > > > >
> > > >
> > > > Great catch. Agree, I fixed the config names to `field.style`.
> > > >
> > > >
> > > > >
> > > > > 2. Some of the examples are off:
> > > > > - TimestampConverter: the input in the second example ("when field
> > > names
> > > > > include dots") doesn't contain a field with a dotted name
> > > > > - ValueToKey: the config in the third example ("when field names
> > > include
> > > > > dots") should probably use "parent..child.k2" as the
> > > > > "transforms.smt1.fields" property
> > > > >
> > > >
> > > > Fixed. Thanks!
> > > >
> > > >
> > > > >
> > > > > 3. RE changes to InsertField:
> > > > > - The InsertField SMT should also come with the new "field.style"
> > > > property
> > > > > in order to preserve backwards compatibility, right? I don't see it
> > > > > included in the example configs for that one, just want to make
> sure
> > > > > - I don't know of any cases where we use snake_case for property
> > names
> > > in
> > > > > Kafka; we should probably 

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Tom Bentley]

Hi Mickael,

Thanks for the KIP, I can see this would be useful.

I guess you could have used optional tagged fields, rather than bumping the
version, but then again I don't see it being particularly advantageous in
this case either.

Kind regards,

Tom

On Tue, 19 Apr 2022 at 10:23, Divij Vaidya  wrote:

> I have a minor suggestion below but overall KIP looks good to me to start a
> vote.
>
> *Reg#6* Would you consider replacing UNKNOWN_SPACE with UNSUPPORTED?
> UNSUPPORTED tells the user explicitly that the value is missing due to
> client/server version mismatch whereas with UNKNOWN_SPACE, the user is left
> wondering whether it is a problem with underlying storage not providing
> space information or something else.
>
> Divij Vaidya
>
>
>
> On Fri, Apr 15, 2022 at 3:40 PM Mickael Maison 
> wrote:
>
> > Hi Luke,
> >
> > 7. I've updated the KIP to clarify these sizes are in bytes.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, Apr 15, 2022 at 12:16 PM Luke Chen  wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > > This is a good improvement.
> > >
> > > (3) +1 for not adding the number of files in the directory. Counting
> the
> > > file numbers should be slow.
> > > (7) Could you make the fields clear in `DescribeLogDirsResponse`, to
> > > mention the returned number is size in Byte (or not?)
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Apr 15, 2022 at 5:27 PM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > 3. Yes that's right. Also the number of file descriptors is really
> not
> > > > a property of log directories. Administrators typically tracked that
> > > > count per process and for the whole operating system.
> > > >
> > > > 5. That's a good point, I've updated the KIP to mention sizes will be
> > > > capped to Long.MAX_VALUE even if the actual storage is larger.
> > > >
> > > > 6. Brokers would never return UNKNOWN_SPACE. When new clients query
> > > > older brokers via the admin API, the admin client will use
> > > > UNKNOWN_SPACE to indicate these values weren't provided by brokers.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, Apr 8, 2022 at 5:00 PM Divij Vaidya  >
> > > > wrote:
> > > > >
> > > > > Thanks for replying. I still have a few lingering
> questions/comments.
> > > > >
> > > > > *Reg#1* Understood. I checked and the underlying system call is
> > statvfs
> > > > for
> > > > > unix systems which should be ok to call here.
> > > > > *Reg#2* Fair point. I checked again and yes, log.dir always means
> > local
> > > > > storage even when tiered storage is enabled.
> > > > > *Reg#3* The rationale for adding these new (size) fields to the
> > > > > `DescribeLogDirs` is to allow the administrator to monitor or
> perhaps
> > > > take
> > > > > automated action based on results. Doesn't monitoring the number of
> > file
> > > > > descriptors fall in the same category of use cases? I am assuming
> > that we
> > > > > want to add the size information in the API response because JVM
> > makes it
> > > > > possible to get this information in a platform agnostic manner
> which
> > is
> > > > not
> > > > > true for open file descriptors, correct?
> > > > > *Reg#4* Agree.
> > > > > *New#5*: As an FYI, Java FileStore API breaks on large storage
> sizes.
> > > > See:
> > > > > https://bugs.openjdk.java.net/browse/JDK-8162520. ElasticSearch
> has
> > been
> > > > > hit by these limitations in the past. For JDK 11, you will probably
> > have
> > > > to
> > > > > add defensive checks such as
> > > > >
> > > >
> >
> https://github.com/opensearch-project/OpenSearch/blob/b74d71fb747cc2873d4c2ffae825944da4d06e1b/server/src/main/java/org/opensearch/monitor/fs/FsProbe.java#L148
> > > > .
> > > > > The documentation of the API mentioned in KIP will also be modified
> > to
> > > > > account for this edge case.
> > > > > *New#6*: Can you please provide an example where the return for
> these
> > > > APIs
> > > > > would be UNKNOWN_SPACE? Doesn't JVM guarantee that this API will
> > > > definitely
> > > > > return results (else it throws an IOException)? I would propose
> that
> > we
> > > > get
> > > > > rid of default since JVM guarantees that this would work on all
> > > > platforms.
> > > > > If it doesn't then it's a bug and should be uncovered via an
> > exception.
> > > > >
> > > > > Also, I would like to volunteer to code review (of course, it would
> > be
> > > > > non-binding) your implementation once this KIP is approved.
> > > > >
> > > > > Regards,
> > > > > Divij Vaidya
> > > > >
> > > > > On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Divij,
> > > > > >
> > > > > > Thanks for taking a look!
> > > > > >
> > > > > > 1. In order to retrieve the sizes, the plan is to use
> > getTotalSpace()
> > > > > > and getUsableSpace() from java.nio.file.FileStore. The
> > implementations
> > > > > > may 

[VOTE] 3.1.1 RC0

[Tom Bentley]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.1.1.

Apache Kafka 3.1.1 is a bugfix release and 29 issues have been fixed
since 3.1.0.

Release notes for the 3.1.1 release:
https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Friday 15 April, 12:00 UTC

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~tombentley/kafka-3.1.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/

* Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
https://github.com/apache/kafka/releases/tag/3.1.1-rc0

* Documentation:
https://kafka.apache.org/31/documentation.html

* Protocol:
https://kafka.apache.org/31/protocol.html

* Successful Jenkins builds for the 3.1 branch:
I will share a link one the build is complete.

/**

Thanks,

Tom

Re: [DISCUSS] Apache Kafka 3.1.1

[Tom Bentley]

Hi Ismael,

I hope to have an RC0 by Friday.

Cheers,

Tom

On Tue, 29 Mar 2022 at 23:20, Ismael Juma  wrote:

> Hi Tom,
>
> When do you expect to publish the first RC?
>
> Ismael
>
> On Thu, Mar 10, 2022 at 1:27 AM Tom Bentley  wrote:
>
> > I think I was supposed to publish the release plan at the same time as
> > calling the vote for the release.
> > Here it is:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.1.1
> >
> > Kind regards,
> >
> > Tom
> >
> > On Thu, 10 Mar 2022 at 08:49, Bruno Cadonna  wrote:
> >
> > > Thank you, Tom!
> > >
> > > +1
> > >
> > > Best,
> > > Bruno
> > >
> > > On 09.03.22 19:55, David Jacot wrote:
> > > > +1. Thanks Tom!
> > > >
> > > > Le mer. 9 mars 2022 à 19:10, Bill Bejeck  a
> écrit :
> > > >
> > > >> Thanks Tom!  It's a +1 for me.
> > > >>
> > > >> -Bill
> > > >>
> > > >> On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma 
> > wrote:
> > > >>
> > > >>> Thanks Tom. +1
> > > >>>
> > > >>> Ismael
> > > >>>
> > > >>>
> > > >>> On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley 
> > > wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> I'd like to volunteer to be the release manager for the 3.1.1
> bugfix
> > > >>>> release.
> > > >>>>
> > > >>>> Kind regards,
> > > >>>>
> > > >>>> Tom
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

Re: Subscribe to Kafka dev mailing list

[Tom Bentley]

Hi,

To subscribe you need to send email to dev-subscr...@kafka.apache.org, not
dev@

Full instructions: https://kafka.apache.org/contact

Kind regards,

Tom

On Tue, 29 Mar 2022 at 01:29, 那一叶零落 <1664637...@qq.com.invalid> wrote:

> Subscribe to Kafka dev mailing list

Re: [DISCUSS] Apache Kafka 3.1.1

[Tom Bentley]

I think I was supposed to publish the release plan at the same time as
calling the vote for the release.
Here it is:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.1.1

Kind regards,

Tom

On Thu, 10 Mar 2022 at 08:49, Bruno Cadonna  wrote:

> Thank you, Tom!
>
> +1
>
> Best,
> Bruno
>
> On 09.03.22 19:55, David Jacot wrote:
> > +1. Thanks Tom!
> >
> > Le mer. 9 mars 2022 à 19:10, Bill Bejeck  a écrit :
> >
> >> Thanks Tom!  It's a +1 for me.
> >>
> >> -Bill
> >>
> >> On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma  wrote:
> >>
> >>> Thanks Tom. +1
> >>>
> >>> Ismael
> >>>
> >>>
> >>> On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley 
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'd like to volunteer to be the release manager for the 3.1.1 bugfix
> >>>> release.
> >>>>
> >>>> Kind regards,
> >>>>
> >>>> Tom
> >>>>
> >>>
> >>
> >
>
>

[DISCUSS] Apache Kafka 3.1.1

[Tom Bentley]

Hi,

I'd like to volunteer to be the release manager for the 3.1.1 bugfix
release.

Kind regards,

Tom

Re: [VOTE] 3.0.1 RC0

[Tom Bentley]

Hi Mickael,

In addition to the results others have posted I've also validated the
checksums and keys, built from source and executed the unit and integration
tests.

Overall I'm +1 (binding).

Thanks,

Tom

On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:

> Hello,
>
>  executed https://github.com/tombentley/kafka-verify-rc - with no issues.
> All checks have passed.
> * checksums, keys
> * unit tests
> * executed few system tests - all passed
>
> +1 (non-binding).
>
> Thank you
>
>
> ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
>
> > Hi Mickael,
> >
> > Thanks for running the release!
> >
> > I did the following:
> >1. Validated the scala 2.13 checksums
> >2. Spot checked the java docs
> >3. Ran the quick start with scala 2.13 (found a minor bug KAFKA-13718
> > , won't block the
> > release)
> >
> > +1 (non-binding).
> >
> > Thank you.
> >
> > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison 
> > wrote:
> >
> > > Here is a successful Jenkins build for the 3.0 branch:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > >
> > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz  wrote:
> > > >
> > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> staging
> > > > Maven repository to run my tests. All seems to work fine, no issues
> > > found.
> > > >
> > > > Thanks
> > > > Jakub
> > > >
> > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison 
> > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > >
> > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> fixed
> > > > > since 3.0.0.
> > > > >
> > > > > Release notes for the 3.0.1 release:
> > > > >
> https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Thursday, March 10, 6pm GMT
> ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.0.1-rc0
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/30/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/30/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > I'll share a link once the build complete
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > >
> >
>
>
> --
> S pozdravom
>  Michal Tóth
>

Re: [DISCUSS] KIP-714: Client metrics and observability

[Tom Bentley]

Hi Ashenafi,

You'll need to unsubscribe from the dev mailing list by sending an email to
dev-unsubscr...@kafka.apache.org. No one else can do this for you.

Kind regards,

Tom

On Tue, 8 Mar 2022 at 04:40, Ashenafi Marcos  wrote:

> Hi,
> Can you please take out my email I’d so that will not be able to receive
> any mail from you.
> Thank you
>
> On Tue, Oct 19, 2021 at 1:30 PM Mickael Maison 
> wrote:
>
> > Hi Magnus,
> >
> > Thanks for the proposal.
> >
> > 1. Looking at the protocol section, isn't "ClientInstanceId" expected
> > to be a field in GetTelemetrySubscriptionsResponseV0? Otherwise, how
> > does a client retrieve this value?
> >
> > 2. In the client API section, you mention a new method
> > "clientInstanceId()". Can you clarify which interfaces are affected?
> > Is it only Consumer and Producer?
> >
> > 3. I'm a bit concerned this is enabled by default. Even if the data
> > collected is supposed to be not sensitive, I think this can be
> > problematic in some environments. Also users don't seem to have the
> > choice to only expose some metrics. Knowing how much data transit
> > through some applications can be considered critical.
> >
> > 4. As a user, how do you know if your application is actively sending
> > metrics? Are there new metrics exposing what's going on, like how much
> > data is being sent?
> >
> > 5. If all metrics are enabled on a regular Consumer or Producer, do
> > you have an idea how much throughput this would use?
> >
> > Thanks
> >
> > On Tue, Oct 19, 2021 at 5:06 PM Magnus Edenhill 
> > wrote:
> > >
> > > Den tis 19 okt. 2021 kl 13:22 skrev Tom Bentley :
> > >
> > > > Hi Magnus,
> > > >
> > > > I reviewed the KIP since you called the vote (sorry for not reviewing
> > when
> > > > you announced your intention to call the vote). I have a few
> questions
> > on
> > > > some of the details.
> > > >
> > > > 1. There's no Javadoc on ClientTelemetryPayload.data(), so I don't
> know
> > > > whether the payload is exposed through this method as compressed or
> > not.
> > > > Later on you say "Decompression of the payloads will be handled by
> the
> > > > broker metrics plugin, the broker should expose a suitable
> > decompression
> > > > API to the metrics plugin for this purpose.", which suggests it's the
> > > > compressed data in the buffer, but then we don't know which codec was
> > used,
> > > > nor the API via which the plugin should decompress it if required for
> > > > forwarding to the ultimate metrics store. Should the
> > ClientTelemetryPayload
> > > > expose a method to get the compression and a decompressor?
> > > >
> > >
> > > Good point, updated.
> > >
> > >
> > >
> > > > 2. The client-side API is expressed as StringOrError
> > > > ClientInstance::ClientInstanceId(int timeout_ms). I understand that
> > you're
> > > > thinking about the librdkafka implementation, but it would be good to
> > show
> > > > the API as it would appear on the Apache Kafka clients.
> > > >
> > >
> > > This was meant as pseudo-code, but I changed it to Java.
> > >
> > >
> > > > 3. "PushTelemetryRequest|Response - protocol request used by the
> > client to
> > > > send metrics to any broker it is connected to." To be clear, this
> means
> > > > that the client can choose any of the connected brokers and push to
> > just
> > > > one of them? What should a supporting client do if it gets an error
> > when
> > > > pushing metrics to a broker, retry sending to the same broker or try
> > > > pushing to another broker, or drop the metrics? Should supporting
> > clients
> > > > send successive requests to a single broker, or round robin, or is
> > that up
> > > > to the client author? I'm guessing the behaviour should be sticky to
> > > > support the rate limiting features, but I think it would be good for
> > client
> > > > authors if this section were explicit on the recommended behaviour.
> > > >
> > >
> > > You are right, I've updated the KIP to make this clearer.
> > >
> > >
> > > > 4. "Mapping the client instance id to an actual application instance
> > > > running on a (virtual) machine can be done by insp

Re: [ANNOUNCE] New committer: Luke Chen

[Tom Bentley]

Congratulations Luke!

On Thu, 10 Feb 2022 at 06:41, Josep Prat 
wrote:

> Congrats Luke!
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Thu, Feb 10, 2022, 07:07 Randall Hauch  wrote:
>
> > Congratulations, Luke!
> >
> > On Wed, Feb 9, 2022 at 11:02 PM Matthias J. Sax 
> wrote:
> >
> > > Congratulations! Glad to have you onboard, Luke!
> > >
> > > -Matthias
> > >
> > > On 2/9/22 16:37, Bill Bejeck wrote:
> > > > Congrats Luke! Well deserved.
> > > >
> > > > -Bill
> > > >
> > > > On Wed, Feb 9, 2022 at 7:25 PM Israel Ekpo 
> > wrote:
> > > >
> > > >> Congratulations Luke!
> > > >>
> > > >> Thank you for your service
> > > >>
> > > >> On Wed, Feb 9, 2022 at 6:22 PM Guozhang Wang 
> > > wrote:
> > > >>
> > > >>> The PMC for Apache Kafka has invited Luke Chen (showuon) as a
> > committer
> > > >> and
> > > >>> we are pleased to announce that he has accepted!
> > > >>>
> > > >>> Luke has been actively contributing to Kafka since early 2020. He
> has
> > > >>> made more than 120 commits on various components of Kafka, with
> > notable
> > > >>> contributions to the rebalance protocol in Consumer and Streams
> > > (KIP-766,
> > > >>> KIP-726, KIP-591, KAFKA-12675 and KAFKA12464, to just name a few),
> as
> > > >> well
> > > >>> as making an impact on improving test stability of the project.
> Aside
> > > >> from
> > > >>> all his code contributions, Luke has been a great participant in
> > > >>> discussions across the board, a very active and helpful reviewer of
> > > other
> > > >>> contributors' works, all of which are super valuable and highly
> > > >> appreciated
> > > >>> by the community.
> > > >>>
> > > >>>
> > > >>> Thanks for all of your contributions Luke. Congratulations!
> > > >>>
> > > >>> -- Guozhang, on behalf of the Apache Kafka PMC
> > > >>>
> > > >> --
> > > >> Israel Ekpo
> > > >> Lead Instructor, IzzyAcademy.com
> > > >> https://www.youtube.com/c/izzyacademy
> > > >> https://izzyacademy.com/
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.2.0 release

[Tom Bentley]

+1, thanks Bruno!

On Sat, 5 Feb 2022 at 07:17, Luke Chen  wrote:

> Thanks Bruno!
>
> +1
>
> Luke
>
> On Sat, Feb 5, 2022 at 10:50 AM Guozhang Wang  wrote:
>
> > Thanks Bruno! +1
> >
> > On Fri, Feb 4, 2022 at 4:14 PM Ismael Juma  wrote:
> >
> > > Thanks for volunteering, Bruno. +1!
> > >
> > > Ismael
> > >
> > > On Fri, Feb 4, 2022 at 7:03 AM Bruno Cadonna 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to volunteer to be the release manager for our next
> > > > feature release, 3.2.0. If there are no objections, I'll send
> > > > out the release plan soon.
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-769: Connect APIs to list all connector plugins and retrieve their configuration definitions

[Tom Bentley]

Hi Mickael,

+1 (binding), thanks!

Tom

On Wed, 26 Jan 2022 at 09:42, Viktor Somogyi-Vass
 wrote:

> Hi Michael,
>
> +1 (non-binding) from me.
>
> Viktor
>
> On Thu, Jan 13, 2022 at 11:32 AM Mickael Maison 
> wrote:
>
> > Bumping this vote.
> >
> > We have 2 non-binding votes so far. Please take a look and let me know
> > if you have any feedback.
> >
> > Thanks,
> > Mickael
> >
> > On Mon, Dec 13, 2021 at 10:50 PM Ryanne Dolan 
> > wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Ryanne
> > >
> > > On Mon, Dec 13, 2021, 4:18 AM Mickael Maison  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on KIP-769 which proposes adding new
> > > > endpoints to the Connect REST API to list all connectors plugins and
> > > > retrieve their configurations.
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > >
> > > > Please take a look and let me know if you have any feedback.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> >
>

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[Tom Bentley]

Hi Luke,

Thanks for the KIP, +1 (binding).

Kind regards,

Tom

On Wed, 19 Jan 2022 at 13:16, Luke Chen  wrote:

> Hi all,
>
> Bump this thread to see if there are other comments to this KIP.
> So far, I have one +1 vote (binding), and need more votes.
>
> Thank you.
> Luke
>
> On Tue, Dec 21, 2021 at 10:33 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > Bump this thread to see if there are other comments to this KIP.
> >
> > Thank you.
> > Luke
> >
> > On Fri, Dec 17, 2021 at 1:27 AM Colin McCabe  wrote:
> >
> >> Thanks for the explanation, Luke. That makes sense.
> >>
> >> best,
> >> Colin
> >>
> >> On Thu, Dec 9, 2021, at 13:31, Guozhang Wang wrote:
> >> > Thanks Luke, in that case I'm +1 on this KIP.
> >> >
> >> > On Thu, Dec 9, 2021 at 1:46 AM Luke Chen  wrote:
> >> >
> >> >> Hi Guozhang,
> >> >>
> >> >> Thanks for your comment.
> >> >>
> >> >> > we need to make sure the old-versioned leader would be able to
> >> ignore the
> >> >> new
> >> >> field during an upgrade e.g. without crashing.
> >> >>
> >> >> Yes, I understand. I'll be careful to make sure it won't crash the
> old
> >> >> versioned leader.
> >> >> But basically, it won't, because we appended the new field into the
> >> last of
> >> >> the ConsumerProtocolSubscription, which means, when read/deserialize
> >> the
> >> >> Subscription metadata, the old versioned leader will just read the
> head
> >> >> part of the data.
> >> >>
> >> >> Thanks for the reminder!
> >> >>
> >> >> Luke
> >> >>
> >> >> On Thu, Dec 9, 2021 at 4:00 AM Guozhang Wang 
> >> wrote:
> >> >>
> >> >> > Hi Luke,
> >> >> >
> >> >> > Thanks for the KIP.
> >> >> >
> >> >> > One thing I'd like to double check is that, since the
> >> >> > ConsumerProtocolSubscription is not auto generated from the json
> >> file, we
> >> >> > need to make sure the old-versioned leader would be able to ignore
> >> the
> >> >> new
> >> >> > field during an upgrade e.g. without crashing. Other than that, the
> >> KIP
> >> >> > lgtm.
> >> >> >
> >> >> > Guozhang
> >> >> >
> >> >> > On Tue, Dec 7, 2021 at 6:16 PM Luke Chen 
> wrote:
> >> >> >
> >> >> > > Hi Colin,
> >> >> > >
> >> >> > > I'm not quite sure if I understand your thoughts correctly.
> >> >> > > If I was wrong, please let me know.
> >> >> > >
> >> >> > > Also, I'm not quite sure how I could lock this feature to a new
> IBP
> >> >> > > version.
> >> >> > > I saw "KIP-584: Versioning scheme for features" is still under
> >> >> > development.
> >> >> > > Not sure if I need to lock the IBP version, how should I do?
> >> >> > >
> >> >> > > Thank you.
> >> >> > > Luke
> >> >> > >
> >> >> > > On Tue, Dec 7, 2021 at 9:41 PM Luke Chen 
> >> wrote:
> >> >> > >
> >> >> > > > Hi Colin,
> >> >> > > >
> >> >> > > > Thanks for your comments. I've updated the KIP to mention about
> >> the
> >> >> KIP
> >> >> > > > won't affect current broker side behavior.
> >> >> > > >
> >> >> > > > > One scenario that we need to consider is what happens during
> a
> >> >> > rolling
> >> >> > > > upgrade. If the coordinator moves back and forth between
> brokers
> >> with
> >> >> > > > different IBPs, it seems that the same epoch numbers could be
> >> reused
> >> >> > for
> >> >> > > a
> >> >> > > > group, if things are done in the obvious manner (old IBP =
> don't
> >> read
> >> >> > or
> >> >> > > > write epoch, new IBP = do)
> >> >> > > >
> >> >> > > > I think this KIP doesn't care about the group epoch number at
> >> all.
> >> >> The
> >> >> > > > subscription metadata is passed from each member to group
> >> >> coordinator,
> >> >> > > and
> >> >> > > > then the group coordinator pass all of them back to the
> consumer
> >> >> lead.
> >> >> > So
> >> >> > > > even if the epoch number is reused in a group, it should be
> >> fine. On
> >> >> > the
> >> >> > > > other hand, the group coordinator will have no idea if the join
> >> group
> >> >> > > > request sent from consumer containing the new subscription
> >> >> "generation"
> >> >> > > > field or not, because group coordinator won't deserialize the
> >> >> metadata.
> >> >> > > >
> >> >> > > > I've added also added them into the KIP.
> >> >> > > >
> >> >> > > > Thank you.
> >> >> > > > Luke
> >> >> > > >
> >> >> > > > On Mon, Dec 6, 2021 at 10:39 AM Colin McCabe <
> cmcc...@apache.org
> >> >
> >> >> > wrote:
> >> >> > > >
> >> >> > > >> Hi Luke,
> >> >> > > >>
> >> >> > > >> Thanks for the explanation.
> >> >> > > >>
> >> >> > > >> I don't see any description of how the broker decides to use
> >> the new
> >> >> > > >> version of ConsumerProtocolSubscription or not. This probably
> >> needs
> >> >> to
> >> >> > > be
> >> >> > > >> locked to a new IBP version.
> >> >> > > >>
> >> >> > > >> One scenario that we need to consider is what happens during a
> >> >> rolling
> >> >> > > >> upgrade. If the coordinator moves back and forth between
> brokers
> >> >> with
> >> >> > > >> different IBPs, it seems that the same epoch numbers could be
> >> reused
> >> >> > > for a
> >> >> > > >> group, if things are done in the 

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Tom Bentley]

To be honest I'm very sceptical that you can evolve (without breaking
compatibility) from a situation of having a large number of plugins which
have already implemented version() to return whatever they please, to a
situation where you can rely on it as the basis for some feature to support
multiple versions of plugins. It would be much safer to introduce a new
versioning concept, perhaps using Jar manifests and
"Specification-Version", than to try to build on Versioned.

So I would prefer Javadoc that clearly said that this version number was a
means for the plugin author to identify the version of their plugin in use
by users, for the purpose of bug reporting. That at least makes is clear
that Connect doesn't do anything with it except pass it through to the API.
It also makes clear that using "undefined" is a perfectly fine
implementation for PossiblyVersioned to have.

Kind regards,

Tom






On Thu, 27 Jan 2022 at 18:37, Chris Egerton 
wrote:

> Hi Mickael,
>
> +1 for the warning at startup, it'd be great if we could get everyone to
> start versioning all of their components. Although given the current state
> of WARN-level logging in Connect (see
> https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much
> noise for most people to notice, unfortunately. Still, better than nothing!
>
> I'm not sure it'd be wise to move quite so quickly with deprecating and
> then removing the PossiblyVersioned interface. In an ideal world everyone
> would update their converters, SMTs, and predicates as soon as they saw the
> warning, but there could be some plugins out there that aren't maintained
> anymore and would become incompatible with a Connect runtime that requires
> them to provide versions. I can imagine there might be some fairly trivial
> but valuable SMTs or converters that were written years ago and haven't
> been touched in some time but are still useful to Connect users today.
>
> I think it'd be fine to log a scary-looking warning message stating that
> the plugin may be incompatible with future versions of Connect, but perhaps
> we can leave the decision about how and when we actually create that
> incompatibility to a future KIP. If we take the example use case of loading
> multiple versions of the same plugin on the same worker, for example, that
> feature would obviously require plugins to provide versions, but we could
> potentially add some failsafe logic in order to still support versionless
> plugins with the understanding that it would not be possible to colocate
> multiple versions of them on the same worker.
>
> Cheers,
>
> Chris
>
> On Thu, Jan 27, 2022 at 7:43 AM Mickael Maison 
> wrote:
>
> > Hi Chris,
> >
> > Thanks for taking another look!
> >
> > That's a good point which I should have mentioned in the KIP. I think
> > we can have a PossiblyVersioned interface that extends Versioned and
> > has the default implementation. I'd like to also print a warning at
> > startup if a plugin does not override version(). Thanks for the
> > suggestion!
> >
> > I'm even considering whether PossiblyVersioned could be deprecated
> > immediately in order to be removed in the next major release. If so,
> > the warning message I mentioned would state that plugins not
> > overriding version() would stop working in the next major release.
> >
> > Thanks,
> > Mickael
> >
>
>