[jira] [Created] (KAFKA-16757) Fix broker re-registration issues around MV 3.7-IV2

2024-05-13 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16757: Summary: Fix broker re-registration issues around MV 3.7-IV2 Key: KAFKA-16757 URL: https://issues.apache.org/jira/browse/KAFKA-16757 Project: Kafka Issue

[jira] [Created] (KAFKA-16649) Fix potential deadlock in DynamicBrokerConfig

2024-04-30 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16649: Summary: Fix potential deadlock in DynamicBrokerConfig Key: KAFKA-16649 URL: https://issues.apache.org/jira/browse/KAFKA-16649 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16624) Don't generate useless PartitionChangeRecord on older MV

2024-04-25 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16624: Summary: Don't generate useless PartitionChangeRecord on older MV Key: KAFKA-16624 URL: https://issues.apache.org/jira/browse/KAFKA-16624 Project: Kafka

[ANNOUNCE] New committer: Igor Soarez

2024-04-24 Thread Colin McCabe
Hi all, The PMC of Apache Kafka is pleased to announce a new Kafka committer, Igor Soarez. Igor has been a Kafka contributor since 2019. In addition to being a regular contributor and reviewer, he has made significant contributions to improving Kafka's JBOD support in KRaft mode. He has also

[jira] [Created] (KAFKA-16509) CurrentControllerId metric is unreliable in ZK mode

2024-04-10 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16509: Summary: CurrentControllerId metric is unreliable in ZK mode Key: KAFKA-16509 URL: https://issues.apache.org/jira/browse/KAFKA-16509 Project: Kafka Issue

[jira] [Created] (KAFKA-16475) Create unit test for TopicImageNode

2024-04-04 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16475: Summary: Create unit test for TopicImageNode Key: KAFKA-16475 URL: https://issues.apache.org/jira/browse/KAFKA-16475 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-16469) Metadata Schema Checker

2024-04-03 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16469: Summary: Metadata Schema Checker Key: KAFKA-16469 URL: https://issues.apache.org/jira/browse/KAFKA-16469 Project: Kafka Issue Type: New Feature

[jira] [Resolved] (KAFKA-16428) Fix bug where config change notification znode may not get created during migration

2024-03-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16428. -- Resolution: Fixed > Fix bug where config change notification znode may not get created dur

[jira] [Resolved] (KAFKA-16411) Correctly migrate default client quota entities in KRaft migration

2024-03-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16411. -- Resolution: Fixed > Correctly migrate default client quota entities in KRaft migrat

[jira] [Created] (KAFKA-16435) Add test for KAFKA-16428

2024-03-27 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16435: Summary: Add test for KAFKA-16428 Key: KAFKA-16435 URL: https://issues.apache.org/jira/browse/KAFKA-16435 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-16428) Fix bug where config change notification znode may not get created during migration

2024-03-26 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16428: Summary: Fix bug where config change notification znode may not get created during migration Key: KAFKA-16428 URL: https://issues.apache.org/jira/browse/KAFKA-16428

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-26 Thread Colin McCabe
Hi all, Thanks for the discussion. Let me respond to a few questions I saw in the thread (hope I didn't miss any!) Feature level 0 is "special" because it effectively means that the feature hasn't been set up. So I could create a foo feature, a bar feature, and a baz feature

Re: [VOTE] 3.6.2 RC1

2024-03-22 Thread Colin McCabe
I have created a PR to fix this here : https://github.com/apache/kafka/pull/15584 best, Colin On Fri, Mar 22, 2024, at 13:28, Colin McCabe wrote: > Sorry but I have to vote -1 > > I tried verifying that the migration quotas bug described in > https://issues.apache.org/jira/browse

[jira] [Created] (KAFKA-16411) Correctly migrate default entities in KRaft migration

2024-03-22 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16411: Summary: Correctly migrate default entities in KRaft migration Key: KAFKA-16411 URL: https://issues.apache.org/jira/browse/KAFKA-16411 Project: Kafka Issue

Re: [VOTE] 3.6.2 RC1

2024-03-22 Thread Colin McCabe
Sorry but I have to vote -1 I tried verifying that the migration quotas bug described in https://issues.apache.org/jira/browse/KAFKA-16222 was fixed, and it appears to still be an issue with 3.6.2 RC1. The quota on the default resource is still getting translated improperly. I am looking into

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-15 Thread Colin McCabe
specifically, I have no metadata version >> > dependencies besides requiring 3.3 to write the feature records and use >> the >> > feature tools. I would suspect all new features would have this >> > requirement. >> > >> > >> > 11. *Basically, if **--release-ver

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-07 Thread Colin McCabe
nt. >> >> >> 11. *Basically, if **--release-version is not used, the command will just >> use the latest* >> >> *production version of every feature. Should we apply that logic to both* >> >> *tools?* >> >> >> How would this work wit

[jira] [Created] (KAFKA-16321) Default directory ids to MIGRATING, not UNASSIGNED

2024-03-01 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16321: Summary: Default directory ids to MIGRATING, not UNASSIGNED Key: KAFKA-16321 URL: https://issues.apache.org/jira/browse/KAFKA-16321 Project: Kafka Issue

[jira] [Resolved] (KAFKA-16216) Reduce batch size for initial metadata load during ZK migration

2024-02-01 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16216. -- Fix Version/s: 3.7.0 Reviewer: Colin McCabe Assignee: David Arthur

[jira] [Created] (KAFKA-16216) Reduce batch size for initial metadata load during ZK migration

2024-02-01 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16216: Summary: Reduce batch size for initial metadata load during ZK migration Key: KAFKA-16216 URL: https://issues.apache.org/jira/browse/KAFKA-16216 Project: Kafka

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-29 Thread Colin McCabe
; fledged discussion on these topics. >> >> > I'm struggling a bit with your phrasing. Are you suggesting that unstable >> MVs would not be able to be in trunk? >> >> Unstable MV should be in trunk. The wording is related to when we can >> promote "unst

[jira] [Created] (KAFKA-16180) Full metadata request sometimes fails during zk migration

2024-01-19 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16180: Summary: Full metadata request sometimes fails during zk migration Key: KAFKA-16180 URL: https://issues.apache.org/jira/browse/KAFKA-16180 Project: Kafka

[jira] [Resolved] (KAFKA-16078) Be more consistent about getting the latest MetadataVersion

2024-01-17 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16078. -- Fix Version/s: 3.7.0 Reviewer: Colin Patrick McCabe Resolution: Fixed

[jira] [Resolved] (KAFKA-16131) Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6

2024-01-17 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16131. -- Resolution: Fixed > Repeated UnsupportedVersionException logged when running Kafka 3.7.0-

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-15 Thread Colin McCabe
On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > I think using feature flags (whether we support a framework and tooling for > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to > this KIP. I think the value of this KIP is that it's trying to propose a > systemic

[jira] [Resolved] (KAFKA-16121) Partition reassignments in ZK migration dual write mode stalled until leader epoch incremented

2024-01-15 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16121. -- Fix Version/s: 3.7.0 Reviewer: Colin McCabe Assignee: David Mao

Re: [VOTE] 3.7.0 RC2

2024-01-15 Thread Colin McCabe
Docs fix discussed in the thread is here: https://github.com/apache/kafka/pull/15193 best, Colin On Sun, Jan 14, 2024, at 23:56, Colin McCabe wrote: > Hi Stanislav, > > Thanks for making the first RC. The fact that it's titled RC2 is > messing with my mind a bit. I hope this

Re: [VOTE] 3.7.0 RC2

2024-01-14 Thread Colin McCabe
Hi Stanislav, Thanks for making the first RC. The fact that it's titled RC2 is messing with my mind a bit. I hope this doesn't make people think that we're farther along than we are, heh. On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote: > *> Nice catch! It does seem like we should have

[jira] [Resolved] (KAFKA-16120) Partition reassignments in ZK migration dual write leaves stray partitions

2024-01-14 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16120. -- Fix Version/s: 3.7.0 Reviewer: Colin McCabe Assignee: David Mao

[jira] [Created] (KAFKA-16126) Kcontroller dynamic configurations may fail to apply at startup

2024-01-14 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16126: Summary: Kcontroller dynamic configurations may fail to apply at startup Key: KAFKA-16126 URL: https://issues.apache.org/jira/browse/KAFKA-16126 Project: Kafka

Re: Logging in Kafka

2024-01-10 Thread Colin McCabe
Hi Mickael, Thanks for bringing this up. If we move to log4j2 in 4.0, is there any work that needs to be done in 3.8? That's probably what we should focus on. P.S. My assumption is that if the log4j2 work misses the train, we'll stick with reload4j in 4.0. Hopefully this won't happen. best,

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-10 Thread Colin McCabe
t; Thanks for this clarification. I think this concrete risk should be >> highlighted in the KIP and in the "unstable.metadata.versions.enable" >> documentation. >> >> In the test plan, should we also have one system test checking that >> "features with a stab

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-01-09 Thread Colin McCabe
On Tue, Jan 9, 2024, at 17:07, Jason Gustafson wrote: > Hi Jose, > > Thanks for the KIP! A few initial questions below: > > 1. In the user experience section, the user is expected to provide supply > the UUID for each voter. I'm assuming this is the directory.id coming from > KIP-858. I thought it

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-09 Thread Colin McCabe
t it was production ready in 3.7 ... hence it belonged (and still belongs) in an unstable MV, until that changes (hopefully soon :) ) best, Colin > --Proven > > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe wrote: > >> Hi Justine, >> >> Yes, this is an important point to

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-01-09 Thread Colin McCabe
Hi José, Thanks for the revisions. I'm really excited to see this going forward for Kafka 3.8. One important piece of feedback that a lot of people have given me is that they really want auto-formatting in KRaft mode. In other words, they want to start up a process and just have it do the

[jira] [Resolved] (KAFKA-16094) BrokerRegistrationRequest.logDirs field must be ignorable

2024-01-09 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16094. -- Fix Version/s: 3.7.0 Resolution: Fixed > BrokerRegistrationRequest.logDirs field m

Re: Apache Kafka 3.7.0 Release

2024-01-09 Thread Colin McCabe
KAFKA-16094 has been fixed and backported to 3.7. Colin On Mon, Jan 8, 2024, at 14:52, Colin McCabe wrote: > On an unrelated note, I found a blocker bug related to upgrades from > 3.6 (and earlier) to 3.7. > > The JIRA is here: > https://issues.apache.org/jira/browse/KAFK

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-09 Thread Colin McCabe
et's clarify if we are ok with reordering unstable metadata versions :) > > Justine > > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: > >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: >> > Hey all, >> > >> > I was wondering h

Re: Kafka trunk test & build stability

2024-01-09 Thread Colin McCabe
Sorry, but to put it bluntly, the current build setup isn't good enough at partial rebuilds that build caching would make sense. All Kafka devs have had the experience of needing to clean the build directory in order to get a valid build. The scala code esspecially seems to have this issue.

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
On an unrelated note, I found a blocker bug related to upgrades from 3.6 (and earlier) to 3.7. The JIRA is here: https://issues.apache.org/jira/browse/KAFKA-16094 Fix here: https://github.com/apache/kafka/pull/15153 best, Colin On Mon, Jan 8, 2024, at 14:47, Colin McCabe wrote: >

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
, Jan 8, 2024, at 14:09, Ismael Juma wrote: > All modules are published to Sonatype - that's a requirement. You may be > missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`. > > Ismael > > On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe wrote: > >> Hi

[jira] [Created] (KAFKA-16094) 3.7 brokers must be able to register with 3.6 and earlier controllers

2024-01-08 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16094: Summary: 3.7 brokers must be able to register with 3.6 and earlier controllers Key: KAFKA-16094 URL: https://issues.apache.org/jira/browse/KAFKA-16094 Project: Kafka

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
IPs/Features the community identified that should be >> >> > present in a 3.x version so they can safely migrate to a potential 4.0 >> >> > version >> >> > - The timeline of 4.0 (starting after branching, not after release) >> >> > -

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
tood what they mean by gradle metadata - it's > not the Kafka metadata module. > > Ismael > > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe wrote: > >> Oops, hit send too soon. I see that #15127 was already merged. So we >> should no longer be publishing :metadata as

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-08 Thread Colin McCabe
nge up until 3.7.0 is released. Once it gets released, it's never unreleased. We just move on to 3.7.1. Same thing here. best, Colin > > Thanks, > > Justine > > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe wrote: > >> Hi Proven, >> >> Thanks for the KIP. I

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-08 Thread Colin McCabe
Hi Proven, Thanks for the KIP. I think there is a need for this capability, for those of us who deploy from trunk (or branches dervied from trunk). With regard to "unstable.metadata.versions.enable": is this going to be a documented configuration, or an internal one? I am guessing we want it

Re: [DISCUSS] KIP-1015: Limit number of ssl connections in brokers

2024-01-08 Thread Colin McCabe
Hi zw, As you yourself wrote in the rejected alternatives section, the existing listener-specific connection limit already lets administrators limit the number of SSL connections (assuming that one listener is SSL and another is not). I don't understand the objection to just using that

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
eded in 3.x to make migration possible, now is the time to speak up. best, Colin > > [1] https://lists.apache.org/thread/k3bbz7k0hsb4y0b2xn1lh7bjrmjowmcq > > Thanks! > Greg > > On Fri, Jan 5, 2024 at 2:20 PM Colin McCabe wrote: >> >> Hi Justine, >>

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
Oops, hit send too soon. I see that #15127 was already merged. So we should no longer be publishing :metadata as part of the clients artifacts, right? thanks, Colin On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote: > Hi Apporv, > > Please remove the metadata module from any

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
1 > > > On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe wrote: > >> metadata is an internal gradle module. It is not used by clients. So I >> don't see why you would want to publish it (unless I'm misunderstanding >> something). >> >> best, >>

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
east another minor >> > release in 3.x to enable and ease migration to a potential 4.0 version >> > >> > I always mention potential in terms of 4.0 as we don't know what will be >> in >> > there yet, and this KIP's scope is not meant to define this. >> > &g

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
know what will be in > there yet, and this KIP's scope is not meant to define this. > > Best, > > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma wrote: > >> I agree with Colin. Also, 4.0 would be started after 3.8 is branched (not >> after 3.8.0 is released). The rest look

Re: Apache Kafka 3.7.0 Release

2024-01-05 Thread Colin McCabe
metadata is an internal gradle module. It is not used by clients. So I don't see why you would want to publish it (unless I'm misunderstanding something). best, Colin On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote: > Thanks for reporting the blockers, folks. Good job finding. > > I

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
ufer 3-7, 10117 Berlin >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> Amtsgericht Charlottenburg, HRB 209739 B >> >> On Fri, Jan 5, 2024, 00:24 Colin McCabe wrote: >> >> > Hi all, >> > >> > Since this has been open for a few

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
the required ones best, Colin On Fri, Jan 5, 2024, at 13:27, Colin McCabe wrote: > Hi, > > Thanks for calling the vote, Josep. > > I re-checked this, and saw something that we missed updating. One thing > we talked about earlier is that KIP-966 is actually not required for > 3.8. Wha

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
n the road to 4.0 thread that may be worth >> having here. >> >> On Fri, Jan 5, 2024 at 1:25 AM Josep Prat >> wrote: >> >> > Hi Colin, >> > Sorry for being quiet these last days (PTO). >> > >> > I will start the vote thread right away. &g

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
Hi, Thanks for calling the vote, Josep. I re-checked this, and saw something that we missed updating. One thing we talked about earlier is that KIP-966 is actually not required for 3.8. What is required is some way of enabling unclean leader election by default in KRaft. (Which could be

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-04 Thread Colin McCabe
with a tentative schedule. https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0 Please let me know if these dates make sense -- they're just proposals right now. best, Colin On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote: > On Thu, Dec 28, 2023, at 18:17, Justine Olshan wr

Re: [DISCUSS] Road to Kafka 4.0

2024-01-04 Thread Colin McCabe
te: >> >> > Hi, >> > >> > I've not seen replies about log4j2. >> > The plan was to deprecated the appender (KIP-719) and switch to log4j2 >> > (KIP-653). >> > >> > While reload4j works well, I'd still be in favor of switching to >

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2023-12-28 Thread Colin McCabe
> Yeah. There is a bunch of docs and quickstart cleanup that we should do. I don't think any of it is a blocker for 3.8 or 4.0, but the new year is a good time to clean things up. best, Colin > Justine > > On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe wrote: > >> Hi Josep,

Re: Apache Kafka 3.7.0 Release

2023-12-28 Thread Colin McCabe
Just to update this thread, everything in KAFKA-14127 is done now. A few tasks got moved to a separate umbrella JIRA. Some folks are going to do more testing, both manual and automated, in the next week or two. I think this will give us a good indicator of stability and what we need to fix.

[jira] [Resolved] (KAFKA-14127) KIP-858: Handle JBOD broker disk failure in KRaft

2023-12-28 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-14127. -- Resolution: Fixed > KIP-858: Handle JBOD broker disk failure in KR

[jira] [Resolved] (KAFKA-15650) Data-loss on leader shutdown right after partition creation?

2023-12-28 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15650. -- Resolution: Not A Problem > Data-loss on leader shutdown right after partition creat

[jira] [Created] (KAFKA-16061) JBOD follow-ups

2023-12-28 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16061: Summary: JBOD follow-ups Key: KAFKA-16061 URL: https://issues.apache.org/jira/browse/KAFKA-16061 Project: Kafka Issue Type: Improvement Reporter

Re: [DISCUSS] Road to Kafka 4.0

2023-12-28 Thread Colin McCabe
Hi all, Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for a Kafka 3.8.x release" email thread. Colin On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote: > Hi Divij, > > Thanks for the feedback. I agree that having a 3.8 release is > beneficial but some of

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2023-12-28 Thread Colin McCabe
Hi Josep, Thanks for the KIP. Based on the discussions we had previously, I agree that we need a 3.8. It would be good to link to KIP-833 in the motivation section, since this KIP builds on that one. Also, I think we should mention in KIP-1012 that 3.8 will be a general-purpose release that

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

2023-12-28 Thread Colin McCabe
Congratulations, Divij! best, Colin On Thu, Dec 28, 2023, at 11:38, Bruno Cadonna wrote: > Congratulations Divij! Well deserved! > > Best, > Bruno > > On 12/27/23 12:45 PM, Luke Chen wrote: >> Hi, Everyone, >> >> Divij has been a Kafka committer since June, 2023. He has remained very >> active

Re: [DISCUSS] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2023-12-26 Thread Colin McCabe
Hi Ismael, +1 from me. Looking at the list of languages features for JDK17, from a developer productivity standpoint, the biggest wins are probably pattern matching and java.util.HexFormat. Also, Java 11 is getting long in the tooth, even though we never adopted it. It was released 6 years

[jira] [Created] (KAFKA-15979) Add KIP-1001 CurrentControllerId metric

2023-12-06 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15979: Summary: Add KIP-1001 CurrentControllerId metric Key: KAFKA-15979 URL: https://issues.apache.org/jira/browse/KAFKA-15979 Project: Kafka Issue Type

[jira] [Created] (KAFKA-15980) Add KIP-1001 CurrentControllerId metric

2023-12-06 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15980: Summary: Add KIP-1001 CurrentControllerId metric Key: KAFKA-15980 URL: https://issues.apache.org/jira/browse/KAFKA-15980 Project: Kafka Issue Type

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-12-04 Thread Colin McCabe
Yes, this should be CLUSTER_ACTION on CLUSTER, to be consistent with our other internal RPCs. best, Colin On Mon, Dec 4, 2023, at 04:28, Viktor Somogyi-Vass wrote: > Hi Igor, > > I'm just reading through your KIP and noticed that the new protocol you > created doesn't say anything about ACLs

[jira] [Created] (KAFKA-15956) MetadataShell must take the directory lock when reading

2023-12-01 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15956: Summary: MetadataShell must take the directory lock when reading Key: KAFKA-15956 URL: https://issues.apache.org/jira/browse/KAFKA-15956 Project: Kafka

[jira] [Resolved] (KAFKA-15311) Fix docs about reverting to ZooKeeper mode during KRaft migration

2023-11-29 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15311. -- Fix Version/s: 3.7.0 Resolution: Fixed > Fix docs about reverting to ZooKeeper m

[jira] [Created] (KAFKA-15922) Add MetadataVersion for JBOD

2023-11-28 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15922: Summary: Add MetadataVersion for JBOD Key: KAFKA-15922 URL: https://issues.apache.org/jira/browse/KAFKA-15922 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-15860) ControllerRegistration must be written out to the metadata image

2023-11-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15860. -- Fix Version/s: 3.7.0 Resolution: Fixed > ControllerRegistration must be written

Re: [DISCUSS] Road to Kafka 4.0

2023-11-27 Thread Colin McCabe
eeds to fade away, just like JDK 6 support or the old Scala producer. best, Colin > BR, > Anton > > Den tors 23 nov. 2023 kl 05:48 skrev Colin McCabe : > >> On Tue, Nov 21, 2023, at 19:30, Luke Chen wrote: >> > Yes, KIP-853 and disk failure support are both very impor

Re: [DISCUSS] Road to Kafka 4.0

2023-11-22 Thread Colin McCabe
fer. But, on the KRaft side, I still maintain that nothing is missing except JBOD, which we already have a plan for. best, Colin > Thank you. > Luke > > On Wed, Nov 22, 2023 at 2:54 AM Colin McCabe wrote: > >> On Tue, Nov 21, 2023, at 03:47, Josep Prat wrote: >> &g

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

2023-11-21 Thread Colin McCabe
e/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1 > > Ismael > > On Tue, Nov 21, 2023, 12:21 PM Colin McCabe wrote: > >> Hi Ismael, >> >> Can we state somewhere that the message.format.version configuration will >> be gone in 4.0? We

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

2023-11-21 Thread Colin McCabe
Hi Ismael, Can we state somewhere that the message.format.version configuration will be gone in 4.0? We only will support one message format version (for now, at least). If we do want more versions later, I don't think we'll want to configure them via a static config. best, Colin On Tue,

Re: [VOTE] KIP-1001; CurrentControllerId Metric

2023-11-21 Thread Colin McCabe
> wrote: >> >> The KIP makes sense. +1 >> >> On Mon, Nov 20, 2023 at 12:37 PM David Arthur >> wrote: >> >> > Thanks Colin, >> > >> > +1 from me >> > >> > -David >> > >> > On Tue, Nov 14, 2

Re: [DISCUSS] Road to Kafka 4.0

2023-11-21 Thread Colin McCabe
ethods on top of the usual > work we have. For this reason I think there would be enough time > between releasing 3.8.0 and 4.0.0. > > What do you all think? > > Best, > Josep Prat > > On 2023/11/20 20:03:18 Colin McCabe wrote: >> Hi Josep, >> >> I

[jira] [Created] (KAFKA-15860) ControllerRegistration must be written out to the metadata image

2023-11-20 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15860: Summary: ControllerRegistration must be written out to the metadata image Key: KAFKA-15860 URL: https://issues.apache.org/jira/browse/KAFKA-15860 Project: Kafka

Re: [DISCUSS] Road to Kafka 4.0

2023-11-20 Thread Colin McCabe
version before 4.0.x. > > What do other think in this regard? > > Best, > > On 2023/11/14 20:47:10 Colin McCabe wrote: >> On Tue, Nov 14, 2023, at 04:37, Anton Agestam wrote: >> > Hi Colin, >> > >> > Thank you for your thoughtful and comprehens

[VOTE] KIP-1001; CurrentControllerId Metric

2023-11-14 Thread Colin McCabe
Hi all, I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. Take a look here: https://cwiki.apache.org/confluence/x/egyZE best, Colin

Re: [DISCUSS] Road to Kafka 4.0

2023-11-14 Thread Colin McCabe
with both KRaft and ZK supporting this feature to ease the migration out of > ZK. > The problem is, supporting multiple controller implementations is a huge burden. So we don't want to extend the 3.x release past the point that's needed to complete all the must-dos (SCRAM, delegatio

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

2023-11-14 Thread Colin McCabe
cess role provided." > > What do you think? > > Thanks, > Hailey > > On Thu, Nov 9, 2023 at 4:03 PM Colin McCabe wrote: > >> Hi Hailey, >> >> Thanks for the KIP. >> >> It feels clunky to have to pass an absolute path to the configuration file

[jira] [Resolved] (KAFKA-15532) ZkWriteBehindLag should not be reported by inactive controllers

2023-11-13 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15532. -- Resolution: Fixed > ZkWriteBehindLag should not be reported by inactive controll

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

2023-11-09 Thread Colin McCabe
Hi Hailey, Thanks for the KIP. It feels clunky to have to pass an absolute path to the configuration file when starting the broker or controller. I think we should consider one of two alternate options: 1. Use JMXtool to examine the running kafka.Kafka processes. I believe ID is available

[DISCUSS] KIP-1001: Add CurrentControllerId Metric

2023-11-09 Thread Colin McCabe
Hi all, I would like to start the discussion on a KIP I wrote about adding a new CurrentControllerIdMetric. See: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1001%3A+Add+CurrentControllerId+Metric best, Colin

Re: [DISCUSS] Road to Kafka 4.0

2023-11-09 Thread Colin McCabe
Hi Anton, It rarely makes sense to scale up and down the number of controller nodes in the cluster. Only one controller node will be active at any given time. The main reason to use 5 nodes would be to be able to tolerate 2 failures instead of 1. At Confluent, we generally run KRaft with 3

Re: [VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

2023-11-06 Thread Colin McCabe
-1 If the goal is not to have so much config logging, let's add a configuration key that suppresses log4j in AbstractConfig. Then plugins can set that configuration if they want before invoking the superclass constructors. (Or in the interest of allowing people to be verbose if they want,

[jira] [Resolved] (KAFKA-15754) The kafka-storage tool can generate UUID starting with "-"

2023-10-30 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15754. -- Resolution: Invalid kafka-storage tool can not, in fact, generate uuids starting

Re: [DISCUSS] Road to Kafka 4.0

2023-10-27 Thread Colin McCabe
n though I removed the blocker tag from some of these, thanks for taking a look at our backlog. That always helps us improve. If you find something that should be fixed for kraft, go ahead and add the "kraft" tag so we can find it more easily. best, Colin On Fri, Oct 27, 2023, at 15:01

Re: [DISCUSS] Road to Kafka 4.0

2023-10-27 Thread Colin McCabe
Hi Luke, As Ismael said, the only thing on that list that is a 4.0 blocker is KIP-858: JBOD support. It is the last remaining feature gap with ZK mode. We previously agreed that it needed to be implemented so that users of this feature could migrate to KRaft as planned. The good news is that

[jira] [Resolved] (KAFKA-15704) ControllerRegistrationRequest must set ZkMigrationReady field if appropriate

2023-10-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15704. -- Resolution: Fixed > ControllerRegistrationRequest must set ZkMigrationReady fi

[jira] [Created] (KAFKA-15704) ControllerRegistrationRequest must set ZkMigrationReady field if appropriate

2023-10-27 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15704: Summary: ControllerRegistrationRequest must set ZkMigrationReady field if appropriate Key: KAFKA-15704 URL: https://issues.apache.org/jira/browse/KAFKA-15704 Project

Re: [DISCUSS] How to detect (and prevent) complex bugs in Kafka?

2023-10-24 Thread Colin McCabe
Hi Divij, I've worked on several projects that had a "debug mode." It was something that a lot of old-fashioned C and C++ projects would do. Usually implemented through an ASSERT macro or similar that was defined away when in "production mode" I didn't like this back then, and still don't like

Re: UncleanLeaderElectionsPerSec metric and Raft

2023-10-24 Thread Colin McCabe
Hi Neil, Yes, I think we should probably report the UncleanLeaderElectionsPerSec metric in KRaft. We don't have it currently. We do have the concept of unclean leader election in KRaft, but it has to be triggered by the leader election tool currently. We've been talking about adding

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-09 Thread Colin McCabe
On Fri, Oct 6, 2023, at 18:30, Igor Soarez wrote: > Hi Colin, > >> I would call #2 LOST. It was assigned in the past, but we don't know where. >> I see that you called this OFFLINE). This is not really normal... >> it should happen only when we're migrating from ZK mode to KRaft mode, >> or going

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-06 Thread Colin McCabe
Thanks for posting these notes, Igor. I think we should definitely distinguish between these two cases: 1. this replica hasn't been assigned to a storage directory 2. we don't know which storage directory this replica was assigned to in the past I would call #1 UNASSIGNED. I see that you

  1   2   3   4   5   6   7   8   9   10   >