[jira] [Created] (KAFKA-16667) KRaftMigrationDriver gets stuck after successive failovers

2024-05-04 Thread David Arthur (Jira)
David Arthur created KAFKA-16667: Summary: KRaftMigrationDriver gets stuck after successive failovers Key: KAFKA-16667 URL: https://issues.apache.org/jira/browse/KAFKA-16667 Project: Kafka

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

2024-04-18 Thread David Arthur
Hi Fred, thanks for the KIP. Seems like a useful improvement. As others have mentioned, I think we should avoid exposing Record in this way. Using ConsumerRecord seems okay, but maybe not the best fit for this case (for the reasons Matthias gave). Maybe we could create a new container interface

[jira] [Created] (KAFKA-16539) Can't update specific broker configs in pre-migration mode

2024-04-11 Thread David Arthur (Jira)
David Arthur created KAFKA-16539: Summary: Can't update specific broker configs in pre-migration mode Key: KAFKA-16539 URL: https://issues.apache.org/jira/browse/KAFKA-16539 Project: Kafka

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-04 Thread David Arthur
Andrew, thanks for the KIP! This is a pretty exciting effort. I've finally made it through the KIP, still trying to grok the whole thing. Sorry if some of my questions are basic :) Concepts: 70. Does the Group Coordinator communicate with the Share Coordinator over RPC or directly in-process?

[jira] [Created] (KAFKA-16468) Listener not found error in SendRPCsToBrokersEvent

2024-04-03 Thread David Arthur (Jira)
David Arthur created KAFKA-16468: Summary: Listener not found error in SendRPCsToBrokersEvent Key: KAFKA-16468 URL: https://issues.apache.org/jira/browse/KAFKA-16468 Project: Kafka Issue

[jira] [Created] (KAFKA-16466) QuorumController is swallowing some exception messages

2024-04-02 Thread David Arthur (Jira)
David Arthur created KAFKA-16466: Summary: QuorumController is swallowing some exception messages Key: KAFKA-16466 URL: https://issues.apache.org/jira/browse/KAFKA-16466 Project: Kafka Issue

[jira] [Created] (KAFKA-16463) Automatically delete metadata log directory on ZK brokers

2024-04-02 Thread David Arthur (Jira)
David Arthur created KAFKA-16463: Summary: Automatically delete metadata log directory on ZK brokers Key: KAFKA-16463 URL: https://issues.apache.org/jira/browse/KAFKA-16463 Project: Kafka

[jira] [Created] (KAFKA-16446) Log slow controller events

2024-03-28 Thread David Arthur (Jira)
David Arthur created KAFKA-16446: Summary: Log slow controller events Key: KAFKA-16446 URL: https://issues.apache.org/jira/browse/KAFKA-16446 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-16180) Full metadata request sometimes fails during zk migration

2024-03-14 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-16180. -- Resolution: Fixed > Full metadata request sometimes fails during zk migrat

[jira] [Resolved] (KAFKA-16171) Controller failover during ZK migration can prevent metadata updates to ZK brokers

2024-03-13 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-16171. -- Resolution: Fixed > Controller failover during ZK migration can prevent metadata upda

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-28 Thread David Arthur
Andrew/Jose, I like the suggested Flow API. It's also similar to the stream observers in GPRC. I'm not sure we should expose something as complex as the Flow API directly in KafkaAdminClient, but certainly we can provide a similar interface. --- Cancellations: Another thing not yet discussed is

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-23 Thread David Arthur
ve the >pagination on huge topics for their partitions. >2. To avoid OOM, we should only fetch the new topics when we need them >and release the used topics. Especially the main use case of looping the >topic list is when the client prints all the topics. > > So, to better

Re: Github build queue

2024-02-09 Thread David Arthur
gt;> > > > that would not be too hard. > >> > > > > >> > > > Ismael > >> > > > > >> > > > On Fri, Feb 9, 2024 at 9:03 AM Greg Harris > >> >> > > > > >> > > > wrote: > >> > > >

Re: Github build queue

2024-02-09 Thread David Arthur
gt; > > > Hi David, > > > > > > +1 on that strategy. > > > > > > I see several flaky tests that aren't marked with @Tag("integration") > > > or @IntegrationTest, and I think those would make using the unitTest > > > target ineffecti

Re: Github build queue

2024-02-09 Thread David Arthur
> (or else the PR won't be merged). Sounds like you are suggesting to skip > > the tests for the merge queue validation. Could we perhaps include the > unit > > tests as well? That would incentivize us to ensure the unit tests are > fast > > and reliable. Getting the integra

Github build queue

2024-02-09 Thread David Arthur
stats, we are averaging under 40 commits per week. Assuming those primarily come in on weekdays, that's 8 commits per day. If we just run "gradlew check -x tests" for the merge queue job, I don't think we'd get backlogged. Thoughts? David -- David Arthur

[jira] [Created] (KAFKA-16206) ZkConfigMigrationClient tries to delete topic configs twice

2024-01-29 Thread David Arthur (Jira)
David Arthur created KAFKA-16206: Summary: ZkConfigMigrationClient tries to delete topic configs twice Key: KAFKA-16206 URL: https://issues.apache.org/jira/browse/KAFKA-16206 Project: Kafka

[jira] [Created] (KAFKA-16205) Reduce number of metadata requests during hybrid mode

2024-01-29 Thread David Arthur (Jira)
David Arthur created KAFKA-16205: Summary: Reduce number of metadata requests during hybrid mode Key: KAFKA-16205 URL: https://issues.apache.org/jira/browse/KAFKA-16205 Project: Kafka Issue

[jira] [Created] (KAFKA-16171) Controller failover during ZK migration can lead to controller unavailability for ZK brokers

2024-01-19 Thread David Arthur (Jira)
David Arthur created KAFKA-16171: Summary: Controller failover during ZK migration can lead to controller unavailability for ZK brokers Key: KAFKA-16171 URL: https://issues.apache.org/jira/browse/KAFKA-16171

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2024-01-08 Thread David Arthur
; > > > On Tue, Jan 2, 2024 at 4:30 PM Ismael Juma > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I would like to start a vote on KIP-1013. > > > > > > > > > > > > > > > > As stated in the discussion thread, this KIP was proposed > > after the > > > > > KIP > > > > > > > > freeze for Apache Kafka 3.7, but it is purely a documentation > > > > update > > > > > > (if we > > > > > > > > decide to adopt it) and I believe it would serve our users > > best if > > > > we > > > > > > > > communicate the deprecation for removal sooner (i.e. 3.7) > > rather > > > > than > > > > > > later > > > > > > > > (i.e. 3.8). > > > > > > > > > > > > > > > > Please take a look and cast your vote. > > > > > > > > > > > > > > > > Link: > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510 > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > -- David Arthur

[jira] [Created] (KAFKA-16078) InterBrokerProtocolVersion defaults to non-production MetadataVersion

2024-01-03 Thread David Arthur (Jira)
David Arthur created KAFKA-16078: Summary: InterBrokerProtocolVersion defaults to non-production MetadataVersion Key: KAFKA-16078 URL: https://issues.apache.org/jira/browse/KAFKA-16078 Project: Kafka

Re: [DISCUSS] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2023-12-26 Thread David Arthur
Thanks, Ismael. I'm +1 on the proposal. Does this KIP essentially replace KIP-750? On Tue, Dec 26, 2023 at 3:57 PM Ismael Juma wrote: > Hi Colin, > > A couple of comments: > > 1. It is true that full support for OpenJDK 11 from Red Hat will end on > October 2024 (extended life support will

Re: Kafka trunk test & build stability

2023-12-26 Thread David Arthur
S2. We’ve looked into this before, and it wasn’t possible at the time with JUnit. We commonly set a timeout on each test class (especially integration tests). It is probably worth looking at this again and seeing if something has changed with JUnit (or our usage of it) that would allow a global

[jira] [Created] (KAFKA-16020) Time#waitForFuture should tolerate nanosecond overflow

2023-12-15 Thread David Arthur (Jira)
David Arthur created KAFKA-16020: Summary: Time#waitForFuture should tolerate nanosecond overflow Key: KAFKA-16020 URL: https://issues.apache.org/jira/browse/KAFKA-16020 Project: Kafka Issue

[jira] [Created] (KAFKA-16007) ZK migrations can be slow for large clusters

2023-12-13 Thread David Arthur (Jira)
David Arthur created KAFKA-16007: Summary: ZK migrations can be slow for large clusters Key: KAFKA-16007 URL: https://issues.apache.org/jira/browse/KAFKA-16007 Project: Kafka Issue Type

Re: [VOTE] 3.6.1 RC0

2023-12-04 Thread David Arthur
Commit will crash. Considering these two factors, I don't strongly feel like we need to block the release for this fix. -David On Mon, Dec 4, 2023 at 10:49 AM David Arthur wrote: > Mickael, > > I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while > investigating a lo

Re: [VOTE] 3.6.1 RC0

2023-12-04 Thread David Arthur
Mickael, I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while investigating a log corruption issue on the controller. I'm still investigating the issue to see how far back this goes, but I think this could be a blocker. Essentially, the bug is that the controller does not treat a

[jira] [Created] (KAFKA-15968) QuorumController does not treat CorruptRecordException as fatal

2023-12-04 Thread David Arthur (Jira)
David Arthur created KAFKA-15968: Summary: QuorumController does not treat CorruptRecordException as fatal Key: KAFKA-15968 URL: https://issues.apache.org/jira/browse/KAFKA-15968 Project: Kafka

Re: [VOTE] KIP-1001; CurrentControllerId Metric

2023-11-20 Thread David Arthur
Thanks Colin, +1 from me -David On Tue, Nov 14, 2023 at 3:53 PM Colin McCabe wrote: > Hi all, > > I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. > > Take a look here: > https://cwiki.apache.org/confluence/x/egyZE > > best, > Colin > -- -David

[jira] [Resolved] (KAFKA-15825) KRaft controller writes empty state to ZK after migration

2023-11-14 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15825. -- Resolution: Fixed This bug was fixed as part of KAFKA-15605 > KRaft controller writes em

[jira] [Created] (KAFKA-15825) KRaft controller writes empty state to ZK after migration

2023-11-14 Thread David Arthur (Jira)
David Arthur created KAFKA-15825: Summary: KRaft controller writes empty state to ZK after migration Key: KAFKA-15825 URL: https://issues.apache.org/jira/browse/KAFKA-15825 Project: Kafka

[jira] [Resolved] (KAFKA-15605) Topics marked for deletion in ZK are incorrectly migrated to KRaft

2023-11-14 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15605. -- Fix Version/s: 3.7.0 Resolution: Fixed > Topics marked for deletion in

[jira] [Created] (KAFKA-15799) ZK brokers incorrectly handle KRaft metadata snapshots

2023-11-08 Thread David Arthur (Jira)
David Arthur created KAFKA-15799: Summary: ZK brokers incorrectly handle KRaft metadata snapshots Key: KAFKA-15799 URL: https://issues.apache.org/jira/browse/KAFKA-15799 Project: Kafka Issue

[jira] [Created] (KAFKA-15698) KRaft mode brokers should clean up stray partitions from migration

2023-10-26 Thread David Arthur (Jira)
David Arthur created KAFKA-15698: Summary: KRaft mode brokers should clean up stray partitions from migration Key: KAFKA-15698 URL: https://issues.apache.org/jira/browse/KAFKA-15698 Project: Kafka

[jira] [Created] (KAFKA-15648) QuorumControllerTest#testBootstrapZkMigrationRecord is flaky

2023-10-19 Thread David Arthur (Jira)
David Arthur created KAFKA-15648: Summary: QuorumControllerTest#testBootstrapZkMigrationRecord is flaky Key: KAFKA-15648 URL: https://issues.apache.org/jira/browse/KAFKA-15648 Project: Kafka

[jira] [Created] (KAFKA-15605) Topic marked for deletion are incorrectly migrated to KRaft

2023-10-13 Thread David Arthur (Jira)
David Arthur created KAFKA-15605: Summary: Topic marked for deletion are incorrectly migrated to KRaft Key: KAFKA-15605 URL: https://issues.apache.org/jira/browse/KAFKA-15605 Project: Kafka

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-11 Thread David Arthur
One thing we should consider is a static config to totally enable/disable the ELR feature. If I understand the KIP correctly, we can effectively disable the unclean recovery by setting the recovery strategy config to "none". This would make development and rollout of this feature a bit smoother.

Re: Apache Kafka 3.6.0 release

2023-10-05 Thread David Arthur
; > > which may > > > > > > > > > > > > > cause confusion for users, or poor behavior in > > clients. > > > > > > > > > > > > > Using the other REST API endpoints appears > > unaffected.

[jira] [Created] (KAFKA-15552) Duplicate Producer ID blocks during ZK migration

2023-10-05 Thread David Arthur (Jira)
David Arthur created KAFKA-15552: Summary: Duplicate Producer ID blocks during ZK migration Key: KAFKA-15552 URL: https://issues.apache.org/jira/browse/KAFKA-15552 Project: Kafka Issue Type

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-05 Thread David Arthur
Hey, just chiming in regarding the ZK migration piece. Generally speaking, one of the design goals of the migration was to have minimal changes on the ZK brokers and especially the ZK controller. Since ZK mode is our safe/well-known fallback mode, we wanted to reduce the chances of introducing

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-03 Thread David Arthur
Calvin, thanks for the KIP! I'm getting up to speed on the discussion. I had a few questions 57. When is the CleanShutdownFile removed? I think it probably happens after registering with the controller, but it would be good to clarify this. 58. Since the broker epoch comes from the controller,

[jira] [Created] (KAFKA-15532) ZkWriteBehindLag should not be reported by inactive controllers

2023-10-03 Thread David Arthur (Jira)
David Arthur created KAFKA-15532: Summary: ZkWriteBehindLag should not be reported by inactive controllers Key: KAFKA-15532 URL: https://issues.apache.org/jira/browse/KAFKA-15532 Project: Kafka

Re: [VOTE] 3.6.0 RC0

2023-09-19 Thread David Arthur
ish.dugg...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Greg, > > > > > > > Thanks for reporting the KafkaConnect issue. I replied to this > issue > > > > > &g

Re: [VOTE] 3.6.0 RC0

2023-09-18 Thread David Arthur
w runs of unit/integration tests. You can see the latest at > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will > continue > running a few more iterations. > System tests: > We will send an update once we have the results. > > Thanks, > Satish. > -- David Arthur

[jira] [Resolved] (KAFKA-15450) Disable ZK migration when JBOD configured

2023-09-12 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15450. -- Resolution: Fixed > Disable ZK migration when JBOD configu

Re: Apache Kafka 3.6.0 release

2023-09-12 Thread David Arthur
e/kafka/pull/14370. > > > > > > > > I understand if there are concerns about last minute changes to this > > API > > > > and we can hold off if that makes the most sense. > > > > If we take that route, I think we should still keep verification for >

Re: Apache Kafka 3.6.0 release

2023-09-11 Thread David Arthur
; > On Fri, 8 Sept 2023 at 19:23, Ismael Juma wrote: > > > > > > Hi Satish, > > > > > > Do you have a sense of when we'll publish RC0? > > > > > > Thanks, > > > Ismael > > > > > > On Fri, Sep 8, 2023 at 6:27 AM David Arthur

[jira] [Created] (KAFKA-15450) Disable ZK migration when JBOD configured

2023-09-11 Thread David Arthur (Jira)
David Arthur created KAFKA-15450: Summary: Disable ZK migration when JBOD configured Key: KAFKA-15450 URL: https://issues.apache.org/jira/browse/KAFKA-15450 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-15441) Broker sessions can time out during ZK migration

2023-09-08 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15441. -- Resolution: Fixed > Broker sessions can time out during ZK migrat

Re: Apache Kafka 3.6.0 release

2023-09-08 Thread David Arthur
Quick update on my two blockers: KAFKA-15435 is merged to trunk and cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will hopefully get it merged today. -David On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko wrote: > Hi Satish and all, > > I wonder if

[jira] [Resolved] (KAFKA-15435) KRaft migration record counts in log message are incorrect

2023-09-08 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15435. -- Resolution: Fixed > KRaft migration record counts in log message are incorr

Re: Apache Kafka 3.6.0 release

2023-09-06 Thread David Arthur
. Thanks! David On Tue, Sep 5, 2023 at 8:13 PM Satish Duggana wrote: > Hi David, > Thanks for bringing this issue to this thread. > I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker. > > Thanks, > Satish. > > On Tue, 5 Sept 2023 at 21:29, David Arthur

[jira] [Created] (KAFKA-15441) Broker sessions can time out during ZK migration

2023-09-06 Thread David Arthur (Jira)
David Arthur created KAFKA-15441: Summary: Broker sessions can time out during ZK migration Key: KAFKA-15441 URL: https://issues.apache.org/jira/browse/KAFKA-15441 Project: Kafka Issue Type

Re: Apache Kafka 3.6.0 release

2023-09-05 Thread David Arthur
> > > > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > > >> > > > > >> > > > > > > > > > > > > > wro

[jira] [Created] (KAFKA-15435) KRaft migration record counts in log message are incorrect

2023-09-05 Thread David Arthur (Jira)
David Arthur created KAFKA-15435: Summary: KRaft migration record counts in log message are incorrect Key: KAFKA-15435 URL: https://issues.apache.org/jira/browse/KAFKA-15435 Project: Kafka

[jira] [Created] (KAFKA-15389) MetadataLoader may publish an empty image on first start

2023-08-21 Thread David Arthur (Jira)
David Arthur created KAFKA-15389: Summary: MetadataLoader may publish an empty image on first start Key: KAFKA-15389 URL: https://issues.apache.org/jira/browse/KAFKA-15389 Project: Kafka

[jira] [Created] (KAFKA-15381) Controller waiting for migration should only allow failover when transactions are supported

2023-08-18 Thread David Arthur (Jira)
David Arthur created KAFKA-15381: Summary: Controller waiting for migration should only allow failover when transactions are supported Key: KAFKA-15381 URL: https://issues.apache.org/jira/browse/KAFKA-15381

[jira] [Created] (KAFKA-15374) ZK migration fails on configs for default broker resource

2023-08-17 Thread David Arthur (Jira)
David Arthur created KAFKA-15374: Summary: ZK migration fails on configs for default broker resource Key: KAFKA-15374 URL: https://issues.apache.org/jira/browse/KAFKA-15374 Project: Kafka

[jira] [Resolved] (KAFKA-15263) KRaftMigrationDriver can run the migration twice

2023-07-28 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15263. -- Resolution: Fixed > KRaftMigrationDriver can run the migration tw

[jira] [Created] (KAFKA-15263) KRaftMigrationDriver can run the migration twice

2023-07-27 Thread David Arthur (Jira)
David Arthur created KAFKA-15263: Summary: KRaftMigrationDriver can run the migration twice Key: KAFKA-15263 URL: https://issues.apache.org/jira/browse/KAFKA-15263 Project: Kafka Issue Type

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

2023-07-26 Thread David Arthur
Thanks for driving this KIP, Colin! +1 binding -David On Wed, Jul 26, 2023 at 8:58 AM Divij Vaidya wrote: > +1 (binding) > > -- > Divij Vaidya > > > On Wed, Jul 26, 2023 at 2:56 PM ziming deng > wrote: > > > > +1 (binding) from me. > > > > Thanks for the KIP! > > > > -- > > Ziming > > > > >

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-07-21 Thread David Arthur
Hey Colin, thanks for the KIP! Some questions 1) "This registration will include information about the endpoints which they possess" Will this include all endpoints, or only those configured in "advertised.listeners" 2) "Periodically, each controller will check that the controller registration

[jira] [Created] (KAFKA-15230) ApiVersions data between controllers is not reliable

2023-07-21 Thread David Arthur (Jira)
David Arthur created KAFKA-15230: Summary: ApiVersions data between controllers is not reliable Key: KAFKA-15230 URL: https://issues.apache.org/jira/browse/KAFKA-15230 Project: Kafka Issue

[jira] [Created] (KAFKA-15196) Additional ZK migration metrics

2023-07-17 Thread David Arthur (Jira)
David Arthur created KAFKA-15196: Summary: Additional ZK migration metrics Key: KAFKA-15196 URL: https://issues.apache.org/jira/browse/KAFKA-15196 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-15137) Don't log the entire request in KRaftControllerChannelManager

2023-06-30 Thread David Arthur (Jira)
David Arthur created KAFKA-15137: Summary: Don't log the entire request in KRaftControllerChannelManager Key: KAFKA-15137 URL: https://issues.apache.org/jira/browse/KAFKA-15137 Project: Kafka

[jira] [Resolved] (KAFKA-15098) KRaft migration does not proceed and broker dies if authorizer.class.name is set

2023-06-22 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15098. -- Resolution: Fixed > KRaft migration does not proceed and broker dies if authorizer.class.n

[jira] [Resolved] (KAFKA-15109) ISR shrink/expand issues on ZK brokers during migration

2023-06-22 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15109. -- Resolution: Fixed > ISR shrink/expand issues on ZK brokers during migrat

[jira] [Created] (KAFKA-15109) ISR not expanding on ZK brokers during migration

2023-06-20 Thread David Arthur (Jira)
David Arthur created KAFKA-15109: Summary: ISR not expanding on ZK brokers during migration Key: KAFKA-15109 URL: https://issues.apache.org/jira/browse/KAFKA-15109 Project: Kafka Issue Type

Re: [DISCUSS] Partial CI builds - Reducing flakiness with fewer tests

2023-06-14 Thread David Arthur
the added tests is flaky, but that's my > discretion. We can adopt a project policy of reverting whatever we > can, but I don't think that's a more welcoming or productive project > than we have now. > > Greg > > On Tue, Jun 13, 2023 at 7:24 AM David Arthur wrote: > > >

Re: [ANNOUNCE] New committer: Divij Vaidya

2023-06-13 Thread David Arthur
Congrats Divij! On Tue, Jun 13, 2023 at 12:34 PM Igor Soarez wrote: > Congratulations Divij! > > -- > Igor > -- -David

Re: [DISCUSS] Partial CI builds - Reducing flakiness with fewer tests

2023-06-13 Thread David Arthur
; > > I think this would be more likely to pass than the full suite due to > > > > the fewer tests failing probabilistically, and would improve the > > > > signal-to-noise ratio of the summary pass/fail marker on GitHub. This > > > > should also be shorter to execute than the full suite, allowing for > > > > faster cycle-time than the current full suite encourages. > > > > > > > > This would also strengthen the incentive for contributors > specializing > > > > in a module to de-flake tests, as they are rewarded with a tangible > > > > improvement within their area of the project. Currently, even the > > > > modules with the most reliable tests receive consistent CI failures > > > > from other less reliable modules. > > > > > > > > I believe this is possible, even if there isn't an off-the-shelf > > > > solution for it. We can learn of the changed files via a git diff, > map > > > > that to modules containing those files, and then execute the tests > > > > just for those modules with gradle. GitHub also permits showing > > > > multiple "checks" so that we can emit both the full-suite and partial > > > > test results. > > > > > > > > Thanks, > > > > Greg > > > > > -- David Arthur

Re: [VOTE] 3.5.0 RC1

2023-06-12 Thread David Arthur
ted upon: > > > >> > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > >> > > > > > >> > > * Javadoc: > > > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/ > > > >> > > > > > >> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag: > > > >> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1 > > > >> > > > > > >> > > * Documentation: > > > >> > > https://kafka.apache.org/35/documentation.html > > > >> > > > > > >> > > * Protocol: > > > >> > > https://kafka.apache.org/35/protocol.html > > > >> > > > > > >> > > * Successful Jenkins builds for the 3.5 branch: > > > >> > > Unit/integration tests: I'm struggling to get all tests to pass > > in the > > > >> > > same build. I'll run a few more builds to ensure each test pass > at > > > >> > > least once in the CI. All tests passed locally. > > > >> > > System tests: The build is still running, I'll send an update > > once I > > > >> > > have the results. > > > >> > > > > > >> > > Thanks, > > > >> > > Mickael > > > >> > > > > > >> > > > > > > > > > > > > -- > > > > [image: Aiven] <https://www.aiven.io> > > > > > > > > *Josep Prat* > > > > Open Source Engineering Director, *Aiven* > > > > josep.p...@aiven.io | +491715557497 > > > > aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud> > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > > > *Aiven Deutschland GmbH* > > > > Alexanderufer 3-7, 10117 Berlin > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > -- David Arthur

Re: [DISCUSS] Regarding Old PRs

2023-06-09 Thread David Arthur
Hey all, I just wanted to bump this one more time before I merge this PR (thanks for the review, Josep!). I'll merge it at the end of the day today unless anyone has more feedback. Thanks! David On Wed, Jun 7, 2023 at 8:50 PM David Arthur wrote: > I filed KAFKA-15073 for this. Here is a pa

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

2023-06-08 Thread David Arthur
te: > > On Thu, Jun 8, 2023, at 10:00, David Arthur wrote: > >> Colin, thanks for the KIP! These all seem like pretty useful additions. > A > >> few quick questions > >> > >> 1) Will the value of TimedOutBrokerHeartbeatCount be zero for inactive >

Re: [VOTE] KIP-938: Add more metrics for measuring KRaft performance

2023-06-08 Thread David Arthur
Colin, thanks for the KIP! These all seem like pretty useful additions. A few quick questions 1) Will the value of TimedOutBrokerHeartbeatCount be zero for inactive controllers? Will the value reset to zero after an election, or only process restart? 2) Does HandleLoadSnapshotCount include the

Re: [DISCUSS] Regarding Old PRs

2023-06-07 Thread David Arthur
n Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491715557497 > > w: aiven.io > > e: josep.p...@aiven.io > > On Wed, Jun 7, 2023, 20:28 David

[jira] [Created] (KAFKA-15073) Automation for old/inactive PRs

2023-06-07 Thread David Arthur (Jira)
David Arthur created KAFKA-15073: Summary: Automation for old/inactive PRs Key: KAFKA-15073 URL: https://issues.apache.org/jira/browse/KAFKA-15073 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] Regarding Old PRs

2023-06-07 Thread David Arthur
aintainer. > > > > If you look at my last email I described a flow (but AFAIU it will > work > > > > only if a human executes it) where the situation you point out would > be > > > > covered. > > > > > > > > ——— > > > > J

Re: [DISCUSS] Regarding Old PRs

2023-06-02 Thread David Arthur
maybe `stale` again > >> -- Reviewer: ping some reviewers that have experience or lately touched > >> this piece of the codebase, add a label `reviewer needed` or something > >> similar > >> - PRs that have `stale` label after X days, will be closed. > >> > >> Regarding the comments about only committers and collaborators being > able > >> to label PRs, this is true, not everyone can do this. However, this > could > >> be a great opportunity for the newly appointed contributors to exercise > >> their new permissions :) > >> > >> Let me know if it makes sense to you all. > >> > >> Best, > -- David Arthur

Re: [VOTE] 3.5.0 RC0

2023-06-02 Thread David Arthur
Olshan > >> > wrote: > >> > > > >> > > FYI -- I just saw this PR regarding a dependency for ARM. We may > want > >> to > >> > > get this in for 3.5 as well. It should be quick. > >> > > > >> > > http

Re: [VOTE] 3.5.0 RC0

2023-05-31 Thread David Arthur
gt; > Thanks, > > Mickael > > > > On Tue, May 30, 2023 at 10:32 AM Mickael Maison > > wrote: > > > > > > Hi Greg, > > > > > > Thanks for the heads up, this indeed looks like something we want in > > > 3.5. I've replied in the

Re: [VOTE] 3.5.0 RC0

2023-05-27 Thread David Arthur
; > > > > operations > > > > > > > > > > We are working on PRs for these issues and will get them in soon, > we > > > > think! > > > > > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for

Re: [VOTE] 3.5.0 RC0

2023-05-23 Thread David Arthur
Mickael, we have some migration fixes on trunk, is it okay to cherry-pick these to 3.5? KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator KAFKA-15009 Handle new ACLs in KRaft snapshot during migration There is another issue KAFKA-15010 that I'm also investigating to determine

[jira] [Created] (KAFKA-15017) New ClientQuotas are not written to ZK from snapshot

2023-05-23 Thread David Arthur (Jira)
David Arthur created KAFKA-15017: Summary: New ClientQuotas are not written to ZK from snapshot Key: KAFKA-15017 URL: https://issues.apache.org/jira/browse/KAFKA-15017 Project: Kafka Issue

Re: Migration from zookeeper to kraft not working

2023-05-18 Thread David Arthur
Elena, Did you provision a KRaft controller quorum before restarting the brokers? If you don't mind, could you create a JIRA and attach the config files used for the brokers before/after the migration along with the controller configs? Please include the sequence of steps you took in the JIRA as

Re: [DISCUSS] Apache Kafka 3.5.0 release

2023-05-06 Thread David Arthur
ould I move all of them to the next > release? > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the > migration. Should I move them too? > > Thanks, > Mickael > > On Fri, May 5, 2023 at 4:27 PM David Arthur > wrote: > > > > Hey Mickael, my two ZK mig

[jira] [Resolved] (KAFKA-14805) KRaft Controller shouldn't allow metadata updates before migration starts

2023-05-06 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-14805. -- Resolution: Fixed > KRaft Controller shouldn't allow metadata updates before migration sta

[jira] [Resolved] (KAFKA-14840) Handle KRaft snapshots in dual-write mode

2023-05-06 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-14840. -- Fix Version/s: (was: 3.4.1) Resolution: Fixed > Handle KRaft snapshots in d

Re: [DISCUSS] Apache Kafka 3.5.0 release

2023-05-05 Thread David Arthur
t; > https://issues.apache.org/jira/browse/KAFKA-14946 > > > > https://github.com/apache/kafka/pull/13653 > > > > > > > > Thanks. > > > > Luke > > > > > > > > > > > > > > > > On Fri, Apr 28, 2023 at 4:18 PM

Re: Question: CI Pipeline in Kafka GitHub Pull Requests & Github Actions Usage

2023-05-04 Thread David Arthur
Hello, Aaron and welcome to the project! If you look towards the bottom of a Pull Request there will be a section reporting the status of the Checks for the latest commit. There is a "Details" link that takes you to the Jenkins job for that PR. The default Jenkins view is the new UI (called Blue

[jira] [Created] (KAFKA-14964) ClientQuotaMetadataManager should not suppress exceptions

2023-05-03 Thread David Arthur (Jira)
David Arthur created KAFKA-14964: Summary: ClientQuotaMetadataManager should not suppress exceptions Key: KAFKA-14964 URL: https://issues.apache.org/jira/browse/KAFKA-14964 Project: Kafka

Re: [DISCUSS] Adding non-committers as Github collaborators

2023-04-28 Thread David Arthur
. > > >> > > >> There is actually also a way to allow people to re-trigger Jenkins > jobs: > > >> https://github.com/apache/kafka/pull/13578 > > >> > > >> Retriggering test is a little bit more sensitive as our resources are > > >> limite

Re: [DISCUSS] Recommendations for managing long-running projects

2023-04-28 Thread David Arthur
Kirk, 1) I would check out the project management features built into GitHub, e.g., labels, milestones, and projects https://docs.github.com/en/issues/planning-and-tracking-with-projects/learning-about-projects/about-projects . 2) Ultimately, the contributions will need to follow the normal PR

Re: [DISCUSS] Apache Kafka 3.5.0 release

2023-04-27 Thread David Arthur
Hey Mickael, I have one major ZK migration improvement (KAFKA-14805) that landed in trunk this week that I'd like to merge to 3.5 (once we fix some test failures it introduced). After that, I have another PR for KAFKA-14840 which is essentially a huge bug in the ZK migration logic that needs to

[DISCUSS] Adding non-committers as Github collaborators

2023-04-27 Thread David Arthur
Hey folks, I stumbled across this wiki page from the infra team that describes the various features supported in the ".asf.yaml" file: https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features One section that looked particularly interesting was

Re: Adding reviewers with Github actions

2023-04-27 Thread David Arthur
lot more convenient to have it in the commit than having to follow > > links, etc. > > > > David Arthur also wrote a script to help with this step, I believe. > > > > Ismael > > > > On Tue, Apr 18, 2023, 9:29 AM Divij Vaidya > > wrote: > > > > > D

[jira] [Created] (KAFKA-14939) Only expose ZkMigrationState metric if metadata.version supports it

2023-04-26 Thread David Arthur (Jira)
David Arthur created KAFKA-14939: Summary: Only expose ZkMigrationState metric if metadata.version supports it Key: KAFKA-14939 URL: https://issues.apache.org/jira/browse/KAFKA-14939 Project: Kafka

[jira] [Created] (KAFKA-14934) KafkaClusterTestKit makes FaultHandler accessible

2023-04-25 Thread David Arthur (Jira)
David Arthur created KAFKA-14934: Summary: KafkaClusterTestKit makes FaultHandler accessible Key: KAFKA-14934 URL: https://issues.apache.org/jira/browse/KAFKA-14934 Project: Kafka Issue Type

[jira] [Created] (KAFKA-14918) KRaft controller sending ZK controller RPCs to KRaft brokers

2023-04-18 Thread David Arthur (Jira)
David Arthur created KAFKA-14918: Summary: KRaft controller sending ZK controller RPCs to KRaft brokers Key: KAFKA-14918 URL: https://issues.apache.org/jira/browse/KAFKA-14918 Project: Kafka

[jira] [Resolved] (KAFKA-14796) Migrate ZK ACLs to KRaft

2023-04-17 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-14796. -- Resolution: Fixed Removed the 3.4.1 fix version since we're probably not back-porting

  1   2   3   4   5   6   >