for the new RC.
>
>- I checked the signatures and checksums
>- Built from source
>- Ran though the quickstart
>- Spot checked the docs
>
> +1(binding) from me modulo the previous issues pointed out.
>
> Thanks,
> Bill
>
> On Wed, Oct 30, 2024 at
David Arthur created KAFKA-17902:
Summary: Flaky Test kafka.admin.DeleteTopicTest
testAddPartitionDuringDeleteTopic
Key: KAFKA-17902
URL: https://issues.apache.org/jira/browse/KAFKA-17902
Project
[
https://issues.apache.org/jira/browse/KAFKA-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur reopened KAFKA-15918:
--
https://ge.apache.org/scans/tests?search.names=CI%20workflow%2CGit%20repository
; > - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> >> > >> > > - KIP-950: Tiered Storage Disablement
> >> > >> > > - KIP-956: Tiered Storage Quotas
> >> > >> > >
> >> > >> > > Release notes for the 3.9.0 release:
> >> > >> > >
> >> > >>
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/RELEASE_NOTES.html
> >> > >> > >
> >> > >> > > *** Please download, test and vote by October 30, 2024.
> >> > >> > >
> >> > >> > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> >> > >> > > https://kafka.apache.org/KEYS
> >> > >> > >
> >> > >> > > * Release artifacts to be voted upon (source and binary):
> >> > >> > > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/
> >> > >> > >
> >> > >> > > * Docker release artifacts to be voted upon:
> >> > >> > > apache/kafka:3.9.0-rc5
> >> > >> > > apache/kafka-native:3.9.0-rc5
> >> > >> > >
> >> > >> > > * Maven artifacts to be voted upon:
> >> > >> > >
> >> >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > >> > >
> >> > >> > > * Javadoc:
> >> > >> > >
> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/javadoc/
> >> > >> > >
> >> > >> > > * Documentation:
> >> > >> > > https://kafka.apache.org/39/documentation.html
> >> > >> > >
> >> > >> > > * Protocol:
> >> > >> > > https://kafka.apache.org/39/protocol.html
> >> > >> > >
> >> > >> > > * Tag to be voted upon (off 3.9 branch) is the 3.9.0-rc5 tag:
> >> > >> > > https://github.com/apache/kafka/releases/tag/3.9.0-rc5
> >> > >> > >
> >> > >> > > * Successful Docker Image Github Actions Pipeline for 3.9
> branch:
> >> > >> > > Docker Build Test Pipeline (JVM):
> >> > >> > > https://github.com/apache/kafka/actions/runs/11535300463
> >> > >> > > Docker Build Test Pipeline (Native):
> >> > >> > > https://github.com/apache/kafka/actions/runs/11535328957
> >> > >> > >
> >> > >> > > Thanks to everyone who helped with this release candidate,
> either
> >> by
> >> > >> > > contributing code, testing, or documentation.
> >> > >> > >
> >> > >> > > Regards,
> >> > >> > > Colin
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
--
David Arthur
ctories
>
> [3]
>
> https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#ConfigurationCompatibility
>
> [4]
>
> https://logging.apache.org/log4j/2.x/manual/configuration.html#java-properties-features
>
> [5] https://logging.apache.org/xml/ns/
>
>
--
David Arthur
David Arthur created KAFKA-17886:
Summary: BrokerLifecycleManagerTest
testAlwaysSendsAccumulatedOfflineDirs hanging in CI
Key: KAFKA-17886
URL: https://issues.apache.org/jira/browse/KAFKA-17886
David Arthur created KAFKA-17767:
Summary: Capture tests run on trunk into test catalog
Key: KAFKA-17767
URL: https://issues.apache.org/jira/browse/KAFKA-17767
Project: Kafka
Issue Type: Sub
David Arthur created KAFKA-17766:
Summary: TopicBasedRemoteLogMetadataManager stuck in close
Key: KAFKA-17766
URL: https://issues.apache.org/jira/browse/KAFKA-17766
Project: Kafka
Issue Type
David Arthur created KAFKA-17712:
Summary: Run recently added integration tests in quarantine
Key: KAFKA-17712
URL: https://issues.apache.org/jira/browse/KAFKA-17712
Project: Kafka
Issue
[
https://issues.apache.org/jira/browse/KAFKA-15914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur reopened KAFKA-15914:
--
This has re-occurred on trunk
[https://ge.apache.org/scans/tests?search.names=CI%20workflow
David Arthur created KAFKA-17684:
Summary: Run tests on Java 11 and Java 21
Key: KAFKA-17684
URL: https://issues.apache.org/jira/browse/KAFKA-17684
Project: Kafka
Issue Type: Sub-task
cript"
> >> >>> "Pre-print"
> >> >>> "Published"
> >> >>>
> >> >>>
> >> >>
> >> >> The need to standardize terminology also means that, sorry, you have
> >> to choose. :) This is actually a feedback I often give on KIPs. People
> like
> >> to add sections that say "maybe we'll do X, maybe we'll do Y." But to
> make
> >> progress on the KIP, you have to choose either X or Y and put the other
> one
> >> in the "rejected alternatives" section.
> >> >>
> >> >> I think our purpose in choosing names should be clarity for users and
> >> developers. That's why I suggested "not implemented", "alpha", "beta",
> >> "production ready". I am curious what your thoughts are about these.
> >> >>
> >> >> best,
> >> >> Colin
> >>
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io/>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io | +491715557497
> > aiven.io <https://www.aiven.io/> |
> > <https://www.facebook.com/aivencloud>
> > <https://www.linkedin.com/company/aiven/> <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io/>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io | +491715557497
> aiven.io <https://www.aiven.io/> | <
> https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/> <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>
--
David Arthur
David Arthur created KAFKA-17680:
Summary: StandbyTaskEOSMultiRebalanceIntegrationTest caused build
failure
Key: KAFKA-17680
URL: https://issues.apache.org/jira/browse/KAFKA-17680
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-17673.
--
Resolution: Fixed
Fixed in https://github.com/apache/kafka/pull/17336
> Gradle Build Scan
ou, David
>
> Best,
> TengYao
>
> 吳岱儒 於 2024年9月30日 週一，下午6:01寫道：
>
> > +1 (non-binding)
> > Thanks David, I do love this flaky tag.
> >
> > TJ
> >
>
--
David Arthur
nistration of groups.
> > > This KIP enhances the command-line tools to make it easier to
> administer
> > > groups on clusters with a variety of types of groups.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1043%3A+Administration+of+groups
> > > >
> > > > Thanks.
> > > > Andrew
> > >
> >
>
--
David Arthur
Progress has continued over the past weeks to improve our builds. Thanks to
everyone who has contributed to this effort so far. Here is an update.
*Jenkins Build Disabled*
We have removed the Jenkinsfile from trunk which has disabled the Jenkins
build. There was much rejoicing throughout the land.
David Arthur created KAFKA-17673:
Summary: Gradle Build Scan PR check not showing
Key: KAFKA-17673
URL: https://issues.apache.org/jira/browse/KAFKA-17673
Project: Kafka
Issue Type: Bug
David Arthur created KAFKA-17672:
Summary: Run quarantined tests separately
Key: KAFKA-17672
URL: https://issues.apache.org/jira/browse/KAFKA-17672
Project: Kafka
Issue Type: Sub-task
ould we see the reports (quarantined tests, etc) mentioned in the
> KIP?
> 2. The quarantine process seems to only apply to integration tests. What's
> our recommendation for flaky unit tests?
>
> Jun
>
> On Thu, Sep 26, 2024 at 1:34 PM David Arthur wrote:
>
> > If th
gt; Chia-Ping
>
> On 2024/09/26 21:37:17 David Arthur wrote:
> > Josep, I've filed KAFKA-17628 and submitted a PR to partially automate
> the
> > workflow approval. With this change, we just need a committer to add a
> > label one time to a PR, then it will get auto
>> >> I have implemented a patch to avoid these
> >> UnsupportedOperationExceptions,
> >> >> and effectively make Kafka compatible with Java 23 by default.
> >> >> Issue: https://issues.apache.org/jira/browse/KAFKA-17078 and the
> >> associated
&
4, 7:47 p.m. Matthias J. Sax
> wrote:
> >
> > > +1
> > >
> > > On 9/26/24 3:38 PM, Chia-Ping Tsai wrote:
> > > > +1
> > > >
> > > > nit: Could you please add the KIP link to KAFKA-17629
> > > >
> > > > David Arthur
I would like to call a vote for KIP-1090. Please take a moment to review
the proposal and cast your vote.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1090+Flaky+Test+Management
Thanks!
David A
David Arthur created KAFKA-17629:
Summary: KIP-1090 Flaky Test Management
Key: KAFKA-17629
URL: https://issues.apache.org/jira/browse/KAFKA-17629
Project: Kafka
Issue Type: Improvement
t;>>> > Do you see any potential impact if we backport the change to
> those?
> > > >>>>
> > > >>>> In my opinion, the main concern is that non-trunk PRs can't
> > > effectively
> > > >>>> leverage the cache, meaning
If there is no more feedback on this, I'll go ahead and move to a vote.
-David
On Sun, Sep 22, 2024 at 11:04 AM Chia-Ping Tsai wrote:
>
>
> > David Arthur 於 2024年9月22日 晚上10:07 寫道：
> >
> > Q2: Yes, I think we should run the quarantined tests on all CI builds
David Arthur created KAFKA-17628:
Summary: Automate workflow approvals for CI
Key: KAFKA-17628
URL: https://issues.apache.org/jira/browse/KAFKA-17628
Project: Kafka
Issue Type: Improvement
ithub-ci is triggered by trunk branch only, and we have
> >>>> not
> >>>> tested it on non-trunk branch yet. Given that 3.9.0 and 3.8.1 releases
> >>>> are
> >>>> processing, we could continue using Jenkins CI to avoid the additional
> >>>> ove
Today, we disabled the Jenkins build on trunk. With this change, we should
now be expecting all green status checks on PRs before merging. Of course,
flaky tests still exist, but generally speaking we should have green builds
(see KIP-1090 for some plans on flaky tests).
Any committer or "collabor
David Arthur created KAFKA-17607:
Summary: Add CI step to verify LICENSE-binary
Key: KAFKA-17607
URL: https://issues.apache.org/jira/browse/KAFKA-17607
Project: Kafka
Issue Type
David Arthur created KAFKA-17606:
Summary: Include Rat errors in GitHub workflow summary
Key: KAFKA-17606
URL: https://issues.apache.org/jira/browse/KAFKA-17606
Project: Kafka
Issue Type
David Arthur created KAFKA-17603:
Summary: KRaftMigrationDriverTest testNoDualWriteBeforeMigration
became flaky
Key: KAFKA-17603
URL: https://issues.apache.org/jira/browse/KAFKA-17603
Project: Kafka
,
> > > > > > David
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > [image: Aiven] <https://www.aiven.io>
> > > > >
> > > > > *Josep Prat*
> > > > > Open Source Engineering Director, *Aiven*
> > > > > josep.p...@aiven.io | +491715557497
> > > > > aiven.io <https://www.aiven.io> | <
> > > > https://www.facebook.com/aivencloud
> > > > > >
> > > > > <https://www.linkedin.com/company/aiven/> <
> > > > > https://twitter.com/aiven_io>
> > > > > *Aiven Deutschland GmbH*
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > Anna Richardson, Kenneth Chen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > >
> > >
>
--
David Arthur
ncluded by PR’s CI? If yes, could we (committers
> and author) ignore the failure of quarantine tests? If not, should we
> remove the retry from PR flow as the non-quarantine tests should be stable?
>
> Best,
> Chia-Ping
--
David Arthur
run the tests only for the modules changed by PR. That is
> used to reduce the usage of GitHub runner. However, it is hard to ensure
> the “change” of client module won’t impact the tests of core modules.
> Hence, should we run all module tests if some important modules are touched
> by
le
> > doesn't have any integration tests and all of the tests are meant to
> > be deterministic. It would be dangerous to the protocol's correctness
> > and the consistency of the cluster metadata to allow contributors to
> > mark tests as flaky in the raft module in
David Arthur created KAFKA-17589:
Summary: Move JUnit extensions to test-common module
Key: KAFKA-17589
URL: https://issues.apache.org/jira/browse/KAFKA-17589
Project: Kafka
Issue Type: Sub
David Arthur created KAFKA-17588:
Summary: Remove internal classes from ClusterInstance
Key: KAFKA-17588
URL: https://issues.apache.org/jira/browse/KAFKA-17588
Project: Kafka
Issue Type: Sub
David Arthur created KAFKA-17587:
Summary: Move test infrastructure out of core
Key: KAFKA-17587
URL: https://issues.apache.org/jira/browse/KAFKA-17587
Project: Kafka
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/KAFKA-17307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-17307.
--
Resolution: Duplicate
> Remove junit-platform.properties from test J
ery Gradle Develocity with "7 days
> ago"
> 3) the test cases having fewer builds are viewed as "new tests"
>
> WDYT?
>
> David Arthur 於 2024年9月19日 週四 下午11:56寫道：
>
> > TengYao,
> >
> > > These two mechanisms are independent. We could m
tiered approach, the test might have two
> > states.(Isolated by developer, Quarantined automatically).
> > These two mechanisms are independent. We could manually remove a tag
> from a
> > test, but at the same time, it might still be quarantined.
> > I know the above s
retryable
> > > 2. we don't need to worry that "quarantined.txt" having out-of-date
> test
> > > names
> > > 3. we can require the flaky annotation must have jira link. That means
> > the
> > > PR's author must create the jira link for
David Arthur created KAFKA-17580:
Summary: Java 21 spurious compilation failure
Key: KAFKA-17580
URL: https://issues.apache.org/jira/browse/KAFKA-17580
Project: Kafka
Issue Type: Bug
[
https://issues.apache.org/jira/browse/KAFKA-17559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-17559.
--
Assignee: Kamal Chandraprakash (was: Chia-Ping Tsai)
Resolution: Fixed
Hello, Kafka community!
Looking at the last 7 days of GitHub, we have 59 out of 64 trunk builds
having flaky tests. Excluding timeouts (a separate issue), only 4 builds
out of the last 7 days have failed due to excess test failures. This is
actually a slight improvement when compared with the last
David Arthur created KAFKA-17572:
Summary: Clear build scan PR status when re-running the tests
Key: KAFKA-17572
URL: https://issues.apache.org/jira/browse/KAFKA-17572
Project: Kafka
Issue
David Arthur created KAFKA-17559:
Summary: RemoteLogOffsetReaderTest flaky on trunk
Key: KAFKA-17559
URL: https://issues.apache.org/jira/browse/KAFKA-17559
Project: Kafka
Issue Type: Test
David Arthur created KAFKA-17553:
Summary: DefaultTaskManager.shutdown stalling on Github CI
Key: KAFKA-17553
URL: https://issues.apache.org/jira/browse/KAFKA-17553
Project: Kafka
Issue Type
he.org/confluence/pages/viewpage.action?pageId=318606528
>
> Discussion:
> https://lists.apache.org/thread/wtgt9jql43qmfsmvqcz0y1phc2n08440
>
> Thank you,
>
> Max
>
>
>
--
David Arthur
David Arthur created KAFKA-17542:
Summary: Use actions/labeler for automatic PR labeling
Key: KAFKA-17542
URL: https://issues.apache.org/jira/browse/KAFKA-17542
Project: Kafka
Issue Type
David Arthur created KAFKA-17540:
Summary: Create floating tag on trunk for CI
Key: KAFKA-17540
URL: https://issues.apache.org/jira/browse/KAFKA-17540
Project: Kafka
Issue Type: Task
A lot has been happening with the GitHub Actions build in the past few
weeks. I thought I would share some updates.
*Build Statistics*
Now that we have all PRs builds running the test suite (see note below), we
can do a better comparison between GH and Jenkins
Github Actions
Successful trunk buil
[
https://issues.apache.org/jira/browse/KAFKA-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-15073.
--
Resolution: Fixed
Resolving this since we've merged the change to actually close ol
or actions/stale: https://github.com/actions/stale
Cheers,
David A
On Sat, Jun 10, 2023 at 2:53 AM David Jacot wrote:
> Thanks, David. I left a few comments in the PR.
>
> -David
>
> Le ven. 9 juin 2023 à 15:31, David Arthur .invalid>
> a écrit :
>
> > Hey all, I just
[
https://issues.apache.org/jira/browse/KAFKA-17506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-17506.
--
Resolution: Fixed
> KRaftMigrationDriver initialization race condit
David Arthur created KAFKA-17524:
Summary: StreamThreadTest
shouldReturnErrorIfProducerInstanceIdNotInitialized hanging
Key: KAFKA-17524
URL: https://issues.apache.org/jira/browse/KAFKA-17524
Project
Yash Mayya
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Congratulations Jeff!
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, 9 Sept, 2024, 12:13 David Jacot,
> >>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> The PMC of Apache Kafka is pleased to announce a new Kafka
> >>>>>>>> committer,
> >>>>>>>>>>> Jeff
> >>>>>>>>>>>> Kim.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jeff has been a Kafka contributor since May 2020. In addition
> >>>>>> to
> >>>>>>>>> being
> >>>>>>>>>>>> a regular contributor and reviewer, he has made significant
> >>>>>>>>>>>> contributions to the next generation of the consumer rebalance
> >>>>>>>>>>>> protocol (KIP-848) and to the new group coordinator. He
> >>>>>> authored
> >>>>>>>>>>>> KIP-915 which improved how coordinators can be downgraded. He
> >>>>>>> also
> >>>>>>>>>>>> contributed multiple fixes/improvements to the fetch from
> >>>>>>> follower
> >>>>>>>>>>>> feature.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Congratulations, Jeff!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> David (on behalf of the Apache Kafka PMC)
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >
> >
> >
>
--
David Arthur
David Arthur created KAFKA-17506:
Summary: ZkMigrationFailoverTest is flaky
Key: KAFKA-17506
URL: https://issues.apache.org/jira/browse/KAFKA-17506
Project: Kafka
Issue Type: Test
ours.
Next steps I'd like to take
1) Fully enable the GH workflows for all PRs (not just ones with gh- prefix)
2) Continue investigating the build cache (
https://issues.apache.org/jira/browse/KAFKA-17479)
3) Prioritize fixes for the worst flaky tests
4) Identify tests which are causing bu
[
https://issues.apache.org/jira/browse/KAFKA-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-16045.
--
Resolution: Fixed
> ZkMigrationIntegrationTest.testMigrateTopicDeletion fl
David Arthur created KAFKA-17457:
Summary: Raise minimum MetadataVersion for ZK migration to 3.6
Key: KAFKA-17457
URL: https://issues.apache.org/jira/browse/KAFKA-17457
Project: Kafka
Issue
gt; >
> > >
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io | +491715557497
> > aiven.io <https://www.aiven.io> |
David Arthur created KAFKA-17433:
Summary: Add a CI job to run a single test N times
Key: KAFKA-17433
URL: https://issues.apache.org/jira/browse/KAFKA-17433
Project: Kafka
Issue Type
s that levels can only be
> > >> changed in minors and majors, this means that if we don't say anything
> > >> else, the minimum "baking time" would be 1 minor release. This is the
> > >> technical lower limit. We could mention that
t;>>>>> understand the Docker image is level 4.
> >>>>>>>>>
> >>>>>>>>> Does that make sense? If so I can update the KIP with those
> >>>>>> examples.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Josep Prat
> >>>>>>>>> Open Source Engineering Director, Aiven
> >>>>>>>>> josep.p...@aiven.io | +491715557497 | aiven.io
> >>>>>>>>> Aiven Deutschland GmbH
> >>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >>>>>>>>> Anna Richardson, Kenneth Chen
> >>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>>
> >>>>>>>>> On Sun, Aug 18, 2024, 21:46 Chia-Ping Tsai
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> hi Josep
> >>>>>>>>>>
> >>>>>>>>>> Although I didn't join the discussion before, the KIP is
> >>>>>> interesting
> >>>>>>>> and
> >>>>>>>>>> great to me.
> >>>>>>>>>>
> >>>>>>>>>> one small comment:
> >>>>>>>>>>
> >>>>>>>>>> Could you please add existent features as an example to each
> level
> >>>>>>> for
> >>>>>>>>> the
> >>>>>>>>>> readers who have poor reading (like me) ? For instance, I guess
> >>>>>> the
> >>>>>>> new
> >>>>>>>>>> coordinator is level 3? tiered storage is level 4?
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Josep Prat 於 2024年8月19日 週一
> >>>>>> 上午2:13寫道：
> >>>>>>>>>>
> >>>>>>>>>>> Hi all,
> >>>>>>>>>>> I want to start a discussion for KIP-1081: Graduation Steps for
> >>>>>>>>> Features.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1081%3A+Graduation+Steps+for+Features
> >>>>>>>>>>>
> >>>>>>>>>>> We already had a bit of a discussion here
> >>>>>>>>>>>
> >>>>>> https://lists.apache.org/thread/5z6rxvs9m0bro5ssmtg8qcgdk40882bz
> >>>>>>> and
> >>>>>>>>>> that
> >>>>>>>>>>> materialized into this KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> I deliberately defined the graduation steps without giving them
> >>>>>> a
> >>>>>>>> name,
> >>>>>>>>>> so
> >>>>>>>>>>> we don't go bike-shedding there. There is a separate section
> for
> >>>>>>> the
> >>>>>>>>>> names
> >>>>>>>>>>> of each step. Also an alternative set of names. I'd like to get
> >>>>>>> some
> >>>>>>>>>>> feedback on the steps, and also on the names for the steps.
> >>>>>>>>>>>
> >>>>>>>>>>> Looking forward to your opinions, and hopefully only a tiny bit
> >>>>>> of
> >>>>>>>>>>> bike-shedding :)
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> [image: Aiven] <https://www.aiven.io/>
> >>>>>>>>>>>
> >>>>>>>>>>> *Josep Prat*
> >>>>>>>>>>> Open Source Engineering Director, *Aiven*
> >>>>>>>>>>> josep.p...@aiven.io | +491715557497
> >>>>>>>>>>> aiven.io <https://www.aiven.io/> | <
> >>>>>>>>>> https://www.facebook.com/aivencloud
> >>>>>>>>>>>>
> >>>>>>>>>>> <https://www.linkedin.com/company/aiven/> <
> >>>>>>>>>>> https://twitter.com/aiven_io>
> >>>>>>>>>>> *Aiven Deutschland GmbH*
> >>>>>>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >>>>>>>>>>> Anna Richardson, Kenneth Chen
> >>>>>>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> [image: Aiven] <https://www.aiven.io/>
> >>>>>>>
> >>>>>>> *Josep Prat*
> >>>>>>> Open Source Engineering Director, *Aiven*
> >>>>>>> josep.p...@aiven.io | +491715557497
> >>>>>>> aiven.io <https://www.aiven.io/> | <
> >>>>>> https://www.facebook.com/aivencloud
> >>>>>>>>
> >>>>>>> <https://www.linkedin.com/company/aiven/> <
> >>>>>>> https://twitter.com/aiven_io>
> >>>>>>> *Aiven Deutschland GmbH*
> >>>>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >>>>>>> Anna Richardson, Kenneth Chen
> >>>>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> [image: Aiven] <https://www.aiven.io/>
> >>>>>
> >>>>> *Josep Prat*
> >>>>> Open Source Engineering Director, *Aiven*
> >>>>> josep.p...@aiven.io | +491715557497
> >>>>> aiven.io <https://www.aiven.io/> |
> >>>>> <https://www.facebook.com/aivencloud>
> >>>>> <https://www.linkedin.com/company/aiven/> <
> >>> https://twitter.com/aiven_io>
> >>>>> *Aiven Deutschland GmbH*
> >>>>> Alexanderufer 3-7, 10117 Berlin
> >>>>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >>>>> Anna Richardson, Kenneth Chen
> >>>>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> [image: Aiven] <https://www.aiven.io/>
> >>
> >> *Josep Prat*
> >> Open Source Engineering Director, *Aiven*
> >> josep.p...@aiven.io | +491715557497
> >> aiven.io <https://www.aiven.io/> | <
> https://www.facebook.com/aivencloud>
> >> <https://www.linkedin.com/company/aiven/> <
> https://twitter.com/aiven_io>
> >> *Aiven Deutschland GmbH*
> >> Alexanderufer 3-7, 10117 Berlin
> >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >> Anna Richardson, Kenneth Chen
> >> Amtsgericht Charlottenburg, HRB 209739 B
>
--
David Arthur
-David
On Thu, Aug 22, 2024 at 3:04 PM David Arthur wrote:
> The Github public runners (which we are using) only offer windows, mac,
> and linux (x86_64). It is possible to set up dedicated "self-hosted"
> runners for a project (or org) which would allow whatever architectur
jira/browse/INFRA-26011 if you are
> interested in the details.
>
> I'm wondering if we also get access to other architectures via GitHub
> actions?
>
> Thanks,
> Mickael
>
> On Fri, Aug 16, 2024 at 6:02 PM David Arthur wrote:
> >
> > Josep,
> >
David Arthur created KAFKA-17381:
Summary: Reduce log output from checkstyle task
Key: KAFKA-17381
URL: https://issues.apache.org/jira/browse/KAFKA-17381
Project: Kafka
Issue Type: Task
is to write back the result of the CI run as a comment
> > on
> > > the PR itself. I believe not needing to periodically check CI to see if
> > the
> > > run finished would be a great win. By having CI commenting on the PR
> > > everyone watching the PR (author and reviewers) will get notified when
> > it's
> > > done.
> >
> >
>
--
David Arthur
David Arthur created KAFKA-17352:
Summary: Upgrade Gradle Enterprise to develocity plugin
Key: KAFKA-17352
URL: https://issues.apache.org/jira/browse/KAFKA-17352
Project: Kafka
Issue Type
Hey everyone,
Over the past several months (years, maybe?) I've tinkered around with
GitHub Actions as a possible alternative to Jenkins for Apache Kafka CI. I
think it is time to actually give it an earnest try.
We have already done some work with GH Actions. Namely the Docker build and
the "sta
[
https://issues.apache.org/jira/browse/KAFKA-15330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-15330.
--
Resolution: Won't Fix
Closing since it is not recommended to perform a migration on 3
er/txn requests) and operator of the cluster
> need them to take a decision. The pagination just provides them with a way
> to escape the timeout problem with large clusters. So am not sure adding
> throttling during such time would be wise.
> Am happy to add section for throttli
gt; > > have nothing to do. It's opt-in as you need to set cordoned.log.dirs
> > > on some brokers to get the new behavior. If you don't want it anymore,
> > > you should unset cordoned.log.dirs. Can you explain why this would not
> > > work?
> &g
; cordoned
> > > > > > > > log dir?
> > > > > > > > From the current java doc of the interface, it doesn't look
> > > right:
> > > > > > > > "Get the default directory for new partitions placed in a
> given
> > > > > > broker."
> > > > > > > >
> > > > > > > > 4. Currently, if a broker is registered and then go offline.
> In
> > > this
> > > > > > state,
> > > > > > > > the controller will still distribute partitions to this
> broker.
> > > > > > > > So, if now, the broker get startup with "cordoned.log.dirs"
> set,
> > > what
> > > > > > will
> > > > > > > > happen?
> > > > > > > > Will the newly assigned partitions be created successfully or
> > > not?
> > > > > > > >
> > > > > > > > 5. I think after a log dir get cordoned, we can always
> uncordon
> > > it,
> > > > > > right?
> > > > > > > > I think we should mention it in the KIP.
> > > > > > > >
> > > > > > > > 6. If a broker is startup with "cordoned.log.dirs" set, and
> does
> > > that
> > > > > > mean
> > > > > > > > the internal topics partitions (ex: __consumer_offsets)
> cannot be
> > > > > > created,
> > > > > > > > either?
> > > > > > > > Also, if this log dir is happen to be the metadata log dir,
> what
> > > will
> > > > > > > > happen to the metadata topic creation?
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > Luke
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jul 9, 2024 at 12:12 AM Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > Thanks for taking a look.
> > > > > > > > >
> > > > > > > > > - Yes you're right, I meant AlterPartitionReassignments.
> Fixed.
> > > > > > > > > - That's a good idea. I was expecting users to discover
> > > cordoned log
> > > > > > > > > directories by describing broker configurations. But being
> > > able to
> > > > > > > > > also get this information when describing log directories
> makes
> > > > > > sense.
> > > > > > > > > I've added that to the KIP.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 5, 2024 at 8:05 AM Haruki Okada <
> > > ocadar...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > Thank you for the KIP.
> > > > > > > > > > The motivation sounds make sense to me.
> > > > > > > > > >
> > > > > > > > > > I have a few questions:
> > > > > > > > > >
> > > > > > > > > > - [nits] "AlterPartitions request" in Error handling
> section
> > > is
> > > > > > > > > > "AlterPartitionReassignments request" actually, right?
> > > > > > > > > > - Don't we need to include cordoned information in
> > > DescribeLogDirs
> > > > > > > > > response
> > > > > > > > > > too? Some tools (e.g. CruiseControl) need to have a way
> to
> > > know
> > > > > > which
> > > > > > > > > > broker/log-dirs are cordoned to generate partition
> > > reassignment
> > > > > > > > proposal.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > 2024年7月4日(木) 22:57 Mickael Maison <
> mickael.mai...@gmail.com
> > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I'd like to start a discussion on KIP-1066 that
> introduces
> > > a
> > > > > > > > mechanism
> > > > > > > > > > > to cordon log directories and brokers.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Mickael
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Okada Haruki
> > > > > > > > > > ocadar...@gmail.com
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > >
>
--
David Arthur
y nice improvement for
> administering
> >>> large clusters.
> >>>>>
> >>>>> AS1: Besides topics, the most numerous resources in Kafka clusters in
> >>> my experience
> >>>>> are consumer groups. Would it be possible to extend the KIP to cover
> >>> ListGroups while
> >>>>> you’re in here? I’ve heard of clusters with truly vast numbers of
> >>> groups. This is also
> >>>>> potentially a sign of a misbehaving or poorly written clients.
> Getting
> >>> a page of groups
> >>>>> with a massive ItemsLeftToFetch would be nice.
> >>>>>
> >>>>> AS2: A tiny nit: The versions for the added fields are incorrect in
> >>> some cases.
> >>>>>
> >>>>> AS3: I don’t quite understand the cursor for
> >>> OffsetFetchRequest/Response.
> >>>>> It looks like the cursor is (topic, partition), but not group ID.
> Does
> >>> the cursor
> >>>>> apply to all groups in the request, or is group ID missing?
> >>>>>
> >>>>> AS4: For the remaining request/response pairs, the cursor makes sense
> >>> to me,
> >>>>> but I do wonder whether `NextCursor` should be at the top level of
> the
> >>> responses
> >>>>> instead, like DescribeTopicPartitionsResponse.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>>> On 27 Jun 2024, at 14:05, Omnia Ibrahim
> >>> wrote:
> >>>>>>
> >>>>>> Hi everyone, I would like to start a discussion thread for KIP-1062
> >>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1062%3A+Introduce+Pagination+for+some+requests+used+by+Admin+API
> >>>>>>
> >>>>>>
> >>>>>> Thanks
> >>>>>> Omnia
> >>>
> >>>
> >>>
> >
>
>
--
David Arthur
[
https://issues.apache.org/jira/browse/KAFKA-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-16463.
--
Resolution: Fixed
> Automatically delete metadata log directory on ZK brok
David Arthur created KAFKA-16667:
Summary: KRaftMigrationDriver gets stuck after successive failovers
Key: KAFKA-16667
URL: https://issues.apache.org/jira/browse/KAFKA-16667
Project: Kafka
Hi Fred, thanks for the KIP. Seems like a useful improvement.
As others have mentioned, I think we should avoid exposing Record in this
way.
Using ConsumerRecord seems okay, but maybe not the best fit for this case
(for the reasons Matthias gave).
Maybe we could create a new container interface
David Arthur created KAFKA-16539:
Summary: Can't update specific broker configs in pre-migration mode
Key: KAFKA-16539
URL: https://issues.apache.org/jira/browse/KAFKA-16539
Project:
Andrew, thanks for the KIP! This is a pretty exciting effort.
I've finally made it through the KIP, still trying to grok the whole thing.
Sorry if some of my questions are basic :)
Concepts:
70. Does the Group Coordinator communicate with the Share Coordinator over
RPC or directly in-process?
David Arthur created KAFKA-16468:
Summary: Listener not found error in SendRPCsToBrokersEvent
Key: KAFKA-16468
URL: https://issues.apache.org/jira/browse/KAFKA-16468
Project: Kafka
Issue
David Arthur created KAFKA-16466:
Summary: QuorumController is swallowing some exception messages
Key: KAFKA-16466
URL: https://issues.apache.org/jira/browse/KAFKA-16466
Project: Kafka
Issue
David Arthur created KAFKA-16463:
Summary: Automatically delete metadata log directory on ZK brokers
Key: KAFKA-16463
URL: https://issues.apache.org/jira/browse/KAFKA-16463
Project: Kafka
David Arthur created KAFKA-16446:
Summary: Log slow controller events
Key: KAFKA-16446
URL: https://issues.apache.org/jira/browse/KAFKA-16446
Project: Kafka
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/KAFKA-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-16180.
--
Resolution: Fixed
> Full metadata request sometimes fails during zk migrat
[
https://issues.apache.org/jira/browse/KAFKA-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Arthur resolved KAFKA-16171.
--
Resolution: Fixed
> Controller failover during ZK migration can prevent metadata updates
Andrew/Jose, I like the suggested Flow API. It's also similar to the stream
observers in GPRC. I'm not sure we should expose something as complex as
the Flow API directly in KafkaAdminClient, but certainly we can provide a
similar interface.
---
Cancellations:
Another thing not yet discussed is h
also want to have the
>pagination on huge topics for their partitions.
>2. To avoid OOM, we should only fetch the new topics when we need them
>and release the used topics. Especially the main use case of looping the
>topic list is when the client prints all the topics.
&
> >> like
> >> > > > that would not be too hard.
> >> > > >
> >> > > > Ismael
> >> > > >
> >> > > > On Fri, Feb 9, 2024 at 9:03 AM Greg Harris
> >> >> > > >
> >> > > > wrote:
>
te:
> >
> > > Hi David,
> > >
> > > +1 on that strategy.
> > >
> > > I see several flaky tests that aren't marked with @Tag("integration")
> > > or @IntegrationTest, and I think those would make using the unitTest
> > &
he validation checks to be
> reliable
> > (or else the PR won't be merged). Sounds like you are suggesting to skip
> > the tests for the merge queue validation. Could we perhaps include the
> unit
> > tests as well? That would incentivize us to ensure the unit tests are
>
b stats, we are averaging under 40 commits per week.
Assuming those primarily come in on weekdays, that's 8 commits per day. If
we just run "gradlew check -x tests" for the merge queue job, I don't think
we'd get backlogged.
Thoughts?
David
--
David Arthur
David Arthur created KAFKA-16206:
Summary: ZkConfigMigrationClient tries to delete topic configs
twice
Key: KAFKA-16206
URL: https://issues.apache.org/jira/browse/KAFKA-16206
Project: Kafka
David Arthur created KAFKA-16205:
Summary: Reduce number of metadata requests during hybrid mode
Key: KAFKA-16205
URL: https://issues.apache.org/jira/browse/KAFKA-16205
Project: Kafka
Issue
David Arthur created KAFKA-16171:
Summary: Controller failover during ZK migration can lead to
controller unavailability for ZK brokers
Key: KAFKA-16171
URL: https://issues.apache.org/jira/browse/KAFKA-16171
;
> > > > > > > On Tue, Jan 2, 2024 at 4:30 PM Ismael Juma
> > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to start a vote on KIP-1013.
> > > > > > > >
> > > > > > > > As stated in the discussion thread, this KIP was proposed
> > after the
> > > > > KIP
> > > > > > > > freeze for Apache Kafka 3.7, but it is purely a documentation
> > > > update
> > > > > > (if we
> > > > > > > > decide to adopt it) and I believe it would serve our users
> > best if
> > > > we
> > > > > > > > communicate the deprecation for removal sooner (i.e. 3.7)
> > rather
> > > > than
> > > > > > later
> > > > > > > > (i.e. 3.8).
> > > > > > > >
> > > > > > > > Please take a look and cast your vote.
> > > > > > > >
> > > > > > > > Link:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
--
David Arthur
David Arthur created KAFKA-16078:
Summary: InterBrokerProtocolVersion defaults to non-production
MetadataVersion
Key: KAFKA-16078
URL: https://issues.apache.org/jira/browse/KAFKA-16078
Project: Kafka
1 - 100 of 708 matches
Mail list logo
