KAFKA-15567 Review request

[Haruki Okada]

Hi!

I found that ReplicaFetcherThreadBenchmark (and other few benchmarks) is
not working now and reported as KAFKA-15567.

I submitted a patch for that in https://github.com/apache/kafka/pull/14513.

Could anyone review the PR?


Thanks,

--

Okada Haruki
ocadar...@gmail.com

Re: Apache Kafka 3.7.0 Release

[Chris Egerton]

Hi Stanislav,

Thanks for putting this together! I think the "Ensure that release
candidates include artifacts for the new Connect test-plugins module"
section (which I'm guessing was copied over from the 3.6.0 release plan?)
can be removed; we made sure that those artifacts were present for 3.6.0,
and I don't anticipate any changes that would make them likelier to be
accidentally dropped in subsequent releases than any other Maven artifacts
that we publish.

Also, can we add KIP-976 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-976%3A+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect)
to the release plan? The vote thread for it passed last week and I've
published a complete PR (https://github.com/apache/kafka/pull/14538), so it
shouldn't be too difficult to get things merged in time for 3.7.0.

Cheers,

Chris

On Sat, Oct 14, 2023 at 3:26 PM Stanislav Kozlovski
 wrote:

> Thanks for letting me drive it, folks.
>
> I've created the 3.7.0 release page here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0
> It outlines the key milestones and important dates for the release.
>
> In particular, since the last two releases slipped their originally
> targeted release date by taking an average of 46 days after code freeze (as
> opposed to the minimum which is 14 days), I pulled the dates forward to try
> and catch up with the original release schedule.
> You can refer to the last release during the Christmas holiday season -
> Apache
> Kafka 3.4
>  -
> to
> see sample dates.
>
> The currently proposed dates are:
>
> *KIP Freeze - 18th November *(Saturday)
> *This is 1 month and four days from now - rather short - but I'm afraid is
> the only lever that's easy to pull forward.*
> As usual, a KIP must be accepted by this date in order to be considered for
> this release. Note, any KIP that may not be implemented in a week, or that
> might destabilize the release, should be deferred.
>
> *Feature Freeze - 8th December* (Friday)
> *This follows 3 weeks after the KIP Freeze, as has been the case in our
> latest releases.*
> By this point, we want all major features to be merged & us to be working
> on stabilisation. Minor features should have PRs, the release branch should
> be cut; anything not in this state will be automatically moved to the next
> release in JIRA
>
> *Code Freeze - 20th December* (Wednesday)
>
> *Critically, this is before the holiday season and ends in the middle of
> the week, to give contributors more time and flexibility to address any
> last-minute without eating into the time people usually take holidays. It
> comes 12 days after the Feature Freeze.This is two days shorter than the
> usual code freeze window. I don't have a strong opinion and am open to
> extend it to Friday, or trade off a day/two with the KF<->FF date range.*
>
> *Release -* *after January 3rd*.
> *It comes after a minimum of two weeks of stabilization, so the earliest we
> can start releasing is January 3rd. We will move as fast as we can and aim
> completing it as early in January as possible.*
>
> As for the initially-populated KIPs in the release plan, I did the
> following:
>
> I kept 4 KIPs that were mentioned in 3.6, saying they would have minor
> parts finished in 3.7 (as the major ones went out in 3.6)
> - KIP-405 Tiered Storage mentioned a major part went out with 3.6 and the
> remainder will come with 3.7
> - KIP-890 mentioned Part 1 shipped in 3.6. I am assuming the remainder will
> come in 3.7, and have contacted the author to confirm.
> - KIP-926 was partially implemented in 3.6. I am assuming the remainder
> will come in 3.7, and have contacted the author to confirm.
> - KIP-938 mentioned that the majority was completed and a small remainder
> re: ForwardingManager metrics will come in 3.7. I have contacted the author
> to confirm.
>
> I then went through the JIRA filter which looks at open issues with a Fix
> Version of 3.7 and added KIP-770, KIP-858, and KIP-980.
> I also found a fair amount of JIRAs that were targeting the 3.7 release but
> consecutively had no activity on them for the past few releases. For most
> of those, I pinged the author and explicitly asked if it's going to aim to
> make it to 3.7. I have not included those here and will not until I hear
> confirmation.
>
> Please review the plan and provide any additional information or updates
> regarding KIPs that target this release version (3.7).
> If you have authored any KIPs that have an inaccurate status in the list,
> or are not in the list and should be, or are in the list and should not be
> - please inform me in this thread so that I can keep the document accurate
> and up to date.
>
> Excited to get this release going!
>
> All the best,
> Stanislav
>
> On Tue, Oct 10, 2023 at 9:12 AM Bruno Cadonna  wrote:
>
> > Thanks Stan!
> >
> > +1
> >
> > Best,
> > Bruno
> >
> > On 10/10/23 7:24 AM, Luke Chen wrote:
> > > Thanks Stanislav!
> 

[jira] [Resolved] (KAFKA-15249) Verify Connect test-plugins artifact is published to Maven Central

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-15249.
---
Resolution: Done

> Verify Connect test-plugins artifact is published to Maven Central
> --
>
> Key: KAFKA-15249
> URL: https://issues.apache.org/jira/browse/KAFKA-15249
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Affects Versions: 3.6.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Blocker
>
> In KAFKA-14759 we created a separate {{connect/test-plugins}} module to store 
> all testing-only Connect plugins and removed those plugins from existing 
> Connect modules.
> These testing-only plugins are intentionally excluded from the project's 
> release file (which can be generated with {{{}./gradlew releaseTarGz{}}}) 
> however, some users may still be relying on them for testing environments.
> Although we should refrain from distributing these testing-only plugins with 
> our out-of-the-box distribution of Connect, we should still ensure that 
> they're available on an opt-in basis to users who would like to continue 
> using them. This can be accomplished by publishing them to [Maven 
> Central|https://search.maven.org/], like we do with our other modules.
> This will probably happen automatically during the next release (3.6.0) with 
> no further action required. This ticket is just here as a reminder to verify 
> that the artifacts are present in the staging Maven repo when release 
> candidates are published for voting.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Stanislav Kozlovski]

Thanks for letting me drive it, folks.

I've created the 3.7.0 release page here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0
It outlines the key milestones and important dates for the release.

In particular, since the last two releases slipped their originally
targeted release date by taking an average of 46 days after code freeze (as
opposed to the minimum which is 14 days), I pulled the dates forward to try
and catch up with the original release schedule.
You can refer to the last release during the Christmas holiday season - Apache
Kafka 3.4
 - to
see sample dates.

The currently proposed dates are:

*KIP Freeze - 18th November *(Saturday)
*This is 1 month and four days from now - rather short - but I'm afraid is
the only lever that's easy to pull forward.*
As usual, a KIP must be accepted by this date in order to be considered for
this release. Note, any KIP that may not be implemented in a week, or that
might destabilize the release, should be deferred.

*Feature Freeze - 8th December* (Friday)
*This follows 3 weeks after the KIP Freeze, as has been the case in our
latest releases.*
By this point, we want all major features to be merged & us to be working
on stabilisation. Minor features should have PRs, the release branch should
be cut; anything not in this state will be automatically moved to the next
release in JIRA

*Code Freeze - 20th December* (Wednesday)

*Critically, this is before the holiday season and ends in the middle of
the week, to give contributors more time and flexibility to address any
last-minute without eating into the time people usually take holidays. It
comes 12 days after the Feature Freeze.This is two days shorter than the
usual code freeze window. I don't have a strong opinion and am open to
extend it to Friday, or trade off a day/two with the KF<->FF date range.*

*Release -* *after January 3rd*.
*It comes after a minimum of two weeks of stabilization, so the earliest we
can start releasing is January 3rd. We will move as fast as we can and aim
completing it as early in January as possible.*

As for the initially-populated KIPs in the release plan, I did the
following:

I kept 4 KIPs that were mentioned in 3.6, saying they would have minor
parts finished in 3.7 (as the major ones went out in 3.6)
- KIP-405 Tiered Storage mentioned a major part went out with 3.6 and the
remainder will come with 3.7
- KIP-890 mentioned Part 1 shipped in 3.6. I am assuming the remainder will
come in 3.7, and have contacted the author to confirm.
- KIP-926 was partially implemented in 3.6. I am assuming the remainder
will come in 3.7, and have contacted the author to confirm.
- KIP-938 mentioned that the majority was completed and a small remainder
re: ForwardingManager metrics will come in 3.7. I have contacted the author
to confirm.

I then went through the JIRA filter which looks at open issues with a Fix
Version of 3.7 and added KIP-770, KIP-858, and KIP-980.
I also found a fair amount of JIRAs that were targeting the 3.7 release but
consecutively had no activity on them for the past few releases. For most
of those, I pinged the author and explicitly asked if it's going to aim to
make it to 3.7. I have not included those here and will not until I hear
confirmation.

Please review the plan and provide any additional information or updates
regarding KIPs that target this release version (3.7).
If you have authored any KIPs that have an inaccurate status in the list,
or are not in the list and should be, or are in the list and should not be
- please inform me in this thread so that I can keep the document accurate
and up to date.

Excited to get this release going!

All the best,
Stanislav

On Tue, Oct 10, 2023 at 9:12 AM Bruno Cadonna  wrote:

> Thanks Stan!
>
> +1
>
> Best,
> Bruno
>
> On 10/10/23 7:24 AM, Luke Chen wrote:
> > Thanks Stanislav!
> >
> > On Tue, Oct 10, 2023 at 3:05 AM Josep Prat 
> > wrote:
> >
> >> Thanks Stanislav!
> >>
> >> ———
> >> Josep Prat
> >>
> >> Aiven Deutschland GmbH
> >>
> >> Alexanderufer 3-7, 10117 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> m: +491715557497
> >>
> >> w: aiven.io
> >>
> >> e: josep.p...@aiven.io
> >>
> >> On Mon, Oct 9, 2023, 20:05 Chris Egerton 
> wrote:
> >>
> >>> +1, thanks Stanislav!
> >>>
> >>> On Mon, Oct 9, 2023, 14:02 Bill Bejeck  wrote:
> >>>
>  +1
> 
>  Thanks, Stanislav!
> 
>  -Bill
> 
>  On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma  wrote:
> 
> > Thanks for volunteering Stanislav!
> >
> > Ismael
> >
> > On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski
> >  wrote:
> >
> >> Hey all!
> >>
> >> I would like to volunteer to be the release manager driving the
> >> next
> >> release - Apache Kafka *3.7.0*.
> >>
> >> If there are no objections, I will start and share a release 

[jira] [Resolved] (KAFKA-14175) KRaft Upgrades Part 2

[Stanislav Kozlovski (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stanislav Kozlovski resolved KAFKA-14175.
-
Resolution: Won't Fix

> KRaft Upgrades Part 2
> -
>
> Key: KAFKA-14175
> URL: https://issues.apache.org/jira/browse/KAFKA-14175
> Project: Kafka
>  Issue Type: New Feature
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
>
> This is the parent issue for KIP-778 tasks which were not completed for the 
> 3.3 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

[Philip Nee]

Hi Erik,

Thanks for the KIP, again.  I am also very much interested in the idea of
this KIP, and I want to let you know that we are rewriting the kafka
consumer using an event-driven approach, so I think the new impl would make
this KIP much easier to implement.  In a nutshell, the network IO will
become completely asynchronous to the application thread, so that the
blocking APIs won't stale the network send/receive.  In the new impl, the
main role of poll are 1. check if there are any background events such as
error or callback invocation, 2. notify the background that the user is
polling, and 3. check if there is any data to return to the user.
Because the background thread and application thread are inherently working
in an async fashion, it is possible to continue to process and commit
during the revocation period; however, we have to be very careful with the
state of partition ownership as you mentioned in the KIP.

Please keep an eye out on the new consumer implementation, in case if you
are interested in digging in, it is the PrototypeKafkaConsumer module.  It
is not fully functional but we are working full speed to get this to a good
state.

Thanks - I am still reading to KIP and your previous KIP to see if I can
make more constructive suggestions here.
P

On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten
 wrote:

> Hello David,
>
> Thanks, I am happy to hear we agree on the problem. All the tiny details
> of an implementation are less important.
>
> I will read KIP-848 first to answer you question about its relation with
> KIP-983. But for sure it makes sense to complete the implementation of
> KIP-848 first.
>
> Kind regards,
>  Erik.
>
>
> Op 13-10-2023 om 21:00 schreef David Jacot:
> > Hi Erik,
> >
> > Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the
> > weaknesses that you point out in it. I will continue to read it.
> >
> > For your information, we are working full speed on implementing KIP-848
> > while also changing the internal threading model of consumer. Those
> changes
> > are already extremely large so I would rather prefer to complete them
> > before adding more on top of them. Moreover, I think that this KIP should
> > build on top of KIP-848 now. Would you agree with this?
> >
> >
> > Best,
> > David
> >
> > Le ven. 13 oct. 2023 à 20:44, Erik van Oosten .invalid>
> > a écrit :
> >
> >> Thanks Philip,
> >>
> >> No worries, I am not in a hurry. Knowing this is not forgotten is enough
> >> for me. If there is anything I can do to help the process please let me
> >> know.
> >>
> >> Kind regards,
> >>   Erik.
> >>
> >>
> >> Op 13-10-2023 om 20:29 schreef Philip Nee:
> >>> Hi Erik,
> >>>
> >>> Sorry for the delay, I have not finished reviewing the KIP, but I also
> >> have
> >>> not forgotten about it!
> >>>
> >>> In general, KIP review process can be lengthy, so I think mailing list
> is
> >>> the best bet to get the committer's attention.
> >>>
> >>> P
> >>>
> >>> On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten
> >>>   wrote:
> >>>
>  Hi client developers,
> 
>  The text is updated so that it is more clear that you can only use
>  auto-commit when doing synchronous processing (approach 1). I am
>  assuming that auto-commit commits whatever was consumed in the
> previous
>  poll.
> 
>  I am wondering why this KIP doesn't get more attention. Is async
>  processing not something that the kafka client wants to support?
> 
>  Kind regards,
> Erik.
> 
> 
>  Op 25-09-2023 om 18:17 schreef Erik van Oosten:
> > Hi Viktor,
> >
> > Good questions!
> >
> > 1. Auto-commits would only work with approach 1 in the KIP. Any async
> > solution is incompatible with auto-commits. Do you think the text
> will
> > improve when this is mentioned?
> >
> > 2. That is entirely correct. If you use async commits you can await
> > completion by doing a single sync commit with an empty offsets Map
> > (this will work as of Kafka 3.6.0).
> >
> > Is there anything I can do to make the text clearer?
> >
> > Kind regards,
> >   Erik.
> >
> >
> > Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass:
> >> Hi Erik,
> >>
> >> I'm still trying to wrap my head around the KIP, however I have a
> few
> >> questions that weren't clear to me regarding offset commits:
> >> 1. Would auto-commits interfere with the behavior defined in your
> KIP
> >> or
> >> would it work the same as manual commits?
> >> 2. As I see you don't separate offset commits by whether they're
> sync
> >> or
> >> async. For sync commits timing isn't really a problem but how would
> >> you
> >> change work in case of async offset commits? There can be a few
> >> caveats
> >> there as you may not know whether a commit is finished or not until
> >> your
> >> callback is called.
> >>
> >> Thanks,
> >> Viktor
> >>
> >> On 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Guozhang Wang]

Thanks for the summary, that looks good to me.

Guozhang

On Fri, Oct 13, 2023 at 8:57 PM Colt McNealy  wrote:
>
> Hello there!
>
> Thanks everyone for the comments. There's a lot of back-and-forth going on,
> so I'll do my best to summarize what everyone's said in TLDR format:
>
> 1. Rename `onStandbyUpdateStart()` -> `onUpdateStart()`,  and do similarly
> for the other methods.
> 2. Keep `SuspendReason.PROMOTED` and `SuspendReason.MIGRATED`.
> 3. Remove the `earliestOffset` parameter for performance reasons.
>
> If that's all fine with everyone, I'll update the KIP and we—well, mostly
> Edu (:  —will open a PR.
>
> Cheers,
> Colt McNealy
>
> *Founder, LittleHorse.dev*
>
>
> On Fri, Oct 13, 2023 at 7:58 PM Eduwer Camacaro 
> wrote:
>
> > Hello everyone,
> >
> > Thanks for all your feedback for this KIP!
> >
> > I think that the key to choosing proper names for this API is understanding
> > the terms used inside the StoreChangelogReader. Currently, this class has
> > two possible states: ACTIVE_RESTORING and STANDBY_UPDATING. In my opinion,
> > using StandbyUpdateListener for the interface fits better on these terms.
> > Same applies for onUpdateStart/Suspended.
> >
> > StoreChangelogReader uses "the same mechanism" for active task restoration
> > and standby task updates, but this is an implementation detail. Under
> > normal circumstances (no rebalances or task migrations), the changelog
> > reader will be in STANDBY_UPDATING, which means it will be updating standby
> > tasks as long as there are new records in the changelog topic. That's why I
> > prefer onStandbyUpdated instead of onBatchUpdated, even if it doesn't 100%
> > align with StateRestoreListener, but either one is fine.
> >
> > Edu
> >
> > On Fri, Oct 13, 2023 at 8:53 PM Guozhang Wang 
> > wrote:
> >
> > > Hello Colt,
> > >
> > > Thanks for writing the KIP! I have read through the updated KIP and
> > > overall it looks great. I only have minor naming comments (well,
> > > aren't naming the least boring stuff to discuss and that takes the
> > > most of the time for KIPs :P):
> > >
> > > 1. I tend to agree with Sophie regarding whether or not to include
> > > "Standby" in the functions of "onStandbyUpdateStart/Suspended", since
> > > it is also more consistent with the functions of
> > > "StateRestoreListener" where we do not name it as
> > > "onStateRestoreState" etc.
> > >
> > > 2. I know in community discussions we sometimes say "a standby is
> > > promoted to active", but in the official code / java docs we did not
> > > have a term of "promotion", since what the code does is really recycle
> > > the task (while keeping its state stores open), and create a new
> > > active task that takes in the recycled state stores and just changing
> > > the other fields like task type etc. After thinking about this for a
> > > bit, I tend to feel that "promoted" is indeed a better name for user
> > > facing purposes while "recycle" is more of a technical detail inside
> > > the code and could be abstracted away from users. So I feel keeping
> > > the name "PROMOTED" is fine.
> > >
> > > 3. Regarding "earliestOffset", it does feel like we cannot always
> > > avoid another call to the Kafka API. And on the other hand, I also
> > > tend to think that such bookkeeping may be better done at the app
> > > level than from the Streams' public API level. I.e. the app could keep
> > > a "first ever starting offset" per "topic-partition-store" key, and a
> > > when we have rolling restart and hence some standby task keeps
> > > "jumping" from one client to another via task assignment, the app
> > > would update this value just one when it finds the
> > > ""topic-partition-store" was never triggered before. What do you
> > > think?
> > >
> > > 4. I do not have a strong opinion either, but what about
> > "onBatchUpdated" ?
> > >
> > >
> > > Guozhang
> > >
> > > On Wed, Oct 11, 2023 at 9:31 PM Colt McNealy 
> > wrote:
> > > >
> > > > Sohpie—
> > > >
> > > > Thank you very much for such a detailed review of the KIP. It might
> > > > actually be longer than the original KIP in the first place!
> > > >
> > > > 1. Ack'ed and fixed.
> > > >
> > > > 2. Correct, this is a confusing passage and requires context:
> > > >
> > > > One thing on our list of TODO's regarding reliability is to determine
> > how
> > > > to configure `session.timeout.ms`. In our Kubernetes Environment, an
> > > > instance of our Streams App can be terminated, restarted, and get back
> > > into
> > > > the "RUNNING" Streams state in about 20 seconds. We have two options
> > > here:
> > > > a) set session.timeout.ms to 30 seconds or so, and deal with 20
> > seconds
> > > of
> > > > unavailability for affected partitions, but avoid shuffling Tasks; or
> > b)
> > > > set session.timeout.ms to a low value, such as 6 seconds (
> > > > heartbeat.interval.ms of 2000), and reduce the unavailability window
> > > during
> > > > a rolling bounce but incur an "extra" rebalance. There are several
> > > > 

[jira] [Created] (KAFKA-15607) Possible NPE is thrown in MirrorCheckpointTask

[hudeqi (Jira)]

hudeqi created KAFKA-15607:
--

 Summary: Possible NPE is thrown in MirrorCheckpointTask
 Key: KAFKA-15607
 URL: https://issues.apache.org/jira/browse/KAFKA-15607
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.8.1
Reporter: hudeqi
Assignee: hudeqi


In the `syncGroupOffset` method, if `targetConsumerOffset.get(topicPartition)` 
gets null, then the calculation of `latestDownstreamOffset` will throw NPE. 
This usually occurs in this situation: a group consumed a topic in the target 
cluster previously. Later, the group offset of some partitions was reset to -1, 
the `OffsetAndMetadata` of these partitions was null.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2288

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

[Erik van Oosten]

Hello David,

Thanks, I am happy to hear we agree on the problem. All the tiny details 
of an implementation are less important.


I will read KIP-848 first to answer you question about its relation with 
KIP-983. But for sure it makes sense to complete the implementation of 
KIP-848 first.


Kind regards,
    Erik.


Op 13-10-2023 om 21:00 schreef David Jacot:

Hi Erik,

Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the
weaknesses that you point out in it. I will continue to read it.

For your information, we are working full speed on implementing KIP-848
while also changing the internal threading model of consumer. Those changes
are already extremely large so I would rather prefer to complete them
before adding more on top of them. Moreover, I think that this KIP should
build on top of KIP-848 now. Would you agree with this?


Best,
David

Le ven. 13 oct. 2023 à 20:44, Erik van Oosten
a écrit :


Thanks Philip,

No worries, I am not in a hurry. Knowing this is not forgotten is enough
for me. If there is anything I can do to help the process please let me
know.

Kind regards,
  Erik.


Op 13-10-2023 om 20:29 schreef Philip Nee:

Hi Erik,

Sorry for the delay, I have not finished reviewing the KIP, but I also

have

not forgotten about it!

In general, KIP review process can be lengthy, so I think mailing list is
the best bet to get the committer's attention.

P

On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten
  wrote:


Hi client developers,

The text is updated so that it is more clear that you can only use
auto-commit when doing synchronous processing (approach 1). I am
assuming that auto-commit commits whatever was consumed in the previous
poll.

I am wondering why this KIP doesn't get more attention. Is async
processing not something that the kafka client wants to support?

Kind regards,
   Erik.


Op 25-09-2023 om 18:17 schreef Erik van Oosten:

Hi Viktor,

Good questions!

1. Auto-commits would only work with approach 1 in the KIP. Any async
solution is incompatible with auto-commits. Do you think the text will
improve when this is mentioned?

2. That is entirely correct. If you use async commits you can await
completion by doing a single sync commit with an empty offsets Map
(this will work as of Kafka 3.6.0).

Is there anything I can do to make the text clearer?

Kind regards,
  Erik.


Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass:

Hi Erik,

I'm still trying to wrap my head around the KIP, however I have a few
questions that weren't clear to me regarding offset commits:
1. Would auto-commits interfere with the behavior defined in your KIP

or

would it work the same as manual commits?
2. As I see you don't separate offset commits by whether they're sync

or

async. For sync commits timing isn't really a problem but how would

you

change work in case of async offset commits? There can be a few

caveats

there as you may not know whether a commit is finished or not until

your

callback is called.

Thanks,
Viktor

On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten
  wrote:


Hi all,

I would like to start the discussion on KIP-983: Full speed async
processing during rebalance [1].

The idea is that we can prevent the drop in throughput during a
cooperative rebalance.

I am curious to your ideas and comments.

Kind regards,
Erik.

[1]



https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com