Re: [ANNOUNCE] New committer: Igor Soarez

[Christo Lolov]

Congratulations Igor :) !

On Thu, 25 Apr 2024 at 17:07, Igor Soarez  wrote:

> Thanks everyone, I'm very honoured to join!
>
> --
> Igor
>

Re: Confluence edit access

[Claude Warren]

My Confluence ID is "claude"

On Thu, Apr 25, 2024 at 8:40 PM Matthias J. Sax  wrote:

> What's your wiki ID? We can grant write access on our side if you have
> already an account.
>
> -Matthias
>
> On 4/25/24 4:06 AM, Claude Warren wrote:
> > I would like to get edit access to the Kafka confluence so that I can
> work
> > on KIP-936.  Can someone here do that or do I need to go through Infra?
> >
> > Claude
> >
>


-- 
LinkedIn: http://www.linkedin.com/in/claudewarren

[VOTE] KIP-1023: Follower fetch from tiered offset

[Abhijeet Kumar]

Hi All,

I would like to start the vote for KIP-1023 - Follower fetch from tiered
offset

The KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+tiered+offset

Regards.
Abhijeet.

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

[Abhijeet Kumar]

Thank you all for your comments. As all the comments in the thread are
addressed, I am starting a Vote thread for the KIP. Please have a look.

Regards.
Abhijeet.



On Thu, Apr 25, 2024 at 6:08 PM Luke Chen  wrote:

> Hi, Abhijeet,
>
> Thanks for the update.
>
> I have no more comments.
>
> Luke
>
> On Thu, Apr 25, 2024 at 4:21 AM Jun Rao  wrote:
>
> > Hi, Abhijeet,
> >
> > Thanks for the updated KIP. It looks good to me.
> >
> > Jun
> >
> > On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar <
> > abhijeet.cse@gmail.com>
> > wrote:
> >
> > > Hi Jun,
> > >
> > > Please find my comments inline.
> > >
> > >
> > > On Thu, Apr 18, 2024 at 3:26 AM Jun Rao 
> > wrote:
> > >
> > > > Hi, Abhijeet,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 1. I am wondering if we could achieve the same result by just
> lowering
> > > > local.retention.ms and local.retention.bytes. This also allows the
> > newly
> > > > started follower to build up the local data before serving the
> consumer
> > > > traffic.
> > > >
> > >
> > > I am not sure I fully followed this. Do you mean we could lower the
> > > local.retention (by size and time)
> > > so that there is little data on the leader's local storage so that the
> > > follower can quickly catch up with the leader?
> > >
> > > In that case, we will need to set small local retention across brokers
> in
> > > the cluster. It will have the undesired
> > > effect where there will be increased remote log fetches for serving
> > consume
> > > requests, and this can cause
> > > degradations. Also, this behaviour (of increased remote fetches) will
> > > happen on all brokers at all times, whereas in
> > > the KIP we are restricting the behavior only to the newly bootstrapped
> > > brokers and only until the time it fully builds
> > > the local logs as per retention defined at the cluster level.
> > > (Deprioritization of the broker could help reduce the impact
> > >  even further)
> > >
> > >
> > > >
> > > > 2. Have you updated the KIP?
> > > >
> > >
> > > The KIP has been updated now.
> > >
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Apr 9, 2024 at 3:36 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 to Jun for adding the consumer fetching from a follower scenario
> > > > > also to the existing section that talked about the drawback when a
> > > > > node built with last-tiered-offset has become a leader. As Abhijeet
> > > > > mentioned, we plan to have a follow-up KIP that will address these
> by
> > > > > having a deprioritzation of these brokers. The deprioritization of
> > > > > those brokers can be removed once they catchup until the local log
> > > > > retention.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Tue, 9 Apr 2024 at 14:08, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Abhijeet,
> > > > > >
> > > > > > Thanks for the KIP to improve the tiered storage feature!
> > > > > >
> > > > > > Questions:
> > > > > > 1. We could also get the "pending-upload-offset" and epoch via
> > remote
> > > > log
> > > > > > metadata, instead of adding a new API to fetch from the leader.
> > Could
> > > > you
> > > > > > explain why you choose the later approach, instead of the former?
> > > > > > 2.
> > > > > > > We plan to have a follow-up KIP that will address both the
> > > > > > deprioritization
> > > > > > of these brokers from leadership, as well as
> > > > > > for consumption (when fetching from followers is allowed).
> > > > > >
> > > > > > I agree with Jun that we might need to make it clear all possible
> > > > > drawbacks
> > > > > > that could have. So, could we add the drawbacks that Jun
> mentioned
> > > > about
> > > > > > the performance issue when consumer fetch from follower?
> > > > > >
> > > > > > 3. Could we add "Rejected Alternatives" section to the end of the
> > KIP
> > > > to
> > > > > > add some of them?
> > > > > > Like the "ListOffsetRequest" approach VS
> > > > "Earliest-Pending-Upload-Offset"
> > > > > > approach, or getting the "Earliest-Pending-Upload-Offset" from
> > remote
> > > > log
> > > > > > metadata... etc.
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 9, 2024 at 2:25 PM Abhijeet Kumar <
> > > > > abhijeet.cse@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Christo,
> > > > > > >
> > > > > > > Please find my comments inline.
> > > > > > >
> > > > > > > On Fri, Apr 5, 2024 at 12:36 PM Christo Lolov <
> > > > christolo...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Abhijeet and Jun,
> > > > > > > >
> > > > > > > > I have been mulling this KIP over a bit more in recent days!
> > > > > > > >
> > > > > > > > re: Jun
> > > > > > > >
> > > > > > > > I wasn't aware we apply 2.1 and 2.2 for reserving new
> > timestamps
> > > -
> > > > in
> > > > > > > > retrospect it should have been fairly obvious. I would need
> to
> > go
> > > > an
> > > > > > > update
> > > > > > > > 

Re: [DISCUSS] KIP-932: Queues for Kafka

[Jun Rao]

Hi, Andrew,

Thanks for the reply.

123. "Rather than add the group epoch to the ShareGroupHeartbeat, I have
decided to go for TopicPartitions in ShareGroupHeartbeatRequest which
mirrors ConsumerGroupHeartbeatRequest."
ShareGroupHeartbeat.MemberEpoch is the group epoch, right? Is that enough
for confirming the receipt of the new assignment?

125. This also means that "Alter share group offsets" needs to write a
ShareGroupPartitionMetadata record, if the partition is not already
initialized.

140. In the table for "Delete share group offsets", we need to add a step
to write a ShareGroupPartitionMetadata record with DeletingTopics.

141. Hmm, ShareGroupMemberMetadata is stored in the __consumer_offsets
topic, which is a compacted topic, right?

143. So, the client sends DescribeShareGroupOffsets requests to GC, which
then forwards it to SC?

147. I guess a client only knows the rebalance triggered by itself, but not
the ones triggered by other members or topic/partition changes?

Jun

On Thu, Apr 25, 2024 at 4:19 AM Andrew Schofield 
wrote:

> Hi Jun,
> Thanks for the response.
>
> 123. Of course, ShareGroupHearbeat started off as ConsumerGroupHeartbeat
> and then unnecessary fields were removed. In the network issue case,
> there is not currently enough state being exchanged to be sure an
> assignment
> was received.
>
> Rather than add the group epoch to the ShareGroupHeartbeat, I have decided
> to go for TopicPartitions in ShareGroupHeartbeatRequest which mirrors
> ConsumerGroupHeartbeatRequest. It means the share group member does
> confirm the assignment it is using, and that can be used by the GC to
> safely
> stop repeating the assignment in heartbeat responses.
>
> 125. Ah, yes. This is indeed something possible with a consumer group
> and share groups should support it too. This does of course imply that
> ShareGroupPartitionMetadataValue needs an array of partitions, not
> just the number.
>
> 140. Yes, good spot. There is an inconsistency here in consumer groups
> where you can use AdminClient.deleteConsumerGroupOffsets at the
> partition level, but kafka-consumer-groups.sh --delete only operates
> at the topic level.
>
> Personally, I don’t think it’s sensible to delete offsets at the partition
> level only. You can reset them, but if you’re actively using a topic with
> a share group, I don’t see why you’d want to delete offsets rather than
> reset. If you’ve finished using a topic with a share group and want to
> clean
> up, use delete.
>
> So, I’ve changed the AdminClient.deleteConsumerGroupOffsets to be
> topic-based and the RPCs behind it.
>
> The GC reconciles the cluster state with the ShareGroupPartitionMetadata
> to spot deletion of topics and the like. However, when the offsets for
> a topic were deleted manually, the topic very like still exists so
> reconciliation
> alone is not going to be able to continue an interrupted operation that
> has started. So, I’ve added DeletingTopics back into
> ShareGroupPartitionMetadata for this purpose. It’s so failover of a GC
> can continue where it left off rather than leaving fragments across the
> SCs.
>
> 141. That is not required. Because this is not a compacted topic, it is
> not necessary to write tombstones for every key. As long as there is a
> clear and unambiguous record for the deletion of the group, that is enough.
> The tombstone for ShareGroupPartitionMetadata is theoretically not
> required but it’s a single record, rather than one per member, so I prefer
> to leave it as a record that the interactions with the SC have been
> completed.
>
> 142.
> 142.1. It will prompt the user to confirm they want to continue.
> This is in common with `kafka-consumer-groups.sh` which historically
> has defaulted to --dry-run behaviour, but is due to change to prompting
> if neither --dry-run nor --execute is specified “in a future major
> release”.
>
> 142.2. It should support partition-level reset, but only topic-level
> delete.
> I have updated the usage text accordingly. This is in common with
> kafka-consumer-groups.sh.
>
> 142.3. --dry-run displays the operation that would be executed.
>
> 142.4. The valid values are: Dead, Empty, Stable. Added to the
> usage text.
>
> 143. DescribeShareGroupOffsets is served by the group coordinator
> for this kind of reason.
>
> 144. That’s the default. If you haven’t asked to release or reject, it
> accepts.
> This is analogous to fetching and committing offsets in a consumer group.
>
> 145. I think this is a good idea, but I would prefer to defer it until a
> future
> metrics KIP that I have planned. In KIP-932, I have added basic metrics
> only.
> For example, you’ll see that there’s no concept of lag yet, which surely
> will have relevance for share groups. I plan to create and deliver the
> metrics KIP before share groups are declared ready for production.
> I want the new metrics to be developed with the experience of running
> the code.
>
> 146. Milliseconds. KIP updated.
>
> 147. There is a 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2847

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

[Matthias J. Sax]

Thanks for the details!


107: I like `numProcessingThreads()` proposal.


108(a): it was not really a concern, but it was rather a question if we 
could/should simplify it, so it's easier to implement a custom task 
assignor. But if we believe that it's an integral component, I am fine 
with leaving the responsibility to request follow-up rebalances to the user.


108(b): Similar to 108(a). But I am ok to give control to the user too.


115: SGTM.



-Matthias

On 4/25/24 3:45 PM, Sophie Blee-Goldman wrote:

104. Fair enough -- also happy to defer to Rohan on this (or Bruno if he
feels super strongly)

107. That's a good point . Ultimately the task load should reflect the
processing capacity, and that's something that will exist in both the new
and old threading model. I like #processingCapacity for the name. And for
the javadocs, I think we can just say something generic enough that it will
be accurate in both the old and new model and won't need to be updated
(since, as you mentioned before, we always forget to update random
javadocs).

As for the NOTE, however, I do feel that is necessary. At the very least,
it's necessary if we rename the method to something with the word
"capacity", since that tends to imply a "maximum limit" rather than a
"minimum load". In fact I think the original method was named capacity and
that's why I added this NOTE to the javadoc in the first place, to avoid
confusion. I can see why it doesn't make sense if the name doesn't have the
word "capacity" in it though.

But I do think #processingCapacity feels appropriate. I guess a reasonable
alternative would just be #numProcessingThreads, which is both descriptive
enough to not need the NOTE in the javadocs and also accurately describes
both the old one-consumer-per-StreamThread model and the new one (it's just
a matter of what "processing thread" means will change). So how about we
just call it #numProcessingThreads and in the javadocs we can say it's like
a capacity that should correspond to the relative "weight" of assigned
tasks relative to other KafkaStreams clients? WDYT?

108. Yes, I 100% believe that a custom assignor should be able to schedule
a followup rebalance:

108(1) For one thing, imo the HATaskAssignor shouldn't be a "special case"
but should effectively look just like any other custom assignor. And it
definitely needs the ability to schedule followups. How would the
StreamsPartitionAssignor figure out whether to schedule a followup
rebalance for the HAAssignor? From the StreamsPArtitionAssignor POV it just
sees "active" and "standby" tasks -- It doesn't know whether some of those
standby tasks are actually "warmup tasks". And anyways, I think users
should be able to request a followup rebalance. What is the concern over
giving custom assignors control over this? Imo if people abuse this and
shoot themselves in the foot by scheduling followup rebalances for no
reason, that's on them. As always -- "simple things should be easy,
difficult things should be possible"

108(2) This is a fair question, although I still believe we should give the
custom assignor full control over the scheduled followup rebalance
timeline. Again, because more advanced things should be possible -- and
it's not like having this API return an Instant makes things more
complicated for people who want to do simple things, users are free to
ignore this completely and returning an Instant doesn't feel more difficult
than returning an enum. So why restrict this?

To take a more specific example: let's say users have a complicated set of
metrics they use to determine task placement. Sometimes these metrics are
unavailable, in which case they want to schedule a followup rebalance but
may want to implement a backoff/retry rather than simply scheduling an
immediate followup. I know for a fact that immediate followup rebalances
triggered by Kafka Streams can actually cause issues in some cases (see
KAFKA-14382  and
KAFKA-14419  -- same
root cause but note that the second issue is still unresolved to this day,
and I know someone besides the issue reporter who has repeatedly been
affected by it). Giving users the ability to back off and schedule smarter
"immediate" followups if/when they run into issues seems like a sufficient
motivation to me. For another: perhaps these complex metrics are advanced
enough to be able to predict when a given task will be "warmed up" (to take
the HAAssignor example) or otherwise be able to compute an exact time at
which to cut over to a new task assignment. In this case it would be
necessary to have full flexibility over when the followup was triggered.

115. I see what you mean here. I was originally thinking that way, but was
worried that users might "accidentally" catch whatever exception we throw
if the task lag computation fails and not know how to handle it. But I
suppose we can just say in the javadocs that you 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2846

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 460306 lines...]
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldClearTaskTimeoutOnProcessed() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldClearTaskTimeoutOnProcessed() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldUnassignTaskWhenRequired() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldUnassignTaskWhenRequired() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldClearTaskReleaseFutureOnShutdown() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldClearTaskReleaseFutureOnShutdown() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldProcessTasks() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldProcessTasks() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldPunctuateStreamTime() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldPunctuateStreamTime() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldShutdownTaskExecutor() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldShutdownTaskExecutor() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() 
STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() 
PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldPunctuateSystemTime() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldPunctuateSystemTime() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldNotFlushOnException() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > shouldNotFlushOnException() PASSED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() STARTED
[2024-04-25T20:32:08.431Z] 
[2024-04-25T20:32:08.431Z] Gradle Test Run :streams:test > Gradle Test Executor 
94 > DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() PASSED
[2024-04-25T20:32:08.431Z] 

[jira] [Created] (KAFKA-16626) Uuid to String for subscribed topic names in assignment spec

[Ritika Reddy (Jira)]

Ritika Reddy created KAFKA-16626:


 Summary: Uuid to String for subscribed topic names in assignment 
spec
 Key: KAFKA-16626
 URL: https://issues.apache.org/jira/browse/KAFKA-16626
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ritika Reddy
Assignee: Ritika Reddy


In creating the assignment spec from the existing consumer subscription 
metadata, quite some time is spent in converting the String to a Uuid

Change from Uuid to String for the subscribed topics in assignment spec and 
convert on the fly



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

[Sophie Blee-Goldman]

104. Fair enough -- also happy to defer to Rohan on this (or Bruno if he
feels super strongly)

107. That's a good point . Ultimately the task load should reflect the
processing capacity, and that's something that will exist in both the new
and old threading model. I like #processingCapacity for the name. And for
the javadocs, I think we can just say something generic enough that it will
be accurate in both the old and new model and won't need to be updated
(since, as you mentioned before, we always forget to update random
javadocs).

As for the NOTE, however, I do feel that is necessary. At the very least,
it's necessary if we rename the method to something with the word
"capacity", since that tends to imply a "maximum limit" rather than a
"minimum load". In fact I think the original method was named capacity and
that's why I added this NOTE to the javadoc in the first place, to avoid
confusion. I can see why it doesn't make sense if the name doesn't have the
word "capacity" in it though.

But I do think #processingCapacity feels appropriate. I guess a reasonable
alternative would just be #numProcessingThreads, which is both descriptive
enough to not need the NOTE in the javadocs and also accurately describes
both the old one-consumer-per-StreamThread model and the new one (it's just
a matter of what "processing thread" means will change). So how about we
just call it #numProcessingThreads and in the javadocs we can say it's like
a capacity that should correspond to the relative "weight" of assigned
tasks relative to other KafkaStreams clients? WDYT?

108. Yes, I 100% believe that a custom assignor should be able to schedule
a followup rebalance:

108(1) For one thing, imo the HATaskAssignor shouldn't be a "special case"
but should effectively look just like any other custom assignor. And it
definitely needs the ability to schedule followups. How would the
StreamsPartitionAssignor figure out whether to schedule a followup
rebalance for the HAAssignor? From the StreamsPArtitionAssignor POV it just
sees "active" and "standby" tasks -- It doesn't know whether some of those
standby tasks are actually "warmup tasks". And anyways, I think users
should be able to request a followup rebalance. What is the concern over
giving custom assignors control over this? Imo if people abuse this and
shoot themselves in the foot by scheduling followup rebalances for no
reason, that's on them. As always -- "simple things should be easy,
difficult things should be possible"

108(2) This is a fair question, although I still believe we should give the
custom assignor full control over the scheduled followup rebalance
timeline. Again, because more advanced things should be possible -- and
it's not like having this API return an Instant makes things more
complicated for people who want to do simple things, users are free to
ignore this completely and returning an Instant doesn't feel more difficult
than returning an enum. So why restrict this?

To take a more specific example: let's say users have a complicated set of
metrics they use to determine task placement. Sometimes these metrics are
unavailable, in which case they want to schedule a followup rebalance but
may want to implement a backoff/retry rather than simply scheduling an
immediate followup. I know for a fact that immediate followup rebalances
triggered by Kafka Streams can actually cause issues in some cases (see
KAFKA-14382  and
KAFKA-14419  -- same
root cause but note that the second issue is still unresolved to this day,
and I know someone besides the issue reporter who has repeatedly been
affected by it). Giving users the ability to back off and schedule smarter
"immediate" followups if/when they run into issues seems like a sufficient
motivation to me. For another: perhaps these complex metrics are advanced
enough to be able to predict when a given task will be "warmed up" (to take
the HAAssignor example) or otherwise be able to compute an exact time at
which to cut over to a new task assignment. In this case it would be
necessary to have full flexibility over when the followup was triggered.

115. I see what you mean here. I was originally thinking that way, but was
worried that users might "accidentally" catch whatever exception we throw
if the task lag computation fails and not know how to handle it. But I
suppose we can just say in the javadocs that you can/should not catch it
and/or rethrow to allow Streams to recover and re-attempt. I agree we
should have Streams just handle this transparently for users and not
require them to rebuild the assignment on their own. I'll add this to the
javadocs -- and I don't think we need to introduce a new exception type
even. We have the "TaskAssignmentException" already which behaves similarly
now -- ie if there's an error during assignment, we throw this and return
the same assignment back and schedule a followup.

So 

[jira] [Created] (KAFKA-16625) Reverse Lookup Partition to Member in Assignors

[Ritika Reddy (Jira)]

Ritika Reddy created KAFKA-16625:


 Summary: Reverse Lookup Partition to Member in Assignors
 Key: KAFKA-16625
 URL: https://issues.apache.org/jira/browse/KAFKA-16625
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ritika Reddy
Assignee: Ritika Reddy


Calculating unassigned partitions within the Uniform assignor is a costly 
process, this can be improved by using a reverse lookup map between 
topicIdPartition and the member



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Kirk True]

Hi Alieh,

Thanks for the updates!

Comments inline...


> On Apr 25, 2024, at 1:10 PM, Alieh Saeedi  
> wrote:
> 
> Hi all,
> 
> Thanks a lot for the constructive feedbacks!
> 
> 
> 
> Addressing some of the main concerns:
> 
> 
> - The `RecordTooLargeException` can be thrown by broker, producer and
> consumer. Of course, the `ProducerExceptionHandler` interface is introduced
> to affect only the exceptions thrown from the producer. This KIP very
> specifically means to provide a possibility to manage the
> `RecordTooLargeException` thrown from the Producer.send() method. Please
> see “Proposed Changes” section for more clarity. I investigated the issue
> there thoroughly. I hope it can explain the concern about how we handle the
> errors as well.
> 
> 
> 
> - The problem with Callback: Methods of Callback are called when the record
> sent to the server is acknowledged, while this is not the desired time for
> all exceptions. We intend to handle exceptions beforehand.

I guess it makes sense to keep the expectation for when Callback is invoked 
as-is vs. shoehorning more into it.

> - What if the custom handler returns RETRY for `RecordTooLargeException`? I
> assume changing the producer configuration at runtime is possible. If so,
> RETRY for a too large record is valid because maybe in the next try, the
> too large record is not poisoning any more. I am not 100% sure about the
> technical details, though. Otherwise, we can consider the RETRY as FAIL for
> this exception. Another solution would be to consider a constant number of
> times for RETRY which can be useful for other exceptions as well.

It’s not presently possible to change the configuration of an existing Producer 
at runtime. So if a record hits a RecordTooLargeException once, no amount of 
retrying (with the current Producer) will change that fact. So I’m still a 
little stuck on how to handle a response of RETRY for an “oversized” record. 

> - What if the handle() method itself throws an exception? I think
> rationally and pragmatically, the behaviour must be exactly like when no
> custom handler is defined since the user actually did not have a working
> handler.

I’m not convinced that ignoring an errant handler is the right choice. It then 
becomes a silent failure that might have repercussions, depending on the 
business logic. A user would have to proactively trawls through the logs for 
WARN/ERROR messages to catch it.

Throwing a hard error is pretty draconian, though…

> - Why not use config parameters instead of an interface? As explained in
> the “Rejected Alternatives” section, we assume that the handler will be
> used for a greater number of exceptions in the future. Defining a
> configuration parameter for each exception may make the configuration a bit
> messy. Moreover, the handler offers more flexibility.

Agreed that the logic-via-configuration approach is weird and limiting. Forget 
I ever suggested it ;)

I’d think additional background in the Motivation section would help me 
understand how users might use this feature beyond a) skipping “oversized” 
records, and b) not retrying missing topics. 

> Small change:
> 
> -ProductionExceptionHandlerResponse -> Response for brevity and simplicity.
> Could’ve been HandlerResponse too I think!

The name change sounds good to me.

Thanks Alieh!

> 
> 
> I thank you all again for your useful questions/suggestions.
> 
> I would be happy to hear more of your concerns, as stated in some feedback.
> 
> Cheers,
> Alieh
> 
> On Wed, Apr 24, 2024 at 12:31 AM Justine Olshan
>  wrote:
> 
>> Thanks Alieh for the updates.
>> 
>> I'm a little concerned about the design pattern here. It seems like we want
>> specific usages, but we are packaging it as a generic handler.
>> I think we tried to narrow down on the specific errors we want to handle,
>> but it feels a little clunky as we have a generic thing for two specific
>> errors.
>> 
>> I'm wondering if we are using the right patterns to solve these problems. I
>> agree though that we will need something more than the error classes I'm
>> proposing if we want to have different handling be configurable.
>> My concern is that the open-endedness of a handler means that we are
>> creating more problems than we are solving. It is still unclear to me how
>> we expect to handle the errors. Perhaps we could include an example? It
>> seems like there is a specific use case in mind and maybe we can make a
>> design that is tighter and supports that case.
>> 
>> Justine
>> 
>> On Tue, Apr 23, 2024 at 3:06 PM Kirk True  wrote:
>> 
>>> Hi Alieh,
>>> 
>>> Thanks for the KIP!
>>> 
>>> A few questions:
>>> 
>>> K1. What is the expected behavior for the producer if it generates a
>>> RecordTooLargeException, but the handler returns RETRY?
>>> K2. How do we determine which Record was responsible for the
>>> UnknownTopicOrPartitionException since we get that response when
>> sending  a
>>> batch of records?
>>> K3. What is the expected behavior if the 

[jira] [Created] (KAFKA-16624) Don't generate useless PartitionChangeRecord on older MV

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-16624:


 Summary: Don't generate useless PartitionChangeRecord on older MV
 Key: KAFKA-16624
 URL: https://issues.apache.org/jira/browse/KAFKA-16624
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe
Assignee: Colin McCabe


Fix a case where we could generate useless PartitionChangeRecords on metadata 
versions older than 3.6-IV0. This could happen in the case where we had an ISR 
with only one broker in it, and we were trying to go down to a fully empty ISR. 
In this case, PartitionChangeBuilder would block the record to going down to a 
fully empty ISR (since that is not valid in these pre-KIP-966 metadata 
versions), but it would still emit the record, even though it had no effect.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Alieh Saeedi]

Hi all,

Thanks a lot for the constructive feedbacks!



Addressing some of the main concerns:


- The `RecordTooLargeException` can be thrown by broker, producer and
consumer. Of course, the `ProducerExceptionHandler` interface is introduced
to affect only the exceptions thrown from the producer. This KIP very
specifically means to provide a possibility to manage the
`RecordTooLargeException` thrown from the Producer.send() method. Please
see “Proposed Changes” section for more clarity. I investigated the issue
there thoroughly. I hope it can explain the concern about how we handle the
errors as well.



- The problem with Callback: Methods of Callback are called when the record
sent to the server is acknowledged, while this is not the desired time for
all exceptions. We intend to handle exceptions beforehand.


- What if the custom handler returns RETRY for `RecordTooLargeException`? I
assume changing the producer configuration at runtime is possible. If so,
RETRY for a too large record is valid because maybe in the next try, the
too large record is not poisoning any more. I am not 100% sure about the
technical details, though. Otherwise, we can consider the RETRY as FAIL for
this exception. Another solution would be to consider a constant number of
times for RETRY which can be useful for other exceptions as well.


- What if the handle() method itself throws an exception? I think
rationally and pragmatically, the behaviour must be exactly like when no
custom handler is defined since the user actually did not have a working
handler.


- Why not use config parameters instead of an interface? As explained in
the “Rejected Alternatives” section, we assume that the handler will be
used for a greater number of exceptions in the future. Defining a
configuration parameter for each exception may make the configuration a bit
messy. Moreover, the handler offers more flexibility.



Small change:

-ProductionExceptionHandlerResponse -> Response for brevity and simplicity.
Could’ve been HandlerResponse too I think!


I thank you all again for your useful questions/suggestions.

I would be happy to hear more of your concerns, as stated in some feedback.

Cheers,
Alieh

On Wed, Apr 24, 2024 at 12:31 AM Justine Olshan
 wrote:

> Thanks Alieh for the updates.
>
> I'm a little concerned about the design pattern here. It seems like we want
> specific usages, but we are packaging it as a generic handler.
> I think we tried to narrow down on the specific errors we want to handle,
> but it feels a little clunky as we have a generic thing for two specific
> errors.
>
> I'm wondering if we are using the right patterns to solve these problems. I
> agree though that we will need something more than the error classes I'm
> proposing if we want to have different handling be configurable.
> My concern is that the open-endedness of a handler means that we are
> creating more problems than we are solving. It is still unclear to me how
> we expect to handle the errors. Perhaps we could include an example? It
> seems like there is a specific use case in mind and maybe we can make a
> design that is tighter and supports that case.
>
> Justine
>
> On Tue, Apr 23, 2024 at 3:06 PM Kirk True  wrote:
>
> > Hi Alieh,
> >
> > Thanks for the KIP!
> >
> > A few questions:
> >
> > K1. What is the expected behavior for the producer if it generates a
> > RecordTooLargeException, but the handler returns RETRY?
> > K2. How do we determine which Record was responsible for the
> > UnknownTopicOrPartitionException since we get that response when
> sending  a
> > batch of records?
> > K3. What is the expected behavior if the handle() method itself throws an
> > error?
> > K4. What is the downside of adding an onError() method to the Producer’s
> > Callback interface vs. a new mechanism?
> > K5. Can we change “ProducerExceptionHandlerResponse" to just “Response”
> > given that it’s an inner enum?
> > K6. Any recommendation for callback authors to handle different behavior
> > for different topics?
> >
> > I’ll echo what others have said, it would help me understand why we want
> > another handler class if there were more examples in the Motivation
> > section. As it stands now, I agree with Chris that the stated issues
> could
> > be solved by adding two new configuration options:
> >
> > oversized.record.behavior=fail
> > retry.on.unknown.topic.or.partition=true
> >
> > What I’m not yet able to wrap my head around is: what exactly would the
> > logic in the handler be? I’m not very imaginative, so I’m assuming they’d
> > mostly be if-this-then-that. However, if they’re more complicated, I’d
> have
> > other concerns.
> >
> > Thanks,
> > Kirk
> >
> > > On Apr 22, 2024, at 7:38 AM, Alieh Saeedi  >
> > wrote:
> > >
> > > Thank you all for the feedback!
> > >
> > > Addressing the main concern: The KIP is about giving the user the
> ability
> > > to handle producer exceptions, but to be more conservative and avoid
> > future
> > > issues, we decided 

Re: Confluence edit access

[Matthias J. Sax]

What's your wiki ID? We can grant write access on our side if you have 
already an account.


-Matthias

On 4/25/24 4:06 AM, Claude Warren wrote:

I would like to get edit access to the Kafka confluence so that I can work
on KIP-936.  Can someone here do that or do I need to go through Infra?

Claude

[jira] [Created] (KAFKA-16623) KafkaAsyncConsumer system tests warn about revoking partitions that weren't previously assigned

[Kirk True (Jira)]

Kirk True created KAFKA-16623:
-

 Summary: KafkaAsyncConsumer system tests warn about revoking 
partitions that weren't previously assigned
 Key: KAFKA-16623
 URL: https://issues.apache.org/jira/browse/KAFKA-16623
 Project: Kafka
  Issue Type: Bug
  Components: clients, consumer, system tests
Affects Versions: 3.8.0
Reporter: Kirk True
Assignee: Kirk True
 Fix For: 3.8.0


When running system tests for the KafkaAsyncConsumer, we occasionally see this 
warning:

{noformat}
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
  File 
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py",
 line 38, in _protected_worker
self._worker(idx, node)
  File 
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/verifiable_consumer.py",
 line 304, in _worker
handler.handle_partitions_revoked(event, node, self.logger)
  File 
"/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/verifiable_consumer.py",
 line 163, in handle_partitions_revoked
(tp, node.account.hostname)
AssertionError: Topic partition TopicPartition(topic='test_topic', partition=0) 
cannot be revoked from worker20 as it was not previously assigned to that 
consumer
{noformat}

It is unclear what is causing this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Igor Soarez

[Igor Soarez]

Thanks everyone, I'm very honoured to join!

--
Igor

[jira] [Created] (KAFKA-16622) Mirromaker2 first Checkpoint not emitted until consumer group fully catches up once

[Edoardo Comar (Jira)]

Edoardo Comar created KAFKA-16622:
-

 Summary: Mirromaker2 first Checkpoint not emitted until consumer 
group fully catches up once
 Key: KAFKA-16622
 URL: https://issues.apache.org/jira/browse/KAFKA-16622
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.6.2, 3.7.0, 3.8.0
Reporter: Edoardo Comar
 Attachments: edo-connect-mirror-maker-sourcetarget.properties

We observed an excessively delayed emission of the MM2 Checkpoint record.
It only gets created when the source consumer reaches the end of a topic. This 
does not seem reasonable.

In a very simple setup :

Tested with a standalone single process MirrorMaker2 mirroring between two 
single-node kafka clusters(mirromaker config attached) with quick refresh 
intervals (eg 5 sec) and a small offset.lag.max (eg 10)

create a single topic in the source cluster
produce data to it (e.g. 1 records)
start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec between 
polls which commits after each poll

watch the Checkpoint topic in the target cluster

bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
  --topic source.checkpoints.internal \
  --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
   --from-beginning

-> no record appears in the checkpoint topic until the consumer reaches the end 
of the topic (ie its consumer group lag gets down to 0).







--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2845

[Apache Jenkins Server]

See 

Re: [PR] MINOR: remove reference to IRC channel [kafka-site]

[via GitHub]

mimaison merged PR #599:
URL: https://github.com/apache/kafka-site/pull/599


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-16568) Add JMH Benchmarks for assignor performance testing

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16568.
-
Fix Version/s: 3.8.0
   Resolution: Fixed

> Add JMH Benchmarks for assignor performance testing 
> 
>
> Key: KAFKA-16568
> URL: https://issues.apache.org/jira/browse/KAFKA-16568
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ritika Reddy
>Assignee: Ritika Reddy
>Priority: Major
> Fix For: 3.8.0
>
>
> The 3 benchmarks that are being used to test the performance and efficiency 
> of the consumer group rebalance process.
>  * Client Assignors (assign method)
>  * Server Assignors (assign method)
>  * Target Assignment Builder (build method)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[PR] MINOR: remove reference to IRC channel [kafka-site]

[via GitHub]

raboof opened a new pull request, #599:
URL: https://github.com/apache/kafka-site/pull/599

   The channel on freenode seems abandoned (6 users, no topic).
   
   The channel on libera.chat is explicitly unofficial.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-932: Queues for Kafka

[David Jacot]

Hi Andrew,

Thanks for your responses and sorry for my late reply.

001: Makes sense. One thing that we could consider here is to allow
converting an empty classic or consumer group to a share group. If already
do this between empty classic and consumer groups.

004: I see. It would be great to call it out in the KIP.

006: I view `group.coordinator.rebalance.protocols` and `group.version` as
two different concepts. The former is purely here to enable/disable
protocols where the latter is mainly here to gate the versions of records
that we persist. Both are indeed required to enable the new consumer group
protocol. The reason is that we could imagine having a new version to the
`group.version` for different purposes (e.g. queues) but one may not want
to enable the new consumer protocol. Unless we have a strong reason not to
use these two, I would use them from day 1. This is actually something that
I got wrong in KIP-848, I think.

007: I see that you extend ShareGroupPartitionAssignor from
PartitionAssignor. I wonder if we should separate them because it means
that a ShareGroupPartitionAssignor could be accidentally used by a consumer
group. They won't be meant to be, I think.

009: I see. Thanks for clarifying.

017: I think that it should rather be num-partitions but in a different
group, the group of the share coordinator.

018/019: Would it be better to put them in a different group then (e.g.
share-coordinator-metrics)? This is confusing to have metrics crossing
group boundaries in my opinion.

021: "The state epoch is used to fence against writes to earlier versions
of a share-partition state". I understand that this protects the share
leader from overriding the state when it was altered by the admin APIs.
However, it does not seem to protect the state from being altered by, for
instance, a zombie share leader. A simple example is when the leadership of
a partition is moved from broker A to broker B. In this case, broker A
should not be permitted to alter the state anymore. The trick is that the
former leader may not be aware of the change yet. I think that the share
coordinator should prevent this. Don't you agree?

022: Would it make sense to rename `group.share.state.topic.*` configs to
use `share.coordinator.*` prefix as they are effectively share coordinator
matters.

023: Is there a mechanism to prevent fenced members from the group
coordinator point of view to not fetch from the share leaders? Or do we
assume that clients will do the right thing in this case? By this, I meant
that they will stop fetching immediately. This part is not clear in the KIP
or I missed it.

Best,
David

On Thu, Apr 25, 2024 at 1:19 PM Andrew Schofield 
wrote:

> Hi Jun,
> Thanks for the response.
>
> 123. Of course, ShareGroupHearbeat started off as ConsumerGroupHeartbeat
> and then unnecessary fields were removed. In the network issue case,
> there is not currently enough state being exchanged to be sure an
> assignment
> was received.
>
> Rather than add the group epoch to the ShareGroupHeartbeat, I have decided
> to go for TopicPartitions in ShareGroupHeartbeatRequest which mirrors
> ConsumerGroupHeartbeatRequest. It means the share group member does
> confirm the assignment it is using, and that can be used by the GC to
> safely
> stop repeating the assignment in heartbeat responses.
>
> 125. Ah, yes. This is indeed something possible with a consumer group
> and share groups should support it too. This does of course imply that
> ShareGroupPartitionMetadataValue needs an array of partitions, not
> just the number.
>
> 140. Yes, good spot. There is an inconsistency here in consumer groups
> where you can use AdminClient.deleteConsumerGroupOffsets at the
> partition level, but kafka-consumer-groups.sh --delete only operates
> at the topic level.
>
> Personally, I don’t think it’s sensible to delete offsets at the partition
> level only. You can reset them, but if you’re actively using a topic with
> a share group, I don’t see why you’d want to delete offsets rather than
> reset. If you’ve finished using a topic with a share group and want to
> clean
> up, use delete.
>
> So, I’ve changed the AdminClient.deleteConsumerGroupOffsets to be
> topic-based and the RPCs behind it.
>
> The GC reconciles the cluster state with the ShareGroupPartitionMetadata
> to spot deletion of topics and the like. However, when the offsets for
> a topic were deleted manually, the topic very like still exists so
> reconciliation
> alone is not going to be able to continue an interrupted operation that
> has started. So, I’ve added DeletingTopics back into
> ShareGroupPartitionMetadata for this purpose. It’s so failover of a GC
> can continue where it left off rather than leaving fragments across the
> SCs.
>
> 141. That is not required. Because this is not a compacted topic, it is
> not necessary to write tombstones for every key. As long as there is a
> clear and unambiguous record for the deletion of the group, that is enough.
> The 

[jira] [Created] (KAFKA-16621) Alter MirrorSourceConnector offsets dont work

[yuzhou (Jira)]

yuzhou created KAFKA-16621:
--

 Summary: Alter MirrorSourceConnector offsets dont work
 Key: KAFKA-16621
 URL: https://issues.apache.org/jira/browse/KAFKA-16621
 Project: Kafka
  Issue Type: Bug
  Components: connect
Reporter: yuzhou
 Attachments: image-2024-04-25-21-28-37-375.png

In connect-offsets topic:

the offsets wrote by connector, key is 
`\{"cluster":"A","partition":2,"topic":"topic"}`

after alter offsets, the key is  
`\{"partition":2,"topic":"topic","cluster":"A"}`

!image-2024-04-25-21-28-37-375.png!

in Worker.globalOffsetBackingStore.data, both two keys exist, because the are 
different strings:

!image-2024-04-25-21-33-51-892.png!

So alter offsets is not succussful, because when get offsets from 
globalOffsetBackingStore, always returns the first one.
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Greg,

Thanks for taking a close look at the KIP.

1/2) I understand your concern about leaking resources. I've played a
bit more with the code and I think we should be able to handle the
closing of the metrics internally rather than delegating it to the
user code. I built a small PoC inspired by your MonitorablePlugin
class example and that looked fine. I think we can even keep that
class internal. I updated the KIP accordingly.

3) An earlier version of the proposal used connector and task contexts
to allow them to retrieve their PluginMetrics instance. In a previous
comment Chris suggested switching to implementing Monitorable for
consistency. I think both approaches have pros and cons. I agree with
you that implementing Monitorable with cause compatibility issues with
older Connect runtimes. For that reason, I'm leaning towards
reintroducing the context mechanism. However we would still have this
issue with Converters/Transformations/Predicates. I think it's
typically a bit less problematic with these plugins but it's worth
considering the different approaches. If we can't agree on an approach
we can exclude Connect from this proposal and revisit it at a later
point.

4) If this KIP is accepted, I plan to follow up with another KIP to
make MirrorMaker use this mechanism instead of the custom metrics
logic it currently uses.

Thanks,
Mickael




On Wed, Apr 24, 2024 at 9:03 PM Mickael Maison  wrote:
>
> Hi Matthias,
>
> I'm not sure making the Monitorable interface Closeable really solves the 
> issue.
> Ultimately you need to understand the lifecycle of a plugin to
> determine when it make sense to close it and which part of the code is
> responsible for doing it. I'd rather have this described properly in
> the interface of the plugin itself than it being a side effect of
> implementing Monitorable.
>
> Looking at Streams, as far as I can tell the only pluggable interfaces
> that are Closeable today are the Serdes. It seems Streams can accept
> Serdes instances created by the user [0]. In that case, I think it's
> probably best to ignore Streams in this KIP. Nothing should prevent
> Streams for adopting it, in a way that makes sense for Streams, in a
> future KIP if needed.
>
> 0: 
> https://github.com/apache/kafka/blob/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/wordcount/WordCountDemo.java#L84
>
> Thanks,
> Mickael
>
>
>
>
>
> On Fri, Feb 9, 2024 at 1:15 AM Greg Harris  
> wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP, this looks like a great change!
> >
> > 1. I see that one of my concerns was already discussed, and appears to
> > have been concluded with:
> >
> > > I considered Chris' idea of automatically removing metrics but decided to 
> > > leave that responsibility to the plugins.
> >
> > After chasing resource leaks for the last few years, I've internalized
> > that preventing leaks through careful implementation is always
> > inadequate, and that leaks need to be prevented by design.
> > If a leak is possible in a design, then we should count on it
> > happening somewhere as a certainty, and should be prepared for the
> > behavior afterwards.
> >
> > Chris already brought up one of the negative behaviors: Connect
> > plugins which are cancelled may keep their metrics open past the point
> > that a replacement plugin is instantiated.
> > This will have the effect of showing incorrect metrics, which is
> > harmful and confusing for operators.
> > If you are constantly skeptical of the accuracy of your metrics, and
> > there is no "source of truth" to verify against, then what use are the
> > metrics?
> >
> > I think that managing the lifecycle of the PluginMetrics on the
> > framework side would be acceptable if we had an internal class like
> > the following, to keep a reference to the metrics adjacent to the
> > plugin:
> > class MonitorablePlugin implements Supplier, Closeable {
> > MonitorablePlugin(T plugin, PluginMetrics metrics);
> > }
> > I already believe that we need similar wrapper classes in Connect [1]
> > to manage classloader swapping & exception safety, and this simpler
> > interface could be applied to non-connect call-sites that don't need
> > to swap the classloader.
> >
> > 2. Your "MyInterceptor" class doesn't have a "metrics" field, and
> > doesn't perform a null-check on the field in close().
> > Keeping the PluginMetrics as an non-final instance variable in every
> > plugin implementation is another burden on the plugin implementations,
> > as they will need to perform null checks in-case the metrics are never
> > initialized, such as in a test environment.
> > It also shows that without the Closeable interface, plugins may not
> > need the PluginMetrics object after the initial setup if they only
> > have a fixed set of sensors that need to be made instance fields.
> >
> > 3. I realized when writing the above that this implicitly sets a
> > minimum framework version for plugins, as the Monitorable interface
> > must exist in 

[jira] [Created] (KAFKA-16620) Kraft quorum cannot be formed if all controllers are restarted at the same time

[Gantigmaa Selenge (Jira)]

Gantigmaa Selenge created KAFKA-16620:
-

 Summary: Kraft quorum cannot be formed if all controllers are 
restarted at the same time
 Key: KAFKA-16620
 URL: https://issues.apache.org/jira/browse/KAFKA-16620
 Project: Kafka
  Issue Type: Bug
Reporter: Gantigmaa Selenge


Controller quorum cannot seem to form at all after accidentally restarting all 
controller nodes at the same time in a test environment. This is reproducible, 
happens almost everytime when restarting all controller nodes of the cluster. 

Started a cluster with 3 controller nodes and 3 broker nodes. After restarting 
the controller nodes, one of them becomes the active controller but resigns due 
to fetch timeout. The quorum leadership bounces off like this between the nodes 
indefinitely. 
The controller.quorum.fetch.timeout.ms was set to the default of 2 seconds. 
Logs from an active controller:
```
2024-04-17 14:00:48,250 INFO [QuorumController id=0] Becoming the active 
controller at epoch 34, next write offset 1116. 
(org.apache.kafka.controller.QuorumController) 
[quorum-controller-0-event-handler]
2024-04-17 14:00:48,250 WARN [QuorumController id=0] Performing controller 
activation. Loaded ZK migration state of NONE. 
(org.apache.kafka.controller.QuorumController) 
[quorum-controller-0-event-handler]
2024-04-17 14:00:48,701 INFO [RaftManager id=0] Node 1 disconnected. 
(org.apache.kafka.clients.NetworkClient) [kafka-0-raft-outbound-request-thread]
2024-04-17 14:00:48,701 WARN [RaftManager id=0] Connection to node 1 
(my-cluster-controller-1.my-cluster-kafka-brokers.roller.svc.cluster.local/10.244.0.68:9090)
 could not be established. Node may not be available. 
(org.apache.kafka.clients.NetworkClient) [kafka-0-raft-outbound-request-thread]
2024-04-17 14:00:48,776 DEBUG [UnifiedLog partition=__cluster_metadata-0, 
dir=/var/lib/kafka/data/kafka-log0] Flushing log up to offset 1117 
(exclusive)with recovery point 1117, last flushed: 1713362448239,  current 
time: 1713362448776,unflushed: 1 (kafka.log.UnifiedLog) [kafka-0-raft-io-thread]
2024-04-17 14:00:49,277 DEBUG [UnifiedLog partition=__cluster_metadata-0, 
dir=/var/lib/kafka/data/kafka-log0] Flushing log up to offset 1118 
(exclusive)with recovery point 1118, last flushed: 1713362448777,  current 
time: 
...
2024-04-17 14:01:35,934 DEBUG [UnifiedLog partition=__cluster_metadata-0, 
dir=/var/lib/kafka/data/kafka-log0] Flushing log up to offset 1200 
(exclusive)with recovery point 1200, last flushed: 1713362489371,  current 
time: 1713362495934,unflushed: 1 (kafka.log.UnifiedLog) [kafka-0-raft-io-thread]
2024-04-17 14:01:36,121 INFO [RaftManager id=0] Did not receive fetch request 
from the majority of the voters within 3000ms. Current fetched voters are []. 
(org.apache.kafka.raft.LeaderState) [kafka-0-raft-io-thread]
2024-04-17 14:01:36,223 WARN [QuorumController id=0] Renouncing the leadership 
due to a metadata log event. We were the leader at epoch 34, but in the new 
epoch 35, the leader is (none). Reverting to last stable offset 1198. 
(org.apache.kafka.controller.QuorumController) 
[quorum-controller-0-event-handler]
2024-04-17 14:01:36,223 INFO [QuorumController id=0] 
failAll(NotControllerException): failing writeNoOpRecord(152156824). 
(org.apache.kafka.deferred.DeferredEventQueue) 
[quorum-controller-0-event-handler]
2024-04-17 14:01:36,223 INFO [QuorumController id=0] writeNoOpRecord: event 
failed with NotControllerException in 6291037 microseconds. 
(org.apache.kafka.controller.QuorumController) 
[quorum-controller-0-event-handler]
```
Logs from the follower:
```
024-04-17 14:00:48,242 INFO [RaftManager id=2] Completed transition to 
FollowerState(fetchTimeoutMs=2000, epoch=34, leaderId=0, voters=[0, 1, 2], 
highWatermark=Optional[LogOffsetMetadata(offset=1113, 
metadata=Optional.empty)], fetchingSnapshot=Optional.empty) from 
Voted(epoch=34, votedId=0, voters=[0, 1, 2], electionTimeoutMs=1794) 
(org.apache.kafka.raft.QuorumState) [kafka-2-raft-io-thread]
2024-04-17 14:00:48,242 INFO [QuorumController id=2] In the new epoch 34, the 
leader is 0. (org.apache.kafka.controller.QuorumController) 
[quorum-controller-2-event-handler]
2024-04-17 14:00:48,247 DEBUG [UnifiedLog partition=__cluster_metadata-0, 
dir=/var/lib/kafka/data/kafka-log2] Flushing log up to offset 1116 
(exclusive)with recovery point 1116, last flushed: 1713362442238,  current 
time: 1713362448247,unflushed: 2 (kafka.log.UnifiedLog) [kafka-2-raft-io-thread]
2024-04-17 14:00:48,777 DEBUG [UnifiedLog partition=__cluster_metadata-0, 
dir=/var/lib/kafka/data/kafka-log2] Flushing log up to offset 1117 
(exclusive)with recovery point 1117, last flushed: 1713362448249,  current 
time: 1713362448777,unflushed: 1 (kafka.log.UnifiedLog) [kafka-2-raft-io-thread]
2024-04-17 14:00:49,278 DEBUG [UnifiedLog partition=__cluster_metadata-0, 
dir=/var/lib/kafka/data/kafka-log2] Flushing log up to offset 

[jira] [Created] (KAFKA-16619) Unnecessary warning : "Loaded ZK migration state of NONE"

[Jira]

F Méthot created KAFKA-16619:


 Summary: Unnecessary warning : "Loaded ZK migration state of NONE"
 Key: KAFKA-16619
 URL: https://issues.apache.org/jira/browse/KAFKA-16619
 Project: Kafka
  Issue Type: Improvement
  Components: controller
Affects Versions: 3.6.2
Reporter: F Méthot


When we launch a fresh cluster of Kafka and Kraft Controller, no zookeeper 
involved.

We get this warning in the log:

[2024-04-15 03:44:33,881] WARN [QuorumController id=3] Performing controller 
activation. Loaded ZK migration state of NONE. 
(org.apache.kafka.controller.QuorumController)

 

Our project has no business with Zookeeper, seeing this message prompted us to 
investigate and spend time looking up this warning to find an explanation.

We have that setting

{_}zookeeper.metadata.migration.enable{_}=false

and we still get that warning.

In future version, to avoid further confusion this message should not be showed 
when zookeeper is not involved at all .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

[Loic Greffier]

Hi Matthias,

I have updated the KIP regarding points 103 and 108.

103.
I have suggested a new `ImmutableHeaders` interface to deal with the
immutability concern of the headers, which is basically the `Headers`
interface without the write accesses.

public interface ImmutableHeaders {
Header lastHeader(String key);
Iterable headers(String key);
Header[] toArray();
}

The `Headers` interface can be updated accordingly:

public interface Headers extends ImmutableHeaders, Iterable {
//…
}

Loïc

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

[Luke Chen]

Hi, Abhijeet,

Thanks for the update.

I have no more comments.

Luke

On Thu, Apr 25, 2024 at 4:21 AM Jun Rao  wrote:

> Hi, Abhijeet,
>
> Thanks for the updated KIP. It looks good to me.
>
> Jun
>
> On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> wrote:
>
> > Hi Jun,
> >
> > Please find my comments inline.
> >
> >
> > On Thu, Apr 18, 2024 at 3:26 AM Jun Rao 
> wrote:
> >
> > > Hi, Abhijeet,
> > >
> > > Thanks for the reply.
> > >
> > > 1. I am wondering if we could achieve the same result by just lowering
> > > local.retention.ms and local.retention.bytes. This also allows the
> newly
> > > started follower to build up the local data before serving the consumer
> > > traffic.
> > >
> >
> > I am not sure I fully followed this. Do you mean we could lower the
> > local.retention (by size and time)
> > so that there is little data on the leader's local storage so that the
> > follower can quickly catch up with the leader?
> >
> > In that case, we will need to set small local retention across brokers in
> > the cluster. It will have the undesired
> > effect where there will be increased remote log fetches for serving
> consume
> > requests, and this can cause
> > degradations. Also, this behaviour (of increased remote fetches) will
> > happen on all brokers at all times, whereas in
> > the KIP we are restricting the behavior only to the newly bootstrapped
> > brokers and only until the time it fully builds
> > the local logs as per retention defined at the cluster level.
> > (Deprioritization of the broker could help reduce the impact
> >  even further)
> >
> >
> > >
> > > 2. Have you updated the KIP?
> > >
> >
> > The KIP has been updated now.
> >
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Apr 9, 2024 at 3:36 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > +1 to Jun for adding the consumer fetching from a follower scenario
> > > > also to the existing section that talked about the drawback when a
> > > > node built with last-tiered-offset has become a leader. As Abhijeet
> > > > mentioned, we plan to have a follow-up KIP that will address these by
> > > > having a deprioritzation of these brokers. The deprioritization of
> > > > those brokers can be removed once they catchup until the local log
> > > > retention.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Tue, 9 Apr 2024 at 14:08, Luke Chen  wrote:
> > > > >
> > > > > Hi Abhijeet,
> > > > >
> > > > > Thanks for the KIP to improve the tiered storage feature!
> > > > >
> > > > > Questions:
> > > > > 1. We could also get the "pending-upload-offset" and epoch via
> remote
> > > log
> > > > > metadata, instead of adding a new API to fetch from the leader.
> Could
> > > you
> > > > > explain why you choose the later approach, instead of the former?
> > > > > 2.
> > > > > > We plan to have a follow-up KIP that will address both the
> > > > > deprioritization
> > > > > of these brokers from leadership, as well as
> > > > > for consumption (when fetching from followers is allowed).
> > > > >
> > > > > I agree with Jun that we might need to make it clear all possible
> > > > drawbacks
> > > > > that could have. So, could we add the drawbacks that Jun mentioned
> > > about
> > > > > the performance issue when consumer fetch from follower?
> > > > >
> > > > > 3. Could we add "Rejected Alternatives" section to the end of the
> KIP
> > > to
> > > > > add some of them?
> > > > > Like the "ListOffsetRequest" approach VS
> > > "Earliest-Pending-Upload-Offset"
> > > > > approach, or getting the "Earliest-Pending-Upload-Offset" from
> remote
> > > log
> > > > > metadata... etc.
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > >
> > > > > On Tue, Apr 9, 2024 at 2:25 PM Abhijeet Kumar <
> > > > abhijeet.cse@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Christo,
> > > > > >
> > > > > > Please find my comments inline.
> > > > > >
> > > > > > On Fri, Apr 5, 2024 at 12:36 PM Christo Lolov <
> > > christolo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Abhijeet and Jun,
> > > > > > >
> > > > > > > I have been mulling this KIP over a bit more in recent days!
> > > > > > >
> > > > > > > re: Jun
> > > > > > >
> > > > > > > I wasn't aware we apply 2.1 and 2.2 for reserving new
> timestamps
> > -
> > > in
> > > > > > > retrospect it should have been fairly obvious. I would need to
> go
> > > an
> > > > > > update
> > > > > > > KIP-1005 myself then, thank you for giving the useful
> reference!
> > > > > > >
> > > > > > > 4. I think Abhijeet wants to rebuild state from
> > > latest-tiered-offset
> > > > and
> > > > > > > fetch from latest-tiered-offset + 1 only for new replicas (or
> > > > replicas
> > > > > > > which experienced a disk failure) to decrease the time a
> > partition
> > > > spends
> > > > > > > in under-replicated state. In other words, a follower which has
> > > just
> > > > > > fallen
> > > > > > > out of ISR, but has local data will 

Re: Permissions to contribute to Apache Kafka

[Mickael Maison]

Hi,

I've granted you contributor permissions in Jira and Confluence.
Thanks for your interest in Kafka!

Mickael

On Thu, Apr 25, 2024 at 5:47 AM Rajdeep Sahoo
 wrote:
>
> Hi team ,
> Please find my wiki id and jira id mentioned below. Requesting you to grant
> access so that I will be able to contribute to apache kafka.
>
> *wiki id*: rajdeepsahoo2012
> *jira id*: rajdeep_sahoo
>
> Thanks ,
> Rajdeep sahoo

Re: [ANNOUNCE] New committer: Igor Soarez

[Randall Hauch]

Congratulations, Igor!

Randall

On Thu, Apr 25, 2024 at 12:51 AM Viktor Somogyi-Vass
 wrote:

> Congrats Igor!
>
> On Thu, Apr 25, 2024, 07:01 Bruno Cadonna  wrote:
>
> > Congrats!
> >
> > Best,
> > Bruno
> >
> > Am 25. April 2024 05:18:19 MESZ schrieb Yash Mayya  >:
> > >Congratulations Igor!
> > >
> > >On Wed, 24 Apr, 2024, 23:36 Colin McCabe,  wrote:
> > >
> > >> Hi all,
> > >>
> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Igor
> > >> Soarez.
> > >>
> > >> Igor has been a Kafka contributor since 2019. In addition to being a
> > >> regular contributor and reviewer, he has made significant
> contributions
> > to
> > >> improving Kafka's JBOD support in KRaft mode. He has also contributed
> to
> > >> discussing and reviewing many KIPs such as KIP-690, KIP-554, KIP-866,
> > and
> > >> KIP-938.
> > >>
> > >> Congratulations, Igor!
> > >>
> > >> Thanks,
> > >>
> > >> Colin (on behalf of the Apache Kafka PMC)
> > >>
> >
>

Re: [VOTE] KIP-910: Update Source offsets for Source Connectors without producing records

[Sagar]

Hey All,

Bumping the vote thread after a long time!

Thanks!
Sagar.

On Fri, Feb 2, 2024 at 4:24 PM Sagar  wrote:

> Thanks Yash!
>
> I am hoping to have this released in 3.8 so it would be good to get the
> remaining 2 votes.
>
> Thanks!
> Sagar.
>
>
> On Tue, Jan 30, 2024 at 3:18 PM Yash Mayya  wrote:
>
>> Hi Sagar,
>>
>> Thanks for the KIP and apologies for the extremely long delay here! I
>> think
>> we could do with some wordsmithing on the Javadoc for the new
>> `SourceTask::updateOffsets` method but that can be taken care of in the
>> PR.
>>
>> +1 (binding)
>>
>> Thanks,
>> Yash
>>
>> On Wed, Nov 15, 2023 at 11:43 PM Sagar  wrote:
>>
>> > Hey all,
>> >
>> > Bumping this vote thread again after quite a while.
>> >
>> > Thanks!
>> > Sagar.
>> >
>> > On Wed, Sep 6, 2023 at 3:58 PM Sagar  wrote:
>> >
>> > > Hi All,
>> > >
>> > > Based on the latest discussion thread, it appears as if all open
>> > questions
>> > > have been answered.
>> > >
>> > > Hopefully now we are in a state where we can close out on the Voting
>> > > process.
>> > >
>> > > Thanks everyone for the great feedback.
>> > >
>> > > Thanks!
>> > > Sagar.
>> > >
>> > > On Fri, Aug 18, 2023 at 9:00 AM Sagar 
>> wrote:
>> > >
>> > >> Hi All,
>> > >>
>> > >> Bumping the voting thread again.
>> > >>
>> > >> Thanks!
>> > >> Sagar.
>> > >>
>> > >> On Wed, Aug 2, 2023 at 4:43 PM Sagar 
>> wrote:
>> > >>
>> > >>> Attaching the KIP link for reference:
>> > >>>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
>> > >>>
>> > >>> Thanks!
>> > >>> Sagar.
>> > >>>
>> > >>> On Wed, Aug 2, 2023 at 4:37 PM Sagar 
>> > wrote:
>> > >>>
>> >  Hi All,
>> > 
>> >  Calling a Vote on KIP-910 [1]. I feel we have converged to a
>> > reasonable
>> >  design. Ofcourse I am open to any feedback/suggestions and would
>> > address
>> >  them.
>> > 
>> >  Thanks!
>> >  Sagar.
>> > 
>> > >>>
>> >
>>
>

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Jun,
Thanks for the response.

123. Of course, ShareGroupHearbeat started off as ConsumerGroupHeartbeat
and then unnecessary fields were removed. In the network issue case,
there is not currently enough state being exchanged to be sure an assignment 
was received.

Rather than add the group epoch to the ShareGroupHeartbeat, I have decided
to go for TopicPartitions in ShareGroupHeartbeatRequest which mirrors
ConsumerGroupHeartbeatRequest. It means the share group member does
confirm the assignment it is using, and that can be used by the GC to safely
stop repeating the assignment in heartbeat responses.

125. Ah, yes. This is indeed something possible with a consumer group
and share groups should support it too. This does of course imply that
ShareGroupPartitionMetadataValue needs an array of partitions, not
just the number.

140. Yes, good spot. There is an inconsistency here in consumer groups
where you can use AdminClient.deleteConsumerGroupOffsets at the
partition level, but kafka-consumer-groups.sh --delete only operates
at the topic level.

Personally, I don’t think it’s sensible to delete offsets at the partition
level only. You can reset them, but if you’re actively using a topic with
a share group, I don’t see why you’d want to delete offsets rather than
reset. If you’ve finished using a topic with a share group and want to clean
up, use delete.

So, I’ve changed the AdminClient.deleteConsumerGroupOffsets to be
topic-based and the RPCs behind it.

The GC reconciles the cluster state with the ShareGroupPartitionMetadata
to spot deletion of topics and the like. However, when the offsets for
a topic were deleted manually, the topic very like still exists so 
reconciliation
alone is not going to be able to continue an interrupted operation that
has started. So, I’ve added DeletingTopics back into
ShareGroupPartitionMetadata for this purpose. It’s so failover of a GC
can continue where it left off rather than leaving fragments across the
SCs.

141. That is not required. Because this is not a compacted topic, it is
not necessary to write tombstones for every key. As long as there is a
clear and unambiguous record for the deletion of the group, that is enough.
The tombstone for ShareGroupPartitionMetadata is theoretically not
required but it’s a single record, rather than one per member, so I prefer
to leave it as a record that the interactions with the SC have been completed.

142.
142.1. It will prompt the user to confirm they want to continue.
This is in common with `kafka-consumer-groups.sh` which historically
has defaulted to --dry-run behaviour, but is due to change to prompting
if neither --dry-run nor --execute is specified “in a future major release”.

142.2. It should support partition-level reset, but only topic-level delete.
I have updated the usage text accordingly. This is in common with
kafka-consumer-groups.sh.

142.3. --dry-run displays the operation that would be executed.

142.4. The valid values are: Dead, Empty, Stable. Added to the
usage text.

143. DescribeShareGroupOffsets is served by the group coordinator
for this kind of reason.

144. That’s the default. If you haven’t asked to release or reject, it accepts.
This is analogous to fetching and committing offsets in a consumer group.

145. I think this is a good idea, but I would prefer to defer it until a future
metrics KIP that I have planned. In KIP-932, I have added basic metrics only.
For example, you’ll see that there’s no concept of lag yet, which surely
will have relevance for share groups. I plan to create and deliver the
metrics KIP before share groups are declared ready for production.
I want the new metrics to be developed with the experience of running
the code.

146. Milliseconds. KIP updated.

147. There is a membership state machine in the client that
changes states as the ShareGroupHeartbeat requests and responses
flow. The duration of a rebalance will be shorter from the point of
view of the share-group consumer because it doesn’t have to worry about
rebalance callbacks and committing offsets as the partitions move
around, but the overall flow is very similar. So, it’s the state transitions
that drive the collection of the rebalance metrics.

148. Strangely, none of the existing uses of records-per-request-avg
actually have records-per-request-max. I tend to err on the side of
consistency, but can’t think of any reason not to add this. Done.

149. There are several error codes for WriteShareGroupStateResponse:

NOT_COORDINATOR - This is not the share coordinator you’re looking for.
COORDINATOR_NOT_AVAILABLE - The SC can’t 
COORDINATOR_LOAD_IN_PROGRESS - The SC is replaying the topic.
GROUP_ID_NOT_FOUND - The SC doesn’t have state for this group.
UNKNOWN_TOPIC_OR_PARTITION - The SC doesn’t have state for this
topic-partition.
FENCED_STATE_EPOCH - The write has the wrong state epoch.
INVALID_REQUEST - There was a problem with the request.


Thanks,
Andrew


> On 24 Apr 2024, at 19:10, Jun Rao  wrote:
> 
> Hi, 

Confluence edit access

[Claude Warren]

I would like to get edit access to the Kafka confluence so that I can work
on KIP-936.  Can someone here do that or do I need to go through Infra?

Claude

Re: [VOTE] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

[Nikhil Ramakrishnan]

Thank you Justine, Christo, Andrew, and Luke for the vote! I will keep
this vote open for about 3 more days in case there are any more
comments or suggestions.


Thanks,
Nikhil

On Wed, Apr 24, 2024 at 1:01 PM Luke Chen  wrote:
>
> Hi Nikhil,
> Thanks for the KIP.
>
> +1 from me.
>
> Luke
>
> On Mon, Apr 22, 2024 at 7:41 PM Andrew Schofield 
> wrote:
>
> > Hi Nikhil,
> > Thanks for the KIP. Looks good to me.
> >
> > +1 (non-binding)
> >
> > Thanks,
> > Andrew
> >
> > > On 22 Apr 2024, at 09:17, Christo Lolov  wrote:
> > >
> > > Heya Nikhil,
> > >
> > > Thanks for the proposal, as mentioned before it makes sense to me!
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > Christo
> > >
> > > On Sat, 20 Apr 2024 at 00:25, Justine Olshan
> > 
> > > wrote:
> > >
> > >> Hey Nikhil,
> > >>
> > >> I meant to comment on the discussion thread, but my draft took so long,
> > you
> > >> opened the vote.
> > >>
> > >> Regardless, I just wanted to say that it makes sense to me. +1 (binding)
> > >>
> > >> Justine
> > >>
> > >> On Fri, Apr 19, 2024 at 7:22 AM Nikhil Ramakrishnan <
> > >> ramakrishnan.nik...@gmail.com> wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> I would like to start a voting thread for KIP-1037: Allow
> > >>> WriteTxnMarkers API with Alter Cluster Permission
> > >>> (
> > >>>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1037%3A+Allow+WriteTxnMarkers+API+with+Alter+Cluster+Permission
> > >>> )
> > >>> as there have been no objections on the discussion thread.
> > >>>
> > >>> For comments or feedback please check the discussion thread here:
> > >>> https://lists.apache.org/thread/bbkyt8mrc8xp3jfyvhph7oqtjxl29xmn
> > >>>
> > >>> Thanks,
> > >>> Nikhil
> > >>>
> > >>
> >
> >

[DISCUSS] Apache Kafka 3.7.1 release

[Igor Soarez]

Hi everyone,

I'd like to volunteer to be the release manager for a 3.7.1 release.

Please keep in mind, this would be my first release, so I might have some 
questions,
and it might also take me a bit longer to work through the release process.
So I'm thinking a good target would be toward the end of May.

Please let me know your thoughts and if there are any objections or concerns.

Thanks,

--
Igor

[jira] [Created] (KAFKA-16618) Update the RPC for ConsumerGroupHeartbeatRequest and ConsumerGroupHeartbeatResponse

[Phuc Hong Tran (Jira)]

Phuc Hong Tran created KAFKA-16618:
--

 Summary: Update the RPC for ConsumerGroupHeartbeatRequest and 
ConsumerGroupHeartbeatResponse
 Key: KAFKA-16618
 URL: https://issues.apache.org/jira/browse/KAFKA-16618
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Reporter: Phuc Hong Tran
Assignee: Phuc Hong Tran
 Fix For: 4.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)