[jira] [Resolved] (KAFKA-17509) Introduce a delayed action queue to complete purgatory actions outside purgatory code.

2024-10-04 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17509. - Fix Version/s: 4.0.0 Resolution: Fixed Merged the PR to trunk. > Introduce a delayed act

Re: [DISCUSS] KIP-1090 Flaky Tests 👻

2024-09-30 Thread Jun Rao
Hi, David, Thanks for the report. 1. Where could we see the reports (quarantined tests, etc) mentioned in the KIP? 2. The quarantine process seems to only apply to integration tests. What's our recommendation for flaky unit tests? Jun On Thu, Sep 26, 2024 at 1:34 PM David Arthur wrote: > If t

[jira] [Resolved] (KAFKA-17076) logEndOffset could be lost due to log cleaning

2024-09-24 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17076. - Fix Version/s: 4.0.0 Assignee: Vincent Jiang Resolution: Fixed Merged the PR to trunk

[jira] [Resolved] (KAFKA-17400) Introduce a purgatory to deal with share fetch requests that cannot be completed instantaneously

2024-09-11 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17400. - Fix Version/s: 4.0.0 Resolution: Fixed merged the PR to trunk. > Introduce a purgatory to d

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

2024-08-23 Thread Jun Rao
Hi, Kamal, Thanks for the KIP. Just a couple of minor comments. Since we use the purgatory name as the metric tag, could you spell out the exact new purgatory name for remote list offset? private val metricsTags = Map("delayedOperation" -> purgatoryName).asJava metricsGroup.newGauge("Purgator

Re: [VOTE] KIP-932: Queues for Kafka

2024-08-15 Thread Jun Rao
ent if required. Let me know if that’s OK. > > Thanks, > Andrew > > > > On 14 Aug 2024, at 00:00, Jun Rao wrote: > > > > Hi, Andrew, > > > > Thanks for the update. A couple of followup questions. > > > > 1. ShareGroupPartitionMetadataValue: Is

Re: [VOTE] KIP-932: Queues for Kafka

2024-08-13 Thread Jun Rao
> > > On 21 May 2024, at 15:35, Andrew Schofield > wrote: > > > > Hi Jun, > > All the client metrics are standard. None are required. > > > > I’ve updated the KIP accordingly. > > > > Thanks, > > Andrew > > > >> On 15 May 2024, at

[jira] [Resolved] (KAFKA-16746) Add support for share acknowledgement request in KafkaApis

2024-08-12 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-16746. - Fix Version/s: 4.0.0 Resolution: Fixed merged the PR to trunk > Add support for sh

[jira] [Resolved] (KAFKA-16745) Add support for share fetch request in KafkaApis

2024-08-06 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-16745. - Fix Version/s: 4.0.0 Resolution: Fixed Merged the PR to trunk. > Add support for share fe

[jira] [Resolved] (KAFKA-17230) Kafka consumer client doesn't report node request-latency metrics

2024-08-01 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17230. - Fix Version/s: 3.9.0 Resolution: Fixed merged the PR to trunk > Kafka consumer client does

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-07-30 Thread Jun Rao
Thanks for updating the KIP, Justine. Jun On Tue, Jul 30, 2024 at 1:37 PM Justine Olshan wrote: > I added this update to the end of the section Colin added. > > Justine > > On Tue, Jul 30, 2024 at 11:01 AM Jun Rao wrote: > > > Hi, Colin, > > > > Tha

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-07-30 Thread Jun Rao
9:25 PM Colin McCabe wrote: > Hi Jun, > > Just to close the loop on this... the KIP now mentions both > ApiVersionResponse and BrokerRegistrationRequest. > > best, > Colin > > On Mon, Jul 8, 2024, at 14:57, Jun Rao wrote: > > Hi, Colin, > > > > Thanks

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

2024-07-24 Thread Jun Rao
Hi, Mickael, Thanks for the KIP. 10. A common case is only one log dir per broker. Could we support sth like --add-config cordoned.log.dirs=* to make it more convenient for this case? 11. Since we changed the metadata record format, should we gate the new configuration based on a new metadata ve

Re: [VOTE] KIP-939: Support Participation in 2PC

2024-07-22 Thread Jun Rao
Hi, Artem, Thanks for the KIP. +1 from me. Jun On Wed, May 29, 2024 at 9:41 AM Jason Gustafson wrote: > +1 Thanks for the KIP! > > On Wed, Mar 13, 2024 at 5:13 AM Omnia Ibrahim > wrote: > > > I had a look at the discussion thread and the KIP looks exciting. > > +1 non-binding > > > > Best > >

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

2024-07-15 Thread Jun Rao
Hi, Christo, Thanks for the KIP and sorry for the late reply. Since this KIP changes the format of TopicRecord, it would be useful to add a section on upgrade based on bumped metadata.version. Jun On Wed, May 15, 2024 at 2:47 AM Luke Chen wrote: > Hi Christo, > > Thanks for the explanation. >

Re: [DISCUSS] KIP-890 Server Side Defense

2024-07-10 Thread Jun Rao
ue in the record based on the EndTxn version. > > Justine > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao wrote: > > > Hi, Justine, > > > > Thanks for the reply. > > > > 120. If the broker sends TV Y for the finalized version in > > ApiVersionResponse, bu

Re: [DISCUSS] KIP-890 Server Side Defense

2024-07-10 Thread Jun Rao
k. I think it is > ok to write it to disk though. Let me know if you think otherwise. > > Justine > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao wrote: > > > Hi, Justine, > > > > Thanks for the update and sorry for the late reply. > > > > 120. I am wonde

Re: [DISCUSS] KIP-890 Server Side Defense

2024-07-10 Thread Jun Rao
f we did want to gate it on TV, I think the idea is to move away > >> from MV gating inter broker protocols. Ideally we can get to a state > where > >> MV is just used for metadata changes. > >> > >> I think some of this discussion might fit more with the featu

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-07-08 Thread Jun Rao
m/apache/kafka/pull/16421 > so that we change the minVersion=0 to minVersion=1 in older > ApiVersionsResponses. > > I hope we can get this in soon and unblock the features that are waiting > for it! > > best, > Colin > > On Wed, Jul 3, 2024, at 10:55, Jun Rao wrote: > &g

[jira] [Created] (KAFKA-17076) logEndOffset could be lost due to log cleaning

2024-07-03 Thread Jun Rao (Jira)
Jun Rao created KAFKA-17076: --- Summary: logEndOffset could be lost due to log cleaning Key: KAFKA-17076 URL: https://issues.apache.org/jira/browse/KAFKA-17076 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-07-03 Thread Jun Rao
> lose 0. I suppose we'd have to special case metadata.version being set to > > 1, since that was NOT equivalent to it being "off" > > > > best, > > Colin > > > > > > On Mon, Jul 1, 2024, at 10:11, Jun Rao wrote: > > > Hi, Dav

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-07-01 Thread Jun Rao
rsion=1 that does basically nothing and > group.version=2 that enables the new protocol. I suppose that we could do > the same for the other features. I agree that it is less elegant but it > would avoid all the backward compatibility issues. > > Best, > David > > On F

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-06-28 Thread Jun Rao
client from returning the finalized > versions of those features to the old client. > > So basically we have a choice between missing information in > Admin.describeFeatures and wrong information. I would lean towards the > missing information path, but I guess we should try out an old

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-06-27 Thread Jun Rao
is finalized at version 0. However, since downgrading is less frequent than upgrading, this approach seems slightly more consistent. No matter what approach we take, it would be useful to document this inconsistency to the old client. Thanks, Jun On Wed, Jun 26, 2024 at 1:18 PM Jun Rao wrote

Re: [DISCUSS] KIP-1014: Managing Unstable Features in Apache Kafka

2024-06-26 Thread Jun Rao
lso, how does a user test an EA feature in a release? It's inconvenient for a user to change code and recompile the binary. Jun On Wed, Jun 26, 2024 at 1:38 PM Colin McCabe wrote: > On Wed, Jun 26, 2024, at 13:16, Jun Rao wrote: > > Hi, Colin, > > > > Thanks for the rep

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-06-26 Thread Jun Rao
ures, > we > > can't return 0 versions and we won't be able to see group version set. > > However, the new api will return 0 and the group version correctly. > > > > Let me know if this is consistent with your thoughts, Colin. > > > > Justine > &g

Re: [DISCUSS] KIP-1014: Managing Unstable Features in Apache Kafka

2024-06-26 Thread Jun Rao
set to true? How do we plan to enforce that? Jun On Wed, Jun 26, 2024 at 12:52 PM Colin McCabe wrote: > On Wed, Jun 26, 2024, at 12:09, Jun Rao wrote: > > Hi, Colin, > > > > Thanks for restarting the discussion. A few comments. > > > > 1. "An unstable

Re: [DISCUSS] KIP-1014: Managing Unstable Features in Apache Kafka

2024-06-26 Thread Jun Rao
Hi, Colin, Thanks for restarting the discussion. A few comments. 1. "An unstable RPC version can be changed at any time, until it becomes stable." What's our recommendation to non-java developers? Should they start building a new version of an RPC until it is stable? Should we explicitly mark u

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-06-24 Thread Jun Rao
gt; solution: > > > >> > > > >> - There will be a new v4 for ApiVersionsRequest > > > >> > > > >> - Clients that sent v4 will promise to correctly handle ranges that > > > start > > > >> with 0, such as [0, 1] > > >

[jira] [Resolved] (KAFKA-16541) Potential leader epoch checkpoint file corruption on OS crash

2024-06-05 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-16541. - Fix Version/s: 4.0.0 Resolution: Fixed Merged the PR to trunk. > Potential leader ep

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

2024-05-22 Thread Jun Rao
hat > require changes, we can address them in the PR or in case it's already > merged, afterwards. > > Best, > Ivan > > [1] https://github.com/apache/kafka/pull/13277 > > On Thu, May 16, 2024, at 01:52, Jun Rao wrote: > > Hi, Ivan, > > > > You

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

2024-05-15 Thread Jun Rao
what you meant or I > misunderstand? > As for the KIP-932, I'll have a closer look. > > Ivan > > > On Wed, May 15, 2024, at 20:14, Jun Rao wrote: > > Hi, Ivan, > > > > Thanks for the KIP. +1 > > > > Just a minor comment. Should we add metadata.reco

Re: [VOTE] KIP-932: Queues for Kafka

2024-05-15 Thread Jun Rao
s for the nice KIP, it will allow other messaging use cases to be > >> onboarded to Kafka. > >> > >> +1 from me. > >> > >> Satish. > >> > >> On Tue, 7 May 2024 at 03:41, Jun Rao wrote: > >>> > >>> Hi, Andrew, > >>&

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

2024-05-15 Thread Jun Rao
Hi, Ivan, Thanks for the KIP. +1 Just a minor comment. Should we add metadata.recovery.strategy to the Streams and the newly introduced ShareConsumer (KIP-932) too? Jun On Wed, May 8, 2024 at 11:35 AM Manikumar wrote: > Thanks for the KIP. > > +1 (binding). > > On Wed, Apr 17, 2024 at 7:50 PM

Re: [VOTE] KIP-932: Queues for Kafka

2024-05-06 Thread Jun Rao
Hi, Andrew, Thanks for the KIP. +1 Jun On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar wrote: > Thanks Andrew, > > +1 (binding) > > Edo > > On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole > wrote: > > > > Hi Andrew > > > > + 1 (Non-Binding) > > > > This will be great addition to Kafka > > > > O

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-06 Thread Jun Rao
ains because it adds support for share > groups. > This is needed to list share groups for the admin client and the > command-line > tools. > > Thanks, > Andrew > > > On 6 May 2024, at 19:26, Jun Rao wrote: > > > > Hi, Andrew, > > > > Removing

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-06 Thread Jun Rao
custom group types. > This all needs sorting out, I think. > > I propose to remove AdminClient.listGroups() from this KIP, and put > it in the administration KIP. > > Let me know what you think. > > Thanks, > Andrew > > > > On 6 May 2024, at 18:04, Jun Rao wrot

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-06 Thread Jun Rao
sp of > delivery, I’ll update the KIP with the final value. > > 164. KRaft only. All the RPCs are “broker” only. None of the code will > be merged until after 3.8 has branched. > > Thanks, > Andrew > > > On 4 May 2024, at 00:12, Jun Rao wrote: > > > > Hi,

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-03 Thread Jun Rao
ked the text so that the simple > assignor will take into account existing assignment information when > it has it, which would just minimise unnecessary churn of (b). > > 158. I’ve changed it to ReadShareGroupStateSummary. > > Thanks, > Andrew > > > > On

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-03 Thread Jun Rao
es. I see. Of course, I want the names as consistent and > understandable too. I suggest renaming > ReadShareGroupOffsetsState to ReadShareGroupStateSummary. > I haven’t changed the KIP yet, so let me know if that’s OK. > > Thanks, > Andrew > > > On 2 May 2024, at 22:18, Jun Rao wrote

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-02 Thread Jun Rao
y a leader change needs > to force a ShareSnapshot. I’ve added leaderEpoch to the ShareUpdate. > > 158. Although ReadShareGroupOffsetsState is a bit of a mouthful, > having “State” in the name makes it clear that this one the family of > inter-broker RPCs served by the share coordinator

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-01 Thread Jun Rao
143. Yes, that’s right. > > 147. The measurement is certainly from the point of view > of the client, but it’s driven by sending and receiving heartbeats > rather than whether the client triggered the rebalance itself. > The client decides when it enters and leaves reconciliation > of

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

2024-04-26 Thread Jun Rao
Hi, Abhijeet, Thanks for the KIP. +1 Jun On Thu, Apr 25, 2024 at 10:30 PM Abhijeet Kumar wrote: > Hi All, > > I would like to start the vote for KIP-1023 - Follower fetch from tiered > offset > > The KIP is here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-25 Thread Jun Rao
add this. Done. > > 149. There are several error codes for WriteShareGroupStateResponse: > > NOT_COORDINATOR - This is not the share coordinator you’re looking for. > COORDINATOR_NOT_AVAILABLE - The SC can’t > COORDINATOR_LOAD_IN_PROGRESS - The SC is replaying the topic. > GROUP_ID_NOT_FOUND - The

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-04-24 Thread Jun Rao
Hi, Abhijeet, Thanks for the updated KIP. It looks good to me. Jun On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar wrote: > Hi Jun, > > Please find my comments inline. > > > On Thu, Apr 18, 2024 at 3:26 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > >

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-24 Thread Jun Rao
up state accordingly. > > > I have made one more change to the KIP. The SharePartitionAssignor > interface has been renamed to > o.a.k.coordinator.group.assignor.ShareGroupPartitionAssignor and it > now extends the PartitionAssignor interface. It’s essentially a marker > of which parti

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-23 Thread Jun Rao
rbroker RPC latency of the read, they’re likely to be very close. > > Later, for a cluster which is using a custom persister, the > share-coordinator > metrics would likely not be reported, and the persister would have its own > metrics. > > 137.3: Correct. Fixed. > > 137

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-22 Thread Jun Rao
roup level, the group > configuration > will only be permitted to name an assignor which is in this list. For now, > there > is no group configuration for assignor, so all groups get the one and only > assignor in the list. > > 135. It’s the number of threads per broker. For a c

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-19 Thread Jun Rao
ing the configuration to enable the feature in the mailing > list with > David Jacot also, so I anticipate a bit of change in this area still. > > Thanks, > Andrew > > > On 15 Apr 2024, at 23:34, Jun Rao wrote: > > > > Hi, Andrew, > > > > Thanks

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-04-17 Thread Jun Rao
> I have tried to clarify this in my reply to Jun. > > > > > > > The follower needs to build the local data starting from the offset > > > > Earliest-Pending-Upload-Offset. Hence it needs the offset and the > > > > corresponding leader-epoch. > > > >

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-15 Thread Jun Rao
It makes evolution and extension of the API much easier. Also, it matches > the precedent set by ListConsumerGroupOffsetSpec. > > 113. ListConsumerGroupsResults.errors() is the same. I think you just have > to look in the exception details and the same pattern is being followed > her

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-04-12 Thread Jun Rao
Hi, Justine, Thanks for the KIP. +1 Jun On Wed, Apr 10, 2024 at 9:13 AM José Armando García Sancio wrote: > Hi Justine, > > +1 (binding) > > Thanks for the improvement. > -- > -José >

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-12 Thread Jun Rao
Justine Olshan wrote: > Updated. :) > Thanks for the reviews > > Justine > > On Thu, Apr 11, 2024 at 11:01 AM Jun Rao wrote: > > > Hi, Justine, > > > > Thanks for the updated KIP. > > > > Perhaps it's better to name the new config >

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-11 Thread Jun Rao
l config was never actually approved in any KIP. But we can say > it is deprecated. > I can change the config name. > > Justine > > On Thu, Apr 11, 2024 at 8:52 AM Jun Rao wrote: > > > Hi, Justine, > > > > Thanks for the updated KIP. > > > > Wo

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-11 Thread Jun Rao
KIP-1014 has been inactive recently. I can > update my KIP and mention this change on that discussion thread. > > Justine > > On Tue, Apr 9, 2024 at 9:01 AM Jun Rao wrote: > > > Hi, Justine, > > > > A single config makes sense to me too. We just need to reach

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-09 Thread Jun Rao
Hi, Andrew, Thanks for the reply. A few more comments. 41. 41.1 How does the partition leader obtain the group epoch to set WriteShareGroupStateRequest.StateEpoch? 41.2 What's the benefit of having the group coordinator initialize the state and the partition leader set the SPSO? It seems simpler

[jira] [Resolved] (KAFKA-16485) Fix broker metrics to follow kebab/hyphen case

2024-04-09 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-16485. - Fix Version/s: 3.8.0 Resolution: Fixed Merged the PR to trunk. > Fix broker metrics to fol

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-09 Thread Jun Rao
that makes sense, I will update the KIP. > > Justine > > On Mon, Apr 8, 2024 at 3:20 PM Jun Rao wrote: > > > Hi, Justine, > > > > Thanks for the updated KIP. > > > > One more question related to KIP-1014. It introduced a new > > config unst

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-08 Thread Jun Rao
David, we agreed > that > >> > transaction version is simpler and conveys more than just protocol > >> changes > >> > (flexible records for example) > >> > > >> > I will update the KIP as well as KIP-890 > >> > > >> > Thank

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

2024-04-04 Thread Jun Rao
Hi, Abhijeet, Thanks for the KIP. Left a few comments. 1. "A drawback of using the last-tiered-offset is that this new follower would possess only a limited number of locally stored segments. Should it ascend to the role of leader, there is a risk of needing to fetch these segments from the remot

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-03 Thread Jun Rao
Hi, Andrew, Thanks for the updated KIP. A few more comments. 40. It would be useful to document the flow for serving the initial ShareGroupHeartbeatRequest. The coordinator needs to (1) write the ConsumerGroupMetadata record, the ConsumerGroupMemberMetadata record, the initial ShareGroupPartition

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-02 Thread Jun Rao
7;t need to be populated for TV 1. We don't > have the same need to retain this since it is written directly to the > transaction log in InitProducerId. It is only needed for KIP-890 part 2 / > TV 2. > > 22. We can do that. > > Justine > > On Tue, Apr 2, 2024 at 10:41 

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-02 Thread Jun Rao
s epoch but we explained in this KIP > how we can figure this out. > > Justine > > > > On Mon, Apr 1, 2024 at 3:56 PM Jun Rao wrote: > > > Hi, Justine, > > > > Thanks for the updated KIP. A couple of more comments. > > > > 20. Could we show the

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-04-01 Thread Jun Rao
do that. I can include this context in > the > > KIP. > > > > I'm happy to modify his 1 and 2 to 0 and 1. > > > > Justine > > > > On Thu, Mar 28, 2024 at 10:57 AM Jun Rao > wrote: > > > >> Hi, David, > >> >

Re: [VOTE] KIP-853: KRaft Controller Membership Changes

2024-03-29 Thread Jun Rao
Hi, Jose, Thanks for the KIP. +1 Jun On Fri, Mar 29, 2024 at 9:55 AM José Armando García Sancio wrote: > Hi all, > > I would like to call a vote to adopt KIP-853. > > KIP: https://cwiki.apache.org/confluence/x/nyH1D > Discussion thread: > https://lists.apache.org/thread/6o3sjvcb8dx1ozqfpltb7p0

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-28 Thread Jun Rao
Hi, Jose, Thanks for the explanation. Other than depending on KIP-1022 to be approved, the KIP looks good to me now. Jun On Thu, Mar 28, 2024 at 2:56 PM José Armando García Sancio wrote: > Hi Jun, > > See my comments below. > > On Thu, Mar 28, 2024 at 11:09 AM Jun Rao wro

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-28 Thread Jun Rao
le brokers reporting the same metric. For example, if they don't report the same value (e.g. because one broker is catching up), how does a user know which value is correct? Thanks, Jun On Thu, Mar 28, 2024 at 10:56 AM José Armando García Sancio wrote: > Hi Jun, > > On Thu, Mar

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-28 Thread Jun Rao
cepted. Another use > > case is the Queues KIP. I think that we should also use this new flag to > > gate it. > > > > Best, > > David > > > > On Thu, Mar 28, 2024 at 1:14 AM Jun Rao > wrote: > > > >> Hi, Justine, > >> > >> T

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-28 Thread Jun Rao
> > On Wed, Mar 27, 2024 at 2:26 PM Jun Rao wrote: > > 55.1 How does the broker and non-leader controller know the pending > voters? > > They are in the log. Pending voter sets are VotersRecords between the > HWM and the LEO. The leader will make sure that there is at

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-27 Thread Jun Rao
he IV strings, but I wasn't sure if there would > be some disagreement with the decision. Not sure if that breaks > compatibility etc. Happy to hear everyone's thoughts. > > Justine > > On Wed, Mar 27, 2024 at 3:36 PM Jun Rao wrote: > > > Hi, Justine, > > >

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-27 Thread Jun Rao
Hi, Justine, Thanks for the reply. Having "kafka-feature dependencies" seems enough to me. We don't need to include the dependencies in the output of "kafka-feature describe". We only support "dependencies" in kafka-feature, not kafka-storage. We probably should do the same for "version-mapping"

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-27 Thread Jun Rao
Hi, Jose, Thanks for the reply. 55.1 How does the broker and non-leader controller know the pending voters? Jun On Wed, Mar 27, 2024 at 1:03 PM José Armando García Sancio wrote: > Hi Jun, > > Thanks for the feedback. See my comments below. > > On Mon, Mar 25, 2024 at 2:21 PM

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-27 Thread Jun Rao
nd finalized versions? > > Feature: transaction.protocol.version SupportedMinVersion: 0 > SupportedMaxVersion: 2 FinalizedVersionLevel: 1 Epoch: 3 Dependencies: > metadata.version=4 > > On Wed, Mar 27, 2024 at 11:14 AM Jun Rao wrote: > > > Hi, Justine, > > > > Ye

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-27 Thread Jun Rao
version dependencies) > > Justine > > On Wed, Mar 27, 2024 at 10:28 AM Jun Rao wrote: > > > Hi, Colin, > > > > Thanks for the comments. It's fine if we want to keep the --metadata flag > > if it's useful. Then we should add the same flag f

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-27 Thread Jun Rao
Hi, Colin, Thanks for the comments. It's fine if we want to keep the --metadata flag if it's useful. Then we should add the same flag for kafka-storage for consistency, right? Hi, Justine, Thanks for the reply. How will a user know about the dependencies among features? Should we expose them in

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-26 Thread Jun Rao
o the storage tool as it is > not necessary. The reason it is not removed is purely for backwards > compatibility. Colin had strong feelings about not removing any flags. > > Justine > > On Mon, Mar 25, 2024 at 5:01 PM Jun Rao wrote: > > > Hi, Justine, > > > > T

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-25 Thread Jun Rao
Hi, Justine, Thanks for the updated KIP. A few more comments. 10. Could we describe what RPCs group.coordinator.version controls? 12. --metadata METADATA is not removed from kafka-features. Do we have a justification for this? If so, should we add that to kafka-storage to be consistent? 14. The

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-25 Thread Jun Rao
and tags. 57. Could we remove --release-version 3.8 in the upgrade example? Jun On Mon, Mar 25, 2024 at 11:54 AM José Armando García Sancio wrote: > Hi Jun, > > See my comments below. > > On Fri, Mar 22, 2024 at 1:30 PM Jun Rao wrote: > > 54. Admin.addMetadataVoter: I

[jira] [Resolved] (KAFKA-15950) Serialize broker heartbeat requests

2024-03-25 Thread Jun Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-15950. - Fix Version/s: 3.8.0 Resolution: Fixed merged the PR to trunk. > Serialize broker heartb

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-22 Thread Jun Rao
Hi, Jose, Thanks for the reply. A few more comments. 54. Admin.addMetadataVoter: It seems that Endpoint shouldn't include securityProtocol since it's not in DescribeQuorumResponse. 55. Metrics: 55.1 It would be useful to be clear whether they are reported by the controller leader, all controller

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-14 Thread Jun Rao
part doesn't read well. 53.5 the the current extra the 53.6 it will be discover discover => discovered 53.7 it would beneficial beneficial => be beneficial Jun On Mon, Mar 11, 2024 at 10:39 AM José Armando García Sancio wrote: > Hi Jun > > Thanks for the feedback. Se

Re: [VOTE] KIP-956: Tiered Storage Quotas

2024-03-08 Thread Jun Rao
Hi, Abhijeet, Thanks for the KIP. +1 Jun On Fri, Mar 8, 2024 at 3:44 AM Abhijeet Kumar wrote: > Hi All, > > I would like to start the vote for KIP-956 - Tiered Storage Quotas > > The KIP is here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas > > Regards. >

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-06 Thread Jun Rao
or the receiver in Vote and BeginQuorumEpoch requests, but not in EndQuorumEpoch, Fetch and FetchSnapshot. Could you explain how they are used? Jun On Wed, Mar 6, 2024 at 8:53 AM José Armando García Sancio wrote: > Hi Jun, > > See my comments below. > > On Tue, Mar 5, 2024 at 2:57 PM Ju

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-05 Thread Jun Rao
uld you explain whether it is truly needed? Before this KIP, KRaft already supports replacing a disk on the voter node, right? Jun On Mon, Mar 4, 2024 at 2:55 PM José Armando García Sancio wrote: > Hi Jun, > > Thanks for the feedback. See my comments below. > > On Fri, Mar 1, 2024 at 1

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

2024-03-04 Thread Jun Rao
le-time-max > metrics we have for Kafak Consumers? > The Avg and Max are computed over the (sliding) window as defined by the > configuration metrics.sample.window.ms on the server. > > (Also, I will update the config and metric names to be consistent) > > Regards. > > On Thu

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-01 Thread Jun Rao
s are > supported. If any version is not supported the tool will throw an error. We > can also issue a warning if the given command (if supported by the cluster) > will downgrade a feature. One option is also to require a yes/no prompt or > flag to allow downgrades. The downgrade tool would

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-03-01 Thread Jun Rao
etadata.log.dir? Jun On Fri, Mar 1, 2024 at 10:51 AM José Armando García Sancio wrote: > Hi Jun, > > Thanks for the feedback. See my comments below. > > On Tue, Feb 27, 2024 at 11:27 AM Jun Rao wrote: > > 30. Who controls RPCs like Fetch, FetchSnapshot, DescribeQuorum RPC

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-02-28 Thread Jun Rao
Hi, Justine, Thanks for the KIP. A few comments below. 10. Currently, MV controls records, inter-broker RPCs and code logic. With more features, would each of those be controlled by a separate feature or multiple features. For example, is the new transaction record format controlled only by MV wi

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2024-02-28 Thread Jun Rao
imeout could be confusing, and a large enough > timeout could still produce the undesirable effects for the cloud > operations (so we kind of get worst of both options -- we don't provide > guarantees and still have impact on operations). > > -Artem > > On Fri, Feb 23, 2024 a

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

2024-02-28 Thread Jun Rao
og.remote:type=RemoteLogManager, name=RemoteCopyThrottleTime -> The > duration of time required at a given moment to bring the observed remote > copy rate within the allowed limit, by preventing further copies. > > Regards. > > On Wed, Feb 28, 2024 at 12:28 AM Jun Rao wrote: > > &g

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-02-27 Thread Jun Rao
ApiKeys or kraft.version feature in ApiVersions response for deciding whether to send AddVoter requests? Jun On Tue, Feb 27, 2024 at 8:20 AM José Armando García Sancio wrote: > Hi Jun, > > Thanks for the comments. See my comments below. > > On Mon, Feb 26, 2024 at 4:48 PM Jun Rao wr

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

2024-02-27 Thread Jun Rao
I will update the KIP based on the > discussion) > > Regards, > Abhijeet. > > On Tue, Feb 27, 2024 at 2:49 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Sorry for the late reply. It seems that you haven't updated the KIP based > > on the discussion? On

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-02-26 Thread Jun Rao
Fri, Feb 23, 2024 at 1:39 PM Jun Rao wrote: > > 15.1 "In this configuration, the local replica needs to use > > CONTROLLER_PLAINTEXT because that is what is supported by all of the > > voters." > >Hmm, my understanding is that what listener to use can be m

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

2024-02-26 Thread Jun Rao
uld you explain the usage of this additional quota? Thanks, Jun On Mon, Feb 12, 2024 at 11:08 AM Abhijeet Kumar wrote: > Comments inline > > On Wed, Dec 6, 2023 at 1:12 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the KIP. A few comments. > > >

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-02-23 Thread Jun Rao
Hi, Jose, Thanks for the reply. 15. VotersRecord: Ok. I got the need for including the listener name in the endpoint. Currently, controller.quorum.voters can only specify one endpoint per voter. So, we can only support one listener. It's fine if we want to extend that in this KIP. 15.1 "In this c

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2024-02-23 Thread Jun Rao
r features that > the customers may want to take advantage of. > > If we enabled keepPreparedTxn=true even without 2PC, then we could enable > case 3b without the need to fall back to reflection, so we could get rid of > reflection-based logic and just have a single implem

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2024-02-21 Thread Jun Rao
t uses reflection, if it moves to use public APIs > provided by KIP-939 it'll break. > > > 32. Ok. That's the kafka metric. In that case, the metric name has a > group and a name. There is no type and no package name. > > Is this a suggestion to change or confirmation t

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

2024-02-21 Thread Jun Rao
Hi, Mickael, Thanks for the updated KIP. There is a typo. The KIP says that it adds a new option compression.snappy.level, but later says that Snappy is excluded. Otherwise, the changes look good to me. Jun On Wed, Feb 7, 2024 at 6:40 AM Mickael Maison wrote: > Hi Divij, > > Thanks for brin

Re: [VOTE] KIP-1019: Expose method to determine Metric Measurability

2024-02-21 Thread Jun Rao
Hi, Apoorv, Thanks for the KIP. +1 Jun On Mon, Feb 19, 2024 at 2:32 PM Apoorv Mittal wrote: > Hi, > I’d like to start the voting for KIP-1019: Expose method to determine > Metric Measurability. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metri

Re: [Discuss] KIP-1019: Expose method to determine Metric Measurability

2024-02-21 Thread Jun Rao
ka/common/metrics/KafkaMetric.java#L36 > > > for > Measurable and Gauge hence went ahead only exposing what is currently > needed. > > Regards, > Apoorv Mittal > +44 7721681581 > > > On Tue, Feb 20, 2024 at 10:58 PM Jun Rao wrote: > > > Hi, Apoorv, >

  1   2   3   4   5   6   7   8   9   10   >