Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

ffset-syncs.enabled=false > > BTW, I'm +1 to this KIP, and I noticed my previous comments are related to > code. Hence, please feel free to open votes. We can have discussion about > the code later. > > > > Omnia Ibrahim 於 2024年5月21日 週二 下午9:10寫道： >

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

g records to offset-syncs topic won't be created if > emit.offset-syncs.enabled=false > > 3) Should we simplify the SourceRecord if emit.offset-syncs.enabled=false? > Maybe that can get a bit performance improvement. > > Best, > Chia-Ping > > On 2024/04/08 10:03:50 Omnia

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

because we are >>>> running out of memory. We should persist that data to disk and >> consider >>>> the disk as the source of truth for state. >>>> >>>> Claude >>>> >>>> >>>> On Wed, May 15, 2024 at 7:42 

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

t;> from the cache if we didn’t receive heartbeat will be more simpler and >> step >>> on right direction to improve idempotent logic and maybe try to make PID >>> get reused between session which will implement a real idempotent >> producer >>> instead of

Re: [VOTE] KIP-932: Queues for Kafka

Thanks for the KIP Andrew +1 none binding from me > On 16 May 2024, at 14:23, David Jacot wrote: > > Hi Andrew, > > Thanks for the KIP! This is really exciting! +1 (binding) from me. > > One note regarding the partition assignor interface changes that you > proposed, it would be great to get

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

Also in the rejection alternatives you listed an approved KIP which is a bit confusing can you move this to motivations instead > On 15 May 2024, at 14:35, Claude Warren wrote: > > This is a proposal that should solve the OOM problem on the servers without > some of the other proposed KIPs

Re: [DISCUSS] KIP-1044: A proposal to change idempotent producer -- server implementation

Hi Claude Thanks for raising this KIP. It is an interesting idea. I had a quick review for the KIP and I have few notes 10. > The issue is that the number of PIDs that need to be tracked has exploded and > has resulted in OOM failures that cause the entire cluster to crash. There > are

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hi Justine Thanks for the feedback > So consider a case where there is a storm for a given principal. We could > have a large mass of short lived producers in addition to some > "well-behaved" ones. My understanding is that if the "well-behaved" one > doesn't produce as frequently ie less than

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hi Igor, thanks for the feedback and sorry for the late response. > 10 Given the goal is to prevent OOMs, do we also need to > limit the number of KafkaPrincipals in use? None of the Kafka quotas ever limited number of KafkaPrincipals and I don’t really think this is the issue as you just need

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

/nxp395zmvc0s8r4ohg91kdb19dxsbxlt if you have time. Thanks Omnia > On 6 Jun 2023, at 15:04, Omnia Ibrahim wrote: > > Thanks, Luke for the feedback > >> 1. how do we store value in bloom filter? It's unclear from this KIP that >> what we store inside bloom filter, and how

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

.001 instead of being hardcoded. I think the KIP is now in better shape to ask others for a review. Omnia > On 24 Apr 2024, at 14:48, Omnia Ibrahim wrote: > > Hi Glaude sorry that it took me a while to respond. I finally had time to > look into your implementation here > https:/

[jira] [Created] (KAFKA-16638) Align the naming convention for config and default variables in *Config classes

Omnia Ibrahim created KAFKA-16638: - Summary: Align the naming convention for config and default variables in *Config classes Key: KAFKA-16638 URL: https://issues.apache.org/jira/browse/KAFKA-16638

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

Thanks for the KIP. +1 non-binding from me > On 26 Apr 2024, at 06:29, Abhijeet Kumar wrote: > > Hi All, > > I would like to start the vote for KIP-1023 - Follower fetch from tiered > offset > > The KIP is here: >

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

Thanks Igor, I’ll conclude the vote with 3 binding votes from Mickael Maison, Chris Egerton and Igor Soarez. Thanks everyone > On 24 Apr 2024, at 15:11, Igor Soarez wrote: > > Hi Omnia, > > Thanks for your answers, and I see you've updated the KIP so thanks for the > changes too. > > +1

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

ain and is added to the newest bloom filter and the >> associated metric is updated. >> >> I believe at this point the metric is incorrect, the PID has been counted >> 2x, when it has been in use for the entire time. >> >> The "track" method that I added solves

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

I updated the KIP as well to briefly explain how offset.lag.max would help latency. Please let me know if the KIP now looks better? > On 24 Apr 2024, at 11:49, Omnia Ibrahim wrote: > > Thanks Igor for the suggestions. I updated the KIP with some of them. > >> s/recored

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

ay that > MirrorCheckpointConnector is disabled but MirrorSourceConnector is not? > Could you also clarify where the additional cost comes from? > > 14. This is probably more ignorance of mine: it doesn't seem obvious in > the KIP how increasing offset.lag.max to INT_MAX helps red

Re: [DISCUSS] KIP-1039: Disable automatic topic creation for MirrorMaker2 consumers

Hi Aaron, You mentioned that there is no public interface changes however changing the default value of a config should be considered as a public change. You can check other KIP where we changed the default config value for a reference. Can you please list what is the impact of changing the

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi Alieh, Thanks for the KIP! I have couple of comments - You mentioned in the KIP motivation, > Another example for which a production exception handler could be useful is > if a user tries to write into a non-existing topic, which returns a retryable > error code; with infinite retries, the

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

;> >> +1 (binding), thanks for the KIP! >> >> Mickael >> >> On Fri, Apr 12, 2024 at 9:01 AM Omnia Ibrahim >> wrote: >>> >>> Hi everyone, I would like to start a voting thread for KIP-1031: Control >> offset translation in MirrorSourc

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Hi Ivan, Thanks for the KIP this is a very nice feature to have. +1(non-binding) Omnia > On 15 Apr 2024, at 14:33, Andrew Schofield wrote: > > Thanks for the KIP > > +1 (non-binding) > > Andrew > >> On 15 Apr 2024, at 14:16, Chris Egerton wrote: >> >> Hi Ivan, >> >> Thanks for the KIP.

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hi Claude, Thanks for the implementation of the LayeredBloomFilter in apache commons. > Define a new configuration option "producer.id.quota.window.count" as > the number of windows active in window.size.seconds. What is the different between “producer.id.quota.window.count” and

[VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

Hi everyone, I would like to start a voting thread for KIP-1031: Control offset translation in MirrorSourceConnector https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector For comments or feedback please check the discussion thread here

Re: [DISCUSS] KIP-932: Queues for Kafka

er of topic-partitions being consumed in the group, and the number of >>> in-flight records, but not >>> the number of members. We should be talking about no more than kilobytes >>> per topic-partition. >>> * For the share-partition leader, the memory consump

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

c/org/apache/kafka/connect/connector/Connector.html#validate(java.util.Map) > > Cheers, > > Chris > > On Thu, Apr 4, 2024 at 8:07 PM Omnia Ibrahim > wrote: > >> Thanks Chris for the feedback >>> 1. It'd be nice to mention that increasing the max offset lag to IN

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

d, Apr 3, 2024, 20:41 Luke Chen wrote: > >> Hi Omnia, >> >> Thanks for the KIP! >> It LGTM! >> But I'm not an expert of MM2, it would be good to see if there is any other >> comment from MM2 experts. >> >> Thanks. >> Luke >> >

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Andrew, Thanks for the KIP it is definitely an interesting read. I have few questions As the KIP proposing extending `AdminClient.incrementalAlterConfigs` to add an explicit `group.type` what would this means for DR feature in MM2 offering? Right now MM2 sync consumer group offsets from

Re: [VOTE] KIP-981: Manage Connect topics with custom implementation of Admin

an one-by-one updating the > Java APIs to use this new plugin. > This KIP appearing as a follow-up to KIP-787 is evidence that the > problem is more general than the proposed solution. > > At this time I'm -1 for this proposal. I'm happy to discuss this more > in the DISCUSS th

[DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

Hi everyone, I would like to start a discussion thread for KIP-1031: Control offset translation in MirrorSourceConnector https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector Thanks Omnia

Re: [VOTE] KIP-939: Support Participation in 2PC

I had a look at the discussion thread and the KIP looks exciting. +1 non-binding Best Omnia On 1 Dec 2023, at 19:06, Artem Livshits wrote: Hello, This is a voting thread for https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC . The KIP proposes extending

[VOTE] KIP-981: Manage Connect topics with custom implementation of Admin

Hi everyone, I would like to start the vote on KIP-981: Manage Connect topics with custom implementation of Admin https://cwiki.apache.org/confluence/display/KAFKA/KIP-981%3A+Manage+Connect+topics+with+custom+implementation+of+Admin Thanks Omnia

[jira] [Created] (KAFKA-16254) Allow MM2 to fully disable offset sync feature

Omnia Ibrahim created KAFKA-16254: - Summary: Allow MM2 to fully disable offset sync feature Key: KAFKA-16254 URL: https://issues.apache.org/jira/browse/KAFKA-16254 Project: Kafka Issue Type

Re: [DISCUSS] KIP-981: Manage Connect topics with custom implementation of Admin

Hi everyone, I'm still keen on implementing this. I'll start a vote if I don't hear back in the next few days Thanks Omnia On Mon, Jan 15, 2024 at 8:12 PM Omnia Ibrahim wrote: > Hi It has been a while! but can I have a feedback on this. It would be > nice to unify this between MM2 and C

[jira] [Resolved] (KAFKA-16162) New created topics are unavailable after upgrading to 3.7

[ https://issues.apache.org/jira/browse/KAFKA-16162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim resolved KAFKA-16162. --- Fix Version/s: 3.7.0 Resolution: Fixed > New created topics are unavailable af

[jira] [Resolved] (KAFKA-14616) Topic recreation with offline broker causes permanent URPs

[ https://issues.apache.org/jira/browse/KAFKA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim resolved KAFKA-14616. --- Fix Version/s: 3.7.0 Assignee: Colin McCabe Resolution: Fixed > To

Re: [DISCUSS] KIP-1016 Make MM2 heartbeats topic name configurable

what was mentioned in the rejected alternatives in KIP-690 isn’t valid anymore. > On 2 Feb 2024, at 15:23, Omnia Ibrahim wrote: > > Hi > Thanks for the KIP. I don’t know if you had a look before into the rejection > alternatives in KIP-690 > https://cwiki.apache.org/conflu

Re: [DISCUSS] KIP-1016 Make MM2 heartbeats topic name configurable

Hi Thanks for the KIP. I don’t know if you had a look before into the rejection alternatives in KIP-690

Re: [VOTE] 3.7.0 RC2

Hi Stan and Gaurav, Just to clarify some points mentioned here before KAFKA-14616: I raised a year ago so it's not related to JBOD work. It is rather a blocker bug for KRAFT in general. The PR from Colin should fix this. Am not sure if it is a blocker for 3.7 per-say as it was a major bug

Re: [DISCUSS] KIP-981: Manage Connect topics with custom implementation of Admin

Hi It has been a while! but can I have a feedback on this. It would be nice to unify this between MM2 and Connect as well. Thanks On Thu, Oct 19, 2023 at 3:14 PM Omnia Ibrahim wrote: > Hi, any thoughts on this kip? > > Thanks > > On Tue, Sep 19, 2023 at 6:04 PM Omnia Ibrahim &

[jira] [Resolved] (KAFKA-15858) Broker stays fenced until all assignments are correct

[ https://issues.apache.org/jira/browse/KAFKA-15858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim resolved KAFKA-15858. --- Resolution: Won't Fix `BrokerHeartbeatManager.calculateNextBrokerState` already keeps

[jira] [Created] (KAFKA-15895) Move DynamicBrokerConfig to server module

Omnia Ibrahim created KAFKA-15895: - Summary: Move DynamicBrokerConfig to server module Key: KAFKA-15895 URL: https://issues.apache.org/jira/browse/KAFKA-15895 Project: Kafka Issue Type: Sub

Re: [DISCUSS] KIP-981: Manage Connect topics with custom implementation of Admin

Hi, any thoughts on this kip? Thanks On Tue, Sep 19, 2023 at 6:04 PM Omnia Ibrahim wrote: > Hi everyone, > I want to start the discussion of the KIP-981 to extend Connect to use > org.apache.kafka.clients.admin.ForwardingAdminClient instead of > KafkaAdminClient > https://c

[DISCUSS] KIP-981: Manage Connect topics with custom implementation of Admin

Hi everyone, I want to start the discussion of the KIP-981 to extend Connect to use org.apache.kafka.clients.admin.ForwardingAdminClient instead of KafkaAdminClient https://cwiki.apache.org/confluence/display/KAFKA/KIP-981%3A+Manage+Connect+topics+with+custom+implementation+of+Admin Thanks for

[jira] [Created] (KAFKA-15478) Update connect to use ForwardingAdmin

Omnia Ibrahim created KAFKA-15478: - Summary: Update connect to use ForwardingAdmin Key: KAFKA-15478 URL: https://issues.apache.org/jira/browse/KAFKA-15478 Project: Kafka Issue Type: New

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

> time (1 hr). It also creates a TimestampedBloomFilter to track the time. > If we need an explicit mechanism to remove filters from the LayerManager we > can probably add one. > > I hope this answers your questions. > > I am currently working on getting layered Bloom filte

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Thanks, Mickael. Am closing the voting now with 3 binding votes (from Chris, Greg and Mickale) and 1 non-binding (from Federico). On Wed, Aug 2, 2023 at 2:52 PM Mickael Maison wrote: > +1 (binding) > > Thanks for the KIP > > On Tue, Aug 1, 2023 at 1:26 PM Omnia Ibrahim > wro

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

to backport this to 3.1 - 3.5, so that MM2 > users can upgrade in minor version increments after those patch > releases go out. > > I'm +1 (binding). > > Thanks, > Greg > > On Mon, Jul 24, 2023 at 7:21 AM Omnia Ibrahim > wrote: > > > > Hi Chris, I updated the

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

changes in KIPs, I think as long as we can describe the > changes we're proposing in the "Public Interfaces" section, that should be > enough for this KIP > > Cheers, > > Chris > > On Mon, Jul 24, 2023 at 2:04 AM Federico Valeri > wrote: > > > +1 (no

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

a vote thread soon. > > > > Cheers, > > > > Chris > > > > On Fri, Jul 21, 2023 at 10:08 AM Omnia Ibrahim > > wrote: > > > > > Hi Chris and Federico, > > > thinking about I think Chris's concern is valid and the big

[VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hi everyone, I would like to open a vote for KIP-949. The proposal is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy .

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

> > would have to be ported over to the LegacyReplicationPolicy class as > > well. > > > > > > Perhaps I'm missing something; are there other benefits of introducing > a > > > separate replication policy class? > > > > > > Cheers, >

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

n find an easy migration path for users who are relying > > on it, then I'd be happy to deprecate and schedule for removal. > > > > Cheers, > > > > Chris > > > > On Tue, Jul 18, 2023 at 12:54 PM Omnia Ibrahim > > wrote: > > > > > Hi Federic

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

ving the same behavior for internal and > non-internal topics, then it should be fine, and we won't need to keep > an additional configuration around. Wdyt? > > Cheers > Fede > > On Fri, Jul 14, 2023 at 1:51 PM Omnia Ibrahim > wrote: > > > > Hi Chris, I added a

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

serts. > > > > In addition, the number of Layered Bloom filter instances can be reduced > > by hashing the Kafka Principal and the ID together into the Bloom filter > to > > look for. > > > > [1] https://github.com/Claudenw/BloomFilters/tree/layered &

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

(including > binding and non-binding voters) before implementing it. > > Cheers, > > Chris > > On Thu, Jul 13, 2023 at 7:13 AM Omnia Ibrahim > wrote: > >> Hi Chris, >> The implementation should be very small so backporting this to 3.1 and 3.2 >> wou

[Vote] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hi Everyone! I would like to start the vote on KIP-949 details is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy Thanks Omnia

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

ning parallel forks. Obviously this shouldn't be taken as a blanket > precedent but in this case it seems like the benefits may outweigh the > costs. What are your thoughts? > > Cheers, > > Chris > > On Wed, Jul 12, 2023 at 9:05 AM Omnia Ibrahim > wrote: > > > Hi Chris,

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

s don't necessarily have to be codified in the KIP, but they're > worth taking into account when considering how to design any functional > changes in order to better try to gauge how well this could go for our > users. > > [1] - https://kafka.apache.org/documentation.html#upgrade &

[DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hi everyone, I want to start the discussion of the KIP-949. The proposal is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

eads occupation which could make other background tasks to > delay > 3. Why the default producer.id.quota.window.size.seconds is 1 hour? > * Unlike other quota types (1 second) > > Thanks, > > 2023年6月6日(火) 23:55 Omnia Ibrahim : > > > Hi everyone, > > I want to start the d

[DISCUSS] KIP-936: Throttle number of active PIDs

Hi everyone, I want to start the discussion of the KIP-936 to throttle the number of active PIDs per KafkaPrincipal. The proposal is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs

[jira] [Created] (KAFKA-15063) Throttle number of active PIDs

Omnia Ibrahim created KAFKA-15063: - Summary: Throttle number of active PIDs Key: KAFKA-15063 URL: https://issues.apache.org/jira/browse/KAFKA-15063 Project: Kafka Issue Type: New Feature

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

indow.num is unclear to me? > "The number of samples to retain in memory for alter producer id quotas" > What's the "samples" mean here? Do you mean we only track the top 11 > kafkaPrinciple usage each window? > > Finally, I think this KIP is good to have an offic

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

new Kafka Principal? Is that simply not covered by > throttling? > > Thanks, > Justine > > On Wed, May 31, 2023 at 10:08 AM Omnia Ibrahim > wrote: > >> Hi Justine and Luke, >> >> I started a KIP draft here >> https://cwiki.apache.org/confluence/display/KAFKA/KI

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

Omnia Ibrahim wrote: > > *Hi Justine, * > > *My initial thought of throttling the initProducerId was to get ripped off > the problem at the source (which creates too many PIDs per client) and fail > faster but if having this on the produce request level is easier this > should b

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

r throttling the produce > request instead? It may be hard to get any metrics from the transaction > coordinator where the initProducerId request is handled. > > Justine *On Tue, Feb 14, 2023 at 9:29 AM Omnia Ibrahim > wrote:* > Hey Justine, > > If I understand your message corr

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

Hi Luke and Justine, Are there any thoughts or updates on this? I would love to help with this as we are hitting this more frequently now. best, On Mon, Oct 31, 2022 at 6:15 PM Omnia Ibrahim wrote: > Hi Luke and Justine, > >> For (3), you said: >> > - I have some conc

[jira] [Created] (KAFKA-14616) Topic recreation with offline broker causes permanent URPs

Omnia Ibrahim created KAFKA-14616: - Summary: Topic recreation with offline broker causes permanent URPs Key: KAFKA-14616 URL: https://issues.apache.org/jira/browse/KAFKA-14616 Project: Kafka

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

> *similar to what we have with client quota? This should reduce the > concern > > about throttling good clients, right? > > > > --> Yes, I mean KafkaPrinciple (sorry, I didn't make it clear) > > Yes, We were thinking about throttling by KafkaPrinciple. Client Id is >

[jira] [Created] (KAFKA-14344) Build EmbeddedKafkaCluster with common configs used for all clients

Omnia Ibrahim created KAFKA-14344: - Summary: Build EmbeddedKafkaCluster with common configs used for all clients Key: KAFKA-14344 URL: https://issues.apache.org/jira/browse/KAFKA-14344 Project: Kafka

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

Hi Luke & Justine, Thanks for looking into this issue, we have been experiencing this because of rouge clients as well. > 3. Having a limit to the number of active producer IDs (sort of like an LRU >cache) >-> The idea here is that if we hit a misconfigured client, we will expire >the older

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

Close the voting with 3 +1 binding and +1 non-binding. Thanks, everyone On Wed, Jul 6, 2022 at 8:16 PM David Jacot wrote: > Thanks for the KIP, Omnia! +1 (binding) > > On Wed, Jul 6, 2022 at 5:02 PM Omnia Ibrahim > wrote: > > > > Hi, > > Can we have one last bin

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

Hi, Can we have one last binding vote for this KIP, please? Omnia On Tue, Jun 28, 2022 at 3:36 PM Omnia Ibrahim wrote: > Thanks, Tom, I have updated the KIP to reflect these minor points. > > On Tue, Jun 28, 2022 at 10:58 AM Tom Bentley wrote: > >> Hi again Omnia, >

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

; it. Surely only one is needed? It would also be a good idea to show the > > > documentation that forward.admin.class will have. > > > > > > Kind regards, > > > > > > Tom > > > > > >> On Tue, 21 Jun 2022 at 17:06, Omn

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

for your patience on this. Kind regards, > > Tom > > On Tue, 28 Jun 2022 at 10:14, Omnia Ibrahim > wrote: > > > Hi, > > I did a small change to the KIP interface to address some discussions on > > the KIP. If all is good now can I get more votes on this, please? >

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

Hi, I did a small change to the KIP interface to address some discussions on the KIP. If all is good now can I get more votes on this, please? Thanks Omnia On Tue, Jun 21, 2022 at 10:34 AM Omnia Ibrahim wrote: > Hi, > Can I get more votes on this, please? > > Thanks > > On

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

he source cluster and creating them on the destination cluster? E.g. the > one should be done one way, but the other using a different mechanism? > > Kind regards, > > Tom > > > On Mon, 20 Jun 2022 at 17:13, Omnia Ibrahim > wrote: > > > Hi Chris, sorry for the la

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

gt; wrote: > > > > Hi Omnia, > > > > I think the approach you settled on is the best option, this will > > allow integrating MirrorMaker in more environments. > > > > +1 binding > > > > Thanks for the KIP (and your persistence!) > >

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

easier to > modify the MM2 codebase exclusively and not have to worry about touching > the Connect framework as well given the possibility for unintended > consequences for other connectors with the latter. Wondering if there's a > distinction on the feature front as well that mak

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

hould+allow+source+connectors+to+set+topic-specific+settings+for+new+topics > [2] - > > https://github.com/apache/kafka/blob/4c9eeef5b2dff9a4f0977fbc5ac7eaaf930d0d0e/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceTask.java > > Cheers, > > Chris > >

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

Hi, Can I get a vote on this, please? Thanks On Wed, May 25, 2022 at 11:15 PM Omnia Ibrahim wrote: > Hi, > I'd like to start a vote on KIP-787 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-787 > %3A+MM2+Interface+to+manage+Kafka+resources > <https://cwiki.apache.org/

[Vote] KIP-787 - MM2 Interface to manage Kafka resources

Hi, I'd like to start a vote on KIP-787 https://cwiki.apache.org/confluence/display/KAFKA/KIP-787 %3A+MM2+Interface+to+manage+Kafka+resources Thanks Omnia

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi everyone, If there's no major concern anymore, I'll start the voting process. On Fri, May 20, 2022 at 5:58 PM Omnia Ibrahim wrote: > Hi Colin, > > >Thanks for the clarification. I agree it's reasonable for people to want > to use their own implementations of Admin. And we coul

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

ve updated the KIP to introduce ForwardingAdmin as well. regards, Omnia On Mon, May 16, 2022 at 9:31 PM Colin McCabe wrote: > On Mon, May 16, 2022, at 10:24, Omnia Ibrahim wrote: > > Hi Colin, > > > > Thanks for your reply. > > > > This KIP doesn’t aim to s

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

a custom implementation. If you still feel this KIP should be closed but can understand its motivation I can close this one and create a new one. Thanks, Omnia On Fri, May 13, 2022 at 6:10 PM Colin McCabe wrote: > On Wed, May 11, 2022, at 15:07, Omnia Ibrahim wrote: > > Hi Colin, &g

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

I updated the KIP to reflect the options we have been discussing since Oct 2021 for people who didn't read the discussion thread. On Wed, May 11, 2022 at 11:07 PM Omnia Ibrahim wrote: > Hi Colin, > I don't mind the idea of MM2 users implementing the AdminClient interface. > However,

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

mary concern here. The KIP is trying to give MM2 more flexibility to allow organisations to integrate MM2 with their resource management system as they see fit without forcing them to disable most MM2 features. Hope this make sense and clear it up. On Wed, May 11, 2022 at 9:09 PM Colin McCabe wrote

Re: [DISCUSS] KIP-781: Allow MirrorMaker 2 to override the client configurations

Hi Dongjin, nice spot for this bug. I wonder if this really needs a KIP as it seems like a clear bug in MM2's interpretation for the config. If you're going to keep the KIP can you please add some details in the Public interface or Proposed changes to list which part of MM2 codebase will change to

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

. Please let me know what do you think. On Mon, Feb 7, 2022 at 3:26 PM Omnia Ibrahim wrote: > Hi Mickael. Thanks for the feedback. I address some of your points below. > > *> This seems to address a relatively advanced and specific use case* > The main point of the KIP is that MM2 is m

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

nce between partitionCounts and newPartitions? > Should we have separate methods for creating topics and partitions? > - Do we really need createCompactedTopic()? > - Instead of updateTopicConfigs() and updateAcls() should we use the > "alter" prefix to stay consistent with Admin? > &

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

> Well I'm convinced! Thanks for looking into it. > > Ryanne > > On Wed, Oct 27, 2021, 8:49 AM Omnia Ibrahim > wrote: > > > I checked the difference between the number of methods in the Admin > > interface and the number of methods MM2 invokes from Admin, and this

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources Kafka/KIPs

Hi Can I get a vote on this, please? Best On Mon, Nov 22, 2021 at 4:59 PM Omnia Ibrahim wrote: > Hi All, > > Just thought of bumping this voting thread again to see if we can form a > consensus around this. > > Thanks > > > On Tue, Nov 16, 2021 at 2:16 PM Omni

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources Kafka/KIPs

Hi All, Just thought of bumping this voting thread again to see if we can form a consensus around this. Thanks On Tue, Nov 16, 2021 at 2:16 PM Omnia Ibrahim wrote: > Hi, > I'd like to start a vote on KIP-787 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-787%3A+MM2+

Re: Track topic deletion state without ZK

best, > Colin > > On Tue, Nov 2, 2021, at 09:24, Omnia Ibrahim wrote: > > Hi Colin, thanks for your response. > > Regards your point that the topic gets deleted immediately, I got that we > > do this if the cluster is healthy. > > However, if there's a

[Vote] KIP-787 - MM2 Interface to manage Kafka resources Kafka/KIPs

Hi, I'd like to start a vote on KIP-787 https://cwiki.apache.org/confluence/display/KAFKA/KIP-787%3A+MM2+Interface+to+manage+Kafka+resources Thanks Omnia

Re: Track topic deletion state without ZK

: > Hi Omnia, > > It is not necessary to know which topics are marked for deletion in when > in KRaft mode, because topic deletion happens immediately. > > best, > Colin > > On Thu, Oct 28, 2021, at 06:57, Omnia Ibrahim wrote: > > Hi, > > > > Kafka topi

[jira] [Created] (KAFKA-13416) Add topic ids to any metrics that has topic and partition tags

Omnia Ibrahim created KAFKA-13416: - Summary: Add topic ids to any metrics that has topic and partition tags Key: KAFKA-13416 URL: https://issues.apache.org/jira/browse/KAFKA-13416 Project: Kafka

[jira] [Created] (KAFKA-13415) Track topic deletion state without ZK

Omnia Ibrahim created KAFKA-13415: - Summary: Track topic deletion state without ZK Key: KAFKA-13415 URL: https://issues.apache.org/jira/browse/KAFKA-13415 Project: Kafka Issue Type

Track topic deletion state without ZK

Hi, Kafka topicCommand used to report which topic is marked for deletion by checking the znode on zookeeper; this feature has been deprecated without replacement as part of KAFKA-12596 Remove deprecated --zookeeper in topicCommands . Also as far

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

could be done for Admin as well. I agree your proposal is > more ergonomic, but maybe it's okay to have a little friction in such > fringe use-cases. > > Ryanne > > > On Tue, Oct 26, 2021, 6:23 AM Omnia Ibrahim > wrote: > > > Hey Ryanne, Thanks fo the quick feedba

  1   2   >