Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

@Gwen, can you please have a second look at the proposal and my explanation for why this approach is better for MM2? On Thu, Jun 24, 2021 at 5:55 PM Mickael Maison wrote: > Hi Omnia, > > I think the current proposal makes sense. Thanks for driving this feature. > > On Mon, Jun 21, 2021 at 11:51

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Omnia, I think the current proposal makes sense. Thanks for driving this feature. On Mon, Jun 21, 2021 at 11:51 PM Ryanne Dolan wrote: > > Omnia, I agree with you that allowing users to specify the whole topic name > via configuration is likely to create problems. MM2 must distinguish betwee

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Omnia, I agree with you that allowing users to specify the whole topic name via configuration is likely to create problems. MM2 must distinguish between internal topics from different clusters, and pushing that complexity into configuration sounds really complicated. I like the ReplicationPolicy a

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Any thoughts on this KIP? On Thu, Jun 17, 2021 at 1:38 PM Omnia Ibrahim wrote: > Another reason why I think adding a configuration for each internal topic is > not a good solution is how MM2 is naming these topics at the moment. > Right now MM2 sets the name of the offset-syncs topic to > mm2

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Another reason why I think adding a configuration for each internal topic is not a good solution is how MM2 is naming these topics at the moment. Right now MM2 sets the name of the offset-syncs topic to mm2-offset-syncs..internal and for checkpoints is .checkpoints.internal so the name has a patter

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi folks, let me try to clarify some of your concerns and questions. Mickael: Have you considered making names changeable via configurations? > Gwen: may be missing something, but we are looking at 3 new configs (one > for each topic). And this rejected alternative is basically identical to > wha

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hey Gwen, Thanks for having a look into the KIP, regard your question > is there a concrete case that shows why configuration is more complicated > than implementing a plugin? > The case I had in mind is where an organisation with medium-to-large size have multiple teams running their own Kafka c

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hey, sorry for arriving late, but I have a question about the rejected alternative involving configuration. I may be missing something, but we are looking at 3 new configs (one for each topic). And this rejected alternative is basically identical to what Connect already does (you can choose names

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Ryanne, Can you vote for it here https://www.mail-archive.com/dev@kafka.apache.org/msg113575.html as well, please? On Fri, Apr 30, 2021 at 12:44 AM Ryanne Dolan wrote: > Thanks Omnia. lgtm! > > Ryanne > > On Thu, Apr 29, 2021 at 10:50 AM Omnia Ibrahim > wrote: > >> I updated the KIP >> >> On

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Thanks Omnia. lgtm! Ryanne On Thu, Apr 29, 2021 at 10:50 AM Omnia Ibrahim wrote: > I updated the KIP > > On Thu, Apr 29, 2021 at 4:43 PM Omnia Ibrahim > wrote: > >> Sure, this would make it easier, we can make these functions returns the >> original behaviour (.checkpoints.internal, >> "mm2-of

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

I updated the KIP On Thu, Apr 29, 2021 at 4:43 PM Omnia Ibrahim wrote: > Sure, this would make it easier, we can make these functions returns the > original behaviour (.checkpoints.internal, > "mm2-offset-syncs..internal", heartbeat) without any > customisation using `replication.policy.separato

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Sure, this would make it easier, we can make these functions returns the original behaviour (.checkpoints.internal, "mm2-offset-syncs..internal", heartbeat) without any customisation using `replication.policy.separator` and use the separator in the DefaultReplicationPolicy On Wed, Apr 28, 2021 at

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Thanks Omnia, makes sense to me. > Customers who have their customised ReplicationPolicy will need to add the definition of their internal topics naming convention I wonder should we include default impls in the interface to avoid that requirement? Ryanne On Sun, Apr 25, 2021, 2:20 PM Omnia Ibr

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael and Ryanne, I updated the KIP to add these methods to the ReplicationPolicy instead of an extra interface to simplify the changes. Please have a look and let me know your thoughts. Thanks On Tue, Mar 30, 2021 at 7:19 PM Omnia Ibrahim wrote: > *(sorry forgot to Replay to All) * > Hi R

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

*(sorry forgot to Replay to All) * Hi Ryanne, It's a valid concern, I was trying to separate the concerns of internal and replicated policy away from each other and to make the code readable as extending ReplicationPolicy to manage both internal and replicated topic is a bit odd. Am not against sim

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael, > - I'm still a bit uneasy with the overlap between these 2 methods as currently `ReplicationPolicy.isInternalTopic` already handles MM2 internal topics. Should we make it only handle Kafka internal topics and `isMM2InternalTopic()` only handle MM2 topics? We can clear the overlap by ma

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Omnia, have we considered just adding methods to ReplicationPolicy? I'm reluctant to add a new class because, as Mickael points out, we'd need to carry it around in client code. Ryanne On Fri, Feb 19, 2021 at 8:31 AM Mickael Maison wrote: > Hi Omnia, > > Thanks for the clarifications. > > - I'm

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Omnia, Thanks for the clarifications. - I'm still a bit uneasy with the overlap between these 2 methods as currently `ReplicationPolicy.isInternalTopic` already handles MM2 internal topics. Should we make it only handle Kafka internal topics and `isMM2InternalTopic()` only handle MM2 topics?

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael, did you have some time to check my answer? On Thu, Jan 21, 2021 at 10:10 PM Omnia Ibrahim wrote: > Hi Mickael, > Thanks for taking another look into the KIP, regards your questions > > - I believe we need both "isMM2InternalTopic" and > `ReplicationPolicy.isInternalTopic` as `Replic

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael, Thanks for taking another look into the KIP, regards your questions - I believe we need both "isMM2InternalTopic" and `ReplicationPolicy.isInternalTopic` as `ReplicationPolicy.isInternalTopic` does check if a topic is Kafka internal topic, while `isMM2InternalTopic` is just focusing i

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Omnia, Thanks for the updates. Sorry for the delay but I have a few more small questions about the API: - Do we really need "isMM2InternalTopic()"? There's already "ReplicationPolicy.isInternalTopic()". If so, we need to explain the difference between these 2 methods. - Is "isCheckpointTopic()

Re: [DISCUSS] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Ryanne, Could you please vote for the KIP here https://www.mail-archive.com/dev@kafka.apache.org/msg113575.html if you are happy with the proposal? Thanks On Thu, Dec 10, 2020 at 8:51 AM Omnia Ibrahim wrote: > Thanks, Ryanne

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael, Did you get time to review the changes to the KIP? If you okay with it could you vote for the KIP here ttps://www.mail-archive.com/dev@kafka.apache.org/msg113575.html ? Thanks On Thu, Dec 10, 2020 at 2:19 PM Omnia Ibrahi

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael, 1) That's right the interface and default implementation will in mirror-connect 2) Renaming the interface should be fine too especially if you planning to move other functionality related to the creation there, I can edit this if you are okay with that please vote for the KIP here http

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Omnia, Thank you for the reply, it makes sense. A couple more comments: 1) I'm assuming the new interface and default implementation will be in the mirror-client project? as the names of some of these topics are needed by RemoteClusterUtils on the client-side. 2) I'm about to open a KIP to s

Re: [DISCUSS] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Thanks, Ryanne for feedback. Could you please vote on the voting thread https://www.mail-archive.com/dev@kafka.apache.org/msg113575.html thanks Omnia On Fri, Dec 4, 2020 at 4:53 PM Ryanne Dolan wrote: > Thanks Omnia, this looks great. I like the approach of introducing another > policy class. >

Re: [DISCUSS] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Thanks Omnia, this looks great. I like the approach of introducing another policy class. Ryanne On Tue, Dec 1, 2020, 9:36 AM Omnia Ibrahim wrote: > Hi everyone > I want to start discussion of the KIP 690, the proposal is here > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Mickael, Thanks for your feedback! Regards your question about having more configurations, I considered adding configuration per each topic however this meant adding more configurations for MM2 which already have so many, also the more complicated and advanced replication pattern you have betwee

Re: [DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi Omnia, Thanks for the KIP. Indeed being able to configure MM2's internal topic names would be a nice improvement. Looking at the KIP, I was surprised you propose an interface to allow users to specify names. Have you considered making names changeable via configurations? If so, we should defin

[DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hey everyone, Please take a look at KIP-690: https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention Thanks for your feedback and support. Omnia

[DISCUSS] KIP-690 Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hey everyone, Please take a look at KIP-690 https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention Thanks for the feedback and support Omnia

[DISCUSS] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi everyone I want to start discussion of the KIP 690, the proposal is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention Thanks for your time and feedback. Omnia

[DISCUSS] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Hi everyone I want to start discussion of the KIP 690, the proposal is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention Thanks for your time and feedback. Omnia